This message was deleted.

what’s your platform, cloud or on-prem? and what’s your firewall setup

You appear to be missing the token?

I used the same machine token and it moved a bit in the process but when i use the other machine token it tells me that the files are incepted with a different token, which it doesn’t make sense. sincerely, Abdullah Alfadhel

copy the the token value from the other server, and pass it as --token when joining this server to the cluster

this is what i did and it yeilds an error:

sudo rke2 server --server <https://172.16.1.11:6443> -t K108fe0d414ce6d89e53b2a188c4a1be2e9d6d293c2864b352b1e1d1c6af067c629::server:d099c75115ea14a14d5dc90bc8a5f011

and it gets stuck at the end and then i canceled it:

INFO[0028] Starting /v1, Kind=Secret controller         
INFO[0028] Updating TLS secret for kube-system/rke2-serving (count: 10): map[<http://listener.cattle.io/cn-10.43.0.1:10.43.0.1|listener.cattle.io/cn-10.43.0.1:10.43.0.1> <http://listener.cattle.io/cn-127.0.0.1:127.0.0.1|listener.cattle.io/cn-127.0.0.1:127.0.0.1> <http://listener.cattle.io/cn-172.16.1.12:172.16.1.12|listener.cattle.io/cn-172.16.1.12:172.16.1.12> <http://listener.cattle.io/cn-__1-f16284:::1|listener.cattle.io/cn-__1-f16284:::1> <http://listener.cattle.io/cn-kubernetes:kubernetes|listener.cattle.io/cn-kubernetes:kubernetes> <http://listener.cattle.io/cn-kubernetes.default:kubernetes.default|listener.cattle.io/cn-kubernetes.default:kubernetes.default> <http://listener.cattle.io/cn-kubernetes.default.svc:kubernetes.default.svc|listener.cattle.io/cn-kubernetes.default.svc:kubernetes.default.svc> <http://listener.cattle.io/cn-kubernetes.default.svc.cluster.local:kubernetes.default.svc.cluster.local|listener.cattle.io/cn-kubernetes.default.svc.cluster.local:kubernetes.default.svc.cluster.local> <http://listener.cattle.io/cn-localhost:localhost|listener.cattle.io/cn-localhost:localhost> <http://listener.cattle.io/cn-soss-m2:soss-m2|listener.cattle.io/cn-soss-m2:soss-m2> <http://listener.cattle.io/fingerprint:SHA1=EC20B204DC47FB082C468868CA1E45DC9D3148BC]|listener.cattle.io/fingerprint:SHA1=EC20B204DC47FB082C468868CA1E45DC9D3148BC]> 
INFO[0028] Running kube-proxy --cluster-cidr=10.42.0.0/16 --conntrack-max-per-core=0 --conntrack-tcp-timeout-close-wait=0s --conntrack-tcp-timeout-established=0s --healthz-bind-address=127.0.0.1 --hostname-override=soss-m2 --kubeconfig=/var/lib/rancher/rke2/agent/kubeproxy.kubeconfig --proxy-mode=iptables 
^CINFO[0933] Proxy done                                    err="context canceled" url="<wss://172.16.1.12:9345/v1-rke2/connect>"
INFO[0933] Shutting down <http://k3s.cattle.io/v1|k3s.cattle.io/v1>, Kind=Addon workers 
W0226 13:28:12.053517   77352 reflector.go:442] <http://k8s.io/client-go@v1.24.9-k3s1/tools/cache/reflector.go:167|k8s.io/client-go@v1.24.9-k3s1/tools/cache/reflector.go:167>: watch of *v1.Endpoints ended with: an error on the server ("unable to decode an event from the watch stream: context canceled") has prevented the request from succeeding
INFO[0933] Shutting down /v1, Kind=Secret workers       
INFO[0933] Shutting down /v1, Kind=Node workers         
FATA[0933] context canceled

copy the token from your first server node, and the rest server/agent nodes should use this token when provisioning the cluster

i did try that, but it didn't work

server port should be 9345 not 6443

ahh good catch, that is also the wrong port for rke2!

i did change the port and this is what happened:

sd@soss-m2:~$ sudo rke2 server --server <https://172.16.1.11:9345> --token K10a7523b5d56f0bdca1e3110e29d4d9f7e4ebde89d0e41cdf51a55ab54daaf23ea::server:fefc31e33c1fedafff7cdc962a9307ec
WARN[0000] not running in CIS mode                      
INFO[0000] Starting rke2 v1.24.10+rke2r1 (1ccdce2571291649b9414af1f269f645c3fe4002) 
INFO[0000] Managed etcd cluster bootstrap already complete and initialized 
INFO[0000] Starting temporary etcd to reconcile with datastore 
{"level":"info","ts":"2023-02-27T09:04:52.773Z","caller":"embed/etcd.go:131","msg":"configuring peer listeners","listen-peer-urls":["<http://127.0.0.1:2400>"]}
{"level":"info","ts":"2023-02-27T09:04:52.774Z","caller":"embed/etcd.go:139","msg":"configuring client listeners","listen-client-urls":["<http://127.0.0.1:2399>"]}
{"level":"info","ts":"2023-02-27T09:04:52.774Z","caller":"embed/etcd.go:308","msg":"starting an etcd server","etcd-version":"3.5.4","git-sha":"Not provided (use ./build instead of go build)","go-version":"go1.19.5 X:boringcrypto","go-os":"linux","go-arch":"amd64","max-cpu-set":8,"max-cpu-available":8,"member-initialized":true,"name":"soss-m2-6797f3ae","data-dir":"/var/lib/rancher/rke2/server/db/etcd-tmp","wal-dir":"","wal-dir-dedicated":"","member-dir":"/var/lib/rancher/rke2/server/db/etcd-tmp/member","force-new-cluster":true,"heartbeat-interval":"500ms","election-timeout":"5s","initial-election-tick-advance":true,"snapshot-count":10000,"snapshot-catchup-entries":5000,"initial-advertise-peer-urls":["<http://127.0.0.1:2400>"],"listen-peer-urls":["<http://127.0.0.1:2400>"],"advertise-client-urls":["<http://127.0.0.1:2399>"],"listen-client-urls":["<http://127.0.0.1:2399>"],"listen-metrics-urls":[],"cors":["*"],"host-whitelist":["*"],"initial-cluster":"","initial-cluster-state":"new","initial-cluster-token":"","quota-size-bytes":2147483648,"pre-vote":true,"initial-corrupt-check":true,"corrupt-check-time-interval":"0s","auto-compaction-mode":"","auto-compaction-retention":"0s","auto-compaction-interval":"0s","discovery-url":"","discovery-proxy":"","downgrade-check-interval":"5s"}
{"level":"info","ts":"2023-02-27T09:04:52.776Z","caller":"etcdserver/backend.go:81","msg":"opened backend db","path":"/var/lib/rancher/rke2/server/db/etcd-tmp/member/snap/db","took":"2.012679ms"}
{"level":"info","ts":"2023-02-27T09:04:53.294Z","caller":"etcdserver/server.go:508","msg":"recovered v2 store from snapshot","snapshot-index":140015,"snapshot-size":"7.1 kB"}
{"level":"info","ts":"2023-02-27T09:04:53.294Z","caller":"etcdserver/server.go:521","msg":"recovered v3 backend from snapshot","backend-size-bytes":3387392,"backend-size":"3.4 MB","backend-size-in-use-bytes":1757184,"backend-size-in-use":"1.8 MB"}
{"level":"info","ts":"2023-02-27T09:04:53.560Z","caller":"etcdserver/raft.go:556","msg":"forcing restart member","cluster-id":"74885ea7cc240892","local-member-id":"6ac10dc079d59789","commit-index":149285}
{"level":"info","ts":"2023-02-27T09:04:53.561Z","logger":"raft","caller":"etcdserver/zap_raft.go:77","msg":"6ac10dc079d59789 switched to configuration voters=(7692444758854440841)"}
{"level":"info","ts":"2023-02-27T09:04:53.561Z","logger":"raft","caller":"etcdserver/zap_raft.go:77","msg":"6ac10dc079d59789 became follower at term 2"}
{"level":"info","ts":"2023-02-27T09:04:53.561Z","logger":"raft","caller":"etcdserver/zap_raft.go:77","msg":"newRaft 6ac10dc079d59789 [peers: [6ac10dc079d59789], term: 2, commit: 149285, applied: 140015, lastindex: 149285, lastterm: 2]"}
{"level":"info","ts":"2023-02-27T09:04:53.561Z","caller":"api/capability.go:75","msg":"enabled capabilities for version","cluster-version":"3.5"}
{"level":"info","ts":"2023-02-27T09:04:53.561Z","caller":"membership/cluster.go:278","msg":"recovered/added member from store","cluster-id":"74885ea7cc240892","local-member-id":"6ac10dc079d59789","recovered-remote-peer-id":"6ac10dc079d59789","recovered-remote-peer-urls":["<https://172.16.1.12:2380>"]}
{"level":"info","ts":"2023-02-27T09:04:53.561Z","caller":"membership/cluster.go:287","msg":"set cluster version from store","cluster-version":"3.5"}
{"level":"warn","ts":"2023-02-27T09:04:53.562Z","caller":"auth/store.go:1220","msg":"simple token is not cryptographically signed"}
{"level":"info","ts":"2023-02-27T09:04:53.562Z","caller":"mvcc/kvstore.go:345","msg":"restored last compact revision","meta-bucket-name":"meta","meta-bucket-name-key":"finishedCompactRev","restored-compact-revision":133869}
{"level":"info","ts":"2023-02-27T09:04:53.570Z","caller":"mvcc/kvstore.go:415","msg":"kvstore restored","current-rev":134453}
{"level":"info","ts":"2023-02-27T09:04:53.571Z","caller":"etcdserver/quota.go:94","msg":"enabled backend quota with default value","quota-name":"v3-applier","quota-size-bytes":2147483648,"quota-size":"2.1 GB"}
{"level":"info","ts":"2023-02-27T09:04:53.571Z","caller":"etcdserver/corrupt.go:46","msg":"starting initial corruption check","local-member-id":"6ac10dc079d59789","timeout":"15s"}
{"level":"info","ts":"2023-02-27T09:04:53.572Z","caller":"etcdserver/corrupt.go:116","msg":"initial corruption checking passed; no corruption","local-member-id":"6ac10dc079d59789"}
{"level":"info","ts":"2023-02-27T09:04:53.572Z","caller":"etcdserver/server.go:842","msg":"starting etcd server","local-member-id":"6ac10dc079d59789","local-server-version":"3.5.4","cluster-id":"74885ea7cc240892","cluster-version":"3.5"}
{"level":"info","ts":"2023-02-27T09:04:53.572Z","caller":"etcdserver/server.go:736","msg":"started as single-node; fast-forwarding election ticks","local-member-id":"6ac10dc079d59789","forward-ticks":9,"forward-duration":"4.5s","election-ticks":10,"election-timeout":"5s"}
{"level":"info","ts":"2023-02-27T09:04:53.576Z","caller":"embed/etcd.go:277","msg":"now serving peer/client/metrics","local-member-id":"6ac10dc079d59789","initial-advertise-peer-urls":["<http://127.0.0.1:2400>"],"listen-peer-urls":["<http://127.0.0.1:2400>"],"advertise-client-urls":["<http://127.0.0.1:2399>"],"listen-client-urls":["<http://127.0.0.1:2399>"],"listen-metrics-urls":[]}
{"level":"info","ts":"2023-02-27T09:04:53.576Z","caller":"embed/etcd.go:581","msg":"serving peer traffic","address":"127.0.0.1:2400"}
{"level":"info","ts":"2023-02-27T09:04:53.577Z","caller":"embed/etcd.go:553","msg":"cmux::serve","address":"127.0.0.1:2400"}
{"level":"info","ts":"2023-02-27T09:04:54.562Z","logger":"raft","caller":"etcdserver/zap_raft.go:77","msg":"6ac10dc079d59789 is starting a new election at term 2"}
{"level":"info","ts":"2023-02-27T09:04:54.562Z","logger":"raft","caller":"etcdserver/zap_raft.go:77","msg":"6ac10dc079d59789 became pre-candidate at term 2"}
{"level":"info","ts":"2023-02-27T09:04:54.562Z","logger":"raft","caller":"etcdserver/zap_raft.go:77","msg":"6ac10dc079d59789 received MsgPreVoteResp from 6ac10dc079d59789 at term 2"}
{"level":"info","ts":"2023-02-27T09:04:54.562Z","logger":"raft","caller":"etcdserver/zap_raft.go:77","msg":"6ac10dc079d59789 became candidate at term 3"}
{"level":"info","ts":"2023-02-27T09:04:54.562Z","logger":"raft","caller":"etcdserver/zap_raft.go:77","msg":"6ac10dc079d59789 received MsgVoteResp from 6ac10dc079d59789 at term 3"}
{"level":"info","ts":"2023-02-27T09:04:54.562Z","logger":"raft","caller":"etcdserver/zap_raft.go:77","msg":"6ac10dc079d59789 became leader at term 3"}
{"level":"info","ts":"2023-02-27T09:04:54.562Z","logger":"raft","caller":"etcdserver/zap_raft.go:77","msg":"raft.node: 6ac10dc079d59789 elected leader 6ac10dc079d59789 at term 3"}
{"level":"info","ts":"2023-02-27T09:04:54.579Z","caller":"etcdserver/server.go:2044","msg":"published local member to cluster through raft","local-member-id":"6ac10dc079d59789","local-member-attributes":"{Name:soss-m2-6797f3ae ClientURLs:[<http://127.0.0.1:2399>]}","request-path":"/0/members/6ac10dc079d59789/attributes","cluster-id":"74885ea7cc240892","publish-timeout":"15s"}
{"level":"info","ts":"2023-02-27T09:04:54.580Z","caller":"embed/serve.go:98","msg":"ready to serve client requests"}
{"level":"info","ts":"2023-02-27T09:04:54.581Z","caller":"embed/serve.go:140","msg":"serving client traffic insecurely; this is strongly discouraged!","address":"127.0.0.1:2399"}
INFO[0002] Defragmenting etcd database                  
{"level":"info","ts":"2023-02-27T09:04:54.586Z","caller":"v3rpc/maintenance.go:89","msg":"starting defragment"}
{"level":"info","ts":"2023-02-27T09:04:54.587Z","caller":"backend/backend.go:497","msg":"defragmenting","path":"/var/lib/rancher/rke2/server/db/etcd-tmp/member/snap/db","current-db-size-bytes":3387392,"current-db-size":"3.4 MB","current-db-size-in-use-bytes":1761280,"current-db-size-in-use":"1.8 MB"}
{"level":"info","ts":"2023-02-27T09:04:54.614Z","caller":"backend/backend.go:549","msg":"finished defragmenting directory","path":"/var/lib/rancher/rke2/server/db/etcd-tmp/member/snap/db","current-db-size-bytes-diff":-1622016,"current-db-size-bytes":1765376,"current-db-size":"1.8 MB","current-db-size-in-use-bytes-diff":-4096,"current-db-size-in-use-bytes":1757184,"current-db-size-in-use":"1.8 MB","took":"27.629055ms"}
{"level":"info","ts":"2023-02-27T09:04:54.614Z","caller":"v3rpc/maintenance.go:95","msg":"finished defragment"}
INFO[0002] etcd temporary data store connection OK      
INFO[0002] Reconciling bootstrap data between datastore and disk 
FATA[0002] Failed to reconcile with temporary etcd: bootstrap data already found and encrypted with different token

Uninstall on this node and start over with the correct server URL and token. The data is out of sync between the two nodes in a way that is easiest to reconcile by just starting over.

I uninstalled using:

/usr/local/bin/rke2-uninstall.sh

and then used this command to install:

curl -sfL <https://get.rke2.io> |  sh -s - server --server=<https://172.16.1.11:9345> --token K10a7523b5d56f0bdca1e3110e29d4d9f7e4ebde89d0e41cdf51a55ab54daaf23ea::server:fefc31e33c1fedafff7cdc962a9307ec

then this command:

systemctl enable rke2-server.service

then this command:

systemctl start rke2-server.service

and then i noticed nothing coming to the first server so i used this command:

rke2 server --server=<https://172.16.1.11:9345> --token K10a7523b5d56f0bdca1e3110e29d4d9f7e4ebde89d0e41cdf51a55ab54daaf23ea::server:fefc31e33c1fedafff7cdc962a9307ec

it did contact the first server but finally i got the same result:

root@soss-m2:~# rke2 server --server=<https://172.16.1.11:9345> --token K10a7523b5d56f0bdca1e3110e29d4d9f7e4ebde89d0e41cdf51a55ab54daaf23ea::server:fefc31e33c1fedafff7cdc962a9307ec
WARN[0000] not running in CIS mode                      
INFO[0000] Starting rke2 v1.24.10+rke2r1 (1ccdce2571291649b9414af1f269f645c3fe4002) 
INFO[0000] Managed etcd cluster bootstrap already complete and initialized 
INFO[0000] Starting temporary etcd to reconcile with datastore 
{"level":"info","ts":"2023-02-27T10:04:49.815Z","caller":"embed/etcd.go:131","msg":"configuring peer listeners","listen-peer-urls":["<http://127.0.0.1:2400>"]}
{"level":"info","ts":"2023-02-27T10:04:49.815Z","caller":"embed/etcd.go:139","msg":"configuring client listeners","listen-client-urls":["<http://127.0.0.1:2399>"]}
{"level":"info","ts":"2023-02-27T10:04:49.815Z","caller":"embed/etcd.go:308","msg":"starting an etcd server","etcd-version":"3.5.4","git-sha":"Not provided (use ./build instead of go build)","go-version":"go1.19.5 X:boringcrypto","go-os":"linux","go-arch":"amd64","max-cpu-set":8,"max-cpu-available":8,"member-initialized":true,"name":"soss-m2-bf2134cb","data-dir":"/var/lib/rancher/rke2/server/db/etcd-tmp","wal-dir":"","wal-dir-dedicated":"","member-dir":"/var/lib/rancher/rke2/server/db/etcd-tmp/member","force-new-cluster":true,"heartbeat-interval":"500ms","election-timeout":"5s","initial-election-tick-advance":true,"snapshot-count":10000,"snapshot-catchup-entries":5000,"initial-advertise-peer-urls":["<http://127.0.0.1:2400>"],"listen-peer-urls":["<http://127.0.0.1:2400>"],"advertise-client-urls":["<http://127.0.0.1:2399>"],"listen-client-urls":["<http://127.0.0.1:2399>"],"listen-metrics-urls":[],"cors":["*"],"host-whitelist":["*"],"initial-cluster":"","initial-cluster-state":"new","initial-cluster-token":"","quota-size-bytes":2147483648,"pre-vote":true,"initial-corrupt-check":true,"corrupt-check-time-interval":"0s","auto-compaction-mode":"","auto-compaction-retention":"0s","auto-compaction-interval":"0s","discovery-url":"","discovery-proxy":"","downgrade-check-interval":"5s"}
{"level":"info","ts":"2023-02-27T10:04:49.817Z","caller":"etcdserver/backend.go:81","msg":"opened backend db","path":"/var/lib/rancher/rke2/server/db/etcd-tmp/member/snap/db","took":"1.10559ms"}
{"level":"info","ts":"2023-02-27T10:04:49.820Z","caller":"etcdserver/server.go:529","msg":"No snapshot found. Recovering WAL from scratch!"}
{"level":"info","ts":"2023-02-27T10:04:49.844Z","caller":"etcdserver/raft.go:556","msg":"forcing restart member","cluster-id":"74885ea7cc240892","local-member-id":"6ac10dc079d59789","commit-index":669}
{"level":"info","ts":"2023-02-27T10:04:49.844Z","logger":"raft","caller":"etcdserver/zap_raft.go:77","msg":"6ac10dc079d59789 switched to configuration voters=()"}
{"level":"info","ts":"2023-02-27T10:04:49.844Z","logger":"raft","caller":"etcdserver/zap_raft.go:77","msg":"6ac10dc079d59789 became follower at term 2"}
{"level":"info","ts":"2023-02-27T10:04:49.844Z","logger":"raft","caller":"etcdserver/zap_raft.go:77","msg":"newRaft 6ac10dc079d59789 [peers: [], term: 2, commit: 669, applied: 0, lastindex: 669, lastterm: 2]"}
{"level":"warn","ts":"2023-02-27T10:04:49.847Z","caller":"auth/store.go:1220","msg":"simple token is not cryptographically signed"}
{"level":"info","ts":"2023-02-27T10:04:49.849Z","caller":"mvcc/kvstore.go:415","msg":"kvstore restored","current-rev":647}
{"level":"info","ts":"2023-02-27T10:04:49.849Z","caller":"etcdserver/quota.go:94","msg":"enabled backend quota with default value","quota-name":"v3-applier","quota-size-bytes":2147483648,"quota-size":"2.1 GB"}
{"level":"info","ts":"2023-02-27T10:04:49.849Z","caller":"etcdserver/corrupt.go:46","msg":"starting initial corruption check","local-member-id":"6ac10dc079d59789","timeout":"15s"}
{"level":"info","ts":"2023-02-27T10:04:49.850Z","caller":"etcdserver/corrupt.go:116","msg":"initial corruption checking passed; no corruption","local-member-id":"6ac10dc079d59789"}
{"level":"info","ts":"2023-02-27T10:04:49.850Z","caller":"etcdserver/server.go:851","msg":"starting etcd server","local-member-id":"6ac10dc079d59789","local-server-version":"3.5.4","cluster-version":"to_be_decided"}
{"level":"info","ts":"2023-02-27T10:04:49.850Z","caller":"etcdserver/server.go:752","msg":"starting initial election tick advance","election-ticks":10}
{"level":"info","ts":"2023-02-27T10:04:49.850Z","logger":"raft","caller":"etcdserver/zap_raft.go:77","msg":"6ac10dc079d59789 switched to configuration voters=(7692444758854440841)"}
{"level":"info","ts":"2023-02-27T10:04:49.850Z","caller":"membership/cluster.go:421","msg":"added member","cluster-id":"74885ea7cc240892","local-member-id":"6ac10dc079d59789","added-peer-id":"6ac10dc079d59789","added-peer-peer-urls":["<https://172.16.1.12:2380>"]}
{"level":"info","ts":"2023-02-27T10:04:49.850Z","caller":"membership/cluster.go:584","msg":"set initial cluster version","cluster-id":"74885ea7cc240892","local-member-id":"6ac10dc079d59789","cluster-version":"3.5"}
{"level":"info","ts":"2023-02-27T10:04:49.851Z","caller":"api/capability.go:75","msg":"enabled capabilities for version","cluster-version":"3.5"}
{"level":"info","ts":"2023-02-27T10:04:49.852Z","caller":"embed/etcd.go:277","msg":"now serving peer/client/metrics","local-member-id":"6ac10dc079d59789","initial-advertise-peer-urls":["<http://127.0.0.1:2400>"],"listen-peer-urls":["<http://127.0.0.1:2400>"],"advertise-client-urls":["<http://127.0.0.1:2399>"],"listen-client-urls":["<http://127.0.0.1:2399>"],"listen-metrics-urls":[]}
{"level":"info","ts":"2023-02-27T10:04:49.852Z","caller":"embed/etcd.go:581","msg":"serving peer traffic","address":"127.0.0.1:2400"}
{"level":"info","ts":"2023-02-27T10:04:49.852Z","caller":"embed/etcd.go:553","msg":"cmux::serve","address":"127.0.0.1:2400"}
{"level":"info","ts":"2023-02-27T10:04:54.846Z","logger":"raft","caller":"etcdserver/zap_raft.go:77","msg":"6ac10dc079d59789 is starting a new election at term 2"}
{"level":"info","ts":"2023-02-27T10:04:54.846Z","logger":"raft","caller":"etcdserver/zap_raft.go:77","msg":"6ac10dc079d59789 became pre-candidate at term 2"}
{"level":"info","ts":"2023-02-27T10:04:54.846Z","logger":"raft","caller":"etcdserver/zap_raft.go:77","msg":"6ac10dc079d59789 received MsgPreVoteResp from 6ac10dc079d59789 at term 2"}
{"level":"info","ts":"2023-02-27T10:04:54.846Z","logger":"raft","caller":"etcdserver/zap_raft.go:77","msg":"6ac10dc079d59789 became candidate at term 3"}
{"level":"info","ts":"2023-02-27T10:04:54.846Z","logger":"raft","caller":"etcdserver/zap_raft.go:77","msg":"6ac10dc079d59789 received MsgVoteResp from 6ac10dc079d59789 at term 3"}
{"level":"info","ts":"2023-02-27T10:04:54.846Z","logger":"raft","caller":"etcdserver/zap_raft.go:77","msg":"6ac10dc079d59789 became leader at term 3"}
{"level":"info","ts":"2023-02-27T10:04:54.846Z","logger":"raft","caller":"etcdserver/zap_raft.go:77","msg":"raft.node: 6ac10dc079d59789 elected leader 6ac10dc079d59789 at term 3"}
{"level":"info","ts":"2023-02-27T10:04:54.848Z","caller":"etcdserver/server.go:2044","msg":"published local member to cluster through raft","local-member-id":"6ac10dc079d59789","local-member-attributes":"{Name:soss-m2-bf2134cb ClientURLs:[<http://127.0.0.1:2399>]}","request-path":"/0/members/6ac10dc079d59789/attributes","cluster-id":"74885ea7cc240892","publish-timeout":"15s"}
{"level":"info","ts":"2023-02-27T10:04:54.848Z","caller":"embed/serve.go:98","msg":"ready to serve client requests"}
{"level":"info","ts":"2023-02-27T10:04:54.850Z","caller":"embed/serve.go:140","msg":"serving client traffic insecurely; this is strongly discouraged!","address":"127.0.0.1:2399"}
INFO[0005] Defragmenting etcd database                  
{"level":"info","ts":"2023-02-27T10:04:54.858Z","caller":"v3rpc/maintenance.go:89","msg":"starting defragment"}
{"level":"info","ts":"2023-02-27T10:04:54.859Z","caller":"backend/backend.go:497","msg":"defragmenting","path":"/var/lib/rancher/rke2/server/db/etcd-tmp/member/snap/db","current-db-size-bytes":1708032,"current-db-size":"1.7 MB","current-db-size-in-use-bytes":1695744,"current-db-size-in-use":"1.7 MB"}
{"level":"info","ts":"2023-02-27T10:04:54.890Z","caller":"backend/backend.go:549","msg":"finished defragmenting directory","path":"/var/lib/rancher/rke2/server/db/etcd-tmp/member/snap/db","current-db-size-bytes-diff":-65536,"current-db-size-bytes":1642496,"current-db-size":"1.6 MB","current-db-size-in-use-bytes-diff":-61440,"current-db-size-in-use-bytes":1634304,"current-db-size-in-use":"1.6 MB","took":"32.07575ms"}
{"level":"info","ts":"2023-02-27T10:04:54.890Z","caller":"v3rpc/maintenance.go:95","msg":"finished defragment"}
INFO[0005] etcd temporary data store connection OK      
INFO[0005] Reconciling bootstrap data between datastore and disk 
FATA[0005] Failed to reconcile with temporary etcd: bootstrap data already found and encrypted with different token

Hmm.. I think you should uninstall and retry again. Instead of adding the args with the install script, you should either create a file at

/etc/rancher/rke2/config.yaml

with the information and run it via systemctl, or just run via the binary. As for the reason as to why all of this is happening: Adding to the install script won’t actually do much for you since it just pulls the tarball and doesn’t run the service. Also, in your example above, what happened there is that you started rke2 (without server/token args since you didn’t have a config file). That then started it on this node as a fresh cluster. Then you tried to run it again on the same node (

rke2 server …

) which would fail regardless due to port conflicts, but in this case actually pointing it to your other server node, so now it doesn’t know which etcd datastore you want it to use. The best thing would be to follow the docs explicitly if you’re getting tripped up: https://docs.rke2.io/install/ha

ok, here is a fresh installation: first command

sudo nano /etc/rancher/rke2/config.yml

and inside it is:

server: <https://192.168.3.180:9345>
token: rke2-demo

then

sudo -i

then

curl -sfL <https://get.rke2.io> | sh -

then

systemctl enable rke2-server.service

then

systemctl start rke2-server.service

the installation is finished and there is nothing came to master1:

test@master1:~$ sudo tcpdump src 192.168.3.176
tcpdump: verbose output suppressed, use -v[v]... for full protocol decode
listening on ens160, link-type EN10MB (Ethernet), snapshot length 262144 bytes

then i tried to run this command:

rke2 server

and the response was:

root@test:~# rke2 server
WARN[0000] not running in CIS mode                      
INFO[0000] Starting rke2 v1.24.10+rke2r1 (1ccdce2571291649b9414af1f269f645c3fe4002) 
INFO[0000] Managed etcd cluster bootstrap already complete and initialized 
INFO[0000] Starting temporary etcd to reconcile with datastore 
{"level":"info","ts":"2023-02-28T05:08:35.112Z","caller":"embed/etcd.go:131","msg":"configuring peer listeners","listen-peer-urls":["<http://127.0.0.1:2400>"]}
{"level":"info","ts":"2023-02-28T05:08:35.112Z","caller":"embed/etcd.go:139","msg":"configuring client listeners","listen-client-urls":["<http://127.0.0.1:2399>"]}
{"level":"info","ts":"2023-02-28T05:08:35.113Z","caller":"embed/etcd.go:308","msg":"starting an etcd server","etcd-version":"3.5.4","git-sha":"Not provided (use ./build instead of go build)","go-version":"go1.19.5 X:boringcrypto","go-os":"linux","go-arch":"amd64","max-cpu-set":4,"max-cpu-available":4,"member-initialized":true,"name":"test-0010ecc4","data-dir":"/var/lib/rancher/rke2/server/db/etcd-tmp","wal-dir":"","wal-dir-dedicated":"","member-dir":"/var/lib/rancher/rke2/server/db/etcd-tmp/member","force-new-cluster":true,"heartbeat-interval":"500ms","election-timeout":"5s","initial-election-tick-advance":true,"snapshot-count":10000,"snapshot-catchup-entries":5000,"initial-advertise-peer-urls":["<http://127.0.0.1:2400>"],"listen-peer-urls":["<http://127.0.0.1:2400>"],"advertise-client-urls":["<http://127.0.0.1:2399>"],"listen-client-urls":["<http://127.0.0.1:2399>"],"listen-metrics-urls":[],"cors":["*"],"host-whitelist":["*"],"initial-cluster":"","initial-cluster-state":"new","initial-cluster-token":"","quota-size-bytes":2147483648,"pre-vote":true,"initial-corrupt-check":true,"corrupt-check-time-interval":"0s","auto-compaction-mode":"","auto-compaction-retention":"0s","auto-compaction-interval":"0s","discovery-url":"","discovery-proxy":"","downgrade-check-interval":"5s"}
{"level":"info","ts":"2023-02-28T05:08:35.116Z","caller":"etcdserver/backend.go:81","msg":"opened backend db","path":"/var/lib/rancher/rke2/server/db/etcd-tmp/member/snap/db","took":"2.726128ms"}
{"level":"info","ts":"2023-02-28T05:08:35.130Z","caller":"etcdserver/server.go:529","msg":"No snapshot found. Recovering WAL from scratch!"}
{"level":"info","ts":"2023-02-28T05:08:35.167Z","caller":"etcdserver/raft.go:556","msg":"forcing restart member","cluster-id":"8f86cad4ee5c8993","local-member-id":"8b8e4494dd110009","commit-index":887}
{"level":"info","ts":"2023-02-28T05:08:35.168Z","logger":"raft","caller":"etcdserver/zap_raft.go:77","msg":"8b8e4494dd110009 switched to configuration voters=()"}
{"level":"info","ts":"2023-02-28T05:08:35.168Z","logger":"raft","caller":"etcdserver/zap_raft.go:77","msg":"8b8e4494dd110009 became follower at term 2"}
{"level":"info","ts":"2023-02-28T05:08:35.168Z","logger":"raft","caller":"etcdserver/zap_raft.go:77","msg":"newRaft 8b8e4494dd110009 [peers: [], term: 2, commit: 887, applied: 0, lastindex: 887, lastterm: 2]"}
{"level":"warn","ts":"2023-02-28T05:08:36.720Z","caller":"auth/store.go:1220","msg":"simple token is not cryptographically signed"}
{"level":"info","ts":"2023-02-28T05:08:36.736Z","caller":"mvcc/kvstore.go:415","msg":"kvstore restored","current-rev":847}
{"level":"info","ts":"2023-02-28T05:08:36.805Z","caller":"etcdserver/quota.go:94","msg":"enabled backend quota with default value","quota-name":"v3-applier","quota-size-bytes":2147483648,"quota-size":"2.1 GB"}
{"level":"info","ts":"2023-02-28T05:08:36.818Z","caller":"etcdserver/corrupt.go:46","msg":"starting initial corruption check","local-member-id":"8b8e4494dd110009","timeout":"15s"}
{"level":"info","ts":"2023-02-28T05:08:36.820Z","caller":"etcdserver/corrupt.go:116","msg":"initial corruption checking passed; no corruption","local-member-id":"8b8e4494dd110009"}
{"level":"info","ts":"2023-02-28T05:08:36.820Z","caller":"etcdserver/server.go:851","msg":"starting etcd server","local-member-id":"8b8e4494dd110009","local-server-version":"3.5.4","cluster-version":"to_be_decided"}
{"level":"info","ts":"2023-02-28T05:08:36.820Z","caller":"etcdserver/server.go:752","msg":"starting initial election tick advance","election-ticks":10}
{"level":"info","ts":"2023-02-28T05:08:36.821Z","logger":"raft","caller":"etcdserver/zap_raft.go:77","msg":"8b8e4494dd110009 switched to configuration voters=(10056050424119623689)"}
{"level":"info","ts":"2023-02-28T05:08:36.821Z","caller":"membership/cluster.go:421","msg":"added member","cluster-id":"8f86cad4ee5c8993","local-member-id":"8b8e4494dd110009","added-peer-id":"8b8e4494dd110009","added-peer-peer-urls":["<https://192.168.3.176:2380>"]}
{"level":"info","ts":"2023-02-28T05:08:36.821Z","caller":"membership/cluster.go:584","msg":"set initial cluster version","cluster-id":"8f86cad4ee5c8993","local-member-id":"8b8e4494dd110009","cluster-version":"3.5"}
{"level":"info","ts":"2023-02-28T05:08:36.821Z","caller":"api/capability.go:75","msg":"enabled capabilities for version","cluster-version":"3.5"}
{"level":"info","ts":"2023-02-28T05:08:36.825Z","caller":"embed/etcd.go:581","msg":"serving peer traffic","address":"127.0.0.1:2400"}
{"level":"info","ts":"2023-02-28T05:08:36.825Z","caller":"embed/etcd.go:553","msg":"cmux::serve","address":"127.0.0.1:2400"}
{"level":"info","ts":"2023-02-28T05:08:36.825Z","caller":"embed/etcd.go:277","msg":"now serving peer/client/metrics","local-member-id":"8b8e4494dd110009","initial-advertise-peer-urls":["<http://127.0.0.1:2400>"],"listen-peer-urls":["<http://127.0.0.1:2400>"],"advertise-client-urls":["<http://127.0.0.1:2399>"],"listen-client-urls":["<http://127.0.0.1:2399>"],"listen-metrics-urls":[]}
{"level":"info","ts":"2023-02-28T05:08:42.669Z","logger":"raft","caller":"etcdserver/zap_raft.go:77","msg":"8b8e4494dd110009 is starting a new election at term 2"}
{"level":"info","ts":"2023-02-28T05:08:42.669Z","logger":"raft","caller":"etcdserver/zap_raft.go:77","msg":"8b8e4494dd110009 became pre-candidate at term 2"}
{"level":"info","ts":"2023-02-28T05:08:42.669Z","logger":"raft","caller":"etcdserver/zap_raft.go:77","msg":"8b8e4494dd110009 received MsgPreVoteResp from 8b8e4494dd110009 at term 2"}
{"level":"info","ts":"2023-02-28T05:08:42.669Z","logger":"raft","caller":"etcdserver/zap_raft.go:77","msg":"8b8e4494dd110009 became candidate at term 3"}
{"level":"info","ts":"2023-02-28T05:08:42.669Z","logger":"raft","caller":"etcdserver/zap_raft.go:77","msg":"8b8e4494dd110009 received MsgVoteResp from 8b8e4494dd110009 at term 3"}
{"level":"info","ts":"2023-02-28T05:08:42.669Z","logger":"raft","caller":"etcdserver/zap_raft.go:77","msg":"8b8e4494dd110009 became leader at term 3"}
{"level":"info","ts":"2023-02-28T05:08:42.669Z","logger":"raft","caller":"etcdserver/zap_raft.go:77","msg":"raft.node: 8b8e4494dd110009 elected leader 8b8e4494dd110009 at term 3"}
{"level":"info","ts":"2023-02-28T05:08:42.670Z","caller":"embed/serve.go:98","msg":"ready to serve client requests"}
{"level":"info","ts":"2023-02-28T05:08:42.670Z","caller":"etcdserver/server.go:2044","msg":"published local member to cluster through raft","local-member-id":"8b8e4494dd110009","local-member-attributes":"{Name:test-0010ecc4 ClientURLs:[<http://127.0.0.1:2399>]}","request-path":"/0/members/8b8e4494dd110009/attributes","cluster-id":"8f86cad4ee5c8993","publish-timeout":"15s"}
{"level":"info","ts":"2023-02-28T05:08:42.671Z","caller":"embed/serve.go:140","msg":"serving client traffic insecurely; this is strongly discouraged!","address":"127.0.0.1:2399"}
INFO[0008] Defragmenting etcd database                  
{"level":"info","ts":"2023-02-28T05:08:42.678Z","caller":"v3rpc/maintenance.go:89","msg":"starting defragment"}
{"level":"info","ts":"2023-02-28T05:08:43.837Z","caller":"backend/backend.go:497","msg":"defragmenting","path":"/var/lib/rancher/rke2/server/db/etcd-tmp/member/snap/db","current-db-size-bytes":3182592,"current-db-size":"3.2 MB","current-db-size-in-use-bytes":3182592,"current-db-size-in-use":"3.2 MB"}
{"level":"info","ts":"2023-02-28T05:08:45.285Z","caller":"backend/backend.go:549","msg":"finished defragmenting directory","path":"/var/lib/rancher/rke2/server/db/etcd-tmp/member/snap/db","current-db-size-bytes-diff":-77824,"current-db-size-bytes":3104768,"current-db-size":"3.1 MB","current-db-size-in-use-bytes-diff":-86016,"current-db-size-in-use-bytes":3096576,"current-db-size-in-use":"3.1 MB","took":"2.606279189s"}
{"level":"info","ts":"2023-02-28T05:08:45.285Z","caller":"v3rpc/maintenance.go:95","msg":"finished defragment"}
{"level":"warn","ts":"2023-02-28T05:08:45.285Z","caller":"v3rpc/interceptor.go:197","msg":"request stats","start time":"2023-02-28T05:08:42.678Z","time spent":"2.607225471s","remote":"127.0.0.1:49848","response type":"/etcdserverpb.Maintenance/Defragment","request count":-1,"request size":-1,"response count":-1,"response size":-1,"request content":""}
{"level":"info","ts":"2023-02-28T05:08:45.491Z","caller":"traceutil/trace.go:171","msg":"trace[1862721267] linearizableReadLoop","detail":"{readStateIndex:890; appliedIndex:890; }","duration":"200.915946ms","start":"2023-02-28T05:08:45.290Z","end":"2023-02-28T05:08:45.491Z","steps":["trace[1862721267] 'read index received'  (duration: 200.907674ms)","trace[1862721267] 'applied index is now lower than readState.Index'  (duration: 6.043µs)"],"step_count":2}
INFO[0011] etcd temporary data store connection OK      
INFO[0011] Reconciling bootstrap data between datastore and disk 
INFO[0011] stopping etcd                                
{"level":"info","ts":"2023-02-28T05:08:45.554Z","caller":"embed/etcd.go:368","msg":"closing etcd server","name":"test-0010ecc4","data-dir":"/var/lib/rancher/rke2/server/db/etcd-tmp","advertise-peer-urls":["<http://127.0.0.1:2400>"],"advertise-client-urls":["<http://127.0.0.1:2399>"]}
{"level":"info","ts":"2023-02-28T05:08:45.554Z","caller":"etcdserver/server.go:1453","msg":"skipped leadership transfer for single voting member cluster","local-member-id":"8b8e4494dd110009","current-leader-member-id":"8b8e4494dd110009"}
FATA[0011] starting kubernetes: preparing server: init cluster datastore and https: listen tcp :9345: bind: address already in use

then i ran the command with the right flags:

rke2 server --server=<https://192.168.3.180:9345> --token=rke2-demo

and i got the error of the token:

FATA[0021] Failed to reconcile with temporary etcd: bootstrap data already found and encrypted with different token

i thought maybe that the error from my token, so i used the generated token:

rke2 server --server=<https://192.168.3.180:9345> --token=K10184dcfa4851e487eaa1756e8bedd18f03cb8c6b204dccf2c27090ba9e699cab5::server:1461a0c608734bad8e13b92971e1e09b

and i got the error of new certificates:

FATA[0000]newer than datastore and could cause a cluster outage. Remove the file(s) from disk and restart to be recreated from datastore.

so i deleted them, and tried again, and this is the error that i got:

FATA[0000] starting kubernetes: preparing server: init cluster datastore and https: listen tcp :9345: bind: address already in use

looks like you have systemd

rke2-server

running, and you are trying to start a new process with

rke2 server

using the same listening port

but if i didn't do that, nothing happens. No what so ever of contact came from master2 to master1 during the installation processes.