https://rancher.com/ logo
#rke2
Title
# rke2
a

adamant-kite-43734

02/26/2023, 11:55 AM
This message was deleted.
a

abundant-noon-17295

02/26/2023, 1:39 PM
what’s your platform, cloud or on-prem? and what’s your firewall setup
c

creamy-pencil-82913

02/26/2023, 7:36 PM
You appear to be missing the token?
w

worried-ram-81084

02/26/2023, 8:18 PM
I used the same machine token and it moved a bit in the process but when i use the other machine token it tells me that the files are incepted with a different token, which it doesn’t make sense. sincerely, Abdullah Alfadhel
c

creamy-pencil-82913

02/26/2023, 8:38 PM
copy the the token value from the other server, and pass it as --token when joining this server to the cluster
w

worried-ram-81084

02/27/2023, 5:29 AM
this is what i did and it yeilds an error:
Copy code
sudo rke2 server --server <https://172.16.1.11:6443> -t K108fe0d414ce6d89e53b2a188c4a1be2e9d6d293c2864b352b1e1d1c6af067c629::server:d099c75115ea14a14d5dc90bc8a5f011
and it gets stuck at the end and then i canceled it:
Copy code
INFO[0028] Starting /v1, Kind=Secret controller         
INFO[0028] Updating TLS secret for kube-system/rke2-serving (count: 10): map[<http://listener.cattle.io/cn-10.43.0.1:10.43.0.1|listener.cattle.io/cn-10.43.0.1:10.43.0.1> <http://listener.cattle.io/cn-127.0.0.1:127.0.0.1|listener.cattle.io/cn-127.0.0.1:127.0.0.1> <http://listener.cattle.io/cn-172.16.1.12:172.16.1.12|listener.cattle.io/cn-172.16.1.12:172.16.1.12> <http://listener.cattle.io/cn-__1-f16284:::1|listener.cattle.io/cn-__1-f16284:::1> <http://listener.cattle.io/cn-kubernetes:kubernetes|listener.cattle.io/cn-kubernetes:kubernetes> <http://listener.cattle.io/cn-kubernetes.default:kubernetes.default|listener.cattle.io/cn-kubernetes.default:kubernetes.default> <http://listener.cattle.io/cn-kubernetes.default.svc:kubernetes.default.svc|listener.cattle.io/cn-kubernetes.default.svc:kubernetes.default.svc> <http://listener.cattle.io/cn-kubernetes.default.svc.cluster.local:kubernetes.default.svc.cluster.local|listener.cattle.io/cn-kubernetes.default.svc.cluster.local:kubernetes.default.svc.cluster.local> <http://listener.cattle.io/cn-localhost:localhost|listener.cattle.io/cn-localhost:localhost> <http://listener.cattle.io/cn-soss-m2:soss-m2|listener.cattle.io/cn-soss-m2:soss-m2> <http://listener.cattle.io/fingerprint:SHA1=EC20B204DC47FB082C468868CA1E45DC9D3148BC]|listener.cattle.io/fingerprint:SHA1=EC20B204DC47FB082C468868CA1E45DC9D3148BC]> 
INFO[0028] Running kube-proxy --cluster-cidr=10.42.0.0/16 --conntrack-max-per-core=0 --conntrack-tcp-timeout-close-wait=0s --conntrack-tcp-timeout-established=0s --healthz-bind-address=127.0.0.1 --hostname-override=soss-m2 --kubeconfig=/var/lib/rancher/rke2/agent/kubeproxy.kubeconfig --proxy-mode=iptables 
^CINFO[0933] Proxy done                                    err="context canceled" url="<wss://172.16.1.12:9345/v1-rke2/connect>"
INFO[0933] Shutting down <http://k3s.cattle.io/v1|k3s.cattle.io/v1>, Kind=Addon workers 
W0226 13:28:12.053517   77352 reflector.go:442] <http://k8s.io/client-go@v1.24.9-k3s1/tools/cache/reflector.go:167|k8s.io/client-go@v1.24.9-k3s1/tools/cache/reflector.go:167>: watch of *v1.Endpoints ended with: an error on the server ("unable to decode an event from the watch stream: context canceled") has prevented the request from succeeding
INFO[0933] Shutting down /v1, Kind=Secret workers       
INFO[0933] Shutting down /v1, Kind=Node workers         
FATA[0933] context canceled
a

abundant-noon-17295

02/27/2023, 5:31 AM
copy the token from your first server node, and the rest server/agent nodes should use this token when provisioning the cluster
w

worried-ram-81084

02/27/2023, 5:32 AM
i did try that, but it didn't work
a

abundant-noon-17295

02/27/2023, 5:36 AM
server port should be 9345 not 6443
☝️ 1
c

creamy-pencil-82913

02/27/2023, 6:34 AM
ahh good catch, that is also the wrong port for rke2!
w

worried-ram-81084

02/27/2023, 9:05 AM
i did change the port and this is what happened:
Copy code
sd@soss-m2:~$ sudo rke2 server --server <https://172.16.1.11:9345> --token K10a7523b5d56f0bdca1e3110e29d4d9f7e4ebde89d0e41cdf51a55ab54daaf23ea::server:fefc31e33c1fedafff7cdc962a9307ec
WARN[0000] not running in CIS mode                      
INFO[0000] Starting rke2 v1.24.10+rke2r1 (1ccdce2571291649b9414af1f269f645c3fe4002) 
INFO[0000] Managed etcd cluster bootstrap already complete and initialized 
INFO[0000] Starting temporary etcd to reconcile with datastore 
{"level":"info","ts":"2023-02-27T09:04:52.773Z","caller":"embed/etcd.go:131","msg":"configuring peer listeners","listen-peer-urls":["<http://127.0.0.1:2400>"]}
{"level":"info","ts":"2023-02-27T09:04:52.774Z","caller":"embed/etcd.go:139","msg":"configuring client listeners","listen-client-urls":["<http://127.0.0.1:2399>"]}
{"level":"info","ts":"2023-02-27T09:04:52.774Z","caller":"embed/etcd.go:308","msg":"starting an etcd server","etcd-version":"3.5.4","git-sha":"Not provided (use ./build instead of go build)","go-version":"go1.19.5 X:boringcrypto","go-os":"linux","go-arch":"amd64","max-cpu-set":8,"max-cpu-available":8,"member-initialized":true,"name":"soss-m2-6797f3ae","data-dir":"/var/lib/rancher/rke2/server/db/etcd-tmp","wal-dir":"","wal-dir-dedicated":"","member-dir":"/var/lib/rancher/rke2/server/db/etcd-tmp/member","force-new-cluster":true,"heartbeat-interval":"500ms","election-timeout":"5s","initial-election-tick-advance":true,"snapshot-count":10000,"snapshot-catchup-entries":5000,"initial-advertise-peer-urls":["<http://127.0.0.1:2400>"],"listen-peer-urls":["<http://127.0.0.1:2400>"],"advertise-client-urls":["<http://127.0.0.1:2399>"],"listen-client-urls":["<http://127.0.0.1:2399>"],"listen-metrics-urls":[],"cors":["*"],"host-whitelist":["*"],"initial-cluster":"","initial-cluster-state":"new","initial-cluster-token":"","quota-size-bytes":2147483648,"pre-vote":true,"initial-corrupt-check":true,"corrupt-check-time-interval":"0s","auto-compaction-mode":"","auto-compaction-retention":"0s","auto-compaction-interval":"0s","discovery-url":"","discovery-proxy":"","downgrade-check-interval":"5s"}
{"level":"info","ts":"2023-02-27T09:04:52.776Z","caller":"etcdserver/backend.go:81","msg":"opened backend db","path":"/var/lib/rancher/rke2/server/db/etcd-tmp/member/snap/db","took":"2.012679ms"}
{"level":"info","ts":"2023-02-27T09:04:53.294Z","caller":"etcdserver/server.go:508","msg":"recovered v2 store from snapshot","snapshot-index":140015,"snapshot-size":"7.1 kB"}
{"level":"info","ts":"2023-02-27T09:04:53.294Z","caller":"etcdserver/server.go:521","msg":"recovered v3 backend from snapshot","backend-size-bytes":3387392,"backend-size":"3.4 MB","backend-size-in-use-bytes":1757184,"backend-size-in-use":"1.8 MB"}
{"level":"info","ts":"2023-02-27T09:04:53.560Z","caller":"etcdserver/raft.go:556","msg":"forcing restart member","cluster-id":"74885ea7cc240892","local-member-id":"6ac10dc079d59789","commit-index":149285}
{"level":"info","ts":"2023-02-27T09:04:53.561Z","logger":"raft","caller":"etcdserver/zap_raft.go:77","msg":"6ac10dc079d59789 switched to configuration voters=(7692444758854440841)"}
{"level":"info","ts":"2023-02-27T09:04:53.561Z","logger":"raft","caller":"etcdserver/zap_raft.go:77","msg":"6ac10dc079d59789 became follower at term 2"}
{"level":"info","ts":"2023-02-27T09:04:53.561Z","logger":"raft","caller":"etcdserver/zap_raft.go:77","msg":"newRaft 6ac10dc079d59789 [peers: [6ac10dc079d59789], term: 2, commit: 149285, applied: 140015, lastindex: 149285, lastterm: 2]"}
{"level":"info","ts":"2023-02-27T09:04:53.561Z","caller":"api/capability.go:75","msg":"enabled capabilities for version","cluster-version":"3.5"}
{"level":"info","ts":"2023-02-27T09:04:53.561Z","caller":"membership/cluster.go:278","msg":"recovered/added member from store","cluster-id":"74885ea7cc240892","local-member-id":"6ac10dc079d59789","recovered-remote-peer-id":"6ac10dc079d59789","recovered-remote-peer-urls":["<https://172.16.1.12:2380>"]}
{"level":"info","ts":"2023-02-27T09:04:53.561Z","caller":"membership/cluster.go:287","msg":"set cluster version from store","cluster-version":"3.5"}
{"level":"warn","ts":"2023-02-27T09:04:53.562Z","caller":"auth/store.go:1220","msg":"simple token is not cryptographically signed"}
{"level":"info","ts":"2023-02-27T09:04:53.562Z","caller":"mvcc/kvstore.go:345","msg":"restored last compact revision","meta-bucket-name":"meta","meta-bucket-name-key":"finishedCompactRev","restored-compact-revision":133869}
{"level":"info","ts":"2023-02-27T09:04:53.570Z","caller":"mvcc/kvstore.go:415","msg":"kvstore restored","current-rev":134453}
{"level":"info","ts":"2023-02-27T09:04:53.571Z","caller":"etcdserver/quota.go:94","msg":"enabled backend quota with default value","quota-name":"v3-applier","quota-size-bytes":2147483648,"quota-size":"2.1 GB"}
{"level":"info","ts":"2023-02-27T09:04:53.571Z","caller":"etcdserver/corrupt.go:46","msg":"starting initial corruption check","local-member-id":"6ac10dc079d59789","timeout":"15s"}
{"level":"info","ts":"2023-02-27T09:04:53.572Z","caller":"etcdserver/corrupt.go:116","msg":"initial corruption checking passed; no corruption","local-member-id":"6ac10dc079d59789"}
{"level":"info","ts":"2023-02-27T09:04:53.572Z","caller":"etcdserver/server.go:842","msg":"starting etcd server","local-member-id":"6ac10dc079d59789","local-server-version":"3.5.4","cluster-id":"74885ea7cc240892","cluster-version":"3.5"}
{"level":"info","ts":"2023-02-27T09:04:53.572Z","caller":"etcdserver/server.go:736","msg":"started as single-node; fast-forwarding election ticks","local-member-id":"6ac10dc079d59789","forward-ticks":9,"forward-duration":"4.5s","election-ticks":10,"election-timeout":"5s"}
{"level":"info","ts":"2023-02-27T09:04:53.576Z","caller":"embed/etcd.go:277","msg":"now serving peer/client/metrics","local-member-id":"6ac10dc079d59789","initial-advertise-peer-urls":["<http://127.0.0.1:2400>"],"listen-peer-urls":["<http://127.0.0.1:2400>"],"advertise-client-urls":["<http://127.0.0.1:2399>"],"listen-client-urls":["<http://127.0.0.1:2399>"],"listen-metrics-urls":[]}
{"level":"info","ts":"2023-02-27T09:04:53.576Z","caller":"embed/etcd.go:581","msg":"serving peer traffic","address":"127.0.0.1:2400"}
{"level":"info","ts":"2023-02-27T09:04:53.577Z","caller":"embed/etcd.go:553","msg":"cmux::serve","address":"127.0.0.1:2400"}
{"level":"info","ts":"2023-02-27T09:04:54.562Z","logger":"raft","caller":"etcdserver/zap_raft.go:77","msg":"6ac10dc079d59789 is starting a new election at term 2"}
{"level":"info","ts":"2023-02-27T09:04:54.562Z","logger":"raft","caller":"etcdserver/zap_raft.go:77","msg":"6ac10dc079d59789 became pre-candidate at term 2"}
{"level":"info","ts":"2023-02-27T09:04:54.562Z","logger":"raft","caller":"etcdserver/zap_raft.go:77","msg":"6ac10dc079d59789 received MsgPreVoteResp from 6ac10dc079d59789 at term 2"}
{"level":"info","ts":"2023-02-27T09:04:54.562Z","logger":"raft","caller":"etcdserver/zap_raft.go:77","msg":"6ac10dc079d59789 became candidate at term 3"}
{"level":"info","ts":"2023-02-27T09:04:54.562Z","logger":"raft","caller":"etcdserver/zap_raft.go:77","msg":"6ac10dc079d59789 received MsgVoteResp from 6ac10dc079d59789 at term 3"}
{"level":"info","ts":"2023-02-27T09:04:54.562Z","logger":"raft","caller":"etcdserver/zap_raft.go:77","msg":"6ac10dc079d59789 became leader at term 3"}
{"level":"info","ts":"2023-02-27T09:04:54.562Z","logger":"raft","caller":"etcdserver/zap_raft.go:77","msg":"raft.node: 6ac10dc079d59789 elected leader 6ac10dc079d59789 at term 3"}
{"level":"info","ts":"2023-02-27T09:04:54.579Z","caller":"etcdserver/server.go:2044","msg":"published local member to cluster through raft","local-member-id":"6ac10dc079d59789","local-member-attributes":"{Name:soss-m2-6797f3ae ClientURLs:[<http://127.0.0.1:2399>]}","request-path":"/0/members/6ac10dc079d59789/attributes","cluster-id":"74885ea7cc240892","publish-timeout":"15s"}
{"level":"info","ts":"2023-02-27T09:04:54.580Z","caller":"embed/serve.go:98","msg":"ready to serve client requests"}
{"level":"info","ts":"2023-02-27T09:04:54.581Z","caller":"embed/serve.go:140","msg":"serving client traffic insecurely; this is strongly discouraged!","address":"127.0.0.1:2399"}
INFO[0002] Defragmenting etcd database                  
{"level":"info","ts":"2023-02-27T09:04:54.586Z","caller":"v3rpc/maintenance.go:89","msg":"starting defragment"}
{"level":"info","ts":"2023-02-27T09:04:54.587Z","caller":"backend/backend.go:497","msg":"defragmenting","path":"/var/lib/rancher/rke2/server/db/etcd-tmp/member/snap/db","current-db-size-bytes":3387392,"current-db-size":"3.4 MB","current-db-size-in-use-bytes":1761280,"current-db-size-in-use":"1.8 MB"}
{"level":"info","ts":"2023-02-27T09:04:54.614Z","caller":"backend/backend.go:549","msg":"finished defragmenting directory","path":"/var/lib/rancher/rke2/server/db/etcd-tmp/member/snap/db","current-db-size-bytes-diff":-1622016,"current-db-size-bytes":1765376,"current-db-size":"1.8 MB","current-db-size-in-use-bytes-diff":-4096,"current-db-size-in-use-bytes":1757184,"current-db-size-in-use":"1.8 MB","took":"27.629055ms"}
{"level":"info","ts":"2023-02-27T09:04:54.614Z","caller":"v3rpc/maintenance.go:95","msg":"finished defragment"}
INFO[0002] etcd temporary data store connection OK      
INFO[0002] Reconciling bootstrap data between datastore and disk 
FATA[0002] Failed to reconcile with temporary etcd: bootstrap data already found and encrypted with different token
c

creamy-pencil-82913

02/27/2023, 9:18 AM
Uninstall on this node and start over with the correct server URL and token. The data is out of sync between the two nodes in a way that is easiest to reconcile by just starting over.
w

worried-ram-81084

02/27/2023, 10:06 AM
I uninstalled using:
Copy code
/usr/local/bin/rke2-uninstall.sh
and then used this command to install:
Copy code
curl -sfL <https://get.rke2.io> |  sh -s - server --server=<https://172.16.1.11:9345> --token K10a7523b5d56f0bdca1e3110e29d4d9f7e4ebde89d0e41cdf51a55ab54daaf23ea::server:fefc31e33c1fedafff7cdc962a9307ec
then this command:
Copy code
systemctl enable rke2-server.service
then this command:
Copy code
systemctl start rke2-server.service
and then i noticed nothing coming to the first server so i used this command:
Copy code
rke2 server --server=<https://172.16.1.11:9345> --token K10a7523b5d56f0bdca1e3110e29d4d9f7e4ebde89d0e41cdf51a55ab54daaf23ea::server:fefc31e33c1fedafff7cdc962a9307ec
it did contact the first server but finally i got the same result:
Copy code
root@soss-m2:~# rke2 server --server=<https://172.16.1.11:9345> --token K10a7523b5d56f0bdca1e3110e29d4d9f7e4ebde89d0e41cdf51a55ab54daaf23ea::server:fefc31e33c1fedafff7cdc962a9307ec
WARN[0000] not running in CIS mode                      
INFO[0000] Starting rke2 v1.24.10+rke2r1 (1ccdce2571291649b9414af1f269f645c3fe4002) 
INFO[0000] Managed etcd cluster bootstrap already complete and initialized 
INFO[0000] Starting temporary etcd to reconcile with datastore 
{"level":"info","ts":"2023-02-27T10:04:49.815Z","caller":"embed/etcd.go:131","msg":"configuring peer listeners","listen-peer-urls":["<http://127.0.0.1:2400>"]}
{"level":"info","ts":"2023-02-27T10:04:49.815Z","caller":"embed/etcd.go:139","msg":"configuring client listeners","listen-client-urls":["<http://127.0.0.1:2399>"]}
{"level":"info","ts":"2023-02-27T10:04:49.815Z","caller":"embed/etcd.go:308","msg":"starting an etcd server","etcd-version":"3.5.4","git-sha":"Not provided (use ./build instead of go build)","go-version":"go1.19.5 X:boringcrypto","go-os":"linux","go-arch":"amd64","max-cpu-set":8,"max-cpu-available":8,"member-initialized":true,"name":"soss-m2-bf2134cb","data-dir":"/var/lib/rancher/rke2/server/db/etcd-tmp","wal-dir":"","wal-dir-dedicated":"","member-dir":"/var/lib/rancher/rke2/server/db/etcd-tmp/member","force-new-cluster":true,"heartbeat-interval":"500ms","election-timeout":"5s","initial-election-tick-advance":true,"snapshot-count":10000,"snapshot-catchup-entries":5000,"initial-advertise-peer-urls":["<http://127.0.0.1:2400>"],"listen-peer-urls":["<http://127.0.0.1:2400>"],"advertise-client-urls":["<http://127.0.0.1:2399>"],"listen-client-urls":["<http://127.0.0.1:2399>"],"listen-metrics-urls":[],"cors":["*"],"host-whitelist":["*"],"initial-cluster":"","initial-cluster-state":"new","initial-cluster-token":"","quota-size-bytes":2147483648,"pre-vote":true,"initial-corrupt-check":true,"corrupt-check-time-interval":"0s","auto-compaction-mode":"","auto-compaction-retention":"0s","auto-compaction-interval":"0s","discovery-url":"","discovery-proxy":"","downgrade-check-interval":"5s"}
{"level":"info","ts":"2023-02-27T10:04:49.817Z","caller":"etcdserver/backend.go:81","msg":"opened backend db","path":"/var/lib/rancher/rke2/server/db/etcd-tmp/member/snap/db","took":"1.10559ms"}
{"level":"info","ts":"2023-02-27T10:04:49.820Z","caller":"etcdserver/server.go:529","msg":"No snapshot found. Recovering WAL from scratch!"}
{"level":"info","ts":"2023-02-27T10:04:49.844Z","caller":"etcdserver/raft.go:556","msg":"forcing restart member","cluster-id":"74885ea7cc240892","local-member-id":"6ac10dc079d59789","commit-index":669}
{"level":"info","ts":"2023-02-27T10:04:49.844Z","logger":"raft","caller":"etcdserver/zap_raft.go:77","msg":"6ac10dc079d59789 switched to configuration voters=()"}
{"level":"info","ts":"2023-02-27T10:04:49.844Z","logger":"raft","caller":"etcdserver/zap_raft.go:77","msg":"6ac10dc079d59789 became follower at term 2"}
{"level":"info","ts":"2023-02-27T10:04:49.844Z","logger":"raft","caller":"etcdserver/zap_raft.go:77","msg":"newRaft 6ac10dc079d59789 [peers: [], term: 2, commit: 669, applied: 0, lastindex: 669, lastterm: 2]"}
{"level":"warn","ts":"2023-02-27T10:04:49.847Z","caller":"auth/store.go:1220","msg":"simple token is not cryptographically signed"}
{"level":"info","ts":"2023-02-27T10:04:49.849Z","caller":"mvcc/kvstore.go:415","msg":"kvstore restored","current-rev":647}
{"level":"info","ts":"2023-02-27T10:04:49.849Z","caller":"etcdserver/quota.go:94","msg":"enabled backend quota with default value","quota-name":"v3-applier","quota-size-bytes":2147483648,"quota-size":"2.1 GB"}
{"level":"info","ts":"2023-02-27T10:04:49.849Z","caller":"etcdserver/corrupt.go:46","msg":"starting initial corruption check","local-member-id":"6ac10dc079d59789","timeout":"15s"}
{"level":"info","ts":"2023-02-27T10:04:49.850Z","caller":"etcdserver/corrupt.go:116","msg":"initial corruption checking passed; no corruption","local-member-id":"6ac10dc079d59789"}
{"level":"info","ts":"2023-02-27T10:04:49.850Z","caller":"etcdserver/server.go:851","msg":"starting etcd server","local-member-id":"6ac10dc079d59789","local-server-version":"3.5.4","cluster-version":"to_be_decided"}
{"level":"info","ts":"2023-02-27T10:04:49.850Z","caller":"etcdserver/server.go:752","msg":"starting initial election tick advance","election-ticks":10}
{"level":"info","ts":"2023-02-27T10:04:49.850Z","logger":"raft","caller":"etcdserver/zap_raft.go:77","msg":"6ac10dc079d59789 switched to configuration voters=(7692444758854440841)"}
{"level":"info","ts":"2023-02-27T10:04:49.850Z","caller":"membership/cluster.go:421","msg":"added member","cluster-id":"74885ea7cc240892","local-member-id":"6ac10dc079d59789","added-peer-id":"6ac10dc079d59789","added-peer-peer-urls":["<https://172.16.1.12:2380>"]}
{"level":"info","ts":"2023-02-27T10:04:49.850Z","caller":"membership/cluster.go:584","msg":"set initial cluster version","cluster-id":"74885ea7cc240892","local-member-id":"6ac10dc079d59789","cluster-version":"3.5"}
{"level":"info","ts":"2023-02-27T10:04:49.851Z","caller":"api/capability.go:75","msg":"enabled capabilities for version","cluster-version":"3.5"}
{"level":"info","ts":"2023-02-27T10:04:49.852Z","caller":"embed/etcd.go:277","msg":"now serving peer/client/metrics","local-member-id":"6ac10dc079d59789","initial-advertise-peer-urls":["<http://127.0.0.1:2400>"],"listen-peer-urls":["<http://127.0.0.1:2400>"],"advertise-client-urls":["<http://127.0.0.1:2399>"],"listen-client-urls":["<http://127.0.0.1:2399>"],"listen-metrics-urls":[]}
{"level":"info","ts":"2023-02-27T10:04:49.852Z","caller":"embed/etcd.go:581","msg":"serving peer traffic","address":"127.0.0.1:2400"}
{"level":"info","ts":"2023-02-27T10:04:49.852Z","caller":"embed/etcd.go:553","msg":"cmux::serve","address":"127.0.0.1:2400"}
{"level":"info","ts":"2023-02-27T10:04:54.846Z","logger":"raft","caller":"etcdserver/zap_raft.go:77","msg":"6ac10dc079d59789 is starting a new election at term 2"}
{"level":"info","ts":"2023-02-27T10:04:54.846Z","logger":"raft","caller":"etcdserver/zap_raft.go:77","msg":"6ac10dc079d59789 became pre-candidate at term 2"}
{"level":"info","ts":"2023-02-27T10:04:54.846Z","logger":"raft","caller":"etcdserver/zap_raft.go:77","msg":"6ac10dc079d59789 received MsgPreVoteResp from 6ac10dc079d59789 at term 2"}
{"level":"info","ts":"2023-02-27T10:04:54.846Z","logger":"raft","caller":"etcdserver/zap_raft.go:77","msg":"6ac10dc079d59789 became candidate at term 3"}
{"level":"info","ts":"2023-02-27T10:04:54.846Z","logger":"raft","caller":"etcdserver/zap_raft.go:77","msg":"6ac10dc079d59789 received MsgVoteResp from 6ac10dc079d59789 at term 3"}
{"level":"info","ts":"2023-02-27T10:04:54.846Z","logger":"raft","caller":"etcdserver/zap_raft.go:77","msg":"6ac10dc079d59789 became leader at term 3"}
{"level":"info","ts":"2023-02-27T10:04:54.846Z","logger":"raft","caller":"etcdserver/zap_raft.go:77","msg":"raft.node: 6ac10dc079d59789 elected leader 6ac10dc079d59789 at term 3"}
{"level":"info","ts":"2023-02-27T10:04:54.848Z","caller":"etcdserver/server.go:2044","msg":"published local member to cluster through raft","local-member-id":"6ac10dc079d59789","local-member-attributes":"{Name:soss-m2-bf2134cb ClientURLs:[<http://127.0.0.1:2399>]}","request-path":"/0/members/6ac10dc079d59789/attributes","cluster-id":"74885ea7cc240892","publish-timeout":"15s"}
{"level":"info","ts":"2023-02-27T10:04:54.848Z","caller":"embed/serve.go:98","msg":"ready to serve client requests"}
{"level":"info","ts":"2023-02-27T10:04:54.850Z","caller":"embed/serve.go:140","msg":"serving client traffic insecurely; this is strongly discouraged!","address":"127.0.0.1:2399"}
INFO[0005] Defragmenting etcd database                  
{"level":"info","ts":"2023-02-27T10:04:54.858Z","caller":"v3rpc/maintenance.go:89","msg":"starting defragment"}
{"level":"info","ts":"2023-02-27T10:04:54.859Z","caller":"backend/backend.go:497","msg":"defragmenting","path":"/var/lib/rancher/rke2/server/db/etcd-tmp/member/snap/db","current-db-size-bytes":1708032,"current-db-size":"1.7 MB","current-db-size-in-use-bytes":1695744,"current-db-size-in-use":"1.7 MB"}
{"level":"info","ts":"2023-02-27T10:04:54.890Z","caller":"backend/backend.go:549","msg":"finished defragmenting directory","path":"/var/lib/rancher/rke2/server/db/etcd-tmp/member/snap/db","current-db-size-bytes-diff":-65536,"current-db-size-bytes":1642496,"current-db-size":"1.6 MB","current-db-size-in-use-bytes-diff":-61440,"current-db-size-in-use-bytes":1634304,"current-db-size-in-use":"1.6 MB","took":"32.07575ms"}
{"level":"info","ts":"2023-02-27T10:04:54.890Z","caller":"v3rpc/maintenance.go:95","msg":"finished defragment"}
INFO[0005] etcd temporary data store connection OK      
INFO[0005] Reconciling bootstrap data between datastore and disk 
FATA[0005] Failed to reconcile with temporary etcd: bootstrap data already found and encrypted with different token
g

gray-lawyer-73831

02/27/2023, 4:45 PM
Hmm.. I think you should uninstall and retry again. Instead of adding the args with the install script, you should either create a file at
/etc/rancher/rke2/config.yaml
with the information and run it via systemctl, or just run via the binary. As for the reason as to why all of this is happening: Adding to the install script won’t actually do much for you since it just pulls the tarball and doesn’t run the service. Also, in your example above, what happened there is that you started rke2 (without server/token args since you didn’t have a config file). That then started it on this node as a fresh cluster. Then you tried to run it again on the same node (
rke2 server …
) which would fail regardless due to port conflicts, but in this case actually pointing it to your other server node, so now it doesn’t know which etcd datastore you want it to use. The best thing would be to follow the docs explicitly if you’re getting tripped up: https://docs.rke2.io/install/ha
w

worried-ram-81084

02/28/2023, 5:18 AM
ok, here is a fresh installation: first command
Copy code
sudo nano /etc/rancher/rke2/config.yml
and inside it is:
Copy code
server: <https://192.168.3.180:9345>
token: rke2-demo
then
Copy code
sudo -i
then
Copy code
curl -sfL <https://get.rke2.io> | sh -
then
Copy code
systemctl enable rke2-server.service
then
Copy code
systemctl start rke2-server.service
the installation is finished and there is nothing came to master1:
Copy code
test@master1:~$ sudo tcpdump src 192.168.3.176
tcpdump: verbose output suppressed, use -v[v]... for full protocol decode
listening on ens160, link-type EN10MB (Ethernet), snapshot length 262144 bytes
then i tried to run this command:
Copy code
rke2 server
and the response was:
Copy code
root@test:~# rke2 server
WARN[0000] not running in CIS mode                      
INFO[0000] Starting rke2 v1.24.10+rke2r1 (1ccdce2571291649b9414af1f269f645c3fe4002) 
INFO[0000] Managed etcd cluster bootstrap already complete and initialized 
INFO[0000] Starting temporary etcd to reconcile with datastore 
{"level":"info","ts":"2023-02-28T05:08:35.112Z","caller":"embed/etcd.go:131","msg":"configuring peer listeners","listen-peer-urls":["<http://127.0.0.1:2400>"]}
{"level":"info","ts":"2023-02-28T05:08:35.112Z","caller":"embed/etcd.go:139","msg":"configuring client listeners","listen-client-urls":["<http://127.0.0.1:2399>"]}
{"level":"info","ts":"2023-02-28T05:08:35.113Z","caller":"embed/etcd.go:308","msg":"starting an etcd server","etcd-version":"3.5.4","git-sha":"Not provided (use ./build instead of go build)","go-version":"go1.19.5 X:boringcrypto","go-os":"linux","go-arch":"amd64","max-cpu-set":4,"max-cpu-available":4,"member-initialized":true,"name":"test-0010ecc4","data-dir":"/var/lib/rancher/rke2/server/db/etcd-tmp","wal-dir":"","wal-dir-dedicated":"","member-dir":"/var/lib/rancher/rke2/server/db/etcd-tmp/member","force-new-cluster":true,"heartbeat-interval":"500ms","election-timeout":"5s","initial-election-tick-advance":true,"snapshot-count":10000,"snapshot-catchup-entries":5000,"initial-advertise-peer-urls":["<http://127.0.0.1:2400>"],"listen-peer-urls":["<http://127.0.0.1:2400>"],"advertise-client-urls":["<http://127.0.0.1:2399>"],"listen-client-urls":["<http://127.0.0.1:2399>"],"listen-metrics-urls":[],"cors":["*"],"host-whitelist":["*"],"initial-cluster":"","initial-cluster-state":"new","initial-cluster-token":"","quota-size-bytes":2147483648,"pre-vote":true,"initial-corrupt-check":true,"corrupt-check-time-interval":"0s","auto-compaction-mode":"","auto-compaction-retention":"0s","auto-compaction-interval":"0s","discovery-url":"","discovery-proxy":"","downgrade-check-interval":"5s"}
{"level":"info","ts":"2023-02-28T05:08:35.116Z","caller":"etcdserver/backend.go:81","msg":"opened backend db","path":"/var/lib/rancher/rke2/server/db/etcd-tmp/member/snap/db","took":"2.726128ms"}
{"level":"info","ts":"2023-02-28T05:08:35.130Z","caller":"etcdserver/server.go:529","msg":"No snapshot found. Recovering WAL from scratch!"}
{"level":"info","ts":"2023-02-28T05:08:35.167Z","caller":"etcdserver/raft.go:556","msg":"forcing restart member","cluster-id":"8f86cad4ee5c8993","local-member-id":"8b8e4494dd110009","commit-index":887}
{"level":"info","ts":"2023-02-28T05:08:35.168Z","logger":"raft","caller":"etcdserver/zap_raft.go:77","msg":"8b8e4494dd110009 switched to configuration voters=()"}
{"level":"info","ts":"2023-02-28T05:08:35.168Z","logger":"raft","caller":"etcdserver/zap_raft.go:77","msg":"8b8e4494dd110009 became follower at term 2"}
{"level":"info","ts":"2023-02-28T05:08:35.168Z","logger":"raft","caller":"etcdserver/zap_raft.go:77","msg":"newRaft 8b8e4494dd110009 [peers: [], term: 2, commit: 887, applied: 0, lastindex: 887, lastterm: 2]"}
{"level":"warn","ts":"2023-02-28T05:08:36.720Z","caller":"auth/store.go:1220","msg":"simple token is not cryptographically signed"}
{"level":"info","ts":"2023-02-28T05:08:36.736Z","caller":"mvcc/kvstore.go:415","msg":"kvstore restored","current-rev":847}
{"level":"info","ts":"2023-02-28T05:08:36.805Z","caller":"etcdserver/quota.go:94","msg":"enabled backend quota with default value","quota-name":"v3-applier","quota-size-bytes":2147483648,"quota-size":"2.1 GB"}
{"level":"info","ts":"2023-02-28T05:08:36.818Z","caller":"etcdserver/corrupt.go:46","msg":"starting initial corruption check","local-member-id":"8b8e4494dd110009","timeout":"15s"}
{"level":"info","ts":"2023-02-28T05:08:36.820Z","caller":"etcdserver/corrupt.go:116","msg":"initial corruption checking passed; no corruption","local-member-id":"8b8e4494dd110009"}
{"level":"info","ts":"2023-02-28T05:08:36.820Z","caller":"etcdserver/server.go:851","msg":"starting etcd server","local-member-id":"8b8e4494dd110009","local-server-version":"3.5.4","cluster-version":"to_be_decided"}
{"level":"info","ts":"2023-02-28T05:08:36.820Z","caller":"etcdserver/server.go:752","msg":"starting initial election tick advance","election-ticks":10}
{"level":"info","ts":"2023-02-28T05:08:36.821Z","logger":"raft","caller":"etcdserver/zap_raft.go:77","msg":"8b8e4494dd110009 switched to configuration voters=(10056050424119623689)"}
{"level":"info","ts":"2023-02-28T05:08:36.821Z","caller":"membership/cluster.go:421","msg":"added member","cluster-id":"8f86cad4ee5c8993","local-member-id":"8b8e4494dd110009","added-peer-id":"8b8e4494dd110009","added-peer-peer-urls":["<https://192.168.3.176:2380>"]}
{"level":"info","ts":"2023-02-28T05:08:36.821Z","caller":"membership/cluster.go:584","msg":"set initial cluster version","cluster-id":"8f86cad4ee5c8993","local-member-id":"8b8e4494dd110009","cluster-version":"3.5"}
{"level":"info","ts":"2023-02-28T05:08:36.821Z","caller":"api/capability.go:75","msg":"enabled capabilities for version","cluster-version":"3.5"}
{"level":"info","ts":"2023-02-28T05:08:36.825Z","caller":"embed/etcd.go:581","msg":"serving peer traffic","address":"127.0.0.1:2400"}
{"level":"info","ts":"2023-02-28T05:08:36.825Z","caller":"embed/etcd.go:553","msg":"cmux::serve","address":"127.0.0.1:2400"}
{"level":"info","ts":"2023-02-28T05:08:36.825Z","caller":"embed/etcd.go:277","msg":"now serving peer/client/metrics","local-member-id":"8b8e4494dd110009","initial-advertise-peer-urls":["<http://127.0.0.1:2400>"],"listen-peer-urls":["<http://127.0.0.1:2400>"],"advertise-client-urls":["<http://127.0.0.1:2399>"],"listen-client-urls":["<http://127.0.0.1:2399>"],"listen-metrics-urls":[]}
{"level":"info","ts":"2023-02-28T05:08:42.669Z","logger":"raft","caller":"etcdserver/zap_raft.go:77","msg":"8b8e4494dd110009 is starting a new election at term 2"}
{"level":"info","ts":"2023-02-28T05:08:42.669Z","logger":"raft","caller":"etcdserver/zap_raft.go:77","msg":"8b8e4494dd110009 became pre-candidate at term 2"}
{"level":"info","ts":"2023-02-28T05:08:42.669Z","logger":"raft","caller":"etcdserver/zap_raft.go:77","msg":"8b8e4494dd110009 received MsgPreVoteResp from 8b8e4494dd110009 at term 2"}
{"level":"info","ts":"2023-02-28T05:08:42.669Z","logger":"raft","caller":"etcdserver/zap_raft.go:77","msg":"8b8e4494dd110009 became candidate at term 3"}
{"level":"info","ts":"2023-02-28T05:08:42.669Z","logger":"raft","caller":"etcdserver/zap_raft.go:77","msg":"8b8e4494dd110009 received MsgVoteResp from 8b8e4494dd110009 at term 3"}
{"level":"info","ts":"2023-02-28T05:08:42.669Z","logger":"raft","caller":"etcdserver/zap_raft.go:77","msg":"8b8e4494dd110009 became leader at term 3"}
{"level":"info","ts":"2023-02-28T05:08:42.669Z","logger":"raft","caller":"etcdserver/zap_raft.go:77","msg":"raft.node: 8b8e4494dd110009 elected leader 8b8e4494dd110009 at term 3"}
{"level":"info","ts":"2023-02-28T05:08:42.670Z","caller":"embed/serve.go:98","msg":"ready to serve client requests"}
{"level":"info","ts":"2023-02-28T05:08:42.670Z","caller":"etcdserver/server.go:2044","msg":"published local member to cluster through raft","local-member-id":"8b8e4494dd110009","local-member-attributes":"{Name:test-0010ecc4 ClientURLs:[<http://127.0.0.1:2399>]}","request-path":"/0/members/8b8e4494dd110009/attributes","cluster-id":"8f86cad4ee5c8993","publish-timeout":"15s"}
{"level":"info","ts":"2023-02-28T05:08:42.671Z","caller":"embed/serve.go:140","msg":"serving client traffic insecurely; this is strongly discouraged!","address":"127.0.0.1:2399"}
INFO[0008] Defragmenting etcd database                  
{"level":"info","ts":"2023-02-28T05:08:42.678Z","caller":"v3rpc/maintenance.go:89","msg":"starting defragment"}
{"level":"info","ts":"2023-02-28T05:08:43.837Z","caller":"backend/backend.go:497","msg":"defragmenting","path":"/var/lib/rancher/rke2/server/db/etcd-tmp/member/snap/db","current-db-size-bytes":3182592,"current-db-size":"3.2 MB","current-db-size-in-use-bytes":3182592,"current-db-size-in-use":"3.2 MB"}
{"level":"info","ts":"2023-02-28T05:08:45.285Z","caller":"backend/backend.go:549","msg":"finished defragmenting directory","path":"/var/lib/rancher/rke2/server/db/etcd-tmp/member/snap/db","current-db-size-bytes-diff":-77824,"current-db-size-bytes":3104768,"current-db-size":"3.1 MB","current-db-size-in-use-bytes-diff":-86016,"current-db-size-in-use-bytes":3096576,"current-db-size-in-use":"3.1 MB","took":"2.606279189s"}
{"level":"info","ts":"2023-02-28T05:08:45.285Z","caller":"v3rpc/maintenance.go:95","msg":"finished defragment"}
{"level":"warn","ts":"2023-02-28T05:08:45.285Z","caller":"v3rpc/interceptor.go:197","msg":"request stats","start time":"2023-02-28T05:08:42.678Z","time spent":"2.607225471s","remote":"127.0.0.1:49848","response type":"/etcdserverpb.Maintenance/Defragment","request count":-1,"request size":-1,"response count":-1,"response size":-1,"request content":""}
{"level":"info","ts":"2023-02-28T05:08:45.491Z","caller":"traceutil/trace.go:171","msg":"trace[1862721267] linearizableReadLoop","detail":"{readStateIndex:890; appliedIndex:890; }","duration":"200.915946ms","start":"2023-02-28T05:08:45.290Z","end":"2023-02-28T05:08:45.491Z","steps":["trace[1862721267] 'read index received'  (duration: 200.907674ms)","trace[1862721267] 'applied index is now lower than readState.Index'  (duration: 6.043µs)"],"step_count":2}
INFO[0011] etcd temporary data store connection OK      
INFO[0011] Reconciling bootstrap data between datastore and disk 
INFO[0011] stopping etcd                                
{"level":"info","ts":"2023-02-28T05:08:45.554Z","caller":"embed/etcd.go:368","msg":"closing etcd server","name":"test-0010ecc4","data-dir":"/var/lib/rancher/rke2/server/db/etcd-tmp","advertise-peer-urls":["<http://127.0.0.1:2400>"],"advertise-client-urls":["<http://127.0.0.1:2399>"]}
{"level":"info","ts":"2023-02-28T05:08:45.554Z","caller":"etcdserver/server.go:1453","msg":"skipped leadership transfer for single voting member cluster","local-member-id":"8b8e4494dd110009","current-leader-member-id":"8b8e4494dd110009"}
FATA[0011] starting kubernetes: preparing server: init cluster datastore and https: listen tcp :9345: bind: address already in use
then i ran the command with the right flags:
Copy code
rke2 server --server=<https://192.168.3.180:9345> --token=rke2-demo
and i got the error of the token:
Copy code
FATA[0021] Failed to reconcile with temporary etcd: bootstrap data already found and encrypted with different token
i thought maybe that the error from my token, so i used the generated token:
Copy code
rke2 server --server=<https://192.168.3.180:9345> --token=K10184dcfa4851e487eaa1756e8bedd18f03cb8c6b204dccf2c27090ba9e699cab5::server:1461a0c608734bad8e13b92971e1e09b
and i got the error of new certificates:
Copy code
FATA[0000]newer than datastore and could cause a cluster outage. Remove the file(s) from disk and restart to be recreated from datastore.
so i deleted them, and tried again, and this is the error that i got:
Copy code
FATA[0000] starting kubernetes: preparing server: init cluster datastore and https: listen tcp :9345: bind: address already in use
i did what the documentation asked for workers and it worked! but when i want to join the masters it's just shenanigans!! i'm starting to think that it just hates me 😞 💔
a

abundant-noon-17295

03/02/2023, 2:39 AM
looks like you have systemd
rke2-server
running, and you are trying to start a new process with
rke2 server
using the same listening port
w

worried-ram-81084

03/02/2023, 6:12 AM
but if i didn't do that, nothing happens. No what so ever of contact came from master2 to master1 during the installation processes.
278 Views