worried-ram-81084
02/26/2023, 11:55 AMrke2 server --server <https://172.16.1.11:6443>
and the master didn't join the cluster, what could be the issue?abundant-noon-17295
02/26/2023, 1:39 PMcreamy-pencil-82913
02/26/2023, 7:36 PMworried-ram-81084
02/26/2023, 8:18 PMcreamy-pencil-82913
02/26/2023, 8:38 PMworried-ram-81084
02/27/2023, 5:29 AMsudo rke2 server --server <https://172.16.1.11:6443> -t K108fe0d414ce6d89e53b2a188c4a1be2e9d6d293c2864b352b1e1d1c6af067c629::server:d099c75115ea14a14d5dc90bc8a5f011
and it gets stuck at the end and then i canceled it:
INFO[0028] Starting /v1, Kind=Secret controller
INFO[0028] Updating TLS secret for kube-system/rke2-serving (count: 10): map[<http://listener.cattle.io/cn-10.43.0.1:10.43.0.1|listener.cattle.io/cn-10.43.0.1:10.43.0.1> <http://listener.cattle.io/cn-127.0.0.1:127.0.0.1|listener.cattle.io/cn-127.0.0.1:127.0.0.1> <http://listener.cattle.io/cn-172.16.1.12:172.16.1.12|listener.cattle.io/cn-172.16.1.12:172.16.1.12> <http://listener.cattle.io/cn-__1-f16284:::1|listener.cattle.io/cn-__1-f16284:::1> <http://listener.cattle.io/cn-kubernetes:kubernetes|listener.cattle.io/cn-kubernetes:kubernetes> <http://listener.cattle.io/cn-kubernetes.default:kubernetes.default|listener.cattle.io/cn-kubernetes.default:kubernetes.default> <http://listener.cattle.io/cn-kubernetes.default.svc:kubernetes.default.svc|listener.cattle.io/cn-kubernetes.default.svc:kubernetes.default.svc> <http://listener.cattle.io/cn-kubernetes.default.svc.cluster.local:kubernetes.default.svc.cluster.local|listener.cattle.io/cn-kubernetes.default.svc.cluster.local:kubernetes.default.svc.cluster.local> <http://listener.cattle.io/cn-localhost:localhost|listener.cattle.io/cn-localhost:localhost> <http://listener.cattle.io/cn-soss-m2:soss-m2|listener.cattle.io/cn-soss-m2:soss-m2> <http://listener.cattle.io/fingerprint:SHA1=EC20B204DC47FB082C468868CA1E45DC9D3148BC]|listener.cattle.io/fingerprint:SHA1=EC20B204DC47FB082C468868CA1E45DC9D3148BC]>
INFO[0028] Running kube-proxy --cluster-cidr=10.42.0.0/16 --conntrack-max-per-core=0 --conntrack-tcp-timeout-close-wait=0s --conntrack-tcp-timeout-established=0s --healthz-bind-address=127.0.0.1 --hostname-override=soss-m2 --kubeconfig=/var/lib/rancher/rke2/agent/kubeproxy.kubeconfig --proxy-mode=iptables
^CINFO[0933] Proxy done err="context canceled" url="<wss://172.16.1.12:9345/v1-rke2/connect>"
INFO[0933] Shutting down <http://k3s.cattle.io/v1|k3s.cattle.io/v1>, Kind=Addon workers
W0226 13:28:12.053517 77352 reflector.go:442] <http://k8s.io/client-go@v1.24.9-k3s1/tools/cache/reflector.go:167|k8s.io/client-go@v1.24.9-k3s1/tools/cache/reflector.go:167>: watch of *v1.Endpoints ended with: an error on the server ("unable to decode an event from the watch stream: context canceled") has prevented the request from succeeding
INFO[0933] Shutting down /v1, Kind=Secret workers
INFO[0933] Shutting down /v1, Kind=Node workers
FATA[0933] context canceled
abundant-noon-17295
02/27/2023, 5:31 AMworried-ram-81084
02/27/2023, 5:32 AMabundant-noon-17295
02/27/2023, 5:36 AMcreamy-pencil-82913
02/27/2023, 6:34 AMworried-ram-81084
02/27/2023, 9:05 AMsd@soss-m2:~$ sudo rke2 server --server <https://172.16.1.11:9345> --token K10a7523b5d56f0bdca1e3110e29d4d9f7e4ebde89d0e41cdf51a55ab54daaf23ea::server:fefc31e33c1fedafff7cdc962a9307ec
WARN[0000] not running in CIS mode
INFO[0000] Starting rke2 v1.24.10+rke2r1 (1ccdce2571291649b9414af1f269f645c3fe4002)
INFO[0000] Managed etcd cluster bootstrap already complete and initialized
INFO[0000] Starting temporary etcd to reconcile with datastore
{"level":"info","ts":"2023-02-27T09:04:52.773Z","caller":"embed/etcd.go:131","msg":"configuring peer listeners","listen-peer-urls":["<http://127.0.0.1:2400>"]}
{"level":"info","ts":"2023-02-27T09:04:52.774Z","caller":"embed/etcd.go:139","msg":"configuring client listeners","listen-client-urls":["<http://127.0.0.1:2399>"]}
{"level":"info","ts":"2023-02-27T09:04:52.774Z","caller":"embed/etcd.go:308","msg":"starting an etcd server","etcd-version":"3.5.4","git-sha":"Not provided (use ./build instead of go build)","go-version":"go1.19.5 X:boringcrypto","go-os":"linux","go-arch":"amd64","max-cpu-set":8,"max-cpu-available":8,"member-initialized":true,"name":"soss-m2-6797f3ae","data-dir":"/var/lib/rancher/rke2/server/db/etcd-tmp","wal-dir":"","wal-dir-dedicated":"","member-dir":"/var/lib/rancher/rke2/server/db/etcd-tmp/member","force-new-cluster":true,"heartbeat-interval":"500ms","election-timeout":"5s","initial-election-tick-advance":true,"snapshot-count":10000,"snapshot-catchup-entries":5000,"initial-advertise-peer-urls":["<http://127.0.0.1:2400>"],"listen-peer-urls":["<http://127.0.0.1:2400>"],"advertise-client-urls":["<http://127.0.0.1:2399>"],"listen-client-urls":["<http://127.0.0.1:2399>"],"listen-metrics-urls":[],"cors":["*"],"host-whitelist":["*"],"initial-cluster":"","initial-cluster-state":"new","initial-cluster-token":"","quota-size-bytes":2147483648,"pre-vote":true,"initial-corrupt-check":true,"corrupt-check-time-interval":"0s","auto-compaction-mode":"","auto-compaction-retention":"0s","auto-compaction-interval":"0s","discovery-url":"","discovery-proxy":"","downgrade-check-interval":"5s"}
{"level":"info","ts":"2023-02-27T09:04:52.776Z","caller":"etcdserver/backend.go:81","msg":"opened backend db","path":"/var/lib/rancher/rke2/server/db/etcd-tmp/member/snap/db","took":"2.012679ms"}
{"level":"info","ts":"2023-02-27T09:04:53.294Z","caller":"etcdserver/server.go:508","msg":"recovered v2 store from snapshot","snapshot-index":140015,"snapshot-size":"7.1 kB"}
{"level":"info","ts":"2023-02-27T09:04:53.294Z","caller":"etcdserver/server.go:521","msg":"recovered v3 backend from snapshot","backend-size-bytes":3387392,"backend-size":"3.4 MB","backend-size-in-use-bytes":1757184,"backend-size-in-use":"1.8 MB"}
{"level":"info","ts":"2023-02-27T09:04:53.560Z","caller":"etcdserver/raft.go:556","msg":"forcing restart member","cluster-id":"74885ea7cc240892","local-member-id":"6ac10dc079d59789","commit-index":149285}
{"level":"info","ts":"2023-02-27T09:04:53.561Z","logger":"raft","caller":"etcdserver/zap_raft.go:77","msg":"6ac10dc079d59789 switched to configuration voters=(7692444758854440841)"}
{"level":"info","ts":"2023-02-27T09:04:53.561Z","logger":"raft","caller":"etcdserver/zap_raft.go:77","msg":"6ac10dc079d59789 became follower at term 2"}
{"level":"info","ts":"2023-02-27T09:04:53.561Z","logger":"raft","caller":"etcdserver/zap_raft.go:77","msg":"newRaft 6ac10dc079d59789 [peers: [6ac10dc079d59789], term: 2, commit: 149285, applied: 140015, lastindex: 149285, lastterm: 2]"}
{"level":"info","ts":"2023-02-27T09:04:53.561Z","caller":"api/capability.go:75","msg":"enabled capabilities for version","cluster-version":"3.5"}
{"level":"info","ts":"2023-02-27T09:04:53.561Z","caller":"membership/cluster.go:278","msg":"recovered/added member from store","cluster-id":"74885ea7cc240892","local-member-id":"6ac10dc079d59789","recovered-remote-peer-id":"6ac10dc079d59789","recovered-remote-peer-urls":["<https://172.16.1.12:2380>"]}
{"level":"info","ts":"2023-02-27T09:04:53.561Z","caller":"membership/cluster.go:287","msg":"set cluster version from store","cluster-version":"3.5"}
{"level":"warn","ts":"2023-02-27T09:04:53.562Z","caller":"auth/store.go:1220","msg":"simple token is not cryptographically signed"}
{"level":"info","ts":"2023-02-27T09:04:53.562Z","caller":"mvcc/kvstore.go:345","msg":"restored last compact revision","meta-bucket-name":"meta","meta-bucket-name-key":"finishedCompactRev","restored-compact-revision":133869}
{"level":"info","ts":"2023-02-27T09:04:53.570Z","caller":"mvcc/kvstore.go:415","msg":"kvstore restored","current-rev":134453}
{"level":"info","ts":"2023-02-27T09:04:53.571Z","caller":"etcdserver/quota.go:94","msg":"enabled backend quota with default value","quota-name":"v3-applier","quota-size-bytes":2147483648,"quota-size":"2.1 GB"}
{"level":"info","ts":"2023-02-27T09:04:53.571Z","caller":"etcdserver/corrupt.go:46","msg":"starting initial corruption check","local-member-id":"6ac10dc079d59789","timeout":"15s"}
{"level":"info","ts":"2023-02-27T09:04:53.572Z","caller":"etcdserver/corrupt.go:116","msg":"initial corruption checking passed; no corruption","local-member-id":"6ac10dc079d59789"}
{"level":"info","ts":"2023-02-27T09:04:53.572Z","caller":"etcdserver/server.go:842","msg":"starting etcd server","local-member-id":"6ac10dc079d59789","local-server-version":"3.5.4","cluster-id":"74885ea7cc240892","cluster-version":"3.5"}
{"level":"info","ts":"2023-02-27T09:04:53.572Z","caller":"etcdserver/server.go:736","msg":"started as single-node; fast-forwarding election ticks","local-member-id":"6ac10dc079d59789","forward-ticks":9,"forward-duration":"4.5s","election-ticks":10,"election-timeout":"5s"}
{"level":"info","ts":"2023-02-27T09:04:53.576Z","caller":"embed/etcd.go:277","msg":"now serving peer/client/metrics","local-member-id":"6ac10dc079d59789","initial-advertise-peer-urls":["<http://127.0.0.1:2400>"],"listen-peer-urls":["<http://127.0.0.1:2400>"],"advertise-client-urls":["<http://127.0.0.1:2399>"],"listen-client-urls":["<http://127.0.0.1:2399>"],"listen-metrics-urls":[]}
{"level":"info","ts":"2023-02-27T09:04:53.576Z","caller":"embed/etcd.go:581","msg":"serving peer traffic","address":"127.0.0.1:2400"}
{"level":"info","ts":"2023-02-27T09:04:53.577Z","caller":"embed/etcd.go:553","msg":"cmux::serve","address":"127.0.0.1:2400"}
{"level":"info","ts":"2023-02-27T09:04:54.562Z","logger":"raft","caller":"etcdserver/zap_raft.go:77","msg":"6ac10dc079d59789 is starting a new election at term 2"}
{"level":"info","ts":"2023-02-27T09:04:54.562Z","logger":"raft","caller":"etcdserver/zap_raft.go:77","msg":"6ac10dc079d59789 became pre-candidate at term 2"}
{"level":"info","ts":"2023-02-27T09:04:54.562Z","logger":"raft","caller":"etcdserver/zap_raft.go:77","msg":"6ac10dc079d59789 received MsgPreVoteResp from 6ac10dc079d59789 at term 2"}
{"level":"info","ts":"2023-02-27T09:04:54.562Z","logger":"raft","caller":"etcdserver/zap_raft.go:77","msg":"6ac10dc079d59789 became candidate at term 3"}
{"level":"info","ts":"2023-02-27T09:04:54.562Z","logger":"raft","caller":"etcdserver/zap_raft.go:77","msg":"6ac10dc079d59789 received MsgVoteResp from 6ac10dc079d59789 at term 3"}
{"level":"info","ts":"2023-02-27T09:04:54.562Z","logger":"raft","caller":"etcdserver/zap_raft.go:77","msg":"6ac10dc079d59789 became leader at term 3"}
{"level":"info","ts":"2023-02-27T09:04:54.562Z","logger":"raft","caller":"etcdserver/zap_raft.go:77","msg":"raft.node: 6ac10dc079d59789 elected leader 6ac10dc079d59789 at term 3"}
{"level":"info","ts":"2023-02-27T09:04:54.579Z","caller":"etcdserver/server.go:2044","msg":"published local member to cluster through raft","local-member-id":"6ac10dc079d59789","local-member-attributes":"{Name:soss-m2-6797f3ae ClientURLs:[<http://127.0.0.1:2399>]}","request-path":"/0/members/6ac10dc079d59789/attributes","cluster-id":"74885ea7cc240892","publish-timeout":"15s"}
{"level":"info","ts":"2023-02-27T09:04:54.580Z","caller":"embed/serve.go:98","msg":"ready to serve client requests"}
{"level":"info","ts":"2023-02-27T09:04:54.581Z","caller":"embed/serve.go:140","msg":"serving client traffic insecurely; this is strongly discouraged!","address":"127.0.0.1:2399"}
INFO[0002] Defragmenting etcd database
{"level":"info","ts":"2023-02-27T09:04:54.586Z","caller":"v3rpc/maintenance.go:89","msg":"starting defragment"}
{"level":"info","ts":"2023-02-27T09:04:54.587Z","caller":"backend/backend.go:497","msg":"defragmenting","path":"/var/lib/rancher/rke2/server/db/etcd-tmp/member/snap/db","current-db-size-bytes":3387392,"current-db-size":"3.4 MB","current-db-size-in-use-bytes":1761280,"current-db-size-in-use":"1.8 MB"}
{"level":"info","ts":"2023-02-27T09:04:54.614Z","caller":"backend/backend.go:549","msg":"finished defragmenting directory","path":"/var/lib/rancher/rke2/server/db/etcd-tmp/member/snap/db","current-db-size-bytes-diff":-1622016,"current-db-size-bytes":1765376,"current-db-size":"1.8 MB","current-db-size-in-use-bytes-diff":-4096,"current-db-size-in-use-bytes":1757184,"current-db-size-in-use":"1.8 MB","took":"27.629055ms"}
{"level":"info","ts":"2023-02-27T09:04:54.614Z","caller":"v3rpc/maintenance.go:95","msg":"finished defragment"}
INFO[0002] etcd temporary data store connection OK
INFO[0002] Reconciling bootstrap data between datastore and disk
FATA[0002] Failed to reconcile with temporary etcd: bootstrap data already found and encrypted with different token
creamy-pencil-82913
02/27/2023, 9:18 AMworried-ram-81084
02/27/2023, 10:06 AM/usr/local/bin/rke2-uninstall.sh
and then used this command to install:
curl -sfL <https://get.rke2.io> | sh -s - server --server=<https://172.16.1.11:9345> --token K10a7523b5d56f0bdca1e3110e29d4d9f7e4ebde89d0e41cdf51a55ab54daaf23ea::server:fefc31e33c1fedafff7cdc962a9307ec
then this command:
systemctl enable rke2-server.service
then this command:
systemctl start rke2-server.service
and then i noticed nothing coming to the first server so i used this command:
rke2 server --server=<https://172.16.1.11:9345> --token K10a7523b5d56f0bdca1e3110e29d4d9f7e4ebde89d0e41cdf51a55ab54daaf23ea::server:fefc31e33c1fedafff7cdc962a9307ec
it did contact the first server but finally i got the same result:
root@soss-m2:~# rke2 server --server=<https://172.16.1.11:9345> --token K10a7523b5d56f0bdca1e3110e29d4d9f7e4ebde89d0e41cdf51a55ab54daaf23ea::server:fefc31e33c1fedafff7cdc962a9307ec
WARN[0000] not running in CIS mode
INFO[0000] Starting rke2 v1.24.10+rke2r1 (1ccdce2571291649b9414af1f269f645c3fe4002)
INFO[0000] Managed etcd cluster bootstrap already complete and initialized
INFO[0000] Starting temporary etcd to reconcile with datastore
{"level":"info","ts":"2023-02-27T10:04:49.815Z","caller":"embed/etcd.go:131","msg":"configuring peer listeners","listen-peer-urls":["<http://127.0.0.1:2400>"]}
{"level":"info","ts":"2023-02-27T10:04:49.815Z","caller":"embed/etcd.go:139","msg":"configuring client listeners","listen-client-urls":["<http://127.0.0.1:2399>"]}
{"level":"info","ts":"2023-02-27T10:04:49.815Z","caller":"embed/etcd.go:308","msg":"starting an etcd server","etcd-version":"3.5.4","git-sha":"Not provided (use ./build instead of go build)","go-version":"go1.19.5 X:boringcrypto","go-os":"linux","go-arch":"amd64","max-cpu-set":8,"max-cpu-available":8,"member-initialized":true,"name":"soss-m2-bf2134cb","data-dir":"/var/lib/rancher/rke2/server/db/etcd-tmp","wal-dir":"","wal-dir-dedicated":"","member-dir":"/var/lib/rancher/rke2/server/db/etcd-tmp/member","force-new-cluster":true,"heartbeat-interval":"500ms","election-timeout":"5s","initial-election-tick-advance":true,"snapshot-count":10000,"snapshot-catchup-entries":5000,"initial-advertise-peer-urls":["<http://127.0.0.1:2400>"],"listen-peer-urls":["<http://127.0.0.1:2400>"],"advertise-client-urls":["<http://127.0.0.1:2399>"],"listen-client-urls":["<http://127.0.0.1:2399>"],"listen-metrics-urls":[],"cors":["*"],"host-whitelist":["*"],"initial-cluster":"","initial-cluster-state":"new","initial-cluster-token":"","quota-size-bytes":2147483648,"pre-vote":true,"initial-corrupt-check":true,"corrupt-check-time-interval":"0s","auto-compaction-mode":"","auto-compaction-retention":"0s","auto-compaction-interval":"0s","discovery-url":"","discovery-proxy":"","downgrade-check-interval":"5s"}
{"level":"info","ts":"2023-02-27T10:04:49.817Z","caller":"etcdserver/backend.go:81","msg":"opened backend db","path":"/var/lib/rancher/rke2/server/db/etcd-tmp/member/snap/db","took":"1.10559ms"}
{"level":"info","ts":"2023-02-27T10:04:49.820Z","caller":"etcdserver/server.go:529","msg":"No snapshot found. Recovering WAL from scratch!"}
{"level":"info","ts":"2023-02-27T10:04:49.844Z","caller":"etcdserver/raft.go:556","msg":"forcing restart member","cluster-id":"74885ea7cc240892","local-member-id":"6ac10dc079d59789","commit-index":669}
{"level":"info","ts":"2023-02-27T10:04:49.844Z","logger":"raft","caller":"etcdserver/zap_raft.go:77","msg":"6ac10dc079d59789 switched to configuration voters=()"}
{"level":"info","ts":"2023-02-27T10:04:49.844Z","logger":"raft","caller":"etcdserver/zap_raft.go:77","msg":"6ac10dc079d59789 became follower at term 2"}
{"level":"info","ts":"2023-02-27T10:04:49.844Z","logger":"raft","caller":"etcdserver/zap_raft.go:77","msg":"newRaft 6ac10dc079d59789 [peers: [], term: 2, commit: 669, applied: 0, lastindex: 669, lastterm: 2]"}
{"level":"warn","ts":"2023-02-27T10:04:49.847Z","caller":"auth/store.go:1220","msg":"simple token is not cryptographically signed"}
{"level":"info","ts":"2023-02-27T10:04:49.849Z","caller":"mvcc/kvstore.go:415","msg":"kvstore restored","current-rev":647}
{"level":"info","ts":"2023-02-27T10:04:49.849Z","caller":"etcdserver/quota.go:94","msg":"enabled backend quota with default value","quota-name":"v3-applier","quota-size-bytes":2147483648,"quota-size":"2.1 GB"}
{"level":"info","ts":"2023-02-27T10:04:49.849Z","caller":"etcdserver/corrupt.go:46","msg":"starting initial corruption check","local-member-id":"6ac10dc079d59789","timeout":"15s"}
{"level":"info","ts":"2023-02-27T10:04:49.850Z","caller":"etcdserver/corrupt.go:116","msg":"initial corruption checking passed; no corruption","local-member-id":"6ac10dc079d59789"}
{"level":"info","ts":"2023-02-27T10:04:49.850Z","caller":"etcdserver/server.go:851","msg":"starting etcd server","local-member-id":"6ac10dc079d59789","local-server-version":"3.5.4","cluster-version":"to_be_decided"}
{"level":"info","ts":"2023-02-27T10:04:49.850Z","caller":"etcdserver/server.go:752","msg":"starting initial election tick advance","election-ticks":10}
{"level":"info","ts":"2023-02-27T10:04:49.850Z","logger":"raft","caller":"etcdserver/zap_raft.go:77","msg":"6ac10dc079d59789 switched to configuration voters=(7692444758854440841)"}
{"level":"info","ts":"2023-02-27T10:04:49.850Z","caller":"membership/cluster.go:421","msg":"added member","cluster-id":"74885ea7cc240892","local-member-id":"6ac10dc079d59789","added-peer-id":"6ac10dc079d59789","added-peer-peer-urls":["<https://172.16.1.12:2380>"]}
{"level":"info","ts":"2023-02-27T10:04:49.850Z","caller":"membership/cluster.go:584","msg":"set initial cluster version","cluster-id":"74885ea7cc240892","local-member-id":"6ac10dc079d59789","cluster-version":"3.5"}
{"level":"info","ts":"2023-02-27T10:04:49.851Z","caller":"api/capability.go:75","msg":"enabled capabilities for version","cluster-version":"3.5"}
{"level":"info","ts":"2023-02-27T10:04:49.852Z","caller":"embed/etcd.go:277","msg":"now serving peer/client/metrics","local-member-id":"6ac10dc079d59789","initial-advertise-peer-urls":["<http://127.0.0.1:2400>"],"listen-peer-urls":["<http://127.0.0.1:2400>"],"advertise-client-urls":["<http://127.0.0.1:2399>"],"listen-client-urls":["<http://127.0.0.1:2399>"],"listen-metrics-urls":[]}
{"level":"info","ts":"2023-02-27T10:04:49.852Z","caller":"embed/etcd.go:581","msg":"serving peer traffic","address":"127.0.0.1:2400"}
{"level":"info","ts":"2023-02-27T10:04:49.852Z","caller":"embed/etcd.go:553","msg":"cmux::serve","address":"127.0.0.1:2400"}
{"level":"info","ts":"2023-02-27T10:04:54.846Z","logger":"raft","caller":"etcdserver/zap_raft.go:77","msg":"6ac10dc079d59789 is starting a new election at term 2"}
{"level":"info","ts":"2023-02-27T10:04:54.846Z","logger":"raft","caller":"etcdserver/zap_raft.go:77","msg":"6ac10dc079d59789 became pre-candidate at term 2"}
{"level":"info","ts":"2023-02-27T10:04:54.846Z","logger":"raft","caller":"etcdserver/zap_raft.go:77","msg":"6ac10dc079d59789 received MsgPreVoteResp from 6ac10dc079d59789 at term 2"}
{"level":"info","ts":"2023-02-27T10:04:54.846Z","logger":"raft","caller":"etcdserver/zap_raft.go:77","msg":"6ac10dc079d59789 became candidate at term 3"}
{"level":"info","ts":"2023-02-27T10:04:54.846Z","logger":"raft","caller":"etcdserver/zap_raft.go:77","msg":"6ac10dc079d59789 received MsgVoteResp from 6ac10dc079d59789 at term 3"}
{"level":"info","ts":"2023-02-27T10:04:54.846Z","logger":"raft","caller":"etcdserver/zap_raft.go:77","msg":"6ac10dc079d59789 became leader at term 3"}
{"level":"info","ts":"2023-02-27T10:04:54.846Z","logger":"raft","caller":"etcdserver/zap_raft.go:77","msg":"raft.node: 6ac10dc079d59789 elected leader 6ac10dc079d59789 at term 3"}
{"level":"info","ts":"2023-02-27T10:04:54.848Z","caller":"etcdserver/server.go:2044","msg":"published local member to cluster through raft","local-member-id":"6ac10dc079d59789","local-member-attributes":"{Name:soss-m2-bf2134cb ClientURLs:[<http://127.0.0.1:2399>]}","request-path":"/0/members/6ac10dc079d59789/attributes","cluster-id":"74885ea7cc240892","publish-timeout":"15s"}
{"level":"info","ts":"2023-02-27T10:04:54.848Z","caller":"embed/serve.go:98","msg":"ready to serve client requests"}
{"level":"info","ts":"2023-02-27T10:04:54.850Z","caller":"embed/serve.go:140","msg":"serving client traffic insecurely; this is strongly discouraged!","address":"127.0.0.1:2399"}
INFO[0005] Defragmenting etcd database
{"level":"info","ts":"2023-02-27T10:04:54.858Z","caller":"v3rpc/maintenance.go:89","msg":"starting defragment"}
{"level":"info","ts":"2023-02-27T10:04:54.859Z","caller":"backend/backend.go:497","msg":"defragmenting","path":"/var/lib/rancher/rke2/server/db/etcd-tmp/member/snap/db","current-db-size-bytes":1708032,"current-db-size":"1.7 MB","current-db-size-in-use-bytes":1695744,"current-db-size-in-use":"1.7 MB"}
{"level":"info","ts":"2023-02-27T10:04:54.890Z","caller":"backend/backend.go:549","msg":"finished defragmenting directory","path":"/var/lib/rancher/rke2/server/db/etcd-tmp/member/snap/db","current-db-size-bytes-diff":-65536,"current-db-size-bytes":1642496,"current-db-size":"1.6 MB","current-db-size-in-use-bytes-diff":-61440,"current-db-size-in-use-bytes":1634304,"current-db-size-in-use":"1.6 MB","took":"32.07575ms"}
{"level":"info","ts":"2023-02-27T10:04:54.890Z","caller":"v3rpc/maintenance.go:95","msg":"finished defragment"}
INFO[0005] etcd temporary data store connection OK
INFO[0005] Reconciling bootstrap data between datastore and disk
FATA[0005] Failed to reconcile with temporary etcd: bootstrap data already found and encrypted with different token
gray-lawyer-73831
02/27/2023, 4:45 PM/etc/rancher/rke2/config.yaml
with the information and run it via systemctl, or just run via the binary.
As for the reason as to why all of this is happening: Adding to the install script won’t actually do much for you since it just pulls the tarball and doesn’t run the service. Also, in your example above, what happened there is that you started rke2 (without server/token args since you didn’t have a config file). That then started it on this node as a fresh cluster. Then you tried to run it again on the same node (rke2 server …
) which would fail regardless due to port conflicts, but in this case actually pointing it to your other server node, so now it doesn’t know which etcd datastore you want it to use. The best thing would be to follow the docs explicitly if you’re getting tripped up: https://docs.rke2.io/install/haworried-ram-81084
02/28/2023, 5:18 AMsudo nano /etc/rancher/rke2/config.yml
and inside it is:
server: <https://192.168.3.180:9345>
token: rke2-demo
then
sudo -i
then
curl -sfL <https://get.rke2.io> | sh -
then
systemctl enable rke2-server.service
then
systemctl start rke2-server.service
the installation is finished and there is nothing came to master1:
test@master1:~$ sudo tcpdump src 192.168.3.176
tcpdump: verbose output suppressed, use -v[v]... for full protocol decode
listening on ens160, link-type EN10MB (Ethernet), snapshot length 262144 bytes
then i tried to run this command:
rke2 server
and the response was:
root@test:~# rke2 server
WARN[0000] not running in CIS mode
INFO[0000] Starting rke2 v1.24.10+rke2r1 (1ccdce2571291649b9414af1f269f645c3fe4002)
INFO[0000] Managed etcd cluster bootstrap already complete and initialized
INFO[0000] Starting temporary etcd to reconcile with datastore
{"level":"info","ts":"2023-02-28T05:08:35.112Z","caller":"embed/etcd.go:131","msg":"configuring peer listeners","listen-peer-urls":["<http://127.0.0.1:2400>"]}
{"level":"info","ts":"2023-02-28T05:08:35.112Z","caller":"embed/etcd.go:139","msg":"configuring client listeners","listen-client-urls":["<http://127.0.0.1:2399>"]}
{"level":"info","ts":"2023-02-28T05:08:35.113Z","caller":"embed/etcd.go:308","msg":"starting an etcd server","etcd-version":"3.5.4","git-sha":"Not provided (use ./build instead of go build)","go-version":"go1.19.5 X:boringcrypto","go-os":"linux","go-arch":"amd64","max-cpu-set":4,"max-cpu-available":4,"member-initialized":true,"name":"test-0010ecc4","data-dir":"/var/lib/rancher/rke2/server/db/etcd-tmp","wal-dir":"","wal-dir-dedicated":"","member-dir":"/var/lib/rancher/rke2/server/db/etcd-tmp/member","force-new-cluster":true,"heartbeat-interval":"500ms","election-timeout":"5s","initial-election-tick-advance":true,"snapshot-count":10000,"snapshot-catchup-entries":5000,"initial-advertise-peer-urls":["<http://127.0.0.1:2400>"],"listen-peer-urls":["<http://127.0.0.1:2400>"],"advertise-client-urls":["<http://127.0.0.1:2399>"],"listen-client-urls":["<http://127.0.0.1:2399>"],"listen-metrics-urls":[],"cors":["*"],"host-whitelist":["*"],"initial-cluster":"","initial-cluster-state":"new","initial-cluster-token":"","quota-size-bytes":2147483648,"pre-vote":true,"initial-corrupt-check":true,"corrupt-check-time-interval":"0s","auto-compaction-mode":"","auto-compaction-retention":"0s","auto-compaction-interval":"0s","discovery-url":"","discovery-proxy":"","downgrade-check-interval":"5s"}
{"level":"info","ts":"2023-02-28T05:08:35.116Z","caller":"etcdserver/backend.go:81","msg":"opened backend db","path":"/var/lib/rancher/rke2/server/db/etcd-tmp/member/snap/db","took":"2.726128ms"}
{"level":"info","ts":"2023-02-28T05:08:35.130Z","caller":"etcdserver/server.go:529","msg":"No snapshot found. Recovering WAL from scratch!"}
{"level":"info","ts":"2023-02-28T05:08:35.167Z","caller":"etcdserver/raft.go:556","msg":"forcing restart member","cluster-id":"8f86cad4ee5c8993","local-member-id":"8b8e4494dd110009","commit-index":887}
{"level":"info","ts":"2023-02-28T05:08:35.168Z","logger":"raft","caller":"etcdserver/zap_raft.go:77","msg":"8b8e4494dd110009 switched to configuration voters=()"}
{"level":"info","ts":"2023-02-28T05:08:35.168Z","logger":"raft","caller":"etcdserver/zap_raft.go:77","msg":"8b8e4494dd110009 became follower at term 2"}
{"level":"info","ts":"2023-02-28T05:08:35.168Z","logger":"raft","caller":"etcdserver/zap_raft.go:77","msg":"newRaft 8b8e4494dd110009 [peers: [], term: 2, commit: 887, applied: 0, lastindex: 887, lastterm: 2]"}
{"level":"warn","ts":"2023-02-28T05:08:36.720Z","caller":"auth/store.go:1220","msg":"simple token is not cryptographically signed"}
{"level":"info","ts":"2023-02-28T05:08:36.736Z","caller":"mvcc/kvstore.go:415","msg":"kvstore restored","current-rev":847}
{"level":"info","ts":"2023-02-28T05:08:36.805Z","caller":"etcdserver/quota.go:94","msg":"enabled backend quota with default value","quota-name":"v3-applier","quota-size-bytes":2147483648,"quota-size":"2.1 GB"}
{"level":"info","ts":"2023-02-28T05:08:36.818Z","caller":"etcdserver/corrupt.go:46","msg":"starting initial corruption check","local-member-id":"8b8e4494dd110009","timeout":"15s"}
{"level":"info","ts":"2023-02-28T05:08:36.820Z","caller":"etcdserver/corrupt.go:116","msg":"initial corruption checking passed; no corruption","local-member-id":"8b8e4494dd110009"}
{"level":"info","ts":"2023-02-28T05:08:36.820Z","caller":"etcdserver/server.go:851","msg":"starting etcd server","local-member-id":"8b8e4494dd110009","local-server-version":"3.5.4","cluster-version":"to_be_decided"}
{"level":"info","ts":"2023-02-28T05:08:36.820Z","caller":"etcdserver/server.go:752","msg":"starting initial election tick advance","election-ticks":10}
{"level":"info","ts":"2023-02-28T05:08:36.821Z","logger":"raft","caller":"etcdserver/zap_raft.go:77","msg":"8b8e4494dd110009 switched to configuration voters=(10056050424119623689)"}
{"level":"info","ts":"2023-02-28T05:08:36.821Z","caller":"membership/cluster.go:421","msg":"added member","cluster-id":"8f86cad4ee5c8993","local-member-id":"8b8e4494dd110009","added-peer-id":"8b8e4494dd110009","added-peer-peer-urls":["<https://192.168.3.176:2380>"]}
{"level":"info","ts":"2023-02-28T05:08:36.821Z","caller":"membership/cluster.go:584","msg":"set initial cluster version","cluster-id":"8f86cad4ee5c8993","local-member-id":"8b8e4494dd110009","cluster-version":"3.5"}
{"level":"info","ts":"2023-02-28T05:08:36.821Z","caller":"api/capability.go:75","msg":"enabled capabilities for version","cluster-version":"3.5"}
{"level":"info","ts":"2023-02-28T05:08:36.825Z","caller":"embed/etcd.go:581","msg":"serving peer traffic","address":"127.0.0.1:2400"}
{"level":"info","ts":"2023-02-28T05:08:36.825Z","caller":"embed/etcd.go:553","msg":"cmux::serve","address":"127.0.0.1:2400"}
{"level":"info","ts":"2023-02-28T05:08:36.825Z","caller":"embed/etcd.go:277","msg":"now serving peer/client/metrics","local-member-id":"8b8e4494dd110009","initial-advertise-peer-urls":["<http://127.0.0.1:2400>"],"listen-peer-urls":["<http://127.0.0.1:2400>"],"advertise-client-urls":["<http://127.0.0.1:2399>"],"listen-client-urls":["<http://127.0.0.1:2399>"],"listen-metrics-urls":[]}
{"level":"info","ts":"2023-02-28T05:08:42.669Z","logger":"raft","caller":"etcdserver/zap_raft.go:77","msg":"8b8e4494dd110009 is starting a new election at term 2"}
{"level":"info","ts":"2023-02-28T05:08:42.669Z","logger":"raft","caller":"etcdserver/zap_raft.go:77","msg":"8b8e4494dd110009 became pre-candidate at term 2"}
{"level":"info","ts":"2023-02-28T05:08:42.669Z","logger":"raft","caller":"etcdserver/zap_raft.go:77","msg":"8b8e4494dd110009 received MsgPreVoteResp from 8b8e4494dd110009 at term 2"}
{"level":"info","ts":"2023-02-28T05:08:42.669Z","logger":"raft","caller":"etcdserver/zap_raft.go:77","msg":"8b8e4494dd110009 became candidate at term 3"}
{"level":"info","ts":"2023-02-28T05:08:42.669Z","logger":"raft","caller":"etcdserver/zap_raft.go:77","msg":"8b8e4494dd110009 received MsgVoteResp from 8b8e4494dd110009 at term 3"}
{"level":"info","ts":"2023-02-28T05:08:42.669Z","logger":"raft","caller":"etcdserver/zap_raft.go:77","msg":"8b8e4494dd110009 became leader at term 3"}
{"level":"info","ts":"2023-02-28T05:08:42.669Z","logger":"raft","caller":"etcdserver/zap_raft.go:77","msg":"raft.node: 8b8e4494dd110009 elected leader 8b8e4494dd110009 at term 3"}
{"level":"info","ts":"2023-02-28T05:08:42.670Z","caller":"embed/serve.go:98","msg":"ready to serve client requests"}
{"level":"info","ts":"2023-02-28T05:08:42.670Z","caller":"etcdserver/server.go:2044","msg":"published local member to cluster through raft","local-member-id":"8b8e4494dd110009","local-member-attributes":"{Name:test-0010ecc4 ClientURLs:[<http://127.0.0.1:2399>]}","request-path":"/0/members/8b8e4494dd110009/attributes","cluster-id":"8f86cad4ee5c8993","publish-timeout":"15s"}
{"level":"info","ts":"2023-02-28T05:08:42.671Z","caller":"embed/serve.go:140","msg":"serving client traffic insecurely; this is strongly discouraged!","address":"127.0.0.1:2399"}
INFO[0008] Defragmenting etcd database
{"level":"info","ts":"2023-02-28T05:08:42.678Z","caller":"v3rpc/maintenance.go:89","msg":"starting defragment"}
{"level":"info","ts":"2023-02-28T05:08:43.837Z","caller":"backend/backend.go:497","msg":"defragmenting","path":"/var/lib/rancher/rke2/server/db/etcd-tmp/member/snap/db","current-db-size-bytes":3182592,"current-db-size":"3.2 MB","current-db-size-in-use-bytes":3182592,"current-db-size-in-use":"3.2 MB"}
{"level":"info","ts":"2023-02-28T05:08:45.285Z","caller":"backend/backend.go:549","msg":"finished defragmenting directory","path":"/var/lib/rancher/rke2/server/db/etcd-tmp/member/snap/db","current-db-size-bytes-diff":-77824,"current-db-size-bytes":3104768,"current-db-size":"3.1 MB","current-db-size-in-use-bytes-diff":-86016,"current-db-size-in-use-bytes":3096576,"current-db-size-in-use":"3.1 MB","took":"2.606279189s"}
{"level":"info","ts":"2023-02-28T05:08:45.285Z","caller":"v3rpc/maintenance.go:95","msg":"finished defragment"}
{"level":"warn","ts":"2023-02-28T05:08:45.285Z","caller":"v3rpc/interceptor.go:197","msg":"request stats","start time":"2023-02-28T05:08:42.678Z","time spent":"2.607225471s","remote":"127.0.0.1:49848","response type":"/etcdserverpb.Maintenance/Defragment","request count":-1,"request size":-1,"response count":-1,"response size":-1,"request content":""}
{"level":"info","ts":"2023-02-28T05:08:45.491Z","caller":"traceutil/trace.go:171","msg":"trace[1862721267] linearizableReadLoop","detail":"{readStateIndex:890; appliedIndex:890; }","duration":"200.915946ms","start":"2023-02-28T05:08:45.290Z","end":"2023-02-28T05:08:45.491Z","steps":["trace[1862721267] 'read index received' (duration: 200.907674ms)","trace[1862721267] 'applied index is now lower than readState.Index' (duration: 6.043µs)"],"step_count":2}
INFO[0011] etcd temporary data store connection OK
INFO[0011] Reconciling bootstrap data between datastore and disk
INFO[0011] stopping etcd
{"level":"info","ts":"2023-02-28T05:08:45.554Z","caller":"embed/etcd.go:368","msg":"closing etcd server","name":"test-0010ecc4","data-dir":"/var/lib/rancher/rke2/server/db/etcd-tmp","advertise-peer-urls":["<http://127.0.0.1:2400>"],"advertise-client-urls":["<http://127.0.0.1:2399>"]}
{"level":"info","ts":"2023-02-28T05:08:45.554Z","caller":"etcdserver/server.go:1453","msg":"skipped leadership transfer for single voting member cluster","local-member-id":"8b8e4494dd110009","current-leader-member-id":"8b8e4494dd110009"}
FATA[0011] starting kubernetes: preparing server: init cluster datastore and https: listen tcp :9345: bind: address already in use
then i ran the command with the right flags:
rke2 server --server=<https://192.168.3.180:9345> --token=rke2-demo
and i got the error of the token:
FATA[0021] Failed to reconcile with temporary etcd: bootstrap data already found and encrypted with different token
i thought maybe that the error from my token, so i used the generated token:
rke2 server --server=<https://192.168.3.180:9345> --token=K10184dcfa4851e487eaa1756e8bedd18f03cb8c6b204dccf2c27090ba9e699cab5::server:1461a0c608734bad8e13b92971e1e09b
and i got the error of new certificates:
FATA[0000]newer than datastore and could cause a cluster outage. Remove the file(s) from disk and restart to be recreated from datastore.
so i deleted them, and tried again, and this is the error that i got:
FATA[0000] starting kubernetes: preparing server: init cluster datastore and https: listen tcp :9345: bind: address already in use
abundant-noon-17295
03/02/2023, 2:39 AMrke2-server
running, and you are trying to start a new process with rke2 server
using the same listening portworried-ram-81084
03/02/2023, 6:12 AM