https://rancher.com/ logo
Join Slack
Powered by
This message was deleted.
# k3s
a
This message was deleted.
n
Yes the commands you give will work, but the recommended upgrade process is: 1. Upgrade to the latest version of 1.28 on all nodes 2. Read the https://github.com/kubernetes/kubernetes/blob/master/CHANGELOG/CHANGELOG-1.29.md#urgent-upgrade-notes to make sure nothing in your current cluster is affected by upstream changes between 1.28->1.29 3. Upgrade your nodes to the latest version of 1.29
l
all 3 nodes have etcd aswell, when i do in 1st master node, it is fine but rest of the two node goes to not ready state, with below logs,
Copy code
Mar 14 17:25:40 Master-1 k3s[269250]: {"level":"info","ts":"2024-03-14T17:25:40.985Z","caller":"fileutil/purge.go:44","msg":"started to purge file","dir":"/var/lib/etcd/member/snap","suffix":"snap.db","max":5,"interval":"3>
Mar 14 17:25:40 Master-1 k3s[269250]: {"level":"info","ts":"2024-03-14T17:25:40.985Z","caller":"fileutil/purge.go:44","msg":"started to purge file","dir":"/var/lib/etcd/member/snap","suffix":"snap","max":5,"interval":"30s"}
Mar 14 17:25:40 Master-1 k3s[269250]: {"level":"info","ts":"2024-03-14T17:25:40.985Z","caller":"fileutil/purge.go:44","msg":"started to purge file","dir":"/var/lib/etcd/member/wal","suffix":"wal","max":5,"interval":"30s"}
Mar 14 17:25:40 Master-1 k3s[269250]: {"level":"info","ts":"2024-03-14T17:25:40.985Z","caller":"rafthttp/peer.go:330","msg":"stopping remote peer","remote-peer-id":"4a044d0d43c4b92"}
Mar 14 17:25:40 Master-1 k3s[269250]: {"level":"info","ts":"2024-03-14T17:25:40.985Z","caller":"rafthttp/stream.go:294","msg":"stopped TCP streaming connection with remote peer","stream-writer-type":"unknown stream","remot>
Mar 14 17:25:40 Master-1 k3s[269250]: {"level":"info","ts":"2024-03-14T17:25:40.985Z","caller":"rafthttp/stream.go:294","msg":"stopped TCP streaming connection with remote peer","stream-writer-type":"unknown stream","remot>
Mar 14 17:25:40 Master-1 k3s[269250]: {"level":"info","ts":"2024-03-14T17:25:40.985Z","caller":"rafthttp/pipeline.go:85","msg":"stopped HTTP pipelining with remote peer","local-member-id":"8f50b72f426d4b33","remote-peer-id>
Mar 14 17:25:40 Master-1 k3s[269250]: {"level":"info","ts":"2024-03-14T17:25:40.985Z","caller":"rafthttp/stream.go:442","msg":"stopped stream reader with remote peer","stream-reader-type":"stream MsgApp v2","local-member-i>
Mar 14 17:25:40 Master-1 k3s[269250]: {"level":"info","ts":"2024-03-14T17:25:40.985Z","caller":"rafthttp/stream.go:442","msg":"stopped stream reader with remote peer","stream-reader-type":"stream Message","local-member-id">
Mar 14 17:25:40 Master-1 k3s[269250]: {"level":"info","ts":"2024-03-14T17:25:40.985Z","caller":"rafthttp/peer.go:335","msg":"stopped remote peer","remote-peer-id":"4a044d0d43c4b92"}
Mar 14 17:25:40 Master-1 k3s[269250]: {"level":"info","ts":"2024-03-14T17:25:40.985Z","caller":"rafthttp/peer.go:330","msg":"stopping remote peer","remote-peer-id":"ed0f018489f80367"}
Mar 14 17:25:40 Master-1 k3s[269250]: {"level":"info","ts":"2024-03-14T17:25:40.985Z","caller":"rafthttp/stream.go:294","msg":"stopped TCP streaming connection with remote peer","stream-writer-type":"unknown stream","remot>
Mar 14 17:25:40 Master-1 k3s[269250]: {"level":"info","ts":"2024-03-14T17:25:40.985Z","caller":"rafthttp/stream.go:294","msg":"stopped TCP streaming connection with remote peer","stream-writer-type":"unknown stream","remot>
Mar 14 17:25:40 Master-1 k3s[269250]: {"level":"info","ts":"2024-03-14T17:25:40.985Z","caller":"rafthttp/pipeline.go:85","msg":"stopped HTTP pipelining with remote peer","local-member-id":"8f50b72f426d4b33","remote-peer-id>
Mar 14 17:25:40 Master-1 k3s[269250]: {"level":"info","ts":"2024-03-14T17:25:40.985Z","caller":"rafthttp/stream.go:442","msg":"stopped stream reader with remote peer","stream-reader-type":"stream MsgApp v2","local-member-i>
Mar 14 17:25:40 Master-1 k3s[269250]: {"level":"info","ts":"2024-03-14T17:25:40.985Z","caller":"rafthttp/stream.go:442","msg":"stopped stream reader with remote peer","stream-reader-type":"stream Message","local-member-id">
Mar 14 17:25:40 Master-1 k3s[269250]: {"level":"info","ts":"2024-03-14T17:25:40.985Z","caller":"rafthttp/peer.go:335","msg":"stopped remote peer","remote-peer-id":"ed0f018489f80367"}
Mar 14 17:25:40 Master-1 k3s[269250]: {"level":"warn","ts":"2024-03-14T17:25:40.986Z","caller":"etcdserver/server.go:2747","msg":"server has stopped; skipping GoAttach"}
Mar 14 17:25:40 Master-1 k3s[269250]: {"level":"info","ts":"2024-03-14T17:25:40.987Z","caller":"embed/etcd.go:687","msg":"starting with client TLS","tls-info":"cert = /application-volume/k3s/server/tls/etcd/server-client.c>
Mar 14 17:25:40 Master-1 k3s[269250]: {"level":"info","ts":"2024-03-14T17:25:40.987Z","caller":"embed/etcd.go:586","msg":"serving peer traffic","address":"127.0.0.1:2380"}
Mar 14 17:25:40 Master-1 k3s[269250]: {"level":"info","ts":"2024-03-14T17:25:40.987Z","caller":"embed/etcd.go:586","msg":"serving peer traffic","address":"10.50.0.13:2380"}
Mar 14 17:25:40 Master-1 k3s[269250]: {"level":"info","ts":"2024-03-14T17:25:40.987Z","caller":"embed/etcd.go:558","msg":"cmux::serve","address":"10.50.0.13:2380"}
Mar 14 17:25:40 Master-1 k3s[269250]: {"level":"info","ts":"2024-03-14T17:25:40.987Z","caller":"embed/etcd.go:558","msg":"cmux::serve","address":"127.0.0.1:2380"}
Mar 14 17:25:40 Master-1 k3s[269250]: {"level":"warn","ts":"2024-03-14T17:25:40.988Z","caller":"rafthttp/http.go:145","msg":"failed to process Raft message","local-member-id":"8f50b72f426d4b33","error":"raft: stopped"}
Mar 14 17:25:40 Master-1 k3s[269250]: {"level":"warn","ts":"2024-03-14T17:25:40.988Z","caller":"rafthttp/http.go:413","msg":"failed to find remote peer in cluster","local-member-id":"8f50b72f426d4b33","remote-peer-id-strea>
Mar 14 17:25:40 Master-1 k3s[269250]: {"level":"info","ts":"2024-03-14T17:25:40.988Z","caller":"embed/etcd.go:275","msg":"now serving peer/client/metrics","local-member-id":"8f50b72f426d4b33","initial-advertise-peer-urls":>
Mar 14 17:25:40 Master-1 k3s[269250]: {"level":"info","ts":"2024-03-14T17:25:40.988Z","caller":"embed/etcd.go:762","msg":"serving metrics","address":"<http://127.0.0.1:2381>"}
Mar 14 17:25:40 Master-1 k3s[269250]: {"level":"warn","ts":"2024-03-14T17:25:40.988Z","caller":"rafthttp/http.go:413","msg":"failed to find remote peer in cluster","local-member-id":"8f50b72f426d4b33","remote-peer-id-strea>
Mar 14 17:25:40 Master-1 k3s[269250]: {"level":"warn","ts":"2024-03-14T17:25:40.992Z","caller":"rafthttp/http.go:413","msg":"failed to find remote peer in cluster","local-member-id":"8f50b72f426d4b33","remote-peer-id-strea>
Mar 14 17:25:40 Master-1 k3s[269250]: {"level":"warn","ts":"2024-03-14T17:25:40.992Z","caller":"rafthttp/http.go:413","msg":"failed to find remote peer in cluster","local-member-id":"8f50b72f426d4b33","remote-peer-id-strea>
Mar 14 17:25:40 Master-1 k3s[269250]: time="2024-03-14T17:25:40Z" level=fatal msg="etcd stopped"
used this command only
Copy code
curl -sfL <https://get.k3s.io> | INSTALL_K3S_EXEC='server --cluster-init --tls-san <public-ip> --data-dir /application-volume/k3s --write-kubeconfig-mode 644 --etcd-arg=--data-dir=/var/lib/etcd --etcd-arg=--debug --disable traefik' INSTALL_K3S_VERSION='v1.28.7+k3s1' sh -
so in my case etcd getting removed in other two nodes, how to avoid this ?
n
Are you stopping the k3s service before running these commands?
The manual install docs page need work I'm now realizing. But basically: • Stop the existing K3s services on all nodes • Run the install script commands
It looks like your not stopping the commands, so you have two instances of the k3s trying to run on the same node
l
Copy code
Mar 14 17:25:40 Master-1 k3s[269250]: {"level":"info","ts":"2024-03-14T17:25:40.985Z","caller":"fileutil/purge.go:44","msg":"started to purge file","dir":"/var/lib/etcd/member/snap","suffix":"snap.db","max":5,"interval":"3>
Mar 14 17:25:40 Master-1 k3s[269250]: {"level":"info","ts":"2024-03-14T17:25:40.985Z","caller":"fileutil/purge.go:44","msg":"started to purge file","dir":"/var/lib/etcd/member/snap","suffix":"snap","max":5,"interval":"30s"}
Mar 14 17:25:40 Master-1 k3s[269250]: {"level":"info","ts":"2024-03-14T17:25:40.985Z","caller":"fileutil/purge.go:44","msg":"started to purge file","dir":"/var/lib/etcd/member/wal","suffix":"wal","max":5,"interval":"30s"}
why the purge happening is this expected to make the notready to up, i need to delete the etcd dir and restart the k3s right !!!
Copy code
2]: {"level":"info","ts":"2024-03-14T23:40:18.376Z","caller":"etcdserver/server.go:754","msg":"starting initial election tick advance","election-ticks":10}
Mar 14 23:40:18 Master-1 k3s[322072]: {"level":"info","ts":"2024-03-14T23:40:18.376Z","caller":"fileutil/purge.go:44","msg":"started to purge file","dir":"/var/lib/etcd/member/snap","suffix":"snap.db","max":5,"interv>
Mar 14 23:40:18 Master-1 k3s[322072]: {"level":"warn","ts":"2024-03-14T23:40:18.376Z","caller":"etcdserver/server.go:1127","msg":"server error","error":"the member has been permanently removed from the cluster"}
Mar 14 23:40:18 Master-1 k3s[322072]: {"level":"warn","ts":"2024-03-14T23:40:18.376Z","caller":"etcdserver/server.go:1128","msg":"data-dir used by this member must be removed"}
Mar 14 23:40:18 Master-1 k3s[322072]: {"level":"info","ts":"2024-03-14T23:40:18.376Z","caller":"fileutil/purge.go:44","msg":"started to purge file","dir":"/var/lib/etcd/member/snap","suffix":"snap","max":5,"interval">
Mar 14 23:40:18 Master-1 k3s[322072]: {"level":"warn","ts":"2024-03-14T23:40:18.376Z","caller":"etcdserver/server.go:2085","msg":"failed to publish local member to cluster through raft","local-member-id":"3eda39509b1>
Mar 14 23:40:18 Master-1 k3s[322072]: {"level":"warn","ts":"2024-03-14T23:40:18.376Z","caller":"etcdserver/server.go:2747","msg":"server has stopped; skipping GoAttach"}
Mar 14 23:40:18 Master-1 k3s[322072]: {"level":"warn","ts":"2024-03-14T23:40:18.376Z","caller":"etcdserver/server.go:2075","msg":"stopped publish because server is stopped","local-member-id":"3eda39509b1cb116","local>
Mar 14 23:40:18 Master-1 k3s[322072]: {"level":"info","ts":"2024-03-14T23:40:18.376Z","caller":"fileutil/purge.go:44","msg":"started to purge file","dir":"/var/lib/etcd/member/wal","suffix":"wal","max":5,"interval":">
Mar 14 23:40:18 Master-1 k3s[322072]: {"level":"info","ts":"2024-03-14T23:40:18.377Z","caller":"rafthttp/peer.go:330","msg":"stopping remote peer","remote-peer-id":"ed0f018489f80367"}
Mar 14 23:40:18 Master-1 k3s[322072]: {"level":"info","ts":"2024-03-14T23:40:18.377Z","caller":"rafthttp/stream.go:294","msg":"stopped TCP streaming connection with remote peer","stream-writer-type":"unknown stream",>
Mar 14 23:40:18 Master-1 k3s[322072]: {"level":"info","ts":"2024-03-14T23:40:18.377Z","caller":"rafthttp/stream.go:294","msg":"stopped TCP streaming connection with remote peer","stream-writer-type":"unknown stream",>
Mar 14 23:40:18 Master-1 k3s[322072]: {"level":"info","ts":"2024-03-14T23:40:18.377Z","caller":"rafthttp/pipeline.go:85","msg":"stopped HTTP pipelining with remote peer","local-member-id":"3eda39509b1cb116","remote-p>
Mar 14 23:40:18 Master-1 k3s[322072]: {"level":"info","ts":"2024-03-14T23:40:18.377Z","caller":"rafthttp/stream.go:442","msg":"stopped stream reader with remote peer","stream-reader-type":"stream MsgApp v2","local-me>
Mar 14 23:40:18 Master-1 k3s[322072]: {"level":"info","ts":"2024-03-14T23:40:18.377Z","caller":"rafthttp/stream.go:442","msg":"stopped stream reader with remote peer","stream-reader-type":"stream Message","local-memb>
Mar 14 23:40:18 Master-1 k3s[322072]: {"level":"info","ts":"2024-03-14T23:40:18.377Z","caller":"rafthttp/peer.go:335","msg":"stopped remote peer","remote-peer-id":"ed0f018489f80367"}
Mar 14 23:40:18 Master-1 k3s[322072]: {"level":"warn","ts":"2024-03-14T23:40:18.378Z","caller":"etcdserver/server.go:2747","msg":"server has stopped; skipping GoAttach"}
Mar 14 23:40:18 Master-1 k3s[322072]: {"level":"info","ts":"2024-03-14T23:40:18.378Z","caller":"embed/etcd.go:687","msg":"starting with client TLS","tls-info":"cert = /application-volume/k3s/server/tls/etcd/server-cl>
Mar 14 23:40:18 Master-1 k3s[322072]: {"level":"info","ts":"2024-03-14T23:40:18.378Z","caller":"embed/etcd.go:586","msg":"serving peer traffic","address":"127.0.0.1:2380"}
Mar 14 23:40:18 Master-1 k3s[322072]: {"level":"info","ts":"2024-03-14T23:40:18.379Z","caller":"embed/etcd.go:558","msg":"cmux::serve","address":"127.0.0.1:2380"}
Mar 14 23:40:18 Master-1 k3s[322072]: {"level":"info","ts":"2024-03-14T23:40:18.378Z","caller":"embed/etcd.go:586","msg":"serving peer traffic","address":"10.50.0.13:2380"}
Mar 14 23:40:18 Master-1 k3s[322072]: {"level":"info","ts":"2024-03-14T23:40:18.379Z","caller":"embed/etcd.go:558","msg":"cmux::serve","address":"10.50.0.13:2380"}
Mar 14 23:40:18 Master-1 k3s[322072]: {"level":"warn","ts":"2024-03-14T23:40:18.379Z","caller":"rafthttp/http.go:413","msg":"failed to find remote peer in cluster","local-member-id":"3eda39509b1cb116","remote-peer-id>
Mar 14 23:40:18 Master-1 k3s[322072]: {"level":"info","ts":"2024-03-14T23:40:18.379Z","caller":"embed/etcd.go:275","msg":"now serving peer/client/metrics","local-member-id":"3eda39509b1cb116","initial-advertise-peer->
Mar 14 23:40:18 Master-1 k3s[322072]: {"level":"info","ts":"2024-03-14T23:40:18.379Z","caller":"embed/etcd.go:762","msg":"serving metrics","address":"<http://127.0.0.1:2381>"}
Mar 14 23:40:18 Master-1 k3s[322072]: {"level":"warn","ts":"2024-03-14T23:40:18.379Z","caller":"rafthttp/http.go:413","msg":"failed to find remote peer in cluster","local-member-id":"3eda39509b1cb116","remote-peer-id>
Mar 14 23:40:18 Master-1 k3s[322072]: time="2024-03-14T23:40:18Z" level=fatal msg="etcd stopped"
i have made all 3 nodes up and running and confirmed etcd health is fine, then i have stopped the nodes and upgraded the leader node with above cluster-init with new version k3s, that is fine [1st node] in 2nd and third node, the member is removed when i start the k3s !!!, why this is happening
cc @creamy-pencil-82913
i can see the issue is similar to https://github.com/k3s-io/k3s/issues/7613 but in my case all 3 nodes have same version only, and the final comment is talking about syslink [can anyone say what is the fix] what is the exact steps to upgrade multi-master node cluster [with embedded etcd], can anyone share me the steps i dont see in manual upgrade page it has sufficient details, cc @creamy-pencil-82913 @nutritious-tomato-14686
13 Views
Open in Slack
PreviousNext
Powered by