https://rancher.com/ logo
#rke2
Title
# rke2
a

adamant-kite-43734

10/17/2022, 4:42 PM
This message was deleted.
r

rich-crowd-36987

10/17/2022, 6:08 PM
On Master 1, if I keep the original
/var/lib/rancher/rke2
directory, I get the following panic:
Copy code
Oct 17 18:05:01 <http://k8mst01.espc-nostromo.nos-amc.io|k8mst01.espc-nostromo.nos-amc.io> rke2[27972]: {"level":"info","ts":"2022-10-17T18:05:01.589Z","caller":"rafthttp/pipeline.go:86","msg":"stopped HTTP pipelining with remote peer","local-member-id":"fd35c5c2c37ef2ee","remote-peer-id":"42626d1a170fc6f5"}
Oct 17 18:05:01 <http://k8mst01.espc-nostromo.nos-amc.io|k8mst01.espc-nostromo.nos-amc.io> rke2[27972]: {"level":"info","ts":"2022-10-17T18:05:01.589Z","caller":"rafthttp/stream.go:459","msg":"stopped stream reader with remote peer","stream-reader-type":"stream MsgApp v2","local-member-id":"fd35c5c2c37ef2ee","remote-peer-id":"42626d1a170fc6f5"}
Oct 17 18:05:01 <http://k8mst01.espc-nostromo.nos-amc.io|k8mst01.espc-nostromo.nos-amc.io> rke2[27972]: {"level":"info","ts":"2022-10-17T18:05:01.589Z","caller":"rafthttp/stream.go:459","msg":"stopped stream reader with remote peer","stream-reader-type":"stream Message","local-member-id":"fd35c5c2c37ef2ee","remote-peer-id":"42626d1a170fc6f5"}
Oct 17 18:05:01 <http://k8mst01.espc-nostromo.nos-amc.io|k8mst01.espc-nostromo.nos-amc.io> rke2[27972]: {"level":"info","ts":"2022-10-17T18:05:01.589Z","caller":"rafthttp/peer.go:340","msg":"stopped remote peer","remote-peer-id":"42626d1a170fc6f5"}
Oct 17 18:05:01 <http://k8mst01.espc-nostromo.nos-amc.io|k8mst01.espc-nostromo.nos-amc.io> rke2[27972]: {"level":"info","ts":"2022-10-17T18:05:01.589Z","caller":"rafthttp/transport.go:369","msg":"removed remote peer","local-member-id":"fd35c5c2c37ef2ee","removed-remote-peer-id":"42626d1a170fc6f5"}
Oct 17 18:05:01 <http://k8mst01.espc-nostromo.nos-amc.io|k8mst01.espc-nostromo.nos-amc.io> rke2[27972]: panic: removed all voters
Oct 17 18:05:01 <http://k8mst01.espc-nostromo.nos-amc.io|k8mst01.espc-nostromo.nos-amc.io> rke2[27972]: goroutine 206 [running]:
Oct 17 18:05:01 <http://k8mst01.espc-nostromo.nos-amc.io|k8mst01.espc-nostromo.nos-amc.io> rke2[27972]: <http://go.etcd.io/etcd/raft.(*raft).applyConfChange(0xc0001c0500|go.etcd.io/etcd/raft.(*raft).applyConfChange(0xc0001c0500>, 0x0, 0xc00091dbf0, 0x1, 0x1, 0x0, 0x0, 0x0, 0x0, 0x0, ...)
Oct 17 18:05:01 <http://k8mst01.espc-nostromo.nos-amc.io|k8mst01.espc-nostromo.nos-amc.io> rke2[27972]: /go/pkg/mod/github.com/k3s-io/etcd@v0.5.0-alpha.5.0.20220113195313-6c2233a709e8/raft/raft.go:1514 +0x225
Oct 17 18:05:01 <http://k8mst01.espc-nostromo.nos-amc.io|k8mst01.espc-nostromo.nos-amc.io> rke2[27972]: <http://go.etcd.io/etcd/raft.(*node).run(0xc0014440c0)|go.etcd.io/etcd/raft.(*node).run(0xc0014440c0)>
Oct 17 18:05:01 <http://k8mst01.espc-nostromo.nos-amc.io|k8mst01.espc-nostromo.nos-amc.io> rke2[27972]: /go/pkg/mod/github.com/k3s-io/etcd@v0.5.0-alpha.5.0.20220113195313-6c2233a709e8/raft/node.go:356 +0x845
Oct 17 18:05:01 <http://k8mst01.espc-nostromo.nos-amc.io|k8mst01.espc-nostromo.nos-amc.io> rke2[27972]: created by <http://go.etcd.io/etcd/raft.RestartNode|go.etcd.io/etcd/raft.RestartNode>
Oct 17 18:05:01 <http://k8mst01.espc-nostromo.nos-amc.io|k8mst01.espc-nostromo.nos-amc.io> rke2[27972]: /go/pkg/mod/github.com/k3s-io/etcd@v0.5.0-alpha.5.0.20220113195313-6c2233a709e8/raft/node.go:240 +0x330
Oct 17 18:05:01 <http://k8mst01.espc-nostromo.nos-amc.io|k8mst01.espc-nostromo.nos-amc.io> systemd[1]: rke2-server.service: main process exited, code=exited, status=2/INVALIDARGUMENT
Ultimately got this working. Not sure what the extra difficultly was this time, but I had to use
etcdctl
to manually promote the learner node, and on my first master node also had to include the
server:
cfg option (and eventually promote it too when it tried to join the existing cluster). But here I am now with three master nodes again, so hooray.
176 Views