This message was deleted Rancher Users #rke2

Join Slack

This message was deleted.

# rke2

adamant-kite-43734

10/17/2022, 4:42 PM

This message was deleted.

rich-crowd-36987

10/17/2022, 6:08 PM

On Master 1, if I keep the original

/var/lib/rancher/rke2

directory, I get the following panic:

Copy code

Oct 17 18:05:01 <http://k8mst01.espc-nostromo.nos-amc.io|k8mst01.espc-nostromo.nos-amc.io> rke2[27972]: {"level":"info","ts":"2022-10-17T18:05:01.589Z","caller":"rafthttp/pipeline.go:86","msg":"stopped HTTP pipelining with remote peer","local-member-id":"fd35c5c2c37ef2ee","remote-peer-id":"42626d1a170fc6f5"}
Oct 17 18:05:01 <http://k8mst01.espc-nostromo.nos-amc.io|k8mst01.espc-nostromo.nos-amc.io> rke2[27972]: {"level":"info","ts":"2022-10-17T18:05:01.589Z","caller":"rafthttp/stream.go:459","msg":"stopped stream reader with remote peer","stream-reader-type":"stream MsgApp v2","local-member-id":"fd35c5c2c37ef2ee","remote-peer-id":"42626d1a170fc6f5"}
Oct 17 18:05:01 <http://k8mst01.espc-nostromo.nos-amc.io|k8mst01.espc-nostromo.nos-amc.io> rke2[27972]: {"level":"info","ts":"2022-10-17T18:05:01.589Z","caller":"rafthttp/stream.go:459","msg":"stopped stream reader with remote peer","stream-reader-type":"stream Message","local-member-id":"fd35c5c2c37ef2ee","remote-peer-id":"42626d1a170fc6f5"}
Oct 17 18:05:01 <http://k8mst01.espc-nostromo.nos-amc.io|k8mst01.espc-nostromo.nos-amc.io> rke2[27972]: {"level":"info","ts":"2022-10-17T18:05:01.589Z","caller":"rafthttp/peer.go:340","msg":"stopped remote peer","remote-peer-id":"42626d1a170fc6f5"}
Oct 17 18:05:01 <http://k8mst01.espc-nostromo.nos-amc.io|k8mst01.espc-nostromo.nos-amc.io> rke2[27972]: {"level":"info","ts":"2022-10-17T18:05:01.589Z","caller":"rafthttp/transport.go:369","msg":"removed remote peer","local-member-id":"fd35c5c2c37ef2ee","removed-remote-peer-id":"42626d1a170fc6f5"}
Oct 17 18:05:01 <http://k8mst01.espc-nostromo.nos-amc.io|k8mst01.espc-nostromo.nos-amc.io> rke2[27972]: panic: removed all voters
Oct 17 18:05:01 <http://k8mst01.espc-nostromo.nos-amc.io|k8mst01.espc-nostromo.nos-amc.io> rke2[27972]: goroutine 206 [running]:
Oct 17 18:05:01 <http://k8mst01.espc-nostromo.nos-amc.io|k8mst01.espc-nostromo.nos-amc.io> rke2[27972]: <http://go.etcd.io/etcd/raft.(*raft).applyConfChange(0xc0001c0500|go.etcd.io/etcd/raft.(*raft).applyConfChange(0xc0001c0500>, 0x0, 0xc00091dbf0, 0x1, 0x1, 0x0, 0x0, 0x0, 0x0, 0x0, ...)
Oct 17 18:05:01 <http://k8mst01.espc-nostromo.nos-amc.io|k8mst01.espc-nostromo.nos-amc.io> rke2[27972]: /go/pkg/mod/github.com/k3s-io/etcd@v0.5.0-alpha.5.0.20220113195313-6c2233a709e8/raft/raft.go:1514 +0x225
Oct 17 18:05:01 <http://k8mst01.espc-nostromo.nos-amc.io|k8mst01.espc-nostromo.nos-amc.io> rke2[27972]: <http://go.etcd.io/etcd/raft.(*node).run(0xc0014440c0)|go.etcd.io/etcd/raft.(*node).run(0xc0014440c0)>
Oct 17 18:05:01 <http://k8mst01.espc-nostromo.nos-amc.io|k8mst01.espc-nostromo.nos-amc.io> rke2[27972]: /go/pkg/mod/github.com/k3s-io/etcd@v0.5.0-alpha.5.0.20220113195313-6c2233a709e8/raft/node.go:356 +0x845
Oct 17 18:05:01 <http://k8mst01.espc-nostromo.nos-amc.io|k8mst01.espc-nostromo.nos-amc.io> rke2[27972]: created by <http://go.etcd.io/etcd/raft.RestartNode|go.etcd.io/etcd/raft.RestartNode>
Oct 17 18:05:01 <http://k8mst01.espc-nostromo.nos-amc.io|k8mst01.espc-nostromo.nos-amc.io> rke2[27972]: /go/pkg/mod/github.com/k3s-io/etcd@v0.5.0-alpha.5.0.20220113195313-6c2233a709e8/raft/node.go:240 +0x330
Oct 17 18:05:01 <http://k8mst01.espc-nostromo.nos-amc.io|k8mst01.espc-nostromo.nos-amc.io> systemd[1]: rke2-server.service: main process exited, code=exited, status=2/INVALIDARGUMENT

rich-crowd-36987

10/17/2022, 9:09 PM

Ultimately got this working. Not sure what the extra difficultly was this time, but I had to use

etcdctl

to manually promote the learner node, and on my first master node also had to include the

server:

cfg option (and eventually promote it too when it tried to join the existing cluster). But here I am now with three master nodes again, so hooray.

212 Views

Open in Slack

Previous Next