This message was deleted.
# general
a
This message was deleted.
f
Jul 27 193034 copula-lab-1.cpaas.sifox.ru k3s[509753]: {"level":"info","ts":"2024-07-27T193034.161459+0300","caller":"rafthttp/peer.go:330","msg":"stopping remote peer","remote-peer-id":"bb35ada2b12df342"} Jul 27 193034 copula-lab-1.cpaas.sifox.ru k3s[509753]: {"level":"info","ts":"2024-07-27T193034.161534+0300","caller":"rafthttp/stream.go:294","msg":"stopped TCP streaming connection with remote peer","stream-writer-type":"unknown stream","remote-peer-id":"bb35ada2b12df342"} Jul 27 193034 copula-lab-1.cpaas.sifox.ru k3s[509753]: {"level":"info","ts":"2024-07-27T193034.161656+0300","caller":"rafthttp/stream.go:294","msg":"stopped TCP streaming connection with remote peer","stream-writer-type":"unknown stream","remote-peer-id":"bb35ada2b12df342"} Jul 27 193034 copula-lab-1.cpaas.sifox.ru k3s[509753]: {"level":"info","ts":"2024-07-27T193034.161866+0300","caller":"rafthttp/pipeline.go:85","msg":"stopped HTTP pipelining with remote peer","local-member-id":"d176b6176e13d9b4","remote-peer-id":"bb35ada2b12df342"} Jul 27 193034 copula-lab-1.cpaas.sifox.ru k3s[509753]: {"level":"info","ts":"2024-07-27T193034.161971+0300","caller":"rafthttp/stream.go:442","msg":"stopped stream reader with remote peer","stream-reader-type":"stream MsgApp v2","local-member-id":"d176b6176e13d9b4","remote-peer-id":"bb35ada2b12df342"} Jul 27 193034 copula-lab-1.cpaas.sifox.ru k3s[509753]: {"level":"info","ts":"2024-07-27T193034.162068+0300","caller":"rafthttp/stream.go:442","msg":"stopped stream reader with remote peer","stream-reader-type":"stream Message","local-member-id":"d176b6176e13d9b4","remote-peer-id":"bb35ada2b12df342"} Jul 27 193034 copula-lab-1.cpaas.sifox.ru k3s[509753]: {"level":"info","ts":"2024-07-27T193034.162151+0300","caller":"rafthttp/peer.go:335","msg":"stopped remote peer","remote-peer-id":"bb35ada2b12df342"} Jul 27 193034 copula-lab-1.cpaas.sifox.ru k3s[509753]: {"level":"info","ts":"2024-07-27T193034.162218+0300","caller":"rafthttp/transport.go:355","msg":"removed remote peer","local-member-id":"d176b6176e13d9b4","removed-remote-peer-id":"bb35ada2b12df342"} Jul 27 193034 copula-lab-1.cpaas.sifox.ru k3s[509753]: panic: removed all voters Jul 27 193034 copula-lab-1.cpaas.sifox.ru k3s[509753]: goroutine 155 [running]: Jul 27 193034 copula-lab-1.cpaas.sifox.ru k3s[509753]: go.etcd.io/etcd/raft/v3.(*raft).applyConfChange(0x0?, {0x0, {0xc003e7a4c0, 0x1, 0x1}, {0x0, 0x0, 0x0}}) Jul 27 193034 copula-lab-1.cpaas.sifox.ru k3s[509753]: /go/pkg/mod/github.com/k3s-io/etcd/raft/v3@v3.5.9-k3s1/raft.go:1633 +0x1d4 Jul 27 193034 copula-lab-1.cpaas.sifox.ru k3s[509753]: go.etcd.io/etcd/raft/v3.(*node).run(0xc00105c420) Jul 27 193034 copula-lab-1.cpaas.sifox.ru k3s[509753]: /go/pkg/mod/github.com/k3s-io/etcd/raft/v3@v3.5.9-k3s1/node.go:360 +0xaf7 Jul 27 193034 copula-lab-1.cpaas.sifox.ru k3s[509753]: created by go.etcd.io/etcd/raft/v3.RestartNode Jul 27 193034 copula-lab-1.cpaas.sifox.ru k3s[509753]: /go/pkg/mod/github.com/k3s-io/etcd/raft/v3@v3.5.9-k3s1/node.go:244 +0x24a Jul 27 193034 copula-lab-1.cpaas.sifox.ru systemd[1]: k3s.service: Main process exited, code=exited, status=2/INVALIDARGUMENT Jul 27 193034 copula-lab-1.cpaas.sifox.ru systemd[1]: k3s.service: Failed with result 'exit-code'. -- Subject: Unit failed -- Defined-By: systemd
Copy code
I tried to restore it using this command
[root@copula-lab-1 db]# ls -la total 20 drwx------ 5 root root 4096 Jul 27 15:35 . drwx------. 8 root root 4096 Jul 27 11:35 .. drwx------ 3 root root 4096 Jul 27 15:35 etcd drwx------ 3 root root 4096 Jul 27 15:35 etcd-tmp drwx------ 2 root root 4096 Jul 27 15:35 snapshots [root@copula-lab-1 db]# [root@copula-lab-1 db]# [root@copula-lab-1 db]# [root@copula-lab-1 db]# [root@copula-lab-1 db]# k3s server --cluster-reset --cluster-reset-restore-path=/var/lib/rancher/k3s/server/db/snapshots/etcd-snapshot-copula-lab-1.cpaas.sifox.ru-1721854802 WARN[0000] remove /var/lib/rancher/k3s/agent/etc/k3s-agent-load-balancer.json: no such file or directory WARN[0000] remove /var/lib/rancher/k3s/agent/etc/k3s-api-server-agent-load-balancer.json: no such file or directory INFO[0000] Starting k3s v1.28.3+k3s2 (bbafb86e) INFO[0000] Managed etcd cluster bootstrap already complete and initialized INFO[0000] Starting temporary etcd to reconcile with datastore {"level":"info","ts":"2024-07-27T153530.328902+0300","caller":"embed/etcd.go:127","msg":"configuring peer listeners","listen-peer-urls":["http://127.0.0.1:2400"]} {"level":"info","ts":"2024-07-27T153530.329437+0300","caller":"embed/etcd.go:135","msg":"configuring client listeners","listen-client-urls":["http://127.0.0.1:2399"]} {"level":"info","ts":"2024-07-27T153530.329741+0300","caller":"embed/etcd.go:309","msg":"starting an etcd server","etcd-version":"3.5.9","git-sha":"Not provided (use ./build instead of go build)","go-version":"go1.20.10","go-os":"linux","go-arch":"amd64","max-cpu-set":6,"max-cpu-available":6,"member-initialized":true,"name":"copula-lab-1.cpaas.sifox.ru-c619a0ca","data-dir":"/var/lib/rancher/k3s/server/db/etcd-tmp","wal-dir":"","wal-dir-dedicated":"","member-dir":"/var/lib/rancher/k3s/server/db/etcd-tmp/member","force-new-cluster":true,"heartbeat-interval":"500ms","election-timeout":"5s","initial-election-tick-advance":true,"snapshot-count":10000,"max-wals":0,"max-snapshots":0,"snapshot-catchup-entries":5000,"initial-advertise-peer-urls":["http://127.0.0.1:2400"],"listen-peer-urls":["http://127.0.0.1:2400"],"advertise-client-urls":["http://127.0.0.1:2399"],"listen-client-urls":["http://127.0.0.1:2399"],"listen-metrics-urls":[],"cors":["*"],"host-whitelist":["*"],"initial-cluster":"","initial-cluster-state":"new","initial-cluster-token":"","quota-backend-bytes":2147483648,"max-request-bytes":1572864,"max-concurrent-streams":4294967295,"pre-vote":true,"initial-corrupt-check":true,"corrupt-check-time-interval":"0s","compact-check-time-enabled":false,"compact-check-time-interval":"1m0s","auto-compaction-mode":"","auto-compaction-retention":"0s","auto-compaction-interval":"0s","discovery-url":"","discovery-proxy":"","downgrade-check-interval":"5s"} {"level":"info","ts":"2024-07-27T153530.344058+0300","caller":"etcdserver/backend.go:81","msg":"opened backend db","path":"/var/lib/rancher/k3s/server/db/etcd-tmp/member/snap/db","took":"13.852023ms"}