https://rancher.com/ logo
#rke2
Title
# rke2
b

billions-easter-91774

06/16/2022, 7:17 PM
i upgraded my ubuntu 20.04 to 5.15 kernel and (perhaps also in parallel installing openebs) but since then my etcd no longer works and the ctrl plane is dead. I have apparmor installed. I see error messages like this: "level=info msg="Waiting for containerd startup: rpc error: code = Unimplemented desc = unknown service runtime.v1alpha2.RuntimeService"" And i see error messages like this in etcd:
Copy code
"/health error","output":"{\"health\":\"false\",\"reason\":\"RAFT NO LEADER\"}","status-code":503}
When i try to debug containerd/container with ctr or crictrl i always get that there is no containerd.sock (i can only find
Copy code
containerd.sock.ttrpc
There was a weird issue on 2 ctrl planes: something about not enough filedescriptors. Unfortuna i was not able to see who tried to open too many files, but i'im not sure if this is not more of an containerd issue who starts 2 etcd container constantly My google and debug magic is gone. Any ideas/suggestions?
r

rapid-helmet-86074

06/16/2022, 7:20 PM
Does the file
Copy code
/run/k3s/containerd/containerd.sock
exist? Have you poked around with any of the other commands from https://gist.github.com/superseb/3b78f47989e0dbc1295486c186e944bf ?
b

billions-easter-91774

06/16/2022, 7:22 PM
It exists, i think i should be able to create a config then. let me check out the other commands. tx
r

rapid-helmet-86074

06/16/2022, 7:22 PM
No problem.
b

billions-easter-91774

06/16/2022, 7:30 PM
No luck:
Copy code
/var/lib/rancher/rke2/bin/crictl --runtime-endpoint unix:///run/k3s/containerd/containerd.sock ps -a
FATA[0009] connect: connect endpoint 'unix:///run/k3s/containerd/containerd.sock', make sure you are running as root and the endpoint has been started: context deadline exceeded
r

rapid-helmet-86074

06/16/2022, 7:31 PM
Did you try
Copy code
/var/lib/rancher/rke2/bin/crictl --config /var/lib/rancher/rke2/agent/etc/crictl.yaml ps
The socket directly I only tried with
ctr
but I used the config with
crictl
. Both worked fine for me.
b

billions-easter-91774

06/16/2022, 7:41 PM
i tried the ctr one again and that works now
wtf... whle recreating my environment onto a new setup, i selected the vps with 2gb of ram and it struggled with memory... i distinctly rememeber upgrading that thing in the old env due to this...
98 Views