https://rancher.com/ logo
Join Slack
Powered by
This message was deleted.
# rke2
a
This message was deleted.
c
have you tried the etcd pod logs?
Also, if you’re restoring from a backup and the node name or IP has changed, you would need to do a --cluster-reset, possibly with --cluster-reset-restore to restore from a datastore snapshot.
b
I didn’t have a pod, but the issue has been discovered
The new node is also a new os and I had to change the cgroup driver to systemd
I’m not sure how that could get logged better, but in short it wasn’t able to start pods at a fundamental level it seems
c
That should be autodetected based on the running OS configuration. Had you customized the containerd config template or something?
b
Nope
I’m running 22.04 though, not sure if that’s supported
c
where did you need to change the cgroup driver then?
That’s not directly configurable anywhere in RKE2, unless you’re overriding kubelet args or something
b
Well, the previous nodes were frankenstien centos7 nodes with current kernels
So I think I had to set that as a custom kubelet arg yes
Which is what I had to change
c
ah yeah. If you get out in the weeds with component args then you’re on the hook for maintaining that. We try to autodetect as much as we can.
b
Yeah, I wasn’t in the weeds previously but admittedly centos7 with fresh kernel isn’t exactly normal either
But that didn’t work without intervention as I recall
The tough thing is if containers are flat out failing to start it would be nice to have some feedback if possible
c
It was probably in the containerd log somewhere?
b
Does that not go to the jounald log or just to flat file?
c
no, containerd has its own log file
b
ok, it was probably in there then
I see some stuff in the kubelet logs actually as well
Copy code
E1010 16:08:08.562192   22449 remote_runtime.go:201] "RunPodSandbox from runtime service failed" err="rpc error: code = Unknown desc = failed to create containerd task: failed to create shim task: OCI runtime create failed: runc create failed: expected cgroupsPath to be of format \"slice:prefix:name\" for systemd cgroups, got \"/kubepods/burstable/pod2bfb17ee41d495f34ce04b7101a882c6/77912b0c5a67ab7443d50a97cf1d84e00b862559989ca10bbef398f9e017fc3f\" instead: unknown"
E1010 16:08:08.562232   22449 kuberuntime_sandbox.go:70] "Failed to create sandbox for pod" err="rpc error: code = Unknown desc = failed to create containerd task: failed to create shim task: OCI runtime create failed: runc create failed: expected cgroupsPath to be of format \"slice:prefix:name\" for systemd cgroups, got \"/kubepods/burstable/pod2bfb17ee41d495f34ce04b7101a882c6/77912b0c5a67ab7443d50a97cf1d84e00b862559989ca10bbef398f9e017fc3f\" instead: unknown" pod="kube-system/etcd-172.26.64.22"
E1010 16:08:08.562256   22449 kuberuntime_manager.go:815] "CreatePodSandbox for pod failed" err="rpc error: code = Unknown desc = failed to create containerd task: failed to create shim task: OCI runtime create failed: runc create failed: expected cgroupsPath to be of format \"slice:prefix:name\" for systemd cgroups, got \"/kubepods/burstable/pod2bfb17ee41d495f34ce04b7101a882c6/77912b0c5a67ab7443d50a97cf1d84e00b862559989ca10bbef398f9e017fc3f\" instead: unknown" pod="kube-system/etcd-172.26.64.22"
E1010 16:08:08.562310   22449 pod_workers.go:951] "Error syncing pod, skipping" err="failed to \"CreatePodSandbox\" for \"etcd-172.26.64.22_kube-system(2bfb17ee41d495f34ce04b7101a882c6)\" with CreatePodSandboxError: \"Failed to create sandbox for pod \\\"etcd-172.26.64.22_kube-system(2bfb17ee41d495f34ce04b7101a882c6)\\\": rpc error: code = Unknown desc = failed to create containerd task: failed to create shim task: OCI runtime create failed: runc create failed: expected cgroupsPath to be of format \\\"slice:prefix:name\\\" for systemd cgroups, got \\\"/kubepods/burstable/pod2bfb17ee41d495f34ce04b7101a882c6/77912b0c5a67ab7443d50a97cf1d84e00b862559989ca10bbef398f9e017fc3f\\\" instead: unknown\"" pod="kube-system/etcd-172.26.64.22" podUID=2bfb17ee41d495f34ce04b7101a882c6
Copy code
kubelet-arg:
- --make-iptables-util-chains=false
- --log-file-max-size=20
#- --cgroup-driver=cgroupfs
- --cgroup-driver=systemd
- --max-pods=220
c
K3s already sets that based on the detected cgroup driver. You’re not intended to set it yourself.
Just delete both lines
b
As I say it didn’t work for me and I had to explicitly set it.
But admittedly the nodes I was using were an oddity. I’ll try removing altogether for the 22.04 nodes and see how it goes.
23 Views
Open in Slack
PreviousNext
Powered by