When looking at the yaml config of the cluster, I see this:

machineSelectorConfig:
      - config:
          kubelet-arg:
            - image-gc-low-threshold=70
            - image-gc-high-threshold=75
            - registry-qps=10
            - registry-burst=20
            - >-
              container-runtime-endpoint=unix:///run/k3s/containerd/containerd.sock
          protect-kernel-defaults: false

So maybe this part is overriding the rke2 config.yaml on the node, which maybe was editing manually, I'm not sure.

So my question would then be: what is the right way to use a different container runtime on a certain node?

(on a rke2 prod cluster I see that the container-runtime-endpoint is not set at all. So I guess this is optional. I wonder how it ended up there)

Simply deleting

- >-
              container-runtime-endpoint=unix:///run/k3s/containerd/containerd.sock

from the cluster configuration worked and the worker node works again. I wonder how that line ended up there 🙂

@creamy-pencil-82913 just another question related to this: why is the containerd socket under /run/k3s/containerd? So why 'k3s'? Instead of, I don't know, rke2 or something like that, given that this is an rke2 cluster.

Early on we replaced most of the occurrences of “k3s” with a variable that is set to “rke2" or “k3s” at build time, but we missed that one, and now it’s kinda late to change it