Is it possible that in new RKE2 / GPU-operator setups, the config parameters:

- name: CONTAINERD_RUNTIME_CLASS
        value: nvidia
      - name: CONTAINERD_SET_AS_DEFAULT
        value: "true"
      - name: CONTAINERD_CONFIG
        value: /var/lib/rancher/rke2/agent/etc/containerd/config.toml

are ignored / not working / not supported anymore?

I think you need to use config.toml.tmpl as name of the config file (and maybe config-v3.toml.tmpl). However, we ended with dummy config file and we distribute pre-fabricated config-v3.toml.tmpl, because rke2 afaik does not support nvidia-cdi or nvidia-legacy runtimes.

You are right, it used to be

/var/lib/rancher/rke2/agent/etc/containerd/config.toml.tmpl

but that file does not exist anymore in my setup,...

you need to create it

Oh, really? Manually on all nodes? Hm...ok.

or nvidia toolkit pod will create it for you, but just plain version I think

Yeah, I think it used to be created automatically but it doesn't get created anymore. So,...pretty weird. Will check if that still runs, otherwise I will have to find a different solution 🙂

I think that .tmpl was never created by rke2. it contrary uses it instead of internal toml configuration. so at some point, it needs to be created. If you pass it to the gpu-operator, nvidia-toolkit pod will create it, but only some minimal version. our approach is to collect the correct tmpl file and then distribute it via puppet tool to each gpu node and instruct gpu-operator to touch some dummy file instead and it works as we need.

Hmm,...ok

also be aware that the file was renamed to config-v3.toml.tmpl in recent (containerd 2.0) versions of rke

Ok, so it's only a matter of time until the config.toml.tmpl is not usable anymore.

No, the old temple still works. And will as long as containerd supports the legacy syntax