Hello Rancher team! I have a basic question. I'm working on setting up GPU support on a single-node RKE2 cluster and have run into an issue with the

nvidia-device-plugin-daemonset

. The

nvidia-device-plugin-daemonset

pod is stuck in a

CrashLoopBackOff

loop. The NVIDIA driver is correctly installed on the host, and

nvidia-smi

works as expected. When I run

kubectl describe pod nvidia-device-plugin-daemonset-85bkr -n kube-system

on the failing pod, I get this specific error message:

Events:
  Type     Reason   Age                     From     Message
  ----     ------   ----                    ----     -------
  Warning  BackOff  40m (x1266 over 5h15m)  kubelet  Back-off restarting failed container nvidia-device-plugin-ctr in pod nvidia-device-plugin-daemonset-85bkr_kube-system(3f6e3f80-6add-41bc-b49a-6e0aa8f2af30)
  Normal   Pulled   38m (x59 over 5h15m)    kubelet  Container image "<http://nvcr.io/nvidia/k8s-device-plugin:v0.18.0|nvcr.io/nvidia/k8s-device-plugin:v0.18.0>" already present on machine
  Warning  Failed   23m (x5 over 26m)       kubelet  Error: failed to create containerd task: failed to create shim task: OCI runtime create failed: runc create failed: expected cgroupsPath to be of format "slice:prefix:name" for systemd cgroups, got "/kubepods/besteffort/pod3f6e3f80-6add-41bc-b49a-6e0aa8f2af30/nvidia-device-plugin-ctr" instead
  Normal   Pulled   4m44s (x9 over 26m)     kubelet  Container image "<http://nvcr.io/nvidia/k8s-device-plugin:v0.18.0|nvcr.io/nvidia/k8s-device-plugin:v0.18.0>" already present on machine
  Normal   Created  4m44s (x9 over 26m)     kubelet  Created container: nvidia-device-plugin-ctr
  Warning  BackOff  55s (x117 over 26m)     kubelet  Back-off restarting failed container nvidia-device-plugin-ctr in pod nvidia-device-plugin-daemonset-85bkr_kube-system(3f6e3f80-6add-41bc-b49a-6e0aa8f2af30)

So any pod that requires the GPU basically is pending...

kubectl get pods 
NAME              READY   STATUS    RESTARTS   AGE
nvidia-gpu-test   0/1     Pending   0          5h10m

I have been following this guide here https://docs.rke2.io/advanced?_highlight=gpu#deploy-nvidia-operator any help? thanks and this is the config.toml.tmpl

cat /var/lib/rancher/rke2/agent/etc/containerd/config.toml.tmpl 
{{ template "base" . }}

[plugins."io.containerd.cri.v1.cri".containerd]
  default_runtime_name = "nvidia"

[plugins."io.containerd.cri.v1.runtime".containerd.runtimes.nvidia]
  privileged_without_host_devices = false
  runtime_type = "io.containerd.runc.v2"
  [plugins."io.containerd.cri.v1.runtime".containerd.runtimes.nvidia.options]
    BinaryName = "/usr/bin/nvidia-container-runtime"

Any help?

The docs section you linked to no longer exists. It was moved to a separate page 2 weeks ago. How are you reading docs that haven’t existed for weeks? Have you tried following the updated docs at https://docs.rke2.io/add-ons/gpu_operators ?

@creamy-pencil-82913 indeed I was puzzled brecaue I foun dnotthin talking about NVIDIA.. so my bad...no I have not followed the GPU Operators. Let me try this then! thanks!

ok so when you said you were following that guide… what exactly were you following?

ahh I see. We will have to ask them to update their docs.

thanks