This message was deleted.

e.g.

/run/k3s

on bare metal

drwx--x--x.  3 root           root           system_u:object_r:var_run_t:s0                  60 Dec  2 17:57 k3s

on vm

drwx--x--x.  3 root           root           system_u:object_r:container_var_run_t:s0        60 Nov 22 15:53 k3s

Figure out what’s different about the OS installs between your physical and VM nodes. Different packages? different filesystem layout?

do you have separate disk partitions for

/var/lib/rancher

and/or

/var/lib/kubelet

?

ah the /var mounts have completely different options

# bare metal
/dev/mapper/rl-var      /var                    xfs     defaults,nodev,nosuid 0 0
# vm
/dev/mapper/vg-var      /var                    xfs     defaults        0 0

this is from my RHEL9.4 nodes:

# ls -ldZ /var/lib/kubelet/ /var/lib/rancher/ /run/k3s
drwx--x--x. 3 root root system_u:object_r:container_var_run_t:s0  60 Nov 13 17:14 /run/k3s
drwxr-xr-x. 9 root root system_u:object_r:container_var_lib_t:s0 193 Oct  7 13:02 /var/lib/kubelet/
drwxr-xr-x. 3 root root unconfined_u:object_r:var_lib_t:s0        18 Jul  2 12:28 /var/lib/rancher/

also selinux enforcing and selinux is true in the config.yaml

Yep it was

nosuid

. Thanks both