Hello everyone, I'm running RKE2 v1.24.8 on RHEL 8.6 and I'm trying to move the data directory from the default location (/var/lib/rancher/rke2) to /mnt/data/rke2, which is on another (virtual) disk. I copied the entire directory using

cp -ar /var/lib/rancher/rke2 /mnt/data/

to preserve all attributes, then modified /etc/rancher/rke2/config.yaml and added the line

data-dir: /mnt/data/rke2

. This seems to work on agent/worker nodes. On server nodes, however, it looks like the necessary Kubernetes containers can't start. In the log of rke2-server, I keep getting the message

Waiting to retrieve kube-proxy configuration; server is not ready: <https://127.0.0.1:9345/v1-rke2/readyz>: 500 Internal Server Error

, while /mnt/data/rke2/agent/containerd/containerd.log looks like this:

time="2022-12-19T09:32:29.103553429+01:00" level=info msg="CreateContainer within sandbox \"478658988f888b30063a9127fb124abd38385967b796e15016675930bbb6cf88\" for container &ContainerMetadata{Name:cloud-controller-manager,Attempt:23,}"

time="2022-12-19T09:32:29.154564201+01:00" level=info msg="CreateContainer within sandbox \"478658988f888b30063a9127fb124abd38385967b796e15016675930bbb6cf88\" for &ContainerMetadata{Name:cloud-controller-manager,Attempt:23,} returns container id \"4a3fe66d16a18ac6397dafc147a68b5bbe9bda1d7d4f7f7ce5e7f95e3a49b84b\""

time="2022-12-19T09:32:29.154953048+01:00" level=info msg="StartContainer for \"4a3fe66d16a18ac6397dafc147a68b5bbe9bda1d7d4f7f7ce5e7f95e3a49b84b\""

time="2022-12-19T09:32:29.276767961+01:00" level=info msg="StartContainer for \"4a3fe66d16a18ac6397dafc147a68b5bbe9bda1d7d4f7f7ce5e7f95e3a49b84b\" returns successfully"

time="2022-12-19T09:32:29.691126028+01:00" level=info msg="shim disconnected" id=4a3fe66d16a18ac6397dafc147a68b5bbe9bda1d7d4f7f7ce5e7f95e3a49b84b

time="2022-12-19T09:32:29.691184071+01:00" level=warning msg="cleaning up after shim disconnected" id=4a3fe66d16a18ac6397dafc147a68b5bbe9bda1d7d4f7f7ce5e7f95e3a49b84b namespace=<http://k8s.io|k8s.io>

time="2022-12-19T09:32:29.691196163+01:00" level=info msg="cleaning up dead shim"

time="2022-12-19T09:32:29.708945825+01:00" level=warning msg="cleanup warnings time=\"2022-12-19T09:32:29+01:00\" level=info msg=\"starting signal loop\" namespace=<http://k8s.io|k8s.io> pid=3497353 runtime=io.containerd.runc.v2\n"

time="2022-12-19T09:32:30.049467561+01:00" level=info msg="RemoveContainer for \"9e40fb319e648b964c90cc77975c6cf7400aac36e53eb6354f738ad31995ce3c\""

time="2022-12-19T09:32:30.056618827+01:00" level=info msg="RemoveContainer for \"9e40fb319e648b964c90cc77975c6cf7400aac36e53eb6354f738ad31995ce3c\" returns successfully"

There are similar messages for kube-apiserver, etcd and kube-controller-manager. If I remove the data-dir line from my config, it all works again. Am I doing something wrong here? Some help would be much appreciated.

If you're using selinux, you will likely need to write your own policies to get the correct things allowed. The rke2-selinux policies only cover the default data-dir, not any other one you might decide to use.

Selinux is disabled.

I would probably check out the containerd log then?

I've attached both containerd.log and kubelet.log. To me, it looks like containerd is trying to start some necessary containers, but can't (for some unspecified reason?), and the kubelet is trying to connect to kube-apiserver, but that's not running yet. I don't see any obvious warnings or errors which would tell me what's wrong.

I will admit I have never tried moving the data after the node is up; generally I recommend starting the cluster with the correct paths already set. I kind of wonder if some of the kubeconfig files in the rancher/rke2/agent or rancher/rke2/server/cred directory have the old paths in them - there are some files that we don’t rewrite if they already exist.

Yeah, I believe that will be the root of the problem. I ended up reinstalling each node with the data-dir parameter set before installation and they all work now. Thanks for your help Brad, happy holidays!