Hi, I have RKE2 v1.22.9 with rke2-canal. As the operating system supports ipset 7.15 I'm facing the issue of it not being compatible with hardened calico ipset. According to the tigera documentation, the latest image supports ipset 7.11 which is still incompatible with 7.15. As I'm not using any of the extra security features of hardened-calico (not even sure why I went with this in the first place), I would like to replace it. I've done a CNI replacement in a different cluster (EKS, replaced the stock CNI with weave), but I'm not sure what is the procedure here (if possible). Tigera documentation mentions a possible migration path from canal to calico, but does that apply here? The CNI was installed as part of the RKE2 installation. Does replacing the CNI break anything in the cluster?

Hi @most-hairdresser-42454 😄 glad to see you around. It's been some time.

Ho @careful-piano-35019, it's a long time ago 😄

I'm trying to restore a node that is a control plane node (I rebuilt the node and copied all the data/etc back into place) but it not really starting up. The logs keep saying effectively that etcd hasn't started but I'm not really sure where to look for further debugging? any tips?

Hi @creamy-pencil-82913, It seems with this fixed issue https://github.com/k3s-io/helm-controller/issues/89 , we can use self signed CA in helmchart custom resource, but I do not see any field to provide the custom CA , https://docs.k3s.io/helm#helmchart-field-definitions here, to fetch the chart ?

Hi, I have an RKE2 cluster with restored etcd. After cluster reset, master2 got online but it didn't connect back to Rancher. Checking rancher-system-agent logs show Oct 11 140938 worker2 rancher-system-agent[1210]: time="2022-10-11T140938Z" level=error msg="[K8s] Received secret that was nil" Oct 11 140943 worker2 rancher-system-agent[1210]: time="2022-10-11T140943Z" level=error msg="[K8s] Received secret that was nil" It appears, that the rancher agent does not have the proper token to register the cluster with Rancher. How do I get rancher-system-agent to reconnect?

When setting up a nginx or haproxy LB, do I set that up outside of kubernetes environment? Like as a docker service on the main rancher docker server? In tandem?

So I got nginx working in RKE2, but how do I find out if it was set up using nodeport, clusterIP or LoadBalancer without looking at the config?

I created a Rancher HA setup with RKE2 installed on 3 nodes behind a LB. I have observed that if I reboot one of the nodes, it never rejoins the cluster after rebooting. This is not the case if I use RKE instead of RKE2. In our infrastructure it is possible that one or more nodes could fail. Is this expected behavior with RKE2?

is it possible to set

kube-proxy-replacement

to strict on rancher launched rke2 clusters with cilium ? if yes, could you please tellme how ? thanks

General question about air-gap installs: I have 2 images i always download. which are rke2-images-calico.linux-amd64.tar.gz and rke2.linux-amd64.tar.gz. My aim is to make sure that NO images try to download during initial build of the cluster Any hints or guides to the other images? Any way i can see what was download other then tailing the log during install?

Recently moved some master nodes into different AWS AZs and am now having cert issues:

Oct 14 15:09:01 k8worker05 rke2: time="2022-10-14T15:09:01Z" level=info msg="Connecting to proxy" url="<wss://10.149.5.62:9345/v1-rke2/connect>"
Oct 14 15:09:01 k8worker05 rke2: time="2022-10-14T15:09:01Z" level=error msg="Failed to connect to proxy" error="x509: certificate is valid for 10.149.4.146, 10.149.4.32, 10.149.4.77, 10.43.0.1, 127.0.0.1, not 10.149.5.62"
Oct 14 15:09:01 k8worker05 rke2: time="2022-10-14T15:09:01Z" level=error msg="Remotedialer proxy error" error="x509: certificate is valid for 10.149.4.146, 10.149.4.32, 10.149.4.77, 10.43.0.1, 127.0.0.1, not 10.149.5.62"

Obviously

10.149.5.62

is the new IP and doesn't match what the cert is advertising. I'm stumped however about how the cert is being generated. The

/etc/rancher/rke2/config.yaml

file doesn't have any IP references... There are IPs in

/var/lib/rancher/rke2/server/tls/dynamic-cert.json

though these appear to be the result of some process. Any idea how to regenerate these certs?

Hi, I have a question about the generated rke2.yaml kubeconfig (/etc/rancher/rke2/rke2.yaml). Do you know if it’s possible to override the certificate-authority-data? It seems RKE2-server uses the server/tls/server-ca.crt as certificate-authority-data, am I right? Is it possible to change the path ?

@creamy-pencil-82913, I have logged https://github.com/rancher/rke2/issues/3462 Once I faced the error, thought of trying out crictl command execution on nodes like.

root@server:/home/ubuntu# crictl pull --creds "AWS:eyJwYXlsb2FkIjoieXRSVW5JMzkwRlVneitXNnpPNnJGOGRqYU9yZ0tRbEFIdkF0aGprMjlNTU1JWWdQd095QlJsQ01FUmRCWFVjZlZNNkEyRTdYS3ByeVRwRjhPNWlneStEdEtmcXdrR2tkMnlwM3RNUnFNNG8zOW1xdUsrSlVOemVWWDFUbGEwR1RqdjkyMmtXMWNsVUZuVnJxOEUzM3VubG9wdm5HbVp0a3o2YVdVSGNzM20reDEvbTl1K2dLZTk1ZnhaTnIrdU43SmRyNlBod0Z1TXBMUnNxUzZoZC9rYy9xMmwxbDJRNXk0Nm9scDNtNG9uc29pdjRid1JBMVpIaEdvMDhSS1lac" <http://1234.dkr.ecr.us-west-2.amazonaws.com/mlflow-run:latest|1234.dkr.ecr.us-west-2.amazonaws.com/mlflow-run:latest>

This ended up in error

WARN[0000] image connect using default endpoints: [unix:///var/run/dockershim.sock unix:///run/containerd/containerd.sock unix:///run/crio/crio.sock]. As the default settings are now deprecated, you should set the endpoint instead.
ERRO[0002] connect endpoint 'unix:///var/run/dockershim.sock', make sure you are running as root and the endpoint has been started: context deadline exceeded
ERRO[0004] connect endpoint 'unix:///run/containerd/containerd.sock', make sure you are running as root and the endpoint has been started: context deadline exceeded
FATA[0006] connect: connect endpoint 'unix:///run/crio/crio.sock', make sure you are running as root and the endpoint has been started: context deadline exceeded

to overcome above error, I have added

/etc/crictl.yaml

file with below content.

runtime-endpoint: unix:///run/k3s/containerd/containerd.sock
image-endpoint: unix:///run/k3s/containerd/containerd.sock
timeout: 10

After creating above file, on pod creation images were pulled successfully from AWS ECR.

Greetings, everyone Maybe someone knows how to solve this - I have an RKE2 cluster stuck in Updating, with an error message "Init node not found". By looking at this cluster CRDs, I've found out that it is stuck on trying to restore a nonexistent ETCD snapshot to a (already) nonexistent node. As a result, I cannot do anything with the cluster (like adding more master nodes). How do I get it un-stuck?

nevermind, solved it

Hello all! I recently setup a custom cluster that is being provisioned on Openstack. The provisioning status is stuck in "waiting for agent to checkin and apply initial plan" due to the agent being in a restart loop due to status=11/SEGV. I can't seem to break the cycle and debug further, any guidance or tips of how to proceed or where else to look for further information? rancher-system-agent.service: Main process exited, code=killed, status=11/SEGV rancher-system-agent.service: Failed with result 'signal'. New Install Ubuntu 22.04 LTS Rancher v2.6.8 Openstack RKE2 Firewalls are open ANY<>ANY on each side to avoid any issues.

Follow on question to my attempts to move nodes to different AWS AZs... Tried doing this with another stack and am having less luck! The original cluster had 3 master nodes. Currently 1 of those 3 is functional. On the first master node, if I remove the

/var/lib/rancher/rke2

directory and relaunch

rke2 server

, it appears to create an entirely new cluster (as the process starts successfully, but

kubectl get nodes

only returns itself.) On the second master node, after removing the dir and relaunching the service, it is just showing this loop in the logs:

Oct 17 16:37:09 <http://k8mst02.espc-nostromo.nos-amc.io|k8mst02.espc-nostromo.nos-amc.io> rke2[32216]: time="2022-10-17T16:37:09Z" level=info msg="Failed to test data store connection: this server has not yet been promoted from learner to voting member"
Oct 17 16:37:10 <http://k8mst02.espc-nostromo.nos-amc.io|k8mst02.espc-nostromo.nos-amc.io> rke2[32216]: time="2022-10-17T16:37:10Z" level=info msg="Waiting to retrieve kube-proxy configuration; server is not ready: <https://127.0.0.1:9345/v1-rke2/readyz>: 500 Internal Server Error"

I have a RKE2 1.24.6 cluster - the other day I added etcd-snapshot-schedule-cron: " */6 * * *" etcd-snapshot-retention: 56 to config.yaml and restarted the master node (control-plane,etcd,master) but even after a few days it's still using the default. Can this not be changed after cluster install?

Hey, i have rke2 v1.21 cluster with calico cni. I want to migrate it to cillium with wireguard. Is there any way to do this?

1.23 not 21*

Hello. I'am currently running rke2 1.24.4 and most recently run into several helm chart failing installing due to seccomp issues. Basically i get told that the contain seccomp settings cannot be applied

forbidden seccomp may not be set

- for example the new cert-manager 1.10 introduces https://artifacthub.io/packages/helm/cert-manager/cert-manager/1.10.0#default-security-contexts a new default security context - thus i cannot install it on my rke2 cluster. Same goes with bitnami-wordpress start 15.2.0 which also introduce RuntimeDefault as their default runtime. Is there anything missing in my rke2 configuration or do i miss the point entirely?

Hello Team! I am trying an airgap install of an RKE2 cluster on AlmaLinux 8 VMs. My environment has no upstream DNS server and I left pretty much all of the RKE2's config to default. I have an issue with CoreDNS though.. My problem is that none of my pods are resolving the name of the services I create (could not resolve host: <service name>). Basically CoreDNS seems to be in a failed state My coredns pods are in CrashingLoopBackOff state and the logs are "Plugin/forward: no nameserver found". In this configuration I haven't changed the Corefile yet, and thus I have "forward . /etc/resolv.conf" in the file. Every server node in my cluster has an empty resolv.conf since I have no parent/upstream DNS server. I've tried to add "nameserver 8.8.8.8" to "/etc/resolv.conf", and after deleting the coredns pods so they can get recreated, they went into a running state but the logs are full of error contacting 8.8.8.8 server (obviously) and pods still can't resolve services' name. I've also tried to remove the forward plugin from the Corefile, the coredns pods are correctly running with no error in logs but all of my pods keep running the same name resolution error. I've launched a busybox pod to help me debugging and every nslookup command gives me a connection refused on the IP of the ClusterIP of coredns.

Hello, I am having an issue with RKE2. I have a RKE1 cluster up on rancher using the vsphere provisioner. But when I try to setup an RKE2 cluster it gets stuck on connecting to the machines. Is there a special VM image I need to use for RKE2 or am I doing something wrong.

[Disconnected] Cluster agent is not connected

I'm working on setting up a RKE2 cluster and I have an HA control plane tainted so it doesn't run workloads. Is there an argument to be made to go a step further and have etcd run separately from the other control plane services? Is there a best practice RKE2 deployment page I missed? Thanks in advanced for the help

Only reason I could think of would be if I wanted to scale etcd far larger than I wanted the control plane.

Hey all. Whats the best way to move PVC's and PV's from an RKE1 cluster to a new RKE2 cluster?

Once again I have a question about how to properly shutdown a RKE2 master node 😄 1. Cordon node 2. Drain node 3. systemctl stop rke2-server which ends with the service being faulted I still have etcd, api server and everything else running. From an RKE2 perspective, what is the correct way to shutdown the relevant services? Thinking primarily etcd I guess, but ideally I'd shut down the other core services first and etcd last.

ah 1.24.7 has been released, awesome!