hi all - I have a rke2 cluster running 1.24.12, on rhel 8 using cilium - since the 1.24.12 upgrade (which uplifted cilium to 1.13.0 from 1.12.5) we've had a weird issue that results in some pods becoming uncontactable and the cilium health status check to fail against the pod IP, the host IP cilium health check continues working. I haven't been able to work out to reproduce it, but it's looking like an interaction between cilium and kube-proxy, and iptables somehow

the host itself has no iptables applied other than what k8s has added, it's all NFTables - restarting the cilium pod sometimes helps, restarting the kube-proxy pod sometimes helps, sometimes doing a

nft flush ruleset

and waiting a minute helps, the only reliable answer is to reboot the worker node

not all running pods on the worker node become uncontactable, and they continue to accept new connections, but if I destroy and let a pod reschedule onto the worker, usually it then fails to become contactable over the pod network

the iptables is a hunch because flushing the nft rules sometimes helps, but I have no concrete evidence, maybe the other thing to be aware of is it's ipsec on top of GENEVE as the tunneling protocol

I should mention, from the host and pod itself that is running the cilium health check, I can reach it, it's only crossing the workers east west that fails

Open an issue on GH and attach as much info as possible, I'll ask the CNI guys to take a look

thanks @creamy-pencil-82913 I'll get it onto github tomorrow with as much info as I can collate

Is the

Default - RKE2 Embedded

Pod Security Administration Configuration Template comparable to Privileged or Restricted ?

How can i trigger a certificate rotation of kube-controller-manager and kube-scheduler certificates before they expire ? all the other rke2 certificates seem to have been rotated all nodes have been rebooted, but these 2 still have not rotated. rotate certificates in rancher gui, or rke2 command do not seem to do anything for these 2. this is a rke2 rancher installed kluster, running v1.22.7+rke2r2 kube-controller-manager/kube-controller-manager.crt: Not After : Apr 25 181506 2023 GMT kube-scheduler/kube-scheduler.crt: Not After : Apr 25 181506 2023 GMT

Hey 👋, I am having trouble adding pure worker nodes to my rancher custom cluster with rke2. Adding nodes with all roles (etcd,control,worker) works just fine. But whenever I want to add a worker node only the node won't start. Any ideas? Logs are telling me that some http://127.0.0.1/cacerts connection was reset by the peer

hi, anyone knows which chart is Rancher using for OpenLDAP auth providers ? i would like to install the same for RKE2 this is to integrate with my org AD

Hi. On rke2 what folder do the kube-apiserver logs live?

Can anyone help me find or explain in detailed instructions how to import another cluster so that I can monitor multiple clusters under one Rancher GUI.

My team runs an RKE2 cluster (installed via ansible), to host rancher on to provision and manage downstream clusters. Occasionally, new pods will get stuck in pending waiting for an IP address from calico. This doesn't resolve itself, until we manually restart the calico pods. The calico pods aren't failing healthchecks, and there's nothing error-looking in the logs. We don't have this on the rancher-managed downstream clusters 😞 Has anyone got any ideas what might be causing this or things we can check?

Do you actually need to drain a node before upgrading k8s version? The docs doesn't mention anything about it.

Is there an example on how to install/config multus with Cilium?

like this? https://docs.rke2.io/install/network_options#using-multus-with-cilium

Hi all, I’m looking to upgrade our rke2 clusters and have a couple of questions. • is there a limit to the version jump that can be applied? i.e can we upgrade from v1.22.6+rke2r1 to v1.27.1+rke2r1 or do we need to go in steps? • does containerd get updated as well or is that a separate update? • do the server and worker nodes need to be cordoned before restarting the service? thanks!

I tried cilum as CNI, using rke2-images-cilium.linux-amd64.tar.zst and rke2-images-core.linux-amd64.tar.zst. The "rke2-images-cilium.linux-amd64.txt" includes all of these images: docker.io/rancher/mirrored-cilium-certgen:v0.1.8 docker.io/rancher/mirrored-cilium-cilium:v1.13.0 docker.io/rancher/mirrored-cilium-cilium-etcd-operator:v2.0.7 docker.io/rancher/mirrored-cilium-clustermesh-apiserver:v1.13.0 docker.io/rancher/mirrored-cilium-hubble-relay:v1.13.0 docker.io/rancher/mirrored-cilium-hubble-ui:v0.10.0 docker.io/rancher/mirrored-cilium-hubble-ui-backend:v0.10.0 docker.io/rancher/mirrored-cilium-operator-aws:v1.13.0 docker.io/rancher/mirrored-cilium-operator-azure:v1.13.0 docker.io/rancher/mirrored-cilium-operator-generic:v1.13.0 docker.io/rancher/hardened-cni-plugins:v1.0.1-build20221011 Installation of cilium completed. However, there is no hubble in my installation and enabling it gives me this error: cilium hubble enable ⚠️ Error parsing helm cli secret: unable to retrieve helm values secret kube-system/cilium-cli-helm-values: secrets "cilium-cli-helm-values" not found ⚠️ Proceeding in unknown installation state Error: Unable to enable Hubble: unable to obtain cilium version, no cilium pods found in namespace "kube-system" Any ideas?

Hello, I'm trying to enable feature gate

SizeMemoryBackedVolumes

. It should be enabled by default since k8s 1.22 (I'm running v1.24.10+rke2r1) but if I try to use ⬇️ k8s fails with invalid configuration. I tried adding

--feature-gates=SizeMemoryBackedVolumes=true

to

/etc/rancher/rke2/config.yaml

under

kubelet-arg

and

kube-apiserver-arg

but rke2-server service won't start.

emptyDir:
     medium: Memory
     sizeLimit: 1Gi

It should be possible to switch cni by changing the config and restarting rke2 service right? And maybe manually uninstall old cni. Wanna switch from Calico to cilium

anyone mess around with minio? I'm following: https://documentation.suse.com/trd/kubernetes/html/gs_rancher_minio/index.html and I receive: kubectl directpv info ERROR DirectPV not installed

Asking what I am sure is a stupid question, but I want to update a cluster, changing some arguments to kubelet, and assumed i would try updating the config.yaml and restart the control plane nodes... however using systemctl to stop the rke2-server service seems to have stopped kubelet, but not the other k8s related processes. Is this expected behaviour? Is the "correct" way to perform this restart to use the bundled

rke2-killall.sh

or something else?

OK, stupid question number 2: Im trying to setup a custom audit logfile. i am mounting the volume from the host ok, and I can set the

audit-log-max{age,backup,size}

values in the rke2/config.yaml file, however if i set the

audit-log-path

argument then the kube-apiserver fails to start, in fact containerd doesnt seem to know about it at all. Im running

v1.25.9+rke2r1

which i believe has a recent fix for this in place. When i jump into the container prior to setting the audit-log-path I can see the extra-mount as available, and can create files in that directory. Any hints into debugging this further would be greatly appreciated. The rke2-server logs indicate that it was passed through correctly.

Hi all, hope you are all fine 🙂. I have probably a simple question regarding an RKE2 installation, but can’t really figuring out what is wrong. I have a very simple RKE2 cluster setup, 1 control-plane and 2 worker nodes in combination with the cilium CNI. Somehow pods running in this cluster aren’t able to resolve dns. Some information on the cluster: CoreDNS is running on two nodes control-plane-01 (pod-ip: 10.36.0.116) and worker-01 (pod-ip: 10.36.1.185) with a Service configured on 10.40.0.10 When running a multitool on each of the nodes a default

nslookup

fails with error

;; connection timed out; no servers could be reached

When running

nslookup <http://google.com|google.com> 10.36.0.116

from multitool running on the control-plane-01 host I get a valid response, running

nslookup <http://google.com|google.com> 10.36.1.185

it fails with the connection timed out error. Same behaviour when running this in the multitool pod on the worker-01 host. My guess is that our external firewall is blocking some connections, but I can’t find out which ports I should whitelist. I verified that all the ports mentioned on https://docs.rke2.io/install/requirements#inbound-network-rules (Kubernets specific ones and the Cilium ones) are open and allowed port 53 for TCP and UDP traffic between all the cluster hosts.

Hi Team, I have deployed a Rancher server by following this doc -> https://ranchermanager.docs.rancher.com/getting-started/quick-start-guides/deploy-rancher-manager/aws. Now I am trying to create a new RKE cluster in my cloud but I am not able to do it. I getting an error when as shown below. I am giving the right details but still I am getting 403 issue. Can anyone help?

Screenshot 2023-05-08 at 9.45.15 AM.png

RKE2 with cilium on Ubuntu 22.04 on VMWare is not working, node reachable but endpoints NO. The internal communication between nodes not working. Any support on this?

k8s 1.24 from rancher 2.6.10

👋 Hey, I have a new cluster running in azure with a vmss with autoscaling (cluster version is: v1.25.9+rke2r1 and it was registered from a rancher server v2.7.3), the nodes are removed from rancher and the vmss (thanks to a small operator), however "new" nodes can no longer be added, the error on the new nodes is:

Waiting to retrieve agent configuration; server is not ready: Node password rejected, duplicate hostname or contents of '/etc/rancher/node/password' may not match server node-passwd entry, try enabling a unique node name with the --with-node-id flag

I have searched over in github issues but I couldn't get a reference that worked, the secret for the nodes

<node-name>.node-password.rke2

is also gone, so I'm trying to find any clue to make the cluster operational again, it is stuck and the agent cannot complete the process, any help is appreciated 🙏 , thanks