The release notes for the most recent rke2 version v1.26.0+rke2r1 recommend to use version rke2r2 instead (due to some containerd issue resolved 3 weeks ago), but this version is not released yet, as it seems. I have to install 2 new clusters now, so I wonder which version you would recommend?

Hello, Is rke2 supported on Redhat? If yes, what version of Redhat?

Hello, In RKE2, I have a deployment and setup an Ingress (nginx). But I am getting

413 Request Entity Too Large

Has anyone else experienced this? How to resolve this?

Hi Rancher Team, Is their a roadmap to support Redhat 8.6? From this docs, https://docs.rke2.io/install/requirements, it only supports Redhat 8.5 Thanks!

what is the default lifetime policy for completed pods in RKE2? All our completed pods are getting cleaned up, do we have to modify garbage collection settings in kubelet to avoid that? Sorry for the question, I was coming from the openshift background, completed pods used be there forever unless we run some cleanup job

Is this good or bad?

root@srvl034a:~# grep cgroup /proc/self/mounts
cgroup2 /sys/fs/cgroup cgroup2 rw,nosuid,nodev,noexec,relatime,nsdelegate,memory_recursiveprot 0 0

Hi. A cross post from the Cilium Slack. I just noticed that doing a

kubectl rollout restart daemonset -n kube-system cilium

caused a cluster outage. Output from

journalctl -u rke2-server

Jan 10 11:31:21 k8s-build001-master001.dc1 rke2[980]: time="2023-01-10T11:31:21Z" level=error msg="Remotedialer proxy error" error="websocket: close 1006 (abnormal closure): unexpected EOF"
Jan 10 11:31:21 k8s-build001-master001.dc1 rke2[980]: time="2023-01-10T11:31:21Z" level=info msg="error in remotedialer server [400]: websocket: close 1006 (abnormal closure): unexpected EOF"
Jan 10 11:31:26 k8s-build001-master001.dc1 rke2[980]: time="2023-01-10T11:31:26Z" level=info msg="Connecting to proxy" url="<wss://10.81.23.12:9345/v1-rke2/connect>"
Jan 10 11:31:26 k8s-build001-master001.dc1 rke2[980]: time="2023-01-10T11:31:26Z" level=error msg="Failed to connect to proxy. Empty dialer response" error="dial tcp 10.81.23.12:9345: connect: connection refused"
Jan 10 11:31:26 k8s-build001-master001.dc1 rke2[980]: time="2023-01-10T11:31:26Z" level=error msg="Remotedialer proxy error" error="dial tcp 10.81.23.12:9345: connect: connection refused"
Jan 10 11:31:28 k8s-build001-master001.dc1 rke2[980]: {"level":"warn","ts":"2023-01-10T11:31:28.069Z","logger":"etcd-client","caller":"v3@v3.5.4-k3s1/retry_interceptor.go:62","msg":"retrying of unary invoker failed","target":"<etcd-endpoints://0xc000aace00/127.0.0.1:2379>","attempt":0,"error":"rpc error: code = DeadlineExceeded desc = context deadline exceeded"}
Jan 10 11:31:28 k8s-build001-master001.dc1 rke2[980]: time="2023-01-10T11:31:28Z" level=error msg="Failed to get recorded learner progress from etcd: context deadline exceeded"
Jan 10 11:31:31 k8s-build001-master001.dc1 rke2[980]: time="2023-01-10T11:31:31Z" level=info msg="Connecting to proxy" url="<wss://10.81.23.12:9345/v1-rke2/connect>"
Jan 10 11:32:10 k8s-build001-master001.dc1 rke2[980]: E0110 11:32:10.988495     980 leaderelection.go:330] error retrieving resource lock kube-system/rke2: Get "<https://127.0.0.1:6443/api/v1/namespaces/kube-system/configmaps/rke2>": stream error: stream ID 2425069; INTERNAL_ERROR; received from peer

k8s recovered after about 5 minutes. In the hubble monitor I notice return traffic is denied from etcd port

Jan 10 11:51:29.529: 10.81.23.12:2380 (kube-apiserver) <> 10.81.23.11:32768 (host) policy-verdict:none DENIED (TCP Flags: SYN, ACK)
Jan 10 11:51:29.529: 10.81.23.12:2380 (kube-apiserver) <> 10.81.23.11:32768 (host) Policy denied DROPPED (TCP Flags: SYN, ACK)

Logs from cilium agent

Policy verdict log: flow 0xc18ce21f local EP ID 2998, remote ID kube-apiserver, proto 6, ingress, action deny, match none, 10.81.23.12:2380 -> 10.81.23.11:32770 tcp SYN, ACK
xx drop (Policy denied) flow 0xc18ce21f to endpoint 0, file bpf_host.c line 2147, , identity kube-apiserver->unknown: 10.81.23.12:2380 -> 10.81.23.11:32770 tcp SYN, ACK
Policy verdict log: flow 0xe145a85 local EP ID 2998, remote ID kube-apiserver, proto 6, ingress, action deny, match none, 10.81.23.12:2380 -> 10.81.23.11:32768 tcp SYN, ACK
xx drop (Policy denied) flow 0xe145a85 to endpoint 0, file bpf_host.c line 2147, , identity kube-apiserver->unknown: 10.81.23.12:2380 -> 10.81.23.11:32768 tcp SYN, ACK

Anyone that has encountered similar issues?

Hi, I have configured HA RKE2 using Kube-VIP on 3 Master nodes. The setup came up properly by assigning the VIP address to one of the Master nodes and is working as expected. The issue arises only when two of the master nodes are down, first kubectl command fails to respond and there is no kube-vip address available on the existing single Master node. Please let me know if this is a known issue with kube-vip or I am missing something. Any help would be appreciated.

Hi when we have built a cluster with 3 control planes we are seeing that kubectl commands are taking 15 seconds to complete. Whereas with a cluster with one control plane the response was instant

Hi, I have deployed HA RKE2 using Kube-VIP. The setup is successful when I configured server section of "/etc/rancher/rke2/config.yaml" file with "server: <first server node hostname>:9345" on all the secondary server nodes and the agent nodes. Later I found that some of the documentations suggest using VIP hostname in the server section instead of first server node as "server: <VIP hostname>:9345". I have a doubt in using the VIP hostname because the token we are using is from the first server node. So, I don't find any proper reason in using the VIP hostname with first sever node's token. Please suggest me, which one I need to use in the server section. (first server hostname or VIP hostname)

Greetings. Why could provisioning a fresh RKE2 cluster produce a completely broken setup? Anything that I try to deploy in any namespace besides the default ones ends up with "forbidden: PodSecurityPolicy: unable to admit pod: []" without any sane explanation. Have tried deploying both v1.22.16 and v1.24.8 without any luck. Doesn't matter which default pod security policy is set - both unrestricted and RKE2 default fail. I'm out of ideas at this point.

the weirdest thing is that only affects deployments, statefulsets, daemonsets etc. Deploying the very same pod manually by applying pod manifest works perfectly

deploying to

default

namespace also works

Hello, I am trying to set up rancher on AlmaLinunx/Rocky/Red Hat 8.5 with SELinux and FirewallD enabled. The docker image keeps restarting. Is there anything I can do to make it work with SELinux enforcing?

hi, mistakes were made and I started rke2-server on a machine that was running rke2-agent. I've tried to revert this but I am not able to stop the etcd, cloud-controller-manager, kube-controller-manager, kube-scheduler and kube-api-server pods from trying to start. Is anyone able to point me to some procedure to revert this mistake?

Hello, we have a cluster deployed, k8s 1.22, when we are trying to make use of the

vsphere CSI controller

, we kept getting an i/o timeout when connecting to vsphere and the only way around it was to add our name server to the k8s netpol, obviously we dont want to do that, we deployed another test pod using the exact same netpol however we excluded the name server and it worked fine. We noticed that dnsPolicy on the vsphere CSI controller is

Default

, we updated that too

ClusterFirst

and could see the connection working as expected, however now the pods keep going into.

CrashLoopBackoff

with the following error:

Still connecting to unix:///csi/csi.sock

Any ideas?

Im pretty new to rke2 and kubernetes. Is there any good resources for learning how to do canal/calico policies for your rke2 cluster? Does it require you to have calicoctl CLI?

I have an ongoing issue with RKE2 here. So far I cannot find any way to fix it. This seems to relate to issue 3176 on Github which is still open. Now I want to follow this guide and see if it works. So I need a token for the

config.yaml

file before first starting RKE2 server. How can I generate that K10 token without Kubernetes already up an running?

Hi, I have deployed RKE2 using air-gapped installation procedure and the cluster is up and running. Now I want to change the hostnames of all the nodes (both server and agent nodes). I have manually updated the hostnames of the nodes and rebooted the VMs but the nodes went into "NotReady" state. I have deleted the "NotReady" nodes but unable to delete the server agent. Also I see most of the pods are still scheduled on server agent so unable to delete it. So, please share me any references or share me the steps to rename the hostnames of the RKE2 cluster without impacting any pods. Any help would be appreciated.

Attempting to add agent nodes to an rke2 v1.25.5+rke2r1, and the token which was provided for the original cluster is not being accepted by the agents. Do I need to extract the agent token to add new agents or what? "Waiting to retrieve agent configuration; server is not ready: Node password rejected, duplicate hostname or contents of '/etc/rancher/node/password' may not match server node-passwd entry, try enabling a unique node name with the --with-node-id flag""

I created a Cluster with 3 Node (HA Setup) and a LoadBalancer in front for rke2 and the k8s api I added the IP of the Loadbalancer to

tls-san

Cluster setup works fine, all 3 nodes are in the k8s cluster.

kubectl get node

does work when using the loaadbalancer IP in my kubeconfig. But when I want to execute a

kubectl logs <pod

I get an error

Error from server: Get "<https://10.0.0.4:10250/containerLogs/kube-system/cilium-9cg2l/cilium-agent?follow=true>": x509: certificate is valid for 127.0.0.1, <publicipv4>, <publicipv6>, not 10.0.0.4

the node has a public and internal interface and it seems the internal ip address is nod added to the kubelet certificate. How can I change this?

Does RKE follows same End-of-Life/End-of-Support lifecycle as vanilla k8s ? Or where can I query that information for every RKE release ?

Is there a rancher-api terraform module to deploy an RKE2 cluster? Or does it have to be done within the UI?

@orange-airplane-98016 has left the channel

Does rke2 support RHEL 8.6 and RHEL 9? Not sure if this docs are up-to-date. https://docs.rke2.io/install/requirements#operating-systems

After enabling cis-1.6 profile getting bunch of these pods with the following error: apply-system-agent-upgrader-on-phygatapp24-lab-with-2d753-zfhvx + TMPDIRBASE=/var/lib/rancher/agent/tmp + mkdir -p /host/var/lib/rancher/agent/tmp mkdir: can’t create directory ‘/host/var/lib/rancher/’: No such file or directory

Why is v1.24.10+rke2r1 tagged as a pre-release?

Hi Everyone, I am trying to setup my RKE2 cluster using ansible, but I have two issues that I can't solve. First, the server url to get the CA certs is the IPv6 of the first master that join, and in some cases, the other nodes can't resolve the url and I get the error "no route to host" when starting rke2 server

FATA[0006] starting kubernetes: preparing server: failed to get CA certs: Get "<https://xxxx>:xxxx:xxxx:xxxx:xxxx:xxxx:9345/cacerts": dial tcp [xxxx:xxxx:xxxx:xxxx:xxxx:xxxx]:9345: connect: no route to host

I manage to get it to work by editing the /etc/rancher/rke2/config.yaml.d/50-rancher.yaml file and changing the IPv6 to IPv4. But I got another issue for the worker nodes even after updating the url, the node gets a 401 unauthorized response, when I try to add other master nodes it works fine

FATA[0000] starting kubernetes: preparing server: <https://xxx.xxx.xxx.xxx:9345/v1-rke2/server-bootstrap>: 401 Unauthorized

Thanks for your time

Greetings, is it expected that the psa exemption namespaces listed in the following link should support all CNIs or just the default canal CNI? https://github.com/rancher/rke2/blob/0ab4614b13daa5a49e484249a49012918d46ed22/pkg/rke2/psa.go#L58

hello how can i recover from this