as i can see from https://github.com/rancher/rke2-charts not all versions have egress gateway function

I recently installed Rancher 2.6.5 since RKE2 was coming out of tech preview. I tried installing a downstream custom cluster but the option to have SELinux enforcing was missing (it'd been there but didn't work in 2.6.2). Is that coming back, or will RKE2 with SELinux enforcing always require installing manually and adding to Rancher by importing an existing cluster?

Hi everyone, was anyone able to integrate TCP (or UDP) ingress resources on RKE2 integrated Nginx Ingress Controller?. I can't find a clear recipe for this on the documentation.

Hi Team, I have a question regarding communications between rke2 clusters. I have an rke2 cluster in development with 6 nodes, 3 master and 3 workers, all the development is in a network range (xx.xxx.6) I have installed rancher as master in three other different nodes that is in another network range ( xx.xx.2). Can there be problems when adding the development cluster in rancher if they are in different red ranges and it is not reached?

Where in the docs does it go over why / what values you would set for the tls-san option in cluster.config?

The primary issue seems to be that none of the machines can now find etcd 🥵

Hi, what would be a best practice to manage the Helm Charts (and Helm Chart Configs) manifests used by RKE2? Manifests defined in the /var/lib/rancher/rke2/server/manifests are applied to the cluster. But what if you have multiple masters, should you define the manifest on all master nodes or just on a single node? And how deals the Helm Controller then when two different manifests (with the same name and namespace used) on two different nodes? My best bet is to write the manifests on all nodes, but not sure on that (and I can’t find any documentation on this)

for rke2 on openstack I can recommand https://github.com/remche/terraform-openstack-rke2 (main author of module here) 😉

Is there anything special about deploying grafana and prometheus on rke2? Ive tried to deploy it but it doesnt seem as smooth as vanilla kubernetes. It could be something im doing.

Can I do an array append using HelmChartConfig with a manifest file for RKE2 (i.e. https://docs.rke2.io/helm/#customizing-packaged-components-with-helmchartconfig)? Specifically I'd like to add the

--default-ssl-certificate

argument to

rke2-ingress-nginx-controller

config to spec.template.spec.containers.args array as an append so that all the default arguments are still there without me specifying (as I've noticed they change between versions at times). I know I can replace a value, but it isn't clear if I can specify an array append to what's there in a generic fashion.

i upgraded my ubuntu 20.04 to 5.15 kernel and (perhaps also in parallel installing openebs) but since then my etcd no longer works and the ctrl plane is dead. I have apparmor installed. I see error messages like this: "level=info msg="Waiting for containerd startup: rpc error: code = Unimplemented desc = unknown service runtime.v1alpha2.RuntimeService"" And i see error messages like this in etcd:

"/health error","output":"{\"health\":\"false\",\"reason\":\"RAFT NO LEADER\"}","status-code":503}

When i try to debug containerd/container with ctr or crictrl i always get that there is no containerd.sock (i can only find

containerd.sock.ttrpc

There was a weird issue on 2 ctrl planes: something about not enough filedescriptors. Unfortuna i was not able to see who tried to open too many files, but i'im not sure if this is not more of an containerd issue who starts 2 etcd container constantly My google and debug magic is gone. Any ideas/suggestions?

hello, I'm trying to set up a bare metal rke2 cluster and I am having issues adding new server nodes to my cluster. when I review the logs for rke2-server, I get the following error:

Jun 16 21:53:06 test-rke-server-1 rke2[7641]: F0616 21:53:06.418489    7641 csi_plugin.go:305] Failed to initialize CSINode after retrying: timed out waiting for the condition

Jun 16 21:53:06 test-rke-server-1 rke2[7641]: goroutine 796 [running]:

Jun 16 21:53:06 test-rke-server-1 rke2[7641]: <http://k8s.io/klog/v2.stacks(0xc000a0e101|k8s.io/klog/v2.stacks(0xc000a0e101>, 0xc0002122c0, 0x82, 0x291)

Jun 16 21:53:06 test-rke-server-1 rke2[7641]:         /go/src/kubernetes/vendor/k8s.io/klog/v2/klog.go:1026 +0xb9

Jun 16 21:53:06 test-rke-server-1 rke2[7641]: <http://k8s.io/klog/v2.(*loggingT).output(0x7648520|k8s.io/klog/v2.(*loggingT).output(0x7648520>, 0xc000000003, 0x0, 0x0, 0xc000586540, 0x0, 0x618e831, 0xd, 0x131, 0x0)

Jun 16 21:53:06 test-rke-server-1 rke2[7641]:         /go/src/kubernetes/vendor/k8s.io/klog/v2/klog.go:975 +0x1e5

Jun 16 21:53:06 test-rke-server-1 rke2[7641]: <http://k8s.io/klog/v2.(*loggingT).printf(0x7648520|k8s.io/klog/v2.(*loggingT).printf(0x7648520>, 0xc000000003, 0x0, 0x0, 0x0, 0x0, 0x4efe5d9, 0x2f, 0xc0016f2cb0, 0x1, ...)

Jun 16 21:53:06 test-rke-server-1 rke2[7641]:         /go/src/kubernetes/vendor/k8s.io/klog/v2/klog.go:753 +0x19a

Jun 16 21:53:06 test-rke-server-1 rke2[7641]: <http://k8s.io/klog/v2.Fatalf(...)|k8s.io/klog/v2.Fatalf(...)>

(error trace goes for hundreds of lines) has anyone run into this issue before/know what is wrong? was not able to find much from googling. I am running k8s/rke2 version 1.22.10+rke2r2 on centos 8 stream note that this only happens on the second server node, my first node is fine:

NAME                STATUS   ROLES                       AGE     VERSION

test-rke-server-0   Ready    control-plane,etcd,master   4h44m   v1.22.10+rke2r2

Good morning! In order to pass extra args to kube-controller-manager and kube-scheduler where should those go? In the rke_config in my main.tf? I see that the below are available in the cli. So where do i add them?

--kube-controller-manager-arg value          
 --kube-scheduler-arg value

kinda a shot in the dark here, but does anyone know if you can use a service principal for azure-disk managed StorageClass? https://kubernetes.io/docs/concepts/storage/storage-classes/#azure-disk I'm using a RKE2 deployed on a VMSS in azure and the kube-controller-manager is trying to use the User Managed Identity attached to the VMSS to create PVCs, which doesn't have access to do so, and I'd like to avoid giving it access to do so. I'm using a service principal in the azure cloud conf

Whether Rancher_backups take backup of entire cluster like velero?

Good morning. I have a situation in a cluster with Server & Client version mismatch. Wanted to fix client version mismatch and make client minor version to 21(same as server)

$ kubectl version

Client Version: version.Info{Major:“1”, Minor:“23", GitVersion:“v1.23.6”, GitCommit:“ad3338546da947756e8a88aa6822e9c11e7eac22", GitTreeState:“clean”, BuildDate“2022 04 14T0849:13Z”, GoVersion:“go1.17.9”, Compiler:“gc”, Platform:“linux/amd64”}

Server Version: version.Info{Major:“1", Minor:“21”, GitVersion:“v1.21.5+rke2r2", GitCommit:“aea7bbadd2fc0cd689de94a54e5b7b758869d691”, GitTreeState:“clean”, BuildDate“2021 10 04T2239:02Z”, GoVersion:“go1.16.7b7", Compiler:“gc”, Platform:“linux/amd64"}

WARNING: version difference between client (1.23) and server (1.21) exceeds the supported minor version skew of +/-1

$ kubectl get nodes

NAME STATUS ROLES AGE VERSION

server1 Ready <none> 10d v1.22.9+rke2r2

server2 Ready <none> 10d v1.21.5+rke2r2

server3 Ready,SchedulingDisabled control-plane,etcd,master 58d v1.21.5+rke2r2

I did following steps. 1. Drained server1, 2. rke2-killall.sh in server1 3. Replaced ‘/usr/local/bin/rke2*’ in server1 with server2 binaries 4. started agent in server1 “systemctl start rke2-agent” it became like following.

$ kubectl get nodes

NAME STATUS ROLES AGE VERSION

server1 Ready <none> 10d v1.21.5+rke2r2

server2 Ready <none> 10d v1.21.5+rke2r2

server3 Ready,SchedulingDisabled control-plane,etcd,master 58d v1.21.5+rke2r2

Still it shows client version mismatch. would like to know how to update this Client version mismatch.

$ kubectl version

Client Version: version.Info{Major:“1", Minor:“23”, GitVersion:“v1.23.6", GitCommit:“ad3338546da947756e8a88aa6822e9c11e7eac22”, GitTreeState:“clean”, BuildDate“2022 04 14T0849:13Z”, GoVersion:“go1.17.9", Compiler:“gc”, Platform:“linux/amd64"}

Server Version: version.Info{Major:“1”, Minor:“21", GitVersion:“v1.21.5+rke2r2”, GitCommit:“aea7bbadd2fc0cd689de94a54e5b7b758869d691", GitTreeState:“clean”, BuildDate“2021 10 04T2239:02Z”, GoVersion:“go1.16.7b7”, Compiler:“gc”, Platform:“linux/amd64”}

WARNING: version difference between client (1.23) and server (1.21) exceeds the supported minor version skew of +/-1

Quick question. What's the minimum hardware requirements for rke2 server running Rancher and agents? I was going to use a node with 16gb ram for the control plane and nodes with 8gb for agent nodes. This will be a small production cluster with a total of 9 machines / servers.

hey guys! I am trying to provision rke2 via rancher with custom CNI which is

aws-eni-cni

and having interesting issue, the bootstrap controlplane node starts-up fine but other nodes are failing as they get wrong

"server": "<https://IP:9345>",

that

IP

isn’t the node IP and I have no idea how it was discovered… I did check I have no pods running with such IP on bootstrap node… however that IP is one of the EC2 instance secondary IP addresses… any idea how can I fix this? 🙏

Hi all, does apparmor still need to be disabled before installing? Also, does anyone have a requirements checklist before installing rke2 server and agent?

Has anyone had to prep their ubuntu 20.04 servers to run the agent and server? (for example)

sudo swapoff -a
sudo sed -i '/ swap / s/^\(.*\)$/#\1/g' /etc/fstab

systemctl stop ufw
systemctl disable ufw

create file /etc/sysctl.d/90-rke2.conf
net.ipv4.conf.all.forwarding=1
net.ipv6.conf.all.forwarding=1

modify /etc/resolv.conf
nameserver from 127.0.0.53 to 8.8.8.8
search from <http://attlocal.net|attlocal.net> to <http://my.domain.com|my.domain.com>

hi, i was trying to join a rke2 server node to the initial rke2 server node, but everytime i tried this, the initial server got stuck, i was able to get the

ps

before it become unresponsive:

PID USER      PR  NI    VIRT    RES    SHR S  %CPU  %MEM     TIME+ COMMAND
    833 root      20   0  948732 156860  89276 S 100.3   0.2   0:17.59 rke2
   1426 root      20   0 1045476 317188  74388 S  70.4   0.5   0:02.39 kube-apiserver
   1348 root      20   0   10.7g  62184  24912 S  10.3   0.1   0:00.41 etcd
    992 root      20   0  830948  95468  64436 S   2.3   0.1   0:00.38 kubelet
   1489 root      20   0  758424  50276  35488 S   1.0   0.1   0:00.50 kube-scheduler
    973 root      20   0  766448  58876  38472 S   0.7   0.1   0:00.34 containerd
   1508 root      20   0  752084  35188  27144 S   0.7   0.1   0:00.05 kube-proxy

after a while, i was logged off from the ssh connection:

client_loop: send disconnect: Broken pipe

the server spec looks fine: each server has 6 physical cores, 64gb memory os: ubuntu 22.04 rke2 version: latest stable anyone knows what could be wrong? happy to provider any info

Does this output look like everything is ready to go for rancher install?

NAMESPACE      NAME                                                    READY   STATUS      RESTARTS   AGE
cert-manager   cert-manager-76d44b459c-qfdgq                           1/1     Running     0          4m26s
cert-manager   cert-manager-cainjector-9b679cc6-rnzcc                  1/1     Running     0          4m26s
cert-manager   cert-manager-webhook-57c994b6b9-r7cxs                   1/1     Running     0          4m26s
kube-system    cloud-controller-manager-rke2                           1/1     Running     0          18m
kube-system    etcd-rke2                                               1/1     Running     0          17m
kube-system    helm-install-rke2-canal-zvs8x                           0/1     Completed   0          17m
kube-system    helm-install-rke2-coredns-frgcz                         0/1     Completed   0          17m
kube-system    helm-install-rke2-ingress-nginx-8mvg9                   0/1     Completed   0          17m
kube-system    helm-install-rke2-metrics-server-rp2jz                  0/1     Completed   0          17m
kube-system    kube-apiserver-rke2                                     1/1     Running     0          17m
kube-system    kube-controller-manager-rke2                            1/1     Running     0          18m
kube-system    kube-proxy-rke2                                         1/1     Running     0          18m
kube-system    kube-scheduler-rke2                                     1/1     Running     0          18m
kube-system    rke2-canal-nkrmd                                        2/2     Running     0          16m
kube-system    rke2-coredns-rke2-coredns-547d5499cb-4g4jj              1/1     Running     0          16m
kube-system    rke2-coredns-rke2-coredns-autoscaler-65c9bb465d-m2d8d   1/1     Running     0          16m
kube-system    rke2-ingress-nginx-controller-s977x                     1/1     Running     0          11m
kube-system    rke2-metrics-server-6564db4569-9h8mk                    1/1     Running     0          13m

hi, i was trying to setup a control plane with 3 rke2 servers. the first node was bootstrapped successfully, but when i tried to join the second node. i got the following error. i verified the error is consistent. please see the full log from

journalctl -u rke2-server -f

in the thread. os: ubuntu 22.04 rke2 version: latest stable

hi, i was able to connect to the rke2 cluster locally, however, after i updated some of the rke2 cluster parameters and restart, i can't connect to it locally anymore. but i am still able to connect to it on the sever node. anyone know what could be the cause here?

Hi, I am trying to install RKE2 on a centos stream 8 with 3 server nodes and 3 agent nodes...The installation is successful on the first server node but the installation fails on second and third server nodes due to the configuration mismatch error.

Jun 27 10:17:43 <http://rke2-master2.xxx.xxx.xxx.43.nip.io|rke2-master2.xxx.xxx.xxx.43.nip.io> rke2[38318]: time="2022-06-27T10:17:43-04:00" level=fatal msg="starting kubernetes: preparing server: failed to validate server configuration: critical configuration value mismatch"

First Server configuration:

# BEGIN Adding RKE2 configuration
write-kubeconfig-mode: "0644"
tls-san:
- "<http://rke2-master1.xxx.xxx.xxx.42.nip.io|rke2-master1.xxx.xxx.xxx.42.nip.io>"
node-label:
- "nodetype=master"
node-ip: "xxx.xxx.xxx.42,xxxx:xxx:x:xxx:xxx:xxxx:xxxx:aae"
cluster-cidr: "10.42.0.0/16,2001:cafe:42:0::/56"
service-cidr: "10.43.0.0/16,2001:cafe:42:1::/112"
cluster-dns: "10.43.0.10"
cluster-domain: "<http://rke2-master1.xxx.xxx.xxx.42.nip.io|rke2-master1.xxx.xxx.xxx.42.nip.io>"
cni:
- calico
disable:
- rke2-canal
- rke2-kube-proxy
# END Adding RKE2 configuration

Second Server Configuration:

# BEGIN Adding RKE2 configuration
server: "<https://rke2-master1.xxx.xxx.xxx.42.nip.io:9345>"
token: "K10d463a80c8c1323f30fa6d97fcf91992454a43dc5c544f1c9a0de706b733b51ee::server:f6fd26cafff902300ba021b29b11eddc"
tls-san:
- "<http://rke2-master1.xxx.xxx.xxx.42.nip.io|rke2-master1.xxx.xxx.xxx.42.nip.io>"
node-ip: "xxx.xxx.xxx.43,xxxx:xxx:x:xxx:xxx:xxxx:xxxx:5245"
cni:
- calico
disable:
- rke2-canal
- rke2-kube-proxy
# END Adding RKE2 configuration

There is no firewalld or iptables running on any of the nodes...Please help me in finding the issue here

Greetings, does rke2 have a container registry built-in? our developers are using a container registry on a separate server. thanks!

Greetings! has anyone experienced an error like “Error from server: error dialing backend: … … connect: no route to host” - i found a solution here: https://github.com/rancher/rke2/issues/662 and made the proposed change in /etc/rancher/rke2/config.yaml. It fixed the issue on the first node but on nodes 2 and 3 (setup as rke-agent nodes) we still have the error. Any ideas?

Hi guys, just trying to get some background knowledge on the state of FIPS compliance for some of the CNI plugins. based on the network options RKE2 supports other CNIs on top of Canal. However, only Canal is FIPS compliant:

As of v1.21.2, RKE2 supports selecting a different CNI via the

--cni

flag and comes bundled with several CNIs including Canal (default), Calico, Cilium, and Multus. Of these, only Canal (the default) is rebuilt for FIPS compliance.

I want to understand why RKE2 supports these different CNIs, but doesn't recompile them for FIPS compliance. Doesn't that go against RKE2's ethos of having a fully conformant distribution for US Government sector customers? Are there any options for people that want to use something like Multus, Calico (Enterprise) or Cilium but need all encryption to be FIPS validated?

Hello, I am attempting to install a specific version of RKE2 using:

curl -sfL <https://get.rke2.io> | INSTALL_RKE2_VERSION=1.22.9+rke2r2 sh -

but the installation script reports "No package rke-server-1.22.9-rke2r2 available". This release exists: https://github.com/rancher/rke2/releases/tag/v1.22.9%2Brke2r2 I've tried various combinations of INSTALL_RKE2_CHANNEL, INSTALL_RKE2_VERSION, and INSTALL_RKE2_COMMIT, with no change in behavior. Is this expected, or should I be able to install a specific release?

Hello! I have installed rke2 with 3 nodes, everything works fine, but adding additional ingress-es with different ingressClassName still just shows the nginx 404 error page, am I understanding correctly when I add other ingress controllers it should use them?