Hi, I have successfully installed kube-vip on an rke2 cluster. Now I have 2 sets of kubernetes(rke2) clusters in which one of them acts as an Active cluster and the other as Standby cluster. I want the kube-vip to be assigned to the Active cluster control plane nodes and when the Active cluster fails the kube-vip needs to be assigned to Standby cluster control plane nodes. Please let me know if kube-vip alone can do it or else the same can be achieved thorough a combination of other tools like keepalived.

Hello folks, I'm setting up OIDC authentication on a RKE2 cluster but something is not working, thus I'd like to check the kube-apiserver logs but I cannot find them. I expected them to be in

/var/lib/rancher/rke2/server/logs

but the directory is empty. Do you know where I should look for?

I'm trying to get zfs localpv from openebs running but the installation remarks that you need to update/fix the kubelet path: https://github.com/openebs/zfs-localpv while this apparently works for rancheros, rke2 puts it under a release specific folder. Anyone has an idea how to get it running?

I'm unable to find anything regarding adding node annotations on creation. node-label exists. Any hints?

Good Morning, where can i find information of the mirror charts like rke2-cilum chart? Ist there any version list and when can i expect that the packaged cilium version is up to v1.12?

First post. In a high-availability RKE2 cluster (using kube-vip and 3 servers) should the file /etc/rancher/rke2/config.yaml be identical on the three servers (server1, server2, server3)? It seems to work although server1 lacks the lines for token: and the server: I did 1. Install server1 2. Install kube-vip 3. install server2 and server3 by adding two lines for token: and server: to the file /etc/rancher/rke2/config.yaml I am wondering if I should edit /etc/rancher/rke2/config.yaml on server1 to make it identical to the same file on the server2 and server3? Currently /etc/rancher/rke2/config.yaml on server2 looks something like this:

token: masked
server: <https://cluster.example.com:9345>
tls-san:
- server2
- <http://server2.example.com|server2.example.com>
- <http://cluster.example.com|cluster.example.com>
- 12.34.56.78
disable: rke2-ingress-nginx
disable-kube-proxy: true
cni:
- cilium

(I masked the kubevip IP address and wrote 12.34.56.78 instead)

A clarification regarding the word identical in the question:

In a high-availability RKE2 cluster (using kube-vip and 3 servers)
should the file /etc/rancher/rke2/config.yaml
be identical on the three servers (server1, server2, server3)?

I meant identical except for the section tls-san where the files can differ. For example server1:

tls-san:
- server1
- <http://server1.example.com|server1.example.com>
- <http://cluster.example.com|cluster.example.com>
- 12.34.56.78

server2:

tls-san:
- server2
- <http://server2.example.com|server2.example.com>
- <http://cluster.example.com|cluster.example.com>
- 12.34.56.78

Hi, I need recommendation for loadbalncers for the two use cases mentioned below. I have two sets of kubernetes(RKE2) clusters located in two different geographical locations. use case#1: One of the clusters act as an Active cluster and the other as a Standby cluster. Now I need a recommendation for loadbalancer before these two clusters to route the traffic always to the Active cluster. If the control nodes of the Active cluster fail then the Standby cluster should be elected as Active cluster and the traffic should route to the newly elected Active cluster. use case#2: Both the clusters act as Active clusters. Here I need another recommendation for a loadbalancer to route the traffic to the cluster that responds quicker than the other.

Greetings friends. I have a 3 node RKE2 cluster for a group of rascals (developers). I would like to set up monitoring and get alerts if a node goes down e.g. rke2-server process dies for any reason. In “my days” (yeah im older) we used things like monit to watch these processes. I was wonderfing if there were more ‘cloud-native’ ways to do this? thanks in advance!

Hello, I’m trying to install RKE2 but I have some issues. First, this command doesn’t give any feedback:

~$ sudo curl -sfL <https://get.rke2.io> | sh -

And after that, I receive this error with the second command:

~$ sudo systemctl enable rke2-server.service
Failed to enable unit: Unit file rke2-server.service does not exist.

Any hit on this? Thanks

Hi Everyone, I have a HA rke2 custom cluster which i deployed via Rancher (Downstream). This has 3 Master nodes and 3 workers. So far i see, my

kubeconfig

for this new cluster has my rancher-server as the server endpoint so I suppose the rancher is balancing the connections between me and the masters nodes, is that correct? Additionally, I did not see any configuration on the worker nodes for the

kubelet

to talk to the masters in HA. So it seems to me that e.g

worker-1

can talk only to

master-1

. I have seen people who setup a

ha-proxy

on the worker nodes so they are able to communicate to any of the masters, but here i do not see any setup like that by default.

Hi Rancher Team, I have an RKE2 in production based on v1.21.5+rke2r1. We need to upgrade to v1.23.4+rke2r2 (this is very specific). I upgrade by stopping the rke2 service, uploading the new version and restarting the service. However, this intermittently passed/fails. We have also seen this in our Dev/QA env. i.e Starting/Restarting Rke2 service

ok.... need a sanity check. running rancher standalone right now which I should probably properly deploy 'in' my cluster (not best practice... but shrug) It is 2.6.6 and the local k3s is v1.23.6+k3s1 my downstream cluster is on GKE and is at v1.23.8-gke.1900. I would like to upgrade to RKE2. ignoring my HA issues with Rancher (I'll figure out that with the infamous, 'later')... what is the best upgrade path?

so thought I'd just try what was in https://docs.rke2.io/upgrade/automated_upgrade/ and ended up with this /system-upgrade$ kubectl apply -f https://github.com/rancher/system-upgrade-controller/releases/download/v0.9.1/system-upgrade-controller.yaml -n system-upgrade namespace/system-upgrade unchanged serviceaccount/system-upgrade unchanged clusterrolebinding.rbac.authorization.k8s.io/system-upgrade unchanged configmap/default-controller-env unchanged deployment.apps/system-upgrade-controller unchanged ~/system-upgrade$ kubectl apply -f rke2.yml -n system-upgrade resource mapping not found for name: "server-plan" namespace: "system-upgrade" from "rke2.yml": no matches for kind "Plan" in version "upgrade.cattle.io/v1" ensure CRDs are installed first resource mapping not found for name: "agent-plan" namespace: "system-upgrade" from "rke2.yml": no matches for kind "Plan" in version "upgrade.cattle.io/v1" ensure CRDs are installed first

rke2.yml is the plan from that docs.rke2.io link above.

any ideas? I see the config-map applied in the system-upgrade namespace.

or..... is there a way to pull all the yaml out of Rancher so I can reapply everything other than the cluster.yml if I recreate a cluster and ensure it is rke2 from scratch. I have that option.

@creamy-pencil-82913, Continuing https://rancher-users.slack.com/archives/C01PHNP149L/p1660833392861649 Followed https://docs.rke2.io/backup_restore/#restoring-a-snapshot-to-new-nodes and created a HA cluster, static pods are up. But failing to attach volumes.

>> kubectl get pod -n kube-system

cilium-4xc5q                                            1/1     Running            0          8h
cilium-89vrg                                            1/1     Running            0          8h
cilium-cg8gn                                            1/1     Running            6          8h
cilium-gbbl7                                            1/1     Running            1          8h
cilium-j8s9t                                            1/1     Running            3          8h
cilium-jfs9f                                            1/1     Running            1          179m
cilium-ld9fc                                            1/1     Running            0          8h
cilium-lz2hj                                            1/1     Running            0          8h
cilium-node-init-7ltcv                                  1/1     Running            0          8h
cilium-node-init-gzhvc                                  1/1     Running            0          8h
cilium-node-init-hqnrk                                  1/1     Running            0          179m
cilium-node-init-j2ffd                                  1/1     Running            0          8h
cilium-node-init-j5q52                                  1/1     Running            3          8h
cilium-node-init-mmbjj                                  1/1     Running            0          8h
cilium-node-init-qk6pj                                  1/1     Running            1          8h
cilium-node-init-w87qb                                  1/1     Running            3          8h
cilium-node-init-zfrt9                                  1/1     Running            0          8h
cilium-nxqxb                                            1/1     Running            0          8h
cilium-operator-fccb67dc5-srt76                         1/1     Running            5          8h
cilium-operator-fccb67dc5-wsr5m                         1/1     Running            3          8h
cloud-controller-manager-sv-svr1                           1/1     Running            3          9h
cloud-controller-manager-sv-svr2                           1/1     Running            3          8h
cloud-controller-manager-sv-svr3                           1/1     Running            3          8h
etcd-sv-svr1                                               1/1     Running            8          9h
etcd-sv-svr2                                               1/1     Running            3          8h
etcd-sv-svr3                                               1/1     Running            3          147m
external-dns-dc9dd7d74-h6dqw                            1/1     Running            1          90d
helm-install-rke2-metrics-server-cmgjc                  0/1     CrashLoopBackOff   72         5h40m
kube-apiserver-sv-svr1                                     1/1     Running            1          9h
kube-apiserver-sv-svr2                                     1/1     Running            3          8h
kube-apiserver-sv-svr3                                     1/1     Running            3          140m
kube-controller-manager-sv-svr1                            1/1     Running            3          9h
kube-controller-manager-sv-svr2                            1/1     Running            3          8h
kube-controller-manager-sv-svr3                            1/1     Running            3          8h
kube-proxy-sv-agent3                                         1/1     Running            0          7h40m
kube-proxy-sv-agent4                                         1/1     Running            0          8h
kube-proxy-sv-agent5                                         1/1     Running            0          8h
kube-proxy-sv-agent6                                         1/1     Running            0          8h
kube-proxy-sv-svr1                                         1/1     Running            1          9h
kube-proxy-sv-svr2                                         1/1     Running            3          8h
kube-proxy-sv-svr3                                         1/1     Running            3          8h
kube-proxy-sv-agent1                                          1/1     Running            0          8h
kube-proxy-sv-agent2                                          1/1     Running            0          3h
kube-scheduler-sv-svr1                                     1/1     Running            3          9h
kube-scheduler-sv-svr2                                     1/1     Running            3          8h
kube-scheduler-sv-svr3                                     1/1     Running            3          8h
kube-vip-cloud-provider-0                               1/1     Running            3          8h
kube-vip-ds-5q5qw                                       1/1     Running            3          8h
kube-vip-ds-fw8zv                                       1/1     Running            3          8h
kube-vip-ds-rmqhc                                       1/1     Running            4          8h
metrics-server-8bbfb4bdb-rzpnp                          1/1     Running            5          7h33m
rke2-coredns-rke2-coredns-855c5d9879-9fwhx              1/1     Running            0          5h40m
rke2-coredns-rke2-coredns-855c5d9879-j7wbc              0/1     CrashLoopBackOff   41         3h3m
rke2-coredns-rke2-coredns-autoscaler-7c77dcfb76-hm78m   1/1     Running            3          8h
rke2-ingress-nginx-controller-4kvdx                     1/1     Running            2          8h
rke2-ingress-nginx-controller-8k5z8                     1/1     Running            0          8h
rke2-ingress-nginx-controller-c6r5q                     1/1     Running            0          179m
rke2-ingress-nginx-controller-cx88s                     1/1     Running            0          8h
rke2-ingress-nginx-controller-jl74q                     1/1     Running            1          8h
rke2-ingress-nginx-controller-nr2qp                     1/1     Running            8          8h
rke2-ingress-nginx-controller-p6sfq                     1/1     Running            3          8h
rke2-ingress-nginx-controller-qmbzn                     1/1     Running            0          8h
rke2-ingress-nginx-controller-wj54z                     1/1     Running            0          8h
rke2-metrics-server-5df7d77b5b-b4qlw                    1/1     Running            20         74d

Following are the errors noticed and volumes are not mounted.

E0911 20:16:38.965933   17195 kubelet.go:1701] "Unable to attach or mount volumes for pod; skipping pod" err="unmounted volumes=[data], unattached volumes=[data kube-api-access-ztp4j dshm]: timed out waiting for the condition" pod="cvat/cvat-postgresql-0"

E0911 20:23:07.393663   16782 pod_workers.go:190] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"container\" with CrashLoopBackOff: \"back-off 5m0s restarting failed container=container pod=metadata-grpc-deployment-f8d68f687-5fvbs_kubeflow(d72591f7-e2c4-475f-ad83-fc59c996219a)\"" pod="kubeflow/metadata-grpc-deployment-f8d68f687-5fvbs" podUID=d72591f7-e2c4-475f-ad83-fc59c996219a
I0911 20:23:08.718940   16782 reconciler.go:224] "operationExecutor.VerifyControllerAttachedVolume started for volume \"pvc-62552b22-3e99-4b63-8a56-69519573ae1d\" (UniqueName: \"<http://kubernetes.io/csi/driver.longhorn.io^pvc-62552b22-3e99-4b63-8a56-69519573ae1d\|kubernetes.io/csi/driver.longhorn.io^pvc-62552b22-3e99-4b63-8a56-69519573ae1d\>") pod \"loki-0\" (UID: \"8aef7574-fb66-415f-a130-6b8ec9091672\") "
E0911 20:23:08.724147   16782 nestedpendingoperations.go:335] Operation for "{volumeName:<http://kubernetes.io/csi/driver.longhorn.io^pvc-62552b22-3e99-4b63-8a56-69519573ae1d|kubernetes.io/csi/driver.longhorn.io^pvc-62552b22-3e99-4b63-8a56-69519573ae1d> podName: nodeName:}" failed. No retries permitted until 2022-09-11 20:25:10.724134581 +0000 UTC m=+21624.816950484 (durationBeforeRetry 2m2s). Error: "Volume not attached according to node status for volume \"pvc-62552b22-3e99-4b63-8a56-69519573ae1d\" (UniqueName: \"<http://kubernetes.io/csi/driver.longhorn.io^pvc-62552b22-3e99-4b63-8a56-69519573ae1d\|kubernetes.io/csi/driver.longhorn.io^pvc-62552b22-3e99-4b63-8a56-69519573ae1d\>") pod \"loki-0\" (UID: \"8aef7574-fb66-415f-a130-6b8ec9091672\") "
I0911 20:23:09.829046   16782 reconciler.go:224] "operationExecutor.VerifyControllerAttachedVolume started for volume \"pvc-c6597566-f0c6-40b3-be5b-9d670f51748d\" (UniqueName: \"<http://kubernetes.io/csi/driver.longhorn.io^pvc-c6597566-f0c6-40b3-be5b-9d670f51748d\|kubernetes.io/csi/driver.longhorn.io^pvc-c6597566-f0c6-40b3-be5b-9d670f51748d\>") pod \"harbor-redis-0\" (UID: \"912226dd-12cf-4cb5-a54b-fb831b4e7e73\") "
E0911 20:23:09.831850   16782 nestedpendingoperations.go:335] Operation for "{volumeName:<http://kubernetes.io/csi/driver.longhorn.io^pvc-c6597566-f0c6-40b3-be5b-9d670f51748d|kubernetes.io/csi/driver.longhorn.io^pvc-c6597566-f0c6-40b3-be5b-9d670f51748d> podName: nodeName:}" failed. No retries permitted until 2022-09-11 20:25:11.831837052 +0000 UTC m=+21625.924652956 (durationBeforeRetry 2m2s). Error: "Volume not attached according to node status for volume \"pvc-c6597566-f0c6-40b3-be5b-9d670f51748d\" (UniqueName: \"<http://kubernetes.io/csi/driver.longhorn.io^pvc-c6597566-f0c6-40b3-be5b-9d670f51748d\|kubernetes.io/csi/driver.longhorn.io^pvc-c6597566-f0c6-40b3-be5b-9d670f51748d\>") pod \"harbor-redis-0\" (UID: \"912226dd-12cf-4cb5-a54b-fb831b4e7e73\") "

Any inputs to recover?

Hi All, I am attempting to load images into my airgap environment and running into an issue. I am attempting to install both

cert-manager

and

kube-vip

. I have created two tars: cert-manager:

docker save <http://quay.io/jetstack/cert-manager-cainjector:v1.9.1|quay.io/jetstack/cert-manager-cainjector:v1.9.1>

<http://quay.io/jetstack/cert-manager-controller:v1.9.1|quay.io/jetstack/cert-manager-controller:v1.9.1>

<http://quay.io/jetstack/cert-manager-webhook:v1.9.1|quay.io/jetstack/cert-manager-webhook:v1.9.1>

<http://quay.io/jetstack/cert-manager-ctl:v1.9.1|quay.io/jetstack/cert-manager-ctl:v1.9.1> | gzip > cert-manager.tar.gz

kube-vip:

docker save <http://ghcr.io/kube-vip/kube-vip:v0.5.0|ghcr.io/kube-vip/kube-vip:v0.5.0> | gzip > kube-vip.tar.gz

I have copied both into the images directory:

root@rke-test-cluster-node-0:~# ls /var/lib/rancher/rke2/agent/images/
cert-manager.tar.gz  kube-vip.tar.gz  rke2-images.linux-amd64.tar.zst

Cert-manager

is able to come up without issue, but I run into issues with

kube-vip

.

Failed to pull image "<http://ghcr.io/kube-vip/kube-vip:v0.5.0|ghcr.io/kube-vip/kube-vip:v0.5.0>": rpc error: code = Unknown desc = failed to pull and unpack image "<http://ghcr.io/kube-vip/kube-vip:v0.5.0|ghcr.io/kube-vip/kube-vip:v0.5.0>": failed to resolve reference "<http://ghcr.io/kube-vip/kube-vip:v0.5.0|ghcr.io/kube-vip/kube-vip:v0.5.0>": failed to do request: Head "<https://ghcr.io/v2/kube-vip/kube-vip/manifests/v0.5.0>": dial tcp 140.82.112.33:443: i/o timeout

When listing my available images, the

kube-vip

image seems to be available:

root@rke-test-cluster-node-0:~# /var/lib/rancher/rke2/bin/crictl images | grep -e kube-vip -e cert-manager
<http://ghcr.io/kube-vip/kube-vip|ghcr.io/kube-vip/kube-vip>                                       v0.5.0                         09067696476ff       37.9MB
<http://quay.io/jetstack/cert-manager-cainjector|quay.io/jetstack/cert-manager-cainjector>                        v1.9.1                         11778d29f8cc2       39.2MB
<http://quay.io/jetstack/cert-manager-controller|quay.io/jetstack/cert-manager-controller>                        v1.9.1                         8eaca4249b016       57.2MB
<http://quay.io/jetstack/cert-manager-ctl|quay.io/jetstack/cert-manager-ctl>                               v1.9.1                         0a3af10d53674       50.2MB
<http://quay.io/jetstack/cert-manager-webhook|quay.io/jetstack/cert-manager-webhook>                           v1.9.1                         d3348bcdc1e7e       45.8MB

It seems it is trying to reach out to the internet for the image, rather than use the image available locally. Could someone provide any insight into what settings if any I can look into or steps I can take to further debug this?

Hi All, While I start rke2-server service on first node of HA cluster, i see following lines in journalctl.

Sep 13 17:41:21 svmaster rke2[14824]: time="2022-09-13T17:41:21+09:00" level=info msg="Latest etcd manifest deployed"
Sep 13 17:41:22 svmaster rke2[14824]: {"level":"warn","ts":"2022-09-13T17:41:22.837+0900","caller":"grpclog/grpclog.go:60","msg":"grpc: addrConn.createTransport failed to connect to {<https://127.0.0.1:2379>  <nil> 0 <nil>}. Err :connection error: desc = \"transport: Error while dialing dial tcp 127.0.0.1:2379: connect: connection refused\". Reconnecting..."}

Sep 13 17:43:23 svmaster rke2[21854]: time="2022-09-13T17:43:23+09:00" level=info msg="Stopped tunnel to 127.0.0.1:9345"
Sep 13 17:43:23 svmaster rke2[21854]: time="2022-09-13T17:43:23+09:00" level=info msg="Proxy done" err="context canceled" url="<wss://127.0.0.1:9345/v1-rke2/connect>"
Sep 13 17:43:23 svmaster rke2[21854]: time="2022-09-13T17:43:23+09:00" level=info msg="Connecting to proxy" url="<wss://192.168.7.15:9345/v1-rke2/connect>"
Sep 13 17:43:23 svmaster rke2[21854]: time="2022-09-13T17:43:23+09:00" level=info msg="error in remotedialer server [400]: websocket: close 1006 (abnormal closure): unexpected EOF"
Sep 13 17:43:23 svmaster rke2[21854]: time="2022-09-13T17:43:23+09:00" level=info msg="Handling backend connection request [svmaster]"

And in agent journalctl logs following are displayed.

Sep 13 18:03:02 svagent rke2[2659960]: W0913 18:03:02.914417 2659960 clientconn.go:1223] grpc: addrConn.createTransport failed to connect to {<https://127.0.0.1:2379>  <nil> 0 <nil>}. Err :connection error: desc = "transport: Error while dialing dial tcp 127.0.0.1:2379: connect: connection refused". Reconnecting...

Sep 13 18:03:06 svagent rke2[2659960]: time="2022-09-13T18:03:06+09:00" level=debug msg="Wrote ping"

any inputs to resolve?

Hey! I am trying to install a RKE2 cluster (vsphere provider) and change CNI to Cilium, but it’s stuck on Calico no matter what i do... I have this in my cluster.yaml rkeConfig: controlPlaneConfig: cni: cilium (also tried with none) And on the server it says: sudo cat /etc/rancher/rke2/config.yaml.d/50-rancher.yaml { “cni”: “calico”, } And it’s obviously installing calico pods..

Is there any migration path/utility for switching from RKE to RKE2? I did see a link https://docs.rke2.io/migration/ mentioned in slack earlier, but get a 404 error on the link.

Afternoon all. I’m having DNS issues with a recently installed RKE2 cluster on ESX VMs running Oracle Linux 8. I started with RKE2 1.21.14 but then upgraded to 1.22.13 after I saw a comment in this channel about networking issues on 1.21. The cluster has been happy and no errors when I look at the pods through

kubectl

. However, when I tried to access Rancher, after a successful install, the site never fully loads. I looked through logs and have determined that DNS isn’t working and that is causing the problem. I got a shell inside a container and confirmed that I can ping an IP but not a domain name. I’ve disabled firewalld, temporarily disabled SELinux, and I updated NetworkManager to ignore CNI traffic on all RKE2 nodes. The 3 management nodes and 3 worker nodes have no DNS issues, just the pods that do. I’m not sure what to try next. It appears the issue is communication between pods. Could anyone point me in the right direction on this? I’ve looked for generic coredns troubleshooting and nothing has helped me find the problem yet.

hmm is there no way to override the node names? my vsphere nodes deployed with helmchart rke2+vsphere provider is named after clustername-nodepoolname-random-random. in rke we used Name Prefix and then we could set for example node01 and then if we created 3 nodes we get node01-03

i have a weird issue with rke2 and cilium, when rancher use “system-default-registry”: “ourinternalregistry” the registry is replaced for some cilium images but not all. the init containers ‘mount-cgroup’ and apply-sysctl-overwrites’ part of the cilium agent stay on rancher/mirrored-cilium-cilium:v.12.0 but the rest use ourinternalregistry/rancher/mirrored-cilium-cilium:v.12.0… so i have to manually edit those two init containers for the daemonset to get cilium running

cross post frrom #rke But here it goes: I'm installing RKE2 agent on a windows worker. Should I just use the URL given from ranchers custom cluster -> registration. Or should I follow instructions from https://docs.rke2.io/install/windows_airgap/ (I cant find any other installation documentation for the agent)

When installing rke2 HA with multus and cilium do I need to make the same config across all nodes? Or just one?

I get pods with unknown statuses rn

Hi How to rotate Containerd Logs on RKE2. please suggest me