Hey everyone, I am trying out rke2 on an

Ubuntu 22.04.2 LTS

by going through the QuickStart and I have an issue where my CNI pod doesn't come up because it cannot talk to the api server, (i have tried different cnis and I get the same issue). Right now I have an instance with rke2 setup with these commands

ufw disable
curl -sfL <https://get.rke2.io> |  sh -
systemctl enable rke2-server.service
systemctl start rke2-server.service

I don't see any errors in the journalctl logs but the pod

rke2-canal-___

is stuck in an init crashloopbackoff. and from the logs of the

install-cni

container, I see that it cannot connect to the

kubernetes

service:

2023-05-23 10:36:11.795 [FATAL][1] cni-installer/<nil> <nil>: Unable to create token for CNI kubeconfig error=Post "<https://10.43.0.1:443/api/v1/namespaces/kube-system/serviceaccounts/canal/token>": dial tcp 10.43.0.1:443: i/o timeout

this is my service and endpoints:

root@k8s-master-1:~# kubectl get svc -owide
NAME         TYPE        CLUSTER-IP   EXTERNAL-IP   PORT(S)   AGE   SELECTOR
kubernetes   ClusterIP   10.43.0.1    <none>        443/TCP   14m   <none>

root@k8s-master-1:~# kubectl get endpoints -owide
NAME         ENDPOINTS            AGE
kubernetes   45.76.137.187:6443   15m

I can reach the endpoint

$ kubectl exec -it etcd-k8s-master-1 -nkube-system -- curl -vk <https://45.76.137.187:6443>
....
{
  "kind": "Status",
  "apiVersion": "v1",
  "metadata": {},
  "status": "Failure",
  "message": "Unauthorized",
  "reason": "Unauthorized",
  "code": 401
* Connection #0 to host 45.76.137.187 left intact

but I cannot reach the

kubernetes

service

$ kubectl exec -it etcd-k8s-master-1 -nkube-system -- curl -vk <https://10.43.0.1>
* Uses proxy env variable NO_PROXY == '.svc,.cluster.local,10.42.0.0/16,10.43.0.0/16'
*   Trying 10.43.0.1:443...
* TCP_NODELAY set

Hello folks, I was wondering: what is responsible for creating

/opt/cni/bin/

on agent nodes? I've had a case where an agent node doesn't have it but not sure why

I am curious, for on-prim RKE2 (v1.25 and newer) where PSP is not available, what some of you are using for loadbalancer? Metallb (latest v0.13.9 - if I am reading correctly) does not install on RKE v1.25. has anyone tried servicelb on RKE2?

Hello. I am trying to join a worker node to a rancher deployed rke2 cluster. but this node gives me this error level=error msg="failed to get CA certs: Get \"https://127.0.0.1:6444/cacerts\": EOF " if i try to restart rke2-server after a reboot, i get this error "Cluster CA certificate is not trusted by the host CA bundle, but the token does not include a CA hash. Use the full token from the server's node-token file to enable Cluster CA validation." google have lead me to think the worker can not talk to the controllplane, but using curl to test /cacerts reachabillity, works with ipv4 address, ipv6 address, hostname, and loadbalancer ip and host. anyone have an idea of where to troubleshoot next ?

I’m trying to do a fresh rke2 install on RHEL8.8 (the same way I did it on RHEL8.7) and it seems like something has changed.

systemctl start rke2-server

eventually times out and fails. It seems like it’s waiting for etcd startup which never happens. I’m guessing something changed in the rhel release, but I’m failing to grok what the issue is. Anyone seen this?

Any plans to support this: https://github.com/container-orchestrated-devices/container-device-interface#containerd-configuration I'm not seeing much around about rke2 + cdi

Is there a documentation that describe in detail about the certificates generated by RKE2? Theres a bunch of them under /var/lib/rancher/rke2/agent/ /var/lib/rancher/rke2/server/tls/ /var/lib/rancher/rke2/server/tls/etcd

Hi everybody, we have some problems with a rancher installed rke2 cluster on custom nodes. The setup is working. We after that did some HA tests and when the Rancher server was rebooted the user cluster broke. The etcd on the user cluster lost its quorum and 2 of 3 master nodes crashed. Can someone explain why this happens ? In my understanding a user cluster should survive a reboot of rancher. Any thoughts ?

I swear that I’ve seen a guide that talked about enabling a CIS Profile on an existing RKE2 cluster, but now I can’t seem to find it…am I losing my mind?

HI @here in one of my rancher managed downstream cluster worker node, i am observing error is "failed to get CA certs https://127.0.0.1:6444"

any idea on this?

What are the recommended resources to assign for the different node types when running RKE2? I’ve been transitioning from RKE1, and it looks like the control plane needs more cpu for RKE2, but I haven’t been able to find documentation that recommends reasonable allocations.

More of a Rancher question but, Is anyone running Rancher on RKE2 w/CIS Profile v1.23?

Hi, I'm testing metalLB in a Rancher Downstream cluster but failing to change the proxy-mode. As Rancher deployes the cluster with kube-proxy as a static pod I need to change all kube-proxy.yamls on all nodes. But thus far I'm failing to get it running in ipvs mode. I have the kernel modules loaded but still getting error of

level=warning msg="Running modprobe ip_vs failed with message: ``, error: exec: \"modprobe\": executable file not found in $PATH"

I'm trying to switch to cilium, is there any easy way? if I only change the config it doesn't sync (remove it)

Hi all, I’ve just setup a new simple RKE2 cluster using one node as master (aka server) and 3 nodes as workers. The server node gets the taints

Taints:             <http://node.kubernetes.io/not-ready:NoExecute|node.kubernetes.io/not-ready:NoExecute>
                    <http://node.kubernetes.io/not-ready:NoSchedule|node.kubernetes.io/not-ready:NoSchedule>

Therefore showing up as

NotReady

NAME                            STATUS     ROLES                       AGE     VERSION
k8s-2-master-01.herren5.local   NotReady   control-plane,etcd,master   6d23h   v1.25.9+rke2r1
k8s-2-worker-01.herren5.local   Ready      <none>                      6d23h   v1.25.9+rke2r1
k8s-2-worker-02.herren5.local   Ready      <none>                      6d23h   v1.25.9+rke2r1
k8s-2-worker-03.herren5.local   Ready      <none>                      6d23h   v1.25.9+rke2r1

Shoudn’t the taints be more like ?

<http://node-role.kubernetes.io/etcd=true:NoExecute|node-role.kubernetes.io/etcd=true:NoExecute>
 <http://node-role.kubernetes.io/controlplane=true:NoSchedule|node-role.kubernetes.io/controlplane=true:NoSchedule>

Hello, I have setup a new rke2 (1.25) HA cluster with (2 servers and 1 worker), LB VM (nginx who load balance servers) and VIP that forward trafic to LB. VIP -> LB nginx -> server1 & server2 I have installed my application with helm. This latter have an ingress-nginx. The problem is I cannot access my application using this architecture. Im using canal and all pods are running without any problem. Nginx LB not logging anything in "access.log", neither rke2-ingress-nginx-controller deployed in servers. This is my nginx LB conf :

user nginx;
worker_processes 4;
worker_rlimit_nofile 40000;
error_log /var/log/nginx/error.log;
pid /run/nginx.pid;

# Load dynamic modules. See /usr/share/doc/nginx/README.dynamic.
include /usr/share/nginx/modules/*.conf;

events {
    worker_connections 8192;
}

stream {
upstream backend {
        least_conn;
        server <IP_NODE_1>:9345 max_fails=3 fail_timeout=5s;
        server <IP_NODE_2>:9345 max_fails=3 fail_timeout=5s;
       }

   server {

      listen 9345;
      proxy_pass backend;
   }
    upstream rancher_api {
        least_conn;
        server <IP_NODE_1>:6443 max_fails=3 fail_timeout=5s;
        server <IP_NODE_2>:6443 max_fails=3 fail_timeout=5s;
    }
        server {
        listen     6443;
        proxy_pass rancher_api;
        }
    upstream rancher_http {
        least_conn;
        server <IP_SERVER1>:80 max_fails=3 fail_timeout=5s;
        server <IP_SERVER2>:80 max_fails=3 fail_timeout=5s;
    }
        server {
        listen     80;
        proxy_pass rancher_http;
        }
    upstream rancher_https {
        least_conn;
        server <IP_SERVER1>:443 max_fails=3 fail_timeout=5s;
        server <IP_SERVER2>:443 max_fails=3 fail_timeout=5s;
    }
        server {
        listen     443;
        proxy_pass rancher_https;
        }
}

Please can someone help ?

So I torn down the cluster yesterday because of change in underlying cni, I added config (before or after, I don't remember where I say that I want to kubeProxyReplacement is strict) though I don't know if it worked, I do see kube-proxy component running, and helm install is stuck with Preemption error

Hello folks, is there any way to configure the registry used by RKE2 Pods/charts images? I'd like to force usage of a private proxy registry.

Hello, I'm trying to start a Downstream rke2 cluster, each nodes with specific roles. The etcd and Control Plane nodes stated successfuly, but the worker nodes does not start. After executing the install command provided by Rancher, the system agent starts, but the proccess os rke2-server setup does not start. This is the rancher-system-agent.service journalctl logs:

Which of the Network Rules defined in the Requirements (here: https://docs.rke2.io/install/requirements#inbound-network-rules) do I need to configure for a default RKE installation? I think by default the CNI installed is Canal , so I just added the first 7 and those including Canal in the description. My installation seemed to have issues with networking between pods on different nodes, so I'm revisiting my configuration.

I setup a downstream cluster with nginx ingress using rancher and deployed a microservice. I created an ingress (with SSL termination) which points to the services for the apps. If I then setup an external nginx load balancer what IP address should it point to? I am basically trying to create support for multiple microservices, but having difficulty in setting up the external loadbalancer to pass traffic to my ingress.

Hello everyone, I am trying to connect a Windows agent and it simply doesn't appear in the control plane, how can I debug the issue? I've been following the https://docs.rke2.io/install/quickstart#windows-agent-worker-node-installation and it worked on another environment. All ports seem to be open, token and master hostname a correctly written to

c:/etc/rancher/rke2/config.yaml

Hi everyone, I have a problem. I have created a cluster from rancher on rke2 version v1.25.9+rke2r1. On the machine using the command

/var/lib/rancher/rke2/data/v1.25.9-rke2r1-177f016694ea/bin/ctr --address=/run/k3s/containerd/containerd. sock run --rm -t --runc-binary=/usr/local/nvidia/toolkit/nvidia-container-runtime --env NVIDIA_VISIBLE_DEVICES=all <http://docker.io/nvidia/samples:vectoradd-cuda10.2|docker.io/nvidia/samples:vectoradd-cuda10.2> nvidia-smi

everything works fine. On kubernetes I installed gpu-operator from nvidia which validated correctly on the machine. But when I put the same image from kubernetes I get the log

Failed to allocate device vector A (error code CUDA driver version is insufficient for CUDA runtime version)!
2023-06-09T12:29:00.444604100Z [Vector addition of 50000 elements]

. I have set 2 variables in the operator, specifically

CONTAINERD_CONFIG: /var/lib/rancher/rke2/agent/etc/containerd/config.toml

and

CONTAINERD_SOCKET: /run/k3s/containerd/containerd.sock

. Is anyone able to tell what is wrong that it is not working?

I noticed there was a significant push last week to get arm64 image builds for RKE2 to have arm64 support. How difficult is it to contribute to the CI pipelines building these images? I have free time this weekend to contribute.

Hi, I have deployed postgresql on rke2 cluster pod getting fail due to livenessprobe and rediness probe can any suggest how to fix the error "Warning Unhealthy 67s (x6 over 117s) kubelet Liveness probe failed: 127.0.0.1:5432 - no response Normal Killing 67s kubelet Container postgresql failed liveness probe, will be restarted Warning Unhealthy 43s (x11 over 2m23s) kubelet Readiness probe failed: 127.0.0.1:5432 - no response"

Hi All, I am trying to install tinkerbell with RKE2. The load balancer of boot and stack status is showing pending . I tired to disable the default load balancer by deleting daemonset.apps/rke2-ingress-nginx-controller, service/rke2-ingress-nginx-controller-admission but still seeing same issue. Please help.

Hi all, there is a way to get the RKE2 cluster name as

env

inside a pod?

define cluster name

Cluster is normally not aware of other clusters