What is the correct way to enable audit logging on rke2 ? I can't seem to find anything useful in the documentation, besides some github issue stating i should configure something like this:

audit-policy-file: /etc/rancher/rke2/audit.yaml
kube-apiserver-arg:
  - audit-log-path=/var/lib/rancher/rke2/server/logs/audit.log
  - audit-log-format=json
  - audit-log-maxage=5
  - audit-log-maxbackup=5
  - audit-log-maxsize=100
kube-apiserver-extra-mount: /var/lib/rancher/rke2/server/logs:/var/lib/rancher/rke2/server/logs

is there a way to add windows10 machines to a rke2 cluster as worker-nodes (for windows exclusive workloads) i'm currently running a custom cluster so i'm using the registration url to join windows worker nodes

Hi, I'm trying to add new worker node to a Rancher deployed RKE2-cluster. It has worked fine until now when it fails to add the node. The node gets added to the cluster but the "apply-system-agent-upgrader"-job fails with error "[FATAL] You must select at least one role.". Even tho the --worker is passed in the registeration script. Any ideas what could cause this? In between initial creation and adding the node I've upgraded Rancher from 2.6.8 -> 2.7.1. The k8s version is 1.24.4+rke2r1.

Does anyone know the "Cluster agent is not connected" issue for the RKE2 cluster when after rebooting worker/master nodes on rancher v2.6.3?

Hi, I have created a TLS ingress for an angular app on RKE2. While trying to access JSON file from the UI, the calls are failing with "*ERR_HTTP2_PROTOCOL_ERROR*" error. But the same works fine with a non TLS ingress. The difference I see is that the TLS ingress uses HTTP/2 protocol while non TLS ingress uses HTTP/1.1 protocol. Non TLS Ingress:

[kubenode@master ~]$ curl -kv <http://ne-mgr-ui.xx.xx.xx.xx.nip.io>
* Rebuilt URL to: <http://ne-mgr-ui.xx.xx.xx.xx.nip.io/>
*   Trying xx.xx.xx.xx...
* TCP_NODELAY set
* Connected to <http://ne-mgr-ui.xx.xx.xx.xx.nip.io|ne-mgr-ui.xx.xx.xx.xx.nip.io> (xx.xx.xx.xx) port 80 (#0)
> GET / HTTP/1.1
> Host: <http://ne-mgr-ui.xx.xx.xx.xx.nip.io|ne-mgr-ui.xx.xx.xx.xx.nip.io>
> User-Agent: curl/7.61.1
> Accept: */*
>
< HTTP/1.1 200 OK
< Date: Tue, 04 Apr 2023 10:01:59 GMT

TLS Ingress:

[kubenode@master ~]$ curl -kv <https://ne-mgr-ui.app.com>
* Connection state changed (MAX_CONCURRENT_STREAMS == 128)!
* http2 error: Invalid HTTP header field was received: frame type: 1, stream: 1, name: [            proxy-revalidate, max-age=0], value: []
* HTTP/2 stream 0 was not closed cleanly: PROTOCOL_ERROR (err 1)
* stopped the pause stream!
* Connection #0 to host <http://ne-mgr-ui.app.com|ne-mgr-ui.app.com> left intact
curl: (92) HTTP/2 stream 0 was not closed cleanly: PROTOCOL_ERROR (err 1)

Am not sure how to make the TLS ingress use HTTP/1.1 protocol instead of HTTP2 protocol error. I have tried adding the annotation nginx.ingress.kubernetes.io/use-http2: "false" on the TLS ingress but the ingress still uses HTTP/2 protocol only. I want to disable HTTP/2 protocol on RKE2. Any suggestions would be appreciated.

Hi, Looking for some support, if this is the wrong channel do let me know I have recently updated my Rancher deployment that has been running fine for 310 days I tried using K8S1.23.6/rke2r2 and rancher 1.27.1 I got an error that said the webhooks were not ready on 443 and the cluster was unable to be managed I have now downgraded the rancher version to 1.26.head and the cluster is now manageable, but I cannot add nodes to the cluster i get this error:

curl --insecure -fL https://<IP>/system-agent-install.sh | sudo  sh -s - --server https://<IP>--label '<http://cattle.io/os=linux|cattle.io/os=linux>' --token <token>--ca-checksum <checksum>--etcd --controlplane --node-name <nodeName>
  % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
                                 Dload  Upload   Total   Spent    Left  Speed
100 29790    0 29790    0     0   334k      0 --:--:-- --:--:-- --:--:--  338k
[INFO]  Label: <http://cattle.io/os=linux|cattle.io/os=linux>
[INFO]  Role requested: etcd
[INFO]  Role requested: controlplane
[INFO]  Using default agent configuration directory /etc/rancher/agent
[INFO]  Using default agent var directory /var/lib/rancher/agent
[INFO]  Determined CA is necessary to connect to Rancher
[INFO]  Successfully downloaded CA certificate
[INFO]  Value from https://<IP>/cacerts is an x509 certificate
[INFO]  Successfully tested Rancher connection
[INFO]  Downloading rancher-system-agent binary from https://<IP>/assets/rancher-system-agent-amd64
[INFO]  Successfully downloaded the rancher-system-agent binary.
[INFO]  Downloading rancher-system-agent-uninstall.sh script from https://<IP>/assets/system-agent-uninstall.sh
[INFO]  Successfully downloaded the rancher-system-agent-uninstall.sh script.
[INFO]  Generating Cattle ID
[INFO]  Cattle ID was already detected as <ID>. Not generating a new one.
[ERROR]  500 received while downloading Rancher connection information. Sleeping for 5 seconds and trying again

Hello. Are the rpm repositories down or is it just me? I’ve been getting similar responses for the past 1.5 hours.

404 Not Found
Code: NoSuchKey
Message: The specified key does not exist.
Key: rke2/latest/1.21/repodata/repomd.xml
RequestId: 6CQ27E9HR2YG8B8R
HostId: oa7Wg0AvPXJa6ktBRn6V4WNHriS4KhzbKjI2Y4H5toFuISnzAqQRtSAEQBjBMiZBiG7h6+dKszE=

Hi Rancher Team, how to add tls certificate to the air gap installed rancher.

We have deployed a new #rke2 cluster (version v1.25.3+rke2r1). The pods are failing 'Crashloopbackoff' because of Container runtime error. We are only using RKE2 conatinerd (not systemd) and it is unable to locate cri socket at unix:///var/run/containerd/containerd.sock since it is present at /run/k3s/containerd/containerd.sock. Error message: {"level":"fatal","version":"v1.4.6","scannerId":"infra5cdd91cc f0c5 4204 b371 7a002b65299f<nodename>","containerRuntimeName":"containerd","error":"could not connect to the cri socket unix///var/run/containerd/containerd.sock context deadline exceeded","time":"2023-04-04T123532Z","message":"failed to instantiate container runtime client"} As a workaround, we have created softlink (it works): /var/run/containerd/containerd.sock -> /run/k3s/containerd/containerd.sock Please let us know how to make /run/k3s/containerd/containerd.sock the default cri socket

Hi, I am deploying RKE2 using Air Gapped installation procedure on RHEL8 cluster. Now, I want to add a few configuration options to the nginx-ingress configmap like

data:
  allow-snippet-annotations: "false"
  use-http2: "false"

I want to make the changes before the RKE2 deployment in the configuration files and then start the deployment process. But I am not sure where to make these changes in the Air Gapped installation proces. Any suggestions would be appreciated.

Hi, I'm trying to provision a rke2 cluster with rancher 2.7.1(Air gapped).I have installed a new instance for testing because I have a known issue on my current rancher server in 2.6.8. The VM are well provisioned on my vcenter and boot correctly but nothing happens. I have installed the prerequisites (rke2 common and selinux), maybe I missed something. However if I run the rke2 server command, everything seems to be ok (container, pods...) but my rancher still display error. Do I missed something? Is anyone has already installed a rke2 cluster throw rancher in a Air gapped environment ?

Hello, how can I increase node's pod limit for RKE2? Tried using this config

apiVersion: <http://kubelet.config.k8s.io/v1beta1|kubelet.config.k8s.io/v1beta1>
kind: KubeletConfiguration
maxPods: 250

with in /etc/rancher/rke2/config.yaml but it causes node to be in NotReady state 😞

kubelet-arg:
- config: /etc/rancher/rke2/kubelet-config.yaml

Hello everyone, Could someone help me with redirecting traffic to an external domain? 1. I've purchased a VPS server and installed Rancher version 2.6.2 - a multi-cloud Kubernetes management platform on Ubuntu Server 20.04 LTS. 2. On the server, Docker is running the Rancher image, and it redirects traffic on ports 80 and 443. 3. It generated a URL (194-36-88-21.cloud-xip.com) that is accessible on the Internet, and I can log in to the Rancher UI. ◦ On the 'local' node, created by default, I've installed:NGINX Ingress Controller manually. ◦ Cert-Manager manually. ◦ Created a workload and ingress with the necessary configuration. 4. In the Docker container, I can access my service, but it doesn't work externally. When I use my custom domain, it automatically redirects to https:\194-36-88-21.cloud-xip.com. How can I correctly redirect traffic?

having this issue on on RKE2 cluster. No idea why, but all last deployment were successful

Events:
  Type     Reason          Age                  From               Message
  ----     ------          ----                 ----               -------
  Normal   AddedInterface  35m                  multus             Add eth0 [10.42.0.197/32] from k8s-pod-network
  Normal   Scheduled       35m                  default-scheduler  Successfully assigned netbox/busybox to longvan-node1
  Warning  Failed          35m                  kubelet            Failed to pull image "busybox": rpc error: code = Unknown desc = failed to pull and unpack image "<http://docker.io/library/busybox:latest|docker.io/library/busybox:latest>": failed to resolve reference "<http://docker.io/library/busybox:latest|docker.io/library/busybox:latest>": failed to do request: Head "<https://registry-1.docker.io/v2/library/busybox/manifests/latest>": dial tcp: lookup <http://registry-1.docker.io|registry-1.docker.io> on 127.0.0.53:53: read udp 127.0.0.1:47828->127.0.0.53:53: i/o timeout
  Warning  Failed          35m                  kubelet            Failed to pull image "busybox": rpc error: code = Unknown desc = failed to pull and unpack image "<http://docker.io/library/busybox:latest|docker.io/library/busybox:latest>": failed to resolve reference "<http://docker.io/library/busybox:latest|docker.io/library/busybox:latest>": failed to do request: Head "<https://registry-1.docker.io/v2/library/busybox/manifests/latest>": dial tcp: lookup <http://registry-1.docker.io|registry-1.docker.io> on 127.0.0.53:53: read udp 127.0.0.1:40619->127.0.0.53:53: i/o timeout
  Warning  Failed          34m                  kubelet            Failed to pull image "busybox": rpc error: code = Unknown desc = failed to pull and unpack image "<http://docker.io/library/busybox:latest|docker.io/library/busybox:latest>": failed to resolve reference "<http://docker.io/library/busybox:latest|docker.io/library/busybox:latest>": failed to do request: Head "<https://registry-1.docker.io/v2/library/busybox/manifests/latest>": dial tcp: lookup <http://registry-1.docker.io|registry-1.docker.io> on 127.0.0.53:53: read udp 127.0.0.1:60114->127.0.0.53:53: i/o timeout
  Normal   Pulling         33m (x4 over 35m)    kubelet            Pulling image "busybox"
  Warning  Failed          33m (x4 over 35m)    kubelet            Error: ErrImagePull
  Warning  Failed          33m                  kubelet            Failed to pull image "busybox": rpc error: code = Unknown desc = failed to pull and unpack image "<http://docker.io/library/busybox:latest|docker.io/library/busybox:latest>": failed to resolve reference "<http://docker.io/library/busybox:latest|docker.io/library/busybox:latest>": failed to do request: Head "<https://registry-1.docker.io/v2/library/busybox/manifests/latest>": dial tcp: lookup <http://registry-1.docker.io|registry-1.docker.io> on 127.0.0.53:53: read udp 127.0.0.1:34097->127.0.0.53:53: i/o timeout
  Warning  Failed          33m (x6 over 35m)    kubelet            Error: ImagePullBackOff
  Normal   BackOff         34s (x145 over 35m)  kubelet            Back-off pulling image "busybox"

RKE version: v1.24.7+rke2r1

My cluster is having 3 nodes. If I cordon 2 nodes. This is the another issue I see on the last node.

Events:
  Type     Reason          Age   From               Message
  ----     ------          ----  ----               -------
  Normal   Scheduled       22s   default-scheduler  Successfully assigned netbox/busybox to longvan-node3
  Normal   AddedInterface  21s   multus             Add eth0 [10.42.2.249/32] from k8s-pod-network
  Normal   Pulling         21s   kubelet            Pulling image "busybox"
  Normal   Pulled          15s   kubelet            Successfully pulled image "busybox" in 6.183796754s
  Normal   Created         15s   kubelet            Created container busybox
  Warning  Failed          14s   kubelet            Error: failed to create containerd task: failed to create shim task: OCI runtime create failed: runc create failed: unable to start container process: exec: "–": executable file not found in $PATH: unknown

i'm struggling with increasing the max-pod setting on the agent kubelet.

kubelet-args:

max-pods: 220

shouldn't that be enough in /etc/ranger/rke2/config.yaml and restarting the node / kubelet?

mh the documentation for the cli now says its kubelet-arg; but i tried both non worked 😞

mh i found a slack messge from a few days will try it out; my format seems to be wrong:

kubelet-arg:

- "max-pods=220"

But this seems to become deprecated. will there be support for

KubeletConfiguration

?

Whats the value of encrypting secrets with the AES-CBC provider when the encryption key secret is stored in plaintext? As far as I can tell, that is the default behaviour for RKE2?

Would it be possible to use a config map on the datastore to set a

kube-apiserver-arg

?

The documentation states that

aes-cbc

secret encryption provider is enabled by default. What exactly are the steps to use a different provider say

kms

instead on an existing node? If there is documentation for this can someone point me to it? For context I understand that the EncryptionConfiguration needs to change. But what else is involved there? Should I remove the existing

aes-cbc

section from the existing config? If yes, would key rotation still work (since i assume i need to re-encrypt using the new provider)?

not seen much about this, anyone have any tips?

kubectl -n cattle-system logs -f system-upgrade-controller-976bd7bb4-jv2w7
W0415 01:53:38.040731       1 client_config.go:543] Neither --kubeconfig nor --master was specified.  Using the inClusterConfig.  This might not work.
time="2023-04-15T01:53:38Z" level=fatal msg="Error starting: namespaces \"kube-system\" is forbidden: User \"system:serviceaccount:cattle-system:system-upgrade\" cannot get resource \"namespaces\" in API group \"\" in the namespace \"kube-system\""

rke2 cluster imported manually to rancher

When deploying an RKE2 cluster to vsphere with Rancher, is there a way to specify a datastore cluster instead of a single datastore? We had an issues where the datastore went down and rancher was unable to deploy a new VM until we edited the cluster and changed the datastore for that node pool.

I want to install a RKE2 downstream cluster but do not want nginx ingress controller to bind to host ports 443 and 80 since I will install traefik ingress controller. Is it possible to configure cluster.yaml so that nginx will not occupy those ports? On RKE1 there was a setting in cluster.yaml like "ingress.network_mode: none" on RKE2 I cannot find anything similar

I am trying to install RKE2 following these Quick Start instructions. https://docs.rke2.io/install/quickstart This is the error I am getting Apr 18 145515 rke2 rke2[11426]: time="2023-04-18T145515Z" level=info msg="Tunnel server egress proxy waiting for runtime core to become available" Apr 18 145515 rke2 rke2[11426]: time="2023-04-18T145515Z" level=info msg="Container for etcd is running" Apr 18 145515 rke2 rke2[11426]: time="2023-04-18T145515Z" level=info msg="Container for kube-apiserver not found (no matching container found), retrying" Apr 18 145517 rke2 rke2[11426]: time="2023-04-18T145517Z" level=info msg="Waiting to retrieve kube-proxy configuration; server is not ready: https://127.0.0.1:9345/v1-rke2/readyz: 500 Internal Server Error" Help please

Hi, I’m seeing multiple kube api error on the api server, I have 3 master nodes and I don’t see the same error log on one of the server. Other two kube api servers continuously printing below log in the kube api server. Any of you know what it is and how it fix it?

"Unable to authenticate the request" err="[invalid bearer token, Token has been invalidated]"

Hello everyone. anyone know if there's a way to add a windows 10 worker to an rke2 cluster?

i have some windows based tasks i'd like to run on some windows 10 boxes i have