How can I pass arguments to the kublet configuration with RKE2? I’ve added

kubelet:
  extra_args:
    eviction-hard: >-
      memory.available<100Mi,nodefs.available<10%,imagefs.available<15%,nodefs.inodesFree<5%

to the /etc/rancher/rke2/config.yaml, but it doesn’t seem to effect the worker nodes

Guys, how would you suggest collecting container logs from Windows hosts?

My current set up is rke2, containerd and loki-stack

rke2 Plan CRD for upgrading cluster won’t be deleted automatically when the cluster upgrade completed. What should I do to the

Plan

CRD. should I delete it mannually or just leave it there without touch. Thanks.

pretty early usage with windows nodes so looking for a quick tip if someone has one...we have a couple clusters built but 1 of them seems to not be allowing any traffic from calico interfaces to anything external (not other node IPs, not public internet, not other pod IPs on other nodes). Anyone have any tips on what to look at before I dig out packet capture stuff?

heh sorry, slack won't let me remove from channel 😞

If i deploy a cluster today and i would like to change the cloud-provider-name option to vsphere at a later time, is that possible? Or do i have to redeploy the whole cluster ?

What is the guidance on how to remediate this? etcd cluster "kube-etcd": database size in use on instance 172.16.1.52:2381 is 37.82% of the actual allocated disk space, please run defragmentation (e.g. etcdctl defrag) to retrieve the unused fragmented disk space. How do I auth against etcd, etc? Is there a built in way to do this via rancher + rke2?

hello, I'm using an rke2 powered local cluster (intended for rancher) using OpenSuse Leap 15.4 on our test system to gather experiences with kubernetes I'm running into problems and can't really figure it out myself. Currently the problem is that the

rke2-ingress-controller

on my rke2-agent nodes are not coming up. The one on the rke2-server node is working correctly. when I view the logs of the 2 agent nodes, there is no error, but each container is trying to use

10.43.0.1

as the pod IP, but only the one on my server node succeeds to do so. Can anybody help me with this matter?

New cluster with 3 nodes on RHEL 8.7 - RKE2 v1.25.10 After reboot of one node at a time, when I run kubectl get nodes, I get these:

# kubectl get no
E0622 09:22:58.972153   15285 memcache.go:287] couldn't get resource list for <http://metrics.k8s.io/v1beta1|metrics.k8s.io/v1beta1>: the server is currently unable to handle the request
E0622 09:22:58.987766   15285 memcache.go:121] couldn't get resource list for <http://metrics.k8s.io/v1beta1|metrics.k8s.io/v1beta1>: the server is currently unable to handle the request
E0622 09:22:58.990105   15285 memcache.go:121] couldn't get resource list for <http://metrics.k8s.io/v1beta1|metrics.k8s.io/v1beta1>: the server is currently unable to handle the request
E0622 09:22:58.991244   15285 memcache.go:121] couldn't get resource list for <http://metrics.k8s.io/v1beta1|metrics.k8s.io/v1beta1>: the server is currently unable to handle the request
NAME   STATUS   ROLES                       AGE   VERSION

But everything appears to be okay. Should I ignore these lines in output?

Hello, I am looking to upgrade rke2 clusters from 1.24.x to 1.25x. https://github.com/rancher/rke2/releases/tag/v1.25.0%2Brke2r1 release notes mention this:

Kubernetes v1.25 removes the beta PodSecurityPolicy admission plugin. Please follow the upstream documentation to migrate from PSP if using the built-in PodSecurity Admission Plugin, prior to upgrading to v1.25.0+rke2r1.

While upstream documentation tells me to disable

PodSecurityPolicy

admission plugin. I’ve removed it from

kube-apiserver-arg

enable-admission-plugins=NodeRestriction

but it still somehow added back… I am guessing it is enforced by RKE2 hardening so what I actually got is:

I0622 09:17:49.341327       1 flags.go:64] FLAG: --enable-admission-plugins="[NodeRestriction,PodSecurityPolicy,NodeRestriction]"

I also tried to add

--disable-admission-plugins="[PodSecurityPolicy]"

but this cause api server to fail

"command failed" err="[PodSecurityPolicy] in enable-admission-plugins and disable-admission-plugins overlapped"

So my question is: How do I disable

PodSecurityPolicy

admission plugin to be able to remove all PSP before upgrading to 1.25.x 🙏

Hello, can someone give me a tip on how to change the times for "not ready" and "unavailable"?

Is

rke2-agent

required to run just rancher on RKE2 cluster? What I am trying to do is run a 3 node RKE2 cluster for rancher, these 3 nodes will have

rke2-server

. but am unsure if I will have to deploy

rke2-agent

nodes will be required for rancher application?

Hi, I'm going to upgrade a 3-node rke2 cluster from 1.24.7+rke2r1 to 1.25.9+rke2r1. The cluster is the local cluster for Rancher manager which is running 1.27.4. I have two questions: 1. I want to use Rancher Manager to upgrade the local rke2 cluster. Not sure if this is a supported method. The way I'm going to do this is by going to Cluster Management -> Select the local cluster -> Edit Config. Under the Kubernetes Version drop-down, select v1.25.9+rke2r1. 2. Before I proceed the upgrade, it warns me about the PSP removal in v1.25 and provides me a link for instructions to manually prepare the cluster for the upgrade. The link is https://docs.rke2.io/known_issues#hardened-125. It says I need to update the profile value to cis-1.23 on all nodes. I'm not sure what this means. Is this referring to setting the profile parameter in /etc/rancher/rke2/config.yml? Currently, I don't see the profile parameter in the config file. Any suggestions are greatly appreciated.

How can I reapply the helm chart? I’ve updated the manifest for cilium to enable hubble. But looking at “cilium status” it remains disabled. I’ve restarted the rke2-server but this seems not to have any effect.

in k8s v1.25, podsecuritypolicy is deprecated, and I found that rke2 with v1.24 before will have a default psp for every pod upon it, and if you noticed and did nothing to the cluster, by default, pod will have a psp named

global-unrestricted-psp

. I wonder to know how should I prepare before upgrading my cluster to v1.25. will the remaining of such psp configuration lead to a crack about my app and cluster. How could I delete them? thanks

Hi, I ran into the following problem; I’ve created a RKE2 cluster using Ubuntu 22.04. The machines are assigned an IPv4 using DHCP. On Ubuntu when a node is shutdown / rebooted it releases it’s IP address resulting in a different IP once the machine is up again. RKE2 then start reporting errors as the generated certificates for etcd are not updated.

Failed to test data store connection: this server is a not a member of the etcd cluster. Found [rancher-dev-control-plane-02-abc=https://<old-ip-address:2380

What should be te best way to work around this?

Hi everyone, I am currently facing an issue with etcd metrics, the service listens on localhost only, as opposed to listening on IPs available on the internal network:

Is there any way to listen on 0.0.0.0 for the metrics port?

Hey all. Looking to get a second set of eyes on some RKE2 deployment failures. I am testing out the experimental arm64 support added in 1.27.3 and am seeing some failures trying to get rke2-server to start on my first server node.

Jun 28 20:23:33 rke2-bld-01 rke2[2805]: {"level":"info","ts":"2023-06-28T20:23:33.508Z","logger":"etcd-client","caller":"v3@v3.5.7-k3s1/client.go:210","msg":"Auto sync endpoints failed.","error":"context deadline exceeded"}
Jun 28 20:23:33 rke2-bld-01 rke2[2805]: time="2023-06-28T20:23:33Z" level=error msg="Kubelet exited: exit status 1"
Jun 28 20:23:36 rke2-bld-01 rke2[2805]: time="2023-06-28T20:23:36Z" level=info msg="Waiting to retrieve kube-proxy configuration; server is not ready: <https://127.0.0.1:9345/v1-rke2/readyz>: 500 Internal Server Error"
Jun 28 20:23:39 rke2-bld-01 rke2[2805]: time="2023-06-28T20:23:39Z" level=error msg="Kubelet exited: exit status 1"
Jun 28 20:23:39 rke2-bld-01 rke2[2805]: time="2023-06-28T20:23:39Z" level=info msg="Container for etcd not found (no matching container found), retrying"

I can see the images there:

root@rke2-bld-01:~# /var/lib/rancher/rke2/bin/ctr --address /run/k3s/containerd/containerd.sock --namespace <http://k8s.io|k8s.io> image ls
REF                                                                                                           TYPE                                                      DIGEST                                                                  SIZE      PLATFORMS                           LABELS
<http://docker.io/rancher/hardened-etcd:v3.5.7-k3s1-build20230609|docker.io/rancher/hardened-etcd:v3.5.7-k3s1-build20230609>                                                     application/vnd.docker.distribution.manifest.list.v2+json sha256:faa679c5463e1f42e2a8fff66cf0a1b47e18be09bdf073b8b965642ea4e4a9d5 59.6 MiB  linux/amd64,linux/arm64,linux/s390x io.cri-containerd.image=managed
<http://docker.io/rancher/hardened-etcd@sha256:faa679c5463e1f42e2a8fff66cf0a1b47e18be09bdf073b8b965642ea4e4a9d5|docker.io/rancher/hardened-etcd@sha256:faa679c5463e1f42e2a8fff66cf0a1b47e18be09bdf073b8b965642ea4e4a9d5>       application/vnd.docker.distribution.manifest.list.v2+json sha256:faa679c5463e1f42e2a8fff66cf0a1b47e18be09bdf073b8b965642ea4e4a9d5 59.6 MiB  linux/amd64,linux/arm64,linux/s390x io.cri-containerd.image=managed
<http://docker.io/rancher/hardened-kubernetes:v1.27.3-rke2r1-build20230614|docker.io/rancher/hardened-kubernetes:v1.27.3-rke2r1-build20230614>                                            application/vnd.docker.distribution.manifest.list.v2+json sha256:8ad41bdeae0d1f3f569512380610f822b419b2720d96c03110690563be62fbac 191.4 MiB linux/amd64,linux/arm64,linux/s390x io.cri-containerd.image=managed
<http://docker.io/rancher/hardened-kubernetes@sha256:8ad41bdeae0d1f3f569512380610f822b419b2720d96c03110690563be62fbac|docker.io/rancher/hardened-kubernetes@sha256:8ad41bdeae0d1f3f569512380610f822b419b2720d96c03110690563be62fbac> application/vnd.docker.distribution.manifest.list.v2+json sha256:8ad41bdeae0d1f3f569512380610f822b419b2720d96c03110690563be62fbac 191.4 MiB linux/amd64,linux/arm64,linux/s390x io.cri-containerd.image=managed
sha256:08610ff5575b078662d9a87cf08b8e78c7445cb6440f48757ae127d541dc715c                                       application/vnd.docker.distribution.manifest.list.v2+json sha256:8ad41bdeae0d1f3f569512380610f822b419b2720d96c03110690563be62fbac 191.4 MiB linux/amd64,linux/arm64,linux/s390x io.cri-containerd.image=managed
sha256:d52d36ccbf1ad9932b0e995da0ed5049875db322705a20f572e5f4cd6ed87bb1                                       application/vnd.docker.distribution.manifest.list.v2+json sha256:faa679c5463e1f42e2a8fff66cf0a1b47e18be09bdf073b8b965642ea4e4a9d5 59.6 MiB  linux/amd64,linux/arm64,linux/s390x io.cri-containerd.image=managed

But no containers are running:

root@rke2-bld-01:~# /var/lib/rancher/rke2/bin/ctr --address /run/k3s/containerd/containerd.sock --namespace <http://k8s.io|k8s.io> container ls
CONTAINER    IMAGE    RUNTIME
root@rke2-bld-01:~#

Please let me know if there is any further information I can provide, or if if this would be better suited for a github issue.

Is there a way to create new RKE2 cluster with canal instead of default calico ? And that's from script of RPM install and not from Rancher UI...

canal is default

calico is part of canal though

thanks!

I think Canal is default if you install via script, but I thought I got Calico when I deployed RKE2 via Rancher.

using rancher to deploy rke2 on harvester and have chosen the cilium cni. Does that install include the cilium cli and is it configured with hubble and other bits?

Hi, I want to deploy and use the latest stable version of RKE2 but I see a lot of versions available on the release page. https://github.com/rancher/rke2/releases Can anyone please suggest me the latest stable version of RKE2?

Hi,

Hi, facing an issue like rke2-ingress pods are stuck in container creating state while testing with rke2 stable version like 1.26.6+rke2r1 , And tested on OS- RHEL-8.6, Please find screenshots and suggest the best work around for this