I am trying to install and run rke2 in a SLES 15 SP3 VM and facing issues. I am using the guide here https://docs.rke2.io/install/quickstart/. I have disabled the firewall and enabled IPv4/IPv6 port forwarding in Wicked. Seems the server is running, but I do see errors in the logs. Also when I installed the agent in the same VM I got an error:

msg="failed to get CA certs: Get \"<https://127.0.0.1:6444/cacerts>\": read tcp 127.0.0.1:48636->127.0.0.1:6444: read: connection reset by peer

Curl gives following error:

curl -v <https://127.0.0.1:6444/cacerts>
*  Trying 127.0.0.1:6444...
* TCP_NODELAY set
* Connected to 127.0.0.1 (127.0.0.1) port 6444 (#0)
* ALPN, offering h2
* ALPN, offering http/1.1
* TLSv1.3 (OUT), TLS handshake, Client hello (1):
* OpenSSL SSL_connect: SSL_ERROR_SYSCALL in connection to 127.0.0.1:6444 
* Closing connection 0
curl: (35) OpenSSL SSL_connect: SSL_ERROR_SYSCALL in connection to 127.0.0.1:6444

Anyone know what this might be caused by? curl -k https://127.0.0.1:6443 gives a response.

rke2-server.service - Rancher Kubernetes Engine v2 (server)
   Loaded: loaded (/etc/systemd/system/rke2-server.service; enabled; vendor preset: disabled)
   Active: active (running) since Mon 2022-05-16 20:46:15 PDT; 4min 55s ago
    Docs: <https://github.com/rancher/rke2#readme>
  Process: 48957 ExecStartPre=/bin/sh -xc ! /usr/bin/systemctl is-enabled --quiet nm-cloud-setup.service (code=exited>
  Process: 48960 ExecStartPre=/sbin/modprobe br_netfilter (code=exited, status=0/SUCCESS)
  Process: 48961 ExecStartPre=/sbin/modprobe overlay (code=exited, status=0/SUCCESS)
  Main PID: 48962 (rke2)
   Tasks: 205
   CGroup: /system.slice/rke2-server.service
       ├─ 2652 /var/lib/rancher/rke2/data/v1.22.9-rke2r2-88ecb1441384/bin/containerd-shim-runc-v2 -namespace k8s.>
       ├─ 2673 /pause
       ├─ 2707 /var/lib/rancher/rke2/data/v1.22.9-rke2r2-88ecb1441384/bin/containerd-shim-runc-v2 -namespace k8s.>
       ├─ 2711 /var/lib/rancher/rke2/data/v1.22.9-rke2r2-88ecb1441384/bin/containerd-shim-runc-v2 -namespace k8s.>
       ├─ 2737 /var/lib/rancher/rke2/data/v1.22.9-rke2r2-88ecb1441384/bin/containerd-shim-runc-v2 -namespace k8s.>
       ├─ 2758 /pause
       ├─ 2778 /pause
       ├─ 2813 /var/lib/rancher/rke2/data/v1.22.9-rke2r2-88ecb1441384/bin/containerd-shim-runc-v2 -namespace k8s.>
       ├─ 2836 /var/lib/rancher/rke2/data/v1.22.9-rke2r2-88ecb1441384/bin/containerd-shim-runc-v2 -namespace k8s.>
       ├─ 2841 /pause
       ├─ 2866 /pause
       ├─ 2882 kube-proxy --cluster-cidr=10.42.0.0/16 --conntrack-max-per-core=0 --conntrack-tcp-timeout-close-wa>
       ├─ 3069 etcd --config-file=/var/lib/rancher/rke2/server/db/etcd/config
       ├─ 3761 /var/lib/rancher/rke2/data/v1.22.9-rke2r2-88ecb1441384/bin/containerd-shim-runc-v2 -namespace k8s.>
       ├─ 3782 /pause
       ├─ 3807 /var/lib/rancher/rke2/data/v1.22.9-rke2r2-88ecb1441384/bin/containerd-shim-runc-v2 -namespace k8s.>
       ├─ 3835 /pause
       ├─ 3863 /var/lib/rancher/rke2/data/v1.22.9-rke2r2-88ecb1441384/bin/containerd-shim-runc-v2 -namespace k8s.>
       ├─ 3890 /var/lib/rancher/rke2/data/v1.22.9-rke2r2-88ecb1441384/bin/containerd-shim-runc-v2 -namespace k8s.>
       ├─ 3920 /var/lib/rancher/rke2/data/v1.22.9-rke2r2-88ecb1441384/bin/containerd-shim-runc-v2 -namespace k8s.>
       ├─ 3940 /pause
       ├─ 3941 /pause
       ├─ 4073 /coredns -conf /etc/coredns/Corefile
       ├─38373 kube-apiserver --kubelet-preferred-address-types=InternalIP,ExternalIP,Hostname --allow-privileged>
       ├─42523 /cluster-proportional-autoscaler --namespace=kube-system --configmap=rke2-coredns-rke2-coredns-aut>
       ├─43471 /var/lib/rancher/rke2/data/v1.22.9-rke2r2-88ecb1441384/bin/containerd-shim-runc-v2 -namespace k8s.>
       ├─43493 /pause
       ├─43738 /opt/bin/flanneld --ip-masq --kube-subnet-mgr
       ├─45408 /usr/sbin/runsvdir -P /etc/service/enabled
       ├─45734 runsv felix
       ├─45735 runsv monitor-addresses
       ├─45736 runsv allocate-tunnel-addrs
       ├─45737 runsv node-status-reporter
       ├─45738 runsv cni
       ├─45739 calico-node -monitor-addresses
       ├─45740 calico-node -felix
       ├─45741 calico-node -allocate-tunnel-addrs
       ├─45742 calico-node -status-reporter
       ├─45743 calico-node -monitor-token
       ├─46226 /var/lib/rancher/rke2/data/v1.22.9-rke2r2-88ecb1441384/bin/containerd-shim-runc-v2 -namespace k8s.>
       ├─46261 /pause
       ├─48962 /opt/rke2/bin/rke2 server
       ├─48995 containerd -c /var/lib/rancher/rke2/agent/etc/containerd/config.toml -a /run/k3s/containerd/contai>
       ├─49051 kubelet --volume-plugin-dir=/var/lib/kubelet/volumeplugins --file-check-frequency=5s --sync-freque>
       └─49452 kube-scheduler --permit-port-sharing=true --authentication-kubeconfig=/var/lib/rancher/rke2/server>

May 16 20:50:57 suse-vm rke2[48962]: E0516 20:50:57.578889  48962 memcache.go:101] couldn't get resource list for metr>
May 16 20:51:05 suse-vm rke2[48962]: I0516 20:51:04.992233  48962 trace.go:205] Trace[411526421]: "Reflector ListAndWa>
May 16 20:51:05 suse-vm rke2[48962]: Trace[411526421]: ---"Objects listed" 61358ms (20:51:03.401)
May 16 20:51:05 suse-vm rke2[48962]: Trace[411526421]: [1m2.51858886s] [1m2.51858886s] END
May 16 20:51:08 suse-vm rke2[48962]: time="2022-05-16T20:51:08-07:00" level=info msg="Event(v1.ObjectReference{Kind:\"A>
May 16 20:51:09 suse-vm rke2[48962]: E0516 20:51:09.360628  48962 memcache.go:196] couldn't get resource list for metr>
May 16 20:51:09 suse-vm rke2[48962]: E0516 20:51:09.931690  48962 memcache.go:101] couldn't get resource list for metr>
May 16 20:51:10 suse-vm rke2[48962]: time="2022-05-16T20:51:10-07:00" level=info msg="Starting /v1, Kind=Secret control>
May 16 20:51:10 suse-vm rke2[48962]: time="2022-05-16T20:51:10-07:00" level=info msg="Starting /v1, Kind=Node controlle>
May 16 20:51:10 suse-vm rke2[48962]: I0516 20:51:10.690858  48962 leaderelection.go:248] attempting to acquire leader

Is there anyway to perform an RKE2 snapshot and upload it to 2 buckets?

I am using rke2 v1.21.5 and facing issue in using private registry.

rke2 --version
rke2 version v1.21.5+rke2r2 (9e4acdc6018ae74c36523c99af25ab861f3884da)
go version go1.16.6b7

I have created registries.yaml with aws ECR as private registry

mirrors:
  <http://2341234234.dkr.ecr.yyyyy.amazonaws.com|2341234234.dkr.ecr.yyyyy.amazonaws.com>:
    endpoint:
      - "<https://2341234234.dkr.ecr.yyyyy.amazonaws.com>"
configs:
  "<http://2341234234.dkr.ecr.yyyyy.amazonaws.com|2341234234.dkr.ecr.yyyyy.amazonaws.com>:
    auth:
      username: xxxxx
      password: xxxxx
    tls:
      insecure_skip_verify: true

While I create a pod to pull registry with following image name in pod.yaml, it fails with image pull error.

image: <http://2341234234.dkr.ecr.yyyyy.amazonaws.com/image-name:v1|2341234234.dkr.ecr.yyyyy.amazonaws.com/image-name:v1>

Looking forward some insights

Howdy all 👋 Is it expected that the

kube-controller-manager

metrics on port 10252 are not exposed in RKE2 deployments. We are currently testing

v1.22.9+rke2r2

deployed via Rancher

v2.6.5

and are seeing the following differences from RKE deployments. Thanks a lot. RKE cluster:

# netstat -tulpn | grep kube-contro
tcp6       0      0 :::10252                :::*                    LISTEN      17396/kube-controll
tcp6       0      0 :::10257                :::*                    LISTEN      17396/kube-controll

RKE2 cluster:

# netstat -tulpn | grep kube-contro
tcp        0      0 127.0.0.1:10257         0.0.0.0:*               LISTEN      39961/kube-controll

Hi all! I'm probably blind, but I couldn't find the solution to this problem yet. I'm trying to start my cluster with:

cni: "calico"
disable-kube-proxy: true

and since there's no

kube-proxy

installed the

tigera-operator

can't reach the API

2022/05/19 07:50:02 [ERROR] Get "<https://10.43.0.1:443/api?timeout=32s>": dial tcp 10.43.0.1:443: i/o timeout

I know that I could override it using the

kubernetes-services-endpoint

CM, although I can't inject that during the installation. Any clues?

with rke2 we don't need leader election anymore? https://github.com/rancher/rke2/issues/349. I tried pulling the logic out from https://github.com/rancherfederal/rke2-azure-tf/blob/main/modules/custom_data/files/rke2-init.sh#L51-L71 and am getting an error.

Hello, I'm reading through the rke2 ha documentation. it doesn't talk about the need to add kubectl again for the other server nodes. Is it safe to still install kubectl on the other server nodes?

Hi, I'm a k3s user - just for local testing, my company uses EKS in production. Do Rancher recommend using k3s or rke2 for local testing? Is work on k3s eventually going to stop in preference to rke2? TIA

Morning, I am looking to go from using ubuntu to Suse Leap 15.3. For the hardware I have, the legacy install of ubuntu is what I use. What is the equivalent to install suse leap?

Happy Friday! I’m performing an analysis against RKE2 using the DISA STIGs and I came across this one:

Rule Title: The Kubernetes Controller Manager must use TLS 1.2, at a minimum, to protect the confidentiality of sensitive data during electronic dissemination.

Discussion: The Kubernetes Controller Manager will prohibit the use of SSL and unauthorized versions of TLS protocols to properly secure communication.

The use of unsupported protocol exposes vulnerabilities to the Kubernetes by rogue traffic interceptions, man-in-the-middle attacks, and impersonation of users or services from the container platform runtime, registry, and key store. To enable the minimum version of TLS to be used by the Kubernetes Controller Manager, the setting "tls-min-version" must be set.

Check Text: Change to the /etc/kubernetes/manifests/ directory on the Kubernetes Master Node. Run the command:

grep -i tls-min-version * 

If the setting "tls-min-version" is not configured in the Kubernetes Controller Manager manifest file or it is set to "VersionTLS10" or "VersionTLS11", this is a finding.

Fix Text: Edit the Kubernetes Controller Manager manifest file in the /etc/kubernetes/manifests directory on the Kubernetes Master Node. Set the value of "--tls-min-version" to "VersionTLS12" or higher.

However I don’t see the

tls-min-version

flag set for the

kube-controller-manager

service:

kube-controller-manager --flex-volume-plugin-dir=/var/lib/kubelet/volumeplugins --terminated-pod-gc-threshold=1000 --permit-port-sharing=true --cloud-provider=aws --cloud-config= --allocate-node-cidrs=true --authentication-kubeconfig=/var/lib/rancher/rke2/server/cred/controller.kubeconfig --authorization-kubeconfig=/var/lib/rancher/rke2/server/cred/controller.kubeconfig --bind-address=127.0.0.1 --cluster-cidr=10.0.0.0/12 --cluster-signing-kube-apiserver-client-cert-file=/var/lib/rancher/rke2/server/tls/client-ca.crt --cluster-signing-kube-apiserver-client-key-file=/var/lib/rancher/rke2/server/tls/client-ca.key --cluster-signing-kubelet-client-cert-file=/var/lib/rancher/rke2/server/tls/client-ca.crt --cluster-signing-kubelet-client-key-file=/var/lib/rancher/rke2/server/tls/client-ca.key --cluster-signing-kubelet-serving-cert-file=/var/lib/rancher/rke2/server/tls/server-ca.crt --cluster-signing-kubelet-serving-key-file=/var/lib/rancher/rke2/server/tls/server-ca.key --cluster-signing-legacy-unknown-cert-file=/var/lib/rancher/rke2/server/tls/client-ca.crt --cluster-signing-legacy-unknown-key-file=/var/lib/rancher/rke2/server/tls/client-ca.key --feature-gates=JobTrackingWithFinalizers=true --kubeconfig=/var/lib/rancher/rke2/server/cred/controller.kubeconfig --profiling=false --root-ca-file=/var/lib/rancher/rke2/server/tls/server-ca.crt --secure-port=10257 --service-account-private-key-file=/var/lib/rancher/rke2/server/tls/service.key --use-service-account-credentials=true

I've searched and found a general "no" to this question, but I wonder more specifically about the reasoning / why. I have a busy cluster with high churn rate, my current issues stem from IO latency in the etcd space / control plane. Because rke2 only supports colocated etcd & control plane components, this seems like a pretty severe shortcoming. Is there any advise towards addressing this type of issue with rke2?

@creamy-pencil-82913 @gray-lawyer-73831 In reference to https://rancher-users.slack.com/archives/C01PHNP149L/p1652868532770339 Currently facing below error in pod events after setting up private registry with endpoint.

Events:
  Type     Reason     Age                     From               Message
  ----     ------     ----                    ----               -------
  Normal   Scheduled  21s                     default-scheduler  Successfully assigned default/ecr-pod to sv11
  Normal   BackOff    8s                      kubelet            Back-off pulling image "private-registry/utils-img:latest"
  Warning  Failed     8s                      kubelet            Error: ImagePullBackOff
  Normal   Pulling    <invalid> (x2 over 8s)  kubelet            Pulling image "private-registry/utils-img:latest"
  Warning  Failed     <invalid> (x2 over 8s)  kubelet            Failed to pull image "private-registry/utils-img:latest": rpc error: code = NotFound desc = failed to pull and unpack image "private-registry/utils-img:latest": failed to unpack image on snapshotter overlayfs: unexpected media type text/html for sha256:6066282cb389d5bee17ec9f08335850ece266457a557511d3f5e78eabc34df21: not found  
Warning  Failed     <invalid> (x2 over 8s)  kubelet            Error: ErrImagePull

I am trying to add rewrite as mentioned https://github.com/k3s-io/k3s/issues/5502#issuecomment-1109107238 to overcome this issue. After adding rewrite

/etc/rancher/rke2/registries.yaml

its not reflected in the environment, would like to know rewrite in

/var/lib/rancher/rke2/agent/etc/containerd/config.toml

to overcome this issue. correct me if am wrong.

Hello — Using RKE2 in a multinode air-gapped setup. If I have some custom containers I want to able to run in the cluster, do I need to have the tar files in /var/lib/rancher/rke2/agent/images/ in each of the nodes? or just on the master. Will the pods only be able to pull the image if it exists on the node its trying to run on? Thanks in advance!

Hi everyone, I am having some trouble creating a custom rke2 cluster from Rancher UI and the error message is not really helpful. The rancher-system-agent binary gets downloaded to the nodes but there is an error saying

run.sh

is not found

May 25 13:18:53  rancher-system-agent[4497]: time="2022-05-25T13:18:53+03:00" level=info msg="Pulling image <http://index.docker.io/rancher/system-a|index.docker.io/rancher/system-a>
gent-installer-rke2:v1.23.6-rke2r2"
May 25 13:18:55  rancher-system-agent[4497]: time="2022-05-25T13:18:55+03:00" level=info msg="[Applyinator] Running command: sh [-c run.sh]"
May 25 13:18:55  rancher-system-agent[4497]: time="2022-05-25T13:18:55+03:00" level=info msg="[48304be41fe2284adc2b5937c9d5c9c7b09009b2cae77
6320036286c2f0c3191_0:stderr]: sh: 1: run.sh: not found"
May 25 13:18:55  rancher-system-agent[4497]: time="2022-05-25T13:18:55+03:00" level=info msg="[Applyinator] Command sh [-c run.sh] finished
with err: <nil> and exit code: 127"
May 25 13:18:55  rancher-system-agent[4497]: time="2022-05-25T13:18:55+03:00" level=error msg="error executing instruction 0: <nil>"
May 25 13:18:55  rancher-system-agent[4497]: time="2022-05-25T13:18:55+03:00" level=error msg="error encountered during parsing of last run
time: parsing time \"\" as \"Mon Jan _2 15:04:05 MST 2006\": cannot parse \"\" as \"Mon\""
May 25 13:18:55  rancher-system-agent[4497]: time="2022-05-25T13:18:55+03:00" level=info msg="[Applyinator] No image provided, creating empt
y working directory /var/lib/rancher/agent/work/20220525-131853/48304be41fe2284adc2b5937c9d5c9c7b09009b2cae776320036286c2f0c3191_0"
May 25 13:18:55  rancher-system-agent[4497]: time="2022-05-25T13:18:55+03:00" level=info msg="[Applyinator] Running command: sh [-c rke2 etc
d-snapshot list --etcd-s3=false 2>/dev/null]"
May 25 13:18:55  rancher-system-agent[4497]: time="2022-05-25T13:18:55+03:00" level=info msg="[Applyinator] Command sh [-c rke2 etcd-snapsho
t list --etcd-s3=false 2>/dev/null] finished with err: <nil> and exit code: 127"
May 25 13:18:56  rancher-system-agent[4497]: time="2022-05-25T13:18:56+03:00" level=error msg="error loading x509 client cert/key for probe
kube-apiserver (/var/lib/rancher/rke2/server/tls/client-kube-apiserver.crt//var/lib/rancher/rke2/server/tls/client-kube-apiserver.key): open /var/lib/
rancher/rke2/server/tls/client-kube-apiserver.crt: no such file or directory"

Is there a fully populated config.yaml out there somewhere that I can look at? Or is there an option to produce a blank one from the rke2 executable? I've dug on github but haven't found such an artifact.

When creating RKE2 cluster on Linux, I have /var/lib/rancher separate partition So, wondering what directory is used when a windows worker node is added for RKE2? I also want to create a separate partition there.

Hey all! I’ve been able to set custom configs via:

kube-apiserver-arg

kube-controller-manager-arg

and

kube-scheduler-arg

However, I would like to ensure that a certain config is NOT set that is currently being set by default. For example I would like to unset the

--hostname-override

flag for the

kubelet

service and make sure that flag is not passed when starting the kubelet service

Greetings: I am configuring a cluster using RKE2. I understand that nodes are for the control plane and that agents are for running application workloads. Is there any best-practice advice on mixing/matching on the same node? also what would the memory requirements be? I have only 3 nodes which have blocks of memory that i can adjust. Currently they have 4gb and this is for a dev cluster. Interested in the optimal setup for this small size? 1 node/2 agents, 2 nodes - 1 agent? thanks for any insight!

Hi #rke2! I want to create a high-availability RKE2 cluster where it is necessary to distribute nodes at two sites (e.g. EU-A and EU-B geographically distributed sites). It is recommended to have an odd number of masters in a cluster. But it is necessary for each sites to be failure tolerance. Is it good idea when I place 3 master nodes each sites (EU-A 3 master nodes, EU-B 3 master nodes)? Then fault tolerance is ensured both within and between sites. Am I wrong? I don't want to build multi-cluster RKE2 architecture because of Longhorn.

Hello everyone. Does Rke2 allow the creation of LoadBalancer services by default? I have a rke2 Cluster and I created a LoadBalancer service but when listing the services it is in a

pending

state. Can someone explain to me what to do to enable this option?

Hi, I am having problem with one of the nodes joining rke2 cluster running Rancher 2.6.2 The server has died and was rebuilt. The references to old server were removed from UI. Now when I try to add the node(same IP and name as before) I am getting this:

Jun  1 15:20:33 node-1 rke2[35837]: time="2022-06-01T15:20:33+02:00" level=error msg="Failed to connect to proxy" error="unexpected EOF"

Jun  1 15:20:33 node-1 rke2[35837]: time="2022-06-01T15:20:33+02:00" level=error msg="Remotedialer proxy error" error="unexpected EOF"

Greetings friends. I am trying to install RKE2 on 3 centos8 servers. When i configure the first node with the config.yaml in /etc/rancher/config.yaml it causes the node to crash and not start clean. Ive tried to get the system to recognize the other server nodes but simply cant get it to work. I can get it to work with a single node. Any help would be greatly appreciated.

Hi, We are trying to deploy the RKE2 on a centos7 cluster...containing 1 server node and 3 agent nodes...The complete installation was smooth and able to get all the nodes in "Ready" state. But the issue is that the pods under "calico-system" namespace are going into "CrashLoopBackOff" with "Authorization error" as below,

$ kubectl logs -f pod/calico-node-r2wj8 -n calico-system
Error from server (InternalError): Internal error occurred: Authorization error (user=kube-apiserver, verb=get, resource=nodes, subresource=proxy)

Created a new clusterrolebinding with "system:kubelet-api-admin" role but that did not help. Please help us with the steps to fix this issue and let me know if you need any more details

There are server flags provided to modify some of the items in the generated static pod manifests but is there any way to modify anything outside of that? E.g. liveness probes

Hi, I want to deploy rke2 with dual stack but could not able to find any references for config.yaml. can anyone please share me any reference links of config.yaml for dual stack configuration using calico network. This is the current configuration I am using,

write-kubeconfig-mode: "0644"
tls-san:
  - "<http://master.167.254.204.58.nip.io|master.167.254.204.58.nip.io>"
node-label:
  - "nodetype=master"
cluster-cidr: "10.42.0.0/16"
service-cidr: "10.43.0.0/16"
cluster-dns: "10.43.0.10"
cluster-domain: "<http://master.xxx.xxx.xxx.xxx.nip.io|master.xxx.xxx.xxx.xxx.nip.io>"
cni:
  - calico
disable:
  - rke2-canal
  - rke2-kube-proxy

Is OEL 7.9 compatible with rke2

Hi People. So firewallD has issues with the CNI plugins as per this: https://docs.rke2.io/known_issues/#firewalld-conflicts-with-default-networking my question would be then - what options do I have to enable a firewall on the servers themselves ? so far we just left the firewalls off - but at some point we would need to enable them - suggestions?

hey guys, running rancher 2.6.5 stable release... created RKE2 cluster and also added a windows worker node... however, I see by default, it uses

c:\var\lib\rancher

as directory... is there an option - or a way I can specify something other than c-drive ? perhaps like

d:\var\lib\rancher

? reason I want to do this is to keep users-stuff off the c-drive to prevent c-drive from filling up and crashing the OS.

Hi, I am trying to install RKE2 v1.23.6 on a RHEL8 cluster (1server & 4agents) using the Airgap tarball method. The installation is successful, the nodes are added properly and into Ready state.

kubectl get nodes
NAME           STATUS                     ROLES                       AGE    VERSION
rke2-master    Ready,SchedulingDisabled   control-plane,etcd,master   112m   v1.23.6+rke2r2
rke2-worker1   Ready                      <none>                      108m   v1.23.6+rke2r2
rke2-worker2   Ready                      <none>                      108m   v1.23.6+rke2r2
rke2-worker3   Ready                      <none>                      108m   v1.23.6+rke2r2

All the pods under "kube-system" & "calico-system" namespace are up & Running properly.

kubectl get po -n calico-system
NAME                                       READY   STATUS    RESTARTS   AGE
calico-kube-controllers-69cb5d9c7b-xllwx   1/1     Running   0          113m
calico-node-86pcd                          1/1     Running   0          110m
calico-node-dxtjk                          1/1     Running   0          110m
calico-node-lb8n7                          1/1     Running   0          113m
calico-node-ld8mf                          1/1     Running   0          110m
calico-typha-86c8b747c4-rqfhk              1/1     Running   0          113m
calico-typha-86c8b747c4-xs7zs              1/1     Running   0          110m

kubectl get po -n kube-system
NAME                                                    READY   STATUS      RESTARTS   AGE
cloud-controller-manager-rke2-master                    1/1     Running     0          114m
etcd-rke2-master                                        1/1     Running     0          114m
helm-install-rke2-calico-crd-j44qb                      0/1     Completed   0          114m
helm-install-rke2-calico-hv4xg                          0/1     Completed   1          114m
helm-install-rke2-coredns-btsgx                         0/1     Completed   0          114m
helm-install-rke2-ingress-nginx-c99sw                   0/1     Completed   0          114m
helm-install-rke2-metrics-server-j5gjq                  0/1     Completed   0          114m
kube-apiserver-rke2-master                              1/1     Running     0          113m
kube-controller-manager-rke2-master                     1/1     Running     0          114m
kube-proxy-rke2-master                                  1/1     Running     0          114m
kube-proxy-rke2-worker1                                 1/1     Running     0          110m
kube-proxy-rke2-worker2                                 1/1     Running     0          110m
kube-proxy-rke2-worker3                                 1/1     Running     0          110m
kube-scheduler-rke2-master                              1/1     Running     0          114m
rke2-coredns-rke2-coredns-69c8f974c-gvwqg               1/1     Running     0          7m56s
rke2-coredns-rke2-coredns-69c8f974c-qmq8m               1/1     Running     0          7m56s
rke2-coredns-rke2-coredns-autoscaler-65c9bb465d-4g4sw   1/1     Running     0          114m
rke2-ingress-nginx-controller-2bhbc                     1/1     Running     0          113m
rke2-ingress-nginx-controller-gnpk5                     1/1     Running     0          110m
rke2-ingress-nginx-controller-znh8z                     1/1     Running     0          110m
rke2-ingress-nginx-controller-ztpjf                     1/1     Running     0          110m
rke2-metrics-server-6564db4569-m5kcc                    1/1     Running     0          113m

I have deployed kafka but the pods are going to "CrashLoopBackOff" with a dns error,

java.net.UnknownHostException: zookeeper.msgbus.svc: Temporary failure in name resolution

The logs of coredns pods are throwing a lot of errors as,

[ERROR] plugin/errors: 2 4686730224998678655.5699833978585684595. HINFO: read udp xx.xx.xx.xx:51070->yy.yy.yy.yy:53: read: no route to host
[ERROR] plugin/errors: 2 4686730224998678655.5699833978585684595. HINFO: read udp xx.xx.xx.xx:42332->yy.yy.yy.yy:53: read: no route to host
[ERROR] plugin/errors: 2 4686730224998678655.5699833978585684595. HINFO: read udp xx.xx.xx.xx:57092->yy.yy.yy.yy:53: read: no route to host

I have opened port 53 to allow both tcp and udp traffic and am not about the cause of this issue. Any suggestions would be appreciated and let me know if you need any more debug logs

Hi all. I loved rke2 for a every long time and know i am planning to implement it on a customers site. But the customer thinks somehow that the vanilla Kubernetes has more of a stable development team (because of the fact that it is vanilla). Can anybody help me with reasonable arguments so i can convince the customer to choose for rke2 ?