hi, I have a small cluster of 4 nodes (x3 control plane & etcd & worker and x1 etcd and worker) and all of them shut down. I booted them and the 3 control planes are ok meaning I can kubectl get pods, etc. the 4rth one does not come up, rancher-system-agent is up and running but the rke2-agent is not up. Logs says:

Cluster CA certificate is not trusted by the host CA bundle, but the token does not include a CA hash. Use the full token from the server's node-token file to enable Cluster CA validation.
Waiting to retrieve agent configuration; server is not ready: failed to retrieve configuration from server: <https://127.0.0.1:6444/v1-rke2/config>: 401 Unauthorized

I also have a rke2-server service, I am unsure whether I should try to bring that up too

hi, I have RKE2 cluster deployed in a private network, internet connectivity can be reached through proxy server in aws. Followed https://docs.rke2.io/advanced#configuring-an-http-proxy and set proxy setting. With rke2-coredns using rancher/hardened-coredns:v1.9.3-build20220613 nslookup fails

> kubectl run -it --rm test-pod --image=busybox:1.28 --restart=Never -- /bin/sh
/ # nslookup  kubernetes.default.svc.cluster.local
Server:    10.43.0.10
Address 1: 10.43.0.10 rke2-coredns-rke2-coredns.kube-system.svc.cluster.local

Name:      kubernetes.default.svc.cluster.local
Address 1: 10.43.0.1 kubernetes.default.svc.cluster.local
/ # nslookup  kubernetes
Server:    10.43.0.10
Address 1: 10.43.0.10 rke2-coredns-rke2-coredns.kube-system.svc.cluster.local

Name:      kubernetes
Address 1: 10.43.0.1 kubernetes.default.svc.cluster.local
/ # nslookup  <http://google.com|google.com>
Server:    10.43.0.10
Address 1: 10.43.0.10 rke2-coredns-rke2-coredns.kube-system.svc.cluster.local

nslookup: can't resolve '<http://google.com|google.com>'

has anyone tried to deploy RKE2 cluster on RHEL 9? If so, any issues?

Is there an example of an RKE2 config file (usable by the Rancher UI tool) where the scheduler has been changed to another kube-scheduler image - or something similar?

any insight on this https://cluster-api.sigs.k8s.io/ from Rancher’s point of view?

Is there a skeleton full RKE2 config.yaml or guide to customising images like the scheduler?

Hi All, Anybody is using Calico as the rke2 CNI instead of Canal? How does it compare to the default CNI?

is there a way to limit the outgoing network bandwidth of a pod to some IP address range. great thanks

is there a way to set the default tolerations for all pods run in a namespace in RKE2?

Hey all! Looking to apply some custom configurations to

rke2-coredns

following this guidance here: https://docs.rke2.io/networking#coredns I see a sample helm chart like the following:

apiVersion: <http://helm.cattle.io/v1|helm.cattle.io/v1>
kind: HelmChartConfig
metadata:
  name: rke2-coredns
  namespace: kube-system
spec:
  valuesContent: |-
    nodelocal:
      enabled: true

Where can I find more information regarding all of the possible chart values for RKE2 CoreDNS? Thank you!

Hello! So I am trying to implement the

extraSecrets

config parameter: https://github.com/rancher/rke2-charts/blob/e29071e486dc0b987ed665d9ebe16cfdb681247c/charts/rke2-coredns/rke2-coredns/1.19.401/values.yaml#L231 My CoreDNS Helm looks like this:

apiVersion: <http://helm.cattle.io/v1|helm.cattle.io/v1>
          kind: HelmChartConfig
          metadata:
            name: rke2-coredns
            namespace: kube-system
          spec:
            valuesContent: |-
              servers:
              - zones:
                - zone: .
                port: 53
                plugins:
                - name: errors
                - name: health
                  configBlock: |-
                    lameduck 5s
                - name: ready
                - name: kubernetes
                  parameters: cluster.local in-addr.arpa ip6.arpa
                  configBlock: |-
                    pods insecure
                    fallthrough in-addr.arpa ip6.arpa
                    ttl 30
                - name: prometheus
                  parameters: 0.0.0.0:9153
                - name: forward
                  parameters: . /etc/resolv.conf
                - name: cache
                  parameters: 30
                - name: loop
                - name: reload
                - name: loadbalance
          extraSecrets:
            - name: certs
              mountPath: /etc/ssl/private/certs

However I’m seeing this error in the logs when starting up:

unknown field "extraSecrets"

Mar 11 00:36:53 ip-192-168-0-10.us-gov-east-1.compute.internal rke2[1981]: I0311 00:36:53.877190    1981 event.go:294] "Event occurred" object="kube-system/rke2-coredns-config" fieldPath="" kind="Addon" apiVersion="<http://k3s.cattle.io/v1|k3s.cattle.io/v1>" type="Normal" reason="ApplyingManifest" message="Applying manifest at \"/var/lib/rancher/rke2/server/manifests/rke2-coredns-config.yaml\""
Mar 11 00:36:53 ip-192-168-0-10.us-gov-east-1.compute.internal rke2[1981]: W0311 00:36:53.883804    1981 warnings.go:70] unknown field "extraSecrets"

The secret has been created in the

kube-system

namespace:

$ kubectl get secrets -n kube-system | grep certs
certs                                                              Opaque               2      17m

Neeeevermind, my tab spacing is off 😅

Deploying RKE2 with Metallb Issue - IP gets assigned to the loadbalancer but no ARP entries are showing up under the interfaces, ARP ping and curl fails to the deployed loadbalancer IP + port IP Pool and L2Advertisement config

apiVersion: <http://metallb.io/v1beta1|metallb.io/v1beta1>
kind: IPAddressPool
metadata:
  name: core-net-192.168.92.140-159
  namespace: metallb-system
spec:
  addresses:
  - 192.168.94.140-192.168.94.159
---
apiVersion: <http://metallb.io/v1beta1|metallb.io/v1beta1>
kind: L2Advertisement
metadata:
  name: metallb-pool
  namespace: metallb-system
spec:
  ipAddressPools:
  - core-net-192.168.99.140-159

How can we enable kubeproxy ipvs on the management cluster? (at the moment i only have one cluster with workernodes added to the management cluster)

kubeproxy:
      extra_args:
        ipvs-scheduler: lc
        proxy-mode: ipvs

do i need to deploy a separate cluster with workernodes for this? Thanks a lot for your input, if im on the wrong channel for these questions please let me know, i apologize in advance

Feels like a stupid question - but how do you upgrade Rancher on an RKE2 cluster? I followed the instructions to upgrade RKE2 (https://docs.rke2.io/upgrade/manual_upgrade) and that went fine, but how do I update Rancher...?

how to add the following to the cluster.yaml

kube-proxy-arg:
  - proxy-mode=ipvs
  - ipvs-strict-arp=true

I added the following based on - link but metallb is still not ARPing with the Loadbalancer IP

kubeproxy:
      extra_args:
        ipvs-scheduler: lc
        proxy-mode: ipvs

Rancher v2.7.0 with downstream RKE2 cluster. Not able to import an orphaned cluster into a new instance of Rancher. Import Existing -> Import any Kubernetes cluster. After issuing the import command on the cluster several pods go into CrashLoop and do not recover:

/var/lib/rancher/rke2/bin/kubectl         --kubeconfig /etc/rancher/rke2/rke2.yaml apply -f <https://rancher75182.senode.dev/v3/import/xhctfcnbbt56xvxh6jptq7lzvpw9svd2drkbj5pvm466t5r7zlplqv_c-m-zqcvzlgn.yaml>
<http://clusterrole.rbac.authorization.k8s.io/proxy-clusterrole-kubeapiserver|clusterrole.rbac.authorization.k8s.io/proxy-clusterrole-kubeapiserver> unchanged
<http://clusterrolebinding.rbac.authorization.k8s.io/proxy-role-binding-kubernetes-master|clusterrolebinding.rbac.authorization.k8s.io/proxy-role-binding-kubernetes-master> unchanged
namespace/cattle-system unchanged
serviceaccount/cattle unchanged
<http://clusterrolebinding.rbac.authorization.k8s.io/cattle-admin-binding|clusterrolebinding.rbac.authorization.k8s.io/cattle-admin-binding> unchanged
secret/cattle-credentials-ad9a794 created
<http://clusterrole.rbac.authorization.k8s.io/cattle-admin|clusterrole.rbac.authorization.k8s.io/cattle-admin> unchanged
Warning: spec.template.spec.affinity.nodeAffinity.requiredDuringSchedulingIgnoredDuringExecution.nodeSelectorTerms[0].matchExpressions[0].key: <http://beta.kubernetes.io/os|beta.kubernetes.io/os> is deprecated since v1.14; use "<http://kubernetes.io/os|kubernetes.io/os>" instead
deployment.apps/cattle-cluster-agent configured
service/cattle-cluster-agent unchanged

NAMESPACE             NAME                                                    READY   STATUS             RESTARTS      AGE
calico-system         calico-kube-controllers-f75c97ff6-fvb66                 1/1     Running            0             19m
calico-system         calico-node-6vxmh                                       1/1     Running            0             19m
calico-system         calico-node-d9t8n                                       0/1     Running            0             17m
calico-system         calico-node-khhpr                                       1/1     Running            0             19m
calico-system         calico-node-nmcds                                       0/1     Running            0             17m
calico-system         calico-typha-d65458ffc-97pn9                            1/1     Running            0             17m
calico-system         calico-typha-d65458ffc-p9cj2                            1/1     Running            0             19m
cattle-fleet-system   fleet-agent-6c857b85b5-zff2l                            1/1     Running            0             17m
cattle-system         cattle-cluster-agent-6f588568-dj7ql                     0/1     CrashLoopBackOff   4 (49s ago)   4m9s
cattle-system         cattle-cluster-agent-6f588568-zl55k                     0/1     CrashLoopBackOff   4 (29s ago)   3m53s
kube-system           cloud-controller-manager-sempre1-ctrl                   1/1     Running            0             20m
kube-system           cloud-controller-manager-sempre1-etcd                   1/1     Running            0             20m
kube-system           etcd-sempre1-etcd                                       1/1     Running            0             19m
kube-system           helm-install-rke2-calico-7dxlb                          0/1     Completed          2             20m
kube-system           helm-install-rke2-calico-crd-wzffm                      0/1     Completed          0             20m
kube-system           helm-install-rke2-coredns-zs9rl                         0/1     Completed          0             20m
kube-system           helm-install-rke2-ingress-nginx-gtkv8                   0/1     CrashLoopBackOff   6 (40s ago)   20m
kube-system           helm-install-rke2-metrics-server-blcf4                  0/1     CrashLoopBackOff   6 (51s ago)   20m
kube-system           kube-apiserver-sempre1-ctrl                             1/1     Running            0             20m
kube-system           kube-controller-manager-sempre1-ctrl                    1/1     Running            0             20m
kube-system           kube-proxy-sempre1-ctrl                                 1/1     Running            0             20m
kube-system           kube-proxy-sempre1-etcd                                 1/1     Running            0             20m
kube-system           kube-proxy-sempre1-wrk1                                 1/1     Running            0             17m
kube-system           kube-proxy-sempre1-wrk2                                 1/1     Running            0             17m
kube-system           kube-scheduler-sempre1-ctrl                             1/1     Running            0             20m
kube-system           rke2-coredns-rke2-coredns-58fd75f64b-kfb69              1/1     Running            0             19m
kube-system           rke2-coredns-rke2-coredns-58fd75f64b-rzpsg              1/1     Running            0             20m
kube-system           rke2-coredns-rke2-coredns-autoscaler-768bfc5985-hcf4b   1/1     Running            0             20m
tigera-operator       tigera-operator-586758ccf7-rc9tq                        1/1     Running            0             19m

Looking the logs seems cluster agent is unable to ping the rancher server, but if I do a curl on the same URL it responds with a pong.
ERROR: <https://rancher75182.senode.dev/ping> is not accessible (Could not resolve host: rancher75182.senode.dev)

helm pods report error as well:
/var/lib/rancher/rke2/bin/kubectl --kubeconfig /etc/rancher/rke2/rke2.yaml logs helm-install-rke2-ingress-nginx-gtkv8 -n cattle-system
Error from server (NotFound): pods "helm-install-rke2-ingress-nginx-gtkv8" not found
/var/lib/rancher/rke2/bin/kubectl --kubeconfig /etc/rancher/rke2/rke2.yaml logs helm-install-rke2-metrics-server-blcf4 -n cattle-system
Error from server (NotFound): pods "helm-install-rke2-metrics-server-blcf4" not found

I am back with another issue I am running into... I have everything almost working with my rke2 vsphere cpi/csi enabled cluster, but I am having a problem with the windows node-driver-registrar container... Has anyone run into the following error before?

I0314 12:43:17.862979    7024 main.go:109] "Kubelet registration probe created" path="\\var\\lib\\kubelet\\plugins\\<http://csi.vsphere.vmware.com|csi.vsphere.vmware.com>\\registration"

I0314 12:43:20.196288    7024 main.go:120] Received NotifyRegistrationStatus call: &RegistrationStatus{PluginRegistered:false,Error:RegisterPlugin error -- plugin registration failed with err: rpc error: code = Unavailable desc = connection error: desc = "transport: Error while dialing dial unix \\\\var\\\\lib\\\\kubelet\\\\plugins\\\\<http://csi.vsphere.vmware.com|csi.vsphere.vmware.com>\\\\csi.sock: connect: A socket operation was attempted to an unreachable network.",}

E0314 12:43:20.196288    7024 main.go:122] Registration process failed with error: RegisterPlugin error -- plugin registration failed with err: rpc error: code = Unavailable desc = connection error: desc = "transport: Error while dialing dial unix \\\\var\\\\lib\\\\kubelet\\\\plugins\\\\<http://csi.vsphere.vmware.com|csi.vsphere.vmware.com>\\\\csi.sock: connect: A socket operation was attempted to an unreachable network.", restarting registration container.

But I do see this does exist on the windows node:

PS C:\var\lib\kubelet\plugins\<http://csi.vsphere.vmware.com|csi.vsphere.vmware.com>> ls


    Directory: C:\var\lib\kubelet\plugins\<http://csi.vsphere.vmware.com|csi.vsphere.vmware.com>


Mode                LastWriteTime         Length Name
----                -------------         ------ ----
-a----         3/8/2023   3:18 AM              0 csi.sock

On RHEL 8 VMs... For RKE2, looking at this document, https://docs.rke2.io/install/configuration

Method of installation to use. Default is on RPM-based systems

rpm

, all else

tar

And, with Rancher 2.7.1.... Does it make sense to create RKE2 cluster then import into Rancher or create RKE2 cluster from Rancher? Is one method better over other?

does it make any sense to use private CA signed certs for “internal” comms, ie. kubelets <-> apiserver?

Creating RKE2 custom cluster from Rancher UI 2.7.1, and when I select Cloud Provider to External, I notice that Cloud Provider Config line opens up (screen shot below). If I leave that blank cluster creation does not work. TBH I am not sure what to put there? Is this where I can say "--disable-cloud-controller"?

I'm trying to rebuild a node that was in a rke2 cluster. I've got it rebuilt and rejoined but it's in a notready state because

container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: cni plugin not initialized

but I have disabled deploying cni via rke2 (manual install of cilium). This is a chicken/egg situation that I'm not sure how I haven't run into it before now. How can I get kubelet to come up ready enough to get cni from daemonset in the cluster?

Hello, does anyone know if it is possible to change the default pod/container logs directories from

/var/log/pods

and

/var/log/containers

to something else? Setting

--log-dir

in the kubelet is deprecated unfortunately. Also, I'm not looking for redirection, I need

/var/log/pods

and

/var/log/containers

to have no logs from kubernetes.

@here: is anyone faced/facing the below issue when spinning up kubernetes cluster using rancher 2.7.

level=info msg="[Applyinator] No image provided, creating empty working directory /var/lib/rancher/agent/work/

Hello so I’m quite new to running kubernetes, I have ran it in a test environment and created some apps. However I’m looking to run it in production. How much work is it to maintain a cluster, meaning is it something that requires on going maintenance, or can I just leave it and it will keep running. I know that certs need to be updated for the control plane , is that something that is done automatically with rke2 I think with installation with kubeadm it is not. I really like rke2 I’m just wondering how much operational maintenance is involved. Thanks in advance

I have cluster stuck with waitibg for pboes : kubelet

can someone helpk with this issue

after launching the master, how to edit the node yaml file to include the external ip?

After deploying RKE2 cluster via Rancher 2.7.1... I want to change "Cloud Provider to External" I can do that by edit Yaml and add

cloud-provider-name: external

under

machineSelectorConfig

. This appears to work... My question with this is, do I also need to include

disable-cloud-controller: true

? If so, where does this get added in YAML or the GUI?