This may be more of a CAPI/provisioning question, but is there a good way to bootstrap an RKE2 cluster that can then be managed by Rancher installed on said cluster? We would like to use Rancher's provisioning agents for upgrades and snapshots for all the clusters including the local first cluster.

Hello All, Hope you are doing well. I'm trying to build and release kubernetes itself locally in a private environment (After the code is signed off upstream by the release managers) and wanted to get some info on how rancher does the build and release of kubernetes. Is there any documentation around it. Thanks in advance.

Greetings, I’ve been looking to deploy an RKE2 cluster with CAPI but I am having an extremely hard time finding documentation. I see it reference many times, could someone point me in the right direction?

Rancher can do that for you, its a hybrid cluster api

There is also some experimental k3s provider iirc

you can’t use it without rancher?

You can certainly do the TAR-based install from the RKE2 install shell script and the CAPI install uses that and deploys similarly. I've seen it tossed in an Ansible playbook as well. What are you trying to do with a CAPI install without Rancher that's distinct from just doing a TAR-based install?

hi, I’m fresh to rke2, and any advice on how to deploy a self-host registry on rke2 cluster. My problem is that I cannot make it clear on how rke2 node pull the image in the underhood. More specifically, in my opinion, people can edit the /etc/rke2/config.yaml to set up the default registry for the agent node, which will change the configuration of the config.toml, and it needs to restart the rke2/containerd daemon to make it work. Assuming I understand it clearly. Then what will happen if I deploy a Harbor registry on the rke2 cluster itself and point the registry to the self-hosted harbor one. It seems there will be a dead-lock because of the circular dependency (containerd and harbor). containerd needs to pull image from harbor, while harbor is deployed on rke2 cluster. Could anyone explain something about it for me? Great thanks.

another question is about the practice for maintaining and upgrading the rke2 cluster. More specifically my question is about how to deal with the workload above the nodes when planing a upgrade (which needs to restart rke2/kubelet/containerd?). I cannot make it clear the side-effect and bad things for shutting down the rke2/kubelet/containerd. For example, rook is a ceph storage orchestrator on k8s, however, ceph will automatically replicate data to live disks if some disks get out of the cluster. I wonder to know if I want to upgrade the underlying rke2 cluster, what should I do with workloads like rook? should I drain a node or what to do to keep my workload healthy. I’m really fresh to k8s and rke2, great thanks for you all. Any suggested documentation for me to learn how to operate and maintain a production k8s/rke2 cluster?

hi Team. Is my understanding correct that for RKE2 we don’t need to install docker on master/worker nodes before creating a cluster and specified container runtime will be installed during an installation? thx!

Hi all, I’m adding a node to an existing RKE2 cluster on Rancher but I’m getting the following error in the

rke2-server

logs and I’m at a loss.

"Waiting to retrieve kube-proxy configuration; server is not ready: <https://127.0.0.1:9345/v1-rke2/readyz>: 500 Internal Server Error"

{"level":"warn","ts":"2023-02-09T07:42:48.269-0500","logger":"etcd-client","caller":"v3@v3.5.4-k3s1/retry_interceptor.go:62","msg":"retrying of unary invoker failed","target":"<etcd-endpoints://0xc0005e3a40/127.0.0.1:2379>","attempt":0,"error":"rpc error: code = DeadlineExceeded desc = context deadline exceeded"}

I had no issues when adding the first 2 nodes, just this 3rd node.

I’ve tried Ubuntu 20.04 (which is what the other nodes are running) and I’m now trying on Debian 11 but the same sissue.

Hi experts, I have a 3-node RKE2 cluster created through the Rancher manager UI. The cluster was created and all nodes are up. However, one of the pods is getting "Readiness probe failed" error continuously. I did some digging and here's what I have found but I don't understand it or how to fix it. The problem only appears on node henry-cluster4-pool1-27cb5e2d-fsfxg. Pod kube-apiserver-henry-cluster4-pool1-27cb5e2d-fsfxg has been getting error with the Readiness check. The RKE2 version is v1.24.9+rke2r2 The errors point to the client-cert and client-key. I checked the other 2 nodes and they don't have the same issue. I did not generate/create these certs so they must be handled by Rancher automatically during the cluster creation in the UI. I appreciate any suggestions on how to remediate this issue.

# ./kubectl get nodes
NAME                                  STATUS   ROLES                              AGE   VERSION
henry-cluster4-pool1-27cb5e2d-2k2mw   Ready    control-plane,etcd,master,worker   22h   v1.24.9+rke2r2
henry-cluster4-pool1-27cb5e2d-82llq   Ready    control-plane,etcd,master,worker   22h   v1.24.9+rke2r2
henry-cluster4-pool1-27cb5e2d-fsfxg   Ready    control-plane,etcd,master,worker   22h   v1.24.9+rke2r2

Truncated the output for brevity:

# ./kubectl describe pod kube-apiserver-henry-cluster4-pool1-27cb5e2d-fsfxg -n kube-system
Name:                 kube-apiserver-henry-cluster4-pool1-27cb5e2d-fsfxg
Namespace:            kube-system
Priority:             2000000000
Priority Class Name:  system-cluster-critical
Node:                 henry-cluster4-pool1-27cb5e2d-fsfxg/xxxxxxxxxxx
Start Time:           Wed, 08 Feb 2023 14:06:08 -0500
Labels:               component=kube-apiserver
                      tier=control-plane
Annotations:          <http://kubernetes.io/config.hash|kubernetes.io/config.hash>: 4874a08227e8932676b83ca998a390f3
                      <http://kubernetes.io/config.mirror|kubernetes.io/config.mirror>: 4874a08227e8932676b83ca998a390f3
                      <http://kubernetes.io/config.seen|kubernetes.io/config.seen>: 2023-02-08T13:54:41.543487161-05:00
                      <http://kubernetes.io/config.source|kubernetes.io/config.source>: file
                      <http://kubernetes.io/psp|kubernetes.io/psp>: global-unrestricted-psp
Status:               Running
IP:                   XXXXXXXXXX
IPs:
  IP:           XXXXXXXXXX
Controlled By:  Node/henry-cluster4-pool1-27cb5e2d-fsfxg
Containers:
  kube-apiserver:
    Container ID:  <containerd://07c1f36b907b90784ed1a6d7e4c95c6817018a113ce7a9b555a87660b72e4fc>e
    Image:         <http://index.docker.io/rancher/hardened-kubernetes:v1.24.9-rke2r2-build20230104|index.docker.io/rancher/hardened-kubernetes:v1.24.9-rke2r2-build20230104>
    Image ID:      <http://docker.io/rancher/hardened-kubernetes@sha256:284ed583bf9011db9110b47683084c3238127c49ab937ad31be3efbf5656a0bc|docker.io/rancher/hardened-kubernetes@sha256:284ed583bf9011db9110b47683084c3238127c49ab937ad31be3efbf5656a0bc>
    Port:          <none>
    Host Port:     <none>
    Command:
      kube-apiserver
    Args:
      --allow-privileged=true
      --anonymous-auth=false
      --api-audiences=<https://kubernetes.default.svc.cluster.local>,rke2
      --authorization-mode=Node,RBAC
      --bind-address=0.0.0.0
      --cert-dir=/var/lib/rancher/rke2/server/tls/temporary-certs
      --client-ca-file=/var/lib/rancher/rke2/server/tls/client-ca.crt
      --egress-selector-config-file=/var/lib/rancher/rke2/server/etc/egress-selector-config.yaml
      --enable-admission-plugins=NodeRestriction,PodSecurityPolicy
      --enable-aggregator-routing=true
      --encryption-provider-config=/var/lib/rancher/rke2/server/cred/encryption-config.json
      --etcd-cafile=/var/lib/rancher/rke2/server/tls/etcd/server-ca.crt
      --etcd-certfile=/var/lib/rancher/rke2/server/tls/etcd/client.crt
      --etcd-keyfile=/var/lib/rancher/rke2/server/tls/etcd/client.key
      --etcd-servers=<https://127.0.0.1:2379>
      --feature-gates=JobTrackingWithFinalizers=true
      --kubelet-certificate-authority=/var/lib/rancher/rke2/server/tls/server-ca.crt
      --kubelet-client-certificate=/var/lib/rancher/rke2/server/tls/client-kube-apiserver.crt
      --kubelet-client-key=/var/lib/rancher/rke2/server/tls/client-kube-apiserver.key
      --kubelet-preferred-address-types=InternalIP,ExternalIP,Hostname
      --profiling=false
      --proxy-client-cert-file=/var/lib/rancher/rke2/server/tls/client-auth-proxy.crt
      --proxy-client-key-file=/var/lib/rancher/rke2/server/tls/client-auth-proxy.key
      --requestheader-allowed-names=system:auth-proxy
      --requestheader-client-ca-file=/var/lib/rancher/rke2/server/tls/request-header-ca.crt
      --requestheader-extra-headers-prefix=X-Remote-Extra-
      --requestheader-group-headers=X-Remote-Group
      --requestheader-username-headers=X-Remote-User
      --secure-port=6443
      --service-account-issuer=<https://kubernetes.default.svc.cluster.local>
      --service-account-key-file=/var/lib/rancher/rke2/server/tls/service.key
      --service-account-signing-key-file=/var/lib/rancher/rke2/server/tls/service.key
      --service-cluster-ip-range=10.43.0.0/16
      --service-node-port-range=30000-32767
      --storage-backend=etcd3
      --tls-cert-file=/var/lib/rancher/rke2/server/tls/serving-kube-apiserver.crt
      --tls-private-key-file=/var/lib/rancher/rke2/server/tls/serving-kube-apiserver.key
    State:          Running
      Started:      Wed, 08 Feb 2023 14:12:32 -0500
    Last State:     Terminated
      Reason:       Error
      Exit Code:    137
      Started:      Wed, 08 Feb 2023 13:54:42 -0500
      Finished:     Wed, 08 Feb 2023 14:12:10 -0500
    Ready:          True
    Restart Count:  1
    Requests:
      cpu:      250m
      memory:   1Gi
    Liveness:   exec [kubectl get --server=<https://localhost:6443/> --client-certificate=/var/lib/rancher/rke2/server/tls/client-kube-apiserver.crt --client-key=/var/lib/rancher/rke2/server/tls/client-kube-apiserver.key --certificate-authority=/var/lib/rancher/rke2/server/tls/server-ca.crt --raw=/livez] delay=10s timeout=15s period=10s #success=1 #failure=8
    Readiness:  exec [kubectl get --server=<https://localhost:6443/> --client-certificate=/var/lib/rancher/rke2/server/tls/client-kube-apiserver.crt --client-key=/var/lib/rancher/rke2/server/tls/client-kube-apiserver.key --certificate-authority=/var/lib/rancher/rke2/server/tls/server-ca.crt --raw=/readyz] delay=0s timeout=15s period=5s #success=1 #failure=3
    Startup:    exec [kubectl get --server=<https://localhost:6443/> --client-certificate=/var/lib/rancher/rke2/server/tls/client-kube-apiserver.crt --client-key=/var/lib/rancher/rke2/server/tls/client-kube-apiserver.key --certificate-authority=/var/lib/rancher/rke2/server/tls/server-ca.crt --raw=/livez] delay=10s timeout=5s period=10s #success=1 #failure=24
    Environment:
      FILE_HASH:  65c747a3981e887284258667d87df8f0aac1d5eb238d049859c1986c85b92190
      NO_PROXY:   .svc,.cluster.local,10.42.0.0/16,10.43.0.0/16
      POD_HASH:   0d017805ffe058495a528aa9431ffa9b
    Mounts:
      /etc/ca-certificates from dir1 (rw)
      /etc/ssl/certs from dir0 (rw)
      /var/lib/rancher/rke2/server/cred/encryption-config.json from file1 (ro)
      /var/lib/rancher/rke2/server/db/etcd/name from file0 (ro)
      /var/lib/rancher/rke2/server/etc/egress-selector-config.yaml from file2 (ro)
      /var/lib/rancher/rke2/server/logs from dir2 (rw)
      /var/lib/rancher/rke2/server/tls/client-auth-proxy.crt from file3 (ro)
      /var/lib/rancher/rke2/server/tls/client-auth-proxy.key from file4 (ro)
      /var/lib/rancher/rke2/server/tls/client-ca.crt from file5 (ro)
      /var/lib/rancher/rke2/server/tls/client-kube-apiserver.crt from file6 (ro)
      /var/lib/rancher/rke2/server/tls/client-kube-apiserver.key from file7 (ro)
      /var/lib/rancher/rke2/server/tls/etcd/client.crt from file8 (ro)
      /var/lib/rancher/rke2/server/tls/etcd/client.key from file9 (ro)
      /var/lib/rancher/rke2/server/tls/etcd/server-ca.crt from file10 (ro)
      /var/lib/rancher/rke2/server/tls/request-header-ca.crt from file11 (ro)
      /var/lib/rancher/rke2/server/tls/server-ca.crt from file12 (ro)
      /var/lib/rancher/rke2/server/tls/service.key from file13 (ro)
      /var/lib/rancher/rke2/server/tls/serving-kube-apiserver.crt from file14 (ro)
      /var/lib/rancher/rke2/server/tls/serving-kube-apiserver.key from file15 (ro)
---
---
---

Events:
  Type     Reason     Age                 From     Message
  ----     ------     ----                ----     -------
  Warning  Unhealthy  28m (x32 over 22h)  kubelet  Readiness probe failed: Error from server (InternalError): an error on the server ("[+]ping ok\n[+]log ok\n[-]etcd failed: reason withheld\n[+]informer-sync ok\n[+]poststarthook/start-kube-apiserver-admission-initializer ok\n[+]poststarthook/generic-apiserver-start-informers ok\n[+]poststarthook/priority-and-fairness-config-consumer ok\n[+]poststarthook/priority-and-fairness-filter ok\n[+]poststarthook/start-apiextensions-informers ok\n[+]poststarthook/start-apiextensions-controllers ok\n[+]poststarthook/crd-informer-synced ok\n[+]poststarthook/bootstrap-controller ok\n[+]poststarthook/rbac/bootstrap-roles ok\n[+]poststarthook/scheduling/bootstrap-system-priority-classes ok\n[+]poststarthook/priority-and-fairness-config-producer ok\n[+]poststarthook/start-cluster-authentication-info-controller ok\n[+]poststarthook/aggregator-reload-proxy-client-cert ok\n[+]poststarthook/start-kube-aggregator-informers ok\n[+]poststarthook/apiservice-registration-controller ok\n[+]poststarthook/apiservice-status-available-controller ok\n[+]poststarthook/kube-apiserver-autoregistration ok\n[+]autoregister-completion ok\n[+]poststarthook/apiservice-openapi-controller ok\n[+]poststarthook/apiservice-openapiv3-controller ok\n[+]shutdown ok\nreadyz check failed") has prevented the request from succeeding

When executing the command for the Readiness check I get the following error:

# ./kubectl get --server=<https://localhost:6443/> --client-certificate=/var/lib/rancher/rke2/server/tls/client-kube-apiserver.crt --client-key=/var/lib/rancher/rke2/server/tls/client-kube-apiserver.key --certificate-authority=/var/lib/rancher/rke2/server/tls/server-ca.crt --raw=/readyz
Error in configuration: 
* client-cert-data and client-cert are both specified for default. client-cert-data will override.
* client-key-data and client-key are both specified for default; client-key-data will override

Hi Rancher Team, Will rke2 1.24.x end also support when k8s 1.24 EOL in Jul 2023? https://kubernetes.io/releases/

Hi everyone, I don't know if it should be posted here but I created a rke2 cluster using ansible-role-rke2 and I then imported the cluster to my rancher instance using the kubectl apply method. The cluster is working fine but the Apps part in rancher, when I go there I got a "There are no charts available, have you added any repos?" message, and when I got to the Repositories, I see two that are stuck in "In Progress" status, I tried recreating them refresh ... Nothing works. I am running kubernetes version 1.25.3 with rke2r1. Is anyone aware of such issue ? I have seen similar issues on github without fixes just mentions that it is fixed in next version but the issues are from 2021.

Greetings, the cluster got into a weird state. Everything is working but the master node is stuck at "reconciling". And I don't see an actual namespace called

fleet-default

. Help? 🙃

Hello, I'd like to expose a port (1094) of the rke2 ingress controller in addition to the default ones. Following the nginx documentation and the rke2 doc I tried with a HelmChartConfig like the following:

apiVersion: <http://helm.cattle.io/v1|helm.cattle.io/v1>
kind: HelmChartConfig
metadata:
  name: rke2-ingress-nginx
  namespace: kube-system
spec:
  valuesContent: |-
    controller:
      customPorts:
        - port: 1094
          targetPort: 1094
          protocol: TCP
          name: xrootd

Unfortunately it does not work, am I missing something? Thanks!

have a fresh ubuntu server that I want to use as a jumpstart server to host rancher and tinkerbell and a few other things on k3s or rke2 (preferred) and running into what is probably me overlooking something simple. disk space is ok. running as root. Any other ideas what might be going on?

hi, as we have lots of rke1 installation, will there be an upgrade tools or documentation how to migrade existing rke1 clusters to rke2? How long will the rke distribution will be developed until we must upgrade to rke2? thanks

hey, I’m fresh to rke2 and k8s. How to schedule only a specific workload to some of the nodes? which I mean that for example the node A can only and must take the workload of ‘B’, and any other workload would never be scheduled on the node A and the workload of ‘B’ has an instance on the node A. thanks.

Hi folks, I am trying to use gitlab-agent in an RKE2 cluster, setup via Rancher. CNI: calilco. Problem: Some kubectl commands (e.g. kubectl exec) in the gitlab-runner seem to get stuck. Last words in verbose mode is "Upgrade: SPDY/3.1" (see below), then it runs into a 30sec timeout. There is no such problem for K3s (flannel) and RKE (calico), so I wonder what is the story here?

I0210 14:18:58.178856     320 round_trippers.go:466] curl -v -XPOST  -H "X-Stream-Protocol-Version: <http://v4.channel.k8s.io|v4.channel.k8s.io>" -H "X-Stream-Protocol-Version: <http://v3.channel.k8s.io|v3.channel.k8s.io>" -H "X-Stream-Protocol-Version: <http://v2.channel.k8s.io|v2.channel.k8s.io>" -H "X-Stream-Protocol-Version: <http://channel.k8s.io|channel.k8s.io>" -H "User-Agent: kubectl/v1.24.10 (linux/amd64) kubernetes/5c1d2d4" -H "Authorization: Bearer <masked>" '<https://kas.gitlab.com/k8s-proxy/api/v1/namespaces/sample-kube001/pods/ubuntu/exec?command=echo&command=hello&container=ubuntu&stderr=true&stdout=true>'
I0210 14:18:58.652230     320 round_trippers.go:553] POST <https://kas.gitlab.com/k8s-proxy/api/v1/namespaces/sample-kube001/pods/ubuntu/exec?command=echo&command=hello&container=ubuntu&stderr=true&stdout=true> 101 Switching Protocols in 473 milliseconds
I0210 14:18:58.652260     320 round_trippers.go:570] HTTP Statistics: DNSLookup 0 ms Dial 0 ms TLSHandshake 0 ms Duration 473 ms
I0210 14:18:58.652274     320 round_trippers.go:577] Response Headers:
I0210 14:18:58.652330     320 round_trippers.go:580]     Connection: upgrade
I0210 14:18:58.652349     320 round_trippers.go:580]     Upgrade: SPDY/3.1
I0210 14:18:58.652366     320 round_trippers.go:580]     Via: 1.1 gitlab-agent/v15.9.0-rc1/add168d2
I0210 14:18:58.652383     320 round_trippers.go:580]     Via: gRPC/1.0 gitlab-kas/v15.9.0-rc1/v15.9.0-rc1
I0210 14:18:58.652397     320 round_trippers.go:580]     X-Stream-Protocol-Version: <http://v4.channel.k8s.io|v4.channel.k8s.io>
I0210 14:18:58.652411     320 round_trippers.go:580]     Date: Fri, 10 Feb 2023 14:18:58 GMT
I0210 14:18:58.652458     320 log.go:198] (0xc000276000) (0xc000992640) Create stream
I0210 14:18:58.652474     320 log.go:198] (0xc000276000) (0xc000992640) Stream added, broadcasting: 1
error: Timeout occurred

hey, I wonder to know how to expose TCP service with rke2-ingress-nginx. I cannot find any step by step tutorial. It seems I could edit a configmap like below, but It cannot work. Any suggestion on it. Great thanks.

apiVersion: v1
kind: ConfigMap
metadata:
  name: tcp-services
  namespace: ingress-nginx
data:
  9000: "default/example-go:8080"

I have enabled selinux and most of containers fail to run all of sudden today after 2 months

It looks like it can’t access cgroups, but I fear the problem might be bigger

Huh it's not selinux issue

Ah, my fault 😄 😄

disk-pressure triggered eviction on most pods

I'm installing a cluster of 2 baremetal nodes running RHEL9. I have a Rancher 2.6.9 which already has 2 vm-clusters in it and their working fine. Now when I create this 3rd cluster and register the initial master, it won't become ready. The node stays in state "Waiting for Node Ref" and condition is "Waiting node to become ready and join url to be available". The only warning I see in the logs are in ETCD of "RAFT NO LEADER" and warnings that it cannot connect to some of it's local hosts ports. Any idea what would be my problem here?

Is there any method for pods in kubernetes to access services by URL configured in ingress-nginx controller. which I means not using the service name but something like https://test.domain.com