Is there any relationship between the

token

in config.yaml and the EncryptionConfig? I’m trying to figure out how just the token is sufficient to restore a node from a etcd backup. Is the encryptionConfig also stored in etcd ?

Hello all! I have a new RKE2 cluster in AWS that requires IMDSv2 to be enabled. This allows the cluster to function at the moment. HOWEVER, my issue seem to be when I install CLUSTER-AUTOSCALER. From what I can see the CLUSTER-AUTOSCALER can not assume the IAM role that I have assigned to the nodes, and will fail with the following messages: Failed to regenerate ASG cache: cannot autodiscover ASGs: NoCredentialProviders: no valid providers in chain. Deprecated. AND....... Failed to create AWS Manager: cannot autodiscover ASGs: NoCredentialProviders: no valid providers in chain. Deprecated.

I have tried to force a service account in there, but the service account does not have all the permissions that it needs for the autoscaler to work (and I am not allowed to give that service account the permissions either)

BTW: All of this works just fine on the RKE1 environment.

I'm using rancher I initially set it up with self signed certs, but now I want to switch to let's encrypt. I've already configured cert-manager and fixed the ingress for rancher. How can I set up the mtls certs to use let's encrypt certs as well? This is what i get

E0921 18:35:26.833337  270467 memcache.go:265] couldn't get current server API group list: Get "<https://rancher.internal.nullreference.io/k8s/clusters/local/api?timeout=32s>": tls: failed to verify certificate: x509: certificate signed by unknown authority

cattle cluster agent 2nd replica and on can't resolve a DNS... if I specify dnsPolicy: None + dnsConfig then it works, but I fear it might need internal DNS too

This might be a bit of a shot in the dark, but does anyone have a Calico Network Policy with the minimum required exceptions for a RKE2 & Rancher configuration to keep working?

Hello all, How to display external ip with "kubectl get nodes -o wide" cmd in rke2 - version 1.26.4? I've been tried to add node-ip-external parameter in config.yaml and that config is applied in advertise-address value.

root     2159939  9.8  5.3 1198428 438608 ?      Ssl  Sep24  67:40 kube-apiserver --admission-control-config-file=/etc/rancher/rke2/rke2-pss.yaml --audit-log-path=/var/lib/rancher/rke2/server/logs/audit.log --audit-policy-file=/etc/rancher/rke2/audit-policy.yaml --audit-log-maxage=30 --audit-log-maxbackup=10 --audit-log-maxsize=100 --advertise-address=IP PUBLIC

my config:

server: <https://192.168.0.61:9345>
data-dir: /var/lib/rancher/rke2
tls-san:
  - cluster.local
  - 192.168.0.61
  - 14.225.53.251
node-external-ip: IP PUBLIC

But "kubectl get nodes -o wide" is nothing

NAME        STATUS   ROLES                       AGE     VERSION          INTERNAL-IP    EXTERNAL-IP   OS-IMAGE             KERNEL-VERSION      CONTAINER-RUNTIME
master-01   Ready    control-plane,etcd,master   2d17h   v1.26.4+rke2r1   192.168.0.61   <none>        Ubuntu 22.04.2 LTS   5.15.0-75-generic   <containerd://1.6.19-k3s1>
worker-01   Ready    <none>                      2d17h   v1.26.4+rke2r1   192.168.0.5    <none>        Ubuntu 22.04.2 LTS   5.15.0-75-generic   <containerd://1.6.19-k3>

Thanks all.

Hey , is there any way to use podman as runtime in RKE2 ?

^*Dũng Lê* My guess is that you are not using a LoadBalancer configuration ; so there is nothing that is coordinating that association. (I do not use one, fwiw, so its not exactly a problem) - see https://stackoverflow.com/questions/44110876 or https://github.com/metallb/metallb as options.

I've had no luck getting Calico NP to work with a RKE2 cluster deployed via Rancher.

apiVersion: <http://projectcalico.org/v3|projectcalico.org/v3>

kind: GlobalNetworkPolicy

metadata:

name: default-deny

spec:

namespaceSelector: <http://projectcalico.org/name|projectcalico.org/name> not in  {'kube-system', 'calico-system', 'calico-apiserver', 'default', 'cattle-system', 'cattle-fleet-system', 'cattle-impersonation-system', 'tigera-operator', 'kube-node-lease'}

types:

- Ingress

- Egress

egress:

- action: Allow

protocol: UDP

destination:

selector: 'k8s-app == "kube-dns"'

ports:

- 53

Hi, does anyone know if its possible to programatically determine if a node is an "init node" indicated by the label

<http://rke.cattle.io/init-node|rke.cattle.io/init-node>

via the rancher v3 api? From what I can see this information is only known on the machine plan which from what I can see isn't accessible via the api.

Hi guys , When performing and setting up the backups and the restore for the Rancher downstream clusters can i change the path of the local storage ?? if yes , how can i do that ?

Hey Everyone, i just installed Rke2 cluster and the disk activity seems to be pretty high. I think the journal process is writing about 2.7Mb/second. That is too high. Is there a way to adjust the reduce the log level?

rke2-cilium@1.14.0 Ingress Controller with L2 doesn't respond to LB IP, while separate Gateway API LB IP does, there don't seem to be any logs about it, only when I set loadbalancerMode to shared, and then associated to ingress.

cilium-operator has few 1000s of lines of activity for reconciling the GW API 😱

this means few restarts per second

are there plans to address CVE-2022-1996 in docker.io/rancher/hardened-cluster-autoscaler:v1.8.6-build20230609, I understand it may not be vulnerable but this has been addressed in a newer version of github.com/emicklei/go-restful, and many scanners report this issue with the autoscaler

Hello everyone, When deploying RKE2 on bare metal, I'd like to specify a different disk for RKE2's default root directory (/var/lib/rancher), separate from the OS disk. The main goal behind this is to enhance fault tolerance. As of now, I haven't come across a direct configuration option in RKE2 to change this default directory path. I'm curious, how are you all managing this? Are you using symlinks or any other method?

Any plans on rke2-cilium 1.14.2 stable release?

Hi all , Whenver i am taking an etcd backup manually or recurring backups , it is increasing the size of my vmdk file of the control plane nodes , but on the os it is still showing the expected disk usage as expected. Can anyone help me with this ??

Hello, I am facing the following issue: a kube-proxy pod is stuck in pending state, however crictl reports it is running. I don’t see any errors neigher in rke2-agent service nor in kube-proxy

Hi!. I’ve got a rke2 cluster with etcd backups enabled to s3. I notice that the backups are also stored on disk - is there a way to auto-configure rke2 to delete them after upload?

👋 i'm trying to build a custom

cilium

image from a specific feature branch from upstream (

cilium/cilium

) and deploy that as part of a rke2 deployment using the

rke2-cilium

chart via Rancher Manager. I know how to build the container image and i've pushed that to a private image registry, to deploy this image is no problem since i can just add the image + tag to the

rke2-cilium

chart values. The thing is that the feature branch in the

cilium

repo includes changes to the official Helm chart (adds additional values), this means that i need to push a custom Helm chart to a private chart repo. AFAICT the official

cilium

Helm chart is patched as part of the process to make it into a

rke2-cilium

chart. I can probably apply the same patch and add the changes in the upstream Helm chart, but how can i tell Rancher to now fetch the

rke2-cilium

from my private chart repo?

Hi, I have a rke2 cluster (rancher run by docker 20.10) and I've just setup a harbor registry. But when I try to pull image from harbor it said x509: certificate signed by unknown authority. How can I configure to allow my cluster pull from insecure registry ? Can anyone help me with this ?

Hello Everyone. I have just joined this community and this channel. Thanks in advance for any help anyone can offer. I started trying out rancher recently and deployed the rancher server and 1-node quickstart cluster on Digital Ocean (DO) which went fine. They are currently using self-signed certs I am trying to use the same rancher server to now create a multi node rke2 cluster on DO. OS is ubuntu 22.04, which I checked is in the supported matrix: Provisioner: RKE2 Machine Provider: DigitalOcean Kubernetes Version: v1.26.8+rke2r1 Machine Pools: 2 Machines: 10 First pool has 3 etcd/control plane/worker nodes and the second pool has 7 worker nodes. The issue I am seeing is that once the nodes are created on DO, the state of the first node in the etcd pool goes into Reconciling and gets stuck at "waiting for probes: calico". Anyone faced such an issue and found a solution. Below is the full Provisioning log, which does not go beyond this: [INFO ] waiting for infrastructure ready [INFO ] waiting for at least one control plane, etcd, and worker node to be registered [INFO ] waiting for viable init node [INFO ] configuring bootstrap node(s) pool1-774695d878-q58pf: waiting for agent to check in and apply initial plan [INFO ] configuring bootstrap node(s) pool1-774695d878-q58pf: waiting for probes: calico, etcd, kube-apiserver, kube-controller-manager, kube-scheduler, kubelet [INFO ] configuring bootstrap node(s) pool1-774695d878-q58pf: waiting for probes: calico, etcd, kube-apiserver, kube-controller-manager, kube-scheduler [INFO ] configuring bootstrap node(s) pool1-774695d878-q58pf: waiting for probes: calico, kube-apiserver, kube-controller-manager, kube-scheduler [INFO ] configuring bootstrap node(s) pool1-774695d878-q58pf: waiting for probes: calico

Hello, I am trying to debug and understand for some quite time, what is going on in the background with

rke2-ingress-nginx-controller

I have in my 2 node RKE2 cluster. I think that this behaviour started after I installed second node, I have not experienced this with only single RKE2 node. The issue is, that the

Ingress

resources are continuously refreshing... I was looking into the log of the

rke2-ingress-nginx-controller

and I see there many records like:

I0927 19:15:06.246738       7 event.go:285] Event(v1.ObjectReference{Kind:"Ingress", Namespace:"vault", Name:"vault", UID:"5c2ba786-f9ef-4bfe-8449-2c9cdda1ed29", APIVersion:"<http://networking.k8s.io/v1|networking.k8s.io/v1>", ResourceVersion:"45134625", FieldPath:""}): type: 'Normal' reason: 'Sync' Scheduled for sync
I0927 19:16:05.890433       7 event.go:285] Event(v1.ObjectReference{Kind:"Ingress", Namespace:"vault", Name:"vault", UID:"5c2ba786-f9ef-4bfe-8449-2c9cdda1ed29", APIVersion:"<http://networking.k8s.io/v1|networking.k8s.io/v1>", ResourceVersion:"45135047", FieldPath:""}): type: 'Normal' reason: 'Sync' Scheduled for sync
I0927 19:17:06.437723       7 event.go:285] Event(v1.ObjectReference{Kind:"Ingress", Namespace:"vault", Name:"vault", UID:"5c2ba786-f9ef-4bfe-8449-2c9cdda1ed29", APIVersion:"<http://networking.k8s.io/v1|networking.k8s.io/v1>", ResourceVersion:"45135508", FieldPath:""}): type: 'Normal' reason: 'Sync' Scheduled for sync
I0927 19:18:05.886983       7 event.go:285] Event(v1.ObjectReference{Kind:"Ingress", Namespace:"vault", Name:"vault", UID:"5c2ba786-f9ef-4bfe-8449-2c9cdda1ed29", APIVersion:"<http://networking.k8s.io/v1|networking.k8s.io/v1>", ResourceVersion:"45135925", FieldPath:""}): type: 'Normal' reason: 'Sync' Scheduled for sync
I0927 19:19:06.435988       7 event.go:285] Event(v1.ObjectReference{Kind:"Ingress", Namespace:"vault", Name:"vault", UID:"5c2ba786-f9ef-4bfe-8449-2c9cdda1ed29", APIVersion:"<http://networking.k8s.io/v1|networking.k8s.io/v1>", ResourceVersion:"45136387", FieldPath:""}): type: 'Normal' reason: 'Sync' Scheduled for sync
I0927 19:20:07.327070       7 event.go:285] Event(v1.ObjectReference{Kind:"Ingress", Namespace:"vault", Name:"vault", UID:"5c2ba786-f9ef-4bfe-8449-2c9cdda1ed29", APIVersion:"<http://networking.k8s.io/v1|networking.k8s.io/v1>", ResourceVersion:"45136847", FieldPath:""}): type: 'Normal' reason: 'Sync' Scheduled for sync
I0927 19:21:06.764093       7 event.go:285] Event(v1.ObjectReference{Kind:"Ingress", Namespace:"vault", Name:"vault", UID:"5c2ba786-f9ef-4bfe-8449-2c9cdda1ed29", APIVersion:"<http://networking.k8s.io/v1|networking.k8s.io/v1>", ResourceVersion:"45137287", FieldPath:""}): type: 'Normal' reason: 'Sync' Scheduled for sync
I0927 19:22:07.334879       7 event.go:285] Event(v1.ObjectReference{Kind:"Ingress", Namespace:"vault", Name:"vault", UID:"5c2ba786-f9ef-4bfe-8449-2c9cdda1ed29", APIVersion:"<http://networking.k8s.io/v1|networking.k8s.io/v1>", ResourceVersion:"45137728", FieldPath:""}): type: 'Normal' reason: 'Sync' Scheduled for sync

Working with the ArgoCD, application are literally always Progressing, because of this “Scheduled for sync”. Do you have experience with similar behaviour? I am curious how to debug and resolve this issue. Request coming to

Ingress

resources are slow because of this scheduled sync issue. There are for example 309 count of such events on single

Ingress

resource over short period of time. This is an example of

Ingress

:

apiVersion: <http://networking.k8s.io/v1|networking.k8s.io/v1>
kind: Ingress
metadata:
  annotations:
    <http://cert-manager.io/cluster-issuer|cert-manager.io/cluster-issuer>: letsencrypt
  labels:
    <http://app.kubernetes.io/instance|app.kubernetes.io/instance>: vault
    <http://app.kubernetes.io/managed-by|app.kubernetes.io/managed-by>: Helm
    <http://app.kubernetes.io/name|app.kubernetes.io/name>: vault
    <http://argocd.argoproj.io/instance|argocd.argoproj.io/instance>: vault
    <http://helm.sh/chart|helm.sh/chart>: vault-0.25.0
  name: vault
  namespace: vault
spec:
  ingressClassName: nginx
  rules:
    - host: <http://vault.my.online|vault.my.online>
      http:
        paths:
          - backend:
              service:
                name: vault
                port:
                  number: 8200
            path: /
            pathType: Prefix
  tls:
    - hosts:
        - <http://vault.my.online|vault.my.online>
      secretName: vault-ingress-tls

Hi All, I am running rke2 with rancher version 2.6.8. Under

cattle-logging-system

listing over 300

completed

pods like

rancher-logging-root-fluentd-configcheck-45909352

in completed state. For some reason pods are not disappearing. I do not see any option such

revisionHistoryLimit

to set as value in helm chart in order to remove completed pods automatically. Is there any reason why those pods are still there? Any proper solution for this? Thanks

@creamy-pencil-82913 could you please give any input or pointer on the issue I reported above regarding rke2 cluster deployment on digital ocean getting stuck at "waiting for probes: calico"

Has anyone ever had success adding RKE2 "worker" nodes to a cluster using AWS ASG? If so.....What did you use for the "User Data"