https://rancher.com/ logo
Join Slack
Powered by
Issue after Rancher UCM / downstream K8s upgrade –...
# rke2
q
Issue after Rancher UCM / downstream K8s upgrade – Calico pods disappearing Hi Team, looking for some guidance on a recurring issue we're seeing after upgrading: • Rancher UCM: 2.12 → 2.14.3 • Downstream RKE2/K8s: 1.33.9 → 1.35.6 Since the upgrade, we have randomly seen an issue during RKE2 Helm reconciliation/reinstallation (which can happen during normal RKE2 operations). Following this, the Calico pods disappear and CNI becomes unavailable. When checking the Tigera Installation resource, we see:
Copy code
Error creating / updating namespaces:
admission webhook "rancher.cattle.io.namespaces.create-non-kubesystem"
denied the request: Unauthorized
This prevents the Tigera operator from creating/updating the required Calico resources, leaving the cluster without Calico. Current workaround: 1. Temporarily patch/remove the affected Rancher admission webhook.
Copy code
kubectl patch validatingwebhookconfiguration rancher.cattle.io \                                                                                                                                                            
  --type='json' \
  -p='[{"op":"remove","path":"/webhooks/5"}]'
2. Restart the Tigera operator: 3. kubectl rollout restart deployment/tigera-operator -n tigera-operator 4. Calico resources/pods are then recreated and the cluster recovers. Rancher webhook 
Unauthorized
+ Tigera/Calico disappearance after upgrading to Rancher 2.14.3 / K8s 1.35.x? Any suggestions please
h
did you upgrade directly from 1.33 to 1.35 ? or in steps 1.33 to 1.34 to 1.35
q
it was in steps.. 1.33 -> 1.34 -> 1.35
ucm also from 2.12.3 to 2.13.3 to 2.14.3 And issue even for a new cluster which is created directly with 1.35
@hundreds-evening-84071 @creamy-pencil-82913 - Any thoughts on this issue please? am i missing any RBAC? so far all were working properly. After upgrade seeing this issue very frequently
c
Rancher issue, not rke2. Have you searched for an issue? If you can't find one (also check for closed issues) create one and I can ping rancher folks about it.
q
Sure
unlike this channel , neither of the rancher queries i get responses. lets see this time 🙂
@creamy-pencil-82913 - Instead of removing the validation webhook, was able to fix the error by applying below RBAC on one of the downstream cluster, post which tigera cni installation auto resolved the error and proceeded with creating calico namespaces and calico pods came up. RBAC
Copy code
apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRole
metadata:
  name: tigera-operator-psa-fix
rules:
  - apiGroups:
      - management.cattle.io
    resources:
      - projects
    verbs:
      - updatepsa
---
apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRoleBinding
metadata:
  name: tigera-operator-psa-fix
roleRef:
  apiGroup: rbac.authorization.k8s.io
  kind: ClusterRole
  name: tigera-operator-psa-fix
subjects:
  - kind: ServiceAccount
    name: tigera-operator
    namespace: tigera-operator
When we are in rancher ucm 2.12 we never seen this, as per attached rancher bug - looks this has started from 2.13/2.14 and fixed in 2.15 UCM version - github.com/rancher/rancher/issues/55657 Few questions, please clarify : We’re seeing rke2-calico + all rke2 managed Helm charts across all downstream clusters at approximately 5-day intervals, sometimes with chart version changes (history attached). • Is this expected behavior? Does RKE2 periodically check for chart updates and reconcile them? • Can this reconciliation be controlled or disabled? Sometimes reconciliations appear to fail, and with 
failurePolicy: reinstall
, Calico gets uninstalled and reinstalled. Rancher admission/RBAC error then prevents Calico pods from coming up, causing a CNI outage. please clarify what triggeres these repeated reconciliation? possible to stop it?
.
@creamy-pencil-82913 - Any thoughts on this please on how we can stop rke2 to keep doing this reconciliation? we are hitting a prod issue due to the periodic reconciliation hence seeking your inputs
b
Just a "me too" post: I also saw this today with Rancher 2.14.5 and a managed RKE2 1.33.13 when I changed the OS image of all nodes which caused all nodes to be recreated. I removed the rancher webhook temporarily to fix it. So, imho a bug of rancher webhook not allowing the calico operator to deploy things.
There is github.com/projectcalico/calico/issues/9807 and github.com/projectcalico/calico/pull/14097 . Looks like Calico fixed the issue themselves because there was no action on the Rancher side. From the comments I'd say that this issues is fixed with Calico >=3.32. Not sure which RKE2 version is delivering this version already.
q
yeaah thats true.. my only question would be, is there a way to stop rancher keep doing the helm reconciliation? During a freeze period when there's a need to have no actions being carried onto a prod clusters - above helm reconciliation tends to make disruptive actions. @creamy-pencil-82913 sir - any thoughts on this please
c
what is helm even reconciling? if there are no chart changes, and no value, changes, what is even changing?
I don’t see what the helm controller has to do with this
q
yes same puzzle , not sure why.. i have highlighted revision values, eventhough when there's no changes still it keeps reconciling ❯ helm history rke2-calico -n kube-system REVISION UPDATED STATUS CHART APP VERSION DESCRIPTION 142 Thu Oct 1 001107 2026 superseded rke2-calico-v3.31.300 v3.31.3 Upgrade complete 143 Thu Oct 1 001344 2026 superseded rke2-calico-v3.31.300 v3.31.3 Upgrade complete 144 Mon Oct 5 213416 2026 superseded rke2-calico-v3.31.300 v3.31.3 Upgrade complete 145 Mon Oct 5 214120 2026 superseded rke2-calico-v3.32.100 v3.32.1 Upgrade complete 146 Mon Oct 5 221326 2026 superseded rke2-calico-v3.32.100 v3.32.1 Upgrade complete 147 Mon Oct 5 221503 2026 superseded rke2-calico-v3.32.003 v3.32.0 Upgrade complete 148 Sat Oct 10 221550 2026 superseded rke2-calico-v3.32.003 v3.32.0 Upgrade complete 149 Thu Oct 15 221650 2026 superseded rke2-calico-v3.32.003 v3.32.0 Upgrade complete 150 Tue Oct 20 221750 2026 superseded rke2-calico-v3.32.003 v3.32.0 Upgrade complete 151 Thu Oct 22 200737 2026 deployed rke2-calico-v3.32.003 v3.32.0 Upgrade complete ❯ ❯ helm get values rke2-calico-v3.32.003 -n kube-system --revision 147 Error: release: not found ❯ ❯ ❯ helm get values rke2-calico -n kube-system --revision 147 USER-SUPPLIED VALUES: global: cattle: systemDefaultRegistry: e-car-onprem-docker-virtual.docker clusterCIDR: 192.168.0.0/17 clusterCIDRv4: 192.168.0.0/17 clusterDNS: 192.168.128.10 clusterDomain: cluster.local prime: enabled: false rke2DataDir: /var/lib/rancher/rke2 serviceCIDR: 192.168.128.0/17 systemDefaultIngressClass: ingress-nginx systemDefaultRegistry: e-car-onprem-docker-virtual.docker installation: calicoNetwork: mtu: 1450 ❯ ❯ helm get values rke2-calico -n kube-system --revision 148 USER-SUPPLIED VALUES: global: cattle: systemDefaultRegistry: e-car-onprem-docker-virtual.docker clusterCIDR: 192.168.0.0/17 clusterCIDRv4: 192.168.0.0/17 clusterDNS: 192.168.128.10 clusterDomain: cluster.local prime: enabled: false rke2DataDir: /var/lib/rancher/rke2 serviceCIDR: 192.168.128.0/17 systemDefaultIngressClass: ingress-nginx systemDefaultRegistry: e-car-onprem-docker-virtual.docker installation: calicoNetwork: mtu: 1450 ❯ helm get values rke2-calico -n kube-system --revision 149 USER-SUPPLIED VALUES: global: cattle: systemDefaultRegistry: e-car-onprem-docker-virtual.docker clusterCIDR: 192.168.0.0/17 clusterCIDRv4: 192.168.0.0/17 clusterDNS: 192.168.128.10 clusterDomain: cluster.local prime: enabled: false rke2DataDir: /var/lib/rancher/rke2 serviceCIDR: 192.168.128.0/17 systemDefaultIngressClass: ingress-nginx systemDefaultRegistry: e-car-onprem-docker-virtual.docker installation: calicoNetwork: mtu: 1450 ~/Downloads/kube 
c
All controllers periodically resync, for helm controller this means it will periodically reevaluate the chart and values hash against the job hash and rerun it if necessary. Even with that, helm will only create a new lease if something has changed in the chart or values. What's changing?
q
@creamy-pencil-82913 - Thanks again for your suggestions. We found differences between the existing Helm spec and the newly rendered spec, which is triggering reconciliation and eventually resulting in a failed state. One final question: You mentioned that all controllers periodically resync. Is the resync interval configurable somewhere in RKE2? I am specifically trying to understand why we're seeing this Helm reconciliation approximately every 5 days and where that interval is defined.
c
no, that is just how Kubernetes informers (things that watch for changes and run code on change) work. they periodically re-list resources and trigger an update as if the resource had changed, just to make sure that the current state equals the desired state. This defaults to about every ~10 hours. If you are seeing it happen on the order of days then something else is going on.
every controller that uses informers (and it is pretty much all of them) has a resync interval, and it is not directly configurable.
q
thanks for the confirmation, will check further on my env why it is behaving differently
4 Views
Open in Slack
PreviousNext
Powered by