Anyone run into a similar issue ```Internal error occurred f rancher-users #general

Title

clever-salesmen-35792

03/03/2023, 7:33 PM

Anyone run into a similar issue?

Internal error occurred: failed calling webhook "rancher.cattle.io.secrets": failed to call webhook: Post "<https://rancher-webhook.cattle-system.svc:443/v1/webhook/mutation/secrets?timeout=10s>": context deadline exceeded

For context, our rancher cluster is running on 1.25 (due to an accidental upgrade), which forced us to upgrade to the latest version of rancher and use the RC version. Now some clusters are working, but there are 2 or 3 that are having the above error.

We solved this... Seemed to be firewall issues blocking pot

which is weird because it has been working for a long time....

(and we haven't changed firewall rules)

bitter-breakfast-14780

03/06/2023, 2:13 PM

@clever-salesmen-35792 we're having the same issue after upgrading to Ubuntu 20.02 - where did you need to make the firewall change and how did you notice the block?

clever-salesmen-35792

03/06/2023, 5:19 PM

So running this in all of our clusters helped: https://suse.com/support/kb/doc/?id=000020699 It was actually changed to the following (found from rancher github)

kubectl delete secret -n cattle-system cattle-webhook-tls
kubectl delete <http://mutatingwebhookconfigurations.admissionregistration.k8s.io|mutatingwebhookconfigurations.admissionregistration.k8s.io> --ignore-not-found=true <http://rancher.cattle.io|rancher.cattle.io>
kubectl delete pod -n cattle-system -l app=rancher-webhook

The firewall changes were needed for our private clusters in GKE.

bitter-breakfast-14780

03/06/2023, 6:11 PM

cool i will check that out thank you

so my issue was because i had a dummy registries.yaml that i was loading that k3s was trying to read. removing it solved my issue.

84 Views

#general Join Slack