Hello, i had performance problem with my cluster, specifically with control planes nodes which i have 3 and had to change their volumes for ssd, i removed them by 1 and ran rke up to for every node upgraded. After finishing upgrading seems like every mutating and validating webhooks seems to fail, i see failing webhook logs from kube-apiserver. For example, below is related to logging to rancher GUI:

Failed calling webhook, failing open rancher.cattle.io.clusters.management.cattle.io: failed calling webhook "rancher.cattle.io.clusters.management.cattle.io": failed to call webhook: Post "<https://rancher-webhook.cattle-system.svc:443/v1/webhook/validation/clusters.management.cattle.io?timeout=10s>": dial tcp 10.43.11.133:443: connect: connection refused

W1029 12:24:45.130046       1 dispatcher.go:217] Failed calling webhook, failing closed rancher.cattle.io.userattributes.management.cattle.io: failed calling webhook "rancher.cattle.io.userattributes.management.cattle.io": failed to call webhook: Post "<https://rancher-webhook.cattle-system.svc:443/v1/webhook/validation/userattributes.management.cattle.io?timeout=10s>": dial tcp 10.43.11.133:443: connect: connection refused

kubectl get svc -n cattle-system
NAME              TYPE        CLUSTER-IP     EXTERNAL-IP   PORT(S)          AGE
rancher           ClusterIP   10.43.217.22   <none>        80/TCP,443/TCP   116d
rancher-webhook   ClusterIP   10.43.11.133   <none>        443/TCP          116d

Does anyone have any idea why, if i remove validatingwebhookconfiguration i can login to GUI.