04/04/2023, 10:54 AM
Hi, Looking for some support, if this is the wrong channel do let me know I have recently updated my Rancher deployment that has been running fine for 310 days I tried using K8S1.23.6/rke2r2 and rancher 1.27.1 I got an error that said the webhooks were not ready on 443 and the cluster was unable to be managed I have now downgraded the rancher version to 1.26.head and the cluster is now manageable, but I cannot add nodes to the cluster i get this error:
curl --insecure -fL https://<IP>/ | sudo  sh -s - --server https://<IP>--label '<|>' --token <token>--ca-checksum <checksum>--etcd --controlplane --node-name <nodeName>
  % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
                                 Dload  Upload   Total   Spent    Left  Speed
100 29790    0 29790    0     0   334k      0 --:--:-- --:--:-- --:--:--  338k
[INFO]  Label: <|>
[INFO]  Role requested: etcd
[INFO]  Role requested: controlplane
[INFO]  Using default agent configuration directory /etc/rancher/agent
[INFO]  Using default agent var directory /var/lib/rancher/agent
[INFO]  Determined CA is necessary to connect to Rancher
[INFO]  Successfully downloaded CA certificate
[INFO]  Value from https://<IP>/cacerts is an x509 certificate
[INFO]  Successfully tested Rancher connection
[INFO]  Downloading rancher-system-agent binary from https://<IP>/assets/rancher-system-agent-amd64
[INFO]  Successfully downloaded the rancher-system-agent binary.
[INFO]  Downloading script from https://<IP>/assets/
[INFO]  Successfully downloaded the script.
[INFO]  Generating Cattle ID
[INFO]  Cattle ID was already detected as <ID>. Not generating a new one.
[ERROR]  500 received while downloading Rancher connection information. Sleeping for 5 seconds and trying again
Rancher gives this error when trying to do anything from Cluster Management
Internal error occurred: failed calling webhook "<|>": failed to call webhook: Post "<https://rancher-webhook.cattle-system.svc:443/v1/webhook/validation?timeout=10s>": no endpoints available for service "rancher-webhook"


04/08/2023, 7:50 AM
The rancher-webhook is present. It is possible that the certificate has expired so delete the secret and restart (kill) the rancher webhook pod


04/18/2023, 8:18 AM
I solved the issue by stopping the pod and starting it with a different version of the rancher image. It seemed the specific version (i cannot remember which) had issues