Hi, Looking for some support, if this is the wrong channel do let me know I have recently updated my Rancher deployment that has been running fine for 310 days I tried using K8S1.23.6/rke2r2 and rancher 1.27.1 I got an error that said the webhooks were not ready on 443 and the cluster was unable to be managed I have now downgraded the rancher version to 1.26.head and the cluster is now manageable, but I cannot add nodes to the cluster i get this error:

curl --insecure -fL https://<IP>/system-agent-install.sh | sudo  sh -s - --server https://<IP>--label '<http://cattle.io/os=linux|cattle.io/os=linux>' --token <token>--ca-checksum <checksum>--etcd --controlplane --node-name <nodeName>
  % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
                                 Dload  Upload   Total   Spent    Left  Speed
100 29790    0 29790    0     0   334k      0 --:--:-- --:--:-- --:--:--  338k
[INFO]  Label: <http://cattle.io/os=linux|cattle.io/os=linux>
[INFO]  Role requested: etcd
[INFO]  Role requested: controlplane
[INFO]  Using default agent configuration directory /etc/rancher/agent
[INFO]  Using default agent var directory /var/lib/rancher/agent
[INFO]  Determined CA is necessary to connect to Rancher
[INFO]  Successfully downloaded CA certificate
[INFO]  Value from https://<IP>/cacerts is an x509 certificate
[INFO]  Successfully tested Rancher connection
[INFO]  Downloading rancher-system-agent binary from https://<IP>/assets/rancher-system-agent-amd64
[INFO]  Successfully downloaded the rancher-system-agent binary.
[INFO]  Downloading rancher-system-agent-uninstall.sh script from https://<IP>/assets/system-agent-uninstall.sh
[INFO]  Successfully downloaded the rancher-system-agent-uninstall.sh script.
[INFO]  Generating Cattle ID
[INFO]  Cattle ID was already detected as <ID>. Not generating a new one.
[ERROR]  500 received while downloading Rancher connection information. Sleeping for 5 seconds and trying again

The rancher-webhook is present. It is possible that the certificate has expired so delete the secret and restart (kill) the rancher webhook pod

I solved the issue by stopping the pod and starting it with a different version of the rancher image. It seemed the specific version (i cannot remember which) had issues