hi, we have a few clusters within rancher, and 2 of them in the status "Updating", 2 of 3 master nodes remains in the state "Waiting for probes: kube-controller-manager, kube-scheduler" and i have no idea where to look since all services are up and running in those clusters. Rechecked scheduler and controller-manager for errors but there is nothing. Please help.

generally thats a network connectivity issue / port blocked issue

could you please explain what port and src and dst? unfortunately the output is not very informative

• From Rancher UI, in

local

cluster, I would look at the logs of rancher pods in cattle-system namespace • Ports to open are listed in https://ranchermanager.docs.rancher.com/getting-started/installation-and-upgrade/installation-requirements/port-requirements • If you have access to the downstream clusters (SSH on the nodes), I would look at the logs of cattle-cluster-agent in

cattle-system

namespace

rancher pods logs

There is a process running on the downstream nodes called rancher-system-agent. It is configured to check the probes you’ve mentioned and report their status back to Rancher via a secret. You can check the downstream nodes to ensure that process is running and check its logs.

rancher-system-agent.service does not have any errors, restarting it does not help as well