https://rancher.com/ logo
Title
b

busy-flag-55906

03/23/2023, 10:46 AM
hi, we have a few clusters within rancher, and 2 of them in the status "Updating", 2 of 3 master nodes remains in the state "Waiting for probes: kube-controller-manager, kube-scheduler" and i have no idea where to look since all services are up and running in those clusters. Rechecked scheduler and controller-manager for errors but there is nothing. Please help.
p

polite-piano-74233

03/23/2023, 1:36 PM
generally thats a network connectivity issue / port blocked issue
b

busy-flag-55906

03/23/2023, 3:31 PM
could you please explain what port and src and dst? unfortunately the output is not very informative
c

careful-mouse-42236

03/27/2023, 9:33 AM
• From Rancher UI, in
local
cluster, I would look at the logs of rancher pods in cattle-system namespace • Ports to open are listed in https://ranchermanager.docs.rancher.com/getting-started/installation-and-upgrade/installation-requirements/port-requirements • If you have access to the downstream clusters (SSH on the nodes), I would look at the logs of cattle-cluster-agent in
cattle-system
namespace
b

busy-flag-55906

03/27/2023, 10:20 AM
rancher pods logs
Untitled
cattle-cluster-agent pod logs:
Untitled
rancher-webhook pod:
Untitled
2. all network access seems to be open
3. on the downstream cluster cattle-cluster-agent pod has no errors
b

big-hydrogen-97240

04/03/2023, 12:09 PM
There is a process running on the downstream nodes called rancher-system-agent. It is configured to check the probes you’ve mentioned and report their status back to Rancher via a secret. You can check the downstream nodes to ensure that process is running and check its logs.
b

busy-flag-55906

04/03/2023, 12:31 PM
rancher-system-agent.service does not have any errors, restarting it does not help as well