https://rancher.com/ logo
Title
r

rhythmic-jelly-81455

03/02/2023, 11:56 AM
Hi, my cluster has been stuck in this state for a while. Any help would be greatly appreciated
I got here after trying to replace the 5th node
r

rough-farmer-49135

03/02/2023, 2:33 PM
Looks like your first master node is not finishing. Did you check the logs for the rancher-system-agent service as it suggests?
r

rhythmic-jelly-81455

03/02/2023, 2:53 PM
Yes, I checked. Nothing seems alarming there. Though I did see some warning logs for rke2-server on two of the nodes. It's strange since the etcd port is 2379 but somehow when rke2-server is starting listen-client-urls is being set as 2399
r

rough-farmer-49135

03/02/2023, 3:43 PM
You could check the rke2-server logs and also try
netstat -antp | grep LISTEN
to see what's listening on what port (which'll partially just show you what's up and what's not). Last thing I know for debugging is you can try using crictl ( https://gist.github.com/superseb/3b78f47989e0dbc1295486c186e944bf for cheat sheet ) & check on your static pods to see if things look ok there or not.