https://rancher.com/ logo
Title
q

quaint-soccer-60531

11/22/2022, 12:46 PM
Hi. I accidentally terminated all nodes (masters and workers) of a cluster in Rancher. All but the last master went away. But I cannot get rid of that one, it is stuck there in Removing state, since some hours. How can I recover from that state? My plan was to start a new master and populate its
etcd
from a snapshot of that cluster. But new nodes won't come up and stay in Registering state, probably trying to join any other master nodes that don't exist anymore.
I see this error message in Ranchers log file:
2022/11/22 12:44:08 [ERROR] error syncing 'c-7l6gd/m-xbslc': handler node-controller: waiting for node to be removed from cluster, requeuing
I guess that
node-controller
doesn't run anymore and will never come back to remove a node.
I guess I am in #16859.