This message was deleted.

now all is stuck in this state

are you able to ping? and do you have access to the console? If not probably reach out to your Linux admin team

stonith?

yeah it happened during upgrade to 1.35.4. I see now that the IP address changed on reboot. Do I need to install rke2 manually now and then rancher will get the update?

I think you should be able to just change the IP back to what it was before (reserve it and set it statically so it doesn't happen again)

all 3 cpntrolplanes hanging and cluster is not reachable anymore

Did you fix the IPs on the hosts? Otherwise you're likely looking at an etcd restore.

how did the IP change? do you not have static IPs on these nodes?

guess not. the machines are generated by rancher in vmware

So rancher is aware of VM's it provisioned and can get an IP from them. It's like that in Harvester as well. However, Kubernetes treats IPs as immutable. So if you have DHCP in your stack, you need to either provision those VMs with static IPs, or go back and make a DHCP reservation after they're provisioned. We have to do the same thing in our env.