https://rancher.com/ logo
Title
b

busy-crowd-80458

06/14/2022, 7:28 AM
The primary issue seems to be that none of the machines can now find etcd 🥵
a

alert-potato-16010

06/14/2022, 9:43 AM
Etcd uses host IP addresses therefore you have changed all nodes IP. Not sure if you are going to be able to recover the etcd cluster... You can reconfigure the peers IP addresses (https://etcd.io/docs/v3.3/op-guide/runtime-configuration/ ), but as you don't have quorum, I am not sure if changes will be applied....
v

victorious-analyst-3332

06/14/2022, 1:52 PM
this might be your best bet if preserving as much data as possible is necessary https://docs.rke2.io/backup_restore/#cluster-reset
once you reset/restore from backup and have a single node cluster, you can follow the steps to clear data dirs on the remaining servers and join them again to the new primary
b

busy-crowd-80458

06/15/2022, 1:00 AM
Yeah, we wouldn’t have been so stupid with this cluster if it had anything valuable on it 🙂
this is more of a “hey, we can rebuild, but curious as to if we can avoid rebuilding” question
thanks!