Do you have more info of the cluster? how many nodes there are?
q
quaint-alarm-7893
09/20/2023, 1:29 PM
there are 4 nodes, 3 masters. one worker.
quaint-alarm-7893
09/20/2023, 1:31 PM
the one that got isolated looks like it's faling to restore etcd for the embeded rke2 cluster. that caused a bunch of volumes to fault. when i tried to salvage them about 10 of them wouldnt salvage. after messing with it for a while, i ultimately deleted the replica on the failed node, and then i was able to salvage it.
quaint-alarm-7893
09/20/2023, 1:32 PM
at this point i can either a) remove the node, and the reprovision it, or b) fix the failed node. if you want to make a run at fixing it for documentation purposes, lmk, otherwise i'll nuke it and reinstall it later today.