I see

Failed to get recorded learner progress from etcd: context deadline exceeded

on the master that restarted 🤔

From the logs, what I understand so far is that: • one master node went down for some reason, it took a few secs (maybe minutes but not that long) for the service to be back up • this caused the worker nodes to go in

NotReady

but they didn't "recover" event several hours after the master node being back. And anyway there were other master nodes perfectly fine.

I wonder if this can be related to etcd quorum or something like that, as we have 3 master nodes 🤔

Still not clear why it wouldn't go back to normal by itself

(For the context, I'm more a dev than an ops/sysadmin, just trying to help my teammates here but I might sound silly in some of my question, do not hesitate to state the obvious 🙂 )

this is something that the much-hyped control loop architecture is clearly not healing

perhaps nagios can detect and restart your agents, it can do things these days, that would be legit control loop 🙂

Especially if it happens regularly

Thanks for the help anyway!

I lost a control node once, was massive numbers of dead pods generated from core dns autoscaler malfunction, so check for dead pod accumulation. Had to rebuild that node. Well these were all in one rke1 nodes.