Under what conditions will Harvester promote anoth...
# harvesters
strong-shoe-72392
05/14/2025, 3:17 PMUnder what conditions will Harvester promote another Default Role node to become a Management node (Etcd)? I'm trying to understand what would happen in a scenario where where we spread Management Nodes across two very closely located data closets with low-latency, high bandwidth communication and we lose one data closet (site)? For example, if Site A would go offline, would a Default Role node in Site B get promoted automatically to maintain minimum quorum for Harvester Etcd?
b
bland-article-62755
05/14/2025, 5:56 PMI'm pretty sure it'd only get promoted upon node removal.
bland-article-62755
05/14/2025, 5:56 PMI also don't think there's any way to "demote" a node.
bland-article-62755
05/14/2025, 5:57 PMSo if Site A came back online, you don't want split brain and an ~un~even number for the control plane.
bland-article-62755
05/14/2025, 5:58 PMerr even number
s
strong-shoe-72392
05/14/2025, 5:58 PMSo promotion is ONLY a tool for manual maintenance?
b
bland-article-62755
05/14/2025, 5:59 PMNo, it's still for HA.
bland-article-62755
05/14/2025, 5:59 PMIf your power supply dies or you have a switch go out or something etcd needs to keep running.
bland-article-62755
05/14/2025, 6:01 PMLet's say something bad happens to an etcd node and it's gonna be more than a day or two, you probably want to remove that node out of the cluster and have a new node get promoted to keep up the HA for etcd (and longhorn replicas etc)
bland-article-62755
05/14/2025, 6:02 PMBut Harvester manages that promotion, it's not something you can trigger (unless you are manually removing nodes intentionally)
bland-article-62755
05/14/2025, 6:03 PMIt'll promote the nodes in order they were added to the cluster iirc.
bland-article-62755
05/14/2025, 6:03 PMUnless you have them tagged as a management node, but even then I think it only does 3
s
strong-shoe-72392
05/14/2025, 6:24 PMRight. But since Etcd can only tolerate the loss of (n-1)/2 for quorum. In a scenario where we could potentially lose a whole site, how can we mitigate? Technically losing 3 is more than is allowed for 5 Etcd nodes as well.
strong-shoe-72392
05/14/2025, 6:25 PMThanks for your help and answers by the way. I don't have a lot of time with production/HA configuration of Harvester yet, but I do with RKE2. I know that losing 2 of 3 control plane nodes has caused us issue in the past.
b
bland-article-62755
05/14/2025, 7:08 PMIt depends on how it goes down.
bland-article-62755
05/14/2025, 7:09 PMif siteA loses power and you have one etcd node on B, you won't have qorrum, but etcd will keep running.
bland-article-62755
05/14/2025, 7:10 PMIf you have 2 in A and 1 in B
bland-article-62755
05/14/2025, 7:10 PMWorse would be if you lose connectivity but both sites stay up, and you end up with split brain.
bland-article-62755
05/14/2025, 7:11 PMThis is worth getting support with Suse over and just asking.
s
strong-shoe-72392
05/14/2025, 7:29 PMYep - support is in the works. We've had the same discussions with RKE2 installs. The interesting difference here was Harvester's ability to promote nodes to Managers. I was curious how that changed anything for it's underlying Etcd/HA.
strong-shoe-72392
05/14/2025, 7:29 PMThanks