I have finally added a third node to my Harvester v1.0.0 cluster. (I started with node 003, then added 002, and finally added 001 today). I thought that after adding the third node the cluster would turn into an HA cluster where all three nodes are the control plane with etcd, but that has not happened. Have I done something wrong?

$ kubectl get nodes -o wide --context harvester003
NAME           STATUS   ROLES                       AGE   VERSION          INTERNAL-IP   EXTERNAL-IP   OS-IMAGE           KERNEL-VERSION         CONTAINER-RUNTIME
harvester001   Ready    <none>                      29m   v1.21.7+rke2r1   10.64.0.16    <none>        Harvester v1.0.0   5.3.18-59.37-default   <containerd://1.4.12-k3s1>
harvester002   Ready    <none>                      63d   v1.21.7+rke2r1   10.64.0.17    <none>        Harvester v1.0.0   5.3.18-59.37-default   <containerd://1.4.12-k3s1>
harvester003   Ready    control-plane,etcd,master   84d   v1.21.7+rke2r1   10.64.0.18    <none>        Harvester v1.0.0   5.3.18-59.37-default   <containerd://1.4.12-k3s1>

Last time I read the docs they said that HA was automatically done once the cluster had 4 or more nodes (the first 2 nodes would be the masters).

Oh, so I have to spend another thousand pounds to make a 4 node cluster just to make the first three nodes HA. No. There's something wrong in the logic there. We have to be able to make a 3 node HA cluster.

Don't shoot the messenger 🤣

Indeed. Thanks for re-reading the docs I have read many times.

I think the idea is to avoid running all the control plane services on smaller clusters to avoid over-allocation of resources, but it would definitely be nice to allow operators to adjust that trade-off for themselves by providing a documented mechanism for promoting a node manually.

on 0.3.0, i definitely had HA after adding a 3rd node to the cluster. did that behavior change in 1.0.0?

did that behavior change in 1.0.0?

I think it might have. I think I read and remembered that I needed 3 nodes, so I planned my finances to be able to achieve that. For a minimal HA home-lab I see no point in requiring more than 3 nodes.

@sticky-summer-13450 this should not be the case. Can you point to where you read this? Harvester will do a 3 node HA cluster with the first 3 nodes and the additional nodes will be considered workers-only. If something changed/happened, it would be a bug.

@happy-cat-90847 the first paragraph on https://docs.harvesterhci.io/v1.0/host/host/ says

When there are more than three nodes, the two other nodes that first joined are automatically promoted to management nodes to form a HA cluster.

As I said at the top of this thread - I have a 3 node Harvester v1.0.0 cluster which has is a master and two workers.

$ kubectl get nodes
NAME           STATUS   ROLES                       AGE   VERSION
harvester001   Ready    <none>                      8d    v1.21.7+rke2r1
harvester002   Ready    <none>                      71d   v1.21.7+rke2r1
harvester003   Ready    control-plane,etcd,master   93d   v1.21.7+rke2r1

When I added my 3rd node, it made it HA (I have one cordoned which is why vm01 says SchedulingDisabled)

Did you build with Harvester 1.0.1 or have you upgraded to it?

ahh, it was all built with fresh 1.0.1

Humm - mine was built with 1.0.0 and I have not upgraded yet because I'm a little bit worried, especially with the cluster not being HA...

Have you seen this: https://github.com/harvester/harvester/issues/2194

@adventurous-engine-93026 yes, I created it and linked to it above. Why?

I thought it looked similar to your situation. I couldn't tell it was you though. 🙂

So - seeing as I am in the impossible situation that I have a three node Harvester cluster which has not automatically converted itself into an HA cluster, is there a way I can get it to happen manually - preferably without losing all the workloads? 🙂

As per the note at the top of the docs page "Because Harvester is built on top of Kubernetes and uses etcd as its database, the maximum node fault toleration is one when there are three management nodes." my interpretation is with 3 nodes only 1 will be management and only when you have 4 do 2 more get promoted. Yet @happy-cat-90847 suggests in your bug that he's seeing all 3 mgmt with 3 nodes. I've not yet added my 3rd node, don't want to break my cluster before I've recorded SUSECon session

Yeh - anecdotally, the documentation is wrong. I'd love to get some traction from Suse representatives to confirm or deny that.

There is no need to get caught up on this. If we have 3 nodes, we provide an HA stack. Any further after 3 is only added as a worker node. I don't know what happens when only two are added. I should try it. Any scenario where you have 3 nodes and they are not all cp/etcd/worker it's a bug. If the docs say otherwise, please file a bug.

Thank you - I'm creating a documentation ticket now.

Thank you!

Right - like I said before, you may have hit a bug from a previous version. Did you start from zero with 1.0? Or was a node with a previous version?

Sorry - I don't mean to sound grumpy. It's been a long day, and Harvester is so close to being brilliant!

@sticky-summer-13450 Need your help to dump one more thing. Could you get

kubectl get secrets -n fleet-local -o yaml

and send the output to me privately? Slack is OK. This can help me understand why some Rancher plans are not applied on harvester002 node. (It’s another thing not included in the support bundle. Because the resource name is “secret”) Thanks a lot.

I assume I can do that from anywhere that can access the cluster...