Pods are running for the initially migrated VM's when this was a single node cluster, creating new VM's using a storage class with 3 replicas however results in the volumes being stuck attaching - with only the first node attaching and the VM doesn't get to go up as the volume isn't ready.

Just tried creating a VM with 1 replica - and that would not attach either - it looks like a problem with using anything other than the first node crated in the cluster.

Event-log for a volume being created for a VM with 1 replica, the node this is being created on was added to what was a single node harvester cluster, this now has 3 nodes - but only the first will attach a volume from the look of it.

Ahh - one thing I can see is that I only have one control-plane, as I've added nodes I was expetign these these get promoted as the number of nodes in-creases... Do I actually need to setup as multi-node to make this work, can a single-node cluster be upgraded to multi-node? Hoping I don't need to remove the nodes and start over as a multi-node off the bat and transfer again...

I could not see anything here https://docs.harvesterhci.io/v1.5/advanced/singlenodeclusters/ that says this is possible or not possible

You may be running into an issue with the Longhorn network existing before the new nodes. 🫣 - I haven’t tested that. You could revert back the dedicated Longhorn network change or try to debug it a little more. Have you rebooted the nodes too?

But id say, look at the logs to try determine what is going on. Likely, networking related.

Rebooted the first node, and all the longhorn volumes are now stuck in progress, including those that were single replica on the same node.

I've removed the multi-replica tests while attempting to get the original VM's back up and running.

It wont let me change the storage network because - > admission webhook "validator.harvesterhci.io" denied the request: Please stop all VMs before configuring the storage-network setting But the machines are stopped, I think some are stuck in a start loop ... Figured it out - I need to tell the machines to schedule to the original single node - then they come up! So that explains that... just getting the VM's back up. Will see if once all nodes have had a power cycle things change, if that hasn't worked I'll shut down the VM's and switch to the default storage network to test that as being the issue. Presently only one node (the first) is showing as a control-plane

% kubectl get nodes
NAME   STATUS   ROLES                       AGE     VERSION
n2     Ready    <none>                      3h24m   v1.32.7+rke2r1
n3     Ready    <none>                      4h28m   v1.32.7+rke2r1
n4     Ready    control-plane,etcd,master   4d2h    v1.32.7+rke2r1

After rebooting the three nodes -

% kubectl get nodes
NAME   STATUS   ROLES                       AGE     VERSION
n2     Ready    <none>                      3h40m   v1.32.7+rke2r1
n3     Ready    <none>                      4h43m   v1.32.7+rke2r1
n4     Ready    control-plane,etcd,master   4d2h    v1.32.7+rke2r1

So no promotions yet... just added another node, and thats just joing showing with no roles too... Next will try using default storage network.

@happy-cat-90847 how would the storage network have any impact on Harvester promoting nodes to control-plane? I cant see any way to promote a worker, still digging into the docs - I cant shut the VM's down for a couple of hours as they are in use, but will try the storage network suggestions a bit later.

Happy to help debug this issue, I can open on Github if needed, but I do need a quick resolution - so if this sounds like a problematic edge case then please do let me know as I'll re-install as multi-node in parallel and migrate if thats the case.

I'm setting up a fresh in parallel - taking 2 nodes out of this stack and re-installing fresh to test - see if the single node setup was the issue. If it goes up fine I'll raise an issue on github with some info on this promotion issue. I did configure the storage network early when the single node first came up - its possible that this causes some trouble with promotion of added nodes if the storage isn't working properly. Unfortunately I'm in a bit of a rush to get this sorted now.

fresh setup as multi-node works fine so I reckon there is a bug with promotion of a single node stack to multiple.

Hit different problem this time - but I think it may have explained the issue here, the issue may have been the storage network config after all - should have tried defaults, to see, suspect that to make it HA it would have wanted replicas for the storage and that it may have failed early in its promotion. I think my problem lies with the VLAN config.

The mistake was with VLAN, which if misconfigured will go un-noticed in a single node since it doesn't matter! In my case it was ultimately the wrong ID, I'd expect promotion would work if the VLAN's were setup correctly - or simply don't set them up in single mode node since its not needed - there really is no 'network' to isolate... and you can make changes to this setting with your VM's powered down later!