This message was deleted.

Since Harvester now supports external storage arrays, another (though not ideal) option would be to set up your storage nodes as a separate, non-Harvester/Rancher storage array (Ceph, FreeNAS, NFS, or possibly even Longhorn if you have enough CPU/RAM to support it). You would then set up your blade servers as a 3-node Harvester cluster and then attach the Harvester cluster to the new, external array using your CSI of choice. From there, you can spin up VMs on your 3-node Harvester cluster, but point their disk volumes to the external array. https://docs.harvesterhci.io/v1.7/advanced/csidriver

That seems pretty close to what I'm doing now, all 6 nodes are in the same Harvester cluster but the 3 storage nodes are tainted so no workloads run on them. I use LVM to bundle all the HDDs into one LV and all the SSDs into another then add that directory to Longhorn. What are the pros/cons of what I'm doing vs splitting the storage nodes into their own cluster? That's what I'm really struggling with on this project, I can find a log of documentation on what I can do but I'm not finding a lot on why I would want to do it one way vs another.

Because there's so many different ways of doing things, they mostly have a whole slew of things that boil down to pros and cons. Speaking from experience, we made choices early on that we didn't fully comprehend and only later discovered pros and cons because of issues in our environment that we had no control over, but ended up being important later. Some things to consider: • Are you paying for support? Will you ever pay for support? LH (unaltered) is really the only storage class that's fully supported. Suse will support the integrations working with Harvester, but not the CSI's themselves. Doing 3rd party storage means taking that into consideration. There's also the point that there's no such thing (pricing wise) as a non-workload node. There's Harvester nodes and you pay for longhorn, and you might have to pay extra for longhorn if you expose the Harvester LH storage to a downstream cluster. (node counts and all that jazz) Talking to an account rep might be important here. • Do you have support for or experience running storage clusters? We do pay for Suse support, and a lot of our highest impact/priority tickets were around the storage layer. That being said, my team also has a lot of experience running Ceph clusters, so doing upgrades/troubleshooting/configs were much less of an impact for that aspect. Your mileage may vary. You could also already have a product (Like PureStorage or something) that already has a CSI and would be easy to turnkey some credentials for. • How concerned are you about vendor lock in? Having all your eggs in the Harvester basket might make moving off the platform in the future tough. It's still kubevirt underneath, so with a third party storage, migrating disks/workloads to some $otherHypervisor would be easier. • We initially were thinking that we could have 1 giant cluster (or maybe 2 with a testing one too) and just throw all our hardware in it, but found that wasn't a great idea when it came to different hardware, migrations, and cost. Migration for things between clusters got easier when using third party storage, but your considerations might be different. • Lastly I'll point out that Harvester is intended to be an HCI solution. That doesn't mean that you can't use it in a different fasion, but when you start extending things outside those lines, you'll need to look at your own expertise, or potentially other vendors (or at least products if you want one throat to choke) the further out you go.

• We aren't planning on paying for support, although i'm ready to make the case of paying for a good consultant to get us off the ground. • We have very little experience with storage clusters, right now we are running off a glorified NAS, but that's all we really need • I'm not too concerned about vendor lock in. We are small enough that migrating to something new isn't a major ordeal • Good to know about the multi cluster. I naively though I was going to have one cluster to manage and I've realized it's going to be at least 3, one more wouldn't make a differrent • This is a good point, I'm more wanting Harvester because it's closely integrated with Rancher but if this isn't the right fit for us I'm open to other options.

I think there's a whole slew of node drivers that you could use as an easy turnkey solution: