This message was deleted.

IMHO get an external storage device.

There servers are relatively new and full of high grade disks , they used to run vSAN . We don't have external SAN to use unfortunately.

I've used the recommendations from this guide: https://harvesterhci.io/kb/best_practices_for_optimizing_longhorn_disk_performance In my case, I have three Dell R740XD2's with a mix of SSDs and 10K RPM spinning disks. I've created a few storage classes using disk tags. One is my "performance" tier which is only SSDs with three replicas (used for boot/root drives), another "big and slow" tier which is the spinning drives with two replicas, and the default provided tier (all disks with three replicas). I also make use of

data-locality

to try to keep VMs close to their storage. This, along with a performant SSD/NVMe tier, is essential for running Kubernetes clusters with etcd on top of Harvester. Also, if you are NVMe-heavy (my setup is not), you might look into enabling the Longhorn v2 engine.

Andy, i've been following your Youtube channel on this because I am in the same boat with op. I am assuming with the external storage device you mention this is still connecting to longhorn, probably with a replica of 1 to take advantage of the storage features on external storage? For myself I have a bunch of internal enterprise ssd's spread across my nodes and Longhorn backing images and creating VM's from them is one hurdle after the other since they don't aggregate into a real storage pool. I will say I see Longhorns way of doing storage, and now I see your why recommend the external storage.

in all fairness, I love longhorn. Just not with harvester. There have been too many performance/network issues when using it at scale. I do recommend longhorn for smaller apps that require less data size. It is great for within the VMs. Just not backing them at scale. And to be fair. I run it on single node harvester deployments with zero issues.

imo you need to define how many IOPS you need + other requirements and work backwards. "high performance" and longhorn (v1) don't really mix. I feel like most really high perf use cases shouldn't require generic network replicated block storage, but ymmv

Agreed. I use longhorn on single notes. But when you start scaling to multiple nodes and dozens and dozens of VM’s. That’s where I think you’re running into trouble. And should use an external storage device.

We are currently don't have external storage, I am using inherited servers running VMware vSAN on standard SSDs. no NVMe support, we are unable to upgrade to Longhorn Data Engine V2. I have 10 hosts, each equipped with 8 SSDs, intended to support 400 Kubernetes-based VMs. However, we are experiencing suboptimal performance, specifically regarding high latency. I did most optimizations I can find.

I don’t think your story is unique. I think a lot of people are having the exact same problem.

Awesome, we appreciate the insight guys.

Follow-up question: If I can dedicate two servers to provide external storage, what is the best open-source storage solution for Kubernetes that can run on those remote servers and be consumed by downstream Kubernetes clusters running on Harvester?

You kind of have 2 choices. NFS or iSCSI. I have tested https://github.com/kubernetes-csi/csi-driver-nfs with harvester in the past. It works. You could take one of those servers and use truenas. I would make sure you have as much bandwidth as possible.

We've got 30 K8s clusters that we're gradually migrating over to Harvester. A few of them are high-priority and need top-tier performance and 100% uptime. I’m looking for an open-source CSI plugin that has a really active dev community behind it.

directpv, topolvm, pci passthrough of nvme devices ...