This message was deleted.

we use it, together with our legacy virtualization platforms. It's stable enough, but obviously K8s experience is helpful to be able to troubleshoot and understand how it's working. I wouldn't say it's mature, many features are still in development, but good to see the project is growing, so likely around mid-2026 together with Longhorn v2 should be more suitable for general public. it depends what are your options? the choice is quite limited (unless you need to simply run VMs without HCI part), it's either Proxmox, OpenShift Virtualization, Oracle Virtualization, etc., all have their own issues.

^ agree. We have been using it in prod since 1.2.2. It works, is stable but we find it requires a fair amount of hand holding in many places and lacks user friendly descriptive errors messages and logging. More of our issues seem to be with longhorn compared to harvester. As already said, no solution is perfect and all come with their own issues. We would recommend it and it is regularly getting new releases with features and bug fixes.

not deal breakers just annoyances - longhorn doesn't currently support sharding so you need to be wary of volume size in relation to your physical disks size. (I think will be done in longhorn v2) Snapshots are currently saved to the same disk(s) the volume replica(s) are scheduled on. There is currently no scheduling to ensure that snapshot operations can successfully complete without causing disk space exhaustion. If this happens, all replicas on the disk will fault. (I believe that last point about faulting replicas is fixed) it suffers race conditions in the schedulers so our disks become over provisioned (looks like will be fixed in 1.11) and are not balanced evenly (recall there being some work in this area in latest versions) The harvester csi driver which provisions volumes from the host longhorn to guest rke2 clusters has been quite flaky and volumes often get stuck when the guest rke2 node is draining. We are on harv 1.4 and there has been lots of improvements in harv 1.6 around this so again hoping for a better experience with an upgrade. longhorn PDBs seem to frequently block node drains from completing (believe this is fixed in 1.10) So a common of theme of issues getting fixed

If you are trying to run at that scale I would HIGHLY recommend an external SAN/NAS. Longhorn, even V2, is not efficient enough. It will require a massive amount of bandwidth to maintain it’s replicas.

@acceptable-nest-4738 you can setup a call with me and I can go over things if you plan on getting a support contract. The roadmap is pretty aggressive and there are a ton of fixes going around on every release, which is how we mature. 20 servers is an easy scale for Harvester. Stability is great. The two themes we are catching on are simplifying upgrades and features. Please see the roadmap in the wiki within GitHub.

@acceptable-nest-4738 we didn't proceed with Proxmox as it's storage is supposed to be running on Ceph and mixing Ceph with VM workloads is not the best thing from what I know. We have no Ceph experts and hiring more people isn't really worth it. We've used vSphere in the past too, but since they terminated all the partner contracts, we had to move. Most of the workloads are on Hyper-V/S2D atm (we have a big amount of Windows VMs), but nobody likes to manage it, so we started using Harvester in 2 clusters now. The problem it is just not very polished, it looks nice, but every now and then there are small limitations (e.g. inability to add 2 backup destination, some weird limitations that backups cannot run at the same time, no option to simply backup everything, etc.) and for Longhorn it was mentioned previously, the disk space is limited. Other than that, it does the job, it's free with an option to get support contract later if needed, and I'm thankful to the team that was on AWS summit and handed me a suse chameleon plush toy 🙂 plus I use opensuse myself.

The roadmap is pretty aggressive and there are a ton of fixes going around on every release

Fully agree with this, it's got serious momentum!

As a former employee it is AWESOME to see the work being put into the platform!

> If you are trying to run at that scale I would HIGHLY recommend an external SAN/NAS. Longhorn, even V2, is not efficient enough. It will require a massive amount of bandwidth to maintain it’s replicas. This is true, but I think depends on what kind of hardware you're running. Overall it's decent advice. It may not be an issue if you have beast switches and all your boxes have 4 100Gib interfaces and can be all lagged up. > we didn't proceed with Proxmox as it's storage is supposed to be running on Ceph and mixing Ceph with VM workloads is not the best thing from what I know. That depends™️ on a number of factors. I 100% wouldn't recommend trying running rook for substituting ceph storage in this instance, but if you have a medium ceph cluster (more than 7 nodes but but less than 20) that's external and stand alone with ssd/nvme disks I think it'll be just fine for running VMs. My work has been doing that for years with a different platform and that portion works well. Getting people that understand your storage layer is important though. It doesn't really matter what storage you use, but there needs to be a supported CSI for it. Just make sure you have a supported k8s driver from your solution (pure, ceph, nfs, whatever) and the storage supports the features you need (RWX for live migrations etc) for the VMs.

@thousands-advantage-10804 Thanks, that’s a very interesting point. Could you share a bit more detail on your setup? • What SAN/NAS solution are you using (vendor / protocol)? • Are you completely bypassing Longhorn, or is it still used in some capacity? • Are your VM disks stored entirely on the external storage, or only selected workloads? • How is it integrated with Harvester (CSI, iSCSI, NFS, Fibre Channel)? We’re trying to understand whether an external SAN can realistically replace Longhorn for VM storage at this scale, and what trade-offs you’ve seen in practice.

That being said, having matching hardware for all the nodes in general is helpful. The quirky-ness of different hardware in different nodes can be painful, but honestly, that's true regardless of the hypervisor you're using.

We are currently on 8 nodes with 2x25Gbps LACP'd on each node for just the storage network. When does that not become enough?

@brainy-kilobyte-33711 why not 100Gbps bonded? all kidding aside, 50Gbps is better the recommended 10. In my testing every write had to traverse the storage network 3 times. basically 1 to each replica. You can set the number if replicas to 1, but you loose HA. Using a 3rd-party CSI connecting to a SAN/NAS allows for more efficient network traffic while still providing the same level of HA. In terms of protocols NVMEofTCP is faster than ISCSI, which is faster than NFS. A lot of this will depend on what hardware you have. @acceptable-nest-4738 Yes we are bypassing Longhorn. BUT… here is the cool part. You can use BOTH. meaning. use Longhorn locally with 1 replica and then a SAN/NAS for more HA goodness. You can mix and match depending on the workload. Yes when you use a 3rd party CSI the vms PV is stored on the SAN/NAS. Even the OS image needs to be stored there. This is how Harvester works. You would integrate the 3rd party storage through the use of a CSI. https://docs.harvesterhci.io/v1.5/advanced/csidriver/ here is one example of running Rancher in VMs that are stored on a Pure Storage iscsi FlashArray ( sorry I work there ), with apps also talking to the same array for storage. You can acheive this with Dell or Netapp as well. Or anything with nfs honestly. And as always the more bandwidth you can give the storage network the better.

I am very interested to hear more details if people can share about where the issues with ceph/rook are?

Rook has been a terrible experience for me. It's very NOT lightweight and a resource hog. We had a three node OpenShift cluster and installing rook absolutely burned it to the ground. It, More than once years apart, with multiple versions. Ceph is pretty great, but you need to get beyond the 7ish node limit to not have a bad time. I'm guessing that's likely true for rook but there's so many times that it's just way too much overhead for a stable cluster. For example, we use the ceph rbd (rados block device) helm csi, not the rook-ceph csi that Harvester tests with. Honestly I wish they'd suggest the ceph one instead, because it's the actual upstream project. When you try to connect to an external cluster with RBD it deploys like 50+ resources to the cluster. Why? I don't know. When I deploy the ceph rbd driver it's like 9. I'll take that every day. Not only that, but we def had something go sideways in our testing with the rook driver where some of the bloat caused issues/crashes, which isn't what I'd want for the storage performance.

Thanks, that's great info. Longhorn may have its issues but I'll take that over what you have described.

Thanks for sharing your experiences/suggestions/investigations/expectations... here we also welcome to add issues/requirements on Harvester project https://github.com/harvester/harvester/issues, they will be well checked, answered, tracked ...