Hello everyone! I’ve been using Harvester for over a year in my homelab composed of three beefy but old HP Gen8 servers. The objective was to migrate about fifty workloads from Proxmox VMs running Docker Compose to bare-metal containers and learn about k8s in the process, and I believe I’ve achieved a satisfactory result. My only issue is disk performance; compared to the decent IOPS/bandwidth/latency I can achieve directly on my local SATA/SAS SSD drives, through Longhorn I measure only about a tenth of that, and some workloads are very sensible to such poor performance (in particular etcd in guest clusters, as I have VMs running Rancher and Fleet test environments). In comparison, Harverster’s own etcd (and the entire operating system) is stored in slower SAS 10K HDD drives in RAID5 configuration and it has never failed or logged any abnormality. I’ve tuned Longhorn and the 10G storage network as much as I could, and I’ve even tried to configure all of my guest clusters’ etcd with --unsafe-no-fsync, but my embedded clusters are failing every few hours anyway. I’ve decided to provide the closest thing to NVMe performance that I can on such old hardware, and I’ve found three PCI Express cards called FusionIO (since purchased by SanDisk, and rebranded by HP with the name Workload Accelerator). These are multi-terabyte PCIe drives from more than 10 years ago, but the specs are huge even compared to recent NVMe adapters. I’ve found that the drivers were never included in the Linux tree, but someone ported and has been maintaining the original driver by SanDisk to run in recent versions (https://github.com/RemixVSL/iomemory-vsl4). I don’t know if I’m ready to build by own Harvester images with such driver - or if I even can, without a SUSE subscription - so I’ve tried the next best thing: PCI Passthrough. Unfortunately my FusionIO adventure stopped there, because I’ve had this error since:

vfio-pci 0000:07:00.0: DMAR: Device is ineligible for IOMMU domain attach due to platform RMRR requirement.  Contact your platform vendor.

This is a PCI Passthrough problem known with old hardware and recent Linux kernels, and a workaround has been written by someone (and included in Proxmox since 2023 with a boot flag, because it is a security regression): https://github.com/Aterfax/relax-intel-rmrr/blob/master/patches/add-relaxable-rmrr-5_8_and_up.patch So I’m back to the point I wanted to avoid: I either need to add the iomemory-vsl4 kernel module to the Harvester host, or patch its existing intel-iommu module, and I have no idea on how to proceed with that; I can’t even find the corresponding kernel-devel package. I’ve tried to build a separate iSCSI storage for sensible VMs with democratic-csi, TrueNAS and spare hardware, but the performance is not enough, and I would prefer to use local hardware anyway to consume less power.

I believe that level of “homelab” might be worth considering actual enterprise support. In all seriousness, if you build such a complex home lab and you are not a freelancer, consider raising a request for a proper test lab in your current gig. You should not spend more time fixing homelab then having a life - if it’s work related stuff, on that scale, I believe you should do it on work infra and then check for support under your daily hours. I know building your own sandcastle is most satifsiying but in the end, is it still your sandcastle or a free test env for your employer? For the actual issue I’m afraid I cannot provide anything useful here :/

Thanks Paul, I know it’s not the usual homelab and I’ve definitely profited from my employer already to get most of the network equipment and some of the drives. I’m also waiting for the next customer to give us back Gen10 or better hardware when they migrate, but I will probably always be 10-15 years late anyway. This is my hobby, I already spend thousands every year in maintenance for the privilege of working with enterprise-grade stuff and I leverage it to serve dozens of people in my entourage (hosting many things for family, friends and non-profit associations) as well as thousands of citizens (with projects like NTP Pool, mirrors, public DNS servers, etc.), but what I cannot afford (and my employer can barely afford for itself) is recent servers with the same specs (terabytes of volatile memory, hundreds of cores, etc). I’ve been doing the same for about 25 years, and for now I’ve always found a way - if I have to rebuild my k8s cluster onto a more flexible operating system I will, but of course I’ll try to bend this one first.

One of the things that really hurts disk performance is running longhorn in VMs on Longhorn (Harvester). You are creating 6 writes every time. I would look to consolidating where you can. Also consider setting Longhorn to default to 1 replica. How are you resenting disks to harvester? Hardware raid? External iscsi WILL work with good networking. Can you bond the 10gb nics together?

I also have a guest Longhorn, but it's just for CD testing and nothing runs on it, so it's not a problem if its useless. Both the guest and top Longhorn have 2 replicas, and you are right, they could run a single replica... Because the real drives are set in RAID5 by the hardware controller, so they already have disk failure tolerance. 🤔 I'm planning to bond 2x10GbE but I'll have to invest in more network stuff so it'll have to wait. I'm not sure it's going to help though, I'm always using less than 5% of the current bandwidth. I'll reduce some guest clusters etcd storage to a single replica and see if it helps a lot. Thanks!

1 Longhorn replica will help with performance.

I tried democratic-csi because it integrates well with TrueNAS, but my issue is with the separate hardware, which only has 6x1GbE LACP and 7200 rpm SATA disks, even the local performance is not great. I just run MinIO on it for backups and Fleet Gitlab bootstrap purposes.

AH..

yeah we run "homelab" harvester on ~$300 ryzen beelinks with nvme/ssd + 64GB ram and it's basically fine. imo old server hw is more trouble than its worth except in special circumstances

I am a big fan of minisforum um890pro or MS-01, MS-A2!

All good choices, thank you - but I’ve paid as much for a full HP cluster with the capacity to go over 100 cores and 1TB of RAM, and I’m planning to use them. My scale is just not worth it with mini-pcs. I also have multiple SFP+ ports and the hundred-gigabit backbone to go with, and with the next generation I’ll switch to QSFP+ or better for >40GbE.

maybe time to pay for suse support lol

Maybe

I don’t believe Harvester has support yet, or even Elemental. I also find standard SLES prices to be more expensive than my whole infrastructure maintenance (drives, upgrades…) which makes little sense.

From my own testing I can say that the disks matter quite lot. We saw Longhorn tank performance by 90%+ on 15K SAS and most consumer SATA SSDs. With enterprise SSD we got much healthier numbers. Especially those with a healthy amount of DRAM and PLP.