Happy Harvesting to all! I have couple of storage class questions (using v1.6): 1. When k8's cluster is provisioned by the Harvester CSI driver (through Rancher) is there any benefit in adding the passthrough custom storage class to it https://docs.harvesterhci.io/v1.6/rancher/csi-driver#passthrough-custom-storageclass ? 2. I heard about 'longhorn in longhorn' and it doesn't sound good.. Is the 'harvester' storage class, that comes out of the box in the K8's cluster from question1, good enough to avoid it?

1. yes. You can present the Longhorn on Harvester to the k8s cluster running on vms. This will save you on the complexity of adding another storage layers in on the vms. 2. The passthrough will prevent the longhorn on longhorn problem. You MAY want to use it for testing or tiering of storage. You WILL want to look at 2rd party CSI drivers for external storage. this will help with scaling. Bonus tip, make sure you have the at least 2.5gb network for dev/test and 10gb for prod.

Thanks for clarification @thousands-advantage-10804. I was hoping that I don't need to use any custom storage class because there is some problem with it that it seems like I need to dig in now 😅 When I create a pod and use this custom storage class to provision a volume - it all looks good on the cluster/rancher side, but there is an error in Harvester:

MapVolume.MapPodDevice failed for volume "pvc-dbf52476-201d-488c-b8fd-1c3a9ae14854" : rpc error: code = Internal desc = failed to bind mount "/var/lib/kubelet/plugins/kubernetes.io/csi/volumeDevices/staging/pvc-dbf52476-201d-488c-b8fd-1c3a9ae14854/pvc-dbf52476-201d-488c-b8fd-1c3a9ae14854" at "/var/lib/kubelet/plugins/kubernetes.io/csi/volumeDevices/publish/pvc-dbf52476-201d-488c-b8fd-1c3a9ae14854/6e44a2c0-ff84-4e4c-b0df-1cf685a48fd5": mount failed: exit status 32 Mounting command: mount Mounting arguments: -o bind /var/lib/kubelet/plugins/kubernetes.io/csi/volumeDevices/staging/pvc-dbf52476-201d-488c-b8fd-1c3a9ae14854/pvc-dbf52476-201d-488c-b8fd-1c3a9ae14854 /var/lib/kubelet/plugins/kubernetes.io/csi/volumeDevices/publish/pvc-dbf52476-201d-488c-b8fd-1c3a9ae14854/6e44a2c0-ff84-4e4c-b0df-1cf685a48fd5 Output: mount: /var/lib/kubelet/plugins/kubernetes.io/csi/volumeDevices/publish/pvc-dbf52476-201d-488c-b8fd-1c3a9ae14854/6e44a2c0-ff84-4e4c-b0df-1cf685a48fd5: special device /var/lib/kubelet/plugins/kubernetes.io/csi/volumeDevices/staging/pvc-dbf52476-201d-488c-b8fd-1c3a9ae14854/pvc-dbf52476-201d-488c-b8fd-1c3a9ae14854 does not exist. dmesg(1) may have more information after failed mount system call.

Heads, up. If you pay for support you'll need the longhorn add-on for each of the downstream nodes (including non-worker control plane) if you do the passthrough.

Good to know, Thanks! I'm a free loader for now... It would be good to get to the point when I need/can afford paid support :)

Nothing wrong with trying to run unsupported. We were just caught off guard (we weren't doing passthrough) but that we needed extra licenses for each downstream cluster including controlplane nodes.

I'm comparing performance of the local storage (pod is running on the bare metal rke2 cluster) to the harvester-passthrough one (pod is running in rke2 provisioned by Harvester) using

fio --name=rand-read --ioengine=libaio --iodepth=16 --rw=randread --bs=4k --direct=1 --size=1G --numjobs=4 --runtime=60 --group_reporting

The hardware is identical, one node is Rancher with local rke2 cluster the other is Harvester with rke2. Local storage:

Jobs: 4 (f=4): [r(4)][-.-%][r=1458MiB/s,w=0KiB/s][r=373k,w=0 IOPS][eta 00m:00s]
rand-read: (groupid=0, jobs=4): err= 0: pid=886: Tue Sep  2 22:41:18 2025
   read: IOPS=371k, BW=1447MiB/s (1518MB/s)(4096MiB/2830msec)
    slat (usec): min=5, max=263, avg= 8.84, stdev= 4.60
    clat (usec): min=58, max=894, avg=161.12, stdev=51.77
     lat (usec): min=64, max=901, avg=170.08, stdev=52.07
    clat percentiles (usec):
     |  1.00th=[   94],  5.00th=[  108], 10.00th=[  116], 20.00th=[  126],
     | 30.00th=[  135], 40.00th=[  141], 50.00th=[  149], 60.00th=[  157],
     | 70.00th=[  169], 80.00th=[  190], 90.00th=[  223], 95.00th=[  260],
     | 99.00th=[  359], 99.50th=[  400], 99.90th=[  510], 99.95th=[  562],
     | 99.99th=[  676]
   bw (  KiB/s): min=344808, max=382184, per=25.37%, avg=376002.00, stdev=7881.48, samples=20
   iops        : min=86202, max=95546, avg=94000.50, stdev=1970.37, samples=20
  lat (usec)   : 100=2.32%, 250=91.80%, 500=5.76%, 750=0.11%, 1000=0.01%
  cpu          : usr=12.90%, sys=86.16%, ctx=10206, majf=0, minf=101
  IO depths    : 1=0.1%, 2=0.1%, 4=0.1%, 8=0.1%, 16=100.0%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued rwt: total=1048576,0,0, short=0,0,0, dropped=0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=16

Run status group 0 (all jobs):
   READ: bw=1447MiB/s (1518MB/s), 1447MiB/s-1447MiB/s (1518MB/s-1518MB/s), io=4096MiB (4295MB), run=2830-2830msec

Harvester-Longhorn Passthrough:

Jobs: 4 (f=4): [r(4)][100.0%][r=90.5MiB/s,w=0KiB/s][r=23.2k,w=0 IOPS][eta 00m:00s]
rand-read: (groupid=0, jobs=4): err= 0: pid=660: Tue Sep  2 22:43:59 2025
   read: IOPS=23.7k, BW=92.5MiB/s (96.0MB/s)(4096MiB/44296msec)
    slat (usec): min=2, max=33905, avg= 8.78, stdev=43.28
    clat (usec): min=70, max=64533, avg=2693.15, stdev=862.13
     lat (usec): min=450, max=64554, avg=2702.09, stdev=864.54
    clat percentiles (usec):
     |  1.00th=[ 1631],  5.00th=[ 1876], 10.00th=[ 2008], 20.00th=[ 2180],
     | 30.00th=[ 2311], 40.00th=[ 2409], 50.00th=[ 2540], 60.00th=[ 2671],
     | 70.00th=[ 2835], 80.00th=[ 3064], 90.00th=[ 3589], 95.00th=[ 4113],
     | 99.00th=[ 5145], 99.50th=[ 5669], 99.90th=[ 8848], 99.95th=[10421],
     | 99.99th=[25822]
   bw (  KiB/s): min=20416, max=25728, per=24.95%, avg=23625.16, stdev=1017.91, samples=352
   iops        : min= 5104, max= 6432, avg=5906.27, stdev=254.48, samples=352
  lat (usec)   : 100=0.01%, 250=0.01%, 500=0.01%, 750=0.01%, 1000=0.03%
  lat (msec)   : 2=9.96%, 4=84.12%, 10=5.82%, 20=0.04%, 50=0.01%
  lat (msec)   : 100=0.01%
  cpu          : usr=1.92%, sys=8.33%, ctx=533966, majf=0, minf=96
  IO depths    : 1=0.1%, 2=0.1%, 4=0.1%, 8=0.1%, 16=100.0%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued rwt: total=1048576,0,0, short=0,0,0, dropped=0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=16

Run status group 0 (all jobs):
   READ: bw=92.5MiB/s (96.0MB/s), 92.5MiB/s-92.5MiB/s (96.0MB/s-96.0MB/s), io=4096MiB (4295MB), run=44296-44296msec

Does it make sense?

for single node 1 gb is fine. It is when you have multiple nodes. Longhorn is a chatty thing. You are seeing the fact that longhorn is NOT performant. do you have access to an iscsi NAS?

No, don't have any storage out of the server right now. Is there anything that may help with the drives that are in the node itself? I didn't expect the custom passthrough storage class to have the same performance as out of the box harvester storage class tbh..

Honestly if you have a single node you should reduce the replicas to 1. there is ZERO need for redundancy with only 1 drive backing it all.

Already did. I created initially with 2 because planned to have 3 node rke2 cluster, but now I'm thinking maybe cluster with just one beefed up node would do be better (they all run on the same hardware anyway :)

how many vms do you need to run? I have 2 separate single node installs so I can play with new versions independently.

All my services are running in containers, so i don't even need VM's so much.. but its nice to have an option if I need one.. otherwise would be using just rke2

sounds like you already have your answer. 😄

yeah something like pci passthru or directpv I think are best storage perf? csi-lvm? it'd be nice if there was like a CSI feature table / benchmarks somewhere

I pass a nvme to a vm for truenas. Works great.

Tested the lvm-local-storage, results:

Jobs: 4 (f=4): [r(4)][100.0%][r=461MiB/s][r=118k IOPS][eta 00m:00s]
rand-read: (groupid=0, jobs=4): err= 0: pid=21976: Wed Sep 3 22:39:38 2025
 read: IOPS=115k, BW=449MiB/s (471MB/s)(4096MiB/9114msec)
  slat (nsec): min=1810, max=5630.1k, avg=4182.16, stdev=8905.92
  clat (usec): min=26, max=26334, avg=549.87, stdev=269.73
   lat (usec): min=97, max=26909, avg=554.05, stdev=270.57
  clat percentiles (usec):
   | 1.00th=[ 306], 5.00th=[ 404], 10.00th=[ 445], 20.00th=[ 486],
   | 30.00th=[ 506], 40.00th=[ 519], 50.00th=[ 529], 60.00th=[ 545],
   | 70.00th=[ 562], 80.00th=[ 586], 90.00th=[ 644], 95.00th=[ 701],
   | 99.00th=[ 914], 99.50th=[ 1254], 99.90th=[ 3884], 99.95th=[ 5276],
   | 99.99th=[11469]
  bw ( KiB/s): min=372376, max=481072, per=100.00%, avg=460817.78, stdev=5968.76, samples=72
  iops    : min=93094, max=120268, avg=115204.44, stdev=1492.19, samples=72
 lat (usec)  : 50=0.01%, 100=0.01%, 250=0.44%, 500=25.97%, 750=70.51%
 lat (usec)  : 1000=2.30%
 lat (msec)  : 2=0.58%, 4=0.11%, 10=0.07%, 20=0.01%, 50=0.01%
 cpu     : usr=5.29%, sys=17.08%, ctx=90238, majf=0, minf=113
 IO depths  : 1=0.1%, 2=0.1%, 4=0.1%, 8=0.1%, 16=100.0%, 32=0.0%, >=64=0.0%
   submit  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
   complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0%
   issued rwts: total=1048576,0,0,0 short=0,0,0,0 dropped=0,0,0,0
   latency  : target=0, window=0, percentile=100.00%, depth=16

Run status group 0 (all jobs):
  READ: bw=449MiB/s (471MB/s), 449MiB/s-449MiB/s (471MB/s-471MB/s), io=4096MiB (4295MB), run=9114-9114msec

So it is ~5 times faster than Harvester Longhorn and ~3 times slower than native/node storage. Would be great if you guys can share your setup and performance test(s) results.

Harvester on a minisforum minipc with nvme disk. 1 Longhorn replica from within a VM

[root@tetee ~]# fio --name=rand-read --ioengine=libaio --iodepth=16 --rw=randread --bs=4k --direct=1 --size=1G --numjobs=4 --runtime=60 --group_reporting
rand-read: (g=0): rw=randread, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=libaio, iodepth=16
...
fio-3.36
Starting 4 processes
rand-read: Laying out IO file (1 file / 1024MiB)
rand-read: Laying out IO file (1 file / 1024MiB)
rand-read: Laying out IO file (1 file / 1024MiB)
rand-read: Laying out IO file (1 file / 1024MiB)
Jobs: 3 (f=3): [r(1),_(1),r(2)][95.5%][r=201MiB/s][r=51.5k IOPS][eta 00m:01s]
rand-read: (groupid=0, jobs=4): err= 0: pid=2563: Wed Sep  3 23:22:07 2025
  read: IOPS=50.6k, BW=197MiB/s (207MB/s)(4096MiB/20743msec)
    slat (nsec): min=1993, max=604530, avg=3611.77, stdev=3067.62
    clat (usec): min=174, max=15966, avg=1237.90, stdev=457.75
     lat (usec): min=178, max=15969, avg=1241.51, stdev=457.86
    clat percentiles (usec):
     |  1.00th=[  478],  5.00th=[  676], 10.00th=[  734], 20.00th=[  816],
     | 30.00th=[  914], 40.00th=[ 1090], 50.00th=[ 1270], 60.00th=[ 1369],
     | 70.00th=[ 1450], 80.00th=[ 1565], 90.00th=[ 1713], 95.00th=[ 1860],
     | 99.00th=[ 2638], 99.50th=[ 3130], 99.90th=[ 4752], 99.95th=[ 5407],
     | 99.99th=[ 6194]
   bw (  KiB/s): min=168872, max=270720, per=100.00%, avg=204682.46, stdev=6910.42, samples=162
   iops        : min=42218, max=67680, avg=51170.61, stdev=1727.61, samples=162
  lat (usec)   : 250=0.15%, 500=0.89%, 750=10.77%, 1000=24.31%
  lat (msec)   : 2=60.52%, 4=3.17%, 10=0.19%, 20=0.01%
  cpu          : usr=1.55%, sys=7.92%, ctx=147840, majf=0, minf=111
  IO depths    : 1=0.1%, 2=0.1%, 4=0.1%, 8=0.1%, 16=100.0%, 32=0.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.1%, 32=0.0%, 64=0.0%, >=64=0.0%
     issued rwts: total=1048576,0,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=16

Run status group 0 (all jobs):
   READ: bw=197MiB/s (207MB/s), 197MiB/s-197MiB/s (207MB/s-207MB/s), io=4096MiB (4295MB), run=20743-20743msec

Disk stats (read/write):
  vda: ios=1035959/76, sectors=8287680/1366, merge=0/33, ticks=1254276/154, in_queue=1254430, util=85.17%