https://rancher.com/ logo
Join Slack
Powered by
This message was deleted.
# harvester
a
This message was deleted.
w
Root cause I think*:* SCSI bus for hotplugged volumes. All these optimizations are at the Longhorn level, but our bottleneck is at the KubeVirt VM virtualization layer (SCSI emulation overhead). Even with perfect Longhorn tuning, we're still stuck with SCSI bus = 10x performance hit.
b
What data locality setting are you using?
w
volume is on the same machine as the process
b
Right, so it's set to 
best-effort
?
w
believe so - its the default, patched a volume with best-effort to force but then checked and the workload/volume were on same node
basically noticed database running very slow - and random reads are really bad
Even have a dedicated storage network... its not SCSI - its virtio underneath I think, might be wrong on that - really looking 90% of the speed fro node to cluster is quite a drop... just going to try a few more things with a fresh VM to test some of the volume options and get more benchmarks. Might raise this on the github... hoping its just a config issue! but if this is the reality then I'm going to have to re-thing the storage altogether, and possibly move to bare metal k8s, we could still manage in rancher - install longhorn in the cluster and provision from the volumes on the nodes maybe.... actually I can test that right away as I've got a cluster we've provisioned recently that isn't using harvester that might help!
Just booting off a fresh cluster to test... without harvester... see if loosing a layer helps...
c
are you using RWX or RWO? RWX is nfs on top of iscsi…
w
RWO in this instance
p
ime ~10x slowdown for VM longhorn v1 vs. raw disk is about right?
w
so this is normal?
was about to look at enabling v2 but looks risky...
if this is typical experience then its not very practical - might have to release some drives and set this up differently...
p
yeah not sure about normal but e.g. https://github.com/longhorn/longhorn/wiki/Performance-Benchmark gives a picture of v1 vs. v2
w
how painful is the migration path to v2 ... does enabling v2 mean rebooting nodes?
assuming that it also means new volumes only apply...
p
no clue, presumably reinstalling cluster would be simplest. big v2 tradeoff seems to be its CPU hungry. we mainly use zfs which has its own tradeoffs 😄
w
I think if i read properly - enabling in longhorn only applies to new volumes, but we could create a new node pool, new volumes, migrate workloads - migrate attached volumes and get there the long way... CPU isnt a problem - these machines have plenty of grunt... I've had some quick speeds but only with direct LVM mounts on HDD - which should be much slower!!! I've a MinIO cluster using LVM directly mounted to each instance working as a pool - and that runs pretty well... Brain hurts now - got some workarounds anyway for our dataabse issue since we have so much ram we can cache the DB in RAM and speed no longer an issue - but I'm keen to see if I can make this work or how to migrate to something that can work in the future... Will get a test in rancher+longhorn later tonight without harvester!
g
I've just finished setting up Longhorn V2 on my Harvester Cluster. It wasn't easy. • You need to unbind nvme drivers from the host level, so Longhorn can use it • You need to ensure, that all nvmes are in one group (one controller?) SPDK requires that- I had to reinstall Harvester and re-plan my disks. • Next you need to enable Longhorn V2 and wait for the RKE2 to reboot • Next you probably want to reserve more than one CPU (cpu_mask) by patch Longhorn Settings • Next you want to patch Longhorn nodes with the disks (it won't be available on the UI, since you've called unbind) • And reboot the instance managers • Next you want to configure Storage Classes This is the example of how I group my nvmes in Terraform (you need to plan it per server):
Copy code
harvester_install_device    = "/dev/disk/by-path/pci-0000:82:00.0-nvme-1"
    harvester_install_data_disk = "/dev/disk/by-path/pci-0000:83:00.0-nvme-1"
    spdk_disks = {
      spdk-c2 = "0000:c2:00.0"
      spdk-c3 = "0000:c3:00.0"
      spdk-c4 = "0000:c4:00.0"
      spdk-c5 = "0000:c5:00.0"
    }
It wasn't to hard either, It's mostly missing on the Terraform providers level, so some scripts are required. BUT. The performance at the end is great.
I've run the script:
Copy code
apiVersion: v1
kind: PersistentVolumeClaim
metadata:
  name: fio-v2r1-pvc
spec:
  accessModes:
    - ReadWriteOnce
  storageClassName: longhorn-v2-r1
  resources:
    requests:
      storage: 10Gi
---
apiVersion: v1
kind: Pod
metadata:
  name: fio-v2r1
spec:
  restartPolicy: Never
  containers:
    - name: fio
      image: nixery.dev/shell/fio
      command:
        - fio
        - --name=randwrite
        - --filename=/data/testfile
        - --size=4G
        - --bs=4k
        - --rw=randwrite
        - --iodepth=32
        - --ioengine=libaio
        - --direct=1
        - --numjobs=4
        - --runtime=60
        - --time_based
        - --group_reporting
      volumeMounts:
        - mountPath: /data
          name: data
  volumes:
    - name: data
      persistentVolumeClaim:
        claimName: fio-v2r1-pvc
The results I get:
Copy code
210k IOPS
820 MiB/s write bandwidth
average latency around 609 µs
p99 around 2.2 ms
p99.9 around 12 ms
🎉 1
One thing I've noticed though, is that dd is single threaded and does not saturate the IO when using Longhorn.
a
Why did you have to do the unbinding? If you enable LHv2 from harvester settings (see https://docs.harvesterhci.io/v1.7/advanced/longhorn-v2), your nvme disks (/dev/nvme[xxx]) should then be available to add to hosts via the harvester UI, which will in turn update the LH disk definitions automatically, then LH will automatically use the nvme bdev drivers and take over those disks
w
OK - I've slept on this and now have a theory, and early signs are that this might actually be the issue - looks like longhorn is running on the wrong network despite having a dedicated storage network the IP's the managers are running on are from a 1GiB network - that would explain a 90% speed drop! If this is the case it should be an easy fix!
r
It is not running on wrong network. v2 is running on mgmt (not storage and there is no fix afaik)
w
im not running v2, I'm running 1.9
and mgmt would be fine in this case too - the issue for me is another "tennant" network which runs at 1GiB that should never have been used!
r
yes. I mixed up commenters. v2 engine works only in mgmt.
w
we've 2 dedicated nets - storage and mgmt which are on 10GiB so if i reconfigure lognhorn to run its pods on the right net it should be fine, just making sure all vms have access to that network first!
we'd assumed that this was abstracted by the architecture but it is not!
r
I was originally running multiple nets and then one and now again multiple (because of Suse recommendations). But to be honest I think one net is the easiest option. Everything in harvester is so "slow" that if you have 10G+ network you rarely get network limited (we have 100G for vm and 10G for mgmt atm). Better to just to lacp all 10G interfaces together... Also if you want to run database use pci passthrough disks to vm...
databases usually have own ha mechanism and longhorn is just wasting disk space
w
yep - totally with you - we've learnt that with our Minio and using LVM 1:1 per host with a cluster of vms now - that runs super fast and handles its own HA... its a PITA getting the balance right - think we will do similar for datbases rahter than run in hosted k8s cluster as there is otherwise no way to pin to node the storage and let it do its own thing...
r
we had minio installed directly to harvester kubernetes using directpv
w
will beat that in mind - we split the interfaces for storage to isolate the traffic and seutp a VLAN but only found out today the storage while configured to use it was not using it! Yep - dierect pv looks ideal, but we've already assigned the drives to longhorn so its going to be fun clearing that lot out!
r
now moved to seaweedfs (with metallb , traefik and topolvm) on top of harvester kubernetes
w
were also using metallb - so similar otherwise
seaweed looks very interesting - will check it out 🙂
r
guy maintaining it is fast
often too fast (or maybe I had just bad timing)
but there is no future with minio so...
w
well - not sure its quite that final, but yes - days are certainly numbered and actually it does have limitations!
b
For people installing directly onto harvester rke2, do you have concerns with losing the security a VM gives you at all? We decided against seaweedfs because it did just seem to be one guy so if something happened to him we would need another migration.... Running rook is looking likely, maybe we would do that on harvester and not give all the disks to longhorn.
r
I started with rook and not going back. But I agree that Minio was obvious choice before stupidity and there is no clear winner now. We have multiple levels of security before anything from outside world can have access anything in s3 (which is anyway quite limited). So no and you can of course implement security many ways...
b
did rook prove difficult to manage?
r
just ancient
of course there is more work but still doable
b
We elected to run ceph as a separate cluster. In my experience rook, when it works is great, but if a fire starts there was no real recovery and holy crap did it have an amazing ability to bring down a cluster.
That being said, connecting to ceph with the upstream driver has been great.
b
My reference here is longhorn which has its fair share of gremlins, was rook less reliable?
b
For our hardware? Yes.
Longhorn has had issues for us, rook has made multiple clusters become unavailable.
In fairness, at least 3 of those times were different products (OpenShift, sidero, etc)
They were all POCs but we gave up on recovery after 1/2 a day or so.
b
thanks - useful info
84 Views
Open in Slack
PreviousNext
Powered by