https://rancher.com/ logo
Join Slack
Powered by
Has anyone tried to enable v2 volumes in harvester...
# harvester
b
Has anyone tried to enable v2 volumes in harvester 1.3.0 + longhorn v1.6: 1. I’ve enable the proper amount of huepages 2. I’ve loaded the uio_pci_generic modules and the others are already loaded 3. I’ve run the requirements script on the harvester node and it seems good - I believe cryptsetup is not needed for unecrypted volumes.
Copy code
[INFO]  Required dependencies 'kubectl jq mktemp sort printf' are installed.
[INFO]  All nodes have unique hostnames.
[INFO]  Waiting for longhorn-environment-check pods to become ready (0/1)...
[INFO]  All longhorn-environment-check pods are ready (1/1).
[INFO]  MountPropagation is enabled
[INFO]  Checking kernel release...
[INFO]  Checking iscsid...
[INFO]  Checking multipathd...
[INFO]  Checking packages...
[ERROR] cryptsetup is not found in harvester01.
[INFO]  Checking nfs client...
[INFO]  Checking x86-64 SSE4.2 instruction set...
[INFO]  Checking kernel module nvme_tcp...
[INFO]  Checking kernel module uio_pci_generic...
[INFO]  Checking hugepage...
[INFO]  Cleaning up longhorn-environment-check pods...
[INFO]  Cleanup completed.
1. I’ve forwarded the longhorn interface and enabled v2 engine 2. I’ve tried to create v2 volumes in longhorn ui (and kubectl) and it creates them but fails to schedule replicas. Lots of space available and everything. Any ideas?
m
Hi Kristian Kostecky, Harvester hasn't supported LH v2 yet, however, I remember @great-bear-19718 had some trials before, maybe he can provide some feedback?
b
I actually got it working just now. I forgot to provision drives as block devices vs. file system devices
🙌 1
g
it works fine
some UX items are left which we are working on.. as you found out all deps are already packaged in the OS already
🙌 1
i think we are waiting for longhorn v2 engine to be marked stable in longhorn which is when this is likely to be exposed in harvester ui
b
yeah for sure. I was able to expose it by creating a storage class and it works. However, GA v2 is like a year away
Which is unfortunate, because with NVMes that get 2.5-5GB/s, longhorn v1 volumes are only giving me about 350MB/s 😢
and that is with strict-local and replica=1
I have no idea why there is 90% overhead on iops, bandwidth and latency
but it makes it a non starter to use in production
g
the limitation is from architecture.. iscsi over overlay network
b
yeah. do you know if there are any tunables at any level of the stack to squeeze some more performance out of overlay or iscsi?
g
i will ask our longhorn team, and we are also trying to fast track v2 engine
how was your experience with v2 engine? I used it for a very specific use case internally
b
yeah, I could squeeze out 900 MB/s on write, which was ALMOST there
Reads scale with replicas nicely, so that’s not an issue
it’s really about getting close to native write speeds on replica=1 and then being bandwidth limited by 10-25gbit switches for writes when replica > 1
6 Views
Open in Slack
PreviousNext
Powered by