This message was deleted.

I'm not sure, but.... I'll point this out: I would imagine that this depends on the ingress/egress traffic of your VMs and if they're on separate host interfaces for their VM networks, and if your management network is only running k8s/longhorn traffic. The next thing I would imagine affecting this is what your data replication/tolerance/datalocality is. After that, disk iops is going to make the biggest difference. Do the workloads have a RW or just read?

100%. This is great feedback and I have certainly taken these factors into account. However, I was hoping that there may be some anecdotal reports of people’s cluster sizes and maybe even experiences with this

We have 9 nodes and aren't near network saturation, but have a second interface for the VM networks.

Yeah, so we’re experimenting and would dedicate the longhorn (and admin/mgmt internal net) to 10GB switches w/ dual redudnant nics (trunked @ 2x10GB each node). I mean, we’re looking to deploy a 5 node network and want to do some calculations on how much utilization (and clients of specific workloads) a cluster like this could handle. I know I’m leaving a lot of information out here that would factor in, which is why I’m looking to the world to provide some anchoring. For instance, could you give me an idea of your rough aggregate public facing workload and the resultant metrics on your longhorn storage network? Or anything else that may help me do some calculations?

I don't have access to those metrics, but with two nics, I think you're gonna be ok. But then again, how many Longhorn disks are you expecting? (how many VMs?)

Yes, for sure. A lot of that ends up being dictated by client application workloads that we deploy to the K8s clusters whose nodes are ultimately provisioned on the harvester cluster.

Ah!

and how many disks are provisioned in which way as the storage pool on a longhorn cluster, etc.

And the rke clusters are all using the mgmt network?

yes

Well the RKE nodes will use more of the traffic too for things like etcd

For sure. I wonder if anyone else in this channel has any info or opinions to share?

Heyyy 👋 Harvester PM here at SUSE: We need to do some sort of benchmark and profiling, but it isn't something we actually have right now I'm afraid (at least from our side). It'd be really interesting to see who has done something similar, or, if you end up doing something yourself - if this is something we could potentially collaborate on.

Hey! That is great to hear! I would love to help, but I’m hoping to get some data first before committing to a spend which has a lot of unknowns. Collaboration sounds very interesting! I’d love to help.

Morning! From the Longhorn side?>

I imagine the overhead of harvester itself isn’t too large.

I'm pretty sure it's just whatever the traffic would be for a k3s cluster with x nodes + longhorn + (other downstream clusters or vms if they're using the mgmt network for their backend or network ) Otherwise you're looking for how much network traffic is generated per x amount of disk iops right?

Yep

Well you could also use a dedicated storage network to even further separate the longhorn traffic from the mgmt/k8s traffic and the inter-vm/rke2-guest-k8s-traffic by using 3 separate interfaces and also use a balance tlb or other bond type to even get more throughput, right?