Our Kubernetes and OpenStack clusters are spread across AZs in a region (Dallas/Austin) but the connection is stable enough for etcd to be happy without any reconfiguration. We suspect we are running into a combination of issues relating to hypervisor capacity, possibly some network latency for Longhorn (there was a volume timeout introduced in 1.4.0 that we want to be able to upgrade to and tweak but we can’t right now), and Ceph disk latency.

I have looked over the documentation and the best practices page is kinda the only thing I’ve found in terms of a starting place for our expectations. One of my coworkers swears he read somewhere in the docs that “you should not run Longhorn on top of Ceph” but he cannot find any link proving that. So I’m curious to hear y’alls opinions, as we are using SSDs, and we are using lots of them so I feel like we should be well within operating parameters. Can anyone share any thoughts or other experiences positively or negatively with situations like this? Our next steps (later this year) are to retry Longhorn on physical drives off of a RAID controller, and then attempt a similar trial with a PureStorage device to see if either of those things help improve our results.

I REALLY don’t want to have to worry about site specific storage anymore but these issues have required us to completely abandon this project

And Longhorn works directly on block storage … so I would also think that running it on top of ceph and the like is not to be advised.

For added context, we know there is a certain amount of contention on the underlying Ceph because Graphite could not keep up at all and there is some “slower” database performance on a couple of VMs but the vast majority of the time outside of things like etcd/databases/Graphite/Longhorn, nothing seems to truly care. So I’m trying to see if I can get more specifics on WHY Longhorn might care so much so we can see if we can alleviate those specific issues. But again even when Longhorn is doing almost no IOPS, it still breaks volumes. We have configured CephRBD directly to the OpenStack endpoints and performance is fine.

Our data retention for this is 45 days but we started around May 9th and moved to CephRBD on around June 9th. This screenshot lists one of our clusters that likely had THE MOST load over this time period, again primarily due to Thanos Compaction but even after moving the Compactors off, things that did was less IOPS were still failing/rebuilding/going read-only. We have ~80 Samsung 870 EVOs, on the underlying platform with 3 way replication at the Ceph level but these are of course, shared with everything else.

We have 3 other clusters that have similar access patterns, so a worst case approximation is a total of 4x this graph. Some of these IOPS counts may be inflated compared to actual usage just because Longhorn was almost regularly rebuilding volumes every 2-3 days.

if you already have Ceph, why not use ceph-csi?

We have moved back to Ceph CSI for current reasons but our underlying infrastructure providers do not provide site agnostic Ceph.

I’ll post back here later this year when we get to some of the storage trials we’re planning on attempting