Of course, it is possible that Longhorn is the one using a huge amount of disk space. It could, for example, be creating a bunch of unnecessary replicas of the data on the disk. However, it's hard to understand how draining the node would fix the problem in that case. Draining the node would move all workloads (and Longhorn engines) off of it, but there's no particular reason it would lead to a decrease in Longhorn data consumption. (If Longhorn created a bunch of unnecessary replicas, the drain shouldn't lead to them being removed.) If this situation arises again: • Can we check the consumption of space within the

/var/lib/longhorn

directory on the offending node versus another to confirm the space usage is actually Longhorn's? (In fact, you could do that now on all nodes to see if the Longhorn usage is actually different.) • Can we grab a support bundle so we can see how Longhorn things space it being used at the moment?

root@eweber-v126-worker-9c1451b4-rw5hf:~# du -h -s /var/lib/longhorn
180M    /var/lib/longhorn

root@eweber-v126-worker-9c1451b4-6464j:~# du -h -s /var/lib/longhorn
180M    /var/lib/longhorn

root@eweber-v126-worker-9c1451b4-kgxdq:~# du -h -s /var/lib/longhorn
181M    /var/lib/longhorn

I found the issue, it was due to docker logging, I had not configured any logrotate for the dockerdaemon on the cluster nodes, so one container was filling up the disk with logs.