Can someone point me to the documentation on how t...
# longhorn-storage
s
Can someone point me to the documentation on how to handle disk pressure if a volume got to 95% usage? is there a way to reduce the size of a pvc replica on the node has terabytes of data and the node became unable to schedule work loads
i
cc @little-kangaroo-65735
l
You can have a look here https://longhorn.io/docs/1.9.0/nodes-and-volumes/nodes/multidisk/ What about pvc resize, they can be only expanded. One way could be to create a new smaller PV, then manually copy the data over the it and finally delete the old one.
i
Hello @stocky-leather-992, I have a few questions to clarify before we can properly address the issue: • What exactly does the 95% usage refer to? • How is this value obtained? Is it from the node’s host disk, or from inside the workload using the volume? Since Longhorn doesn't automatically reclaim allocated space, it’s important to clarify these points before we can suggest the next steps.
s
Good morning @icy-agency-38675. the 95% is obtained from the longhorn UI. says node not ready and the usage is 95%. This is a vm on Vmware vsphere. I looked into the volumes under /var/lib/longhorn/replicas (not sure about exact directory) and I saw the volume taking up the that much storage on the affected node.
i
Got it. Need you help • Can you access the workload using the volume and execute
df -h
? • How many snapshots for the volume?
s
I cannot access the workload since it wont come back up. There are no snapshots of this specific pvc since it got too large
Its the elastic logs
@icy-agency-38675 so I managed to add storage to the node. now the longhorn manager pod is giving an error saying failed to upgrade volume recurring job to volume labels. its complaining about upgrading from 1.1.0 to 1.2.0. I never did any upgrade but I did restart the node
https://github.com/longhorn/longhorn/issues/2961 similar to this issue. unfortunatley draining all worker nodes is not an option for me
Fixed the issue. had to unschedule all of the persistant workload and do so much manual stuff. Time to really get that longhorn upgrade. Thank you for the help
🙌 1
👍 1
i
@stocky-leather-992 If there is anything that Longhorn can improve, feel free to create a PR in https://github.com/longhorn/longhorn/issues. Thank you.