https://rancher.com/ logo
Title
l

lively-hospital-24301

08/12/2022, 3:24 PM
Hi folks, we are running Prometheus with persistent volume provided by longhorn. And we have some issues with prometheus compaction. There is some related messages in prometheus logs
level=warn ts=2022-08-11T20:36:59.034Z caller=main.go:849 fs_type=NFS_SUPER_MAGIC msg="This filesystem is not supported and may lead to data corruption and data loss. Please carefully read <https://prometheus.io/docs/prometheus/latest/storage/> to learn more about supported filesystems."
level=error ts=2022-08-12T11:00:13.904Z caller=db.go:821 component=tsdb msg="compaction failed" err="WAL truncation in Compact: create checkpoint: read segments: corruption in segment /prometheus/wal/00000016 at 60187438: unexpected full record"
Could anyone point me to the right direction of troubleshooting this issue?
i

icy-agency-38675

08/15/2022, 3:45 AM
Is it a RWO or RWX volume?
l

lively-hospital-24301

08/15/2022, 8:24 AM
Hi, Derek, it is RWX voluem (Access Mode:ReadWriteMany)
i

icy-agency-38675

08/15/2022, 8:25 AM
Got it. RWX is based on the NFS filesystem. Looks the prometheus complains the NFS filesystem. https://github.com/prometheus/prometheus/issues/8530
l

lively-hospital-24301

08/15/2022, 9:27 AM
Derek, thank you very much for your support. This is unexpected nuance for me. Where did you find information that the Longhorn uses NFS for RWX volumes? What about RWO?
i

icy-agency-38675

08/15/2022, 9:30 AM
RWX volume implementation https://longhorn.io/docs/1.3.1/advanced-resources/rwx-workloads/ RWO should work. You can give it a go.
l

lively-hospital-24301

08/15/2022, 2:01 PM
I’ve checked default value for volume access mode inside rancher-monitoring helm chart which is used to deploy monitoring stack including prometheus and it’s value RWO. Looks like I’ve changed it and forgot but why I did this I don’t remember ))) anyway thank you so much for your help!
👍 1
i

icy-agency-38675

08/16/2022, 2:14 AM
Leaving a note here: NFS is not a POSIX compatible filesystem. Thus, some services might encounter errors while using RWX volumes. cc. @salmon-doctor-9726 @famous-shampoo-18483 @famous-journalist-11332 @kind-alarm-73406
👍 1