Hi folks, we are running Prometheus with persistent volume provided by longhorn. And we have some issues with prometheus compaction. There is some related messages in prometheus logs

level=warn ts=2022-08-11T20:36:59.034Z caller=main.go:849 fs_type=NFS_SUPER_MAGIC msg="This filesystem is not supported and may lead to data corruption and data loss. Please carefully read <https://prometheus.io/docs/prometheus/latest/storage/> to learn more about supported filesystems."
level=error ts=2022-08-12T11:00:13.904Z caller=db.go:821 component=tsdb msg="compaction failed" err="WAL truncation in Compact: create checkpoint: read segments: corruption in segment /prometheus/wal/00000016 at 60187438: unexpected full record"

Could anyone point me to the right direction of troubleshooting this issue?

Is it a RWO or RWX volume?

Hi, Derek, it is RWX voluem (Access Mode:ReadWriteMany)

Got it. RWX is based on the NFS filesystem. Looks the prometheus complains the NFS filesystem. https://github.com/prometheus/prometheus/issues/8530

Derek, thank you very much for your support. This is unexpected nuance for me. Where did you find information that the Longhorn uses NFS for RWX volumes? What about RWO?

RWX volume implementation https://longhorn.io/docs/1.3.1/advanced-resources/rwx-workloads/ RWO should work. You can give it a go.

I’ve checked default value for volume access mode inside rancher-monitoring helm chart which is used to deploy monitoring stack including prometheus and it’s value RWO. Looks like I’ve changed it and forgot but why I did this I don’t remember ))) anyway thank you so much for your help!

Leaving a note here: NFS is not a POSIX compatible filesystem. Thus, some services might encounter errors while using RWX volumes. cc. @salmon-doctor-9726 @famous-shampoo-18483 @famous-journalist-11332 @kind-alarm-73406