Hello wave I am a new user to Longhorn inherited a system an rancher-users #longhorn-storage

Title

# longhorn-storage

mammoth-manchester-45520

11/28/2022, 11:14 PM

Hello! 👋 I am a new user to Longhorn (inherited a system) and am having an issue with one volume which is being mounted and read by 4 different deployments. The PV and PVC exist, but each of the pods has an error similar to this:

Warning  FailedAttachVolume  88s (x10 over 5m43s)  attachdetach-controller  AttachVolume.Attach failed for volume "pvc-5545180c-95e1-48d1-a905-f51380353016" : rpc error: code = DeadlineExceeded desc = volume pvc-5545180c-95e1-48d1-a905-f51380353016 failed to attach to node ip-10-0-3-184

yet otherwise, things generally look healthy. I am not really sure where to start troubleshooting things. I saw one issue that pointed at Linux firewalls causing problems, so I stopped

ufw

(Ubuntu 20.04 system), but that didn't seem to help. I looked at various longhorn pod logs for that node, and didn't notice anything, but any pointers to things to investigate, would be much appreciated. Thanks!

✅ 1

I noticed that there is a

share-manager

pod related to this volume. I wasn't expecting that, but am taking a look at it now, since it doesn't appear to be happy.

Seeing things like this:

Copy code

time="2022-11-28T23:32:53Z" level=debug msg="volume pvc-5545180c-95e1-48d1-a905-f51380353016 device /dev/longhorn/pvc-5545180c-95e1-48d1-a905-f51380353016 contains filesystem of format " encrypted=false volume=pvc-5545180c-95e1-48d1-a905-f51380353016
I1128 23:32:53.373734       1 mount_linux.go:449] Disk "/dev/longhorn/pvc-5545180c-95e1-48d1-a905-f51380353016" appears to be unformatted, attempting to format as type: "ext4" with options: [-F -m0 /dev/longhorn/pvc-5545180c-95e1-48d1-a905-f51380353016]
E1128 23:32:53.375658       1 mount_linux.go:455] format of disk "/dev/longhorn/pvc-5545180c-95e1-48d1-a905-f51380353016" failed: type:("ext4") target:("/export/pvc-5545180c-95e1-48d1-a905-f51380353016") options:("defaults") errcode:(exit status 1) output:(mke2fs 1.43.8 (1-Jan-2018)
/dev/longhorn/pvc-5545180c-95e1-48d1-a905-f51380353016 is apparently in use by the system; will not make a filesystem here!
) 
time="2022-11-28T23:32:53Z" level=warning msg="failed to mount volume" encrypted=false error="format of disk \"/dev/longhorn/pvc-5545180c-95e1-48d1-a905-f51380353016\" failed: type:(\"ext4\") target:(\"/export/pvc-5545180c-95e1-48d1-a905-f51380353016\") options:(\"defaults\") errcode:(exit status 1) output:(mke2fs 1.43.8 (1-Jan-2018)\n/dev/longhorn/pvc-5545180c-95e1-48d1-a905-f51380353016 is apparently in use by the system; will not make a filesystem here!\n) " volume=pvc-5545180c-95e1-48d1-a905-f51380353016
W1128 23:32:53.440853       1 mount_helper_common.go:133] Warning: "/export/pvc-5545180c-95e1-48d1-a905-f51380353016" is not a mountpoint, deleting

Oddly the volume appears perfectly healthy in the UI, despite the fact that

share-manager

is complaining that it can't format the disk .I am a bit surprised that

share-manager

is trying to do this, as I thought that this step would be handled by a different component.

I think I finally got it fixed. Outside of deleting and re-creating things in the right order, I think the issue might have been related to this: https://longhorn.io/kb/troubleshooting-rwx-mount-ownership-set-to-nobody/ Although I did not have that error.

👍 1

118 Views