mammoth-manchester-45520
11/28/2022, 11:14 PMWarning FailedAttachVolume 88s (x10 over 5m43s) attachdetach-controller AttachVolume.Attach failed for volume "pvc-5545180c-95e1-48d1-a905-f51380353016" : rpc error: code = DeadlineExceeded desc = volume pvc-5545180c-95e1-48d1-a905-f51380353016 failed to attach to node ip-10-0-3-184
yet otherwise, things generally look healthy. I am not really sure where to start troubleshooting things. I saw one issue that pointed at Linux firewalls causing problems, so I stopped ufw
(Ubuntu 20.04 system), but that didn't seem to help.
I looked at various longhorn pod logs for that node, and didn't notice anything, but any pointers to things to investigate, would be much appreciated.
Thanks!share-manager
pod related to this volume. I wasn't expecting that, but am taking a look at it now, since it doesn't appear to be happy.time="2022-11-28T23:32:53Z" level=debug msg="volume pvc-5545180c-95e1-48d1-a905-f51380353016 device /dev/longhorn/pvc-5545180c-95e1-48d1-a905-f51380353016 contains filesystem of format " encrypted=false volume=pvc-5545180c-95e1-48d1-a905-f51380353016
I1128 23:32:53.373734 1 mount_linux.go:449] Disk "/dev/longhorn/pvc-5545180c-95e1-48d1-a905-f51380353016" appears to be unformatted, attempting to format as type: "ext4" with options: [-F -m0 /dev/longhorn/pvc-5545180c-95e1-48d1-a905-f51380353016]
E1128 23:32:53.375658 1 mount_linux.go:455] format of disk "/dev/longhorn/pvc-5545180c-95e1-48d1-a905-f51380353016" failed: type:("ext4") target:("/export/pvc-5545180c-95e1-48d1-a905-f51380353016") options:("defaults") errcode:(exit status 1) output:(mke2fs 1.43.8 (1-Jan-2018)
/dev/longhorn/pvc-5545180c-95e1-48d1-a905-f51380353016 is apparently in use by the system; will not make a filesystem here!
)
time="2022-11-28T23:32:53Z" level=warning msg="failed to mount volume" encrypted=false error="format of disk \"/dev/longhorn/pvc-5545180c-95e1-48d1-a905-f51380353016\" failed: type:(\"ext4\") target:(\"/export/pvc-5545180c-95e1-48d1-a905-f51380353016\") options:(\"defaults\") errcode:(exit status 1) output:(mke2fs 1.43.8 (1-Jan-2018)\n/dev/longhorn/pvc-5545180c-95e1-48d1-a905-f51380353016 is apparently in use by the system; will not make a filesystem here!\n) " volume=pvc-5545180c-95e1-48d1-a905-f51380353016
W1128 23:32:53.440853 1 mount_helper_common.go:133] Warning: "/export/pvc-5545180c-95e1-48d1-a905-f51380353016" is not a mountpoint, deleting
share-manager
is complaining that it can't format the disk
.I am a bit surprised that share-manager
is trying to do this, as I thought that this step would be handled by a different component.