https://rancher.com/ logo
Title
q

quiet-area-89381

03/24/2023, 11:48 PM
Hello, longhorn has been giving me headache lately (I'll ask other questions later), but the most recent issue ends up making me being stuck so far. I am deploying jupyterhub, it's been working fine but not today. I get this error for instance when one of the pods is trying to use the PV. I see the PV/PVC created properly. The volume is reported healthy in longhorn UI.
MountVolume.MountDevice failed for volume "pvc-2e1be23b-4021-45d2-878b-d37cd71679dc" : rpc error: code = Internal desc = format of disk "/dev/longhorn/pvc-2e1be23b-4021-45d2-878b-d37cd71679dc" failed: type:("ext4") target:("/var/snap/microk8s/common/var/lib/kubelet/plugins/kubernetes.io/csi/driver.longhorn.io/c072c61d6cbcb55eac5a9120cce1da05cba3ab07bb0b540278a9cfcc67b87887/globalmount") options:("defaults") errcode:(exit status 1) output:(mke2fs 1.46.4 (18-Aug-2021)...
I see the complete log
longhorn-csi-plugin-6bh98 longhorn-csi-plugin 2023-03-24T17:09:28.445862278-07:00 time="2023-03-25T00:09:28Z" level=error msg="NodeStageVolume: err: rpc error: code = Internal desc = format of disk \"/dev/longhorn/pvc-2e1be23b-4021-45d2-878b-d37cd71679dc\" failed: type:(\"ext4\") target:(\"/var/snap/microk8s/common/var/lib/kubelet/plugins/kubernetes.io/csi/driver.longhorn.io/c072c61d6cbcb55eac5a9120cce1da05cba3ab07bb0b540278a9cfcc67b87887/globalmount\") options:(\"defaults\") errcode:(exit status 1) output:(mke2fs 1.46.4 (18-Aug-2021)\n/dev/longhorn/pvc-2e1be23b-4021-45d2-878b-d37cd71679dc is apparently in use by the system; will not make a filesystem here!\n) "
looks very similar to this problem https://github.com/longhorn/longhorn/issues/3177
and problably the manual fix on the kubelet nodes running longhorn https://longhorn.io/kb/troubleshooting-volume-with-multipath/
I do see the multipath on the longhorn devices
I change the config of multipathd on all nodes to blacklist the /dev/sdXXX devices. Now my pods can mount their PV. Success! Not sure why I didn't know about that important setting. I had never seen it anywhere in longhorn main doc/helm chart.
Maybe my other issues are related to that, so I hold on to my other questions for a few days. 😄