Channels
academy
amazon
arm
azure
cabpr
chinese
ci-cd
danish
deutsch
developer
elemental
epinio
espanol
events
extensions
fleet
français
gcp
general
harvester
harvester-dev
hobbyfarm
hypper
japanese
k3d
k3os
k3s
k3s-contributor
kim
kubernetes
kubewarden
lima
logging
longhorn-dev
longhorn-storage
masterclass
mesos
mexico
nederlands
neuvector-security
office-hours
one-point-x
onlinemeetup
onlinetraining
opni
os
ozt
phillydotnet
portugues
rancher-desktop
rancher-extensions
rancher-setup
rancher-wrangler
random
rfed_ara
rio
rke
rke2
russian
s3gw
service-mesh
storage
submariner
supermicro-sixsq
swarm
terraform-controller
terraform-provider-rancher2
terraform-provider-rke
theranchcast
training-0110
training-0124
training-0131
training-0207
training-0214
training-1220
ukranian
v16-v21-migration
vsphere
windows
Title
q
quiet-area-89381
03/24/2023, 11:48 PMHello, longhorn has been giving me headache lately (I'll ask other questions later), but the most recent issue ends up making me being stuck so far. I am deploying jupyterhub, it's been working fine but not today. I get this error for instance when one of the pods is trying to use the PV. I see the PV/PVC created properly. The volume is reported healthy in longhorn UI.
MountVolume.MountDevice failed for volume "pvc-2e1be23b-4021-45d2-878b-d37cd71679dc" : rpc error: code = Internal desc = format of disk "/dev/longhorn/pvc-2e1be23b-4021-45d2-878b-d37cd71679dc" failed: type:("ext4") target:("/var/snap/microk8s/common/var/lib/kubelet/plugins/kubernetes.io/csi/driver.longhorn.io/c072c61d6cbcb55eac5a9120cce1da05cba3ab07bb0b540278a9cfcc67b87887/globalmount") options:("defaults") errcode:(exit status 1) output:(mke2fs 1.46.4 (18-Aug-2021)...I see the complete log
longhorn-csi-plugin-6bh98 longhorn-csi-plugin 2023-03-24T17:09:28.445862278-07:00 time="2023-03-25T00:09:28Z" level=error msg="NodeStageVolume: err: rpc error: code = Internal desc = format of disk \"/dev/longhorn/pvc-2e1be23b-4021-45d2-878b-d37cd71679dc\" failed: type:(\"ext4\") target:(\"/var/snap/microk8s/common/var/lib/kubelet/plugins/kubernetes.io/csi/driver.longhorn.io/c072c61d6cbcb55eac5a9120cce1da05cba3ab07bb0b540278a9cfcc67b87887/globalmount\") options:(\"defaults\") errcode:(exit status 1) output:(mke2fs 1.46.4 (18-Aug-2021)\n/dev/longhorn/pvc-2e1be23b-4021-45d2-878b-d37cd71679dc is apparently in use by the system; will not make a filesystem here!\n) "looks very similar to this problem https://github.com/longhorn/longhorn/issues/3177
and problably the manual fix on the kubelet nodes running longhorn https://longhorn.io/kb/troubleshooting-volume-with-multipath/
I do see the multipath on the longhorn devices
I change the config of multipathd on all nodes to blacklist the /dev/sdXXX devices. Now my pods can mount their PV. Success! Not sure why I didn't know about that important setting. I had never seen it anywhere in longhorn main doc/helm chart.
Maybe my other issues are related to that, so I hold on to my other questions for a few days. 😄