bitter-monkey-66305
02/10/2025, 3:09 PMWarning FailedMount 32s (x23 over 31m) kubelet MountVolume.SetUp failed for volume "pvc-cfd32f84-9d82-413b-b128-d652092345b4" : rpc error: code = Aborted desc = The hot-plug volume pvc-cfd32f84-9d82-413b-b128-d652092345b4 is not ready
error msg. In the Harvester cluster I see that hp-volume-* pod is created but it stuck in ContainerCreating
state with below events:
Type Reason Age From Message
---- ------ ---- ---- -------
Normal Scheduled 20m default-scheduler Successfully assigned stage-lt2-bytc1/hp-volume-njdbl to pl-byt-c1-hp-1-2
Normal SuccessfulMountVolume 20m kubelet MapVolume.MapPodDevice succeeded for volume "pvc-aee63cce-03dc-4a61-a35b-9250667af608" globalMapPath "/var/lib/kubelet/plugins/kubernetes.io/csi/volumeDevices/pvc-aee63cce-03dc-4a61-a35b-9250667af608/dev"
Normal SuccessfulMountVolume 20m kubelet MapVolume.MapPodDevice succeeded for volume "pvc-aee63cce-03dc-4a61-a35b-9250667af608" volumeMapPath "/var/lib/kubelet/pods/dfd363aa-bb46-4cbc-b3c3-3f8da11563c9/volumeDevices/kubernetes.io~csi"
Normal SuccessfulMountVolume 20m kubelet MapVolume.MapPodDevice succeeded for volume "pvc-ec291cc0-cf00-4089-984b-e1adacc15793" globalMapPath "/var/lib/kubelet/plugins/kubernetes.io/csi/volumeDevices/pvc-ec291cc0-cf00-4089-984b-e1adacc15793/dev"
Normal SuccessfulMountVolume 20m kubelet MapVolume.MapPodDevice succeeded for volume "pvc-ec291cc0-cf00-4089-984b-e1adacc15793" volumeMapPath "/var/lib/kubelet/pods/dfd363aa-bb46-4cbc-b3c3-3f8da11563c9/volumeDevices/kubernetes.io~csi"
Normal SuccessfulMountVolume 20m kubelet MapVolume.MapPodDevice succeeded for volume "pvc-bb737df5-3603-42e8-92c2-4d63788909e0" globalMapPath "/var/lib/kubelet/plugins/kubernetes.io/csi/volumeDevices/pvc-bb737df5-3603-42e8-92c2-4d63788909e0/dev"
Normal SuccessfulMountVolume 20m kubelet MapVolume.MapPodDevice succeeded for volume "pvc-bb737df5-3603-42e8-92c2-4d63788909e0" volumeMapPath "/var/lib/kubelet/pods/dfd363aa-bb46-4cbc-b3c3-3f8da11563c9/volumeDevices/kubernetes.io~csi"
Normal SuccessfulMountVolume 20m kubelet MapVolume.MapPodDevice succeeded for volume "pvc-d735d449-c920-4183-8d18-c8ad59438f05" globalMapPath "/var/lib/kubelet/plugins/kubernetes.io/csi/volumeDevices/pvc-d735d449-c920-4183-8d18-c8ad59438f05/dev"
Normal SuccessfulMountVolume 20m kubelet MapVolume.MapPodDevice succeeded for volume "pvc-d735d449-c920-4183-8d18-c8ad59438f05" volumeMapPath "/var/lib/kubelet/pods/dfd363aa-bb46-4cbc-b3c3-3f8da11563c9/volumeDevices/kubernetes.io~csi"
Warning FailedMapVolume 87s (x10 over 19m) kubelet MapVolume.MapPodDevice failed for volume "pvc-bc28369f-7897-4c7e-a829-e854f9ccb217" : rpc error: code = Internal desc = failed to bind mount "/var/lib/kubelet/plugins/kubernetes.io/csi/volumeDevices/staging/pvc-bc28369f-7897-4c7e-a829-e854f9ccb217/pvc-bc28369f-7897-4c7e-a829-e854f9ccb217" at "/var/lib/kubelet/plugins/kubernetes.io/csi/volumeDevices/publish/pvc-bc28369f-7897-4c7e-a829-e854f9ccb217/dfd363aa-bb46-4cbc-b3c3-3f8da11563c9": mount failed: exit status 32
Mounting command: mount
Mounting arguments: -o bind /var/lib/kubelet/plugins/kubernetes.io/csi/volumeDevices/staging/pvc-bc28369f-7897-4c7e-a829-e854f9ccb217/pvc-bc28369f-7897-4c7e-a829-e854f9ccb217 /var/lib/kubelet/plugins/kubernetes.io/csi/volumeDevices/publish/pvc-bc28369f-7897-4c7e-a829-e854f9ccb217/dfd363aa-bb46-4cbc-b3c3-3f8da11563c9
Output: mount: /var/lib/kubelet/plugins/kubernetes.io/csi/volumeDevices/publish/pvc-bc28369f-7897-4c7e-a829-e854f9ccb217/dfd363aa-bb46-4cbc-b3c3-3f8da11563c9: special device /var/lib/kubelet/plugins/kubernetes.io/csi/volumeDevices/staging/pvc-bc28369f-7897-4c7e-a829-e854f9ccb217/pvc-bc28369f-7897-4c7e-a829-e854f9ccb217 does not exist.
dmesg(1) may have more information after failed mount system call.
Any idea of the reason of such behaviour?
Harverster: 1.4.0
Longhorn: 1.7.2