adamant-kite-43734
02/03/2023, 4:52 PMquick-sandwich-76600
02/03/2023, 9:33 PMkubectl get volumes.longhorn.io pvc-xxxxxx -n longhorn-system -o yamlclean-lawyer-76009
02/06/2023, 6:52 AMapiVersion: v1
items:
- apiVersion: v1
kind: PersistentVolumeClaim
metadata:
annotations:
pv.kubernetes.io/bind-completed: "yes"
pv.kubernetes.io/bound-by-controller: "yes"
volume.beta.kubernetes.io/storage-provisioner: driver.longhorn.io
volume.kubernetes.io/storage-provisioner: driver.longhorn.io
creationTimestamp: "2023-02-02T18:50:12Z"
finalizers:
- kubernetes.io/pvc-protection
labels:
app.kubernetes.io/instance: kube-prometheus-stack-prometheus
app.kubernetes.io/managed-by: prometheus-operator
app.kubernetes.io/name: prometheus
operator.prometheus.io/name: kube-prometheus-stack-prometheus
operator.prometheus.io/shard: "0"
prometheus: kube-prometheus-stack-prometheus
name: prometheus-kube-prometheus-stack-prometheus-db-prometheus-kube-prometheus-stack-prometheus-0
namespace: monitoring
resourceVersion: "18457791"
uid: 7707d28d-d4ac-4120-b570-bc4fc50ed95a
spec:
accessModes:
- ReadWriteOnce
resources:
requests:
storage: 20Gi
storageClassName: longhorn
volumeMode: Filesystem
volumeName: pvc-7707d28d-d4ac-4120-b570-bc4fc50ed95a
status:
accessModes:
- ReadWriteOnce
capacity:
storage: 20Gi
phase: Bound
kind: List
metadata:
resourceVersion: ""clean-lawyer-76009
02/06/2023, 3:45 PMWarning FailedMount 59s (x10 over 5m10s) kubelet
MountVolume.MountDevice failed for volume "pvc--6b51e214a520" : rpc error: code = Internal desc = format of disk "/dev/longhorn/pvc-" failed: type:("xfs") target:("/var/lib/kubelet/plugins/kubernetes.io/csi/driver.longhorn.io//globalmount") options:("defaults") errcode:(exit status 1) output:(mkfs.xfs: cannot open /dev/longhorn/pvc-: Device or resource busyclean-lawyer-76009
02/06/2023, 3:50 PMnarrow-egg-98197
02/07/2023, 6:53 AMnarrow-egg-98197
02/07/2023, 7:00 AM/var/lib/kubelet/.../globalmount and /dev/longhorn/pvc-, it is related the NodePublishVolume step in CSI (Container Storage Interface) to bind mount the global directory on a container directory. (ref. https://github.com/kubernetes-csi/docs/issues/24#issuecomment-408342071)clean-lawyer-76009
02/07/2023, 1:44 PMetc/multipath.conf and restarting the service solved the issue.
thanks for your help guys ππ»