clean-lawyer-76009
02/03/2023, 4:52 PMNot ready for workload, Volume Faulted
the output of csi attacher log is:
I0203 10:10:21.981136 1 csi_handler.go:231] Error processing "csi-": failed to attach: rpc error: code = Aborted desc = volume pvc- is not ready for workloads
The folder on the host belongs to root user in group root
How can i drill this issue further down?
Could this be a permission issue?quick-sandwich-76600
02/03/2023, 9:33 PMkubectl get <http://volumes.longhorn.io|volumes.longhorn.io> pvc-xxxxxx -n longhorn-system -o yaml
clean-lawyer-76009
02/06/2023, 6:52 AMapiVersion: v1
items:
- apiVersion: v1
kind: PersistentVolumeClaim
metadata:
annotations:
<http://pv.kubernetes.io/bind-completed|pv.kubernetes.io/bind-completed>: "yes"
<http://pv.kubernetes.io/bound-by-controller|pv.kubernetes.io/bound-by-controller>: "yes"
<http://volume.beta.kubernetes.io/storage-provisioner|volume.beta.kubernetes.io/storage-provisioner>: <http://driver.longhorn.io|driver.longhorn.io>
<http://volume.kubernetes.io/storage-provisioner|volume.kubernetes.io/storage-provisioner>: <http://driver.longhorn.io|driver.longhorn.io>
creationTimestamp: "2023-02-02T18:50:12Z"
finalizers:
- <http://kubernetes.io/pvc-protection|kubernetes.io/pvc-protection>
labels:
<http://app.kubernetes.io/instance|app.kubernetes.io/instance>: kube-prometheus-stack-prometheus
<http://app.kubernetes.io/managed-by|app.kubernetes.io/managed-by>: prometheus-operator
<http://app.kubernetes.io/name|app.kubernetes.io/name>: prometheus
<http://operator.prometheus.io/name|operator.prometheus.io/name>: kube-prometheus-stack-prometheus
<http://operator.prometheus.io/shard|operator.prometheus.io/shard>: "0"
prometheus: kube-prometheus-stack-prometheus
name: prometheus-kube-prometheus-stack-prometheus-db-prometheus-kube-prometheus-stack-prometheus-0
namespace: monitoring
resourceVersion: "18457791"
uid: 7707d28d-d4ac-4120-b570-bc4fc50ed95a
spec:
accessModes:
- ReadWriteOnce
resources:
requests:
storage: 20Gi
storageClassName: longhorn
volumeMode: Filesystem
volumeName: pvc-7707d28d-d4ac-4120-b570-bc4fc50ed95a
status:
accessModes:
- ReadWriteOnce
capacity:
storage: 20Gi
phase: Bound
kind: List
metadata:
resourceVersion: ""
Warning FailedMount 59s (x10 over 5m10s) kubelet
MountVolume.MountDevice failed for volume "pvc--6b51e214a520" : rpc error: code = Internal desc = format of disk "/dev/longhorn/pvc-" failed: type:("xfs") target:("/var/lib/kubelet/plugins/kubernetes.io/csi/driver.longhorn.io//globalmount") options:("defaults") errcode:(exit status 1) output:(mkfs.xfs: cannot open /dev/longhorn/pvc-: Device or resource busy
narrow-egg-98197
02/07/2023, 6:53 AM/var/lib/kubelet/.../globalmount
and /dev/longhorn/pvc-
, it is related the NodePublishVolume step in CSI (Container Storage Interface) to bind mount the global directory on a container directory. (ref. https://github.com/kubernetes-csi/docs/issues/24#issuecomment-408342071)clean-lawyer-76009
02/07/2023, 1:44 PMetc/multipath.conf
and restarting the service solved the issue.
thanks for your help guys 👍🏻