I am still running into some issues with MultiAttachment. It seems some volumes don't get detached properly, or fast enough. I see for instance that kind of error in the logs of the csi-attacher pods

Failed to save detach error to "csi-cf8e26e84124ff4cbbc0598e7949327d8d5f8fbbe83996f006043e22eb70360b": <http://volumeattachments.storage.k8s.io|volumeattachments.storage.k8s.io> "csi-cf8e26e84124ff4cbbc0598e7949327d8d5f8fbbe83996f006043e22eb70360b" not found

I can't figure out if: • it's the root cause of other later attachment on the same pvc/pv • why this error shows up

Hi,

When longhorn in racher upgerade to 1.4.3?

Hi folks, i was trying to deploy a longhorn using fleet

I think some garbage was left on my cluster form an old deployment through rancher interface

no i'm trying again to deploy through rancher apps

2023-08-09T164221.491600267-03:00 time="2023-08-09T194221Z" level=fatal msg="Error starting manager: upgrade resources failed: admission webhook \"validator.longhorn.io\" denied the request: spec and status of disks on node XXXXXX are being syncing and please retry later."

but node XXXXX is not part of this deployment, i'm selection some nodes with labels and nodeselector

there is a way to cleanup all crds and other obscure resources from longhorn? Using unisntall from helm (rancher charts) is not helping

don't bother guys

just figured out

after helm unistall with --no-hooks removed all /var/lib/longhorn/ on nodes + removed all componentes after editing validator and mutating webhooks

everything worked =d

I had this issue happening multiples times in the last few months on our kubernetes cluster 1.26.6 on microk8s with longhorn. It complains about the folder

lost+found

and when I check on the kubelet, that folder is showing a input/output error like in the kubernetes event.

sudo ls -al "/var/snap/microk8s/common/var/lib/kubelet/pods/8721351a-d14e-4da3-81a9-b3e06a701d04/volumes/kubernetes.io~csi/pvc-2e1be23b-4021-45d2-878b-d37cd71679dc/mount/lost+found"
ls: reading directory '/var/snap/microk8s/common/var/lib/kubelet/pods/8721351a-d14e-4da3-81a9-b3e06a701d04/volumes/kubernetes.io~csi/pvc-2e1be23b-4021-45d2-878b-d37cd71679dc/mount/lost+found': Input/output error

The related longhorn volume and replicas are not showing any errors.

unrelated to my issue above, what are the recommendations to upgrade longhorn on a production cluster using 1.26.6? I have version 1.4.0-dev installed now. It was installed using the helm chart from the github repo with only one change in the deployment to make it work with microk8s.

- name: KUBELET_ROOT_DIR
  value: /var/snap/microk8s/common/var/lib/kubelet

I'd like to at least use the last 1.4.x version.

Holly crap I noticed that our prod cluster is not a specific version of longhorn but basically whatever is latest. The helm chart used during installation a long time ago was pointing to master-head.... So now the helm chart says it's version 1.4.0-dev but the actually running version is 1.5.0-dev and eveyrthing is master-head.... a mess! How do I get out of this nicely? backup the volumes and reinstall version 1.5.1 ?

I am trying to upgrade longhorn to 1.5.1 using helm. using

helm upgrade longhorn ./longhorn/chart/ \
  --namespace longhorn-system \
  --values values.yaml

I have the Chart.yaml, questions.yaml and values.yaml in the chart directory, from the 1.5.1 tag on github in my values.yaml I only cusotmize the kubeletRootDir and the loglevel. Helm didn't complain and executed. Now I have no longhorn UI, all containers have the same version, and one provisioner is stuck in crashloop. I definitely need help here...

My upgrade attempt above went very bad. I am now just trying to recover the cluster enough so I can reinstall longhorn and the other apps that were using it. But i am stuck with lot of resources in the longhorn-system namespace. Setting the finalizers to empty and forcing deletes doesn't help...

I feel like I am monopolizing this channel lately 🙂 I believe the answer to the following is no, but in case, I had done backups with the previous deployment of longhorn on a NFS server. Those backups are completely unusable by another deployment of longhorn? can the data be read somehow and mounted on a pod or something?

Apologies if this has been asked before but I was wondering if there needs to be a 3 replica deployment of the csi-* pods for single node k8s clusters? Is 3 replicas arbitrary? Should it scale with the size of the cluster?

$ kubectl get deployment -n longhorn-system 
NAME                       READY   UP-TO-DATE   AVAILABLE   AGE
longhorn-ui                1/1     1            1           54d
longhorn-driver-deployer   1/1     1            1           54d
csi-attacher               3/3     3            3           3h27m
csi-provisioner            3/3     3            3           3h26m
csi-resizer                3/3     3            3           3h26m
csi-snapshotter            3/3     3            3           3h26m

What's the best practice for rebooting Longhorn nodes? We have 20+ Longhorn nodes with 2TB data disks on each, when rebooting a node I've been cordoning/draining in Rancher, then just rebooting it. This puts the attached replicas on that node in an error state and rebuilds them on a new node. That's fine but with 2TB of data and 100 to 200 replicas on a single node that takes a loooong time, and you can't do any more than 2 at a time otherwise you risk data loss.

Question on backups via the Longhorn UI ( or Kubectl backup related objects ) … the Longhorn docs mentions that for CSI snapshots > https://longhorn.io/docs/1.4.3/snapshots-and-backups/csi-snapshot-support/csi-volume-snapshot-associated-with-longhorn-snapshot/#current-limitation < that they only work when the

Volume/Volume's

is attached. Is it the same thing for the

Create Backup

option in the

Volume

tab through the Longhorn UI. As, if I click on an

Detached

Volume

that button is greyed out. Whereas for a

Attached Volume

it’s available. Thank you very much

Hi all! My volume is detach-attach loop. My longhorn is 1.3.3. I want to check volume.img. How can I use method that check volume.img?

Hello everyone, I'm currently working with kubevirt and longhorn, and testing kubevirt recovery with longhorn volume when i shutdown the node's runnning pod. I have to force delete the terminating vm pod. When the new kubevirt pod is scheduled, longhorn take 7 minutes to detach the old volume from the "not ready" node. I don't understand why, can somebody help me ?

hi everyone, anyone know how to deal with: https://github.com/longhorn/longhorn/issues/5388 cannot get PVC to unmount (failed to list snapshot: cannot get client for volume pvc-{id}: more than one engine exists) #5388 running into the issue again, not sure what to do.

Hi guys! We want to use minio s3 backup to restore cluster A backups to cluster B. Is there any best practice on this? We have ugly error messages in the longhorn-manager pods:

time="2023-08-31T09:22:42Z" level=info msg="Found 22 backups in the backup target that do not exist in the cluster and need to be pulled" backupVolume=pvc-609e82a6-853e-43c7-b2d4-cabad74304a6 controller=longhorn-backup-volume node=tst-agent3
time="2023-08-31T09:22:42Z" level=info msg="Listening on 10.42.5.213:9500" node=tst-agent3
time="2023-08-31T09:22:42Z" level=info msg="Debug Server listening on 127.0.0.1:6060" node=tst-agent3
time="2023-08-31T09:22:42Z" level=info msg="Cron is changed from  to 0 0 */7 * *. Next snapshot check job will be executed at 2023-09-01 00:00:00 +0000 UTC" controller=longhorn-node monitor="snapshot monitor" node=tst-agent3
E0831 09:22:43.354415       1 runtime.go:78] Observed a panic: "invalid memory address or nil pointer dereference" (runtime error: invalid memory address or nil pointer dereference)
goroutine 905 [running]:
<http://k8s.io/apimachinery/pkg/util/runtime.logPanic({0x1d7d280|k8s.io/apimachinery/pkg/util/runtime.logPanic({0x1d7d280>, 0x3566c30})
        /go/src/github.com/longhorn/longhorn-manager/vendor/k8s.io/apimachinery/pkg/util/runtime/runtime.go:74 +0x7d
<http://k8s.io/apimachinery/pkg/util/runtime.HandleCrash({0x0|k8s.io/apimachinery/pkg/util/runtime.HandleCrash({0x0>, 0x0, 0xc000fa6c90})
        /go/src/github.com/longhorn/longhorn-manager/vendor/k8s.io/apimachinery/pkg/util/runtime/runtime.go:48 +0x75
panic({0x1d7d280, 0x3566c30})
        /usr/local/go/src/runtime/panic.go:1038 +0x215
<http://github.com/longhorn/longhorn-manager/controller.(*BackupVolumeController).reconcile(0xc000621f80|github.com/longhorn/longhorn-manager/controller.(*BackupVolumeController).reconcile(0xc000621f80>, {0xc000ad08d0, 0x28})
        /go/src/github.com/longhorn/longhorn-manager/controller/backup_volume_controller.go:326 +0x1f2d
<http://github.com/longhorn/longhorn-manager/controller.(*BackupVolumeController).syncHandler(0xc000621f80|github.com/longhorn/longhorn-manager/controller.(*BackupVolumeController).syncHandler(0xc000621f80>, {0xc000ad08c0, 0x8})
        /go/src/github.com/longhorn/longhorn-manager/controller/backup_volume_controller.go:146 +0x118
<http://github.com/longhorn/longhorn-manager/controller.(*BackupVolumeController).processNextWorkItem(0xc000621f80)|github.com/longhorn/longhorn-manager/controller.(*BackupVolumeController).processNextWorkItem(0xc000621f80)>
        /go/src/github.com/longhorn/longhorn-manager/controller/backup_volume_controller.go:128 +0xdb
<http://github.com/longhorn/longhorn-manager/controller.(*BackupVolumeController).worker(...)|github.com/longhorn/longhorn-manager/controller.(*BackupVolumeController).worker(...)>
        /go/src/github.com/longhorn/longhorn-manager/controller/backup_volume_controller.go:118
<http://k8s.io/apimachinery/pkg/util/wait.BackoffUntil.func1(0x7fc3a3ea7e18)|k8s.io/apimachinery/pkg/util/wait.BackoffUntil.func1(0x7fc3a3ea7e18)>
        /go/src/github.com/longhorn/longhorn-manager/vendor/k8s.io/apimachinery/pkg/util/wait/wait.go:155 +0x67
<http://k8s.io/apimachinery/pkg/util/wait.BackoffUntil(0xc0005aa180|k8s.io/apimachinery/pkg/util/wait.BackoffUntil(0xc0005aa180>, {0x234da20, 0xc002c19d70}, 0x1, 0xc0005aa180)
        /go/src/github.com/longhorn/longhorn-manager/vendor/k8s.io/apimachinery/pkg/util/wait/wait.go:156 +0xb6
<http://k8s.io/apimachinery/pkg/util/wait.JitterUntil(0xc000bab7b0|k8s.io/apimachinery/pkg/util/wait.JitterUntil(0xc000bab7b0>, 0x3b9aca00, 0x0, 0x0, 0xc000bab7d0)
        /go/src/github.com/longhorn/longhorn-manager/vendor/k8s.io/apimachinery/pkg/util/wait/wait.go:133 +0x89
<http://k8s.io/apimachinery/pkg/util/wait.Until(0x8506aa|k8s.io/apimachinery/pkg/util/wait.Until(0x8506aa>, 0xc000125d40, 0x0)
        /go/src/github.com/longhorn/longhorn-manager/vendor/k8s.io/apimachinery/pkg/util/wait/wait.go:90 +0x25
created by <http://github.com/longhorn/longhorn-manager/controller.(*BackupVolumeController).Run|github.com/longhorn/longhorn-manager/controller.(*BackupVolumeController).Run>
        /go/src/github.com/longhorn/longhorn-manager/controller/backup_volume_controller.go:112 +0x208
panic: runtime error: invalid memory address or nil pointer dereference [recovered]
        panic: runtime error: invalid memory address or nil pointer dereference
[signal SIGSEGV: segmentation violation code=0x1 addr=0x70 pc=0x18d99cd]

Hello People, we performed a K8s Update 1.24 -> 1.25 using rke1 and this caused some Pods with volumes attached to fail. Now some of them are stuck in container creating with this error:

/dev/longhorn/pvc-9a6a41b0-72f3-4c69-baa0-08b4ddf79fa0 is mounted; will not make a filesystem here!

. We are running on Longhorn 1.4.3

I need to retrieve some data using the replica files. the doc says to do something like this

docker run \
	-v /dev:/host/dev \
	-v /proc:/host/proc \
	-v ${REPLICA_PATH}:/volume \
	--privileged \
	longhornio/longhorn-engine:v1.5.1 \
	launch-simple-longhorn ${VOLUME_ID} ${VOLUME_SIZE}

2 questions: • is there an equivalent for

crictl

? • to avoid the --privileged, can it be done in a totally different folder than /dev or it need devfs?

Hi all, we are testing a scenario where we have a 3 node k3s cluster running longhorn with volume replicas on each node, if one of the nodes becomes unreachable (e.g. because of hardware failure) we have configured pods so that they are rescheduled to another node, the pods on the unreachable node enter

Terminating

state and new pods are provisioned however longhorn is unable to attach volume replica to the newly created pod because Kubernetes reports that it is still attached to the Terminating pod. Any suggestions for how we might overcome this are welcome 🙂

I'm running v1.5.1 on K3s v1.28.2+k3s1 on Alma 9.2. It's a fresh cluster install. The first workload to spin up a volume is running into issues where the PVC shows that it's bound, but the Pod refuses to start. The Pod event log shows that it gets scheduled to a node, but it never starts. I occasionally see an event pop up about storage, but I can't get it to come back now. I dug through some of the longhorn logs and I saw some messages pop up and am not sure what to make of them (will post in thread)