Hello! 👋 I am a new user to Longhorn (inherited a system) and am having an issue with one volume which is being mounted and read by 4 different deployments. The PV and PVC exist, but each of the pods has an error similar to this:

Warning  FailedAttachVolume  88s (x10 over 5m43s)  attachdetach-controller  AttachVolume.Attach failed for volume "pvc-5545180c-95e1-48d1-a905-f51380353016" : rpc error: code = DeadlineExceeded desc = volume pvc-5545180c-95e1-48d1-a905-f51380353016 failed to attach to node ip-10-0-3-184

yet otherwise, things generally look healthy. I am not really sure where to start troubleshooting things. I saw one issue that pointed at Linux firewalls causing problems, so I stopped

ufw

(Ubuntu 20.04 system), but that didn't seem to help. I looked at various longhorn pod logs for that node, and didn't notice anything, but any pointers to things to investigate, would be much appreciated. Thanks!

Hello We have been using

longhorn

for our storage needs. Currently we are dealing with this error , any help appreciated?

Warning  ExternalExpanding  81s   volume_expand  Ignoring the PVC: didn't find a plugin capable of expanding the volume; waiting for an external controller to process this PVC

Any recommendation for using longhorn for vault? anyone using it.

If you're interested in hosting an S3 / object storage service on top of a Longhorn PVC, feel free to check out #s3gw here 🙂 It's a fairly new and currently very experimental project.

Hello, I have a crash on Longhorn CSIAttacher when running 1.3.2 on IBMCloud kubernetes service 1.24. I tried with 1.3.1 and 1.3.0 to get the same crash and anything lower than that wont be supported in that version of kubernetes. Can someone point in what I should be doing to debug.

This is the crash. https://pastebin.com/KveMurJU

I was just trying to run the sample pod-with-pvc.yaml after installing longhorn

Events: Type Reason Age From Message ---- ------ ---- ---- ------- Warning ProvisioningFailed 19m (x11 over 33m) driver.longhorn.io_csi-provisioner-c4b9c676b-pvnmr_8f1b5616-a4a4-4fc3-adb0-03fa6561db21 failed to provision volume with StorageClass “longhorn”: rpc error: code = Internal desc = Bad response statusCode [500]. Status [500 Internal Server Error]. Body: [code=Server Error, detail=, message=unable to create volume: unable to create volume pvc-6211ae03-5cc4-4614-a7ba-f4213a298192: Volume.longhorn.io “pvc-6211ae03-5cc4-4614-a7ba-f4213a298192" is invalid: spec.replicaAutoBalance: Unsupported value: “”: supported values: “ignored”, “disabled”, “least-effort”, “best-effort”] from [http://longhorn-backend:9500/v1/volumes]

hello,how to fix this error [spec.replicaAutoBalance: Unsupported value: “”: supported values: “ignored”, “disabled”, “least-effort”, “best-effort”]

{ “apiVersion”: “v1", “kind”: “PersistentVolumeClaim”, “metadata”: { “annotations”: { “volume.beta.kubernetes.io/storage-provisioner”: “driver.longhorn.io”, “volume.kubernetes.io/storage-provisioner”: “driver.longhorn.io” }, “creationTimestamp”: “2022-12-07T091612Z”, “finalizers”: [ “kubernetes.io/pvc-protection” ], “name”: “test”, “namespace”: “test”, “resourceVersion”: “57176383", “uid”: “6211ae03-5cc4-4614-a7ba-f4213a298192" }, “spec”: { “accessModes”: [ “ReadWriteMany” ], “resources”: { “requests”: { “storage”: “10Gi” } }, “storageClassName”: “longhorn”, “volumeMode”: “Filesystem” }, “status”: { “phase”: “Pending” } }

I need to replace the CNI in my cluster that uses Longhorn (will potentially involve new pod IPs) - is there some common wisdom as to how best do this without explicitly scaling down all individual workloads that consume longhorn storage before doing so? I’ve done some test runs and can see Longhorn consistently being able to rebuild all its volumes before restarting workloads attached to those volumes, but I’m somewhat concerned about possible corruption

Hi. Has anyone successfully used longhorn where /var/lib/longhorn on the worker nodes is stored on an NFS volume?

So I came into Longhorn 1.3.2 and a faulted DR volume. This happens often enough, so I delete it and setup a new DR volume. It created and attached the volume right away, but has not started the restore process itself for 10+ minutes at this point. I have tried to restore the same volume as just "restore latest backup" and it seems to be doing the same thing.

I see a lot of "lost track of snapshot" these days. Is that normal for a backup? (unrelated to the above DR restore)

Hi everyone, we just upgraded our 3rd cluster from Longhorn 1.1.1 to 1.2.6 and it went all smoothly. However, we still see some "old" instance-manager pods. We figured that there is 2 instance-managers per Node running which is the case on every updated cluster but there still seems to be some "leftover" of instance managers, they even use the old image. Is this a known issue, can this be somehow resolved?

Good morning - I had some type of issue overnight that caused all my volumes in Longhorn to become 'missing'. The pvc's show up in Rancher, but the pods wont deploy as the volume is missing when I look for them in Longhorn. I can see the PVC in /var/lib/longhorn/replica but I have no idea how the volume detached from the pvc. I have exported the volume from a single replica using the doc online but is there a way to recreate the volume and point it to the already existing PVC in /var/lib/longhorn/replica? Or, do I need to recreate the longhorn pv/pvc and copy the data in myself?

Good day 🙂 Running Harvester v1.1.1 (Longhorn v1.3.2), fresh install today. When trying to create a Backing Image from an existing Volume, either through Harvester or Longhorn directly (export from a longhorn volume), the process runs, but then dies at the end and starts over. I uploaded a windows 2019 image (worked great) and ran it in a VM to customize it, then sys-preped and shutdown (also took a backup and snapshot for good measure). When I try the above to get a new Backing Image it's failing. I've noticed the Longhorn Manager pods are restarting (so far 5-8 times). Longhorn Manager:

msg="Can't find backing image manager for pod, may be deleted" controller=longhorn-backing-image-manager node=harvester-gtsgd pod=backing-image-manager-d7ad-bfcf

Backing Image Manager:

msg="SyncingFile: failed to get the checksum from a valid config during processing wrap-up, will directly calculated it then"

I'm still fairly novice in this eco-system, but I'll to my best to get more details if pointed in the right direction.

hello, I am adding a new "disk". Should the disks have unique names for every node? Because I set them up with the same

Hello all I am trying to do a deployment with a persistent storage using default longhorn storage class. Initially everything worked and I still see successfully deployed persistent based workload but all new ones are failing with Below errors. AttachVolume.Attach failed for volume “PVC-xxx”: rpc error: code = NotFound desc= node xxxxx not found Any help/suggestions would be appreciated.

is there a way to prevent volumes created on a disk with a tag, when the storageclass used has no diskselector parameter? I have a setup with 2 disks per node - one is tagged and the other is not. I was expecting volumes to be created on the tagged disk only when storageclass with the appropriate diskselector tag is used, but I see that is not the case is this a bug, or expected behavior?

Hi all! I started using longhorn a few days ago on k3s single node setup. I really like it so I decided to expand the cluster to a k3s etcd 3 master nodes setup. I installed ubuntu 22.04 and installed open-iscsi as described in the docs, but when the cluster starts to run the new longhorn pods on the new nodes, the

longhorn-manager

fails to get ready, unable to list/watch resources, showing various errors such as:

W1224 07:44:23.287997       1 client_config.go:617] Neither --kubeconfig nor --master was specified.  Using the inClusterConfig.  This might not work.
time="2022-12-24T07:44:23Z" level=info msg="cannot list the content of the src directory /var/lib/rancher/longhorn/engine-binaries for the copy, will do nothing: failed to execute: nsenter [--mount=/host/proc/1/ns/mnt --net=/host/proc/1/ns/net bash -c ls /var/lib/rancher/longhorn/engine-binaries/*], output , stderr, ls: cannot access '/var/lib/rancher/longhorn/engine-binaries/*': No such file or directory\n, error exit status 2"
time="2022-12-24T07:44:24Z" level=info msg="Skip the leader election for the upgrade since the current Longhorn system is already up to date"
W1224 07:44:24.943618       1 client_config.go:617] Neither --kubeconfig nor --master was specified.  Using the inClusterConfig.  This might not work.
I1224 07:44:24.961708       1 shared_informer.go:240] Waiting for caches to sync for longhorn datastore
W1224 07:44:25.003320       1 reflector.go:324] <http://k8s.io/client-go/informers/factory.go:134|k8s.io/client-go/informers/factory.go:134>: failed to list *v1beta1.CronJob: the server could not find the requested resource
E1224 07:44:25.003407       1 reflector.go:138] <http://k8s.io/client-go/informers/factory.go:134|k8s.io/client-go/informers/factory.go:134>: Failed to watch *v1beta1.CronJob: failed to list *v1beta1.CronJob: the server could not find the requested resource
W1224 07:44:25.020929       1 reflector.go:324] <http://k8s.io/client-go/informers/factory.go:134|k8s.io/client-go/informers/factory.go:134>: failed to list *v1beta1.PodDisruptionBudget: the server could not find the requested resource
E1224 07:44:25.020957       1 reflector.go:138] <http://k8s.io/client-go/informers/factory.go:134|k8s.io/client-go/informers/factory.go:134>: Failed to watch *v1beta1.PodDisruptionBudget: failed to list *v1beta1.PodDisruptionBudget: the server could not find the requested resource
E1224 07:44:25.027597       1 kubernetes_node_controller.go:227] couldn't get kubernetes node master1: node "master1" not found 
time="2022-12-24T07:44:25Z" level=warning msg="Can't find instance manager for pod instance-manager-e-e867fc5d, may be deleted"
time="2022-12-24T07:44:25Z" level=warning msg="Can't find instance manager for pod instance-manager-r-10c2a1ba, may be deleted"

Is it something I did wrong while installing? Any hints what may have gotten wrong?

reply myself for people having the smae issue to me. longhorn works not in a common way. actually you have to install longhorn system components on all worker nodes so that to provision pv to pods. refer to this github issue for solutions https://github.com/longhorn/longhorn/issues/2664

anyone having issues exporting 100gb volumes? i keep getting to 28% and then the manager pod for longhorn craps out.

Node harvester-01 is ready	2.2 mins ago 
Node harvester-01 is ready	2.2 mins ago 
Node harvester-01 is ready	2.2 mins ago 
Node harvester-01 is ready	2.2 mins ago 
Node harvester-01 is down: the manager pod longhorn-manager-fdc28 is not running	2.2 mins ago 
Node harvester-01 is down: the manager pod longhorn-manager-fdc28 is not running	2.2 mins ago 
Node harvester-01 is down: the manager pod longhorn-manager-fdc28 is not running	2.2 mins ago

in the longhorn ui, the backing image its trying to create gets to 28%, then dies, and starts over

Need help with using the CSI driver: In an RKE2 cluster running on Harvester I would like to allow the cluster to create volumes on the longhorn storage created by harvester. I was told that I can use the CSI driver as per https://docs.harvesterhci.io/v1.1/rancher/csi-driver/#deploying-with-harvester-k3s-node-driver. Unfortunately I am unable to get this to work because I created the cluster using Rancher's "Use existing nodes and create a cluster using RKE2/K3s" . Apparently if you already have an existing cluster you can use the method described for "Harvester K3s Node Driver". I am able to deploy the driver, but when the cluster tries to create the volume it fails even though I can see the correct storage driver.

kubectl describe sc
Name:                  harvester
IsDefaultClass:        Yes
Annotations:           <http://meta.helm.sh/release-name=harvester-csi-driver,meta.helm.sh/release-namespace=kube-system,storageclass.kubernetes.io/is-default-class=true|meta.helm.sh/release-name=harvester-csi-driver,meta.helm.sh/release-namespace=kube-system,storageclass.kubernetes.io/is-default-class=true>
Provisioner:           <http://driver.harvesterhci.io|driver.harvesterhci.io>
Parameters:            <none>
AllowVolumeExpansion:  True
MountOptions:          <none>
ReclaimPolicy:         Delete
VolumeBindingMode:     Immediate
Events:                <none>

Hi Ranchers , i'm trying to implement longhorn csi with rancher cluster the case is that i have an encrypted disk with luks already, the provisioned volumes are faulted, once i have removed luks from disk every this came to healthy and i also found that longhorn can encrypt volumes by it self so is the encrypted disk don't work with longhorn and if i need encryption use longhorn encryption, please support ??

hi, We have deployed K3s with Longhorn in a production environment and we are loosing Volumes every other hour. With Default to 3 Replicas , we loose the volumes being unable to coop up with our workload.2ndly, the longhorn starts taking tooo much memory that the node goes out of control and we have to reboot at least 1 node per day.

Can we apply resource limits on the longhorn pod?

or make it more suitable for production workload.

any help

I am okay to pay for any guidance