rancher-users #longhorn-storage

hundreds-airport-66196

02/03/2023, 3:31 PM

Hello, I have a 3-node rke2 cluster. Its using GlusterFS and planning of replacing it with Longhorn. Is longhorn better suited to have it running in another 3-node storage cluster? or it should be ok running in the existing cluster.

sparse-businessperson-74827

02/03/2023, 3:36 PM

Hi, Is there a guide on how to configure nfs export in Longhorn. I have 1.4.0 running and have TVX volume created but cannot figure out how to present it as nfs. Looking at the docs I should have share-manager pod running but I do no have it. Also once nfs export is created can I mount it from outside the cluster?

clean-lawyer-76009

02/03/2023, 4:52 PM

Hey, i installed longhorn v.1.4.0 on a Bare Metal Kubernetes Cluster with RHEL8. When the cluster creates a pv from the longhorn storageclass the pv state is bound, but in longhorn i see the state of the volume is Faulted:

Copy code

Not ready for workload, Volume Faulted

the output of csi attacher log is:

Copy code

I0203 10:10:21.981136       1 csi_handler.go:231] Error processing "csi-": failed to attach: rpc error: code = Aborted desc = volume pvc- is not ready for workloads

The folder on the host belongs to root user in group root How can i drill this issue further down? Could this be a permission issue?

late-needle-80860

02/04/2023, 9:07 PM

https://github.com/longhorn/longhorn/issues/5349 < interesting case I just bumped into.

crooked-cat-21365

02/06/2023, 11:35 AM

Hi folks, what is the recommendation for setting node.session.timeo.replacement_timeout in /etc/iscsi/iscsid.conf? The default is 120 secs, but my Dell servers usually take at least 5 minutes for rebooting, so this setting doesn't seem reasonable.

stocky-article-82001

02/09/2023, 5:18 PM

I’ve got a volume stuck in a detaching -> attached -> failure loop and I can’t salvage it because it keeps looping

stocky-article-82001

02/09/2023, 5:19 PM

any advice is greatly appreciated.

stocky-article-82001

02/09/2023, 5:27 PM

If I do manage to catch the “Salvage” button, I see this

unable to salvage volume pvc-87002c74-b16f-431e-af75-6f9b1f461616: invalid volume state to salvage: attaching

stocky-article-82001

02/09/2023, 5:32 PM

just got

unable to salvage volume pvc-87002c74-b16f-431e-af75-6f9b1f461616: invalid robustness state to salvage: unknown

too

stocky-article-82001

02/09/2023, 5:32 PM

i wonder if it’s too far gone

quaint-alarm-7893

02/10/2023, 6:19 AM

hello everyone, i have a vm (in harvesteR) that got messed up when one of my servers had an issue, and now the VM is off, but the PVC is still attached. how can i detach it so i can bring the vm back online?

Copy code

State: Attached
Health:
No node redundancy: all the healthy replicas are running on the same node Degraded
Ready for workload:Not Ready
Conditions:
restore
scheduled
Frontend:Block Device
Attached Node & Endpoint:
Size:
100 Gi
Actual Size:Unknown
Data Locality:disabled
Access Mode:ReadWriteMany
Backing Image:vdi-image-hzwdh
Backing Image Size:50 Gi
Engine Image:longhornio/longhorn-engine:v1.3.2
Created:a month ago
Encrypted:False
Node Tags:
Disk Tags:
Last Backup:
Last Backup At:
Replicas Auto Balance:ignored
Instance Manager:
instance-manager-e-c2f010c0
instance-manager-e-50d28d2d
Last time used by Pod:10 hours ago
Namespace:vdi
PVC Name:vmname-rootdisk-ossjw
PV Name:pvc-13787ebc-7d4e-41ad-9206-de9a6adb938a
PV Status:Bound
Revision Counter Disabled:False
Last Pod Name:virt-launcher-vmname-mpbn2
Last Workload Name:vmname
Last Workload Type:VirtualMachineInstance

note, it show's its attached to tow instance managers. i've tried rebooting the nodes, and deleting the instance manager pods, no luck...

sparse-fireman-14239

02/10/2023, 1:27 PM

It appears that the Longhorn v1.4.0 Helm chart is now available through Rancher Manager :-)

🎉 1

mysterious-rose-43856

02/10/2023, 5:13 PM

I have longhorn 1.4.0 on 4 clusters or so and have experienced unfortunate downtime that is not expected. I the volumes show as "Attached" but not ready, their workloads will not spin down. I'm investigating, but this is most unfortunate as I'm discovering this mid-migration for some important work loads.

quaint-alarm-7893

02/10/2023, 11:18 PM

question. i have a store class i setup for HDD's since rebooting the node, it seems Longhorn now wasnt so schedule other volumes onto that disk. am i missing something? Disk has a tag of hdd, node has a tag of lff. the class has both of those sets. the volumes being scheduled onto the disk are part of the harvester-longhorn default class. not sure why they keep getting put onto the hdd, but they build to 99%, then sit there forever....

mysterious-rose-43856

02/11/2023, 4:29 PM

My Ongoing issues may be snapshot related.

broad-machine-78396

02/13/2023, 1:29 PM

Hello everyone 👋, what is the best practice for a production cluster to setup longhorn, do i need to have 3 dedicated worker node only for longhorn (with taint) or do i need to schedule pod on normal node ?

👀 1

hundreds-airport-66196

02/14/2023, 6:06 PM

Hi All, I have a 3-node cluster with rke2. Planning of using Longhorn for PVs. How can one volume be access by any of the pods running on each of the nodes and have Read/Write access? Or do I need 3 volumes for each pod and replication takes place behind the scene. Thanks!

loud-helmet-97067

02/16/2023, 1:36 AM

Hi team, We have requirement, where we have provision k3s cluster with x2 master nodes and x3 worker nodes for storage cluster with longhorn and we need this storage cluster to be used / provision in another kubernetes cluster ( workload cluster) PV / PVC claims. The idea is to provide PV/ PVC claims for software stack such as harbor , ELK ( elastic search), custom application workload that need storage for batch processing; Does it make sense to; • Option 1: Connect k3s - storage cluster to workload cluster (2nd k8s cluster) via means of NFS service from ingress ( or any other means) for production work ( i would like to keep longhorn capabilities on top and expose the storage through nfs-server. I saw somewhere that longhorn itself has nfs-server pod as well with longhorn stack) • Option 2: Create workload pods in same k3s - storage cluster having seperate nodes ( by grouping / labeling nodes) to run application workload • Option 3: Any other options?? Any insights/ feedback in to identify the correct approach is highly appreciated.

quaint-alarm-7893

02/22/2023, 2:14 PM

hello everyone. can anyone help on this issue? https://github.com/longhorn/longhorn/issues/5413

👀 1

quaint-alarm-7893

02/23/2023, 9:19 PM

@narrow-egg-98197 you messaged me about his before, any thought? ^

quaint-alarm-7893

02/23/2023, 11:59 PM

sorry, didnt mean to spam the channel^

enough-memory-12110

03/01/2023, 6:48 AM

I am having problems with longhorn 1.2.2: I tried setting up a backup target on a custom S3 endpoint. But we're getting an error:

error listing backup volume names: Timeout executing: /var/lib/longhorn/engine-binaries/rancher-mirrored-longhornio-longhorn-engine-v1.2.2/longhorn [backup ls --volume-only <s3://backups@us-east-1/>], output , stderr, , error <nil>

I tried logging in to the container and checking the probelm from its side and this is what we get:

Copy code

root@longhorn-manager-57lhs:/# /var/lib/longhorn/engine-binaries/rancher-mirrored-longhornio-longhorn-engine-v1.2.2/longhorn backup ls --volume-only <s3://backup/>
ERRO[0020] Fail to list s3: NoCredentialProviders: no valid providers in chain. Deprecated.
	For verbose messaging see aws.Config.CredentialsChainVerboseErrors  pkg=s3

I am sure the credentials in secret are valid and properly formatted ( I've copied minio configuration and adjust to our environment ). Can anyone from here help us diagnose the problem further and solve it? (edited)

blue-painting-33432

03/03/2023, 12:23 AM

Hello, I'm having quite a urgent issue here with a basic problem, anyone online to help? Using longhorn on microk8s with local node storage, it worked fine for a few weeks, but now I restarted a pod and the volume just won't attach anymore, it timeout and restart the pod and keeps looping like this. In the longhorn UI I only see "attached" and "detached" events repeating all the time, but no error.

blue-painting-33432

03/03/2023, 12:23 AM

Copy code

Unable to attach or mount volumes: unmounted volumes=[alertmanager-kube-prometheus-stack-alertmanager-db], unattached volumes=[web-config kube-api-access-8lfhh config-volume config-out tls-assets alertmanager-kube-prometheus-stack-alertmanager-db]: timed out waiting for the condition

blue-painting-33432

03/03/2023, 12:24 AM

any way to increase the timeout waiting for the volume mount or something?

blue-painting-33432

03/03/2023, 12:30 AM

pod remain in pending and containercreating for a few seconds, then immediately goes to deleting and start again, always failing attaching the volume, but longhorn seems to say it is attached with no error

blue-painting-33432

03/03/2023, 1:05 AM

Ok nevermind, I installed an helm chart that installed a bunch of namespaces and ressources I didn't want, including a second prometheus instance in another namespace, and uninstalling the helm release left those ressource in the cluster and seemed to be causing issue with my real prometheus instance volumes. What a mess

acceptable-soccer-28720

03/03/2023, 2:04 AM

Restart of the node and dettach/attach ofmthe volume did not hell to solve the situation

rancher kubectl describe pod gitlab-postgresql-0 -n gitlab

Copy code

Events:
  Type     Reason              Age                  From                     Message
  ----     ------              ----                 ----                     -------
  Normal   Scheduled           14m                  default-scheduler        Successfully assigned gitlab/gitlab-postgresql-0 to vik8scases-w-2
  Warning  FailedMount         10m                  kubelet                  Unable to attach or mount volumes: unmounted volumes=[data], unattached volumes=[kube-api-access-g28wp custom-init-scripts postgresql-password dshm data]: timed out waiting for the condition
  Warning  FailedMount         7m49s                kubelet                  Unable to attach or mount volumes: unmounted volumes=[data], unattached volumes=[postgresql-password dshm data kube-api-access-g28wp custom-init-scripts]: timed out waiting for the condition
  Warning  FailedMount         5m34s                kubelet                  Unable to attach or mount volumes: unmounted volumes=[data], unattached volumes=[dshm data kube-api-access-g28wp custom-init-scripts postgresql-password]: timed out waiting for the condition
  Warning  FailedMount         3m18s (x2 over 12m)  kubelet                  Unable to attach or mount volumes: unmounted volumes=[data], unattached volumes=[data kube-api-access-g28wp custom-init-scripts postgresql-password dshm]: timed out waiting for the condition
  Warning  FailedAttachVolume  2m9s (x6 over 12m)   attachdetach-controller  AttachVolume.Attach failed for volume "pvc-b59dafa1-3efa-44fc-92ba-e2be23e5d4a4" : timed out waiting for external-attacher of <http://driver.longhorn.io|driver.longhorn.io> CSI driver to attach volume pvc-b59dafa1-3efa-44fc-92ba-e2be23e5d4a4
  Warning  FailedMount         64s                  kubelet                  Unable to attach or mount volumes: unmounted volumes=[data], unattached volumes=[custom-init-scripts postgresql-password dshm data kube-api-access-g28wp]: timed out waiting for the condition

plain-breakfast-5576

03/03/2023, 6:02 PM

@famous-journalist-11332 we are seeing deadlock while upgrading from 1.2.2. To 1.2.3. Do u have any Upton this? If not can you plz update the kb Longhorn.io/troubleshooting-engine-upgrading-struck-in-deadlock

rich-shoe-36510

03/08/2023, 11:03 AM

Hey people! One simple question, for which I can't find simple answer. I have running Longhorn (v1.4.0) with one hard drive, everything is fine. Now, I added hard drive, the question is how to add additional disk to the node? Can it be done trough UI, how to bind-mount two hdd mounts to /var/lib/longhorn? Thanks!