https://rancher.com/ logo
Docs
Join the conversationJoin Slack
Channels
academy
amazon
arm
azure
cabpr
chinese
ci-cd
danish
deutsch
developer
elemental
epinio
espanol
events
extensions
fleet
français
gcp
general
harvester
harvester-dev
hobbyfarm
hypper
japanese
k3d
k3os
k3s
k3s-contributor
kim
kubernetes
kubewarden
lima
logging
longhorn-dev
longhorn-storage
masterclass
mesos
mexico
nederlands
neuvector-security
office-hours
one-point-x
onlinemeetup
onlinetraining
opni
os
ozt
phillydotnet
portugues
rancher-desktop
rancher-extensions
rancher-setup
rancher-wrangler
random
rfed_ara
rio
rke
rke2
russian
s3gw
service-mesh
storage
submariner
supermicro-sixsq
swarm
terraform-controller
terraform-provider-rancher2
terraform-provider-rke
theranchcast
training-0110
training-0124
training-0131
training-0207
training-0214
training-1220
ukranian
v16-v21-migration
vsphere
windows
Powered by Linen
longhorn-storage
  • h

    hundreds-airport-66196

    02/03/2023, 3:31 PM
    Hello, I have a 3-node rke2 cluster. Its using GlusterFS and planning of replacing it with Longhorn. Is longhorn better suited to have it running in another 3-node storage cluster? or it should be ok running in the existing cluster.
    c
    • 2
    • 2
  • s

    sparse-businessperson-74827

    02/03/2023, 3:36 PM
    Hi, Is there a guide on how to configure nfs export in Longhorn. I have 1.4.0 running and have TVX volume created but cannot figure out how to present it as nfs. Looking at the docs I should have share-manager pod running but I do no have it. Also once nfs export is created can I mount it from outside the cluster?
    n
    • 2
    • 2
  • c

    clean-lawyer-76009

    02/03/2023, 4:52 PM
    Hey, i installed longhorn v.1.4.0 on a Bare Metal Kubernetes Cluster with RHEL8. When the cluster creates a pv from the longhorn storageclass the pv state is bound, but in longhorn i see the state of the volume is Faulted:
    Not ready for workload, Volume Faulted
    the output of csi attacher log is:
    I0203 10:10:21.981136       1 csi_handler.go:231] Error processing "csi-": failed to attach: rpc error: code = Aborted desc = volume pvc- is not ready for workloads
    The folder on the host belongs to root user in group root How can i drill this issue further down? Could this be a permission issue?
    q
    n
    • 3
    • 7
  • l

    late-needle-80860

    02/04/2023, 9:07 PM
    https://github.com/longhorn/longhorn/issues/5349 < interesting case I just bumped into.
  • c

    crooked-cat-21365

    02/06/2023, 11:35 AM
    Hi folks, what is the recommendation for setting node.session.timeo.replacement_timeout in /etc/iscsi/iscsid.conf? The default is 120 secs, but my Dell servers usually take at least 5 minutes for rebooting, so this setting doesn't seem reasonable.
    n
    • 2
    • 2
  • s

    stocky-article-82001

    02/09/2023, 5:18 PM
    I’ve got a volume stuck in a detaching -> attached -> failure loop and I can’t salvage it because it keeps looping
    n
    • 2
    • 1
  • s

    stocky-article-82001

    02/09/2023, 5:19 PM
    any advice is greatly appreciated.
  • s

    stocky-article-82001

    02/09/2023, 5:27 PM
    If I do manage to catch the “Salvage” button, I see this
    unable to salvage volume pvc-87002c74-b16f-431e-af75-6f9b1f461616: invalid volume state to salvage: attaching
  • s

    stocky-article-82001

    02/09/2023, 5:32 PM
    just got
    unable to salvage volume pvc-87002c74-b16f-431e-af75-6f9b1f461616: invalid robustness state to salvage: unknown
    too
  • s

    stocky-article-82001

    02/09/2023, 5:32 PM
    i wonder if it’s too far gone
  • q

    quaint-alarm-7893

    02/10/2023, 6:19 AM
    hello everyone, i have a vm (in harvesteR) that got messed up when one of my servers had an issue, and now the VM is off, but the PVC is still attached. how can i detach it so i can bring the vm back online?
    State: Attached
    Health:
    No node redundancy: all the healthy replicas are running on the same node Degraded
    Ready for workload:Not Ready
    Conditions:
    restore
    scheduled
    Frontend:Block Device
    Attached Node & Endpoint:
    Size:
    100 Gi
    Actual Size:Unknown
    Data Locality:disabled
    Access Mode:ReadWriteMany
    Backing Image:vdi-image-hzwdh
    Backing Image Size:50 Gi
    Engine Image:longhornio/longhorn-engine:v1.3.2
    Created:a month ago
    Encrypted:False
    Node Tags:
    Disk Tags:
    Last Backup:
    Last Backup At:
    Replicas Auto Balance:ignored
    Instance Manager:
    instance-manager-e-c2f010c0
    instance-manager-e-50d28d2d
    Last time used by Pod:10 hours ago
    Namespace:vdi
    PVC Name:vmname-rootdisk-ossjw
    PV Name:pvc-13787ebc-7d4e-41ad-9206-de9a6adb938a
    PV Status:Bound
    Revision Counter Disabled:False
    Last Pod Name:virt-launcher-vmname-mpbn2
    Last Workload Name:vmname
    Last Workload Type:VirtualMachineInstance
    note, it show's its attached to tow instance managers. i've tried rebooting the nodes, and deleting the instance manager pods, no luck...
    n
    • 2
    • 7
  • s

    sparse-fireman-14239

    02/10/2023, 1:27 PM
    It appears that the Longhorn v1.4.0 Helm chart is now available through Rancher Manager :-)
    🎉 1
    n
    • 2
    • 1
  • m

    mysterious-rose-43856

    02/10/2023, 5:13 PM
    I have longhorn 1.4.0 on 4 clusters or so and have experienced unfortunate downtime that is not expected. I the volumes show as "Attached" but not ready, their workloads will not spin down. I'm investigating, but this is most unfortunate as I'm discovering this mid-migration for some important work loads.
    f
    n
    • 3
    • 16
  • q

    quaint-alarm-7893

    02/10/2023, 11:18 PM
    question. i have a store class i setup for HDD's since rebooting the node, it seems Longhorn now wasnt so schedule other volumes onto that disk. am i missing something? Disk has a tag of hdd, node has a tag of lff. the class has both of those sets. the volumes being scheduled onto the disk are part of the harvester-longhorn default class. not sure why they keep getting put onto the hdd, but they build to 99%, then sit there forever....
    n
    • 2
    • 7
  • m

    mysterious-rose-43856

    02/11/2023, 4:29 PM
    My Ongoing issues may be snapshot related.
  • b

    broad-machine-78396

    02/13/2023, 1:29 PM
    Hello everyone 👋, what is the best practice for a production cluster to setup longhorn, do i need to have 3 dedicated worker node only for longhorn (with taint) or do i need to schedule pod on normal node ?
    👀 1
    p
    n
    • 3
    • 3
  • h

    hundreds-airport-66196

    02/14/2023, 6:06 PM
    Hi All, I have a 3-node cluster with rke2. Planning of using Longhorn for PVs. How can one volume be access by any of the pods running on each of the nodes and have Read/Write access? Or do I need 3 volumes for each pod and replication takes place behind the scene. Thanks!
    n
    • 2
    • 9
  • l

    loud-helmet-97067

    02/16/2023, 1:36 AM
    Hi team, We have requirement, where we have provision k3s cluster with x2 master nodes and x3 worker nodes for storage cluster with longhorn and we need this storage cluster to be used / provision in another kubernetes cluster ( workload cluster) PV / PVC claims. The idea is to provide PV/ PVC claims for software stack such as harbor , ELK ( elastic search), custom application workload that need storage for batch processing; Does it make sense to; • Option 1: Connect k3s - storage cluster to workload cluster (2nd k8s cluster) via means of NFS service from ingress ( or any other means) for production work ( i would like to keep longhorn capabilities on top and expose the storage through nfs-server. I saw somewhere that longhorn itself has nfs-server pod as well with longhorn stack) • Option 2: Create workload pods in same k3s - storage cluster having seperate nodes ( by grouping / labeling nodes) to run application workload • Option 3: Any other options?? Any insights/ feedback in to identify the correct approach is highly appreciated.
    c
    • 2
    • 2
  • q

    quaint-alarm-7893

    02/22/2023, 2:14 PM
    hello everyone. can anyone help on this issue? https://github.com/longhorn/longhorn/issues/5413
    👀 1
  • q

    quaint-alarm-7893

    02/23/2023, 9:19 PM
    @narrow-egg-98197 you messaged me about his before, any thought? ^
    n
    • 2
    • 5
  • q

    quaint-alarm-7893

    02/23/2023, 11:59 PM
    sorry, didnt mean to spam the channel^
  • e

    enough-memory-12110

    03/01/2023, 6:48 AM
    I am having problems with longhorn 1.2.2: I tried setting up a backup target on a custom S3 endpoint. But we're getting an error:
    error listing backup volume names: Timeout executing: /var/lib/longhorn/engine-binaries/rancher-mirrored-longhornio-longhorn-engine-v1.2.2/longhorn [backup ls --volume-only <s3://backups@us-east-1/>], output , stderr, , error <nil>
    I tried logging in to the container and checking the probelm from its side and this is what we get:
    root@longhorn-manager-57lhs:/# /var/lib/longhorn/engine-binaries/rancher-mirrored-longhornio-longhorn-engine-v1.2.2/longhorn backup ls --volume-only <s3://backup/>
    ERRO[0020] Fail to list s3: NoCredentialProviders: no valid providers in chain. Deprecated.
    	For verbose messaging see aws.Config.CredentialsChainVerboseErrors  pkg=s3
    I am sure the credentials in secret are valid and properly formatted ( I've copied minio configuration and adjust to our environment ). Can anyone from here help us diagnose the problem further and solve it? (edited)
    f
    • 2
    • 2
  • b

    blue-painting-33432

    03/03/2023, 12:23 AM
    Hello, I'm having quite a urgent issue here with a basic problem, anyone online to help? Using longhorn on microk8s with local node storage, it worked fine for a few weeks, but now I restarted a pod and the volume just won't attach anymore, it timeout and restart the pod and keeps looping like this. In the longhorn UI I only see "attached" and "detached" events repeating all the time, but no error.
  • b

    blue-painting-33432

    03/03/2023, 12:23 AM
    Unable to attach or mount volumes: unmounted volumes=[alertmanager-kube-prometheus-stack-alertmanager-db], unattached volumes=[web-config kube-api-access-8lfhh config-volume config-out tls-assets alertmanager-kube-prometheus-stack-alertmanager-db]: timed out waiting for the condition
  • b

    blue-painting-33432

    03/03/2023, 12:24 AM
    any way to increase the timeout waiting for the volume mount or something?
  • b

    blue-painting-33432

    03/03/2023, 12:30 AM
    pod remain in pending and containercreating for a few seconds, then immediately goes to deleting and start again, always failing attaching the volume, but longhorn seems to say it is attached with no error
  • b

    blue-painting-33432

    03/03/2023, 1:05 AM
    Ok nevermind, I installed an helm chart that installed a bunch of namespaces and ressources I didn't want, including a second prometheus instance in another namespace, and uninstalling the helm release left those ressource in the cluster and seemed to be causing issue with my real prometheus instance volumes. What a mess
    f
    • 2
    • 1
  • a

    acceptable-soccer-28720

    03/03/2023, 2:04 AM
    Restart of the node and dettach/attach ofmthe volume did not hell to solve the situation
    rancher kubectl describe pod gitlab-postgresql-0 -n gitlab
    Events:
      Type     Reason              Age                  From                     Message
      ----     ------              ----                 ----                     -------
      Normal   Scheduled           14m                  default-scheduler        Successfully assigned gitlab/gitlab-postgresql-0 to vik8scases-w-2
      Warning  FailedMount         10m                  kubelet                  Unable to attach or mount volumes: unmounted volumes=[data], unattached volumes=[kube-api-access-g28wp custom-init-scripts postgresql-password dshm data]: timed out waiting for the condition
      Warning  FailedMount         7m49s                kubelet                  Unable to attach or mount volumes: unmounted volumes=[data], unattached volumes=[postgresql-password dshm data kube-api-access-g28wp custom-init-scripts]: timed out waiting for the condition
      Warning  FailedMount         5m34s                kubelet                  Unable to attach or mount volumes: unmounted volumes=[data], unattached volumes=[dshm data kube-api-access-g28wp custom-init-scripts postgresql-password]: timed out waiting for the condition
      Warning  FailedMount         3m18s (x2 over 12m)  kubelet                  Unable to attach or mount volumes: unmounted volumes=[data], unattached volumes=[data kube-api-access-g28wp custom-init-scripts postgresql-password dshm]: timed out waiting for the condition
      Warning  FailedAttachVolume  2m9s (x6 over 12m)   attachdetach-controller  AttachVolume.Attach failed for volume "pvc-b59dafa1-3efa-44fc-92ba-e2be23e5d4a4" : timed out waiting for external-attacher of <http://driver.longhorn.io|driver.longhorn.io> CSI driver to attach volume pvc-b59dafa1-3efa-44fc-92ba-e2be23e5d4a4
      Warning  FailedMount         64s                  kubelet                  Unable to attach or mount volumes: unmounted volumes=[data], unattached volumes=[custom-init-scripts postgresql-password dshm data kube-api-access-g28wp]: timed out waiting for the condition
    f
    • 2
    • 4
  • p

    plain-breakfast-5576

    03/03/2023, 6:02 PM
    @famous-journalist-11332 we are seeing deadlock while upgrading from 1.2.2. To 1.2.3. Do u have any Upton this? If not can you plz update the kb Longhorn.io/troubleshooting-engine-upgrading-struck-in-deadlock
  • r

    rich-shoe-36510

    03/08/2023, 11:03 AM
    Hey people! One simple question, for which I can't find simple answer. I have running Longhorn (v1.4.0) with one hard drive, everything is fine. Now, I added hard drive, the question is how to add additional disk to the node? Can it be done trough UI, how to bind-mount two hdd mounts to /var/lib/longhorn? Thanks!
Powered by Linen
Title
r

rich-shoe-36510

03/08/2023, 11:03 AM
Hey people! One simple question, for which I can't find simple answer. I have running Longhorn (v1.4.0) with one hard drive, everything is fine. Now, I added hard drive, the question is how to add additional disk to the node? Can it be done trough UI, how to bind-mount two hdd mounts to /var/lib/longhorn? Thanks!
View count: 1