https://rancher.com/ logo
Join the conversationJoin Slack
Channels
academy
amazon
arm
azure
cabpr
chinese
ci-cd
danish
deutsch
developer
elemental
epinio
espanol
events
extensions
fleet
français
gcp
general
harvester
harvester-dev
hobbyfarm
hypper
japanese
k3d
k3os
k3s
k3s-contributor
kim
kubernetes
kubewarden
lima
logging
longhorn-dev
longhorn-storage
masterclass
mesos
mexico
nederlands
neuvector-security
office-hours
one-point-x
onlinemeetup
onlinetraining
opni
os
ozt
phillydotnet
portugues
rancher-desktop
rancher-extensions
rancher-setup
rancher-wrangler
random
rfed_ara
rio
rke
rke2
russian
s3gw
service-mesh
storage
submariner
supermicro-sixsq
swarm
terraform-controller
terraform-provider-rancher2
terraform-provider-rke
theranchcast
training-0110
training-0124
training-0131
training-0207
training-0214
training-1220
ukranian
v16-v21-migration
vsphere
windows
Powered by Linen
longhorn-storage
  • h

    helpful-beard-54962

    05/03/2022, 11:24 AM
    anybody getting this error randomly?
    AttachVolume.Attach failed for volume "pvc-47999557-f6c8-4a99-b594-1b26b18d260d" : rpc error: code = Internal desc = Bad response statusCode [500]. Status [500 Internal Server Error]. Body: [detail=, message=EOF, code=Server Error] from [<http://longhorn-backend:9500/v1/volumes/pvc-47999557-f6c8-4a99-b594-1b26b18d260d?action=attach>]
    and a few redeploys later
    Unable to attach or mount volumes: unmounted volumes=[temp-volume kube-api-access-m4gxw logs packages-volume code-store-volume], unattached volumes=[temp-volume kube-api-access-m4gxw logs packages-volume code-store-volume]: timed out waiting for the condition
    f
    • 2
    • 1
  • b

    big-engine-61989

    05/04/2022, 7:28 PM
    The DaemonSet longhorn-csi-plugin’s taint tolerations are not updated from my helm values.yaml. I’ve added the othertaint toleration after the initial deployment and was just applying the config with helm upgrade and added the toleration in configuration through the UI. Is there a way to get the longhorn-csi-plugin daemonset updated at runtime? (other than manually editing it) Longhorn release is 1.2.4 values.yaml
    taintToleration: "longhorn=true:NoSchedule;othertaint=true:NoSchedule"
    
    defaultSettings:
      taintToleration: "longhorn=true:NoSchedule;othertaint=true:NoSchedule"
    
    longhornManager:
      tolerations:
        - key: "longhorn"
          operator: "Equal"
          value: "true"
          effect: "NoSchedule"
        - key: "othertaint"
          operator: "Equal"
          value: "true"
          effect: "NoSchedule"
    
    longhornDriver:
      tolerations:
        - key: "longhorn"
          operator: "Equal"
          value: "true"
          effect: "NoSchedule"
        - key: "othertaint"
          operator: "Equal"
          value: "true"
          effect: "NoSchedule"
    f
    • 2
    • 2
  • a

    average-gigabyte-2667

    05/07/2022, 9:11 PM
    Hi, I'm researching multi-cloud & multi-cluster Kubernetes. I couldn't found many sources regarding multi-cloud / multi-cluster Kubernetes storage. I would like to do failover of Kubernetes but need storage to be replicated to another cluster, of course with minimal downtime. Longhorn describes it's applicable for multi-cloud. What can you effectively implement regarding multi-cloud and multi-cluster use cases with Longhorn? I saw some sources about Yugabyte + Longhorn, but don't fully grasp the architecture and benefits. Restoring replicas should be faster, but what is the slower alternative?
    k
    • 2
    • 2
  • c

    cool-state-48214

    05/09/2022, 7:35 PM
    hello everybody, hope you are all doin’ well. anybody succeeded in snapshoting a longhorn volume from k10 ? the tool from casten.io said cluster is not csi capable
    g
    b
    • 3
    • 13
  • w

    wonderful-kangaroo-15590

    05/11/2022, 8:45 AM
    hi, my longhorn volume failed to attach to pod/ node. Checking the dmesg log, I'm seeing error like this, do you have any idea what can go wrong here? kubernetes v1.20.15, longhorn v1.2.3, OS RHEL8.4
    f
    • 2
    • 7
  • f

    flaky-coat-75909

    05/11/2022, 2:25 PM
    how can I indicate on specific host where data should be replicated? I want to do something like this: Attach addition node and this node is node for storage how can I achieve it?
    f
    • 2
    • 1
  • w

    wide-easter-7639

    05/11/2022, 4:08 PM
    Hi. I am thinking to implement a XCP-ng cluster, and then kuberntes among all nodes. To provide storage to my workloads/pods I am thinking to use longhorn or other solution. My doubt is if necesary to install longhorn on some nodes dedicating its harddisks specifically to provide storage to other workloads/pods or maybe if it is possible to install longhorn on all the nodes nodes and provide storage to workloads using the local disks. what is the best practice/approach?
    s
    • 2
    • 10
  • s

    sticky-truck-78998

    05/12/2022, 5:42 AM
    Has this burned anyone yet? We just found this setting. I hope it solves the issue for us. Lost a node (1of3)today and manually fixing stuck pods sucks. Luckily our admin was around when it happened. 3 hours later restored.
    f
    w
    • 3
    • 18
  • h

    helpful-beard-54962

    05/15/2022, 10:58 AM
    How can I make longhorn faster on AWS? It's so slow!
    e
    f
    • 3
    • 4
  • h

    hundreds-hairdresser-46043

    05/23/2022, 10:14 AM
    Good Day Fellow ranchers. General question: what is the reasoning of mounting the storage via the UI to a specific node? I saw that once mounted the storage is exposed as /dev/sdX drive on the OS. My reason for this question is what happens when that node is lost/rebooted or offline? We are currently using rook-ceph and it works a little different then longhorn
    f
    • 2
    • 4
  • r

    red-planet-35817

    05/24/2022, 12:50 PM
    Hi all. I fond that RWX (Read Write Many) is an experimental feature (https://longhorn.io/docs/1.2.4/advanced-resources/rwx-workloads/), But what about the ROX (ReadOnlyMany) ? Is natively supported by Longhorn ? Thx a lot for the answer 😜 Baptiste.
    f
    • 2
    • 2
  • g

    great-photographer-94826

    05/24/2022, 2:49 PM
    Hi all! I would like to make a Recurring Job which saves all PVC in my namespace. It is good idea when I add labels (e.g namespace longhorn-test)? How can I achieve it?
    f
    • 2
    • 1
  • f

    flaky-coat-75909

    05/26/2022, 11:50 AM
    it is possible to schedule specific set of node with taint and tollerations instead of nodeSelector like in this example https://longhorn.io/kb/tip-only-use-storage-on-a-set-of-nodes/
    kind: StorageClass
    apiVersion: <http://storage.k8s.io/v1|storage.k8s.io/v1>
    metadata:
      name: my-longhorn-sc
    ...
    parameters:
    ...
      nodeSelector: "storage"
    ?
    f
    • 2
    • 2
  • w

    wide-easter-7639

    05/27/2022, 4:11 PM
    Hi . Is longhorn able to use kubernetes's external iscsi volumes as storage? I have longhorn installed inside a kubernetes cluster. We also have a equallogic disks cabin and also a couple of linux nfs servers (which I want to setup also iscsi daemon). Can the kubernetes's Longhorn use the equalogic disks and the linux iscsi servers as storage to provide volumes to the kubernetes pods?
    c
    • 2
    • 1
  • f

    flaky-coat-75909

    05/28/2022, 2:43 PM
    is longhorn support tracing for jaeger?
    f
    • 2
    • 4
  • f

    flaky-coat-75909

    05/30/2022, 1:10 PM
    For example I have 5 nodes (1master) (4workers) Node0(master) Node1, Node2, Node3, Node4 On Node4 I have a lot of memory (disks) on another hand on nodes Node1, Node2, Node3 I have much less memory (disk) Is longhorn able to share memory from Node4 to pods which are scheduled on Node1,Node2,Node3? or I have to run pod on the node where the longhorn create disk? Is longhorn can handle this scenario?
    f
    • 2
    • 2
  • h

    hundreds-hairdresser-46043

    06/03/2022, 12:26 PM
    general question: if you have 10 nodes and you schedule only 3 nodes for storage usage does that mean it ONLY uses the 3 nodes and no local storage? in relationship to how rook-ceph works?
    f
    • 2
    • 2
  • a

    agreeable-vegetable-79181

    06/08/2022, 2:17 AM
    hi all - getting particularly poor performance on 2 node cluster running k3s with 25G networking between nodes. Seeing 29MB/s to LH. iperf betweeen nodes shows 25Gbps, pod to opposite host is around 7Gbps (not sure why) and write to the disks locally is closer to 250MB/s. Where would you look here?
    h
    f
    • 3
    • 2
  • f

    flaky-coat-75909

    06/09/2022, 8:20 AM
    how can I migrate data in safe way ? for example on disk1 is more than 75% disk usage and I want to move it on disk2 which have more memory
    f
    • 2
    • 1
  • f

    flaky-coat-75909

    06/09/2022, 11:38 AM
    My postgresql are using
    602Mi
    but in frontend ui I see the Actual Size is
    2.26 Gi
    I have 2 snapshots • Size: 531Mi • Size: 907Mi and VolumeHead is 883Mi why snapshots takes so much memory when the total memory of postgresql (602) is less than second snapshot?
    f
    • 2
    • 1
  • f

    flaky-coat-75909

    06/10/2022, 1:15 PM
    can any one explain why in my postgresql pod
    $ df -h
    Filesystem                                              Size  Used Avail Use% Mounted on
    /dev/longhorn/pvc-c6b78d1e-109a-4c44-af17-ae2ad46e28b6   20G  1.5G   19G   8% /bitnami/postgresql
    is
    Used
    1.5G
    $ du -sh /bitnami/postgresql
    1.5G	/bitnami/postgresql
    Where on the frontend-longhorn I see
    /dev/longhorn/pvc-c6b78d1e-109a-4c44-af17-ae2ad46e28b6
    Size: 20 Gi
    Actual Size:10.7 Gi
    Which means it is almost 10times more displaying on frontend And it is probably because I'm doing snapshots On my node in that directory and I see files
    [root@server22 pvc-c6b78d1e-109a-4c44-af17-ae2ad46e28b6-3277a500]# pwd
    /var/lib/longhorn/replicas/pvc-c6b78d1e-109a-4c44-af17-ae2ad46e28b6-3277a500
    [root@server22 pvc-c6b78d1e-109a-4c44-af17-ae2ad46e28b6-3277a500]# du -sh * 
    4.0K	revision.counter
    189M	volume-head-010.img
    4.0K	volume-head-010.img.meta
    4.0K	volume.meta
    531M	volume-snap-035b7630-78a9-4299-af2f-ff3cc8be0f06.img
    4.0K	volume-snap-035b7630-78a9-4299-af2f-ff3cc8be0f06.img.meta
    2.1G	volume-snap-c-bi3bca-c-0dddc3f6.img
    4.0K	volume-snap-c-bi3bca-c-0dddc3f6.img.meta
    2.5G	volume-snap-c-bi3bca-c-1908b8fa.img
    4.0K	volume-snap-c-bi3bca-c-1908b8fa.img.meta
    1.6G	volume-snap-c-bi3bca-c-4b2590fb.img
    4.0K	volume-snap-c-bi3bca-c-4b2590fb.img.meta
    1.9G	volume-snap-c-bi3bca-c-d08a8ddd.img
    4.0K	volume-snap-c-bi3bca-c-d08a8ddd.img.meta
    2.2G	volume-snap-c-bi3bca-c-e827be1c.img
    4.0K	volume-snap-c-bi3bca-c-e827be1c.img.meta
    but why snapshots are taking so much memory while the whole my data is only
    1.5G
    What I'm doing wrong ?
    f
    • 2
    • 1
  • s

    stocky-article-82001

    06/19/2022, 4:48 PM
    I'm having trouble updating longhorn after moving a cluster to a different rancher instance. It was installed via Rancher Apps. Any help is greatly appreciated.
    s
    f
    • 3
    • 2
  • s

    stocky-beard-10620

    06/20/2022, 5:44 PM
    Hello! I'm in the process of upgrading to 1.3.0 (I'm deploying with Helm), but after upgrading, almost all my volumes are stuck in
    Attaching
    and I can't seem to figure out what's the issue, can I get some help troubleshooting? I've tried restarting all pods, with no help. Thanks!
    l
    • 2
    • 14
  • l

    late-needle-80860

    06/20/2022, 5:58 PM
    I’m having issues with Longhorn v1.2.3 prepping the filesystem of a Kafka workload. I’m seeing
    exit 32
    when
    mount … ext4
    is being executed …. and I see this err. in the
    csi-plugin
    Pod
    on the worker node where Longhorn is having the issue.
    Mounting command: mount
    Mounting arguments: -t ext4 -o defaults /dev/longhorn/pvc-77efeb41-63ad-43f6-8cc0-e67b2e820aad /var/lib/kubelet/plugins/kubernetes.io/csi/pv/pvc-77efeb41-63ad-43f6-8cc0-e67b2e820aad/globalmount
    Output: mount: /var/lib/kubelet/plugins/kubernetes.io/csi/pv/pvc-77efeb41-63ad-43f6-8cc0-e67b2e820aad/globalmount: wrong fs type, bad option, bad superblock on /dev/longhorn/pvc-77efeb41-63ad-43f6-8cc0-e67b2e820aad, missing codepage or helper program, or other error.
    
    
    
    time="2022-06-20T14:14:12Z" level=error msg="NodeStageVolume: err: rpc error: code = Internal desc = mount failed: exit status 32\nMounting command: mount\nMounting arguments: -t ext4 -o defaults /dev/longhorn/pvc-77efeb41-63ad-43f6-8cc0-e67b2e820aad /var/lib/kubelet/plugins/kubernetes.io/csi/pv/pvc-77efeb41-63ad-43f6-8cc0-e67b2e820aad/globalmount\nOutput: mount: /var/lib/kubelet/plugins/kubernetes.io/csi/pv/pvc-77efeb41-63ad-43f6-8cc0-e67b2e820aad/globalmount: wrong fs type, bad option, bad superblock on /dev/longhorn/pvc-77efeb41-63ad-43f6-8cc0-e67b2e820aad, missing codepage or helper program, or other error.\n"
    Suggestions are cherished …. thank you very much.
    ✅ 1
    c
    • 2
    • 10
  • s

    stocky-beard-10620

    06/20/2022, 6:50 PM
    After upgrading to 1.3.0, all of my volumes' replicas on all nodes are showing as either
    Stopped
    ,
    Failed
    , or
    Unknown
    . If I inspect a given volume, it tells me that the only replica is "running" on an instance-manager with a name that doesn't match any of the currently running instance-manager pods (neither the
    -e
    nor the
    -r
    ones). Is it possible that during the upgrade, the connection between replicas and instance-managers got lost or out of sync and now it has to be fixed manually?
    l
    s
    f
    • 4
    • 16
  • h

    hundreds-hairdresser-46043

    06/21/2022, 10:53 AM
    has anyone every seen this error and know what causes it? -> Invalid attach limit value 0 cannot be added to CSINode object for "driver.longhorn.io"
    i
    • 2
    • 3
  • f

    flaky-coat-75909

    06/23/2022, 10:53 AM
    root@longhorn-manager:/# curl 10.43.0.3:9500/metrics
    do not return all metrics for example
    longhorn_instance_manager_cpu_usage_millicpu
    or
    longhorn_node_cpu_capacity_millicpu
    Should I fetch it from other pod? on which port ?
    i
    • 2
    • 3
  • h

    helpful-beard-54962

    06/28/2022, 2:54 PM
    I woke up today to Longhorn working insanely slow
    sent 57,683 bytes  received 337 bytes  1,172.12 bytes/sec
    sent 53,839 bytes  received 337 bytes  976.14 bytes/sec
    Anyone else?
    i
    • 2
    • 2
  • r

    ripe-queen-73614

    06/28/2022, 3:17 PM
    I have a cluster of 6 nodes, three master and three worker. I want to use longhorm for persistent volume storage. On which nodes do I have to install longhorm?
    ✅ 1
    i
    • 2
    • 65
  • b

    bumpy-portugal-40754

    06/28/2022, 11:57 PM
    Recently after a powerdown of the complete harvester cluster I got 2 volumes with a stopped replica because... I don't know. Longhorn GUI complained about the fact that 2 manager instances are running for both volumes. They did, surprisingly not on the node with the stopped replica. I killed one manager pod and the volume got rebuild. Any opinion about this "strategy" to solve this issue? How to debug? Are there remains of the old replica left?
    ✅ 1
    a
    c
    • 3
    • 4
Powered by Linen
Title
b

bumpy-portugal-40754

06/28/2022, 11:57 PM
Recently after a powerdown of the complete harvester cluster I got 2 volumes with a stopped replica because... I don't know. Longhorn GUI complained about the fact that 2 manager instances are running for both volumes. They did, surprisingly not on the node with the stopped replica. I killed one manager pod and the volume got rebuild. Any opinion about this "strategy" to solve this issue? How to debug? Are there remains of the old replica left?
✅ 1
a

aloof-hair-13897

06/29/2022, 3:30 AM
Hi, @bumpy-portugal-40754 How did you do power down the harvester cluster and that including the nodes with volumes’ replica? It is normal that you would get stopped replicas if the node containing the replica is down.
Longhorn GUI complained about the fact that 2 manager instances are running for both volumes.
Did you mean instance-manager-e-xxx and instance-manager-r-xxx? Could you provide the support bundle then we could find out what happened.
b

bumpy-portugal-40754

06/29/2022, 11:11 AM
I didn't do the powerdown myself, but I'd suspect it to be graceful. Instance-manager-e... was there twice. The one which is shown in the longhorn gui. I already created a case with suse. Are you able to access it?
c

cuddly-vase-67379

06/29/2022, 12:30 PM
Could you show me where it is?
b

bumpy-portugal-40754

06/29/2022, 1:18 PM
Looks like I was wrong about the way the machines were powered down. It seems that the servers were simply switched off.
View count: 8