rancher-users #longhorn-storage

longhorn-storage

helpful-beard-54962

05/03/2022, 11:24 AM

anybody getting this error randomly?

AttachVolume.Attach failed for volume "pvc-47999557-f6c8-4a99-b594-1b26b18d260d" : rpc error: code = Internal desc = Bad response statusCode [500]. Status [500 Internal Server Error]. Body: [detail=, message=EOF, code=Server Error] from [<http://longhorn-backend:9500/v1/volumes/pvc-47999557-f6c8-4a99-b594-1b26b18d260d?action=attach>]

and a few redeploys later

Unable to attach or mount volumes: unmounted volumes=[temp-volume kube-api-access-m4gxw logs packages-volume code-store-volume], unattached volumes=[temp-volume kube-api-access-m4gxw logs packages-volume code-store-volume]: timed out waiting for the condition

big-engine-61989

05/04/2022, 7:28 PM

The DaemonSet longhorn-csi-plugin’s taint tolerations are not updated from my helm values.yaml. I’ve added the othertaint toleration after the initial deployment and was just applying the config with helm upgrade and added the toleration in configuration through the UI. Is there a way to get the longhorn-csi-plugin daemonset updated at runtime? (other than manually editing it) Longhorn release is 1.2.4 values.yaml

taintToleration: "longhorn=true:NoSchedule;othertaint=true:NoSchedule"

defaultSettings:
  taintToleration: "longhorn=true:NoSchedule;othertaint=true:NoSchedule"

longhornManager:
  tolerations:
    - key: "longhorn"
      operator: "Equal"
      value: "true"
      effect: "NoSchedule"
    - key: "othertaint"
      operator: "Equal"
      value: "true"
      effect: "NoSchedule"

longhornDriver:
  tolerations:
    - key: "longhorn"
      operator: "Equal"
      value: "true"
      effect: "NoSchedule"
    - key: "othertaint"
      operator: "Equal"
      value: "true"
      effect: "NoSchedule"

average-gigabyte-2667

05/07/2022, 9:11 PM

Hi, I'm researching multi-cloud & multi-cluster Kubernetes. I couldn't found many sources regarding multi-cloud / multi-cluster Kubernetes storage. I would like to do failover of Kubernetes but need storage to be replicated to another cluster, of course with minimal downtime. Longhorn describes it's applicable for multi-cloud. What can you effectively implement regarding multi-cloud and multi-cluster use cases with Longhorn? I saw some sources about Yugabyte + Longhorn, but don't fully grasp the architecture and benefits. Restoring replicas should be faster, but what is the slower alternative?

cool-state-48214

05/09/2022, 7:35 PM

hello everybody, hope you are all doin’ well. anybody succeeded in snapshoting a longhorn volume from k10 ? the tool from casten.io said cluster is not csi capable

wonderful-kangaroo-15590

05/11/2022, 8:45 AM

hi, my longhorn volume failed to attach to pod/ node. Checking the dmesg log, I'm seeing error like this, do you have any idea what can go wrong here? kubernetes v1.20.15, longhorn v1.2.3, OS RHEL8.4

flaky-coat-75909

05/11/2022, 2:25 PM

how can I indicate on specific host where data should be replicated? I want to do something like this: Attach addition node and this node is node for storage how can I achieve it?

wide-easter-7639

05/11/2022, 4:08 PM

Hi. I am thinking to implement a XCP-ng cluster, and then kuberntes among all nodes. To provide storage to my workloads/pods I am thinking to use longhorn or other solution. My doubt is if necesary to install longhorn on some nodes dedicating its harddisks specifically to provide storage to other workloads/pods or maybe if it is possible to install longhorn on all the nodes nodes and provide storage to workloads using the local disks. what is the best practice/approach?

sticky-truck-78998

05/12/2022, 5:42 AM

Has this burned anyone yet? We just found this setting. I hope it solves the issue for us. Lost a node (1of3)today and manually fixing stuck pods sucks. Luckily our admin was around when it happened. 3 hours later restored.

helpful-beard-54962

05/15/2022, 10:58 AM

How can I make longhorn faster on AWS? It's so slow!

hundreds-hairdresser-46043

05/23/2022, 10:14 AM

Good Day Fellow ranchers. General question: what is the reasoning of mounting the storage via the UI to a specific node? I saw that once mounted the storage is exposed as /dev/sdX drive on the OS. My reason for this question is what happens when that node is lost/rebooted or offline? We are currently using rook-ceph and it works a little different then longhorn

red-planet-35817

05/24/2022, 12:50 PM

Hi all. I fond that RWX (Read Write Many) is an experimental feature (https://longhorn.io/docs/1.2.4/advanced-resources/rwx-workloads/), But what about the ROX (ReadOnlyMany) ? Is natively supported by Longhorn ? Thx a lot for the answer 😜 Baptiste.

great-photographer-94826

05/24/2022, 2:49 PM

Hi all! I would like to make a Recurring Job which saves all PVC in my namespace. It is good idea when I add labels (e.g namespace longhorn-test)? How can I achieve it?

flaky-coat-75909

05/26/2022, 11:50 AM

it is possible to schedule specific set of node with taint and tollerations instead of nodeSelector like in this example https://longhorn.io/kb/tip-only-use-storage-on-a-set-of-nodes/

kind: StorageClass
apiVersion: <http://storage.k8s.io/v1|storage.k8s.io/v1>
metadata:
  name: my-longhorn-sc
...
parameters:
...
  nodeSelector: "storage"

wide-easter-7639

05/27/2022, 4:11 PM

Hi . Is longhorn able to use kubernetes's external iscsi volumes as storage? I have longhorn installed inside a kubernetes cluster. We also have a equallogic disks cabin and also a couple of linux nfs servers (which I want to setup also iscsi daemon). Can the kubernetes's Longhorn use the equalogic disks and the linux iscsi servers as storage to provide volumes to the kubernetes pods?

flaky-coat-75909

05/28/2022, 2:43 PM

is longhorn support tracing for jaeger?

flaky-coat-75909

05/30/2022, 1:10 PM

For example I have 5 nodes (1master) (4workers) Node0(master) Node1, Node2, Node3, Node4 On Node4 I have a lot of memory (disks) on another hand on nodes Node1, Node2, Node3 I have much less memory (disk) Is longhorn able to share memory from Node4 to pods which are scheduled on Node1,Node2,Node3? or I have to run pod on the node where the longhorn create disk? Is longhorn can handle this scenario?

hundreds-hairdresser-46043

06/03/2022, 12:26 PM

general question: if you have 10 nodes and you schedule only 3 nodes for storage usage does that mean it ONLY uses the 3 nodes and no local storage? in relationship to how rook-ceph works?

agreeable-vegetable-79181

06/08/2022, 2:17 AM

hi all - getting particularly poor performance on 2 node cluster running k3s with 25G networking between nodes. Seeing 29MB/s to LH. iperf betweeen nodes shows 25Gbps, pod to opposite host is around 7Gbps (not sure why) and write to the disks locally is closer to 250MB/s. Where would you look here?

flaky-coat-75909

06/09/2022, 8:20 AM

how can I migrate data in safe way ? for example on disk1 is more than 75% disk usage and I want to move it on disk2 which have more memory

flaky-coat-75909

06/09/2022, 11:38 AM

My postgresql are using

602Mi

but in frontend ui I see the Actual Size is

2.26 Gi

I have 2 snapshots • Size: 531Mi • Size: 907Mi and VolumeHead is 883Mi why snapshots takes so much memory when the total memory of postgresql (602) is less than second snapshot?

flaky-coat-75909

06/10/2022, 1:15 PM

can any one explain why in my postgresql pod

$ df -h
Filesystem                                              Size  Used Avail Use% Mounted on
/dev/longhorn/pvc-c6b78d1e-109a-4c44-af17-ae2ad46e28b6   20G  1.5G   19G   8% /bitnami/postgresql

is Used

1.5G

$ du -sh /bitnami/postgresql
1.5G	/bitnami/postgresql

Where on the frontend-longhorn I see

/dev/longhorn/pvc-c6b78d1e-109a-4c44-af17-ae2ad46e28b6
Size: 20 Gi
Actual Size:10.7 Gi

Which means it is almost 10times more displaying on frontend And it is probably because I'm doing snapshots On my node in that directory and I see files

[root@server22 pvc-c6b78d1e-109a-4c44-af17-ae2ad46e28b6-3277a500]# pwd
/var/lib/longhorn/replicas/pvc-c6b78d1e-109a-4c44-af17-ae2ad46e28b6-3277a500
[root@server22 pvc-c6b78d1e-109a-4c44-af17-ae2ad46e28b6-3277a500]# du -sh * 
4.0K	revision.counter
189M	volume-head-010.img
4.0K	volume-head-010.img.meta
4.0K	volume.meta
531M	volume-snap-035b7630-78a9-4299-af2f-ff3cc8be0f06.img
4.0K	volume-snap-035b7630-78a9-4299-af2f-ff3cc8be0f06.img.meta
2.1G	volume-snap-c-bi3bca-c-0dddc3f6.img
4.0K	volume-snap-c-bi3bca-c-0dddc3f6.img.meta
2.5G	volume-snap-c-bi3bca-c-1908b8fa.img
4.0K	volume-snap-c-bi3bca-c-1908b8fa.img.meta
1.6G	volume-snap-c-bi3bca-c-4b2590fb.img
4.0K	volume-snap-c-bi3bca-c-4b2590fb.img.meta
1.9G	volume-snap-c-bi3bca-c-d08a8ddd.img
4.0K	volume-snap-c-bi3bca-c-d08a8ddd.img.meta
2.2G	volume-snap-c-bi3bca-c-e827be1c.img
4.0K	volume-snap-c-bi3bca-c-e827be1c.img.meta

but why snapshots are taking so much memory while the whole my data is only

1.5G

What I'm doing wrong ?

stocky-article-82001

06/19/2022, 4:48 PM

I'm having trouble updating longhorn after moving a cluster to a different rancher instance. It was installed via Rancher Apps. Any help is greatly appreciated.

stocky-beard-10620

06/20/2022, 5:44 PM

Hello! I'm in the process of upgrading to 1.3.0 (I'm deploying with Helm), but after upgrading, almost all my volumes are stuck in

Attaching

and I can't seem to figure out what's the issue, can I get some help troubleshooting? I've tried restarting all pods, with no help. Thanks!

late-needle-80860

06/20/2022, 5:58 PM

I’m having issues with Longhorn v1.2.3 prepping the filesystem of a Kafka workload. I’m seeing

exit 32

when

mount … ext4

is being executed …. and I see this err. in the

csi-plugin

Pod

on the worker node where Longhorn is having the issue.

Mounting command: mount
Mounting arguments: -t ext4 -o defaults /dev/longhorn/pvc-77efeb41-63ad-43f6-8cc0-e67b2e820aad /var/lib/kubelet/plugins/kubernetes.io/csi/pv/pvc-77efeb41-63ad-43f6-8cc0-e67b2e820aad/globalmount
Output: mount: /var/lib/kubelet/plugins/kubernetes.io/csi/pv/pvc-77efeb41-63ad-43f6-8cc0-e67b2e820aad/globalmount: wrong fs type, bad option, bad superblock on /dev/longhorn/pvc-77efeb41-63ad-43f6-8cc0-e67b2e820aad, missing codepage or helper program, or other error.



time="2022-06-20T14:14:12Z" level=error msg="NodeStageVolume: err: rpc error: code = Internal desc = mount failed: exit status 32\nMounting command: mount\nMounting arguments: -t ext4 -o defaults /dev/longhorn/pvc-77efeb41-63ad-43f6-8cc0-e67b2e820aad /var/lib/kubelet/plugins/kubernetes.io/csi/pv/pvc-77efeb41-63ad-43f6-8cc0-e67b2e820aad/globalmount\nOutput: mount: /var/lib/kubelet/plugins/kubernetes.io/csi/pv/pvc-77efeb41-63ad-43f6-8cc0-e67b2e820aad/globalmount: wrong fs type, bad option, bad superblock on /dev/longhorn/pvc-77efeb41-63ad-43f6-8cc0-e67b2e820aad, missing codepage or helper program, or other error.\n"

Suggestions are cherished …. thank you very much.

✅ 1

stocky-beard-10620

06/20/2022, 6:50 PM

After upgrading to 1.3.0, all of my volumes' replicas on all nodes are showing as either

Stopped

Failed

, or

Unknown

. If I inspect a given volume, it tells me that the only replica is "running" on an instance-manager with a name that doesn't match any of the currently running instance-manager pods (neither the

-e

nor the

-r

ones). Is it possible that during the upgrade, the connection between replicas and instance-managers got lost or out of sync and now it has to be fixed manually?

hundreds-hairdresser-46043

06/21/2022, 10:53 AM

has anyone every seen this error and know what causes it? -> Invalid attach limit value 0 cannot be added to CSINode object for "driver.longhorn.io"

flaky-coat-75909

06/23/2022, 10:53 AM

root@longhorn-manager:/# curl 10.43.0.3:9500/metrics

do not return all metrics for example

longhorn_instance_manager_cpu_usage_millicpu

longhorn_node_cpu_capacity_millicpu

Should I fetch it from other pod? on which port ?

helpful-beard-54962

06/28/2022, 2:54 PM

I woke up today to Longhorn working insanely slow

sent 57,683 bytes  received 337 bytes  1,172.12 bytes/sec
sent 53,839 bytes  received 337 bytes  976.14 bytes/sec

Anyone else?

ripe-queen-73614

06/28/2022, 3:17 PM

I have a cluster of 6 nodes, three master and three worker. I want to use longhorm for persistent volume storage. On which nodes do I have to install longhorm?

✅ 1

bumpy-portugal-40754

06/28/2022, 11:57 PM

Recently after a powerdown of the complete harvester cluster I got 2 volumes with a stopped replica because... I don't know. Longhorn GUI complained about the fact that 2 manager instances are running for both volumes. They did, surprisingly not on the node with the stopped replica. I killed one manager pod and the volume got rebuild. Any opinion about this "strategy" to solve this issue? How to debug? Are there remains of the old replica left?

✅ 1

Title

bumpy-portugal-40754

06/28/2022, 11:57 PM

✅ 1

aloof-hair-13897

06/29/2022, 3:30 AM

Hi, @bumpy-portugal-40754 How did you do power down the harvester cluster and that including the nodes with volumes’ replica? It is normal that you would get stopped replicas if the node containing the replica is down.

Longhorn GUI complained about the fact that 2 manager instances are running for both volumes.

Did you mean instance-manager-e-xxx and instance-manager-r-xxx? Could you provide the support bundle then we could find out what happened.

bumpy-portugal-40754

06/29/2022, 11:11 AM

I didn't do the powerdown myself, but I'd suspect it to be graceful. Instance-manager-e... was there twice. The one which is shown in the longhorn gui. I already created a case with suse. Are you able to access it?

cuddly-vase-67379

06/29/2022, 12:30 PM

Could you show me where it is?

bumpy-portugal-40754

06/29/2022, 1:18 PM

Looks like I was wrong about the way the machines were powered down. It seems that the servers were simply switched off.

View count: 8