Here’s the bundle @cuddly-vase-67379

Hi everybody!👋 Is anyone using multinetwork with multus in longhorn 1.3?

from time to time, my longhorn is crashing and I do not know why It is start for example the Redis stop working (My Redis is using volume based on longhorn) and then almost everything which is using longhorn (and their dependencies) stop working for a while How can I debug it? I have metrics and logs but I do not know where I should look For example, from metrics I know the volumes are inaccessible

So I have 6 VM's (Ubuntu 20.04) with 3 master/3 worker nodes. On the worker nodes I have attached a second 20G disk and here is where I made some assumptions... I learned the Longhorn installs at

/var/lib/longhorn

and I am thinking this is where my provisioned PV's will be created and replica's will be stored. So wanting to use those 20G disk on each, I partitioned the disk, created a logical volume out of it and then formatted it with XFS filesystem which is our standard format. I then mounted that disk at

/mnt/DATA

, created a subfolder named

longhorn

and then symlinked

/mnt/DATA/longhorn -> /var/lib/longhorn

. Is this the correct approach to use? This is the information I am failing to find anywhere including in the Longhorn docs.

Can anyone help me out? Longhorn started behaving strangely. Instance-manage-e pod gets terminated on all nodes every 2-5 minutes. Seeing

Liveness probe failed: dial tcp 10.42.5.68:8500: i/o timeout

when this happens but only happens on these pods none of other pods are impacted. The nodes are not overloaded also

On 1.22.10+rke2r2, longhorn 1.2.4 I've got a bunch of issues like this:

MountVolume.WaitForAttach failed for volume "pvc-8c978b7c-28db-43e9-88d3-f4b43c532891" : volume pvc-8c978b7c-28db-43e9-88d3-f4b43c532891 has GET error for volume attachment csi-c1dd4b0084f8679ff1701abcdc314f2c594e85c0cfb9c287d954b7a6cf3ac4ba: volumeattachments.storage.k8s.io "csi-c1dd4b0084f8679ff1701abcdc314f2c594e85c0cfb9c287d954b7a6cf3ac4ba" is forbidden: User "systemnodeemerald02" cannot get resource "volumeattachments" in API group "storage.k8s.io" at the cluster scope: no relationship found between node 'emerald02' and this object

Hi all, I've been stuck for a week that elasticsearch (ECK) doesn't work for me when using storage longhorm. The volumes are attached and unattached and the logs of the pods show this: [17:52] Warning FailedAttachVolume 10s attachdetach-controller AttachVolume.Attach failed for volume "pvc-70a16c86-2e98-4a7e-a306-ef7238969f85" : rpc error: code = DeadlineExceeded desc = vol

I rebuilt my k3s cluster but didn't wipe the LH disks. all the new PVCs are stuck in "Attaching". I noticed this msg in the events

Persistent Volume pvc-aafbf3de-7b46-43fd-a536-8595a38505f7 started to use/reuse Longhorn volume pvc-aafbf3de-7b46-43fd-a536-8595a38505f7

does this mean LH found the old PVCs from the previous install?

while I'm using longhorn storageClass my CPU and disk IO saturatons are very high how can I debug the reason why it happens? In thread I will give more info

For the default longhorn storage class, should my

replica's

be the same count as the number of work nodes, or would it be one less than the number of worker nodes (n-1)? Thinking if I deploy a Pod/PVC and it gets assigned to a node, wouldn't the replica's only need to exist on the worker nodes that are not running the Pod workload?

Does anyone have an example of a bind mount they setup in their

/etc/fstab

for mounting

/var/lib/longhorn

to a secondary disk to be used exclusively by longhorn so I'm not using the root partition space for any disk provisioning? A little context. I have a secondary disk I want to use for Longhorn PV's. Running on Ubuntu 20.04, secondary disk is configured as an Logical Volume. It has been formatted with XFS filesystem and mounted at

/mnt/DATA

. It is 20G and I only want longhorn to use this 20G when provisioning new PV's.

I need to update longhorn but it was installed on old Rancher Apps and then moved to a different instance of Rancher, so it doesn’t appear under “Apps” anymore. Any ideas are greatly appreciated.

Currently leaning towards just backing up all the volumes, reinstalling longhorn and importing the backups

That's exactly what I had to do. When i asked this question a couple months ago no one had any answers either

Yeah, it could be normal. Can you see if this document clear up the confusion? https://longhorn.io/docs/1.3.0/volumes-and-nodes/volume-size/

Hi guys, I am facing one issue with longhorn. I have a mongodb volume which gets into the loop of attaching and detaching for a pod. Not sure what will be the root cause for this. Any help will be much appreciated

hi folks... I have a volume in Longhorn that has been stuck in deleting for 12+ hours

what does "Scheduling Failure Replica Scheduling Failure" mean?

I guess it means there's no valid place to put one of the replicas?

Hello all. I keep getting an error with one of my pods (StatefulSet) trying to mount a longhorn volume. It's working fine on 2 out of 3 nodes but this one nodes keeps giving me this error. Can anyone help?

Warning  FailedMount             82s (x9 over 3m31s)  kubelet                  MountVolume.MountDevice failed for volume "pvc-113ff49c-1565-489c-9213-14f4d58ae27f" : rpc error: code = Internal desc = format of disk "/dev/longhorn/pvc-113ff49c-1565-489c-9213-14f4d58ae27f" failed: type:("ext4") target:("/var/lib/kubelet/plugins/kubernetes.io/csi/pv/pvc-113ff49c-1565-489c-9213-14f4d58ae27f/globalmount") options:("defaults") errcode:(exit status 1) output:(mke2fs 1.43.8 (1-Jan-2018)
/dev/longhorn/pvc-113ff49c-1565-489c-9213-14f4d58ae27f is apparently in use by the system; will not make a filesystem here!
)

I've removed the app and uninstalled/re-installed Longhorn but whatever pods gets assigned to this one worked node always fails with this message.

Found my solution here. https://github.com/longhorn/longhorn/issues/1210#issuecomment-671689746

After upgrading to 1.3.0, all my volumes are dead - stuck in attaching or detaching. I can see all instance managers are stopped and a bunch of errors in longhorn-manager logs about it not being able to find the last instance managers... Backups? Some exist, I actually upgraded because the backups were failing for larger volumes... How can I figure out what's wrong, the instance manager log just says 'installed'...

can I limit instance-manager cpu usage?

it is created from

- longhorn-manager
        - -d
        - daemon
        - --engine-image
        - longhornio/longhorn-engine:v1.3.0
        - --instance-manager-image
        - longhornio/longhorn-instance-manager:v1_20220611

hey, just a suggestion

we ran into an issue with our Longhorn cluster today, and helpfully there was a guide - https://longhorn.io/kb/troubleshooting-volume-with-multipath/ - with a tweak in it that got us back online

that tweak may be worth including as a suggestion in the install guide by default...

If I understand this line https://github.com/longhorn/longhorn-manager/blob/b88ebf82936c7335c6a3855ab44a39a2bb790d8b/controller/controller_manager.go#L169 I should setup

EngineManagerCPURequest int `json:"engineManagerCPURequest"

to positive value just add to my

kind: Node

Node.spec.engineManagerCPURequest: 200 And a node is come from

<http://longhorn.io/v1beta2|longhorn.io/v1beta2>

nodes

Am right?

Anybody have a way to delete many snapshots at once? I have a bunch leftover from schedules that no longer exist and i don't want to go by hand through 1000+ snapshots.