from time to time my longhorn is crashing and I do not know rancher-users #longhorn-storage

Title

flaky-coat-75909

07/01/2022, 2:00 PM

from time to time, my longhorn is crashing and I do not know why It is start for example the Redis stop working (My Redis is using volume based on longhorn) and then almost everything which is using longhorn (and their dependencies) stop working for a while How can I debug it? I have metrics and logs but I do not know where I should look For example, from metrics I know the volumes are inaccessible

in logs I see something like this (This is probably the issue)

Shutdown SCSI device for target iqn.2019-10.io.longhorn:pvc-280f7155-bee9-4ede-9110-ea0ce977aba3

or from manager

msg=\"device /dev/mapper/pvc-e1918dd4-d8dd-4590-a106-0dd5a9c62e36 is not an active LUKS device: failed to run cryptsetup args: [status pvc-e1918dd4-d8dd-4590-a106-0dd5a9c62e36] output:  error: exit status 4

some more metrics

I have InstanceManager LonghotnManager engine-image-ei csi-attacher csi-provisioner csi-resizer csi-snapshot longhorn-csi-plugin Where I should start debug the flow

late-needle-80860

07/01/2022, 6:09 PM

You don’t mention your version of Longhorn?

flaky-coat-75909

07/01/2022, 7:43 PM

@late-needle-80860

1.2.3

late-needle-80860

07/02/2022, 12:10 AM

And you've checks that this is not something already known and fixed in either 1.2.4 or 1.3.0?

What's the output of

dmesg | tail

and what does

journalctl -xe

give you?

21 Views

#longhorn-storage Join Slack