Hello everyone! I’m having a weird production issue with a longhorn volume in harvester. I have a VM (virt-launcher) with several disks (RWX block devices) and last time I restarted the VM the first disk (the boot image) got “stuck” attached to the host. I see the longhorn.io.volumeattachment, and when I ask to Detach from the longhorn UI - and I have to Force it - it gets immediately attached again on a random node. The virt-launcher is not there (the VM is marked as Off) and when I try to start it I get ‘MapVolume.MapPodDevice failed for volume “pvc-3bab7024-25bd-497b-83a7-e13933454c65" : rpc error: code = FailedPrecondition desc = it appears dev is already mounted’ as well as ‘AttachVolume.Attach failed for volume “pvc-03a41d7f-15b0-44dd-842e-6ee0fa19d219” : volume attachment is being deleted’ errors until I stop it manually again. I’ve launched a clone of the disk and an export to an image out of it, but for now both tasks have been running too long. My last resort is to delete the volume and restore it from backup, but I’d like to know how to troubleshoot this kind of issues. Can anyone point me to some log where I can find what resource is asking to attach the disk (erroneously) and how to make it stop?

Could you provide a support bundle?

I didn’t notice that, thank you! Next time I’ll make sure to look into it - now it’s a little late, I’ve already cloned and removed the offending volume. I will also have to search the archive for private data before I share it, it’s the production cluster.

We will check it soon. cc @little-kangaroo-65735

Thank you, here it is: https://github.com/longhorn/longhorn/issues/11201