07/31/2022, 12:20 AM
After upgrading to 1.3.0, all my volumes are dead - stuck in attaching or detaching. I can see all instance managers are stopped and a bunch of errors in longhorn-manager logs about it not being able to find the last instance managers... Backups? Some exist, I actually upgraded because the backups were failing for larger volumes... How can I figure out what's wrong, the instance manager log just says 'installed'...


08/01/2022, 10:38 AM
Typically Longhorn will deploy and use a newer version of instance manager after the version bump. But Longhorn won’t kill the old instance managers as long as there are engine/replica running in them. In your case, it seems that somehow all the old instance managers directly disappeared… Weird… • For debugging purpose, you can generate the support bundle and send it to us. • Can you try to detach the volumes (and stop the corresponding pods)? If the volumes can be detached, then please apply engine upgrade for these volumes. After that, the volume may be available. If the data inside the volume crashes, you may need to fix it manually or apply snapshot revert/backup restore.