So I, silly me, did not follow https://longhorn.io/docs/1.4.1/volumes-and-nodes/maintenance/#updating-the-node-os-or-container-runtime when I removed a node by first draining it … from a k3s v1.25.5 cluster.
I can create a simple pod that consumes a pvc and that gets up and running and I can exec into the pod and write data to a mount on the created pvc.
But, two of the volumes on the cluster - NOT RWX - cannot get attached.
I’ve hard deleted them but they end up getting deadline exceeded errors after some time and thereby fail to attach.
Tailing all pods in the longhorn-system namespace, with the tail krew plugin, I see that the removed node is mentioned with a log line stating that the engine image is not ready on the removed worker. And of course not - because the worker is gone.
How can the above situation be remediated?
This issue > https://github.com/longhorn/longhorn/issues/5408 is somewhat similar.
Thank you very much
I’m on longhorn version v1.4.0
Tried upgrading longhorn to v1.4.1 and the error remains