This message was deleted.

Version of Longhorn?

It looks like the Longhorn deployment may be pretty unhealthy, but there's not enough to understand exactly what's going on. Some of the messages indicate that the Longhorn CSI plugin cannot reach the

longhorn-backend

service. This may be because all

longhorn-manager

pods are down, or because there is some cluster networking/DNS issue, or for some other reason. Other logs indicate that a volume can't be deleted because it is still attached. Maybe the

csi-attacher

component is also failing to reach the

longhorn-backend

service to detach the volumes? Or maybe you are trying to delete volumes that are still being used by a workload? Feel free to open a GitHub issue at https://github.com/longhorn/longhorn and please be sure to attach a support bundle when you do. That should allow a developer or community member who has gotten pretty good at diving through the totality of the logs to (hopefully) quickly diagnose an issue or two to look at more closely.

RKE2 1.26.10 2r2, Longhorn 102.3.0+up1.5.1, both managed by Rancher v2.7.9. There is also an nfs-provisioner installed. The problem came up during the upgrade from 1.26.8 to 10. 2 of the 3 nodes had to be rebooted during the upgrade (one after the other) due to some internal NFS-server-not-responding problem. Apparently I should had rebooted the third node as well. Longhorn is still running with the default priority class.

Are you saying rebooting the third node resolved the problem? Or is it still occurring? Unfortunately, these details are not enough to diagnose the issue if it is still ongoing. Will need a support bundle for that.

With the reboot of the 3rd node the problem went away.