This message was deleted.

@abundant-hair-58573, sorry, I just want to clarify some terminology so I can understand the situation. When you say you evicted the nodes, do you mean you manipulated the toggles in the Longhorn UI to request replica eviction from them? (Like in the screenshot below?) When you say there are 400 volumes trying to attach at once, do you mean there are 400 replicas rebuilding on other nodes? (I ask because the other scenario I can imagine is that you are using kubectl to drain the nodes or directly using the Kubernetes eviction API. In that case, the volumes are probably trying to detach from the nodes and reattach to other nodes). If it is the first scenario, you can use kubectl to edit the longhorn node object directly and cancel the eviction.

# For example:
kubectl edit -n longhorn-system lhn eweber-v126-worker-9c1451b4-kgxdq
# Set evictionRequested: false.

@faint-sunset-36608 Yes, I set Eviction Requested to True and Node Scheduling to False in the Longhorn UI. There were roughly 400 volumes in attaching status with about 200 in degraded status. So every volume that had a replica on any of those 10 nodes were trying to attach at the same time in order to rebuild the replicas from those nodes. I had already brought 10 new nodes online earlier in the day so there was plenty of storage. I did the same thing a couple weeks ago and it seemed to work fine, but I think 1) there weren't as many replicas on those nodes and 2) I didn't do them all at the exact same time, they were staggered by maybe a couple minutes. The kubernetes API wasn't responsive, I could get to the Longhorn UI briefly when rebooting nodes but the it would freeze up once the control planes froze again. I couldn't add extra control plane nodes to handle the load because all of the other control planes had to be reachable in order to register the new node. I eventually got through it by bringing down each control plane instance, changing the instance type to an m5.2xlarge (64GB of RAM) and then bringing all 5 up at the same time. Even bringing just one up it would get overwhelmed and use all 64GB of memory. It's possible the replicas were rebuilding in the background while all this was going on and I could have just waited it out, but it's unsettling seeing all 5 control planes become unresponsive in a production cluster. Thanks for that command, I'll file that away for next time, although I won't be bringing down that many nodes at once again lol. Just for reference, we had 45 Longhorn nodes at the time, so I didn't think bringing 10 down would overwhelm the kube-apiserver like that.

Ah, thanks for the clarification. I had assumed all volumes were already attached, so I couldn't understand the behavior you described. Yes, looking at the code, I see that: • The number of replicas rebuilding on a node at any given time should be limited by

concurrent-rebuild-limit-per-node

. This setting generally helps to avoid overwhelming disks and network interfaces that running volumes need for I/O. • BUT, there is no limit to the number of volumes the eviction controller might simultaneously attempt to attach when the user takes this kind of action. I too am a bit surprised to see the control plane struggling to keep up like that. However, the volume attachment process does involve Longhorn sending quite a few requests to the API server to track the process and move it along. Would you be willing to open an improvement request regarding this issue? I think we want to specifically look at limiting the number of simultaneous volume attachments the eviction controller can trigger at a time, but including the original issue context (and the impact to your API servers) will help us prioritize it.

Sure thing, thanks for the info.