Hey folks, I have an odd behavior when I drain some nodes via the UI. We have a few pods that take quite a while to shut down fully, and when I drain a node I set it to wait forever, but it seems like there is some hidden timeout. The drain eventually gives up and marks the node as "cordoned" and once the pod finishes terminating I can run the drain again to get the node into a drained state.
If anyone has any suggestion that would be great.