This message was deleted.

Perhaps I misunderstood the proxy/endpoint bits. I added replicas and added node anti-affinity and it seems to be working for me now. Seems I needed a pod on every node to account for wherever the IP may be "listening". I'd still be interested in how this sometimes worked, maybe some kube-proxy internal I'm not getting.

I think, with externalTrafficPolicy Local, if traffic arrives at a node without a pod, it'll simply drop the packet. So if the speaker for the IP you're reaching isn't on a node with the workload, that's expected. Been a while since I had to look at this, though 🤔

For closure, the issue was one of the hosts not having the correct interface configured, so whenever the workload hit that host, the layer2 advertisement would break and not allow the IP to be accessible external to the cluster. Always something silly.