For traffic that comes in to a node that doesn't have a traefik replica running, serviceLB (I'm assuming) routes the traffic to one of the nodes that is running traefik. But the packet that traefik sees has the pod IP of the serviceLB pod as the source, and not the original source IP. This limits the matching space to the ephemeral ports that serviceLB will choose for routing traffic. This is a lot, but it's a lot less than letting the source IP vary. I worry that it is fragile and that something will break, or that it will fail to load balance correctly because two connections that come in from different IPs but happen to use the same port will be forced to use the same backend replica.
For IPv6, serviceLB could just allocate a block of pod IPs and then hash the incoming source IP/port pair to determine the outgoing pod IP/port. This could easily be large enough that you were unlikely to run into problems. For IPv4, the range of pod IPs is small enough that you wouldn't want serviceLB to allocate a large block of them.
I'm not sure there is a solution that will do what I want. In order for traefik to reply correctly, it has to know where to send the response packet. If the incoming connection originated on a different node, the response has to be sent there first so that it can be rewritten going the other direction.
Maybe the solution is to take traefik out of the loop entirely? Does serviceLB have configurable sourceIP/sourcePort to backend affinity with a timeout? If not, is there another LB that does, where I can replace serviceLB with that LB?