This message was deleted.

Have you considered swapping ingress controller? Perhaps a different one in your dev cluster for testing could be a good idea. Yeah, it would likely mean some downtime, but a planned hour of so is likely better than customers seeing a 503.

Hi Josh, thanks for responding - So the 503s are not consistent. We don't believe it is the ingress controller or any of the virtual services (at this stage) as we are able to (both externally and internally) access some pods / services but not others. Our current (not) working theory is that there has been a change in what is and isn't allowed for pod / service specs in the underlying kubernetes or RKE2 between 1.22 and 1.24 and we're falling foul of that. We're ruled out the DNS by using the svc FQDN and been testing curl from both the host and within the CNI (inside a pod).

Right. So what is your 1.22 ingress look like vs a 1.24.x ingress?

Ingress is exactly the same - Istio ingress gateway using sidecar proxy injection and virtual services. Same version for both (1.16.5) to minimise the moving parts (as part of our leap frog approach)

Right. I'm afraid I am not going to be much help then. I tend to stick with the non-sidecar approach and the simple controllers like HAProxy and Nginx.

No problem / thanks for having a stab anyway