My team runs an RKE2 cluster (installed via ansible), to host rancher on to provision and manage downstream clusters. Occasionally, new pods will get stuck in pending waiting for an IP address from calico. This doesn't resolve itself, until we manually restart the calico pods. The calico pods aren't failing healthchecks, and there's nothing error-looking in the logs. We don't have this on the rancher-managed downstream clusters 😞 Has anyone got any ideas what might be causing this or things we can check?

What version of rke2? What do the calico pod logs and containerd logs say on the nodes where the pods are failing to get an IP?

We're running

v1.24.6+rke2r1

. When we try to spin up a pod, calico logs this line 3 times:

Received *Proto.ServiceAccountUpdate update from calculation graph

And then nothing else - until we kill the calico pod 😞