Hey guys. Im using system-upgrade-controller to upgrade my k3s cluster version. I found myself in a rather strange situation. After upgrading my master node from 1.30 to 1.31, the kube-system pods (scheduled on worker node) restarted as expected but they pulled their images from upstream registry instead of waiting for images to load in the local registry. i have the images.tar.zst in the correct folder ie; agent/images folder before k3s boot I was able to confirm that it was an upstream pull by disabling default-endpoint-registry. the pods went to ErrImagePull(which shows they wanna fetch from upstream but it was disallowed) followed by transitioning into running state after images were available in the local registry. (see screenshot)

The kubelet doesn't even get started until after all the images are imported, regardless of version. What do the service logs show during startup?

Thanks @creamy-pencil-82913 for the response.

Oh, I see, you're saying that the server upgrade caused new pods to run on the agents, but the agents hadn't been upgraded yet so they hadn't imported the images

yes thats the exact behavior thats happening

Are you using spegel?

yes Im using embedded registry.

You might consider preloading the images before the upgrade

You mean to move the tar.zst to images folder and restart k3s server before an upgrade? I had only moved tar.zst to images folder in the prepare step. Im expecting cluster upgrade to handle the k3s restart. tho I see the correct behavior in 1.32, What really changed in 1.32? kube-system pods get scheduled/restarted on agents only after the images are completed loaded in the registry.