Hi! I have a question about the embedded registry mirror. I have a setup with a master node that has access to Internet and a private registry (mTLS identified) and this works well. I have added an agent node that has no direct access to the private registry but instead pulls the required images from the master node using the embedded registry mirror. This works well but, sometimes, I can't pull certain images from the master onto the agent even though the image is still present on the master as shown by a

crictl images

command. When the agent tries to pull the image, I get this error in the containerd logs:

time="2025-05-02T09:55:57.977646667Z" level=info msg="PullImage \"<http://my.private-registry.com/rc/myapp/monitorii:2.10.0-rc20\|my.private-registry.com/rc/myapp/monitorii:2.10.0-rc20\>""
time="2025-05-02T09:55:58.195789028Z" level=error msg="PullImage \"<http://my.private-registry.com/rc/myapp/monitorii:2.10.0-rc20\|my.private-registry.com/rc/myapp/monitorii:2.10.0-rc20\>" failed" error="failed to pull and unpack image \"<http://my.private-registry.com/rc/myapp/monitorii:2.10.0-rc20\|my.private-registry.com/rc/myapp/monitorii:2.10.0-rc20\>": failed to copy: httpReadSeeker: failed open: unexpected status code <https://127.0.0.1:6443/v2/rc/myapp/monitorii/blobs/sha256:a3f1dfe736c5f959143f23d75ab522a60be2da902efac236f4fb2a153cc14a5d?ns=my.private-registry.com>: 500 Internal Server Error"
time="2025-05-02T09:55:58.195878522Z" level=info msg="stop pulling image <http://my.private-registry.com/rc/myapp/monitorii:2.10.0-rc20|my.private-registry.com/rc/myapp/monitorii:2.10.0-rc20>: active requests=0, bytes read=8913"

Does anyone have any pointers or suggestions on things I could check? Thanks a lot!

You could enable debug on both nodes and see if you get any additional info on the error. Mentioning the version you're running would also be helpful.

Hi! Thanks for your answer. I enabled debug on both nodes and have a bit more info but not much. I was using 1.29.4 and tried updating to 1.29.15 and now to 1.30.11 to see if there was any difference. As I started from scratch, I did not yet run into the issue again, so we'll see! From what I gathered from the logs, the agent could find the master's peer for Spegel correctly, managed to get the HEAD and GET on the manifest as well as one blob of the image, but subsequent blob fetching failed and the IP mentioned in the logs was no longer that of the master peer but 127.0.0.1 which seems strange? But that could also be normal, I don't really know Spegel's/Containerd's internals

containerd always talks to spegel at the loopback address. spegel in turn will go out to other nodes to fetch content. it acts like a proxy.

I see! But I don't understand why the first blob was ok and the others were not 🤔 If I manually launched a pod on the master node, it was working normally.

hard to know for sure

Thanks for your time anyways! I will report back if I still get strange issues now that I have upgraded k3s