https://rancher.com/ logo
Join Slack
Powered by
This message was deleted.
# rke2
a
This message was deleted.
c
Nope. Up to date and supported by all packaged versions of containerd. What are you seeing that makes you think it's not working?
t
Well that's good. I have a "fake" registry (i.e. one whose hostname doesn't resolve) added with a mirror endpoint pointing to a "real" registry. From the docs, I would assume that "ctr image pull fake.registry/real/image" would get translated to "real.registry/real/image", but in containerd.log, I can see errors like msg="PullImage ..." which show it still using "https://fake.registry/..." as the URL. I can see in containerd.toml that the matching fields have been set.
One complicating factor that just occurred to me: This is behind an proxy with the HTTP(S)_PROXY variables set. The real registry matches NO_PROXY, the fake registry doesn't, but I would assume that that would happen after the "real" URL is built.
c
keep in mind that containerd will always fall back to using the “fake” registry’s default endpoint as a fallback if it cannot pull from the listed endpoints.
t
Ah, I wonder if that's causing a red herring in the logs. I don't suppose there's any way to disable that?
c
nope
just what containerd does
t
Might want to note that in the docs, then. It should always try the mirror endpoints first, though, right?
c
you might need to look further back in the logs to find the initial error it’s encountering when pulling from your configured endpoint
yes
We do link directly to the containerd docs that cover that, from https://docs.rke2.io/install/containerd_registry_configuration#mirrors:
Note: If no endpoint is configured, containerd assumes that the registry can be accessed anonymously via HTTPS on port 443, and is using a certificate trusted by the host operating system. For more information, you may consult the containerd documentation.
which says:
The endpoint is a list that can contain multiple image registry URLs split by commas. When pulling an image from a registry, containerd will try these endpoint URLs one by one, and use the first working one. Please note that if the default registry endpoint is not already specified in the endpoint list, it will be automatically tried at the end with scheme 
https
and path 
v2
, e.g. 
<https://gcr.io/v2>
for 
<http://gcr.io|gcr.io>
.
t
Ah, I assumed that link was only pertinent to the "if no endpoint is configured", which wouldn't apply to this case, since, I do.
Just went back and tried again, no logs showing it using the "real" URL, should those not show up? Does it only ever show the "final" error or something?
c
no, it should show all of them. You are using the containerd bundled with rke2, right? not using your own containerd that you have pointed rke2 at with --container-runtime-endpoint?
what does your registries.yaml look like, and what image are you pulling?
t
Oh, wait, I see the problem now. Those errors are only showing the host, not the actual image, and those got filtered out by the logging filter view.
c
Also, are you pulling it via a pod spec, or just directly with ctr?
t
I had other pods that were correctly pulling directly from the real registry, so I filtered the logs for just those that showed the image in particular, but since the containerd logs only had the image on the "final" error, it looked like it was only attempting the fake registry. I think I can fix it from here. Thanks for your time!
c
gl!
24 Views
Open in Slack
PreviousNext
Powered by