https://rancher.com/ logo
Title
t

thankful-train-2781

02/22/2023, 5:06 PM
Hello. Are these docs current? The "mirrors" section doesn't appear to reflect reality. Specifically, the "endpoint" field seems to be completely disregarded by containerd, despite it being added to the generated toml file, and it always uses the registry component of the image id as the hostname. The "configs" section works just fine. Did "mirrors" get deprecated or something?
c

creamy-pencil-82913

02/22/2023, 5:31 PM
Nope. Up to date and supported by all packaged versions of containerd. What are you seeing that makes you think it's not working?
t

thankful-train-2781

02/22/2023, 5:34 PM
Well that's good. I have a "fake" registry (i.e. one whose hostname doesn't resolve) added with a mirror endpoint pointing to a "real" registry. From the docs, I would assume that "ctr image pull fake.registry/real/image" would get translated to "real.registry/real/image", but in containerd.log, I can see errors like msg="PullImage ..." which show it still using "https://fake.registry/..." as the URL. I can see in containerd.toml that the matching fields have been set.
One complicating factor that just occurred to me: This is behind an proxy with the HTTP(S)_PROXY variables set. The real registry matches NO_PROXY, the fake registry doesn't, but I would assume that that would happen after the "real" URL is built.
c

creamy-pencil-82913

02/22/2023, 5:51 PM
keep in mind that containerd will always fall back to using the “fake” registry’s default endpoint as a fallback if it cannot pull from the listed endpoints.
t

thankful-train-2781

02/22/2023, 5:51 PM
Ah, I wonder if that's causing a red herring in the logs. I don't suppose there's any way to disable that?
c

creamy-pencil-82913

02/22/2023, 5:52 PM
nope
just what containerd does
t

thankful-train-2781

02/22/2023, 5:52 PM
Might want to note that in the docs, then. It should always try the mirror endpoints first, though, right?
c

creamy-pencil-82913

02/22/2023, 5:52 PM
you might need to look further back in the logs to find the initial error it’s encountering when pulling from your configured endpoint
yes
We do link directly to the containerd docs that cover that, from https://docs.rke2.io/install/containerd_registry_configuration#mirrors:
Note: If no endpoint is configured, containerd assumes that the registry can be accessed anonymously via HTTPS on port 443, and is using a certificate trusted by the host operating system. For more information, you may consult the containerd documentation.
which says:
The endpoint is a list that can contain multiple image registry URLs split by commas. When pulling an image from a registry, containerd will try these endpoint URLs one by one, and use the first working one. Please note that if the default registry endpoint is not already specified in the endpoint list, it will be automatically tried at the end with scheme
https
and path
v2
, e.g.
<https://gcr.io/v2>
for
<http://gcr.io|gcr.io>
.
t

thankful-train-2781

02/22/2023, 5:56 PM
Ah, I assumed that link was only pertinent to the "if no endpoint is configured", which wouldn't apply to this case, since, I do.
Just went back and tried again, no logs showing it using the "real" URL, should those not show up? Does it only ever show the "final" error or something?
c

creamy-pencil-82913

02/22/2023, 6:06 PM
no, it should show all of them. You are using the containerd bundled with rke2, right? not using your own containerd that you have pointed rke2 at with --container-runtime-endpoint?
what does your registries.yaml look like, and what image are you pulling?
t

thankful-train-2781

02/22/2023, 6:06 PM
Oh, wait, I see the problem now. Those errors are only showing the host, not the actual image, and those got filtered out by the logging filter view.
c

creamy-pencil-82913

02/22/2023, 6:06 PM
Also, are you pulling it via a pod spec, or just directly with ctr?
t

thankful-train-2781

02/22/2023, 6:08 PM
I had other pods that were correctly pulling directly from the real registry, so I filtered the logs for just those that showed the image in particular, but since the containerd logs only had the image on the "final" error, it looked like it was only attempting the fake registry. I think I can fix it from here. Thanks for your time!
c

creamy-pencil-82913

02/22/2023, 6:09 PM
gl!