Channels
extensions
rancher-extensions
supermicro-sixsq
ozt
os
one-point-x
rio
onlinemeetup
v16-v21-migration
events
onlinetraining
training-1220
training-0110
training-0124
k3s-contributor
submariner
storage
k3os
windows
ci-cd
theranchcast
rfed_ara
nederlands
ukranian
danish
training-0131
masterclass
training-0207
training-0214
terraform-controller
epinio
phillydotnet
swarm
rancher-wrangler
deutsch
russian
chinese
mesos
français
japanese
arm
longhorn-dev
terraform-provider-rke
service-mesh
azure
gcp
academy
random
elemental
espanol
lima
harvester-dev
portugues
kubernetes
kim
hypper
kubewarden
opni
logging
neuvector-security
rancher-setup
developer
office-hours
mexico
cabpr
rke
vsphere
longhorn-storage
k3s
k3d
terraform-provider-rancher2
fleet
amazon
harvester
rke2
s3gw
hobbyfarm
rancher-desktop
general
Title
t
thankful-train-2781
02/22/2023, 5:06 PMHello. Are these docs current? The "mirrors" section doesn't appear to reflect reality. Specifically, the "endpoint" field seems to be completely disregarded by containerd, despite it being added to the generated toml file, and it always uses the registry component of the image id as the hostname. The "configs" section works just fine. Did "mirrors" get deprecated or something?
c
creamy-pencil-82913
02/22/2023, 5:31 PMNope. Up to date and supported by all packaged versions of containerd. What are you seeing that makes you think it's not working?
t
thankful-train-2781
02/22/2023, 5:34 PMWell that's good. I have a "fake" registry (i.e. one whose hostname doesn't resolve) added with a mirror endpoint pointing to a "real" registry. From the docs, I would assume that "ctr image pull fake.registry/real/image" would get translated to "real.registry/real/image", but in containerd.log, I can see errors like
msg="PullImage ..." which show it still using "https://fake.registry/..." as the URL. I can see in containerd.toml that the matching fields have been set.
One complicating factor that just occurred to me: This is behind an proxy with the HTTP(S)_PROXY variables set. The real registry matches NO_PROXY, the fake registry doesn't, but I would assume that that would happen after the "real" URL is built.
c
creamy-pencil-82913
02/22/2023, 5:51 PMkeep in mind that containerd will always fall back to using the “fake” registry’s default endpoint as a fallback if it cannot pull from the listed endpoints.
t
thankful-train-2781
02/22/2023, 5:51 PMAh, I wonder if that's causing a red herring in the logs. I don't suppose there's any way to disable that?
c
creamy-pencil-82913
02/22/2023, 5:52 PMnope
just what containerd does
t
thankful-train-2781
02/22/2023, 5:52 PMMight want to note that in the docs, then. It should always try the mirror endpoints first, though, right?
c
creamy-pencil-82913
02/22/2023, 5:52 PMyou might need to look further back in the logs to find the initial error it’s encountering when pulling from your configured endpoint
yes
We do link directly to the containerd docs that cover that, from https://docs.rke2.io/install/containerd_registry_configuration#mirrors:
Note: If no endpoint is configured, containerd assumes that the registry can be accessed anonymously via HTTPS on port 443, and is using a certificate trusted by the host operating system. For more information, you may consult the containerd documentation.which says:
The endpoint is a list that can contain multiple image registry URLs split by commas. When pulling an image from a registry, containerd will try these endpoint URLs one by one, and use the first working one. Please note that if the default registry endpoint is not already specified in the endpoint list, it will be automatically tried at the end with schemeand pathhttps, e.g.v2for<https://gcr.io/v2>.<http://gcr.io|gcr.io>
t
thankful-train-2781
02/22/2023, 5:56 PMAh, I assumed that link was only pertinent to the "if no endpoint is configured", which wouldn't apply to this case, since, I do.
Just went back and tried again, no logs showing it using the "real" URL, should those not show up? Does it only ever show the "final" error or something?
c
creamy-pencil-82913
02/22/2023, 6:06 PMno, it should show all of them. You are using the containerd bundled with rke2, right? not using your own containerd that you have pointed rke2 at with --container-runtime-endpoint?
what does your registries.yaml look like, and what image are you pulling?
t
thankful-train-2781
02/22/2023, 6:06 PMOh, wait, I see the problem now. Those errors are only showing the host, not the actual image, and those got filtered out by the logging filter view.
c
creamy-pencil-82913
02/22/2023, 6:06 PMAlso, are you pulling it via a pod spec, or just directly with ctr?
t
thankful-train-2781
02/22/2023, 6:08 PMI had other pods that were correctly pulling directly from the real registry, so I filtered the logs for just those that showed the image in particular, but since the containerd logs only had the image on the "final" error, it looked like it was only attempting the fake registry. I think I can fix it from here. Thanks for your time!
c
creamy-pencil-82913
02/22/2023, 6:09 PMgl!