Is there a way to troubleshoot entries (with verbose logging) in

/etc/rancher/k3s/registries.yaml

? I've entered several entries, but it looks like none of them are being honored. I have entries as follows:

mirrors:
  "<http://docker.io|docker.io>":
    endpoint:
      - "<https://registry>.<fqdn>"
    rewrite:
      "(.*)": "library/docker.io/$1"

However pods are not coming up with the following errors:

Failed to pull image "<http://docker.io/bitnami/sealed-secrets-controller:v0.18.1|docker.io/bitnami/sealed-secrets-controller:v0.18.1>": rpc error: code = Unknown desc = failed to pull and unpack image "<http://docker.io/bitnami/sealed-secrets-controller:v0.18.1|docker.io/bitnami/sealed-secrets-controller:v0.18.1>": failed to resolve reference "<http://docker.io/bitnami/sealed-secrets-controller:v0.18.1|docker.io/bitnami/sealed-secrets-controller:v0.18.1>": failed to do request: Head "<https://registry-1.docker.io/v2/library/docker.io/bitnami/sealed-secrets-controller/manifests/v0.18.1>": dial tcp 44.205.64.79:443: connect: no route to host

Similarly,

k3s ctr image pull <http://docker.io/bitnami/sealed-secrets-controller:v0.18.1|docker.io/bitnami/sealed-secrets-controller:v0.18.1>

returns the same error.

look in the containerd log file, in the k3s agent dir

Thanks for the tip. That log shows the following:

time="2022-12-13T13:09:57.955659153Z" level=info msg="PullImage \"<http://docker.io/bitnami/sealed-secrets-controller:v0.18.1\|docker.io/bitnami/sealed-secrets-controller:v0.18.1\>""
time="2022-12-13T13:09:57.988003491Z" level=info msg="trying next host - response was http.StatusNotFound" host=registry.<fqdn>
time="2022-12-13T13:10:00.955694023Z" level=info msg="PullImage \"<http://docker.io/bitnami/sealed-secrets-controller:v0.18.1\|docker.io/bitnami/sealed-secrets-controller:v0.18.1\>""
time="2022-12-13T13:10:00.986493372Z" level=info msg="trying next host - response was http.StatusNotFound" host=registry.<fqdn>

When I check

resolvectl

it shows the correct DNS server. And I can resolve it with

dig

. What could be causing this?

You got a NotFound from the registry, not from DNS. It doesn't have the image you tried to pull. Containerd is then falling back to the normal public endpoint.

Ahhhh, thanks for explaining. I found the cause now. Apparently there was a mismatch in helmcharts between building the environment and deploying it. Issue was entirely user-error 😄