https://rancher.com/ logo
Title
a

abundant-camera-87627

12/12/2022, 10:44 AM
Is there a way to troubleshoot entries (with verbose logging) in
/etc/rancher/k3s/registries.yaml
? I've entered several entries, but it looks like none of them are being honored. I have entries as follows:
mirrors:
  "<http://docker.io|docker.io>":
    endpoint:
      - "<https://registry>.<fqdn>"
    rewrite:
      "(.*)": "library/docker.io/$1"
However pods are not coming up with the following errors:
Failed to pull image "<http://docker.io/bitnami/sealed-secrets-controller:v0.18.1|docker.io/bitnami/sealed-secrets-controller:v0.18.1>": rpc error: code = Unknown desc = failed to pull and unpack image "<http://docker.io/bitnami/sealed-secrets-controller:v0.18.1|docker.io/bitnami/sealed-secrets-controller:v0.18.1>": failed to resolve reference "<http://docker.io/bitnami/sealed-secrets-controller:v0.18.1|docker.io/bitnami/sealed-secrets-controller:v0.18.1>": failed to do request: Head "<https://registry-1.docker.io/v2/library/docker.io/bitnami/sealed-secrets-controller/manifests/v0.18.1>": dial tcp 44.205.64.79:443: connect: no route to host
Similarly,
k3s ctr image pull <http://docker.io/bitnami/sealed-secrets-controller:v0.18.1|docker.io/bitnami/sealed-secrets-controller:v0.18.1>
returns the same error.
1
c

creamy-pencil-82913

12/12/2022, 4:25 PM
look in the containerd log file, in the k3s agent dir
a

abundant-camera-87627

12/13/2022, 1:12 PM
Thanks for the tip. That log shows the following:
time="2022-12-13T13:09:57.955659153Z" level=info msg="PullImage \"<http://docker.io/bitnami/sealed-secrets-controller:v0.18.1\|docker.io/bitnami/sealed-secrets-controller:v0.18.1\>""
time="2022-12-13T13:09:57.988003491Z" level=info msg="trying next host - response was http.StatusNotFound" host=registry.<fqdn>
time="2022-12-13T13:10:00.955694023Z" level=info msg="PullImage \"<http://docker.io/bitnami/sealed-secrets-controller:v0.18.1\|docker.io/bitnami/sealed-secrets-controller:v0.18.1\>""
time="2022-12-13T13:10:00.986493372Z" level=info msg="trying next host - response was http.StatusNotFound" host=registry.<fqdn>
When I check
resolvectl
it shows the correct DNS server. And I can resolve it with
dig
. What could be causing this?
c

creamy-pencil-82913

12/13/2022, 6:10 PM
You got a NotFound from the registry, not from DNS. It doesn't have the image you tried to pull. Containerd is then falling back to the normal public endpoint.
Are you sure that the image exists under your modified name?
a

abundant-camera-87627

12/14/2022, 7:37 AM
Ahhhh, thanks for explaining. I found the cause now. Apparently there was a mismatch in helmcharts between building the environment and deploying it. Issue was entirely user-error 😄