@creamy-pencil-82913 @gray-lawyer-73831 In reference to https://rancher-users.slack.com/archives/C01PHNP149L/p1652868532770339 Currently facing below error in pod events after setting up private registry with endpoint.

Events:
  Type     Reason     Age                     From               Message
  ----     ------     ----                    ----               -------
  Normal   Scheduled  21s                     default-scheduler  Successfully assigned default/ecr-pod to sv11
  Normal   BackOff    8s                      kubelet            Back-off pulling image "private-registry/utils-img:latest"
  Warning  Failed     8s                      kubelet            Error: ImagePullBackOff
  Normal   Pulling    <invalid> (x2 over 8s)  kubelet            Pulling image "private-registry/utils-img:latest"
  Warning  Failed     <invalid> (x2 over 8s)  kubelet            Failed to pull image "private-registry/utils-img:latest": rpc error: code = NotFound desc = failed to pull and unpack image "private-registry/utils-img:latest": failed to unpack image on snapshotter overlayfs: unexpected media type text/html for sha256:6066282cb389d5bee17ec9f08335850ece266457a557511d3f5e78eabc34df21: not found  
Warning  Failed     <invalid> (x2 over 8s)  kubelet            Error: ErrImagePull

I am trying to add rewrite as mentioned https://github.com/k3s-io/k3s/issues/5502#issuecomment-1109107238 to overcome this issue. After adding rewrite

/etc/rancher/rke2/registries.yaml

its not reflected in the environment, would like to know rewrite in

/var/lib/rancher/rke2/agent/etc/containerd/config.toml

to overcome this issue. correct me if am wrong.

configs:
  "<http://2341234234.dkr.ecr.yyyyy.amazonaws.com|2341234234.dkr.ecr.yyyyy.amazonaws.com>:

you're missing the closing

"

here

Added closing

"

but still no luck.

you should not edit the toml file directly as RKE2 will rewrite it on startup.

/var/lib/rancher/rke2/agent/etc/containerd/config.toml

is generated dynamically, based on the contents of

/etc/rancher/rke2/registries.yaml

since because

/etc/rancher/rke2/registries.yaml

are not reflected in the environment added

config.toml.tmpl

If they aren't getting written into toml then you have a problem with your registries.yaml. you should fix that, not create your own template.

sure got it, let me add the registries again and check it

if its just plain bad yaml you should get an error. Its quite possible that it’s valid yaml, but doesn’t do what you want.

wrong registries.yaml

mirrors:
  harbor.local:
    endpoint:
      - "<https://harbor.local.org/>"
configs:
  "harbor.local":
    tls:
      insecure_skip_verify: true