https://rancher.com/ logo
#rke2
Title
# rke2
m

magnificent-vr-88571

05/23/2022, 12:49 PM
@creamy-pencil-82913 @gray-lawyer-73831 In reference to https://rancher-users.slack.com/archives/C01PHNP149L/p1652868532770339 Currently facing below error in pod events after setting up private registry with endpoint.
Copy code
Events:
  Type     Reason     Age                     From               Message
  ----     ------     ----                    ----               -------
  Normal   Scheduled  21s                     default-scheduler  Successfully assigned default/ecr-pod to sv11
  Normal   BackOff    8s                      kubelet            Back-off pulling image "private-registry/utils-img:latest"
  Warning  Failed     8s                      kubelet            Error: ImagePullBackOff
  Normal   Pulling    <invalid> (x2 over 8s)  kubelet            Pulling image "private-registry/utils-img:latest"
  Warning  Failed     <invalid> (x2 over 8s)  kubelet            Failed to pull image "private-registry/utils-img:latest": rpc error: code = NotFound desc = failed to pull and unpack image "private-registry/utils-img:latest": failed to unpack image on snapshotter overlayfs: unexpected media type text/html for sha256:6066282cb389d5bee17ec9f08335850ece266457a557511d3f5e78eabc34df21: not found  
Warning  Failed     <invalid> (x2 over 8s)  kubelet            Error: ErrImagePull
I am trying to add rewrite as mentioned https://github.com/k3s-io/k3s/issues/5502#issuecomment-1109107238 to overcome this issue. After adding rewrite
/etc/rancher/rke2/registries.yaml
its not reflected in the environment, would like to know rewrite in
/var/lib/rancher/rke2/agent/etc/containerd/config.toml
to overcome this issue. correct me if am wrong.
g

gifted-cricket-25537

05/23/2022, 7:21 PM
Copy code
configs:
  "<http://2341234234.dkr.ecr.yyyyy.amazonaws.com|2341234234.dkr.ecr.yyyyy.amazonaws.com>:
you're missing the closing
"
here
m

magnificent-vr-88571

05/23/2022, 10:10 PM
Added closing
"
but still no luck.
RKE2 pickups up only entries from
/var/lib/rancher/rke2/agent/etc/containerd/config.toml
and `/etc/rancher/rke2/registries.yaml`changes are ignored.
c

creamy-pencil-82913

05/23/2022, 10:14 PM
you should not edit the toml file directly as RKE2 will rewrite it on startup.
/var/lib/rancher/rke2/agent/etc/containerd/config.toml
is generated dynamically, based on the contents of
/etc/rancher/rke2/registries.yaml
If the contents of the registries.yaml are not being reflected in the config.toml, it sounds like you have some syntax errors in that file
Have you by any chance customized the config.toml by providing a custom template?
m

magnificent-vr-88571

05/23/2022, 10:32 PM
since because
/etc/rancher/rke2/registries.yaml
are not reflected in the environment added
config.toml.tmpl
c

creamy-pencil-82913

05/23/2022, 10:43 PM
If they aren't getting written into toml then you have a problem with your registries.yaml. you should fix that, not create your own template.
m

magnificent-vr-88571

05/23/2022, 10:44 PM
sure got it, let me add the registries again and check it
yeah, looks syntax mistake in
registries.yaml
Its good to see an error while we restart rke2 services if there is a syntax mistake in
registries.yaml
while generating
config.toml
.
thank you @creamy-pencil-82913
c

creamy-pencil-82913

05/23/2022, 11:07 PM
if its just plain bad yaml you should get an error. Its quite possible that it’s valid yaml, but doesn’t do what you want.
m

magnificent-vr-88571

05/23/2022, 11:11 PM
wrong registries.yaml
Copy code
mirrors:
  harbor.local:
    endpoint:
      - "<https://harbor.local.org/>"
configs:
  "harbor.local":
    tls:
      insecure_skip_verify: true
corrected
Copy code
mirrors:
  "harbor.local":
34 Views