hi guys how to override registry endpoint in rke2 I do creat rancher-users #rke2

Title

loud-eve-73457

02/23/2023, 2:38 AM

hi guys, how to override registry endpoint in rke2. I do create a

registries.yaml

and try to put some things like below to make the kubernetes registry point to own harbor proxy cache, but it doesn’t work and the containerd still tries to pull image from the original one. What’s wrong with my configuration? thanks.

mirrors:
  <http://registry.k8s.io|registry.k8s.io>:
    endpoint:
      - "<https://harbor.in.xxx.tech/registry.k8s.io>"
configs:
  "<http://registry.k8s.io|registry.k8s.io>":
    auth:
      username: robot$xxx
      password: xxxxxxxxxxx

I have checked the

config.toml

generated for containerd, and there existed an entry for

<http://registry.k8s.io|registry.k8s.io>

creamy-pencil-82913

02/23/2023, 3:03 AM

endpoint:
      - "<https://harbor.in.xxx.tech/registry.k8s.io>"

This is not going to work

if you want to put a prefix on the images within the registry at

<http://harbor.in|harbor.in>.xxx.tech

you can’t just put that in the base path of the URL. The registry API is still actually at

<https://harbor.in.xxx.tech/v2>

- you need to look at applying rewrites to your mirror config. This is covered in the docs.

loud-eve-73457

02/23/2023, 3:21 AM

thank your very much. I catch the point, but I am still in confused about the rewrite rules, for which the images specified with its registry name.

for example, to override registry.k8s.io, and try to pull an image of ‘registry.k8s.io/coredns/coredns:v1.9.3’. Which rewrite rule below is correct

the first one:

<http://registry.k8s.io|registry.k8s.io>:
    endpoint:
      - "<https://harbor.in.xxx.tech>"
    rewrite:
      "^<http://registry.k8s.io/(.*)|registry.k8s.io/(.*)>": "<http://registry.k8s.io/$1|registry.k8s.io/$1>"

the second one:

<http://registry.k8s.io|registry.k8s.io>:
    endpoint:
      - "<https://harbor.in.xxx.tech>"
    rewrite:
      "^(.*)": "<http://registry.k8s.io/$1|registry.k8s.io/$1>"

I have figured it out, the second one is the right solution.

Another question. what shoudl I do if I want to keep the original one URL as the backoff choice. It is obviously the

rewrite

rule cannot fit for the override one and the origin one at the same time. How should I do. thanks

<http://docker.io|docker.io>:
    endpoint
      - "<https://harbor.in.xxx.tech>"
      - "<https://hub.docker.com>"
    rewrite:
      "^(.*)": "<http://docker.io/$1|docker.io/$1>"

@creamy-pencil-82913 does rke2 have fallback behavior like you mentioned here: https://github.com/rancher/rke2/issues/915 . However under my test there isn’t any fallback behavior by default. If my private registry failed, there will be a image pull error and rke2 won’t go and find the image from the original default registry like hub.docker.io. SO should there be some special configuration to enable this feature. great thanks

creamy-pencil-82913

02/27/2023, 3:58 AM

Nope, containerd always falls back to the default endpoint. Not that this only helps if you're using your local registry as a mirror for another registry. If you're explicitly pulling from your registry then it can't fall back.

loud-eve-73457

02/27/2023, 6:51 AM

great thanks for your help! Is it possible to achieve fallback for local registry. I’am not explicitly pulling from my local registry, but I override e.g.

<http://docker.io|docker.io>

to use my local registry as backend, thus when I pull

kubectl run busybox

, it goes to

local_registry/library/busybox

to find the image. Is there any way to make it fallback to another backend if my local registry failed with out change the front end url, i.e.

kubectl run busybox

@creamy-pencil-82913

I found an issue that you said that containerd could fallback. https://github.com/rancher/rke2/issues/915

creamy-pencil-82913

02/27/2023, 9:22 AM

If you’ve configured the registries properly, it will try your endpoint first and then fall back to the docker.io default endpoint. Check the containerd.log if you are having issues.

loud-eve-73457

02/27/2023, 9:41 AM

I am confusing about the behavior, and there lacks documentation about this. I’m sorry that I still didn’t know the behavior of the fallback. If I understand your comments correctly, your two comments above describe two different behaviors about the same thing. I will appreciate you very much if you can elaborate it for me more clearly with a concrete example below:

<http://docker.io|docker.io>:
    endpoint:
      - "<https://harbor.in.aitopia.tech>"
    rewrite:
      "^(.*)": "<http://docker.io/$1|docker.io/$1>"

what would it behave if the endpoint list in the endpoint failed. It will fallback to registry-1.docker.io the default one to pull the image, or it will failed pulling. Under my test, it’s the second one, it failed. What should I do to make it behave as the first one. Great thanks and there is few documentation about the configuration of rke2 registry. @creamy-pencil-82913

35 Views

#rke2 Join Slack