magnificent-vr-88571
05/18/2022, 10:08 AMrke2 --version
rke2 version v1.21.5+rke2r2 (9e4acdc6018ae74c36523c99af25ab861f3884da)
go version go1.16.6b7
I have created registries.yaml with aws ECR as private registry
mirrors:
<http://2341234234.dkr.ecr.yyyyy.amazonaws.com|2341234234.dkr.ecr.yyyyy.amazonaws.com>:
endpoint:
- "<https://2341234234.dkr.ecr.yyyyy.amazonaws.com>"
configs:
"<http://2341234234.dkr.ecr.yyyyy.amazonaws.com|2341234234.dkr.ecr.yyyyy.amazonaws.com>:
auth:
username: xxxxx
password: xxxxx
tls:
insecure_skip_verify: true
While I create a pod to pull registry with following image name in pod.yaml, it fails with image pull error.
image: <http://2341234234.dkr.ecr.yyyyy.amazonaws.com/image-name:v1|2341234234.dkr.ecr.yyyyy.amazonaws.com/image-name:v1>
Looking forward some insightslevel=info msg="Using private registry config file at /etc/rancher/rke2/registries.yaml"
/etc/rancher/rke2/registries.yaml
in all agent nodes & restarted service.creamy-pencil-82913
05/18/2022, 5:28 PMmagnificent-vr-88571
05/18/2022, 10:03 PMEvents:
Type Reason Age From Message
---- ------ ---- ---- -------
Normal Scheduled 39m default-scheduler Successfully assigned default/ecr-pod to sv10
Normal Pulling 38m (x4 over 39m) kubelet Pulling image "<http://2341234234.dkr.ecr.yyyyy.amazonaws.com/image-name:v1|2341234234.dkr.ecr.yyyyy.amazonaws.com/image-name:v1>"
Warning Failed 38m (x4 over 39m) kubelet Failed to pull image "<http://2341234234.dkr.ecr.yyyyy.amazonaws.com/image-name:v1|2341234234.dkr.ecr.yyyyy.amazonaws.com/image-name:v1>": rpc error: code = Unknown desc = failed to pull and unpack image "<http://2341234234.dkr.ecr.yyyyy.amazonaws.com/image-name:v1|2341234234.dkr.ecr.yyyyy.amazonaws.com/image-name:v1>": failed to resolve reference "<http://2341234234.dkr.ecr.yyyyy.amazonaws.com/image-name:v1|2341234234.dkr.ecr.yyyyy.amazonaws.com/image-name:v1>": pulling from host <http://2341234234.dkr.ecr.yyyyy.amazonaws.com|2341234234.dkr.ecr.yyyyy.amazonaws.com> failed with status code [manifests slim]: 401 Unauthorized
Warning Failed 38m (x4 over 39m) kubelet Error: ErrImagePull
Warning Failed 38m (x6 over 39m) kubelet Error: ImagePullBackOff
Normal BackOff 4m35s (x151 over 39m) kubelet Back-off pulling image "<http://2341234234.dkr.ecr.yyyyy.amazonaws.com/image-name:v1|2341234234.dkr.ecr.yyyyy.amazonaws.com/image-name:v1>"
creamy-pencil-82913
05/18/2022, 10:07 PMaws ecr login
gives you expire very frequently.magnificent-vr-88571
05/18/2022, 10:08 PM```configs:
"2341234234.dkr.ecr.yyyyy.amazonaws.com:
auth:
username: xxxxx
password: xxxxx
tls:
insecure_skip_verify: true```in /etc/rancher/rke2/registries.yaml
creamy-pencil-82913
05/18/2022, 10:08 PMWhat are you using for the username and password
magnificent-vr-88571
05/18/2022, 10:09 PMcreamy-pencil-82913
05/18/2022, 10:09 PMaws ecr login
, or whatThis command retrieves and displays an authentication token using the GetAuthorizationToken API that you can use to authenticate to an Amazon ECR registry. You can pass the authorization token to the login command of the container client of your preference, such as the Docker CLI. After you have authenticated to an Amazon ECR registry with this command, you can use the client to push and pull images from that registry as long as your IAM principal has access to do so until the token expires. The authorization token is valid for 12 hours.
magnificent-vr-88571
05/18/2022, 10:11 PMcreamy-pencil-82913
05/18/2022, 10:11 PMmagnificent-vr-88571
05/18/2022, 10:12 PMcreamy-pencil-82913
05/18/2022, 10:14 PMmagnificent-vr-88571
05/18/2022, 10:24 PM/etc/rancher/rke2/registries.yaml
with new token for password we need to restart rke2-server too right?creamy-pencil-82913
05/18/2022, 10:25 PMmagnificent-vr-88571
05/18/2022, 10:26 PM/etc/rancher/rke2/registries.yaml
rke2 failedcreamy-pencil-82913
05/18/2022, 10:32 PMmagnificent-vr-88571
05/18/2022, 10:37 PM/etc/rancher/rke2/registries.yaml
should be reflected to all worker nodes by rke2-server would have been good.
not only for ECR but not any registries../var/lib/rancher/rke2/agent/etc/containerd/config.toml
add /etc/rancher/rke2/registries.yaml
with following format allowed to access images from private registry, and on execution of crictl info
we can see the private registry.
config.toml
[plugins.opt]
path = "/var/lib/rancher/rke2/agent/containerd"
[plugins.cri]
stream_server_address = "127.0.0.1"
stream_server_port = "10010"
enable_selinux = false
sandbox_image = "<http://index.docker.io/rancher/pause:3.5|index.docker.io/rancher/pause:3.5>"
[plugins.cri.containerd]
disable_snapshot_annotations = true
snapshotter = "overlayfs"
[plugins.cri.registry.mirrors]
[plugins.cri.registry.mirrors."<http://harbor-local.org|harbor-local.org>"]
endpoint = ["<https://harbor-local.org/>"]
registry
......
registries.yaml
mirrors:
<http://harbor-local.org|harbor-local.org>:
endpoint:
- "<https://harbor-local.org>"
configs:
"<http://harbor-local.org|harbor-local.org>":
auth:
username: xxxx
password: xxxx
tls:
insecure_skip_verify: true
.....
Another rke2 cluster with /var/lib/rancher/rke2/agent/etc/containerd/config.toml
and /etc/rancher/rke2/registries.yaml
with following format were able to connect with private registry and facing on execution of crictl info
we can are not able to see the private registry.
config.toml
version = 2
[plugins]
[plugins."io.containerd.grpc.v1.cri"]
enable_selinux = false
sandbox_image = "<http://index.docker.io/rancher/pause:3.2|index.docker.io/rancher/pause:3.2>"
stream_server_address = "127.0.0.1"
stream_server_port = "10010"
[plugins."io.containerd.grpc.v1.cri".containerd]
default_runtime_name = "nvidia"
disable_snapshot_annotations = true
snapshotter = "overlayfs"
````
registries.yaml
mirrors:
<http://harbor-local.org|harbor-local.org>:
endpoint:
- "<https://harbor-local.org>"
configs:
"<http://harbor-local.org|harbor-local.org>":
auth:
username: xxxx
password: xxxx
tls:
insecure_skip_verify: true
```
Looking for guidance.
RKE2 version in both cluster is
rke2 version v1.21.5+rke2r2 (9e4acdc6018ae74c36523c99af25ab861f3884da)
go version go1.16.6b7
/var/lib/rancher/rke2/agent/etc/containerd/config.toml
to include private registry details.
https://github.com/containerd/cri/blob/release/1.4/docs/registry.md
post these steps make sure to restart rke2 services