I’m trying to get stargz work with private image registry when using k3s and containerd. I got the authentication working using deprecated syntax (

[plugins."io.containerd.snapshotter.v1.stargz".registry.mirrors."*"]

and

[plugins."io.containerd.grpc.v1.cri".registry.mirrors."*"]

; not sure if I need both..?). I’d like to get the authentication work using also the new

config_path

&

certs.d

syntax. Does anyone have insight regarding what’s correct and recommended way? I’ve been checking docs of stargz and containerd/cri, but am a bit lost. Also, although the services eventually boot successfully, I’m currently getting lot of errors, like these during k3s install:

time="2022-07-02T14:51:25.485441124+09:00" level=info msg="Received status code: 401 Unauthorized. Refreshing creds..." key="<http://k8s.io/25/extract-737524037-rTpL|k8s.io/25/extract-737524037-rTpL> sha256:47539da01eebacb627943ac3a63d918b69534684b239edac42ce4c64742fb4fd" mountpoint=/var/lib/rancher/k3s/agent/containerd/io.containerd.snapshotter.v1.stargz/snapshotter/snapshots/17/fs parent="<http://k8s.io/24/sha256:114c5ec8519affa4cba972ba546f900b4803d5863f5242a2daee68aa6133fd7d|k8s.io/24/sha256:114c5ec8519affa4cba972ba546f900b4803d5863f5242a2daee68aa6133fd7d>" src="<http://docker.io/rancher/klipper-helm:v0.7.3-build20220613/sha256:5b8b2ba8c050ccd08bd8f00f15210c40a3f50674a0b9457b3f06870723889cd0|docker.io/rancher/klipper-helm:v0.7.3-build20220613/sha256:5b8b2ba8c050ccd08bd8f00f15210c40a3f50674a0b9457b3f06870723889cd0>"

time="2022-07-02T14:51:26.853585280+09:00" level=warning msg="failed to prepare remote snapshot" error="failed to resolve layer: failed to resolve layer \"sha256:5b8b2ba8c050ccd08bd8f00f15210c40a3f50674a0b9457b3f06870723889cd0\" from \"<http://docker.io/rancher/klipper-helm:v0.7.3-build20220613\|docker.io/rancher/klipper-helm:v0.7.3-build20220613\>": failed to resolve the blob: failed to resolve the source: cannot resolve layer: failed to redirect (host \"<http://registry-1.docker.io|registry-1.docker.io>\", ref:\"<http://docker.io/rancher/klipper-helm:v0.7.3-build20220613\|docker.io/rancher/klipper-helm:v0.7.3-build20220613\>", digest:\"sha256:5b8b2ba8c050ccd08bd8f00f15210c40a3f50674a0b9457b3f06870723889cd0\"): failed to access to the registry with code 404: failed to redirect (host \"<http://registry-jpe2.r-local.net|registry-jpe2.r-local.net>\", ref:\"<http://docker.io/rancher/klipper-helm:v0.7.3-build20220613\|docker.io/rancher/klipper-helm:v0.7.3-build20220613\>", digest:\"sha256:5b8b2ba8c050ccd08bd8f00f15210c40a3f50674a0b9457b3f06870723889cd0\"): failed to access to the registry with code 401: failed to resolve: failed to resolve target" key="<http://k8s.io/25/extract-737524037-rTpL|k8s.io/25/extract-737524037-rTpL> sha256:47539da01eebacb627943ac3a63d918b69534684b239edac42ce4c64742fb4fd" parent="<http://k8s.io/24/sha256:114c5ec8519affa4cba972ba546f900b4803d5863f5242a2daee68aa6133fd7d|k8s.io/24/sha256:114c5ec8519affa4cba972ba546f900b4803d5863f5242a2daee68aa6133fd7d>" remote-snapshot-prepared=false

time="2022-07-02T14:51:37.262725372+09:00" level=error msg="ContainerStatus for \"1e7c670c740754bfa1d6e3c4ca2540025adb6cd8063abeb057911f9535d0f1b3\" failed" error="rpc error: code = NotFound desc = an error occurred when try to find container \"1e7c670c740754bfa1d6e3c4ca2540025adb6cd8063abeb057911f9535d0f1b3\": not found"
time="2022-07-02T14:51:37.263923059+09:00" level=error msg="ContainerStatus for \"5af6495a6eeb8d647ce97e2ece46489961799cb711a85533fd4a38892b0af32e\" failed" error="rpc error: code = NotFound desc = an error occurred when try to find container \"5af6495a6eeb8d647ce97e2ece46489961799cb711a85533fd4a38892b0af32e\": not found"

Is this normal with private registries, or do I possibly have some misconfiguration somewhere…?