https://rancher.com/ logo
Join Slack
Powered by
I am running into a problem where k3s seems to be ...
# general
b
I am running into a problem where k3s seems to be ignoring 
registries.yaml
. I am using k3s through k3d. 
cat /etc/rancher/k3s/registries.yaml
returns:
Copy code
mirrors:
  '*':
    endpoint:
    - <http://k3d-foobar-registry:5000>
    rewrite: {}
  k3d-foobar-registry:5000:
    endpoint:
    - <http://k3d-foobar-registry:5000>
    rewrite: {}
  k3d-foobar-registry:59943:
    endpoint:
    - <http://k3d-foobar-registry:5000>
    rewrite: {}
configs: {}
auths: {}
But `cat /var/lib/rancher/k3s/agent/etc/containerd/certs.d/k3d-foobar-registry:5000/hosts.toml`returns:
Copy code
# File generated by k3s. DO NOT EDIT.

server = "<https://k3d-foobar-registry:5000/v2>"
capabilities = ["pull", "resolve", "push"]


[host]
[host."<http://k3d-foobar-registry:5000/v2>"]
  capabilities = ["pull", "resolve"]
In other words, it's defaulting to using HTTPS in spite of 
registries.yaml
. I checked 
containerd.log
and there is no mention of 
registries.yaml
. It tries using 
https
right away with no attempt to use 
http
. Any idea on how to debug this further? Thanks in advance.
c
that’s not whats happening
b
👀
c
there is a mirror with http as you requested, and then the default (as always) is https
read the k3s registry docs about default endpoints
b
Right. That's what I thought. So instead of the mirror applying to https, it's applying to http. And since no one tries using http then the mirror is ignored. Is that correct?
c
your registry is 
k3d-foobar-registry:5000
. It will try the mirror host first which is http, and if that does not work it will fall back to the default endpoint which is https
b
That's what I thought too, but I don't see it trying http first in 
containerd.log
I also noticed this line in containerd.log which seems to imply that the list of mirrors is empty...
Copy code
time="2025-02-06T14:52:35.921324308Z" level=info msg="Start cri plugin with config {PluginConfig:{ContainerdConfig:{Snapshotter:overlayfs DefaultRuntimeName:runc DefaultRuntime:{Type: Path: Engine: PodAnnotations:[] ContainerAnnotations:[] Root: Options:map[] PrivilegedWithoutHostDevices:false PrivilegedWithoutHostDevicesAllDevicesAllowed:false BaseRuntimeSpec: NetworkPluginConfDir: NetworkPluginMaxConfNum:0 Snapshotter: SandboxMode:} UntrustedWorkloadRuntime:{Type: Path: Engine: PodAnnotations:[] ContainerAnnotations:[] Root: Options:map[] PrivilegedWithoutHostDevices:false PrivilegedWithoutHostDevicesAllDevicesAllowed:false BaseRuntimeSpec: NetworkPluginConfDir: NetworkPluginMaxConfNum:0 Snapshotter: SandboxMode:} Runtimes:map[runc:{Type:io.containerd.runc.v2 Path: Engine: PodAnnotations:[] ContainerAnnotations:[] Root: Options:map[SystemdCgroup:false] PrivilegedWithoutHostDevices:false PrivilegedWithoutHostDevicesAllDevicesAllowed:false BaseRuntimeSpec: NetworkPluginConfDir: NetworkPluginMaxConfNum:0 Snapshotter: SandboxMode:podsandbox}] NoPivot:false DisableSnapshotAnnotations:true DiscardUnpackedLayers:false IgnoreBlockIONotEnabledErrors:false IgnoreRdtNotEnabledErrors:false} CniConfig:{NetworkPluginBinDir:/bin NetworkPluginConfDir:/var/lib/rancher/k3s/agent/etc/cni/net.d NetworkPluginMaxConfNum:1 NetworkPluginSetupSerially:false NetworkPluginConfTemplate: IPPreference:} Registry:{ConfigPath:/var/lib/rancher/k3s/agent/etc/containerd/certs.d Mirrors:map[] Configs:map[] Auths:map[] Headers:map[]} ImageDecryption:{KeyModel:node} DisableTCPService:true StreamServerAddress:127.0.0.1 StreamServerPort:10010 StreamIdleTimeout:4h0m0s EnableSelinux:false SelinuxCategoryRange:1024 SandboxImage:rancher/mirrored-pause:3.6 StatsCollectPeriod:10 SystemdCgroup:false EnableTLSStreaming:false X509KeyPairStreaming:{TLSCertFile: TLSKeyFile:} MaxContainerLogLineSize:16384 DisableCgroup:false DisableApparmor:false RestrictOOMScoreAdj:false MaxConcurrentDownloads:3 DisableProcMount:false UnsetSeccompProfile: TolerateMissingHugetlbController:true DisableHugetlbController:true DeviceOwnershipFromSecurityContext:false IgnoreImageDefinedVolumes:false NetNSMountsUnderStateDir:false EnableUnprivilegedPorts:true EnableUnprivilegedICMP:true EnableCDI:false CDISpecDirs:[/etc/cdi /var/run/cdi] ImagePullProgressTimeout:5m0s DrainExecSyncIOTimeout:0s ImagePullWithSyncFs:false IgnoreDeprecationWarnings:[]} ContainerdRootDir:/var/lib/rancher/k3s/agent/containerd ContainerdEndpoint:/run/k3s/containerd/containerd.sock RootDir:/var/lib/rancher/k3s/agent/containerd/io.containerd.grpc.v1.cri StateDir:/run/k3s/containerd/io.containerd.grpc.v1.cri}"
Sorry for the long paste. It says 
Mirrors:map[]
somewhere near the middle.
c
yes, because the mirrors are not in the config.
they are under 
ConfigPath:/var/lib/rancher/k3s/agent/etc/containerd/certs.d
which is the non-deprecated way to configure mirrors
b
Okay, so let's take a step back... are you saying that the 
hosts.toml
I pasted instructs containerd to try the http endpoint first, and if it fails to use https as fallback? And the reason it's using https is that http is failing? Or are you saying something else? You mentioned reading teh k3s registry docs about default endpoints, but is https://docs.k3s.io/installation/private-registry#default-endpoint-fallback the one you meant?
c
yes
if you see it trying https that means that it is falling back to that, because it was unable to pull from https
if you turn the log level up to trace or debug you will see all the http requests
b
Okay, so if I increase the log level of k3s (or k3d in my case) I should see evidence of the http endpoint being used in containerd.log? Or should I look somewhere else?
I'm going to try adding:
Copy code
[debug]
  level = "debug"
to 
/var/lib/rancher/k3s/agent/etc/containerd/config.toml.tmpl
and restarting the cluster, then I'll check 
/var/lib/rancher/k3s/agent/containerd/containerd.log
again.
@creamy-pencil-82913 You're right. I had to use "trace" logging level instead of "debug" but sure enough it's making the request to HTTP first, gets back HTTP 404 and tries again against HTTPS. Thanks for your help!
c
yep. containerd works pretty well. if it’s not doing what you think it should be doing, the error is almost always on the user side.
it doesn’t always tell you that though ;)
b
👍 though you might want to considering indicating the use of the endpoint at a higher logging level. I've seen this question come up very often on GitHub.
c
I’m not a containerd maintainer, you can find them on CNCF slack or at https://github.com/containerd/containerd
👀 1
b
Thanks
Open in Slack
PreviousNext
Powered by