Hello I am running k3s in docker and as part of my setup I a rancher-users #k3s

Title

miniature-honey-15162

03/23/2023, 8:15 PM

Hello! I am running k3s in docker and as part of my setup, I am using a private docker registry where I am following this documentation and have declared the following YAML file in

/etc/ranger/k3s/

mirrors:
  <http://itregistry.io|itregistry.io>:
    endpoint:
    - <http://it:5000>

However, when I try to create Deployment (that contains an image located at

<http://it:5000>

I get the following error:

failed to resolve reference "<http://itregistry.io/fix-server:0.1.0|itregistry.io/fix-server:0.1.0>": failed to do request: Head "<https://itregistry.io/v2/fix-server/manifests/0.1.0>": dial tcp: lookup <http://itregistry.io|itregistry.io> on 127.0.0.11:53: no such host

More context: • my docker registry is itself running in docker. My k3s and docker registry are in the same docker network. Things I have tried: • I am able to "ping" the registry from the k3s container e.g.

/ # wget --spider <http://it:5000>
Connecting to it:5000 (192.168.32.2:5000)
remote file exists

//...
/ # wget <http://it:5000/v2/fix-server/manifests/0.1.0>
Connecting to it:5000 (192.168.80.2:5000)
saving to '0.1.0'
0.1.0                100% |********************************************************************| 18278  0:00:00 ETA
'0.1.0' saved
/ #

Does anyone see anything immediately wrong or have any suggestions in how to troubleshoot this issue? Thanks in advance!

creamy-pencil-82913

03/23/2023, 8:22 PM

well for starters its /etc/rancher/k3s/registries.yaml not /etc/ranger

miniature-honey-15162

03/23/2023, 8:24 PM

oh that was a typo 🤦‍♂️

/ # cat /etc/rancher/k3s/registries.yaml
mirrors:
  <http://itregistry.io|itregistry.io>:
    endpoint:
    - <http://it:5000>
/ #

Will edit the original post

creamy-pencil-82913

03/23/2023, 8:25 PM

and you have it at that path inside the docker container where you’re running k3s? Not outside the docker container?

I would also check containerd.log to see if there are any other errors in there, the kubelet will often times only show the last in a sequence of errors.

👍 1

miniature-honey-15162

03/23/2023, 8:26 PM

It is within the container.

❯ docker exec (docker ps | rg k3s | awk '{print $1}') cat /etc/rancher/k3s/registries.yaml
mirrors:
  <http://itregistry.io|itregistry.io>:
    endpoint:
    - <http://it:5000>

~ via 🆂
❯

creamy-pencil-82913

03/23/2023, 8:27 PM

it is probably falling back from your manually-configured

<http://it:5000>

endpoint to the default endpoint for some reason

possibly because

it

can’t be resolved within the container either

miniature-honey-15162

03/23/2023, 8:28 PM

I might be naive here, but it looks like

<http://it:5000>

is able to be resolved within the container e.g.

~ via 🆂 took 15s
❯ docker exec -it (docker ps | rg k3s | awk '{print $1}') /bin/sh
/ # wget --spider <http://it:5000>
Connecting to it:5000 (192.168.80.2:5000)
remote file exists
/ #

I'll check containerd.log 🙂

Hrm, nothing too interesting to my eyes there 🤔

time="2023-03-23T20:10:13.197094043Z" level=info msg="PullImage \"<http://itregistry.io/fix-server:0.1.0\|itregistry.io/fix-server:0.1.0\>""
time="2023-03-23T20:10:13.206677710Z" level=info msg="trying next host" error="failed to do request: Head \"<https://itregistry.io/v2/fix-server/manifests/0.1.0>\": dial tcp: lookup <http://itregistry.io|itregistry.io> on 127.0.0.11:53: no such host" host=<http://itregistry.io|itregistry.io>
time="2023-03-23T20:10:13.208168293Z" level=error msg="PullImage \"<http://itregistry.io/fix-server:0.1.0\|itregistry.io/fix-server:0.1.0\>" failed" error="failed to pull and unpack image \"<http://itregistry.io/fix-server:0.1.0\|itregistry.io/fix-server:0.1.0\>": failed to resolve reference \"<http://itregistry.io/fix-server:0.1.0\|itregistry.io/fix-server:0.1.0\>": failed to do request: Head \"<https://itregistry.io/v2/fix-server/manifests/0.1.0>\": dial tcp: lookup <http://itregistry.io|itregistry.io> on 127.0.0.11:53: no such host"
time="2023-03-23T20:10:26.986564050Z" level=info msg="PullImage \"<http://itregistry.io/fix-server:0.1.0\|itregistry.io/fix-server:0.1.0\>""
time="2023-03-23T20:10:27.002840342Z" level=info msg="trying next host" error="failed to do request: Head \"<https://itregistry.io/v2/fix-server/manifests/0.1.0>\": dial tcp: lookup <http://itregistry.io|itregistry.io> on 127.0.0.11:53: no such host" host=<http://itregistry.io|itregistry.io>
time="2023-03-23T20:10:27.004939342Z" level=error msg="PullImage \"<http://itregistry.io/fix-server:0.1.0\|itregistry.io/fix-server:0.1.0\>" failed" error="failed to pull and unpack image \"<http://itregistry.io/fix-server:0.1.0\|itregistry.io/fix-server:0.1.0\>": failed to resolve reference \"<http://itregistry.io/fix-server:0.1.0\|itregistry.io/fix-server:0.1.0\>": failed to do request: Head \"<https://itregistry.io/v2/fix-server/manifests/0.1.0>\": dial tcp: lookup <http://itregistry.io|itregistry.io> on 127.0.0.11:53: no such host"
time="2023-03-23T20:10:49.995474588Z" level=info msg="PullImage \"<http://itregistry.io/fix-server:0.1.0\|itregistry.io/fix-server:0.1.0\>""
time="2023-03-23T20:10:50.018484297Z" level=info msg="trying next host" error="failed to do request: Head \"<https://itregistry.io/v2/fix-server/manifests/0.1.0>\": dial tcp: lookup <http://itregistry.io|itregistry.io> on 127.0.0.11:53: no such host" host=<http://itregistry.io|itregistry.io>

The only other "warnings" or errors I see are

time="2023-03-23T20:09:44.894387419Z" level=info msg="skip loading plugin \"io.containerd.snapshotter.v1.aufs\"..." error="aufs is not supported (modprobe aufs failed: exit status 1 \"modprobe: can't change directory to '5.10.104-linuxkit': No such file or directory\\n\"): skip plugin" type=io.containerd.snapshotter.v1
time="2023-03-23T20:09:44.895323294Z" level=info msg="skip loading plugin \"io.containerd.snapshotter.v1.btrfs\"..." error="path /var/lib/rancher/k3s/agent/containerd/io.containerd.snapshotter.v1.btrfs (ext4) must be a btrfs filesystem to be used with the btrfs snapshotter: skip plugin" type=io.containerd.snapshotter.v1
time="2023-03-23T20:09:44.895367544Z" level=warning msg="failed to load plugin io.containerd.snapshotter.v1.devmapper" error="devmapper not configured"
time="2023-03-23T20:09:44.904731919Z" level=info msg="skip loading plugin \"io.containerd.snapshotter.v1.zfs\"..." error="path /var/lib/rancher/k3s/agent/containerd/io.containerd.snapshotter.v1.zfs must be a zfs filesystem to be used with the zfs snapshotter: skip plugin" type=io.containerd.snapshotter.v1
time="2023-03-23T20:09:44.904792961Z" level=warning msg="could not use snapshotter devmapper in metadata plugin" error="devmapper not configured"

creamy-pencil-82913

03/23/2023, 8:45 PM

if you look at config.toml in the container, do you see the registry endpoint configuration in there?

miniature-honey-15162

03/23/2023, 8:47 PM

I don't think so

[plugins.opt]
  path = "/var/lib/rancher/k3s/agent/containerd"

[plugins.cri]
  stream_server_address = "127.0.0.1"
  stream_server_port = "10010"
  enable_selinux = false
  sandbox_image = "rancher/mirrored-pause:3.6"

[plugins.cri.containerd]
  snapshotter = "overlayfs"
  disable_snapshot_annotations = true


[plugins.cri.cni]
  bin_dir = "/bin"
  conf_dir = "/var/lib/rancher/k3s/agent/etc/cni/net.d"


[plugins.cri.containerd.runtimes.runc]
  runtime_type = "io.containerd.runc.v2"

^^ running

cat $(find . -name config.toml)

in the container

creamy-pencil-82913

03/23/2023, 9:13 PM

did you have the registries.yaml file there when k3s started, or did you put it there after? it needs to be there at startup.

miniature-honey-15162

03/23/2023, 9:24 PM

Arg, I am guessing this is most likely the issue. Let me double check, though.

creamy-pencil-82913

03/23/2023, 9:26 PM

I would recommend mounting it in as a docker volume (-v /foo/registries.yaml:/etc/rancher/k3s/registries.yaml) so that its not created too late, and persists acrosss restarts

if you just copy it in later after its already running, that’s too late

miniature-honey-15162

03/23/2023, 9:41 PM

Ah indeed, just mounting the registry file fixed the issue. Everything works as expected now! Thanks for the help! I'm not sure I would have figured that out on my own - or if I would have, it would have taken me a very long time so I very much appreciate your help 🙂

12 Views

#k3s Join Slack