Hi everyone, I have faced a couple of problems on my k3s cluster contains (3 master nodes + 5 worker nodes). One problem is in name resolutions of container runtime. I have internal DNS server and an internal image registry. All cluster nodes dns configs are set to my internal dns server. On the node I can resolve private image registry dns name with nslookup and everything is ok, BUT when I deploy a pod with an image on private image registry I got this error : "Failed to pull image "reg.mydomain.com/tools/cicd-helper": rpc error: code = Unknown desc = failed to pull and unpack image "reg.mydomain.com/tools/cicd-helper:latest": failed to resolve reference "reg.mydomain.com/tools/cicd-helper:latest": failed to do request: Head "https://reg.mydomain.com/v2/tools/cicd-helper/manifests/latest": dial tcp: lookup reg.mydomain.com: no such host" Same error on the worker node with this command: sudo crictl pull --creds user:pass1 reg.mydomain.com/tools/cicd-helper:latest Anyone can help me with this issue?

Does reg.mydomain.com actually exist on a DNS server somewhere? If it only exist in private DNS, can you confirm that coredns is using that private DNS server as the upstream?

internal registry dns record exists on my internal dns server with private ip, all k3s nodes have access to it and can reach it via nslookup and ping and curl. I tested access and reaching to registry in an existing pod(nslookup) so coredns seems to be OK. even the host os can find the registry itself, it seems to be a container runtime problem that does not use the host os dns config 😕 I don't know what else to do

Theres no reason why containerd shouldn't honor the host's resolv.conf. can you use crictl to pull an image from your private registry?

sudo crictl pull --creds user:pass1 reg.mydomain.com/tools/cicd-helper:latest and get same error. Failed to pull image "reg.mydomain.com/tools/cicd-helper": rpc error: code = Unknown desc = failed to pull and unpack image "reg.mydomain.com/tools/cicd-helper:latest": failed to resolve reference "reg.mydomain.com/tools/cicd-helper:latest": failed to do request: Head "https://reg.mydomain.com/v2/tools/cicd-helper/manifests/latest": dial tcp: lookup reg.mydomain.com: no such host

And you can resolve that same hostname with dig or nslookup on that node?

yes

Hmm I've never seen that. I suspect something is unusual about your node or DNS configuration

Some tools only notice the first three nameservers in resolv.conf, and Kubernetes will add coredns pod at the top I believe, so if the one you're hoping to get hit is the third or farther then some tools may notice and others won't.

🤔 Can you explain more?

There's lots of info about if you search for "Kubernetes DNS ndots" https://stackoverflow.com/questions/65181012/does-alpine-have-known-dns-issue-within-kubernetes

I think it's not related to core dns because inside a running pod everything is fine, crictl on the k3s node can't resolve the internal registry name

What does your resolv.conf look like?

~$ cat /etc/resolv.conf # This file is managed by man:systemd-resolved(8). Do not edit. # # This is a dynamic resolv.conf file for connecting local clients to the # internal DNS stub resolver of systemd-resolved. This file lists all # configured search domains. # # Run "resolvectl status" to see details about the uplink DNS servers # currently in use. # # Third party programs must not access this file directly, but only through the # symlink at /etc/resolv.conf. To manage man:resolv.conf(5) in a different way, # replace this symlink by a static file or a different symlink. # # See man:systemd-resolved.service(8) for details about the supported modes of # operation for /etc/resolv.conf. nameserver 127.0.0.53 options edns0 trust-ad

127.0.0.53 is a localhost IP, so that'd be in the container you got it from (or host if you ran it from the host), so to see DNS config at that point you'd have to check your DNS config (if you don't know what's running DNS on the host

netstat -anp | grep 53

and look for what's listening on UDP & TCP port 53 and maybe

ps auwx | grep ${PID_FROM_NETSTAT}

to see if the command line tells you config location and if not you can start poking around man pages or systemctl and see if you can find it there. Not sure if that'll help or not, but that's all the other info I can think related at the moment.

@creamy-pencil-82913 @rapid-helmet-86074 Thanks, I fixed the issue