https://rancher.com/ logo
Title
b

bright-businessperson-46654

05/10/2022, 6:01 AM
Hi everyone, I have faced a couple of problems on my k3s cluster contains (3 master nodes + 5 worker nodes). One problem is in name resolutions of container runtime. I have internal DNS server and an internal image registry. All cluster nodes dns configs are set to my internal dns server. On the node I can resolve private image registry dns name with nslookup and everything is ok, BUT when I deploy a pod with an image on private image registry I got this error : "Failed to pull image "reg.mydomain.com/tools/cicd-helper": rpc error: code = Unknown desc = failed to pull and unpack image "reg.mydomain.com/tools/cicd-helper:latest": failed to resolve reference "reg.mydomain.com/tools/cicd-helper:latest": failed to do request: Head "https://reg.mydomain.com/v2/tools/cicd-helper/manifests/latest": dial tcp: lookup reg.mydomain.com: no such host" Same error on the worker node with this command: sudo crictl pull --creds user:pass1 reg.mydomain.com/tools/cicd-helper:latest Anyone can help me with this issue?
c

creamy-pencil-82913

05/10/2022, 7:19 AM
Does reg.mydomain.com actually exist on a DNS server somewhere? If it only exist in private DNS, can you confirm that coredns is using that private DNS server as the upstream?
b

bright-businessperson-46654

05/10/2022, 7:52 AM
internal registry dns record exists on my internal dns server with private ip, all k3s nodes have access to it and can reach it via nslookup and ping and curl. I tested access and reaching to registry in an existing pod(nslookup) so coredns seems to be OK. even the host os can find the registry itself, it seems to be a container runtime problem that does not use the host os dns config 😕 I don't know what else to do
c

creamy-pencil-82913

05/10/2022, 8:08 AM
Theres no reason why containerd shouldn't honor the host's resolv.conf. can you use crictl to pull an image from your private registry?
b

bright-businessperson-46654

05/10/2022, 9:16 AM
sudo crictl pull --creds user:pass1 reg.mydomain.com/tools/cicd-helper:latest and get same error. Failed to pull image "reg.mydomain.com/tools/cicd-helper": rpc error: code = Unknown desc = failed to pull and unpack image "reg.mydomain.com/tools/cicd-helper:latest": failed to resolve reference "reg.mydomain.com/tools/cicd-helper:latest": failed to do request: Head "https://reg.mydomain.com/v2/tools/cicd-helper/manifests/latest": dial tcp: lookup reg.mydomain.com: no such host
c

creamy-pencil-82913

05/10/2022, 9:20 AM
And you can resolve that same hostname with dig or nslookup on that node?
b

bright-businessperson-46654

05/10/2022, 9:20 AM
yes
as a temp solution I made my private registry accessible from outside(internet) with public IP. so k3s can find it from internet and pulls image 😕
c

creamy-pencil-82913

05/10/2022, 9:27 AM
Hmm I've never seen that. I suspect something is unusual about your node or DNS configuration
r

rapid-helmet-86074

05/18/2022, 1:59 PM
Some tools only notice the first three nameservers in resolv.conf, and Kubernetes will add coredns pod at the top I believe, so if the one you're hoping to get hit is the third or farther then some tools may notice and others won't.
b

bright-businessperson-46654

05/19/2022, 5:30 AM
🤔 Can you explain more?
c

creamy-pencil-82913

05/19/2022, 5:35 AM
There's lots of info about if you search for "Kubernetes DNS ndots" https://stackoverflow.com/questions/65181012/does-alpine-have-known-dns-issue-within-kubernetes
b

bright-businessperson-46654

05/19/2022, 7:08 AM
I think it's not related to core dns because inside a running pod everything is fine, crictl on the k3s node can't resolve the internal registry name
c

creamy-pencil-82913

05/19/2022, 7:50 AM
What does your resolv.conf look like?
b

bright-businessperson-46654

05/19/2022, 7:59 AM
~$ cat /etc/resolv.conf # This file is managed by man:systemd-resolved(8). Do not edit. # # This is a dynamic resolv.conf file for connecting local clients to the # internal DNS stub resolver of systemd-resolved. This file lists all # configured search domains. # # Run "resolvectl status" to see details about the uplink DNS servers # currently in use. # # Third party programs must not access this file directly, but only through the # symlink at /etc/resolv.conf. To manage man:resolv.conf(5) in a different way, # replace this symlink by a static file or a different symlink. # # See man:systemd-resolved.service(8) for details about the supported modes of # operation for /etc/resolv.conf. nameserver 127.0.0.53 options edns0 trust-ad
k3s -v k3s version v1.21.5+k3s2 (724ef700) go version go1.16.8
~$ sudo crictl -v crictl version v1.21.0-k3s1
r

rapid-helmet-86074

05/19/2022, 2:21 PM
127.0.0.53 is a localhost IP, so that'd be in the container you got it from (or host if you ran it from the host), so to see DNS config at that point you'd have to check your DNS config (if you don't know what's running DNS on the host
netstat -anp | grep 53
and look for what's listening on UDP & TCP port 53 and maybe
ps auwx | grep ${PID_FROM_NETSTAT}
to see if the command line tells you config location and if not you can start poking around man pages or systemctl and see if you can find it there. Not sure if that'll help or not, but that's all the other info I can think related at the moment.
b

bright-businessperson-46654

07/22/2022, 8:35 AM
@creamy-pencil-82913 @rapid-helmet-86074 Thanks, I fixed the issue
👍 1