https://rancher.com/ logo
Title
f

flaky-branch-26433

03/24/2023, 7:52 PM
Hi guys, any idea where I can look for DNS issues? Deployd awx but can't resolve our internal gitlab. I can resolve external addresses and also internal/local IPs - but not internal names. (for example our domain is
domain-xx.local
and
10.2.17.21
is the vm (
server1
running k3s.))
kubectl run -it --rm --restart=Never busybox --image=busybox:1.28 -- nslookup 10.2.17.21
Server:    192.168.246.10
Address 1: 192.168.246.10 kube-dns.kube-system.svc.domain-xx.local

Name:      10.2.17.21
Address 1: 10.2.17.21 server1.domain-xx.local
Lookup with IP is working Now I'm trying to resolve the same system via name:
kubectl run -it --rm --restart=Never busybox --image=busybox:1.28 -- nslookup server1.domain-xx.local
Server:    192.168.246.10
Address 1: 192.168.246.10 kube-dns.kube-system.svc.domain-xx.local

nslookup: can't resolve 'server1.domain-xx.local'
Lookup with name is not working. I'm new to k3s, so bear with me. It's a single k3s node, fresh installed. The host can resolve in both ways. When I'm running:
kubectl run -i --restart=Never --rm test-${RANDOM} --image=ubuntu --overrides='{"kind":"Pod", "apiVersion":"v1", "spec": {"dnsPolicy":"Default"}}' -- sh -c 'cat /etc/resolv.conf'
to check for the /etc/resolv.conf it shows me the correct entries for the search domain and our dns server. kube-dns (coredns) is runnig and dns server is
192.168.246.10
pod:
coredns-597584b69b-9hspl   1/1     Running   1 (65m ago)   87m

svc:
kube-dns   ClusterIP   192.168.246.10   <none>        53/UDP,53/TCP,9153/TCP   87m
p

polite-piano-74233

03/24/2023, 8:04 PM
wouldnt it be
server1.svc.namespace.domain-xx.local
?
or you mean internal as in internal network but not internal k8s
k3s*
f

flaky-branch-26433

03/24/2023, 8:06 PM
Yes, I'm trying to reach our gitlab server, which is another vm (not running k3s). But in this example I'm trying to make a lookup on the k3s host (ip and name). But I can't lookup any other vm with the name.
when I try to lookup for our gitlab (code.domain-xx.local) not working. But the same lookup with the gitlabs IP is working fine and it shows me:
kubectl run -it --rm --restart=Never busybox --image=busybox:1.28 -- nslookup 10.2.17.20
Server:    192.168.246.10
Address 1: 192.168.246.10 kube-dns.kube-system.svc.domain-xx.local

Name:      10.2.17.20
Address 1: 10.2.17.20 code.domain-xx.local
as far as I understood it the lookup should be forwarded to our internal dns server, which should provide the ip/name. (which it's doing outside of k3s). But why ip lookup is working and name lookup not? search domains are correct (in /etc/resolv.conf). Is there a way to check coredns' search domains? Maybe there is something wrong here, as IP lookup works fine. (which is also delivered by upstream dns)
c

creamy-pencil-82913

03/24/2023, 8:15 PM
are you using the same domain for your Kubernetes cluster as your actual domain?
f

flaky-branch-26433

03/24/2023, 8:16 PM
yes, all in one domain
c

creamy-pencil-82913

03/24/2023, 8:17 PM
I wouldn’t do that. coredns assumes it is authoritative for the cluster domain. It isn’t going to forward lookups for the cluster domain out to your primary dns server.
tl;dr don’t set the cluster domain to the same thing as your real domain, it will mask the records in your real domain
that’s why it’s called the cluster domain. It’s just for use in the cluster.
f

flaky-branch-26433

03/24/2023, 8:19 PM
ah wait, what exactly you mean with cluster domain? My 'real' domain is domain-xx.local. This is the domain where our gitlab, dns server and also the k3s host is. I did not specify any special domain in k3s installation.
Where I can check the 'cluster domain' ?
c

creamy-pencil-82913

03/24/2023, 8:19 PM
ok. it can be hard to tell when people redact things.
the default cluster domain is cluster.local
and you showed
kube-dns.kube-system.svc.domain-xx.local
which was indicating that you’d replaced the cluster domain with domain-xx.local because that is NOT the default cluster domain.
so is that what you actually were getting, or was that some over-exuberant redacting of your actual data?
f

flaky-branch-26433

03/24/2023, 8:22 PM
I replaced our real domain with domain-xx.local .
c

creamy-pencil-82913

03/24/2023, 8:23 PM
That suggests that you’re using your real domain as the cluster domain then
f

flaky-branch-26433

03/24/2023, 8:23 PM
ah, got it now. Sorry.
c

creamy-pencil-82913

03/24/2023, 8:23 PM
which you shouldn’t do
f

flaky-branch-26433

03/24/2023, 8:23 PM
yes, i set up the cluster with --cluster-domain domain-xx.local
c

creamy-pencil-82913

03/24/2023, 8:23 PM
yep. don’t do that.
f

flaky-branch-26433

03/24/2023, 8:24 PM
got it, will fix it and try again. Thank you very much!
c

creamy-pencil-82913

03/24/2023, 8:24 PM
so
did not specify any special domain in k3s installation.
was not accurate
f

flaky-branch-26433

03/24/2023, 8:24 PM
any ressources for best practices where I can find such info?
c

creamy-pencil-82913

03/24/2023, 8:24 PM
you should uninstall and reinstall, the cluster domain can’t be changed after the cluster is brought up.
f

flaky-branch-26433

03/24/2023, 8:24 PM
Yes, it was not. Sorry
No problem, will redo it. Sorry again.
c

creamy-pencil-82913

03/24/2023, 8:24 PM
np!
🙌 1
I don’t know that there’s any specific document in either our docs or the upstream Kubernetes docs that says not to set your cluster domain to the same thing as your real domain, I guess the expectation is that people would know not to have conflicting DNS zones.
f

flaky-branch-26433

03/24/2023, 8:28 PM
Yeah, I thougth setting up the same cluster-domain would help. Did not thought about DNS Zones at the moment. Learned my lesson now. 🙂 Will change it to something different.
c

creamy-pencil-82913

03/24/2023, 8:29 PM
it looks like you’ve also customized the cluster and service CIDRs, those don’t overlap with your actual network ranges do they?
f

flaky-branch-26433

03/24/2023, 8:30 PM
I have to do it. The default ranges overlapping some of our used network ranges.
c

creamy-pencil-82913

03/24/2023, 8:36 PM
ok. just make sure that the ranges you picked don’t overlap with any actual ranges, and that the cluster-cidr is large enough to hold as many nodes as you plan on adding.
👍 1