https://rancher.com/ logo
Title
n

nutritious-oxygen-89191

04/20/2023, 2:05 PM
If I start an agent with the
--resolv-conf
flag, how can I verify, that this configuration is actually used?
r

rough-farmer-49135

04/20/2023, 2:12 PM
I'd probably try
kubectl exec --stdin --tty ${POD} -- /bin/bash
for a pod running on the agent that has bash and then poke around looking at /etc/resolv.conf & running nslookup, dig, ping, or whatever else was a good test for my particular concern.
n

nutritious-oxygen-89191

04/20/2023, 2:42 PM
for your command i get
Unable to connect to the server: dial tcp: lookup tdin on 127.0.0.53:53: server misbehaving
but
kubectl exec -it <pod> -- bash
works. However,
/etc/resolv.conf
does not include the
nameservers
I listed in the file supplied to
--resolv.conf
. Also
nslookup <http://google.com|google.com>
fails with
;; connection timed out; no servers could be reached
r

rough-farmer-49135

04/20/2023, 2:56 PM
My guess is you're using a distro that runs local DNS and puts 127.0.0.53 in resolv.conf (newer Ubuntu does that). The problem is that localhost on a pod is the pod itself, not the pod's host. There are multiple ways to get around it, but the one I saw was editing the configMap for coredns (it's in the kube-system namespace). Look for a line saying 'forward . /etc/resolv.conf' and add your preferred DNS IP after it, so 'forward . /etc/resolv.conf 8.8.8.8' if you wanted you use one of Google's DNS as an example.
c

creamy-pencil-82913

04/20/2023, 3:36 PM
dial tcp: lookup tdin on 127.0.0.53:53: server misbehaving
you did
-s tdin
didn’t you
the easiest way to check would be to look at the k3s logs at startup and check for the presence of a
--resolv-conf
flag in the kubelet args
n

nutritious-oxygen-89191

04/20/2023, 4:15 PM
I had
-stdin
instead of
--stdin
🤦‍♂️. That error is gone and I can use the command mentioned by @rough-farmer-49135. The error, however is the same
nslookup <http://google.com|google.com>
fails. My flag with my modified
resolv.conf
is
--resolv-conf=/etc/k3s-resolv.conf
and this also comes up in the k3s logs ( the
k3s-agent.service
also mentions it).
c

creamy-pencil-82913

04/20/2023, 4:49 PM
what are you trying to accomplish by changing the resolv.conf? Are you trying to change where coredns goes for upstream lookups, or change the dns search order or what? Most pods are going to be running with cluster DNS so they will be resolving things via the in-cluster dns service and coredns.
r

rough-farmer-49135

04/20/2023, 4:52 PM
I'm guessing he had similar problems we had with running k3d on some newer Ubuntu that pointed to 127.0.0.53 in resolv.conf . For some of those installs, but not all oddly enough, that ended up blocking pods from being able to resolve host names external to the cluster on our internal network and we had some places where it's needed. The solution we found for those was what I wrote above with the coredns ConfigMap, just passing in our internal network DNS instead of 8.8.8.8. I'm sure there could be other solutions, but that worked and was easy to do.
n

nutritious-oxygen-89191

04/20/2023, 5:31 PM
Basically I am trying to connect my pod to the internet 🙂 . My OS on all 4 nodes is Ubuntu 22.04 (k3s 1.25.6) and a pod on one specific node cannot reach pods on other nodes and vice versa. I read that setting the nameserver in a custom resolv.conf could solve it. I also tried https://ranchermanager.docs.rancher.com/troubleshooting/other-troubleshooting-tips/networking which I think is similar to what I am doing
c

creamy-pencil-82913

04/20/2023, 5:34 PM
a pod on one specific node cannot reach pods on other nodes and vice versa.
That indicates a problem with your CNI traffic, not a problem with DNS
Are the nodes all on the same network? Do you have the correct ports open between nodes for whatever flannel backend you’re using?
n

nutritious-oxygen-89191

04/21/2023, 6:24 AM
Not the same network, but ports should all be open. Are you aware of any test script or command that I could use to test/debug the CNI? ( I am using the default k3s network backend).