https://rancher.com/ logo
Title
p

polite-piano-74233

03/16/2023, 2:30 AM
anyone else run into issues with ubuntu 22.04 and coredns dns queries from pods not working? They work fine in 20.04 but on 22.04 nodes the dns entries cant resolve and the pods crash out.
r

rough-farmer-49135

03/16/2023, 1:01 PM
I've seen it sporadically, but not predictably on 20.04 too. It seems like there's some chance of it hitting any system that runs its own DNS resolver so has 127.0.0.x in /etc/resolv.conf We fixed by editing coredns ConfigMap to include the DNS IP it should be pointed to, then things were fine.
p

polite-piano-74233

03/16/2023, 2:07 PM
hmm, think im getting closer, in my case it looks to be related to flannel etc not properly injecting the ip table routes
r

rough-farmer-49135

03/16/2023, 2:10 PM
That's certainly a worse problem than mine. I can certainly see how coredns would be one of the early canaries for that one, though. I don't recall, but does flannel have pods to poke & check the logs?
And is it missing from the local routing table on all your nodes, or just some?
p

polite-piano-74233

03/16/2023, 2:14 PM
yea the flannel pod pokes at the iptables pretty consistantly, but i dont see any of the entries in its logs its trying to add getting through to the actual iptables routes
for me this is only happening on ubuntu 22.04 nodes, older 20.04 are fine
r

rough-farmer-49135

03/16/2023, 2:21 PM
I don't know if iptables is involved with the routing, I thought that showed up with the
route
command instead.
I assume ufw is off just to cover bases?
p

polite-piano-74233

03/16/2023, 2:21 PM
sorry ip route, your right
r

rough-farmer-49135

03/16/2023, 2:24 PM
Not sure what may have changed with networking and/or network permissions, we ran into some other problems with 22.04 before we even got to Kubernetes and backed off to 20.04 to eliminate the possibility (and our production environment is RHEL clones, so Ubuntu was more just for dev systems and ok to ignore).
p

polite-piano-74233

03/16/2023, 2:26 PM
this may have fixed it, still tshooting
update-alternatives --set iptables /usr/sbin/iptables-legacy
from: https://github.com/k3s-io/k3s/issues/4188 which our very own @creamy-pencil-82913 helped with hah
yup, adding iptables legacy and a reboot, now everything works 100% fine on ubuntu 22.04 (btw this was an in-place upgrade aka do-release-upgrade)
r

rough-farmer-49135

03/16/2023, 2:41 PM
Nice to hear, I'll keep it in mind if I hear similar.
­čÖî 1