Afternoon all. I’m having DNS issues with a recently installed RKE2 cluster on ESX VMs running Oracle Linux 8. I started with RKE2 1.21.14 but then upgraded to 1.22.13 after I saw a comment in this channel about networking issues on 1.21. The cluster has been happy and no errors when I look at the pods through

kubectl

. However, when I tried to access Rancher, after a successful install, the site never fully loads. I looked through logs and have determined that DNS isn’t working and that is causing the problem. I got a shell inside a container and confirmed that I can ping an IP but not a domain name. I’ve disabled firewalld, temporarily disabled SELinux, and I updated NetworkManager to ignore CNI traffic on all RKE2 nodes. The 3 management nodes and 3 worker nodes have no DNS issues, just the pods that do. I’m not sure what to try next. It appears the issue is communication between pods. Could anyone point me in the right direction on this? I’ve looked for generic coredns troubleshooting and nothing has helped me find the problem yet.

Where is DNS failing? Can the pods reach the in-cluster DNS service (coredns)? Can coredns reach its upstream DNS servers?

I enabled logs in the configmap for coredns and then ran

kubectl logs -f …

and tried to do some DNS requests and I don’t see any traffic. At this point, it doesn’t even look like the request hits the nodes.

can you resolve kubernetes records, such as kubernetes.default.svc? that one should always exist.

host <http://updates.suse.com|updates.suse.com>

fails but

host <http://updates.suse.com|updates.suse.com> 1.1.1.1

works.

I like dig better, but sure

OK, so you can’t reach the coredns service. Do you have the vxlan ports open between nodes?

The firewall on the box is off, and there shouldn’t be any restrictions between VMs.

the default MTU for the vxlan interfaces under the Canal CNI is 1450, which should result in an on-the-wire frame size of 1500 when it leaves the node. If you’re running in VMs its possible that vmware is adding further encapsulation on to your traffic and as a result the traffic is exceeding the MTU or is getting fragmented and dropped.

Possible. We’re looking into that issue. Is there anything else I can test/tweak incase it isn’t the MTU issue?

yeah, you can try using a HelmChartConfig to modify the chart values. I’m not sure what will happen if you try to change it on a running cluster; you might need to do it from the get-go or at least kill and recreate all the pods to get it set properly.

Okay, anything else I can try?

https://docs.rke2.io/helm/#customizing-packaged-components-with-helmchartconfig in this case you’d want to set the rke2-canal chart’s

calico.vethuMTU

value

Okay. Thanks!

if MTU doesn't help, might try disabling tx checksum offloading.

sudo ethtool -K flannel.1 tx-checksum-ip-generic off

I've personally only ever seen this issue with Ubuntu, but it's a stab in the dark if nothing else helps

Thanks for that Zach. At this point I’m doing a lot of blind stabbing. 😉

From my understanding of the issues, there's a kernel bug that causes checksumming for UDP traffic of VXLAN to be wrong and the traffic dropped. I've had mixed results in reproduction of the issue. RKE2 Issue: https://github.com/rancher/rke2/issues/1541 Flannel Issue: https://github.com/flannel-io/flannel/issues/1279

Huh. Maybe I’m just using an older kernel where it’s still broken?

It is possible Oracle hasn't backported the patch, it seems it was patched in 5.7, but the bug introduced back in 2006 🙂

Possible. Thank you so much for that help. That just saved me so much time in troubleshooting.