https://rancher.com/ logo
#rke2
Title
# rke2
p

prehistoric-solstice-99854

09/15/2022, 8:58 PM
Afternoon all. I’m having DNS issues with a recently installed RKE2 cluster on ESX VMs running Oracle Linux 8. I started with RKE2 1.21.14 but then upgraded to 1.22.13 after I saw a comment in this channel about networking issues on 1.21. The cluster has been happy and no errors when I look at the pods through
kubectl
. However, when I tried to access Rancher, after a successful install, the site never fully loads. I looked through logs and have determined that DNS isn’t working and that is causing the problem. I got a shell inside a container and confirmed that I can ping an IP but not a domain name. I’ve disabled firewalld, temporarily disabled SELinux, and I updated NetworkManager to ignore CNI traffic on all RKE2 nodes. The 3 management nodes and 3 worker nodes have no DNS issues, just the pods that do. I’m not sure what to try next. It appears the issue is communication between pods. Could anyone point me in the right direction on this? I’ve looked for generic coredns troubleshooting and nothing has helped me find the problem yet.
c

creamy-pencil-82913

09/15/2022, 9:05 PM
Where is DNS failing? Can the pods reach the in-cluster DNS service (coredns)? Can coredns reach its upstream DNS servers?
p

prehistoric-solstice-99854

09/15/2022, 9:07 PM
I enabled logs in the configmap for coredns and then ran
kubectl logs -f …
and tried to do some DNS requests and I don’t see any traffic. At this point, it doesn’t even look like the request hits the nodes.
c

creamy-pencil-82913

09/15/2022, 9:08 PM
can you resolve kubernetes records, such as kubernetes.default.svc? that one should always exist.
Or is it just external records that you can’t hit?
p

prehistoric-solstice-99854

09/15/2022, 9:08 PM
host <http://updates.suse.com|updates.suse.com>
fails but
host <http://updates.suse.com|updates.suse.com> 1.1.1.1
works.
So try
nslookup kubernetes.default.svc
?
c

creamy-pencil-82913

09/15/2022, 9:09 PM
I like dig better, but sure
or
kubernetes.default.svc.cluster.local.
if you want to try a FQDN, without depending on search behavior
p

prehistoric-solstice-99854

09/15/2022, 9:10 PM
Copy code
# dig kubernetes.default.svc

; <<>> DiG 9.9.5-9+deb8u19-Debian <<>> kubernetes.default.svc
;; global options: +cmd
;; connection timed out; no servers could be reached
c

creamy-pencil-82913

09/15/2022, 9:11 PM
OK, so you can’t reach the coredns service. Do you have the vxlan ports open between nodes?
most common cause of that is something is causing inter-node cluster traffic to be dropped
p

prehistoric-solstice-99854

09/15/2022, 9:13 PM
The firewall on the box is off, and there shouldn’t be any restrictions between VMs.
Does xvlan require jumbo frames on the physical network, ie. are there any MTU requirements for physical switches?
It seems we have a Dell switch that is giving us issues with MTUs over 1500.
c

creamy-pencil-82913

09/15/2022, 9:45 PM
the default MTU for the vxlan interfaces under the Canal CNI is 1450, which should result in an on-the-wire frame size of 1500 when it leaves the node. If you’re running in VMs its possible that vmware is adding further encapsulation on to your traffic and as a result the traffic is exceeding the MTU or is getting fragmented and dropped.
p

prehistoric-solstice-99854

09/15/2022, 9:47 PM
Possible. We’re looking into that issue. Is there anything else I can test/tweak incase it isn’t the MTU issue?
Or can I modify that config in my cluster to push the MTU down to 1400?
c

creamy-pencil-82913

09/15/2022, 9:49 PM
yeah, you can try using a HelmChartConfig to modify the chart values. I’m not sure what will happen if you try to change it on a running cluster; you might need to do it from the get-go or at least kill and recreate all the pods to get it set properly.
p

prehistoric-solstice-99854

09/15/2022, 9:49 PM
Okay, anything else I can try?
c

creamy-pencil-82913

09/15/2022, 9:50 PM
https://docs.rke2.io/helm/#customizing-packaged-components-with-helmchartconfig in this case you’d want to set the rke2-canal chart’s
calico.vethuMTU
value
other than raising the MTU on the switch? no
thats where I would start if you are seeing traffic leave one node but not show up on the other
p

prehistoric-solstice-99854

09/15/2022, 9:52 PM
Okay. Thanks!
l

little-actor-95014

09/16/2022, 3:22 PM
if MTU doesn't help, might try disabling tx checksum offloading.
sudo ethtool -K flannel.1 tx-checksum-ip-generic off
I've personally only ever seen this issue with Ubuntu, but it's a stab in the dark if nothing else helps
p

prehistoric-solstice-99854

09/16/2022, 3:28 PM
Thanks for that Zach. At this point I’m doing a lot of blind stabbing. 😉
@little-actor-95014 That fixed my problem!!!!
Now I need to understand what I just did and will this change persist across reboots.
l

little-actor-95014

09/16/2022, 3:39 PM
From my understanding of the issues, there's a kernel bug that causes checksumming for UDP traffic of VXLAN to be wrong and the traffic dropped. I've had mixed results in reproduction of the issue. RKE2 Issue: https://github.com/rancher/rke2/issues/1541 Flannel Issue: https://github.com/flannel-io/flannel/issues/1279
The weirdest part for me has been that I've reinstalled things from scratch to repro and haven't been able to. And that most issues I find that reference the bug and potential workarounds say it was fixed in K8s/RKE2 a year-ish ago
p

prehistoric-solstice-99854

09/16/2022, 3:44 PM
Huh. Maybe I’m just using an older kernel where it’s still broken?
These server nodes are running 4.18 kernel.
l

little-actor-95014

09/16/2022, 3:47 PM
It is possible Oracle hasn't backported the patch, it seems it was patched in 5.7, but the bug introduced back in 2006 🙂
p

prehistoric-solstice-99854

09/16/2022, 3:48 PM
Possible. Thank you so much for that help. That just saved me so much time in troubleshooting.
188 Views