https://rancher.com/ logo
Join Slack
Powered by
This message was deleted.
# rke2
a
This message was deleted.
c
Where is DNS failing? Can the pods reach the in-cluster DNS service (coredns)? Can coredns reach its upstream DNS servers?
p
I enabled logs in the configmap for coredns and then ran 
kubectl logs -f …
and tried to do some DNS requests and I don’t see any traffic. At this point, it doesn’t even look like the request hits the nodes.
c
can you resolve kubernetes records, such as kubernetes.default.svc? that one should always exist.
Or is it just external records that you can’t hit?
p
host <http://updates.suse.com|updates.suse.com>
fails but 
host <http://updates.suse.com|updates.suse.com> 1.1.1.1
works.
So try 
nslookup kubernetes.default.svc
?
c
I like dig better, but sure
or 
kubernetes.default.svc.cluster.local.
if you want to try a FQDN, without depending on search behavior
p
Copy code
# dig kubernetes.default.svc

; <<>> DiG 9.9.5-9+deb8u19-Debian <<>> kubernetes.default.svc
;; global options: +cmd
;; connection timed out; no servers could be reached
c
OK, so you can’t reach the coredns service. Do you have the vxlan ports open between nodes?
most common cause of that is something is causing inter-node cluster traffic to be dropped
p
The firewall on the box is off, and there shouldn’t be any restrictions between VMs.
Does xvlan require jumbo frames on the physical network, ie. are there any MTU requirements for physical switches?
It seems we have a Dell switch that is giving us issues with MTUs over 1500.
c
the default MTU for the vxlan interfaces under the Canal CNI is 1450, which should result in an on-the-wire frame size of 1500 when it leaves the node. If you’re running in VMs its possible that vmware is adding further encapsulation on to your traffic and as a result the traffic is exceeding the MTU or is getting fragmented and dropped.
p
Possible. We’re looking into that issue. Is there anything else I can test/tweak incase it isn’t the MTU issue?
Or can I modify that config in my cluster to push the MTU down to 1400?
c
yeah, you can try using a HelmChartConfig to modify the chart values. I’m not sure what will happen if you try to change it on a running cluster; you might need to do it from the get-go or at least kill and recreate all the pods to get it set properly.
p
Okay, anything else I can try?
c
https://docs.rke2.io/helm/#customizing-packaged-components-with-helmchartconfig in this case you’d want to set the rke2-canal chart’s 
calico.vethuMTU
value
other than raising the MTU on the switch? no
thats where I would start if you are seeing traffic leave one node but not show up on the other
p
Okay. Thanks!
l
if MTU doesn't help, might try disabling tx checksum offloading. 
sudo ethtool -K flannel.1 tx-checksum-ip-generic off
I've personally only ever seen this issue with Ubuntu, but it's a stab in the dark if nothing else helps
p
Thanks for that Zach. At this point I’m doing a lot of blind stabbing. 😉
@little-actor-95014 That fixed my problem!!!!
Now I need to understand what I just did and will this change persist across reboots.
l
From my understanding of the issues, there's a kernel bug that causes checksumming for UDP traffic of VXLAN to be wrong and the traffic dropped. I've had mixed results in reproduction of the issue. RKE2 Issue: https://github.com/rancher/rke2/issues/1541 Flannel Issue: https://github.com/flannel-io/flannel/issues/1279
The weirdest part for me has been that I've reinstalled things from scratch to repro and haven't been able to. And that most issues I find that reference the bug and potential workarounds say it was fixed in K8s/RKE2 a year-ish ago
p
Huh. Maybe I’m just using an older kernel where it’s still broken?
These server nodes are running 4.18 kernel.
l
It is possible Oracle hasn't backported the patch, it seems it was patched in 5.7, but the bug introduced back in 2006 🙂
p
Possible. Thank you so much for that help. That just saved me so much time in troubleshooting.
303 Views
Open in Slack
PreviousNext
Powered by