https://rancher.com/ logo
Join Slack
Powered by
This message was deleted.
# k3s
a
This message was deleted.
✅ 1
c
you sure you have the vxlan ports open between nodes?
and vxlan traffic between nodes isn’t getting dropped for other reasons?
w
how can i check that? i have no firewall rules. the nodes are all vanilla ubuntu hosts
c
are they all on the same network?
w
all on the same router basically yeah
c
What kind of infrastructure are they running on?
vmware, physical, etc
w
physical computers
how do i debug the vxlan stuff? i had read about that on a few github issues. my network skills are not super strong
c
ufw/firewalld is disabled?
w
correct
c
what do you mean by “same router basically”
on flat networks and physical hosts, there aren’t usually any issues with vxlan, it should just work
w
theyre all connected to the same router / switch. think homelab
mmm
c
sometimes there are issues with checksum offload on vmware, or kernel modules missing on raspberry pi
but other than that its usually fine
w
mmm, one of the nodes is in a virtualbox VM with bridged network. again 
kubectl port-forward
works tho, it's just pod tries to access service no go
c
ah, so not all physical after all
w
well whats the difference between 
kubectl port-forward
and a pod accessing a service? dont they both do the same thing basically via kube proxy?
if i switch my flannel config to use wireguard, would that likely sidestep anything virtualbox is doing / vxlan filtering ?
c
no, not at all the same really
w
oh derp
c
port-forward connects to the pod via a reverse tunnel from the apiserver to the kubelet
pods connecting to services uses iptables rules from kube-proxy, and the CNI network overlay
completely different paths
w
oh huh
so port forward ... is encrypted? or its like some sort of network sent thru a TLS connection?
c
tunneled over http
w
mmm ok
c
from kubectl, to the apiserver, to the kubelet
CNI not involved at all
w
got it. pod connecting to service is not tunneled
unless using wireguard with flannel?
c
pod connecting to a service just uses the CNI overlay, which with vxlan is encapsulation with no encryption
just a very small extra header on the packet basically
w
ah ok that makes more sense
it's been a decade since i took ugrad networking ok thank you! so the issue im facing, trying to muck with iptables might not be a great idea for me? maybe virtualbox is doing something that's interfering? i've had a good experience with k3s in another set-up with just ubuntu hosts no virtualization at all, different cluster different times
if i run k8s with 
flannel-backend=wireguard
then that's tunneling everything thru encrypted wireguard effectively? maybe i should just try that. i dont need the wg tunnels today but some day i might, e.g. hybrid cloud set-up
c
yeah there’s a lot more overhead to it
you might just try disabling checksum offload on all the nodes, see if it’s that?
Copy code
ethtool --offload eth0 rx off tx off
on all the nodes
can also see if
Copy code
ethtool -K eth0 gso off
helps
you’re sure that the one on a VM is actually bridged, and not using NAT through the host’s IP?
w
yeah it's def not NAT. i can ssh to the VM and hit other services and everything from external machines to it's IP. cool thanks i have never used 
ethtool
let me try that out. i kinda wanted to give 
wg
a spin anyways too. thanks for helping me understand diff between proxy / tunnel and the overlay, i had mistaken that the proxy / tunnel always gets used
i guess kube proxy is mainly for load balanced stuff. service doesn't have LB. makes sense
c
kube-proxy handles pods accessing service clusterip addresses
w
welp i was using a release that was older than i thought 1.25.12. when i try v1.28.1+k3s1 now, it works (!). i did try the 
ethtool
stuff and re-installed 1.25.12 several times so im pretty sure the ethtool stuff didn't fix (tho i do see in github issues, several from years ago, it's a fix for many people). the issue was that pod ip was routable but not service ip. seems several people have seen this, e.g. https://github.com/k3s-io/k3s/issues/1638#issuecomment-615088410 any idea on what in ... flannel? might have changed in k8s 1.25.12 -> 1.28.1 to mitigate this particular issue? i've got ubuntu 20.04 and i never touched the iptables stuff, which seems to be related. it sounds like flannel had some mac address issues that got merged but that was i think a while ago? will keep testing a bit welp so i was using a release that was fairly old i guess, 
v1.28.1+k3s1
3 Views
Open in Slack
PreviousNext
Powered by