https://rancher.com/ logo
Powered by
#k3s
Title
w

worried-receptionist-18982

09/18/2023, 11:09 PM
dear k3s users, i'm trying to debug some service connection issues. when i use the kubectl proxy 
kubectl port-forward
i can get any agent node to connect to a k3s service just fine. but if i start a pod, the pod can't seem to access the service. is there a good way to debug this issue? im using default flannel config, maybe it's an iptables issue?
✅ 1
c

creamy-pencil-82913

09/18/2023, 11:15 PM
you sure you have the vxlan ports open between nodes?
and vxlan traffic between nodes isn’t getting dropped for other reasons?
w

worried-receptionist-18982

09/18/2023, 11:16 PM
how can i check that? i have no firewall rules. the nodes are all vanilla ubuntu hosts
c

creamy-pencil-82913

09/18/2023, 11:16 PM
are they all on the same network?
w

worried-receptionist-18982

09/18/2023, 11:16 PM
all on the same router basically yeah
c

creamy-pencil-82913

09/18/2023, 11:16 PM
What kind of infrastructure are they running on?
vmware, physical, etc
w

worried-receptionist-18982

09/18/2023, 11:16 PM
physical computers
how do i debug the vxlan stuff? i had read about that on a few github issues. my network skills are not super strong
c

creamy-pencil-82913

09/18/2023, 11:17 PM
ufw/firewalld is disabled?
w

worried-receptionist-18982

09/18/2023, 11:17 PM
correct
c

creamy-pencil-82913

09/18/2023, 11:17 PM
what do you mean by “same router basically”
on flat networks and physical hosts, there aren’t usually any issues with vxlan, it should just work
w

worried-receptionist-18982

09/18/2023, 11:18 PM
theyre all connected to the same router / switch. think homelab
mmm
c

creamy-pencil-82913

09/18/2023, 11:18 PM
sometimes there are issues with checksum offload on vmware, or kernel modules missing on raspberry pi
but other than that its usually fine
w

worried-receptionist-18982

09/18/2023, 11:18 PM
mmm, one of the nodes is in a virtualbox VM with bridged network. again 
kubectl port-forward
works tho, it's just pod tries to access service no go
c

creamy-pencil-82913

09/18/2023, 11:19 PM
ah, so not all physical after all
w

worried-receptionist-18982

09/18/2023, 11:20 PM
well whats the difference between 
kubectl port-forward
and a pod accessing a service? dont they both do the same thing basically via kube proxy?
if i switch my flannel config to use wireguard, would that likely sidestep anything virtualbox is doing / vxlan filtering ?
c

creamy-pencil-82913

09/18/2023, 11:20 PM
no, not at all the same really
w

worried-receptionist-18982

09/18/2023, 11:20 PM
oh derp
c

creamy-pencil-82913

09/18/2023, 11:21 PM
port-forward connects to the pod via a reverse tunnel from the apiserver to the kubelet
pods connecting to services uses iptables rules from kube-proxy, and the CNI network overlay
completely different paths
w

worried-receptionist-18982

09/18/2023, 11:21 PM
oh huh
so port forward ... is encrypted? or its like some sort of network sent thru a TLS connection?
c

creamy-pencil-82913

09/18/2023, 11:22 PM
tunneled over http
w

worried-receptionist-18982

09/18/2023, 11:22 PM
mmm ok
c

creamy-pencil-82913

09/18/2023, 11:22 PM
from kubectl, to the apiserver, to the kubelet
CNI not involved at all
w

worried-receptionist-18982

09/18/2023, 11:22 PM
got it. pod connecting to service is not tunneled
unless using wireguard with flannel?
c

creamy-pencil-82913

09/18/2023, 11:23 PM
pod connecting to a service just uses the CNI overlay, which with vxlan is encapsulation with no encryption
just a very small extra header on the packet basically
w

worried-receptionist-18982

09/18/2023, 11:23 PM
ah ok that makes more sense
it's been a decade since i took ugrad networking ok thank you! so the issue im facing, trying to muck with iptables might not be a great idea for me? maybe virtualbox is doing something that's interfering? i've had a good experience with k3s in another set-up with just ubuntu hosts no virtualization at all, different cluster different times
if i run k8s with 
flannel-backend=wireguard
then that's tunneling everything thru encrypted wireguard effectively? maybe i should just try that. i dont need the wg tunnels today but some day i might, e.g. hybrid cloud set-up
c

creamy-pencil-82913

09/18/2023, 11:30 PM
yeah there’s a lot more overhead to it
you might just try disabling checksum offload on all the nodes, see if it’s that?
Copy code
ethtool --offload eth0 rx off tx off
on all the nodes
can also see if
Copy code
ethtool -K eth0 gso off
helps
you’re sure that the one on a VM is actually bridged, and not using NAT through the host’s IP?
w

worried-receptionist-18982

09/18/2023, 11:34 PM
yeah it's def not NAT. i can ssh to the VM and hit other services and everything from external machines to it's IP. cool thanks i have never used 
ethtool
let me try that out. i kinda wanted to give 
wg
a spin anyways too. thanks for helping me understand diff between proxy / tunnel and the overlay, i had mistaken that the proxy / tunnel always gets used
i guess kube proxy is mainly for load balanced stuff. service doesn't have LB. makes sense
c

creamy-pencil-82913

09/18/2023, 11:38 PM
kube-proxy handles pods accessing service clusterip addresses
w

worried-receptionist-18982

09/19/2023, 10:29 PM
welp i was using a release that was older than i thought 1.25.12. when i try v1.28.1+k3s1 now, it works (!). i did try the 
ethtool
stuff and re-installed 1.25.12 several times so im pretty sure the ethtool stuff didn't fix (tho i do see in github issues, several from years ago, it's a fix for many people). the issue was that pod ip was routable but not service ip. seems several people have seen this, e.g. https://github.com/k3s-io/k3s/issues/1638#issuecomment-615088410 any idea on what in ... flannel? might have changed in k8s 1.25.12 -> 1.28.1 to mitigate this particular issue? i've got ubuntu 20.04 and i never touched the iptables stuff, which seems to be related. it sounds like flannel had some mac address issues that got merged but that was i think a while ago? will keep testing a bit welp so i was using a release that was fairly old i guess, 
v1.28.1+k3s1
3 Views
Powered by