https://rancher.com/ logo
Title
b

bored-horse-3670

03/20/2023, 2:28 PM
Hey, I had a k3s that's been running on an ubuntu VM for a couple years now. (single node) I noticed that it recently stopped forwarding egress traffic. I tried switching the flannel mode from the default vxlan to the wireguard-native type. It is definitely using wireguard, but the egress traffic still times out. It's a proxmox VM connected to a bridge device on the proxmox host. The weird thing is that the packets leaving the ubuntu VM still have the container's address set as the source ip. I tried restarting k3s and also tried restarting the whole host. I saw a few similar issues on the github issue tracker. I'll include my iptables-save output in a thread.
iptables-save.txt
running a packet trace on the VM host where the ubuntu VM is running: packets seen when doing
dig <http://google.com|google.com> @8.8.8.8
on the ubuntu VM that is running k3s:
10:29:53.210020 IP 192.168.168.129.44953 > dns.google.domain: 48216+ [1au] A? <http://google.com|google.com>. (51)
10:29:53.238070 IP dns.google.domain > 192.168.168.129.44953: 48216 1/0/1 A 142.250.80.46 (55)
packets seen when doing
dig <http://google.com|google.com> @8.8.8.8
in a kube container running on the same VM:
10:30:29.737570 IP 10.42.0.106.55863 > dns.google.domain: 14949+ [1au] A? <http://google.com|google.com>. (51)
10:30:34.741290 IP 10.42.0.106.59193 > dns.google.domain: 14949+ [1au] A? <http://google.com|google.com>. (51)
The 10.42.0.106 pod address shouldn't be here at all.
output of
sudo ifconfig
output of
sudo route -n
I do recognize that there's both the flannel-wg and flannel1.1 interfaces. I think something is interfering with the iptables rules since I only see ingress rules present in the iptables output. ... and sure enough the k3s agent logs have stuff like this:
Mar 20 02:31:25 kessler k3s[1052]: E0320 02:31:25.949679    1052 iptables.go:291] Failed to bootstrap IPTables: failed to setup iptables-restore payload: failed to check rule existence: running [/sbin/iptables -t nat -C FLANNEL-POSTRTG -s 10.42.0.0/16 ! -d 224.0.0.0/4 -m comment --comment flanneld masq -j MASQUERADE  --wait]: exit status 2: Bad argument `'
Mar 20 02:31:25 kessler k3s[1052]: Try `iptables -h' or 'iptables --help' for more information.
Mar 20 02:31:26 kessler k3s[1052]: I0320 02:31:26.001850    1052 iptables.go:405] Some iptables rules are missing; deleting and recreating rules
running that command manually shows part of the error was cut off:
# /sbin/iptables -t nat -C FLANNEL-POSTRTG -s 10.42.0.0/16 ! -d 224.0.0.0/4 -m comment --comment flanneld masq -j MASQUERADE  --wait
Bad argument `masq'
Try `iptables -h' or 'iptables --help' for more information.
... lol someone did the same exact 8.8.8.8 packet capture I did when they opened this github issue: https://github.com/k3s-io/k3s/issues/7096
and the fix suggested to use the
--prefer-bundled-bin
option did the trick. It's probably time to upgrade the ubuntu distro too.
👀 1
c

creamy-pencil-82913

03/20/2023, 5:11 PM
Can you comment on that issue to mention what version of Ubuntu you are on? That would be a useful data point