03/20/2023, 2:28 PM
Hey, I had a k3s that's been running on an ubuntu VM for a couple years now. (single node) I noticed that it recently stopped forwarding egress traffic. I tried switching the flannel mode from the default vxlan to the wireguard-native type. It is definitely using wireguard, but the egress traffic still times out. It's a proxmox VM connected to a bridge device on the proxmox host. The weird thing is that the packets leaving the ubuntu VM still have the container's address set as the source ip. I tried restarting k3s and also tried restarting the whole host. I saw a few similar issues on the github issue tracker. I'll include my iptables-save output in a thread.
running a packet trace on the VM host where the ubuntu VM is running: packets seen when doing
dig <|> @
on the ubuntu VM that is running k3s:
10:29:53.210020 IP > 48216+ [1au] A? <|>. (51)
10:29:53.238070 IP > 48216 1/0/1 A (55)
packets seen when doing
dig <|> @
in a kube container running on the same VM:
10:30:29.737570 IP > 14949+ [1au] A? <|>. (51)
10:30:34.741290 IP > 14949+ [1au] A? <|>. (51)
The pod address shouldn't be here at all.
output of
sudo ifconfig
output of
sudo route -n
I do recognize that there's both the flannel-wg and flannel1.1 interfaces. I think something is interfering with the iptables rules since I only see ingress rules present in the iptables output. ... and sure enough the k3s agent logs have stuff like this:
Mar 20 02:31:25 kessler k3s[1052]: E0320 02:31:25.949679    1052 iptables.go:291] Failed to bootstrap IPTables: failed to setup iptables-restore payload: failed to check rule existence: running [/sbin/iptables -t nat -C FLANNEL-POSTRTG -s ! -d -m comment --comment flanneld masq -j MASQUERADE  --wait]: exit status 2: Bad argument `'
Mar 20 02:31:25 kessler k3s[1052]: Try `iptables -h' or 'iptables --help' for more information.
Mar 20 02:31:26 kessler k3s[1052]: I0320 02:31:26.001850    1052 iptables.go:405] Some iptables rules are missing; deleting and recreating rules
running that command manually shows part of the error was cut off:
# /sbin/iptables -t nat -C FLANNEL-POSTRTG -s ! -d -m comment --comment flanneld masq -j MASQUERADE  --wait
Bad argument `masq'
Try `iptables -h' or 'iptables --help' for more information.
... lol someone did the same exact packet capture I did when they opened this github issue:
and the fix suggested to use the
option did the trick. It's probably time to upgrade the ubuntu distro too.
👀 1


03/20/2023, 5:11 PM
Can you comment on that issue to mention what version of Ubuntu you are on? That would be a useful data point