Hey I had a k3s that s been running on an ubuntu VM for a co rancher-users #k3s

Title

bored-horse-3670

03/20/2023, 2:28 PM

Hey, I had a k3s that's been running on an ubuntu VM for a couple years now. (single node) I noticed that it recently stopped forwarding egress traffic. I tried switching the flannel mode from the default vxlan to the wireguard-native type. It is definitely using wireguard, but the egress traffic still times out. It's a proxmox VM connected to a bridge device on the proxmox host. The weird thing is that the packets leaving the ubuntu VM still have the container's address set as the source ip. I tried restarting k3s and also tried restarting the whole host. I saw a few similar issues on the github issue tracker. I'll include my iptables-save output in a thread.

iptables-save.txt

running a packet trace on the VM host where the ubuntu VM is running: packets seen when doing

dig <http://google.com|google.com> @8.8.8.8

on the ubuntu VM that is running k3s:

10:29:53.210020 IP 192.168.168.129.44953 > dns.google.domain: 48216+ [1au] A? <http://google.com|google.com>. (51)
10:29:53.238070 IP dns.google.domain > 192.168.168.129.44953: 48216 1/0/1 A 142.250.80.46 (55)

packets seen when doing

dig <http://google.com|google.com> @8.8.8.8

in a kube container running on the same VM:

10:30:29.737570 IP 10.42.0.106.55863 > dns.google.domain: 14949+ [1au] A? <http://google.com|google.com>. (51)
10:30:34.741290 IP 10.42.0.106.59193 > dns.google.domain: 14949+ [1au] A? <http://google.com|google.com>. (51)

The 10.42.0.106 pod address shouldn't be here at all.

output of

sudo ifconfig

ifconfig

output of

sudo route -n

route

I do recognize that there's both the flannel-wg and flannel1.1 interfaces. I think something is interfering with the iptables rules since I only see ingress rules present in the iptables output. ... and sure enough the k3s agent logs have stuff like this:

Mar 20 02:31:25 kessler k3s[1052]: E0320 02:31:25.949679    1052 iptables.go:291] Failed to bootstrap IPTables: failed to setup iptables-restore payload: failed to check rule existence: running [/sbin/iptables -t nat -C FLANNEL-POSTRTG -s 10.42.0.0/16 ! -d 224.0.0.0/4 -m comment --comment flanneld masq -j MASQUERADE  --wait]: exit status 2: Bad argument `'
Mar 20 02:31:25 kessler k3s[1052]: Try `iptables -h' or 'iptables --help' for more information.
Mar 20 02:31:26 kessler k3s[1052]: I0320 02:31:26.001850    1052 iptables.go:405] Some iptables rules are missing; deleting and recreating rules

running that command manually shows part of the error was cut off:

# /sbin/iptables -t nat -C FLANNEL-POSTRTG -s 10.42.0.0/16 ! -d 224.0.0.0/4 -m comment --comment flanneld masq -j MASQUERADE  --wait
Bad argument `masq'
Try `iptables -h' or 'iptables --help' for more information.

... lol someone did the same exact 8.8.8.8 packet capture I did when they opened this github issue: https://github.com/k3s-io/k3s/issues/7096

and the fix suggested to use the

--prefer-bundled-bin

option did the trick. It's probably time to upgrade the ubuntu distro too.

👀 1

creamy-pencil-82913

03/20/2023, 5:11 PM

Can you comment on that issue to mention what version of Ubuntu you are on? That would be a useful data point

3 Views

#k3s Join Slack