I wanted to give an update on my k3s problem (NOW SOLVED) and request more information on how I can write a better test case that demonstrates this problem so I can file a bug. Essentially, my understanding of this problem is when you have two different default routes with the same metrics then Linux will choose one of those default routes and k3s (or flannel or the vxlan configuration) will choose another default route. They have different rules for disambiguation in this case. This means that sometimes you will have EVERYTHING work outside of k3s, and everything inside of k3s that needs to communicate with the outside world broke. I've written up a description of this here, but I would like assistance if someone knows how to create the dummy interfaces and default routes required to demonstrate this problem outside of terraform. https://devops.stackexchange.com/q/16161/18965

k3os had this issue when I had 2 interfaces one public and one private, and then quit altogether, because default route was put on the wrong interface eth1 instead of eth0

Right, but if there are two default routes, wouldn't it be a bug to have Linux choose one and k3s choose a different one?

This is not really a k3s bug, but rather the network manager (which is some exotic)

so where would a file a bug such that the network manager for k3s is picking a different default route then the one Linux picks?

This is k3os I used, this is not really tied to k3s directly

Right, but I am seeing it on k3s right now, and I've never used k3os. I'm not very aware of the tech that's going on in k3s.

Otherwise default routes are overriden, you could potentially make more "default" routes, if you have multiple IP addresses with different subnets

I don't follow =(

I don't know the specifics of the problem 😄 sorry

It's all on the devops.se link above. Basically, I have three default routes. One gets chosen by Linux and works fine. The other gets chosen by k3s. It's not the same one. It doesn't work. I don't believe k3s should be using a different default route from the host.

iptables -t nat -nvL for masquerade