Hi folks! I've been trying to set up k3s on a few spare machines running in my homelab. 1 control-plane node and 2 agent nodes. k3s & flannel start just fine, and the agent nodes can join the cluster just fine. However, pods can't route out of their network, even to the kubernetes API. i'm at a loss as to why that may be the case, can someone help me troubleshoot what's going on?

So are you saying that your pod's trying to access something outside your k3s cluster and it fails to get the resource? My first question is if DNS works, as I've seen DNS fail inside a kubernetes cluster more than occasionally.

I found out what I missed, I misconfigured my firewall (nftables). DNS didn't even work, it was all traffic coming out of the pod CIDR range... because I was blocking forwarded packets. That is, I had this sort of firewall:

table inet my_firewall {
  chain my_forward_rules {
    type filter hook forward;
    policy drop;
  }
}

I did set the policy correctly on my control-plane node so I wasn't paying attention to that part.

Happens to us all at times.

Do you think it would be a possible feature to add to

k3s check-config

?

I think part of the problem is that firewall in general is chancy with Kubernetes. As I recall vanilla Kubernetes tells you to turn off the firewall & I know RKE2 at least is explicitly incompatible with firewalld. So I'm not sure how easy or difficult it'd be to programmatically figure out what's fine & what's not. You could easily have a "WARNING - your firewall isn't disabled, make sure it doesn't cause problems..."-sort of message, but I'm not sure past that.

Yeah. I was thinking maybe the command could spin up a random network-namespaced subprocess and have it check it can route out successfully.

No clue, I'm not a k3s dev, so maybe it's easier than I expect. However, with all Linux firewalls (such as iptables, nftables, ufw, & firewalld) all just being frontends & management daemons for the kernel's netfilter, which is also used for handling internal networks & NATs & bridges & routing & all that happens with container networking that's where I'm not sure how much you could or couldn't do with it.