https://rancher.com/ logo
#k3s
Title
# k3s
a

agreeable-alarm-7502

03/01/2023, 9:22 PM
Hi folks! I've been trying to set up k3s on a few spare machines running in my homelab. 1 control-plane node and 2 agent nodes. k3s & flannel start just fine, and the agent nodes can join the cluster just fine. However, pods can't route out of their network, even to the kubernetes API. i'm at a loss as to why that may be the case, can someone help me troubleshoot what's going on?
1
r

rough-farmer-49135

03/02/2023, 2:36 PM
So are you saying that your pod's trying to access something outside your k3s cluster and it fails to get the resource? My first question is if DNS works, as I've seen DNS fail inside a kubernetes cluster more than occasionally.
You should be able to use
kubectl exec
to get a shell inside the pod to try things out (at least if it's still running and it wasn't a fatal error).
a

agreeable-alarm-7502

03/02/2023, 7:05 PM
I found out what I missed, I misconfigured my firewall (nftables). DNS didn't even work, it was all traffic coming out of the pod CIDR range... because I was blocking forwarded packets. That is, I had this sort of firewall:
Copy code
table inet my_firewall {
  chain my_forward_rules {
    type filter hook forward;
    policy drop;
  }
}
I did set the policy correctly on my control-plane node so I wasn't paying attention to that part.
I feel very silly about overlooking that 😅
r

rough-farmer-49135

03/02/2023, 7:06 PM
Happens to us all at times.
a

agreeable-alarm-7502

03/02/2023, 7:07 PM
Do you think it would be a possible feature to add to
k3s check-config
?
r

rough-farmer-49135

03/02/2023, 7:09 PM
I think part of the problem is that firewall in general is chancy with Kubernetes. As I recall vanilla Kubernetes tells you to turn off the firewall & I know RKE2 at least is explicitly incompatible with firewalld. So I'm not sure how easy or difficult it'd be to programmatically figure out what's fine & what's not. You could easily have a "WARNING - your firewall isn't disabled, make sure it doesn't cause problems..."-sort of message, but I'm not sure past that.
a

agreeable-alarm-7502

03/02/2023, 7:10 PM
Yeah. I was thinking maybe the command could spin up a random network-namespaced subprocess and have it check it can route out successfully.
r

rough-farmer-49135

03/02/2023, 7:12 PM
No clue, I'm not a k3s dev, so maybe it's easier than I expect. However, with all Linux firewalls (such as iptables, nftables, ufw, & firewalld) all just being frontends & management daemons for the kernel's netfilter, which is also used for handling internal networks & NATs & bridges & routing & all that happens with container networking that's where I'm not sure how much you could or couldn't do with it.
2 Views