https://rancher.com/ logo
Powered by
Title
a

able-mechanic-45652

10/24/2022, 9:07 AM
i need a bit of debugging help, seem our k3s cluster is unable to resolve names correctly. Probably issue with worker iptables or something, pods running on same worker as coredns resolve names correctly but pods on other workers don't seem to be able to resolve clusters internal names so seem unable to connect to coredns
b

bland-account-99790

10/24/2022, 1:13 PM
can you share your k3s config please?
or 
kubectl get nodes -o yaml | grep "<http://k3s.io/node-args|k3s.io/node-args>"
, which gives the config parameters you used
a

able-mechanic-45652

10/24/2022, 1:56 PM
# /usr/local/bin/k3s kubectl get nodes -o yaml | grep "k3s.io/node-args" k3s.io/node-args: '["agent","--server","https://kube-master.domain.fi:6443"]' k3s.io/node-args: '["server","--disable-cloud-controller","--no-deploy","traefik","--node-taint","k3s-controlplane=true:NoExecute"]' k3s.io/node-args: '["agent","--server","https://kube-master.domain.fi:6443"]'
IPTables-Reject-Input: IN=eth0 OUT= MAC=... SRC=<worked#1 ip> DST=<worker#2 ip> LEN=137 TOS=0x00 PREC=0x00 TTL=64 ID=33173 PROTO=UDP SPT=53112 DPT=8472 LEN=117
I got above errors in journal on both worker nodes so it seems the iptables is blocking ... is it flannel traffic?
$ sudo cat /etc/sysconfig/iptables
*filter
:INPUT ACCEPT [0:0]
:FORWARD ACCEPT [0:0]
:OUTPUT ACCEPT [51:25393]
-A INPUT -m state --state RELATED,ESTABLISHED -j ACCEPT
-A INPUT -p icmp -j ACCEPT
-A INPUT -i lo -j ACCEPT
-A INPUT -p tcp -m state --state NEW -m tcp --dport 22 -j ACCEPT
-A INPUT -p tcp -m state --state NEW -m tcp -s <worker#1 ip>,<worker#2 ip> --dport 8472 -j ACCEPT
-A INPUT -p tcp -m state --state NEW -m tcp --dport 30000:32767 -j ACCEPT
-A INPUT -s 10.42.0.0/16 -j ACCEPT
-A INPUT -d 10.42.0.0/16 -j ACCEPT
-A INPUT -j LOG --log-prefix "IPTables-Reject-Input: " --log-level 4
-A INPUT -j REJECT --reject-with icmp-host-prohibited
-A FORWARD -s 10.42.0.0/16 -j ACCEPT
-A FORWARD -d 10.42.0.0/16 -j ACCEPT
-A FORWARD -j LOG --log-prefix "IPTables-Reject-Forward: " --log-level 4
-A FORWARD -j REJECT --reject-with icmp-host-prohibited
-A OUTPUT -o lo -j ACCEPT
COMMIT
*nat
:PREROUTING ACCEPT [17:964]
:INPUT ACCEPT [0:0]
:OUTPUT ACCEPT [0:0]
:POSTROUTING ACCEPT [0:0]
COMMIT
the default iptables config would seem to allow traffic on 8472 port though
b

bland-account-99790

10/24/2022, 3:00 PM
Can you try pinging coredns from the node where it runs and from another node? Let' check if it is a connectivity issue
a

able-mechanic-45652

10/24/2022, 3:04 PM
actually I found a bruteforce method, https://github.com/k3s-io/k3s/issues/535#issuecomment-907789701 Seemed to fix the connectivity issues for now
b

bland-account-99790

10/24/2022, 4:17 PM
interesting. You are using Oracle Cloud?
a

able-mechanic-45652

10/24/2022, 4:18 PM
No, centos 7 vm's on vmware. Should probably test if these changes work after boot as well
3 Views
#k3sJoin Slack
Powered by