This message was deleted.
# k3sa
adamant-kite-43734
09/05/2023, 8:16 PMThis message was deleted.
c
creamy-pencil-82913
09/05/2023, 8:18 PMk3s doesn’t stop container processes on either server, or agent
creamy-pencil-82913
09/05/2023, 8:18 PMthat doesn’t mean the process won’t crash for other reasons if the kubelet or apiserver goes away, but it doesn’t stop them.
creamy-pencil-82913
09/05/2023, 8:19 PMhave you opened an issuse with the cilium folks? I’m not seeing anything here k3s specific
w
worried-jackal-89144
09/06/2023, 7:07 PMNot yet. I was wondering which side is the source of issue here.
worried-jackal-89144
09/06/2023, 8:19 PM@creamy-pencil-82913 I tested it a bit more and it seems that this line from k3s install script:
iptables-save | grep -v KUBE- | grep -iv flannel | $SUDO iptables-restoreworried-jackal-89144
09/06/2023, 8:25 PMMaybe a flag for the script to disable this step would be useful. If someone is not using flannel then this step is obsolete and will do nothing
c
creamy-pencil-82913
09/06/2023, 8:27 PMbreaking it how?
creamy-pencil-82913
09/06/2023, 8:29 PMRemoving the kubelet and flannel iptables rules shouldn’t break anything…
creamy-pencil-82913
09/06/2023, 8:29 PMyou can always set INSTALL_K3S_SKIP_ENABLE or INSTALL_K3S_SKIP_START to skip that bit
w
worried-jackal-89144
09/06/2023, 8:32 PM"breaking it how?" - all connectivity of the VM is broken, and no services can work until:
• reset of vm is done
• or I run commands for cleanup of cilium stuff mentioned here: https://docs.k3s.io/installation/network-options#custom-cni
worried-jackal-89144
09/06/2023, 8:33 PM"Removing the kubelet and flannel iptables rules shouldn’t break anything…" - my idea is that maybe it is related somehow to the order of rules? Not that easy to debug...
c
creamy-pencil-82913
09/06/2023, 8:33 PMah right, I forgot that cilium does dumb things with the host networking
🙂 1
w
worried-jackal-89144
09/06/2023, 8:33 PM"you can always set INSTALL_K3S_SKIP_ENABLE or INSTALL_K3S_SKIP_START to skip that bit" - this is interesting idea
c
creamy-pencil-82913
09/06/2023, 8:34 PMyou’ll have to start the service yourself afterwards, but that shouldn’t be too bad
w
worried-jackal-89144
09/06/2023, 8:36 PMThat's pretty easy with ansible. I'll give it a try. Thanx
l
late-needle-80860
09/06/2023, 9:12 PM@worried-jackal-89144 I would suggest you to use the no kube proxy mode of cilium. Then iptables use is reduced DRASTICALLY. Further, are you actually disabling flannel entirely when you use cilium?
w
worried-jackal-89144
09/06/2023, 9:25 PM@late-needle-80860 Yes, flannel is disabled, and I'm using cilium's kube-proxy (I tried with and without it, but the result was the same). The easiest solution is to use install script, install k3s with INSTALL_K3S_SKIP_START=true, and then restart it explicitly
l
late-needle-80860
09/07/2023, 7:11 AMI would discuss this over on the Cilium slack community as well .. it seems weird. I’m running K3s on v1.27.4+k3s1 as well … have no such issue with Cilium v1.14
late-needle-80860
09/07/2023, 7:12 AMWe don’t configure the Pod CIDR though on-prem .. as we don’t need to. However, I can’t see that affecting node network connection in any negative way
late-needle-80860
09/07/2023, 7:14 AMWe use:
• bpf masquerading
• auto direct node routes
• replace the kube proxy in strict mode
• we set the ipv4 native routing cidr
• we disable tunnel mode
late-needle-80860
09/07/2023, 7:14 AMall the above is of course Cilium conf.