Sounds like stale iptables rules... Hard to say without seeing the nodes - but I would compare the
iptables-save
output on a broken node vs a working one.
h
happy-branch-33441
09/26/2022, 8:28 PM
Yes, I think it’s stale iptables rules-you are correct. To workaround this I’m restarting k3s on startup now every 30s for 10 minutes (since restarts appear HA, this proves not too disruptive and things settle after about 5 minutes on the new machine from a restored AMI.)
happy-branch-33441
09/26/2022, 8:28 PM
I wonder if there’s a way to convince k3s to refresh iptables rules / detect changes in the networking interfaces without restarting k3s?
happy-branch-33441
09/26/2022, 8:30 PM
The other option I’ve thought of is doing an offline install of k3s on each boot, since installation is rather fast (faster than restarting k3s periodically on reboot and waiting for things to settle.) Feels quite hacky though