https://rancher.com/ logo
Title
h

happy-branch-33441

09/23/2022, 12:57 AM
Hi folks! I’m have a pretty basic setup (single machine running k3s, nothing fancy), my cluster runs fine and everything is good. But if I create an AMI/snapshot of the entire machine, and then restore it to another, DNS/networking on all pods is broken and doesn’t recover automatically. As soon as I restart k3s (
systemctl restart k3s
or reboot the machine) it all comes back, networking is fine again. It’s quite annoying to have to restart k3s manually after snapshot+restore of a machine, though - any idea why this would happen or how I can debug further?
b

bulky-sunset-52084

09/24/2022, 4:27 PM
Sounds like stale iptables rules... Hard to say without seeing the nodes - but I would compare the
iptables-save
output on a broken node vs a working one.
h

happy-branch-33441

09/26/2022, 8:28 PM
Yes, I think it’s stale iptables rules-you are correct. To workaround this I’m restarting k3s on startup now every 30s for 10 minutes (since restarts appear HA, this proves not too disruptive and things settle after about 5 minutes on the new machine from a restored AMI.)
I wonder if there’s a way to convince k3s to refresh iptables rules / detect changes in the networking interfaces without restarting k3s?
The other option I’ve thought of is doing an offline install of k3s on each boot, since installation is rather fast (faster than restarting k3s periodically on reboot and waiting for things to settle.) Feels quite hacky though