https://rancher.com/ logo
Join Slack
Powered by
This message was deleted.
# k3d
a
This message was deleted.
g
This happens reproducibly every time – at least on that system. I have a feeling it could be related to 
live-restore
since otherwise the containers would of course be restarted during the docker daemon restart. But I have not tested that hypothesis.
I can confirm it only happens (to me) when having the Docker daemon’s 
live-restore
enabled. Without it (after setting the containers’ restart policy to 
always
) the containers come up with working DNS after the daemon restart. (If the restart policy results in the containers staying stopped, the issue is of course avoided altogether.)
r
That sounds almost sounds like either it opens a socket and expects it to remain open and never retries opening it or the docker network gateway IP changes, but neither of those sound like what I'd guess/assume behavior to be/do.
Although you might check the network gateway. If the live-restore keeps the old network the newly started daemon might create its own new network on a different IP range?
That's the closest I can think of to making sense, though it still seems dumb/wrong.
g
The gateway IP stays the same (the first IP in the subnet, ….129 here).
r
I guess the third option is maybe on service start it configures something in the docker daemon that doesn't end up that way by default on the daemon restarting? Not sure what, though.
g
To be honest I have no idea how that DNS service works anyway (is that part of the docker daemon? a central DNS server shared across all docker networks? separate ones?). At this point I’m just glad it works.
r
I have no clue what the K3D_FIX_DNS=1 does, when we ran into similar we just updated the coredns ConfigMap to toss in a valid nameserver and that seems to work (until our ArgoCD-deployed app pushes a Prometheus change that overwrites the coredns ConfigMap and we have to go fix it again).
My understanding is https://github.com/k3d-io/k3d/issues/209 is the GitHub issue around which K3D_FIX_DNS was developed. I think that also had the fix we ended up with (
kubectl edit -n kube-system configmap coredns
and then change line "forward . /etc/resolv.conf" to "forward . /etc/resolv.conf 8.8.8.8" (or replace Google's DNS there with the appropriate internal one).
I think that's about all the info I have on that stuff.
250 Views
Open in Slack
PreviousNext
Powered by