This message was deleted.

Is this using Kubernetes? kubelet has some garbage collection thresholds, and once they are reached it will delete images that are not currently in use by containers. The way to prevent this is to frequently delete build caches. Look into

docker system prune

to reclaim space.

I do have K8s enabled. I guess I just need to run prune more regularly. I was hoping there'd be a way to keep it from culling images I want to keep.

There is really not much you can do except keeping sufficient free disk space. Or to keep a container running that references the image. The behaviour is documented at Garbage Collection | Kubernetes⁠

Claude said to create a daemonset that references the images I want to keep in initContainers. Do you think that will work?

The initContainers force the image to be present and "used" without actually running anything.

I'm using dockerd. It said containerd has an

io.cri-containerd.pinned=pinned

label

I think the DaemonSet will work. Kubelet will not delete images that are in use by a running container.

Thanks, good to know. I just have a couple base images I use to build others and it's annoying to have to rebuild those when then get culled.

Note that kubelet only deletes images; it cannot cleanup any other data. If you prune e.g. the build cache more frequently, you may have a higher chance of kubelet not triggering any image deletes.

Sounds like I need to make a script for I can run when things are calm that deletes the daemonset, docker system prune, rebuild the images, and apply the daemonset. Of course, I always hit this when tracking down some other problem, and have to take a time hit while I rebuild the base images I may have built a few hours ago.