I keep running into this problem, where on some of...
# k3s
h
I keep running into this problem, where on some of my k3s nodes, i will run “k3s ctr images import” to get all my images into each of my nodes ctr image repos and this has been fine. But lately for some reason on some of my nodes and some of my images, they dissapear from the nodes image repo… and its totally random i feel. Like which node and image this happens to…. Any thoughts?
c
Unused images that are manually imported will get pruned if you are low on disk space.
Make sure you don't ever get below 85% free if you don't want the kubelet to start pruning
If you use the automatic importing via the images dir, it will add a label to the imported images that should prevent them from getting cleaned up. But then you'll just be out of disk space.
h
Thankyou for feedback!
@creamy-pencil-82913 when i go on your official k3s website and I look at the “Requirements” section. There is no mention of: “Make sure you don’t ever get below 85% free if you dont want kubelet to start pruning”
c
This isn't a k3s thing... this is how the kubelet works in every Kubernetes distribution
https://kubernetes.io/docs/reference/command-line-tools-reference/kubelet/ > --image-gc-high-threshold int32 Default: 85 > The percent of disk usage after which image garbage collection is always run. Values must be within the range [0, 100], To disable image garbage collection, set to 100. > --image-gc-low-threshold int32 Default: 80 > The percent of disk usage before which image garbage collection is never run. Lowest disk usage to garbage collect to. Values must be within the range [0, 100] and should not be larger than that of --image-gc-high-threshold.
We generally don't document things that are just part of core Kubernetes behavior
If your disk usage gets over the high threshold, it will prune until it is under the low threshold. Defaults are 85/80. This is not a k3s requirement, this just how the kubelet has always worked.
a
@creamy-pencil-82913 With high disk usage, does kubelet remove unused images that were pulled by a pod ie; they werent manually imported via ctr img import.
c
It will remove any image that isn't in active use and isn't pinned
1