https://rancher.com/ logo
Title
s

sticky-megabyte-50644

08/04/2022, 5:44 PM
đź‘‹ Hello, team! I'm facing issues with RKE2 on air-gapped environment. "Failed to create sandbox", because its unable to pull rancher/pause:3.6. I'm pretty sure this image was part of the original RKE2 deployment
c

creamy-pencil-82913

08/04/2022, 6:08 PM
that image is included in the airgap tarballs. Did you download them and place them in the agent/images directory on your nodes, or import them into your private registry mirror?
s

sticky-megabyte-50644

08/04/2022, 6:09 PM
no - I used the official TF module which sets up an AWS launch configuration, then I closed off the airgap.
I'll try putting the tarball (uncompressed if I understand correctly?) under /var/lib/rancher/rke2/agent/images
c

creamy-pencil-82913

08/04/2022, 6:10 PM
I don’t think that would be sufficient. There’s more to an airgap install than just installing things and then cutting it off from the internet. See https://docs.rke2.io/install/airgap/
s

sticky-megabyte-50644

08/04/2022, 6:29 PM
ty
can I ask a last question, is that airgap tarball loaded into containerd on agent bootup time? if so, whenever the worker node gets populated with other images, when the image cleanup takes place, what excludes system images (like pause) from the cleanup?
c

creamy-pencil-82913

08/04/2022, 6:42 PM
pause should be protected from GC by the kubelet itself, but the kubelet didn’t do a good job of that prior to 1.24
s

sticky-megabyte-50644

08/04/2022, 6:43 PM
so how do we mitigate? upgrade to 1.24?
c

creamy-pencil-82913

08/04/2022, 6:43 PM
the best way to make sure that doesn’t happen is to make sure you don’t run out of disk space and tune your GH thresholds, or use a private registry mirror so that if GC does prune your images they can be pulled again.
s

sticky-megabyte-50644

08/04/2022, 6:43 PM
ok thats unerstandable, thank you!!!