https://rancher.com/ logo
Join Slack
Powered by
This message was deleted.
# k3s
a
This message was deleted.
c
I already added a test for it, yes. It'll be fixed for December.
This is an intentional change on the part of the Spegel maintainer that we will have to work around.
✅ 1
You can look at the upstream Spegel issue for more information.
And to be clear it works fine if you pull from a registry via CRI interface, it is just locally imported content that cannot be shared due to missing attributes.
a
images in k3s-airgap-images-amd64.tar.zst are the affected ones right?
c
If you are dropping them in the images folder, that would be affected yes. If you're loading them into your own private registry and then resharing them from there, that will work fine.
It is not any specific image or tarball that is affected. It is the method of loading the image.
a
Yes got it. we do load them into agent/images folders before k3s startup
@creamy-pencil-82913 not to be nosy here, im just trynna understand the current flow. I saw your commit '_Fix spegel ready checks to give server more time to find a peer_'. Were we running into issues where spegel wasn't getting ready in time (prolly multinode cases)?
Copy code
Switching to the containerd snapshotter allows us to keep the original manifest digest when pulling and saving image tarballs.
We're not using upstream k3s-airgap-images-amd64.tar.zst instead we remove some unwanted images from this and repackage them(docker load + docker save operation). Do we have to move towards the containerd snapshotter too? I think today im using the legacy snapshotter (v1).
Copy code
[root@ip-10-151-21-17 ec2-user]# sudo ctr -n k8s.io images inspect docker.io/rancher/mirrored-pause:3.6 docker.io/rancher/mirrored-pause:3.6 │ Created: 2025-11-22 10:36:33.459764905 +0000 UTC │ Updated: 2025-11-22 12:12:01.002561018 +0000 UTC │ Label "io.cattle.k3s.pinned": "pinned" │ Label "io.cri-containerd.image": "managed" │ Label "io.cri-containerd.pinned": "pinned" └── application/vnd.oci.image.manifest.v1+json @sha256:16974531848218d24822bf606be022d030ab8c9b05b2ecf11076c4c1c6885c95 (399 bytes) ├── application/vnd.oci.image.config.v1+json @sha256:6270bb605e12e581514ada5fd5b3216f727db55dc87d5889c790e4c760683fee (901 bytes) └── application/vnd.oci.image.layer.v1.tar @sha256:1021ef88c7974bfff89c5a0ec4fd3160daac6c48a075f74cff721f85dd104e68 (684544 bytes)
c
the ready checks are more important for rke2, where basically the first thing it does is pull the runtime image to extract binaries for containerd and kubelet. but it is also useful for k3s. Sometime spegel would not finish bootstrapping the p2p mesh before the first images were pulled - and for rke2 this meant a failed startup, or for k3s it would either take longer due to retrying, or just go to the external registry. So this should fix that.
✅ 1
for the snapshotter change - containerd’s legacy snapshotter flattens manifest lists when you pull and save. You end up with a single-platform image in the tarball, instead of a manifest list (potentially with multiple platforms, or maybe just one - depending on what you’ve pulled) Other tools like containerd (or containerd snapshotter for docker), crane, skopeo, etc maintain the manifest list when pulling and saving. So this makes sure that the image stays a manifest list, and the manifest digest doesn’t change, when we pull and save for the tarballs.
✅ 1
a
Thanks @creamy-pencil-82913. is the snapshotter change a breaking change for my usecase? images saved in tarball via legacy snapshotter wont work/participate in spegel image sharing? We use k3s exclusively on linux amd64 machines + embedded registry with disable-default-registry-endpoint: true on agent nodes. registries.yaml has docker.io entry
c
No... tarballs are tarballs. They'll just contain a proper manifest list in future releases.
if you keep using the legacy snapshotter, the digest you see for a given tag won’t match the digest shown on the upstream registry. that’s the only difference
✅ 1
a
Oh thats fine, we retag chainguard images to docker.io images anyway 🙂
3 Views
Open in Slack
PreviousNext
Powered by