This message was deleted Rancher Users #k3s

Join Slack

This message was deleted.

# k3s

adamant-kite-43734

02/26/2026, 3:55 PM

This message was deleted.

average-appointment-24470

02/26/2026, 4:01 PM

@creamy-pencil-82913 im having the same issue with suc upgrades aswell. This is a multi node setup where agent nodes use embedded registry to pull docker.io images like mirrored-pause since default registry fallback is disabled. restarting k3s fixes it however manually running a command is not a viable option for me. Previously 1.33.5 worked fine for me. docker.io images are loaded in master node via /agent/images folder and these images have repo digest + repo tags both as per crictl image -v

creamy-pencil-82913

02/26/2026, 4:15 PM

What version are you running when this happens? Sounds like spegel isn't indexing seeing the content as available in the store until it is restarted. If this is not already fixed in a newer release, I'd recommend opening an issue with the spegel project.

average-appointment-24470

02/26/2026, 4:25 PM

Ran 1.34.4 and 1.35.1 both. Had same issues

average-appointment-24470

02/26/2026, 4:33 PM

https://github.com/spegel-org/spegel/issues/1159 could this be related? is it worth moving back to spegel 0.5 until issues are fixed since its inoperable?

creamy-pencil-82913

02/26/2026, 5:32 PM

Just to confirm, you are running with no upstream registry, relying ONLY on spegel to provide images from the server to the agent? Are you restarting the server, or the agent to fix it?

average-appointment-24470

02/26/2026, 5:33 PM

Yes disable-default-registry-endpoint is true and docker.io is set in registries.yaml so secondary nodes have no way to pull apart from the already loaded images (via agent/images folder) on the master node

average-appointment-24470

02/26/2026, 5:34 PM

Restarting the k3s server i am

average-appointment-24470

02/26/2026, 5:43 PM

Copy code

[root@ip-10-151-21-17 ec2-user]# crictl image -v | grep "mirrored-pause" -A 7
RepoTags: <http://docker.io/rancher/mirrored-pause:3.6|docker.io/rancher/mirrored-pause:3.6>
RepoDigests: <http://docker.io/rancher/mirrored-pause@sha256:16974531848218d24822bf606be022d030ab8c9b05b2ecf11076c4c1c6885c95|docker.io/rancher/mirrored-pause@sha256:16974531848218d24822bf606be022d030ab8c9b05b2ecf11076c4c1c6885c95>
Size: 685844
Uid: value:65535
Pinned: true

creamy-pencil-82913

02/26/2026, 7:04 PM

run both nodes with --debug and see what’s in the logs when the pull fails. it does not sound like that upstream issue with stale nodes.

average-appointment-24470

02/28/2026, 12:41 PM

@creamy-pencil-82913 logs attached. ive done a grep on "registry|p2p|distributed|spegel"

k3s-agent.txt k3s-server.txt

average-appointment-24470

02/28/2026, 12:45 PM

logs of interest might be --server node. this is mirrored-pause sha. Is the spegel's content scan running before the import finishes adding labels ?

Copy code

Feb 28 12:11:42 ip-10-151-22-85.us-west-2.compute.internal k3s[1259]: time="2026-02-28T12:11:42Z" level=error msg="skipping content that cant be converted to reference" error="no distribution source labels found for sha256:16974531848218d24822bf606be022d030ab8c9b05b2ecf11076c4c1c6885c95" logger=spegel

Feb 28 12:11:44 ip-10-151-22-85.us-west-2.compute.internal k3s[1259]: time="2026-02-28T12:11:44Z" level=info msg="Imported docker.io/rancher/mirrored-pause:3.6"
Feb 28 12:11:44 ip-10-151-22-85.us-west-2.compute.internal k3s[1259]: time="2026-02-28T12:11:44.175861088Z" level=info msg="ImageDelete event name:\"docker.io/rancher/mirrored-pause@sha256:16974531848218d24822bf606be022d030ab8c9b05b2ecf11076c4c1c6885c95\""
Feb 28 12:11:44 ip-10-151-22-85.us-west-2.compute.internal k3s[1259]: time="2026-02-28T12:11:44Z" level=info msg="Tagged docker.io/rancher/mirrored-pause@sha256:16974531848218d24822bf606be022d030ab8c9b05b2ecf11076c4c1c6885c95"
Feb 28 12:11:44 ip-10-151-22-85.us-west-2.compute.internal k3s[1259]: time="2026-02-28T12:11:44.177859448Z" level=info msg="ImageCreate event name:\"docker.io/rancher/mirrored-pause@sha256:16974531848218d24822bf606be022d030ab8c9b05b2ecf11076c4c1c6885c95\" labels:{key:\"io.cattle.k3s.pinned\" value:\"pinned\"} labels:{key:\"io.cri-containerd.image\" value:\"managed\"} labels:{key:\"io.cri-containerd.pinned\" value:\"pinned\"}"

-- agent node

Copy code

Feb 28 12:30:48 ip-10-151-21-17.us-west-2.compute.internal k3s[11230]: 2026-02-28T12:30:48.989Z        DEBUG        dht        go-libp2p-kad-dht@v0.36.0/routing.go:579        1 provider entries
Feb 28 12:30:48 ip-10-151-21-17.us-west-2.compute.internal k3s[11230]: 2026-02-28T12:30:48.989Z        DEBUG        dht        go-libp2p-kad-dht@v0.36.0/routing.go:584        got provider: {QmNjkaUu3SzT5VARhuVX9CqzJ9mgLS6xLaNWMAWmCWDfbb: [/ip4/10.151.22.85/tcp/5000]}
Feb 28 12:30:48 ip-10-151-21-17.us-west-2.compute.internal k3s[11230]: 2026-02-28T12:30:48.989Z        DEBUG        dht        go-libp2p-kad-dht@v0.36.0/routing.go:586        using provider: {QmNjkaUu3SzT5VARhuVX9CqzJ9mgLS6xLaNWMAWmCWDfbb: [/ip4/10.151.22.85/tcp/5000]}
Feb 28 12:30:48 ip-10-151-21-17.us-west-2.compute.internal k3s[11230]: 2026-02-28T12:30:48.989Z        DEBUG        dht        go-libp2p-kad-dht@v0.36.0/routing.go:606        got closer peers: 0 []
Feb 28 12:30:48 ip-10-151-21-17.us-west-2.compute.internal k3s[11230]: 2026-02-28T12:30:48.989Z        DEBUG        dht        go-libp2p-kad-dht@v0.36.0/dht.go:712        peer found        {"peer": "QmNjkaUu3SzT5VARhuVX9CqzJ9mgLS6xLaNWMAWmCWDfbb"}
Feb 28 12:30:48 ip-10-151-21-17.us-west-2.compute.internal k3s[11230]: time="2026-02-28T12:30:48Z" level=info handler=mirror ip=127.0.0.1 latency=3.18146ms logger=spegel method=HEAD path=/v2/rancher/mirrored-pause/manifests/3.6 registry=docker.io status=200
Feb 28 12:30:48 ip-10-151-21-17.us-west-2.compute.internal k3s[11230]: 2026-02-28T12:30:48.995Z        DEBUG        dht        go-libp2p-kad-dht@v0.36.0/routing.go:506        finding providers        {"cid": "bafkreibyxxlewz3hvobjgxst7vgdgsp55f275mx3kmjwdwbasqcnzohd74", "mh": "bciqdrpowjntwpk4csnpfh7kmgne732lv72zpwuytmhmcbfae3s4oh7y"}
Feb 28 12:30:48 ip-10-151-21-17.us-west-2.compute.internal k3s[11230]: 2026-02-28T12:30:48.996Z        DEBUG        dht        go-libp2p-kad-dht@v0.36.0/routing.go:579        0 provider entries
Feb 28 12:30:48 ip-10-151-21-17.us-west-2.compute.internal k3s[11230]: 2026-02-28T12:30:48.996Z        DEBUG        dht        go-libp2p-kad-dht@v0.36.0/routing.go:606        got closer peers: 0 []
Feb 28 12:30:48 ip-10-151-21-17.us-west-2.compute.internal k3s[11230]: 2026-02-28T12:30:48.996Z        DEBUG        dht        go-libp2p-kad-dht@v0.36.0/dht.go:712        peer found        {"peer": "QmNjkaUu3SzT5VARhuVX9CqzJ9mgLS6xLaNWMAWmCWDfbb"}
Feb 28 12:30:48 ip-10-151-21-17.us-west-2.compute.internal k3s[11230]: time="2026-02-28T12:30:48Z" level=error error="MANIFEST_UNKNOWN could not find peer for sha256:16974531848218d24822bf606be022d030ab8c9b05b2ecf11076c4c1c6885c95" handler=mirror ip=127.0.0.1 latency=1.338045ms logger=spegel method=GET path="/v2/rancher/mirrored-pause/manifests/sha256:16974531848218d24822bf606be022d030ab8c9b05b2ecf11076c4c1c6885c95" registry=docker.io status=404
Feb 28 12:30:49 ip-10-151-21-17.us-west-2.compute.internal k3s[11230]: {"cniVersion":"1.0.0","forceAddress":true,"hairpinMode":true,"ipMasq":false,"ipam":{"ranges":[[{"subnet":"10.42.1.0/24"}]],"routes":[{"dst":"10.42.0.0/16"}],"type":"host-local"},"isDefaultGateway":true,"isGateway":true,"mtu":8951,"name":"cbr0","type":"bridge"}2026-02-28T12:30:49.989Z        DEBUG        dht        go-libp2p-kad-dht@v0.36.0/routing.go:506        finding providers        {"cid": "bafkreig5lsn4dh3jstov5pvfqvqqjhjp2yq26zry5rk2maeqqxt5efsgee", "mh": "bciqn2xe3ygpwtfg5l27klblbasos7vrbv5tdr3cvuyajbbph2ilemii"}
Feb 28 12:47:25 ip-10-151-21-17.us-west-2.compute.internal k3s[11230]: time="2026-02-28T12:47:25.993666390Z" level=info msg="fetch failed after status: 404 Not Found" host="127.0.0.1:6443"
Feb 28 12:47:25 ip-10-151-21-17.us-west-2.compute.internal k3s[11230]: time="2026-02-28T12:47:25.995614907Z" level=info msg="stop pulling image docker.io/rancher/mirrored-pause:3.6: active requests=0, bytes read=0"

average-appointment-24470

02/28/2026, 1:01 PM

agent logs after did a

systemctl restart k3s

on server node

Copy code

Feb 28 12:47:39 ip-10-151-21-17.us-west-2.compute.internal k3s[11230]: time="2026-02-28T12:47:39Z" level=info handler=mirror ip=127.0.0.1 latency=2.782411ms logger=spegel method=GET path="/v2/rancher/mirrored-pause/blobs/sha256:6270bb605e12e581514ada5fd5b3216f727db55dc87d5889c790e4c760683fee" registry=<http://docker.io|docker.io> status=200
Feb 28 12:47:39 ip-10-151-21-17.us-west-2.compute.internal k3s[11230]: 2026-02-28T12:47:39.045Z        DEBUG        dht        go-libp2p-kad-dht@v0.36.0/routing.go:506        finding providers        {"cid": "bafkreicqdg4mkpnvmlar43h6cy2s2hhlwajwlacdyonqycr5op6iy7rblq", "mh": "bciqfagnyyu63kywbdzwp4frvfuooxmatmwaehq43bqfd2474rr7ccxa"}
Feb 28 12:47:39 ip-10-151-21-17.us-west-2.compute.internal k3s[11230]: 2026-02-28T12:47:39.046Z        DEBUG        dht        go-libp2p-kad-dht@v0.36.0/routing.go:579        1 provider entries
Feb 28 12:47:39 ip-10-151-21-17.us-west-2.compute.internal k3s[11230]: 2026-02-28T12:47:39.046Z        DEBUG        dht        go-libp2p-kad-dht@v0.36.0/routing.go:584        got provider: {QmNjkaUu3SzT5VARhuVX9CqzJ9mgLS6xLaNWMAWmCWDfbb: [/ip4/10.151.22.85/tcp/5000]}
Feb 28 12:47:39 ip-10-151-21-17.us-west-2.compute.internal k3s[11230]: 2026-02-28T12:47:39.046Z        DEBUG        dht        go-libp2p-kad-dht@v0.36.0/routing.go:586        using provider: {QmNjkaUu3SzT5VARhuVX9CqzJ9mgLS6xLaNWMAWmCWDfbb: [/ip4/10.151.22.85/tcp/5000]}
Feb 28 12:47:39 ip-10-151-21-17.us-west-2.compute.internal k3s[11230]: 2026-02-28T12:47:39.046Z        DEBUG        dht        go-libp2p-kad-dht@v0.36.0/routing.go:606        got closer peers: 0 []
Feb 28 12:47:39 ip-10-151-21-17.us-west-2.compute.internal k3s[11230]: 2026-02-28T12:47:39.046Z        DEBUG        dht        go-libp2p-kad-dht@v0.36.0/dht.go:712        peer found        {"peer": "QmNjkaUu3SzT5VARhuVX9CqzJ9mgLS6xLaNWMAWmCWDfbb"}
Feb 28 12:47:39 ip-10-151-21-17.us-west-2.compute.internal k3s[11230]: time="2026-02-28T12:47:39Z" level=info handler=mirror ip=127.0.0.1 latency=6.668203ms logger=spegel method=GET path="/v2/rancher/mirrored-pause/blobs/sha256:1021ef88c7974bfff89c5a0ec4fd3160daac6c48a075f74cff721f85dd104e68" registry=<http://docker.io|docker.io> status=200
Feb 28 12:47:39 ip-10-151-21-17.us-west-2.compute.internal k3s[11230]: time="2026-02-28T12:47:39.078891676Z" level=info msg="ImageCreate event name:\"<http://docker.io/rancher/mirrored-pause:3.6\|docker.io/rancher/mirrored-pause:3.6\>" labels:{key:\"io.cri-containerd.image\" value:\"managed\"}"
Feb 28 12:47:39 ip-10-151-21-17.us-west-2.compute.internal k3s[11230]: time="2026-02-28T12:47:39.080936387Z" level=info msg="stop pulling image <http://docker.io/rancher/mirrored-pause:3.6|docker.io/rancher/mirrored-pause:3.6>: active requests=0, bytes read=685844"
Feb 28 12:47:39 ip-10-151-21-17.us-west-2.compute.internal k3s[11230]: time="2026-02-28T12:47:39.083205225Z" level=info msg="ImageCreate event name:\"sha256:6270bb605e12e581514ada5fd5b3216f727db55dc87d5889c790e4c760683fee\" labels:{key:\"io.cri-containerd.image\" value:\"managed\"}"
Feb 28 12:47:39 ip-10-151-21-17.us-west-2.compute.internal k3s[11230]: time="2026-02-28T12:47:39.085894795Z" level=info msg="Pulled image \"rancher/mirrored-pause:3.6\" with image id \"sha256:6270bb605e12e581514ada5fd5b3216f727db55dc87d5889c790e4c760683fee\", repo tag \"<http://docker.io/rancher/mirrored-pause:3.6\|docker.io/rancher/mirrored-pause:3.6\>", repo digest \"<http://docker.io/rancher/mirrored-pause@sha256:16974531848218d24822bf606be022d030ab8c9b05b2ecf11076c4c1c6885c95\|docker.io/rancher/mirrored-pause@sha256:16974531848218d24822bf606be022d030ab8c9b05b2ecf11076c4c1c6885c95\>", size \"685844\" in 101.802548ms"
Feb 28 12:47:39 ip-10-151-21-17.us-west-2.compute.internal k3s[11230]: time="2026-02-28T12:47:39.085895445Z" level=info msg="ImageCreate event name:\"<http://docker.io/rancher/mirrored-pause@sha256:16974531848218d24822bf606be022d030ab8c9b05b2ecf11076c4c1c6885c95\|docker.io/rancher/mirrored-pause@sha256:16974531848218d24822bf606be022d030ab8c9b05b2ecf11076c4c1c6885c95\>" labels:{key:\"io.cri-containerd.image\" value:\"managed\"}"
Feb 28 12:47:39 ip-10-151-21-17.us-west-2.compute.internal k3s[11230]: time="2026-02-28T12:47:39Z" level=info msg="OCI event" logger=spegel ref="<http://docker.io/rancher/mirrored-pause@sha256:16974531848218d24822bf606be022d030ab8c9b05b2ecf11076c4c1c6885c95|docker.io/rancher/mirrored-pause@sha256:16974531848218d24822bf606be022d030ab8c9b05b2ecf11076c4c1c6885c95>" type=CREATE
Feb 28 12:47:39 ip-10-151-21-17.us-west-2.compute.internal k3s[11230]: time="2026-02-28T12:47:39Z" level=info msg="OCI event" logger=spegel ref="<http://docker.io/rancher/mirrored-pause@sha256:6270bb605e12e581514ada5fd5b3216f727db55dc87d5889c790e4c760683fee|docker.io/rancher/mirrored-pause@sha256:6270bb605e12e581514ada5fd5b3216f727db55dc87d5889c790e4c760683fee>" type=CREATE
Feb 28 12:47:39 ip-10-151-21-17.us-west-2.compute.internal k3s[11230]: time="2026-02-28T12:47:39Z" level=info msg="OCI event" logger=spegel ref="<http://docker.io/rancher/mirrored-pause@sha256:1021ef88c7974bfff89c5a0ec4fd3160daac6c48a075f74cff721f85dd104e68|docker.io/rancher/mirrored-pause@sha256:1021ef88c7974bfff89c5a0ec4fd3160daac6c48a075f74cff721f85dd104e68>" type=CREATE
Feb 28 12:47:39 ip-10-151-21-17.us-west-2.compute.internal k3s[11230]: time="2026-02-28T12:47:39Z" level=info msg="OCI event" logger=spegel ref="<http://docker.io/rancher/mirrored-pause:3.6|docker.io/rancher/mirrored-pause:3.6>" type=CREATE
Feb 28 12:47:39 ip-10-151-21-17.us-west-2.compute.internal k3s[11230]: 2026-02-28T12:47:39.107Z        DEBUG        dht        go-libp2p-kad-dht@v0.36.0/dht.go:712        peer found        {"peer": "QmNjkaUu3SzT5VARhuVX9CqzJ9mgLS6xLaNWMAWmCWDfbb"}
Feb 28 12:47:39 ip-10-151-21-17.us-west-2.compute.internal k3s[11230]: 2026-02-28T12:47:39.107Z        DEBUG        dht/netsize        netsize/netsize.go:258        Running garbage collection
Feb 28 12:47:39 ip-10-151-21-17.us-west-2.compute.internal k3s[11230]: time="2026-02-28T12:47:39.107640827Z" level=info msg="connecting to shim 2b74bc4234814333ac4bf76114b2b76db4d7a01a1507ee581126eeb1323ed7f5" address="unix:///run/containerd/s/65bf7e11a044e584cd8fe85dcfcadc9ec6200d01c06b73b1001b602ade8383b8" namespace=<http://k8s.io|k8s.io> protocol=ttrpc version=3

average-appointment-24470

03/02/2026, 8:06 PM

@creamy-pencil-82913 PTAL

creamy-pencil-82913

03/02/2026, 8:12 PM

please upload the full log somewhere. just putting a couple lines here in chat isn’t particularly helpful. Those logs do show it providing content from

10.151.22.85

though.

average-appointment-24470

03/02/2026, 9:06 PM

@creamy-pencil-82913 PTAL these are full logs from a fresh install + node join

Copy code

[root@ip-10-151-21-17 securiti-appliance-installer]# crictl pull docker.io/rancher/mirrored-pause:3.6
DEBU[0000] Asset dir /mnt/rancher/k3s/data/4a9584f4b72a328dc77a958636e91442257c1890aaa66f0508b3604bf676dbdd 
DEBU[0000] Running /mnt/rancher/k3s/data/4a9584f4b72a328dc77a958636e91442257c1890aaa66f0508b3604bf676dbdd/bin/crictl [crictl pull docker.io/rancher/mirrored-pause:3.6] 
E0302 21:25:39.789454   67571 log.go:32] "PullImage from image service failed" err="rpc error: code = NotFound desc = failed to pull and unpack image \"docker.io/rancher/mirrored-pause:3.6\": failed to copy: httpReadSeeker: failed open: content at <https://127.0.0.1:6443/v2/rancher/mirrored-pause/manifests/sha256:16974531848218d24822bf606be022d030ab8c9b05b2ecf11076c4c1c6885c95?ns=docker.io> not found: not found" image="docker.io/rancher/mirrored-pause:3.6"
FATA[0000] pulling image: rpc error: code = NotFound desc = failed to pull and unpack image "docker.io/rancher/mirrored-pause:3.6": failed to copy: httpReadSeeker: failed open: content at <https://127.0.0.1:6443/v2/rancher/mirrored-pause/manifests/sha256:16974531848218d24822bf606be022d030ab8c9b05b2ecf11076c4c1c6885c95?ns=docker.io> not found: not found

k3s-server-full.txt k3s-agent-full.txt

average-appointment-24470

03/03/2026, 6:03 PM

@creamy-pencil-82913 gentle reminder for the above

clever-house-68727

03/12/2026, 5:36 PM

@creamy-pencil-82913 gentle reminder ^^

➕ 1

creamy-pencil-82913

03/12/2026, 6:38 PM

https://github.com/spegel-org/spegel/issues/1159#issuecomment-3983152730

creamy-pencil-82913

03/12/2026, 6:40 PM

please track upstream in spegel. I don’t see anything to address in k3s.

average-appointment-24470

03/13/2026, 8:06 AM

As per Philip

average-appointment-24470

03/13/2026, 8:06 AM

We did change how images are imported (added labels & digests). Could that be the issue?

creamy-pencil-82913

03/13/2026, 9:29 AM

who is we

creamy-pencil-82913

03/13/2026, 9:30 AM

If you use the k3s images dir to auto-import tarballs, it adds the exact same labels and digests as the containerd CRI server does when images are pulled from a registry. You can see that with ctr.

creamy-pencil-82913

03/13/2026, 9:31 AM

see https://github.com/k3s-io/k3s/pull/13221

creamy-pencil-82913

03/13/2026, 9:39 AM

Are you doing something weird to try to preload content into the containerd image store? The first time K3s starts up, spegel says

Copy code

Mar 02 20:45:30 ip-10-151-22-85.us-west-2.compute.internal k3s[27498]: time="2026-03-02T20:45:30Z" level=error msg="skipping content that cant be converted to reference" error="no distribution source labels found for sha256:16974531848218d24822bf606be022d030ab8c9b05b2ecf11076c4c1c6885c95" logger=spegel

and then later when K3s actually imports the image it says:

Copy code

Mar 02 20:45:31 ip-10-151-22-85.us-west-2.compute.internal k3s[27498]: time="2026-03-02T20:45:31Z" level=info msg="Imported <http://docker.io/rancher/mirrored-pause:3.6|docker.io/rancher/mirrored-pause:3.6>"
Mar 02 20:45:31 ip-10-151-22-85.us-west-2.compute.internal k3s[27498]: time="2026-03-02T20:45:31.959001920Z" level=info msg="ImageUpdate event name:\"<http://docker.io/rancher/local-path-provisioner:v0.0.30\|docker.io/rancher/local-path-provisioner:v0.0.30\>" labels:{key:\"io.cattle.k3s.pinned\" value:\"pinned\"} labels:{key:\"io.cri-containerd.pinned\" value:\"pinned\"}"
Mar 02 20:45:31 ip-10-151-22-85.us-west-2.compute.internal k3s[27498]: time="2026-03-02T20:45:31Z" level=info msg="Tagged <http://docker.io/rancher/mirrored-pause@sha256:16974531848218d24822bf606be022d030ab8c9b05b2ecf11076c4c1c6885c95|docker.io/rancher/mirrored-pause@sha256:16974531848218d24822bf606be022d030ab8c9b05b2ecf11076c4c1c6885c95>"

so that sounds like somehow the content was already there before k3s started, but without any labels? how did it get there?

average-appointment-24470

03/13/2026, 9:47 AM

Copy code

If you use the k3s images dir to auto-import tarballs

Yes using that only, nothing additional on top. 'we' as in k3s.

Copy code

Are you doing something weird to try to preload content into the containerd image store?

Not that im aware of, I did a fresh install which meant deleting k3s and everything that comes with it

creamy-pencil-82913

03/13/2026, 9:50 AM

I don’t think you did a complete job of deleting k3s. Did you use the uninstall script?

creamy-pencil-82913

03/13/2026, 9:51 AM

There are a bunch of weird errors about things still existing

creamy-pencil-82913

03/13/2026, 9:51 AM

Copy code

Mar 02 20:45:33 ip-10-151-22-85.us-west-2.compute.internal k3s[27498]: time="2026-03-02T20:45:33Z" level=error msg="error when handling containerd event" error="image manifest has not been deleted: All attempts fail:\n#1: manifest with digest sha256:a359a4959a2a46cea0603f6c133c14b20a0270e28e2eb88920becce4f38db7d9 still exists\n#2: manifest with digest sha256:a359a4959a2a46cea0603f6c133c14b20a0270e28e2eb88920becce4f38db7d9 still exists\n#3: manifest with digest sha256:a359a4959a2a46cea0603f6c133c14b20a0270e28e2eb88920becce4f38db7d9 still exists\n#4: manifest with digest sha256:a359a4959a2a46cea0603f6c133c14b20a0270e28e2eb88920becce4f38db7d9 still exists\n#5: manifest with digest sha256:a359a4959a2a46cea0603f6c133c14b20a0270e28e2eb88920becce4f38db7d9 still exists\n#6: manifest with digest sha256:a359a4959a2a46cea0603f6c133c14b20a0270e28e2eb88920becce4f38db7d9 still exists\n#7: manifest with digest sha256:a359a4959a2a46cea0603f6c133c14b20a0270e28e2eb88920becce4f38db7d9 still exists\n#8: manifest with digest sha256:a359a4959a2a46cea0603f6c133c14b20a0270e28e2eb88920becce4f38db7d9 still exists\n#9: manifest with digest sha256:a359a4959a2a46cea0603f6c133c14b20a0270e28e2eb88920becce4f38db7d9 still exists\n#10: manifest with digest sha256:a359a4959a2a46cea0603f6c133c14b20a0270e28e2eb88920becce4f38db7d9 still exists" logger=spegel
Mar 02 20:45:33 ip-10-151-22-85.us-west-2.compute.internal k3s[27498]: I0302 20:45:33.982100   27498 controllermanager.go:781] "Started controller" controller="root-ca-certificate-publisher-controller"
Mar 02 20:45:33 ip-10-151-22-85.us-west-2.compute.internal k3s[27498]: I0302 20:45:33.982143   27498 publisher.go:107] "Starting root CA cert publisher controller"
Mar 02 20:45:33 ip-10-151-22-85.us-west-2.compute.internal k3s[27498]: I0302 20:45:33.982149   27498 shared_informer.go:349] "Waiting for caches to sync" controller="crt configmap"
Mar 02 20:45:34 ip-10-151-22-85.us-west-2.compute.internal k3s[27498]: I0302 20:45:34.080060   27498 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="089cc42a-2c63-47c8-9ea3-9cd331b0c6a4" path="/var/lib/kubelet/pods/089cc42a-2c63-47c8-9ea3-9cd331b0c6a4/volumes"
Mar 02 20:45:34 ip-10-151-22-85.us-west-2.compute.internal k3s[27498]: I0302 20:45:34.080396   27498 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="10cbf0d1-916e-407a-bcf1-bd242f1922c5" path="/var/lib/kubelet/pods/10cbf0d1-916e-407a-bcf1-bd242f1922c5/volumes"
Mar 02 20:45:34 ip-10-151-22-85.us-west-2.compute.internal k3s[27498]: I0302 20:45:34.080494   27498 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="1604fc8f-06a9-4a9b-90f1-6828037b27dd" path="/var/lib/kubelet/pods/1604fc8f-06a9-4a9b-90f1-6828037b27dd/volumes"
Mar 02 20:45:34 ip-10-151-22-85.us-west-2.compute.internal k3s[27498]: I0302 20:45:34.080702   27498 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="1a3562ba-2324-4cad-a2cf-06bb5159af83" path="/var/lib/kubelet/pods/1a3562ba-2324-4cad-a2cf-06bb5159af83/volumes"
Mar 02 20:45:34 ip-10-151-22-85.us-west-2.compute.internal k3s[27498]: I0302 20:45:34.080987   27498 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="268b4228-d0de-4a29-b920-839c8c1b2ba0" path="/var/lib/kubelet/pods/268b4228-d0de-4a29-b920-839c8c1b2ba0/volumes"
Mar 02 20:45:34 ip-10-151-22-85.us-west-2.compute.internal k3s[27498]: I0302 20:45:34.081179   27498 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="33a97a39-8ed1-412d-b462-889107465f39" path="/var/lib/kubelet/pods/33a97a39-8ed1-412d-b462-889107465f39/volumes"
Mar 02 20:45:34 ip-10-151-22-85.us-west-2.compute.internal k3s[27498]: I0302 20:45:34.081371   27498 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="33d3dac3-83b9-4fdb-bd24-75c2ef694da6" path="/var/lib/kubelet/pods/33d3dac3-83b9-4fdb-bd24-75c2ef694da6/volumes"
Mar 02 20:45:34 ip-10-151-22-85.us-west-2.compute.internal k3s[27498]: I0302 20:45:34.081802   27498 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="3556b6cd-6e13-4011-854a-7010f9686fb6" path="/var/lib/kubelet/pods/3556b6cd-6e13-4011-854a-7010f9686fb6/volumes"
Mar 02 20:45:34 ip-10-151-22-85.us-west-2.compute.internal k3s[27498]: I0302 20:45:34.082032   27498 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="3564c8dd-351e-4d70-9d8c-7777dfdf59c4" path="/var/lib/kubelet/pods/3564c8dd-351e-4d70-9d8c-7777dfdf59c4/volumes"
Mar 02 20:45:34 ip-10-151-22-85.us-west-2.compute.internal k3s[27498]: I0302 20:45:34.082189   27498 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="368a591a-7178-4d6b-b4f4-5d65fe0899b2" path="/var/lib/kubelet/pods/368a591a-7178-4d6b-b4f4-5d65fe0899b2/volumes"
Mar 02 20:45:34 ip-10-151-22-85.us-west-2.compute.internal k3s[27498]: I0302 20:45:34.082364   27498 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="36b4d916-4650-4075-864b-ea6c897f99b8" path="/var/lib/kubelet/pods/36b4d916-4650-4075-864b-ea6c897f99b8/volumes"
Mar 02 20:45:34 ip-10-151-22-85.us-west-2.compute.internal k3s[27498]: I0302 20:45:34.082565   27498 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="3be3e987-c214-48c6-8a6b-984f84638d0c" path="/var/lib/kubelet/pods/3be3e987-c214-48c6-8a6b-984f84638d0c/volumes"

and so on

creamy-pencil-82913

03/13/2026, 9:52 AM

Copy code

Mar 02 20:45:34 ip-10-151-22-85.us-west-2.compute.internal k3s[27498]: E0302 20:45:34.086893   27498 kubelet_volumes.go:263] "There were many similar errors. Turn up verbosity to see them." err="orphaned pod \"10cbf0d1-916e-407a-bcf1-bd242f1922c5\" found, but failed to rmdir() subpath at path /var/lib/kubelet/pods/10cbf0d1-916e-407a-bcf1-bd242f1922c5/volume-subpaths/shared-data/jdbc-proxy/1: remove /var/lib/kubelet/pods/10cbf0d1-916e-407a-bcf1-bd242f1922c5/volume-subpaths/shared-data/jdbc-proxy/1: device or resource busy" numErrs=2

creamy-pencil-82913

03/13/2026, 9:52 AM

looks like you had stuff left over from a previous install

average-appointment-24470

03/13/2026, 9:52 AM

k3s-uninstall.sh rm -rf /mnt/rancher (custom datadir) rm -rf /etc/rancher Im following these steps

average-appointment-24470

03/13/2026, 9:54 AM

Also once the k3s adds labels etc to that image. shouldn't spegel automatically pick it up?

creamy-pencil-82913

03/13/2026, 9:56 AM

yes, the logs show that

creamy-pencil-82913

03/13/2026, 9:58 AM

there are a bunch of weird logs that show spegel getting OCI events from content being loaded into containerd, but then complaining that the content is not found when it tries to actually read it from containerd

👀 1

creamy-pencil-82913

03/13/2026, 10:00 AM

you might turn up the containerd log level and see if there is anything interesting in there

average-appointment-24470

03/13/2026, 10:04 AM

Copy code

# /mnt rancher/k3s/agent/etc/containerd/config.toml.tmpl                                                                                                         
  [debug]                                                                                                                                                              
    level = "debug"

This needs to be added ?

creamy-pencil-82913

03/13/2026, 10:04 AM

no. it is an env var

creamy-pencil-82913

03/13/2026, 10:05 AM

CONTAINERD_LOG_LEVEL=debug or CONTAINERD_LOG_LEVEL=trace in the k3s service env

✅ 1

average-appointment-24470

03/13/2026, 10:07 AM

Somehow restarting k3s fixes everything so ill have to fresh install and repro the issue. it'll take some time, thanks

average-appointment-24470

03/13/2026, 10:59 AM

with debug on for both k3s and containerd

new-k3s-agent.txt new-server-full.txt

average-appointment-24470

03/13/2026, 11:05 AM

containerd.log

average-appointment-24470

03/13/2026, 11:09 AM

containerd-agent.log

average-appointment-24470

03/13/2026, 11:33 AM

server

Copy code

[root@ip-10-151-22-85 tmp]#   curl -sk <https://127.0.0.1:6443/v2/rancher/mirrored-pause/manifests/sha256:16974531848218d24822bf606be022d030ab8c9b05b2ecf11076c4c1c6885c95?ns=docker.io>       
{"schemaVersion":2,"mediaType":"application/vnd.oci.image.manifest.v1+json","config":{"mediaType":"application/vnd.oci.image.config.v1+json","digest":"sha256:6270bb605e12e581514ada5fd5b3216f727db55dc87d5889c790e4c760683fee","size":901},"layers":[{"mediaType":"application/vnd.oci.image.layer.v1.tar","digest":"sha256:1021ef88c7974bfff89c5a0ec4fd3160daac6c48a075f74cff721f85dd104e68","size":684544}]}[root@ip-10-151-22-85 tmp]#

agent

Copy code

[root@ip-10-151-21-17 ingest]#  curl -sk -I <https://127.0.0.1:6443/v2/rancher/mirrored-pause/manifests/3.6?ns=docker.io>  
HTTP/2 200 
cache-control: no-cache, private
content-type: application/vnd.oci.image.manifest.v1+json
docker-content-digest: sha256:16974531848218d24822bf606be022d030ab8c9b05b2ecf11076c4c1c6885c95
content-length: 399
date: Fri, 13 Mar 2026 11:29:24 GMT

[root@ip-10-151-21-17 ingest]#  curl -sk <https://127.0.0.1:6443/v2/rancher/mirrored-pause/manifests/sha256:16974531848218d24822bf606be022d030ab8c9b05b2ecf11076c4c1c6885c95?ns=docker.io>
{"errors":[{"code":"MANIFEST_UNKNOWN","detail":{"attempts":0},"message":"could not find peer for sha256:16974531848218d24822bf606be022d030ab8c9b05b2ecf11076c4c1c6885c95"}]}[root@ip-10-151-21-17 ingest]#

creamy-pencil-82913

03/13/2026, 4:04 PM

ok so it has it and will serve it locally but other node can’t find it over p2p

creamy-pencil-82913

03/13/2026, 4:04 PM

please report upstream

average-appointment-24470

03/13/2026, 4:29 PM

https://github.com/spegel-org/spegel/issues/1159 Is this tracking the same? Were you able to find something in the containerd logs? Not sure if i mentioned, 1.33.5 worked fine without issues

42 Views

Open in Slack

Previous Next