https://rancher.com/ logo
Title
m

melodic-hamburger-23329

09/30/2022, 7:34 AM
Containerd still fails to boot after manual k3s upgrade… Any way to avoid this happening?
time="2022-09-30T16:30:47.729342645+09:00" level=fatal msg="failed to create new snapshotter" error="failed to restore remote snapshot: failed to prepare remote snapshot: sha256:08b10ee4e4d584086d7203095776335fc5f3a541402bb19e89e908096b30df2e: failed to resolve layer: failed to resolve layer \"sha256:a42e3d1ba15a55b32c4b95cd3486aab3103d7b685b471ce68130d718c16b4e88\" from \"...\": failed to resolve the blob: failed to resolve the source: cannot resolve layer: failed to redirect (host \"...\", ref:\"...\", digest:\"sha256:a42e3d1ba15a55b32c4b95cd3486aab3103d7b685b471ce68130d718c16b4e88\"): failed to access to the registry with code 404: failed to resolve: failed to resolve target"
Kind of impossible to do upgrades if need to basically recreate cluster every time :S Am I possibly doing something wrong? Steps: • download latest k3s binary and put to /usr/local/bin • systemctl stop k3s (or k3s-killall.sh; not sure which one recommended?) • systemctl start k3s (or rerun install script; same result)
hm, rerun with updated install script and it seems it was able to recover the containers. Not sure if using old install script was the culprit
c

creamy-pencil-82913

09/30/2022, 7:49 AM
I have literally never seen that error from containerd. It’s trying to access a registry while creating the snapshotter during startup?
and it got a 404 from it?
m

melodic-hamburger-23329

09/30/2022, 7:50 AM
I’m using internal registry and basically block access to dockerhub, etc. Maybe the mirror configuration somehow gets messed up during upgrade
c

creamy-pencil-82913

09/30/2022, 7:51 AM
are you also using one of the non-standard snapshotters? btrfs or stargz or something?
I would probably go create an issue on the containerd GH repo if you see it again.
m

melodic-hamburger-23329

09/30/2022, 7:51 AM
stargz yes
c

creamy-pencil-82913

09/30/2022, 7:52 AM
yeah, probably a bug in that then.
m

melodic-hamburger-23329

09/30/2022, 7:52 AM
hmm
c

creamy-pencil-82913

09/30/2022, 7:52 AM
I would grab the full containerd logs and create an issue
I don’t think hardly anyone uses stargz
so they’d probably appreciate the report
☝️ 1
b

bulky-sunset-52084

10/01/2022, 2:36 AM
Also if it makes your life easier (kinda unrelated) rancher has a tool for upgrading k3s clusters in an automated fashion: https://github.com/rancher/system-upgrade-controller
m

melodic-hamburger-23329

10/02/2022, 11:21 PM
@bulky-sunset-52084 Yes, I tried it but couldn’t find working configuration