Containerd still fails to boot after manual k3s upgrade… Any way to avoid this happening?

time="2022-09-30T16:30:47.729342645+09:00" level=fatal msg="failed to create new snapshotter" error="failed to restore remote snapshot: failed to prepare remote snapshot: sha256:08b10ee4e4d584086d7203095776335fc5f3a541402bb19e89e908096b30df2e: failed to resolve layer: failed to resolve layer \"sha256:a42e3d1ba15a55b32c4b95cd3486aab3103d7b685b471ce68130d718c16b4e88\" from \"...\": failed to resolve the blob: failed to resolve the source: cannot resolve layer: failed to redirect (host \"...\", ref:\"...\", digest:\"sha256:a42e3d1ba15a55b32c4b95cd3486aab3103d7b685b471ce68130d718c16b4e88\"): failed to access to the registry with code 404: failed to resolve: failed to resolve target"

Kind of impossible to do upgrades if need to basically recreate cluster every time :S Am I possibly doing something wrong? Steps: • download latest k3s binary and put to /usr/local/bin • systemctl stop k3s (or k3s-killall.sh; not sure which one recommended?) • systemctl start k3s (or rerun install script; same result)

I have literally never seen that error from containerd. It’s trying to access a registry while creating the snapshotter during startup?

I’m using internal registry and basically block access to dockerhub, etc. Maybe the mirror configuration somehow gets messed up during upgrade

are you also using one of the non-standard snapshotters? btrfs or stargz or something?

stargz yes

yeah, probably a bug in that then.

hmm

I would grab the full containerd logs and create an issue

Also if it makes your life easier (kinda unrelated) rancher has a tool for upgrading k3s clusters in an automated fashion: https://github.com/rancher/system-upgrade-controller

@bulky-sunset-52084 Yes, I tried it but couldn’t find working configuration