steep-continent-12582
08/01/2022, 5:40 AMoverlay
snapshotter, understandably seems to not work here
• Trying the native
snapshotter, some containers throwing errors like:
Failed to create pod sandbox: rpc error: code = Unknown desc = failed to create containerd container: copying of parent failed: failed to copy file info for /var/lib/rancher/k3s/agent/containerd/io.containerd.snapshotter.v1.native/snapshots/new-4170709921: failed to chown /var/lib/rancher/k3s/agent/containerd/io.containerd.snapshotter.v1.native/snapshots/new-4170709921: lchown /var/lib/rancher/k3s/agent/containerd/io.containerd.snapshotter.v1.native/snapshots/new-4170709921: invalid argument
• Other times (and I haven't figured out the pattern here, I thought it was under overlay config but seems not), I get this kind of similar but different error, where the native snapshotter is not mentioned, seems this one relates to the pull/unpack rather than the starting:
Failed to pull image "<http://quay.io/prometheus/busybox|quay.io/prometheus/busybox>": rpc error: code = Unknown desc = failed to pull and unpack image "<http://quay.io/prometheus/busybox:latest|quay.io/prometheus/busybox:latest>": failed to extract layer sha256:c7412c2a678786efc84fe4bd0b1a79ecca47457b0eb4d4bceb1f79d6b4f75695: mount callback failed on /var/lib/rancher/k3s/agent/containerd/tmpmounts/containerd-mount4066054735: failed to Lchown "/var/lib/rancher/k3s/agent/containerd/tmpmounts/containerd-mount4066054735/bin" for UID 0, GID 0: lchown /var/lib/rancher/k3s/agent/containerd/tmpmounts/containerd-mount4066054735/bin: invalid argument: unknown
• For both of the above, my GitHub searches seemed to indicate they're caused by not running on an ext4 filesystem, which is true in my situation.
• In some cases I can't start any containers in a pod, in others I can start most except busybox
• I recalled that on docker, I believe I was using devmapper
which seemed to work fine and did not have this problem, but as far as I can tell, devmapper
is either not included at all in the k3s version of containerd, or just not configured (some places say it's not included, but I see errors in the containerd.log and ctr plugin ls
output that kind of indicate it's there but not working. I'm also not 100% sure if it would fix my problem.
So, my questions (and thanks in advance) are:
• How come only the busybox image so far has this particular problem? I've gotten a bunch of others to work so far.
• Is there some tweak to the native
snapshotter that can be done to allow things to work?
• If not, is there a simply way to get devmapper
working?creamy-pencil-82913
08/01/2022, 6:30 AMsteep-continent-12582
08/01/2022, 6:51 AMearly-sundown-21192
08/03/2022, 11:23 PMsteep-continent-12582
08/03/2022, 11:25 PM/var/lib/k3s/agent/containerd
and leave the rest on NFS, rather than try to figure out iSCSI boot at this time
I have it working on one node since last night and so far, it seems to work well, just need to repeat for the others 🙂