Also know that I will run services like kafka (strimzi), mongo (via its operator), and many more within this cluster. I'm looking for a cohesive design top to bottom, and bottom to top.

For ceph, I am currently using Rook.

So I'm keen to roll in my private registry, and the ability to deploy custom images, with appropriate architectural decisions.

My current instance of k3s uses K8S 1.23.

[root@master ~]# k version
Client Version: <http://version.Info|version.Info>{Major:"1", Minor:"23", GitVersion:"v1.23.6+k3s1", GitCommit:"418c3fa858b69b12b9cefbcff0526f666a6236b9", GitTreeState:"clean", BuildDate:"2022-04-28T22:16:18Z", GoVersion:"go1.17.5", Compiler:"gc", Platform:"linux/amd64"}
Server Version: <http://version.Info|version.Info>{Major:"1", Minor:"23", GitVersion:"v1.23.6+k3s1", GitCommit:"418c3fa858b69b12b9cefbcff0526f666a6236b9", GitTreeState:"clean", BuildDate:"2022-04-28T22:16:18Z", GoVersion:"go1.17.5", Compiler:"gc", Platform:"linux/amd64"}
[root@master ~]# ps -aef | grep containerd
root         830     809  0 05:09 ?        00:00:03 containerd -c /var/lib/rancher/k3s/agent/etc/containerd/config.toml -a /run/k3s/containerd/containerd.sock --state /run/k3s/containerd --root /var/lib/rancher/k3s/agent/containerd
root        1635       1  0 05:10 ?        00:00:00 /var/lib/rancher/k3s/data/8c2b0191f6e36ec6f3cb68e2302fcc4be850c6db31ec5f8a74e4b3be403101d8/bin/containerd-shim-runc-v2 -namespace <http://k8s.io|k8s.io> -id a5d8946a726fcdcbf8a64a70be5590eec1f0af118c2198aea7d212c46186282e -address /run/k3s/containerd/containerd.sock
root        1638       1  0 05:10 ?        00:00:00 /var/lib/rancher/k3s/data/8c2b0191f6e36ec6f3cb68e2302fcc4be850c6db31ec5f8a74e4b3be403101d8/bin/containerd-shim-runc-v2 -namespace <http://k8s.io|k8s.io> -id f8188c13fccc568ca7187eac4ec3f5bb6331f5382a8198991d6ff75d79b6c8d7 -address /run/k3s/containerd/containerd.sock
root        1643       1  0 05:10 ?        00:00:00 /var/lib/rancher/k3s/data/8c2b0191f6e36ec6f3cb68e2302fcc4be850c6db31ec5f8a74e4b3be403101d8/bin/containerd-shim-runc-v2 -namespace <http://k8s.io|k8s.io> -id 2e978082556affe6fb140f434ec78101af9c2b3a3ad3046b5dd9c52df9b89f07 -address /run/k3s/containerd/containerd.sock
root        2048       1  0 05:10 ?        00:00:00 /var/lib/rancher/k3s/data/8c2b0191f6e36ec6f3cb68e2302fcc4be850c6db31ec5f8a74e4b3be403101d8/bin/containerd-shim-runc-v2 -namespace <http://k8s.io|k8s.io> -id 566c4edd126b06a79e9c620f8e85df2eb57626205312d687b3e9ded8b2aaed4c -address /run/k3s/containerd/containerd.sock
root        2071       1  0 05:10 ?        00:00:00 /var/lib/rancher/k3s/data/8c2b0191f6e36ec6f3cb68e2302fcc4be850c6db31ec5f8a74e4b3be403101d8/bin/containerd-shim-runc-v2 -namespace <http://k8s.io|k8s.io> -id 9f7654696892c5c94f00b4bff7b63190f7b83e198a471e26536585f10031dd88 -address /run/k3s/containerd/containerd.sock
root        2266       1  0 05:10 ?        00:00:00 /var/lib/rancher/k3s/data/8c2b0191f6e36ec6f3cb68e2302fcc4be850c6db31ec5f8a74e4b3be403101d8/bin/containerd-shim-runc-v2 -namespace <http://k8s.io|k8s.io> -id 3899836b870aaa914b39a9b2dca79c728303b62e3a77e7e6392ead797c39e052 -address /run/k3s/containerd/containerd.sock
root        4696    4204  0 05:40 pts/0    00:00:00 grep --color=auto containerd

So these are my basic questions for now with all that lead in: 1. Should I use containerd, or CRI-O? 2. Does k3s support both? 3. What tools (other than docker) should I consider for building compliant container images? 4. Rather than deploy a private docker registry in my k3s cluster is there an alternative recommended registry (e.g. artifactory, nexus, etc.) that I should consider using?

#4, k3s still has no private registry (there has been some internal discussion on this "lack" of a feature, but nothing concrete has been decided), I'm not sure if other users have a preferred registry, docker registry should continue to work just fine.

For #1 and #2, just use the built in containerd that k3s comes with. You technically can rip it out and use your own container runtime using the

--container-runtime-endpoint

, but that's a huge pain and breaks other default components like flannel.