Hi I m building a brand new k3s cluster Currently I have it rancher-users #k3s

Title

handsome-toddler-59547

06/10/2022, 9:35 AM

Hi. I'm building a brand new k3s cluster. Currently I have it up and running along with Ceph. I will be installing a variety of opensource services (e.g. NGINX). I'll also be setting up custom built services. In a past K3S and Vanilla K8S clusters I also instantiated a private docker registry within the cluster to serve custom images for those custom services. I know in recent releases k8s has stepped away from the docker toolsuite and is solidifying around containerd, and CRI-O. Since I have flexibility to build the exact cluster I want, I am looking for advice. Questions I have? Should I use containerd, or CRI-O? Does k3s support both? What tools (other than docker) should I consider for building compliant container images? Rather than deploy a private docker registry in my k3s cluster is there an alternative recommended registry that I should consider using?

As an aside, if I need to rip down my current k3s server and stand it back up, I am happy to do that. I've been focusing on getting this cluster "right" with the latest and greatest tech stack that aligns with the way organizations are building clusters of the future.

Also know that I will run services like kafka (strimzi), mongo (via its operator), and many more within this cluster. I'm looking for a cohesive design top to bottom, and bottom to top.

For ceph, I am currently using Rook.

So I'm keen to roll in my private registry, and the ability to deploy custom images, with appropriate architectural decisions.

My current instance of k3s uses K8S 1.23.

[root@master ~]# k version
Client Version: <http://version.Info|version.Info>{Major:"1", Minor:"23", GitVersion:"v1.23.6+k3s1", GitCommit:"418c3fa858b69b12b9cefbcff0526f666a6236b9", GitTreeState:"clean", BuildDate:"2022-04-28T22:16:18Z", GoVersion:"go1.17.5", Compiler:"gc", Platform:"linux/amd64"}
Server Version: <http://version.Info|version.Info>{Major:"1", Minor:"23", GitVersion:"v1.23.6+k3s1", GitCommit:"418c3fa858b69b12b9cefbcff0526f666a6236b9", GitTreeState:"clean", BuildDate:"2022-04-28T22:16:18Z", GoVersion:"go1.17.5", Compiler:"gc", Platform:"linux/amd64"}
[root@master ~]# ps -aef | grep containerd
root         830     809  0 05:09 ?        00:00:03 containerd -c /var/lib/rancher/k3s/agent/etc/containerd/config.toml -a /run/k3s/containerd/containerd.sock --state /run/k3s/containerd --root /var/lib/rancher/k3s/agent/containerd
root        1635       1  0 05:10 ?        00:00:00 /var/lib/rancher/k3s/data/8c2b0191f6e36ec6f3cb68e2302fcc4be850c6db31ec5f8a74e4b3be403101d8/bin/containerd-shim-runc-v2 -namespace <http://k8s.io|k8s.io> -id a5d8946a726fcdcbf8a64a70be5590eec1f0af118c2198aea7d212c46186282e -address /run/k3s/containerd/containerd.sock
root        1638       1  0 05:10 ?        00:00:00 /var/lib/rancher/k3s/data/8c2b0191f6e36ec6f3cb68e2302fcc4be850c6db31ec5f8a74e4b3be403101d8/bin/containerd-shim-runc-v2 -namespace <http://k8s.io|k8s.io> -id f8188c13fccc568ca7187eac4ec3f5bb6331f5382a8198991d6ff75d79b6c8d7 -address /run/k3s/containerd/containerd.sock
root        1643       1  0 05:10 ?        00:00:00 /var/lib/rancher/k3s/data/8c2b0191f6e36ec6f3cb68e2302fcc4be850c6db31ec5f8a74e4b3be403101d8/bin/containerd-shim-runc-v2 -namespace <http://k8s.io|k8s.io> -id 2e978082556affe6fb140f434ec78101af9c2b3a3ad3046b5dd9c52df9b89f07 -address /run/k3s/containerd/containerd.sock
root        2048       1  0 05:10 ?        00:00:00 /var/lib/rancher/k3s/data/8c2b0191f6e36ec6f3cb68e2302fcc4be850c6db31ec5f8a74e4b3be403101d8/bin/containerd-shim-runc-v2 -namespace <http://k8s.io|k8s.io> -id 566c4edd126b06a79e9c620f8e85df2eb57626205312d687b3e9ded8b2aaed4c -address /run/k3s/containerd/containerd.sock
root        2071       1  0 05:10 ?        00:00:00 /var/lib/rancher/k3s/data/8c2b0191f6e36ec6f3cb68e2302fcc4be850c6db31ec5f8a74e4b3be403101d8/bin/containerd-shim-runc-v2 -namespace <http://k8s.io|k8s.io> -id 9f7654696892c5c94f00b4bff7b63190f7b83e198a471e26536585f10031dd88 -address /run/k3s/containerd/containerd.sock
root        2266       1  0 05:10 ?        00:00:00 /var/lib/rancher/k3s/data/8c2b0191f6e36ec6f3cb68e2302fcc4be850c6db31ec5f8a74e4b3be403101d8/bin/containerd-shim-runc-v2 -namespace <http://k8s.io|k8s.io> -id 3899836b870aaa914b39a9b2dca79c728303b62e3a77e7e6392ead797c39e052 -address /run/k3s/containerd/containerd.sock
root        4696    4204  0 05:40 pts/0    00:00:00 grep --color=auto containerd

So these are my basic questions for now with all that lead in: 1. Should I use containerd, or CRI-O? 2. Does k3s support both? 3. What tools (other than docker) should I consider for building compliant container images? 4. Rather than deploy a private docker registry in my k3s cluster is there an alternative recommended registry (e.g. artifactory, nexus, etc.) that I should consider using?

careful-piano-35019

06/10/2022, 12:18 PM

K3s comes with its own containerd

nutritious-tomato-14686

06/10/2022, 6:05 PM

For #3, you could looking into nerdctl for building containers with containerd instead of docker, or (shameless plug) look into rancher desktop which bundles nerdctl to build and test container images. But docker remains a solid choice.

#4, k3s still has no private registry (there has been some internal discussion on this "lack" of a feature, but nothing concrete has been decided), I'm not sure if other users have a preferred registry, docker registry should continue to work just fine.

For #1 and #2, just use the built in containerd that k3s comes with. You technically can rip it out and use your own container runtime using the

--container-runtime-endpoint

, but that's a huge pain and breaks other default components like flannel.

handsome-toddler-59547

06/10/2022, 7:46 PM

good feedback guys.. ty for guidance

3 Views

#k3s Join Slack