Hi, I have two questions to ask: 1. If I just need kube-apiserver (to create CRDs), not other K8s functionalities (e.g. Pods, Services, etc.), could I customize K3s to start only kube-apiserver (no controller-manager, scheduler, etc.)? 2. Which control plane datastore choice can provider better performance and scale for a large number of CRDs (e.g. several millions of CRs)?

1. yes 2. probably etcd, but with several million resources you are probably going to have some issues no matter what. the kubernetes datastore is NOT intended to be used as a generic document store.

Thanks for the reply! I thought sqlite or postgres can scale better than etcd? In my tests, I saw performance issues with ~500K CRs with etcd, but we also suspect that can be due to how K8s configures etcd and we may be able to tune to improve.

posgres is in my experience the poorest performing backend for kine

We saw etcd data compaction can take long with large number of CRs created. I think K8s requests etcd compaction every 5 mins.

If you REALLY want to try, you should probably plan on hosting multiple datastores and use the apiserver

etcd-servers-overrides

to distribute your resources across them.

Got your point. In our case, we originally develop our services only for K8s and then naturally used CRD as API/storage. But now we are extending the services for other use cases, but want not to make big code changes, and so looking at if we can get a performant apiserver. We are also considering implementing our own apiserver with the K8s apiserver lib if that can scale higher.

you might look at https://github.com/kcp-dev/kcp

Good to know! I feel I learnt a lot from you. Thanks for all the advices!

@creamy-pencil-82913 do you know how well compatible K3s is with the KCP project? Seems you can of course use K3s as the main physical cluster as they call it in the KCP project … and the go from there … but is there something in the project in regards to K3s compatibility one should think about - as far as you know?

they have their own apiserver equivalent I believe? that might be a better fit for what you’re trying to do than running k3s with all the bundled stuff disabled.

I still love what I get with k3s even while having to disable some bundles stuff.