https://rancher.com/ logo
#k3s
Title
# k3s
a

acceptable-king-56965

01/13/2023, 5:17 PM
Hi, I have two questions to ask: 1. If I just need kube-apiserver (to create CRDs), not other K8s functionalities (e.g. Pods, Services, etc.), could I customize K3s to start only kube-apiserver (no controller-manager, scheduler, etc.)? 2. Which control plane datastore choice can provider better performance and scale for a large number of CRDs (e.g. several millions of CRs)?
c

creamy-pencil-82913

01/13/2023, 5:20 PM
1. yes 2. probably etcd, but with several million resources you are probably going to have some issues no matter what. the kubernetes datastore is NOT intended to be used as a generic document store.
a

acceptable-king-56965

01/13/2023, 5:22 PM
Thanks for the reply! I thought sqlite or postgres can scale better than etcd? In my tests, I saw performance issues with ~500K CRs with etcd, but we also suspect that can be due to how K8s configures etcd and we may be able to tune to improve.
c

creamy-pencil-82913

01/13/2023, 5:22 PM
posgres is in my experience the poorest performing backend for kine
sqlite or mysql are better
but either way I think you might be using the wrong tool for the job
🎯 1
a

acceptable-king-56965

01/13/2023, 5:27 PM
We saw etcd data compaction can take long with large number of CRs created. I think K8s requests etcd compaction every 5 mins.
c

creamy-pencil-82913

01/13/2023, 5:27 PM
If you REALLY want to try, you should probably plan on hosting multiple datastores and use the apiserver
etcd-servers-overrides
to distribute your resources across them.
the compaction overhead will depend on the rate at which you are creating/deleting/updating resources. not on the number of resources you have.
1
a

acceptable-king-56965

01/13/2023, 5:29 PM
Got your point. In our case, we originally develop our services only for K8s and then naturally used CRD as API/storage. But now we are extending the services for other use cases, but want not to make big code changes, and so looking at if we can get a performant apiserver. We are also considering implementing our own apiserver with the K8s apiserver lib if that can scale higher.
c

creamy-pencil-82913

01/13/2023, 5:29 PM
many of the core Kubernetes authors are working on that
a

acceptable-king-56965

01/13/2023, 5:30 PM
Good to know! I feel I learnt a lot from you. Thanks for all the advices!
l

late-needle-80860

01/14/2023, 6:16 PM
@creamy-pencil-82913 do you know how well compatible K3s is with the KCP project? Seems you can of course use K3s as the main physical cluster as they call it in the KCP project … and the go from there … but is there something in the project in regards to K3s compatibility one should think about - as far as you know?
c

creamy-pencil-82913

01/14/2023, 9:25 PM
they have their own apiserver equivalent I believe? that might be a better fit for what you’re trying to do than running k3s with all the bundled stuff disabled.
l

late-needle-80860

01/15/2023, 5:31 PM
I still love what I get with k3s even while having to disable some bundles stuff.
10 Views