https://rancher.com/ logo
Title
c

creamy-waiter-66684

09/29/2022, 6:16 PM
Hello team! I'm investigating k3s as a potential solution for storing a large amount of configuration information for various service teams in my organization. We're considering using Custom Resource Definitions (CRDs) in the API server. K8s was not viable due to the 8GB limit etcd imposes. While the external database feature of k3s solves the size limit, I noticed while inserting a large number of very large Customer Resources (CRs), the k3s server begins to eat a large amount of the systems RAM. I'm inserting around 1 million of these large CRs on an EC2 instance with over 128GB of RAM. At around 140,000 objects inserted, the k3s server is eating over 70% of the systems RAM. Is this intended? I assume k3s server is caching the inserted objects? Is there anyway to alleviate the large amount of memory usage? Even after stopping the insertions, the RAM utilization still sits around 60 or 70 percent (goes does 20-30% from peak).
c

creamy-pencil-82913

09/29/2022, 9:34 PM
I think you might be exceeding the design goals of both etcd and kubernetes. If you’ve got so much data that etd itself is refusing to store it, simply switching the storage backend to kine+sql probably isn’t going to be a material improvement.
Sounds like perhaps some other rbac-enabled kv store or document store might be a better fit for your use case?
c

creamy-waiter-66684

09/30/2022, 5:30 PM
Hmmm
We wanted to take advantage of some of the benefits of the API server (provided API, version conversion of CRs, etc).
Is this amount of memory usage by k3s expected though?
Does k3s cache these objects as they are inserted?
c

creamy-pencil-82913

09/30/2022, 6:01 PM
not K3s specifically no, but Kubernetes does do a lot of caching yes.
You might try putting some memory limits on the k3s systemd unit to encourage it to be a little more conservative but at the end of the day you’re just kinda using it in a way it wasn’t really designed for.
It’s not meant to be a document/blob store. It’s an orchestration platform.
You might look at https://github.com/kcp-dev/kcp if all you want is CRDs without the orchestration bit?
it’s experimental though