https://rancher.com/ logo
Title
h

handsome-toddler-59547

06/16/2022, 9:48 PM
Simple question for the k3s experts. I've been using k3s for about a half year now. Prior to that I had setup my prior kubernetes clusters using "vanilla kubernetes".. all my efforts so far have been non-production. I know k3s is designed to be lightweight. Does this mean that it is potentially not a good platform for "load heavy" production platforms? How would you characterize when to choose an alternative k8s environment? I'll be driving towards making production decisions in the future. I like k3s very much. But I need to keep my eyes open to all system architectural and performance concerns. Any thoughts would be appreciated.
c

creamy-pencil-82913

06/16/2022, 9:58 PM
it should scale up fine. It is definitely suitable for production use. The places you might be limited are when you’re dealing with very large clusters that, if using upstream Kubernetes, you might want dedicated hosts to run just specific control-plane components due to the cpu/memory requirements. If you don’t think you’ll ever get that large, you should be fine.
You should decide ahead of time if you’re going to go with kine+sqlite, kine+external sql datastore, or etcd.
h

handsome-toddler-59547

06/16/2022, 10:00 PM
I see.. how large would you say is "very large"
in terms of the decision you highlight.. kine, sqlite.. etc.. i'm not sure i understand how to judge whats best. Do you have a recommendation relative to how to think of that choice
c

creamy-pencil-82913

06/16/2022, 10:03 PM
you also need to know whether your workload is going to be fairly static, and just sit there running… or if there’s going to be a lot of churn with pods coming and going and such. Will you deploy an app that makes heavy use of Kubernetes resources and is going to create/update/destroy things all the time and create a bunch of work for the apiserver and datastore, or are you just using Kubernetes to orchestrate a more traditional app that won’t touch Kubernete itself?
Its really hard to give abstract performance guidance. You need to know what you’re going to do with the cluster in order to size the various components properly.
h

handsome-toddler-59547

06/16/2022, 10:05 PM
no doubt. i recognized that as an unfair question. yet i've architected a variety of systems and there are bullets that can be highlighted relative to how one might think about the sizing. anyways I fully appreciate your point and will certainly be digging further as I drive towards such decisions
i understand the points you made about the static vs dynamic nature of the services
my assumption is that our cluster will grow over time.. as services are added.. it will scale up fairly significantly (at least from my perspective).. but i see 10s of worker nodes.. not 100s and certainly not 1000s
c

creamy-pencil-82913

06/16/2022, 10:07 PM
sqlite is good for small to medium clusters where you will only have a single server with moderate disk iops. external SQL is good if you have existing database infrastructure or a managed database service (RDS) that you can leverage for HA and backups. etcd is good if you want everything to be self-contained, use etcd snapshots for backup/restore, and can throw some reasonably performant SSD at the servers.
10s of workers should be fine for any datastore. I would worry more about the pod count and churn, and whether or not you’re going to deploy apps that put extra load on the apiserver via custom resources.
h

handsome-toddler-59547

06/16/2022, 10:08 PM
i see.. i've been driving towards an architecture where as much as possible is running on k8s.. for example though i started initially with services like ceph and docker and git external to the cluster.. i'm now running such services on the cluster.. i currently lean towards using etcd.. it seems like the default.. and it seems powerful.. but again i dont pretend to understand all the tradeoffs
and by docker i mean a private docker registry
so my cluster will have a variety of services.. nginx.. kafka .. mongo .. postgres.. and then a variety of custom services that will scale up
or should i say .. scale out ...
other services like knative and istio .. will also be there