This message was deleted.
# k3sa
adamant-kite-43734
07/26/2023, 6:43 AMThis message was deleted.
w
worried-jackal-89144
07/26/2023, 7:21 AMDo you know that this 1000 pods is per node?
Where those performance issues occur?
Did you really manage to start 1000 pods on one of nodes?
This config means on avarage 8m cores and 32M per pod (not including the resources needed for kubelet, kubeproxy and underlying OS).
Is your CNI given big enough IP pool for nodes to assign 1k Cluster IPs? I By default each node gets /24 subnet so you won't be able to create correctly over 255 pods.
worried-jackal-89144
07/26/2023, 7:27 AMhttps://kubernetes.io/docs/setup/best-practices/cluster-large/
Kubernetes itself recomends not more than 110 pods per node.
✅ 2
b
broad-dawn-76458
07/26/2023, 7:31 AMYes.
I want 1000 pods per node!
For the IP question, yes I run kubelet.config with the parameter below.
--cluster-cidr 192.0.0.0/8broad-dawn-76458
07/26/2023, 7:37 AMI know the Kubernetes recommends 110 pods per node but I can't find more information in k3s forum.
Actually, I found this link that have some discussion about the pods number per node.
https://github.com/kubernetes/kubernetes/issues/23349
broad-dawn-76458
07/26/2023, 8:08 AMActually, the system exhausts over 83.6% of the CPU resources when running just 500 pods per node.
The cost of system maintenance appears to be high.
Please refer to the picture below.
BTW, 668.7/800 = 83.6%
broad-dawn-76458
07/27/2023, 1:40 AMbroad-dawn-76458
07/27/2023, 1:58 AM[Q] To clarify my question, I'm looking for any parameters that could help reduce the CPU cost of the k3s system. I understand this might cause some side effects, but I'm interested in gathering more information at this stage.
w
worried-jackal-89144
07/27/2023, 7:10 PMThis node seems to run k3s server and it consumes over 250% cpu. Maybe you can have dedicated pool of nodes for controllers with less resources, and some workers with k3s-agent installed to run the workloads.
You can try disabling some FeatureGates that you don't use. But with 8 nodes, each with 500 pods, means 4k pods - usage of 2.5 cores for a controlplane seems not a bad result.
You can take a look on those articles:
https://openai.com/research/scaling-kubernetes-to-2500-nodes
https://openai.com/research/scaling-kubernetes-to-7500-nodes
Maybe you can find some hints relevant for your use case.
🙌 1
82 Views