Hi there, what can the issue if every pod scheduled to same worker every time we run CI pipeline job? the cluster base 4 workers, has the same resources, there is no taint or nodeSelector configured

It sounds like normal behavior, since the kubernetes scheduler is not breaking any rules. If you want the pods on each node you could set node selectors on the pods or create a Daemonset.

isnt the scheduler assign where there free resources and do the load balancing?

the scheduler will assign them to wherever there are free resources

so its weird because we see that it schedule always to a specific worker and reach to high load

can you share the pod spec?

NAME CPU(cores) CPU% MEMORY(bytes) MEMORY% worker-01 639m 0% 27327Mi 36% worker-02 2199m 3% 34688Mi 48% worker-03 1128m 1% 28053Mi 38% worker-04 987m 1% 27513Mi 38%

if you cordone worker 2, do the pods get scheduled to other nodes?

spec: containers: - args: - while true; do sleep 30; done; command: - /bin/sh - -c image: zuul-image-repo imagePullPolicy: Always name: zuul resources: {} terminationMessagePath: /dev/termination-log terminationMessagePolicy: File - mountPath: /var/run/secrets/kubernetes.io/serviceaccount name: kube-api-access-nllmw readOnly: true dnsPolicy: ClusterFirst enableServiceLinks: true nodeName: worker-02 preemptionPolicy: PreemptLowerPriority priority: 0 restartPolicy: Always schedulerName: default-scheduler securityContext: {} serviceAccount: default serviceAccountName: default terminationGracePeriodSeconds: 30 tolerations: - effect: NoExecute key: node.kubernetes.io/not-ready operator: Exists tolerationSeconds: 300 - effect: NoExecute key: node.kubernetes.io/unreachable operator: Exists tolerationSeconds: 300

On the spec, I saw that the pod doesn't have a resource quota. You can add a quota which should limit how many of them will get scheduled

maybe its because that pods installed via ansible plaubook and there is no any deployment/daemonset in cluster?

I think it is normal behavior because I can't find anything in the kubernetes documentation that guarantees that the scheduler will distribute pods evenly. There is a feature called topology spread constraints that can also be used to equally distribute nodes. https://kubernetes.io/docs/concepts/scheduling-eviction/topology-spread-constraints/ But I think the easiest way to do it would be to add resource quotas to the pod spec. The resource quotas are a kubernetes best practice, while the topology constraints are a more advanced feature.

if i understand correctly, Resource Quota set resource limit (cpu,mem) for pods