even if you only output the name field the apiserver still has to do all the work of retrieving and decoding it

and also secrets are encrypted at rest within the etcd datastore, so it’s gotta decrypt it

today morning, without chaning anything (same cluster)

kubectl get secrets -n kube-system -o name  0,31s user 0,03s system 85% cpu 0,388 total

the same request takes 0.388 so either this was an ISP issue or a time based issue on the cluster. I would usually go for ISP here, but why would configmaps be fast if it is an ISP

Could it be, that repeated calls (over minutes and hours) to secrets will rate limit somehow?

could it be, that deploying via terraform an saving the cluster state in as a k8s secret create somehow etc version churn or similar, causing this - and overnight it is reconsiled?

just checked, within the day yesterday i created 1029164479 − 1028604964 ≈ 559,000 revisions which seems a lot. That TF state is really hitting that hard, is it?

comparing the metrics yesterday: get secrets: sum=1641s count=1,406,707 → avg ~1.2ms each but that is an extrem number of gets. TF would get once per deployment - so this rather seems something within the cluster failing to reconsile or similar Since etcd seems to reconsile every ten minutes, this could hardly be the 'over night it fixed itself' cause. I restarted the cluster mid 'issue' yesterday (the control plane) and it did not fix the extreme slowdown. So somehow the load in memory cannot be explained here.:

i enabled audit logs and digged further. digging down even more it seems it is PGO that is constanly writing secrets and fails to reconsile

found the issue withing the pgo operator constantly hammering the secrects and this solved the issue