This message was deleted.
# fleeta
adamant-kite-43734
06/30/2023, 8:57 PMThis message was deleted.
i
important-oxygen-86641
07/03/2023, 9:41 AMfleet is using helm to deploy resources. We use the default secret backend storage in helm, that means that helm history is stored in secrets.
We noticed too many requests to fetch secrets were done when fetching the helm history. We added a cache in https://github.com/rancher/fleet/pull/1485 which will be introduced in a future release.
This is something we are still investigating, and will add more improvements in the future.
a
acceptable-wolf-12708
07/04/2023, 5:02 AMOkay, thank you for the response. Not sure if it fits the code changes unless getting helm release would get all secrets from the cluster regardless of namespace as that seemed to be what was happening. A good test case would be to create 10k unrelated namespaces with 5 secrets each just to see if it happens with the changes.
i
important-oxygen-86641
07/04/2023, 4:14 PMAs far as I am aware the fleet-agent does not get all secrets. It gets the helm release secrets multiple times, which can be quite big as they contain the resources of the helm release. That's what this PR fixes.
a
acceptable-wolf-12708
07/04/2023, 5:08 PMThat don't explain it as the etcd read traffic increases as the amount of secrets in the cluster increase and it ended up with constant 100MB/s read traffic
i
important-oxygen-86641
07/05/2023, 7:19 AMIt might be a different issue then. Please create an issue in github in the fleet repo with more info about your environment, and we will investigate it. Thanks!
a
acceptable-wolf-12708
07/05/2023, 8:08 AMI'll try to create a repro for it, will create the issue if I can reliably reproduce it, thanks!
👍 1
acceptable-wolf-12708
07/06/2023, 10:23 AMI think I found out something, when fleet-agent v0.6.0 logs "getting history for release xxx-management-system-agent" it seems to do the full secret reading of the cluster and that release does it constantly for some reason, v0.7.0 doesn't do this but I think when it does that on startup it reads all the secrets in the cluster anyway which explains a sudden spike on startup, it still reads all of them but not in a loop
acceptable-wolf-12708
07/06/2023, 10:25 AMSo it seems getting the Helm secrets isn't limited to some namespaces but it fetches them from all for some reason
acceptable-wolf-12708
07/06/2023, 10:30 AMI'll create a proper issue now that I have something more tangible
acceptable-wolf-12708
07/06/2023, 11:16 AMi
important-oxygen-86641
07/06/2023, 1:06 PMThanks!
2 Views