I got sharded Prometheus partially deployed - 5 shards now...
busy-crowd-80458
02/06/2024, 12:32 PM
but it means I only get 1/5th of my metrics 🙂
g
gifted-cricket-25537
02/06/2024, 3:23 PM
We replaced the rancher-monitoring application with the kube-prometheus-stack as it had many bugs.
As for this many pods, you need to have a long list of metric drops, but that still could overwhelm Prometheus. We're currently looking into VictoriaMetrics to overcome this. It also has a "stack" helm chart to install.
gifted-cricket-25537
02/06/2024, 3:24 PM
for metric drops look into the tsdb-status page of Prometheus and start weeding out the metrics you don't need.