Hi, I just read about monitoring v2 implemnted for rancher2 above 2.6. My goal is to have separate observability cluster that will collect metrics from all our k8s clusters. Is it possible to use existing monitoring solution from rancher to get this goal? I see there is some extra pushprox application. Is it capable to collect from it a metrics from different clusters (on separate networks) and deliver to this observability cluster? Or should I do some sort of prometheus setup in federation mode to have some primary-secondary model?

Yes, of course. You only have to: 1. Deploy Prometheus chart in your clusters 2. Configure Grafana in one of your clusters (central monitoring cluster) to use the prometheus of other clusters as datasource. For this, you must expose the Prometheus in other clusters using Ingress and protect them, e.g. by IP address whitelists.

ok but the rancher monitoring charts are something that can be installed by manual way or should it be done by rancher api? Also what about the federation mode, I suppose that when we expose prometeush and have no a master one then then there is a risk of losing metrics if cluster collapse, So it's like to have a monitoring on the same cluster, so we will now it's down but not know why till it's back.

You can deploy manually via Rancher UI or via Terraform. We chose to deploy via rancher2 Terraform provider, which at the end uses the API

I mean here about the situation where your cluster is not avaiable, ok you have the data on the storage but when you login to grafana there is a risk that prometeus that use that storage is on high load and grafana will not be able to get the data and display it so till fix of your cluster you will not see your metrics. That's why my plan was to use prometeus in federation mode so the main one will scrap the metrics from anothers till they work and by that that I should see them

To be honest, I don't know what you mean with federation mode

Ah... As always I would start with a simple deployment and then if there is a good reason to extend it

huh, yea but I have limited decision here so try to figure out what's possible.

You don't deploy the internal (old) monitoring by Terraform. You should deploy the chart using the Terraform chart resource

you mean this one? https://registry.terraform.io/providers/rancher/rancher2/latest/docs/resources/app_v2

Yes

ok, do you know if this resource do something extra or just act as a helm in backround so take the values and deploy chart?

In my understanding, it acts like a Helm client. What resource is helm_release?

Yes, you could also use the Helm provider. But because we also do some other tasks with Rancher, we decided to use the Rancher2 terraform resource

ok wil play a bit on both to see what match more my needs. I used monitoring v1 in the past and it works well more or less but here and now the descition was change to use dedicated cluster just for monitoring purpose and wonder how to setup all of that.