This message was deleted.

our clusters don't have enough resources to run prometheus/grafana, so we have a different server for that.

Well you need prometheus locally to scrape metrics but it can provide those metrics to an external grafana i'm sure... just keep in mind that latecy will play a factor in getting data in a timely manner.

The node-exporter doesn’t take much resources. We are very short on I/O so we keep the Prometheus DB on another server

It might be possible if you install Grafana in the cattle-monitoring-system ns. Rancher just proxies from the downstream cluster. Grafana is not "installed" in the rancher cluster in any way

You mean I would need still Grafana in the managed clusters then.

Yes rancher-monitoring installs the stack in the downstream cluster by default and the ln creates a link to proxy data back and forth (a bit of an oversimplification)

I have rancher in its own single node cluster now

def not HA but that is A-OK

Haha yeah hardware constrains are hard now…

I COMPLETELY understand hahah

so let me recap what I should try. • install grafana on the clusters in a namespace called

cattle-monitoring-system

• grafana set up with the datasource to our external prometheus server • node_exporter on all kubernetes nodes pulled by prometheus And nothing else special? Rancher would notice grafana is in the cluster and would autmatically show metrics on its UI?

Well no a few things we release a 'customized' version of Grafana (our custom dashboards) as part of our rancher-monitoring chart. This chart contains Prometheus, alert-manager and Grafana and installs in its own cattle-monitoring-system namespace. This chart is custom made so the UI can interact with it... Now you could break this chart up a bit and install only Grafana in one cluster and Prometheus in all of your resource constrained clusters. Then use that Grafana install as a central hub. But to be completely honest you're in untested territory with this one. Rancher-monitoring is designed to be the full stack installed in each downstream cluster... So it'll take a bit of hacking to split Prometheus and Grafana across multiple clusters

that's what I thought. I'll avoid the complexity for now and keep everything outside of the clusters. Thanks a lot for the time and conversation.