Hey folks, I'm trying to find the "right" path/way to setup grafana, prometheus, telegraf and monitoring.
I have a bunch of services which can either talk to telegraf and push metrics or they offer the monitoring endpoint for prometheus themselves (doesn't really matter ultimately) but I'm a bit confused of how to do it right:
I'm using Rancher, Rancher has a Monitoring section with a lot of Kubernetes monitoring and an existing Prometheus and Grafana but
My main questions and looking for help are:
• I'm afraid of using that also for my app specific dashboards and metrics collection, is this wrong and i should use the Rancher/Cattle-Monitoring stuff?
• Do I install Telegraf, Prometheus, Grafana on my own and crawl the kubernetes prometheus from that instance to build a unified dashboard?
• If I don't use Telegraf but the typical /metrics endpoint, how do i configure prometheus / k8s manifests to automatically collect the data?
I found really a lot of stuff in regards to Grafana, Prometheus, Rancher Monitoring and such but not really a good/best practices guide for monitoring Kubernetes & Applications/Services on top of Kubernetes.
I would appreciated any hints 🙂.