Hi, I am trying to setup kube prometheus stack on K3s and I am having a lot of issues. By default kube controller, proxy and etcd services are not mapped to any container (with the latest version of the stack) and I have empty dashboards for those services in Grafana. Also I read that K3s is exposing metrics from all these in each of them so for the ones which work I will get duplicated data in the Prometheus. So is there custom version of kube prometheus stack which works with K3s and if not, are there any workarounds to get it to work with the latest prom stack version?

there are no containers for those things because they don’t run in containers. They all just run in the main k3s process.

Thanks for a quick response. Issue is that prom stack is quite complex and it creates services in Kubernetes which map to containers for normal Kube distribution. Then in turn they configure Prometheus to pull from those services, and Grafana to pull from the Prometheus using labels in those containers for Prometheus QL. In short to do what you describe I would have to change quite some stuff in the prom stack. That's why I am asking if anyone have already done it as both K3s and prom stack are quite widespread.

just comment out all but one of the control-plane scrapers

I also checked this thread where you actually answered some stuff: https://github.com/k3s-io/k3s/issues/3619 however even if I set the k3s config as is there I still don't understand how hardcoding the IPs in the endpoints for each service makes sense. If these run in main k3s process should I just get current Node IP and put it there (considering I bind them to 0.0.0.0)? What if there are multiple nodes? When you say "if you scrape the metrics port" do you mean I can basically scrape any of those services regardless and they will return me metrics for all of them (as opposed in k8s where they are all separated)? In that case I guess I will have to change Grafana dashboards as they use labels to run queries which are separated per service.