Heya brain-trust. I’m trying to find out workloads on some of our clusters are using API’s that are being removed, from the kubernetes docs https://kubernetes.io/blog/2020/09/03/warnings/#deprecation-warnings i found this handy Prometheus query:

apiserver_requested_deprecated_apis{removed_release="1.22"} * on(group,version,resource,subresource)
group_right() apiserver_request_total

However when i try it i get the following error:

Error executing query: found duplicate series for the match group {group="<http://apiextensions.k8s.io|apiextensions.k8s.io>", resource="customresourcedefinitions", version="v1beta1"} on the left hand-side of the operation: [{__name__="apiserver_requested_deprecated_apis", endpoint="https", group="<http://apiextensions.k8s.io|apiextensions.k8s.io>", instance="172.30.101.215:6443", job="apiserver", namespace="default", removed_release="1.22", resource="customresourcedefinitions", service="kubernetes", version="v1beta1"}, {__name__="apiserver_requested_deprecated_apis", component="k3s-server", endpoint="metrics", group="<http://apiextensions.k8s.io|apiextensions.k8s.io>", instance="172.30.101.215:10250", job="k3s-server", metrics_path="/metrics", namespace="cattle-monitoring-system", pod="pushprox-k3s-server-client-6stpf", removed_release="1.22", resource="customresourcedefinitions", service="pushprox-k3s-server-client", version="v1beta1"}];many-to-many matching not allowed: matching labels must be unique on one side

Being a novice in PromQL i don’t quite get how to resolve this.. any pointers? 🙂