https://rancher.com/ logo
Join Slack
Powered by
This message was deleted.
# harvester
a
a
thanks, I will take a look
from the SB, there is no 
prometheus-kubevirt-rules
I will check the possible cause
w
I don't know if it was something quirky with out install but I had to manually create the service monitor. Raised the issue incase anyone else gets the same happen as re-creating the monitor does get the stats working again, at least I think it does - Grafana has charts now! The cluster metrics did always show, it was the virtual machine metrics that for some reason showed no data.
a
I am checking why the prometheus-kubevirt-rules is missing
give me some time to figure out the possible cause
w
No worries, Thanks for looking 🙂
a
🤝
have you ever manually delete the 
cattle-monitoring-system
namespace ?
w
Not to my knowledge - I did disable the add on and re-enable it to try and get it working. It certainly wouldn't have been deleted intentionally.
Would enabling the add-on not check that its resources are created?
a
the 
virt-operator
logs show the 
servicemonitor
was there when it started
Copy code
2025-11-21T18:07:47.181007922Z {"component":"virt-operator","level":"info","msg":"servicemonitor is defined","pos":"application.go:218","timestamp":"2025-11-21T18:07:47.180972Z"}

2025-11-21T18:07:47.271919503Z {"component":"virt-operator","level":"info","msg":"prometheusrule is defined","pos":"application.go:232","timestamp":"2025-11-21T18:07:47.271864Z"}
w
That must have been from when we upgraded. The VM detail pages still show "no-data" btw (added screens on issue).
a
I delete it, the 
virt-operator
creates it again very quickly
Copy code
harv41:/home/rancher # kk delete servicemonitor -n cattle-monitoring-system prometheus-kubevirt-rules
<http://servicemonitor.monitoring.coreos.com|servicemonitor.monitoring.coreos.com> "prometheus-kubevirt-rules" deleted from cattle-monitoring-system namespace
harv41:/home/rancher # kk get servicemonitor -A
NAMESPACE                  NAME                                          AGE
cattle-monitoring-system   prometheus-kubevirt-rules                     1s
w
Di you want me to try that?
a
you can do this on your env
it just re-create the target object
Copy code
harv41:/home/rancher # kk logs -n harvester-system virt-operator-54974f88c8-2xp4q | grep servicem -i
{"component":"virt-operator","level":"info","msg":"serviceMonitor prometheus-kubevirt-rules created","pos":"prometheus.go:49","timestamp":"2025-12-09T15:20:39.354224Z"}
w
There is no kubevirt-rules -
Copy code
% kubectl get servicemonitor -n cattle-monitoring-system 
NAME                                          AGE
rancher-monitoring-alertmanager               3d1h
rancher-monitoring-apiserver                  3d1h
rancher-monitoring-coredns                    3d1h
rancher-monitoring-grafana                    3d1h
rancher-monitoring-kube-state-metrics         3d1h
rancher-monitoring-kubelet                    3d1h
rancher-monitoring-operator                   3d1h
rancher-monitoring-prometheus                 3d1h
rancher-monitoring-prometheus-node-exporter   3d1h
rancher-monitoring-windows-exporter           3d1h
a
try kill the 
virt-operator
pod
w
I had recreated them here -
Copy code
% kubectl get servicemonitor -n harvester-system         
NAME                  AGE
kubevirt-metrics      43h
service-monitor-cdi   17d
I think - thats the servicemonitor I added as per the issue - i can delete that if needed
a
remove your manual one
and kill the 
virt-operator
pod
it should create things automatically
w
we now have a new harvester-system/virt-operator-5dfbbd6957-g7dbc tail of its log
Copy code
{"component":"virt-operator","level":"info","msg":"Loaded install strategy for kubevirt version 1.5.2-150700.3.5.2 into cache","pos":"kubevirt.go:893","timestamp":"2025-12-09T15:29:04.690781Z"}
{"component":"virt-operator","level":"info","msg":"service kubevirt-prometheus-metrics patched","pos":"core.go:143","timestamp":"2025-12-09T15:29:04.699033Z"}
{"component":"virt-operator","level":"info","msg":"service virt-api patched","pos":"core.go:143","timestamp":"2025-12-09T15:29:04.700875Z"}
{"component":"virt-operator","level":"info","msg":"service kubevirt-operator-webhook patched","pos":"core.go:143","timestamp":"2025-12-09T15:29:04.702117Z"}
{"component":"virt-operator","level":"info","msg":"service virt-exportproxy patched","pos":"core.go:143","timestamp":"2025-12-09T15:29:04.703845Z"}
{"component":"virt-operator","level":"info","msg":"validatingAdmissionPolicyBinding kubevirt-node-restriction-binding patched","pos":"admissionregistration.go:394","timestamp":"2025-12-09T15:29:04.705733Z"}
{"component":"virt-operator","level":"info","msg":"validatingAdmissionPolicy kubevirt-node-restriction-policy patched","pos":"admissionregistration.go:454","timestamp":"2025-12-09T15:29:04.714575Z"}
{"component":"virt-operator","level":"info","msg":"Kept old CA certificates for a duration of at least 4h48m0s, dropping them now.","pos":"secrets.go:421","timestamp":"2025-12-09T15:29:04.714828Z"}
{"component":"virt-operator","level":"warning","msg":"json: cannot unmarshal object into Go value of type jsonpatch.Patch","pos":"apps.go:425","timestamp":"2025-12-09T15:29:04.754683Z"}
{"component":"virt-operator","level":"info","msg":"Kubevirt namespace (harvester-system) labels are in sync","pos":"core.go:59","timestamp":"2025-12-09T15:29:04.768706Z"}
{"component":"virt-operator","kind":"KubeVirt","level":"info","msg":"All KubeVirt resources created","name":"kubevirt","namespace":"harvester-system","pos":"kubevirt.go:1081","timestamp":"2025-12-09T15:29:04.786104Z","uid":"b30302e9-cf48-4d33-860e-078bfd27f626"}
{"component":"virt-operator","kind":"KubeVirt","level":"info","msg":"All KubeVirt components ready","name":"kubevirt","namespace":"harvester-system","pos":"kubevirt.go:1085","timestamp":"2025-12-09T15:29:04.786145Z","uid":"b30302e9-cf48-4d33-860e-078bfd27f626"}
No data for vms yet.
Copy code
% kubectl get servicemonitor -n cattle-monitoring-system 
NAME                                          AGE
rancher-monitoring-alertmanager               3d1h
rancher-monitoring-apiserver                  3d1h
rancher-monitoring-coredns                    3d1h
rancher-monitoring-grafana                    3d1h
rancher-monitoring-kube-state-metrics         3d1h
rancher-monitoring-kubelet                    3d1h
rancher-monitoring-operator                   3d1h
rancher-monitoring-prometheus                 3d1h
rancher-monitoring-prometheus-node-exporter   3d1h
rancher-monitoring-windows-exporter           3d1h
No prometheus-kubevirt-rules in that list 👆
a
any output when 
kubectl logs -n harvester-system virt-operator-54974f88c8-2xp4q | grep servicem -i
w
Copy code
% kubectl logs -n harvester-system virt-operator-5dfbbd6957-g7dbc | grep servicem -i
{"component":"virt-operator","level":"info","msg":"servicemonitor is defined","pos":"application.go:218","timestamp":"2025-12-09T15:28:45.564316Z"}
{"component":"virt-operator","level":"info","msg":"STARTING informer operatorServiceMonitorInformer","pos":"virtinformers.go:363","timestamp":"2025-12-09T15:28:45.574503Z"}
( new pod name is virt-operator-5dfbbd6957-g7dbc )
a
it said 
servicemonitor is defined
let me check
kubectl get servicemonitor -A
w
Copy code
craig@Craigs-Mac-Studio ~ % kubectl get servicemonitor -A
NAMESPACE                  NAME                                          AGE
cattle-fleet-system        monitoring-fleet-controller                   3d1h
cattle-fleet-system        monitoring-gitops-controller                  3d1h
cattle-monitoring-system   rancher-monitoring-alertmanager               3d1h
cattle-monitoring-system   rancher-monitoring-apiserver                  3d1h
cattle-monitoring-system   rancher-monitoring-coredns                    3d1h
cattle-monitoring-system   rancher-monitoring-grafana                    3d1h
cattle-monitoring-system   rancher-monitoring-kube-state-metrics         3d1h
cattle-monitoring-system   rancher-monitoring-kubelet                    3d1h
cattle-monitoring-system   rancher-monitoring-operator                   3d1h
cattle-monitoring-system   rancher-monitoring-prometheus                 3d1h
cattle-monitoring-system   rancher-monitoring-prometheus-node-exporter   3d1h
cattle-monitoring-system   rancher-monitoring-windows-exporter           3d1h
cattle-system              rancher                                       3d1h
harvester-system           service-monitor-cdi                           17d
longhorn-system            longhorn-prometheus-servicemonitor            17d
Some screens
a
Copy code
kubectl logs -n harvester-system virt-operator-5dfbbd6957-g7dbc | grep prometheusrule -i
could you run above also
in old log, I see
Copy code
2025-11-21T18:07:47.271919503Z {"component":"virt-operator","level":"info","msg":"prometheusrule is defined","pos":"application.go:232","timestamp":"2025-11-21T18:07:47.271864Z"}
👀 1
w
Copy code
craig@Craigs-Mac-Studio ~ % kubectl logs -n harvester-system virt-operator-5dfbbd6957-g7dbc | grep prometheusrule -i
{"component":"virt-operator","level":"info","msg":"prometheusrule is defined","pos":"application.go:232","timestamp":"2025-12-09T15:28:45.567444Z"}
{"component":"virt-operator","level":"info","msg":"STARTING informer OperatorPrometheusRuleInformer","pos":"virtinformers.go:363","timestamp":"2025-12-09T15:28:45.574662Z"}
craig@Craigs-Mac-Studio ~ %
👍 1
@ancient-pizza-13099 do you need anything more from me, are there charts that needs re-applying to get this working? any idea why these are missing?
a
@worried-state-78253 I am a bit busy recently, did not have enough time to dig deeper, but will do
w
Many thanks - just let me know if you want me to try and figure out a patch, happy to wait though if this is likely to be something simple to fix with the enable/disable action
I'm busy on other things at present myself but more than happy to assist where I can
👍 1
10 Views
Open in Slack
PreviousNext
Powered by