Hi guys, We have some customers that run mostly GPU heavy tasks and would like to monitor the usage of each NVidia GPU. Any software/tools recommendations for monitoring VMs (Ubuntu Server) that are using GPUs through PCI passthrough? I would like to avoid installing active agents (e.g. Zabbix active agent) into the VMs, if possible. Thanks

If you're using the Nvidia GPU operator in the downstream clusters you can enable the prometheus exporter to extract stats

Thanks, We are using the built-in PCI passthrough feature of Harvester. As far as I understand that is not using the nvidia GPU operator.

These are just bog standard VM's and not k8s nodes?

Hi David. These are VMs created in Harvester using the Create button on the Virtual Machines page. These VMs have one or more GPUs attached using the built-in PCI passthrough function of Harvester. Usually these VMs are running Ubuntu Server (20.04, 22.04). Hope this clarifies it.

i assume you are not running k8s on these VM's

No k8s running on those VMs. Usually, it's just Ubuntu Server with some machine learning (ML) libraries (e.g. Pytorch, Tensorflow) running ML tasks on the GPUs.