Is anyone doing any monitoring where you're shutting down VMs that are idle/not being used? How are you determining what's in use and what's not?

Not doing it but be aware that stopped VMs and detached disks can stop longhorn's "Replica auto balance disk pressure threshold" from working https://github.com/longhorn/longhorn/issues/10837

my limited experience has been it works best to measure some app parameter, e.g. 1 VM can do 50 jobs and now I have 200 jobs so +3 VMs, now I have 25 jobs so spin back to 1 VM. vs trying to tell if the VM is "too busy" directly

For us the overall problem will likely be X students or faculty will provision a box and expect it to just be running for 14 months and never actually check on it or use it.

we've played a bit w/ knative for kube scale-to-zero, but not w/ VMs. but I would think something like that maybe, where a service listens and boots the VM on incoming connection? but yeah how to tell if its being used to spin down w/o knowing the workload sounds hard

Yeah. I noted that the MOTD for Ubuntu Focal (20.4 LTS) had

Users logged in:         0

So maybe there's a way qemu-guest-agent can grok/report it? Out side of that it might just be that after 2 weeks or so we shut it down if the CPU is near idle or something.

yeah interesting, seems so