we are trying to present NVIDIA L4 GPU as vGPUs to harvester nodes so that VMs and RKE2 clussters on harvester can use GPUs. We were able to download nvidia drivers using trail license and deploy them on harvester. But now when we attach the vGPUs to VMs in harvester they can go into a loop (start and off) when we poweron. The events (attached image) show us connection refused errors related fds. Anyone got a similar issue while presenting GPUs to harvester VMs?

I've never had luck with the vGPU stuff. I've always stuck to pcie passthrough to get it to the VM. Personally I'd recommend pcie passthrough to a VM and do a custom cluster, or I'd set them up as elemental nodes if you need scaling or something automated.

Make sure you are using v1.8.2 or v1.7.3. there have been a lot of pci/vgpu fixes…

Also great advice

@bland-article-62755 we did try passthrough but then VMs can't share the GPU resources thus limiting the virtualization capabilities.

Granted, it's been a while since I checked, but you couldn't do something like a live migration with vgpu. So once the machine was up, it was stuck on that node anyways and practically there was no way to keep a workload up with vGPU vs passthough. We figured that if a VM had an entire card would could still do timesharing or split it up at that level vs Hypervisor. We're still dogfooding it but having 1 node per card as a VM in a cluster gets us a lot of the flexibility we actually needed. Not trying to say our solution will work for you, it's just where we ended up because the vGPU at the hypervisor level ended up being less shiny or stable compared to what we needed at our scale.

I need to test the migration also. For us it is due to this bug. we did to 17.3 but that didnt fix then fix in this one helped but I guess we have to upgrade to 1.8.x now. github.com/harvester/harvester/issues/9399

I asked the old AI slop:

KubeVirt explicitly documents that VMs using VFIO devices (which is how vGPUs are typically exposed via vGPU device plugins or SR-IOV) are not live-migratable.
The VirtualMachine spec has no knob to "make" a vGPU migratable; the limitation is at the hypervisor/device level, not KubeVirt-specific.
If you use NVIDIA vGPU with the KubeVirt GPU device plugin, the resulting VM will have a vfio device, and KubeVirt's migration preconditions will mark it non-migratable.

and

Why vGPU Live Migration Fails in KubeVirtHostdev Passthrough Limitations: KubeVirt exposes vGPUs to VMs as mediated devices (mdev) via PCI passthrough (VFIO). The KubeVirt controller inherently flags any VM with a hostdev attachment (including vGPUs and SR-IOV network cards) as LiveMigratable=False.  State Serialization: Standard live migration copies RAM and CPU state to the target host. KubeVirt cannot serialize the live memory and processing state held on the physical GPU hardware to stream it between the source virt-launcher pod and the destination pod.Pod Architecture: Live migration in KubeVirt works by spinning up a new target pod, streaming state to it, and switching execution over. Safely transferring a live slice of physical GPU hardware between two isolated container namespaces without guest OS disruption is a technical hurdle KubeVirt has not yet solved.

Which tracks with what I remember, but who knows.

would be nice to have a support bundle so we can check the full pod logs etc