This message was deleted.

Did you check the document https://docs.harvesterhci.io/v1.3/advanced/vgpusupport#technical-deep-dive ?

Yes, I read through this, but it was not of particular help. I have also removed the P4000 and reset the pcidevice-controller addon, which correctly rescans the P40s. I am able to pass the p40's through to a vm without issue and the driver I am using was received from NVIDIA. The issues is that harvester is not recognizing the p40s as a SR-IOV GPU Device. One question: under nvidia-driver-toolkit, where is the 'Image Tag' value coming from? is this set by harvester os, nvidia, or arbitrary?

value comes from harvester packaging of addon, and tag is closely tied to release of harvester as it contains the kernel header versions which match exactly the underlying OS

@great-bear-19718, thanks for the info on the tag field. According to NVIDIA and Dell documentation, the P40 is vGPU capable, so I am guessing this is a Harvester issue, or possible, I selected the wrong grid driver when registering with NVIDIA, or having more than 1 device installed is not allowed. I selected the standard linux driver. I am just not familiar enough with GPU drivers and linux to troubleshoot this. See Section (3.4.1 PowerEdge T640>Tesla P40 vGPU Matrix: https://dl.dell.com/manuals/common/dell-emc-poweredge-nvidia-vmware-vsphere.pdf )

P40 vGPU is a different class of vGPU.. driver handles it differentlly.. it is not a sriov vgpu device