If on every node the devices of your selector are assigned to

vfio-pci

, you won't have a problem wherever it sits.