Anyone experience with creating a cluster via vSphere with some nodes having access to the GPU attached to a physical host?

Not with Rancher, though we have done it with kubeadm after manually connecting the GPU with PCI-passthrough. Not sure which aspect of this you are talking about.

Currently we create our cluster infrastructure via the Rancher vSphere provider. But using this approach I don't see any options to dynamically configure a PCI device. So my current thought is to create a Virtual Machine Template using a PCI device (for example via Packer and store it in the vSphere Content Library) and use this template to create my infrastructure. Just curious if someone else solved it on a different way of got thought about my approach

That's exactly the way we did it. We have three machine pools, one control-plane, one data-plane, and one data-GPU. data-GPU is created using the previously created VM template with the GPU attached. You'll have to set some settings dynamically when creating the nodes from the VM template though, as there are e. g. memory reservations necessary. Does that help?

Thank you for the reply. You know of other settings that needs to be configured dynamically?

You need to set the following settings (cfgparams) in the VM options: • sched.mem.min: this needs to be the memory size of your VM as the GPU VM needs all of the memory by reservation • sched.mem.minSize: this needs to be the memory size of your VM as the GPU VM needs all of the memory by reservation • pciPassthru.64bitMMIOSizeGB: check this article

"disk.enableUUID=TRUE",
"sched.mem.pin=TRUE",
"sched.mem.min=<memory size of your VM>",
"sched.mem.minSize=<memory size of your VM>",

"pciPassthru.use64bitMMIO=TRUE",
"pciPassthru.64bitMMIOSizeGB=<depends on the GPU you use>"

Hope that helps