https://rancher.com/ logo
Join Slack
Powered by
This message was deleted.
# harvester
a
This message was deleted.
r
Hi @hallowed-analyst-40330! It would be nice to have a support bundle file for us to see what’s happening on your cluster 🙂
g
the no-iommu group likely means vt-d is disabled in bios
the check was written to ensure unexpected failures when trying to bind to vfio driver which needs the iommu group
once you turn on vt-d you may need to delete all pcidevices objects and controller will rescan and recreate the with the correct iommu group info and then nic should be passed through
🙌 1
h
I had KVM working fine in debian and am now using proxmox as it was time critical to get this KVM cluster up. So I no longer had the cluster. VT-D was enable in bios this i know as well as iommu so that was not the issue.
g
if you try again.. 
kubectl get pcidevice $devicename -o yaml
there should be a iommu group in the status..
h
Ah ok. I will try this today on a spare ssd
Where do I get that?
g
the device you are trying to pass through 
kubectl get pcidevices $deviceName -o yaml
h
Thank you very much for your help thus far.
I will do that once I have harvestor back up on this machine
I did this and got
Copy code
- apiVersion: devices.harvesterhci.io/v1beta1
  kind: PCIDevice
  metadata:
    annotations:
      harvesterhci.io/pcideviceDriver: r8169
    creationTimestamp: "2023-10-23T14:25:01Z"
    generation: 1
    labels:
      nodename: main-homeserver
    name: main-homeserver-000043000
    resourceVersion: "263021"
    uid: 092f3053-3a6c-4ade-b25e-b98f850dd2d4
  spec: {}
  status:
    address: "0000:43:00.0"
    classId: "0200"
    description: 'Ethernet controller: Realtek Semiconductor Co., Ltd. RTL8111/8168/8411
      PCI Express Gigabit Ethernet Controller'
    deviceId: "8161"
    iommuGroup: ""
    kernelDriverInUse: r8169
    nodeName: main-homeserver
    resourceName: realtek.com/RTL8111_8168_8411_PCI_EXPRESS_GIGABIT_ETHERNET_CONTROLLER
    vendorId: 10ec
g
there is no iommu group in the CR
likely VT/d is disabled in bios
h
Ah ok let me check
I don't think that the issues. As sr-iov and svm(amds iommu and virtualization) are both enabled
Nx is also enable(no-execute page protection function)
64-bit pci decoding is also enabled
g
the linux kernel is not detecting iommu groups
and that is likely due to VT/D missing
i am not sure of the machine type or bios config.. but likely that is what is missing
if VT/D was enabled later.. you can delete the pcidevice.. and it will be recreated on next scan
and it will pick up the new iommu group
h
All of the above has been enabled since install of rancher 1.2.0 which I just did a clean install of today
g
like i said before no iommu group is detected on the device.. for device passthrough to work.. it needs to be bound to vfio driver which needs iommu group info
h
Would there be any reason why other devices would have iommu groups but a specific device would not?
I ask as gpu passthrough works no problem
g
not sure myself.. i dont have this nic to verify locally
h
That is fair. If I add the nic to the network section am I able to use it as an adapter for vm?
g
you cant pass a physical nic directly to vm that way
it has to be done via pci passthrough
is this a usb card?
h
No it is pci
But I am trying to run pfsense in harvester and I need to have a wan nic. The other nic is the management which will be lan
The management nic I am not passing through
g
are there empty slots in the machine
where you could move this card to?
h
I have already tried moving it around it does not work
g
I will ask internally if anyone has this nic available
h
Ok thank you
I know on normal kvm all I do is create a bridge for this nic
But I don't know if that is.possible in harvestor yet
g
bridge is not same as passing the device to vm
if you just want a bridge.. then create a cluster network using this NIC
h
Well passthrough is the best option but bridge could work as well
g
create a vm network using new cluster network..
then harvester will be using this nic to create bridge interfaces
h
Thank you
2 Views
Open in Slack
PreviousNext
Powered by