rhythmic-painter-76998
04/26/2023, 7:56 AMred-king-19196
04/27/2023, 2:50 AMharvester-pcidevices-controller
DaemonSet running on the cluster. You can check the Pod’s log by executing kubectl -n harvester-system logs harvester-pcidevices-controller-xxxxx
to see if there’s any error message. And if it’s possible, please grab a support bundle file and post it here. We can help look into it. Thanksrhythmic-painter-76998
04/27/2023, 2:54 AMtime="2023-04-27T02:51:31Z" level=info msg="Adding harvester01-000001000 to KubeVirt list of permitted devices"
time="2023-04-27T02:51:31Z" level=info msg="Enabling passthrough for PDC: harvester01-000001000"
time="2023-04-27T02:51:31Z" level=info msg="Binding device harvester01-000001000 [10de 2204] to vfio-pci"
time="2023-04-27T02:51:31Z" level=info msg="Binding device 0000:01:00.0 vfio-pci"
time="2023-04-27T02:51:31Z" level=error msg="error syncing 'harvester01-000001000': handler PCIDeviceClaimReconcile: error writing to bind file: write /sys/bus/pci/drivers/vfio-pci/bind: invalid argument, requeuing"
btw where or how can I grab bundle file?red-king-19196
04/27/2023, 2:57 AMrhythmic-painter-76998
04/27/2023, 3:12 AMred-king-19196
04/27/2023, 3:14 AMrhythmic-painter-76998
04/27/2023, 3:24 AMgmehta3/pcidevices:dev
Digest:sha256:e86f89562a997dd54677eee9e945a1cadefa8cfa103267edf01d380ede39c9a5
) should be included.great-bear-19718
04/27/2023, 4:16 AMrhythmic-painter-76998
04/27/2023, 4:23 AMgreat-bear-19718
04/27/2023, 4:24 AMrhythmic-painter-76998
04/27/2023, 4:26 AMgreat-bear-19718
04/27/2023, 4:26 AMls -l /sys/bus/pci/drivers/vfio-pci/
rhythmic-painter-76998
04/27/2023, 4:31 AMsudo ls -l /sys/bus/pci/drivers/vfio-pci/
total 0
--w------- 1 root root 4096 Apr 27 02:00 bind
lrwxrwxrwx 1 root root 0 Apr 27 04:31 module -> ../../../../module/vfio_pci
--w------- 1 root root 4096 Apr 27 02:00 new_id
--w------- 1 root root 4096 Apr 27 04:31 remove_id
--w------- 1 root root 4096 Apr 27 02:00 uevent
--w------- 1 root root 4096 Apr 27 04:31 unbind
great-bear-19718
04/27/2023, 4:34 AMecho "10de 2204" > /sys/bus/pci/drivers/vfio-pci/new_id
echo "0000:01:00.0" > /sys/bus/pci/drivers/vfio-pci/bind
rhythmic-painter-76998
04/27/2023, 4:37 AMgreat-bear-19718
04/27/2023, 4:37 AMrhythmic-painter-76998
04/27/2023, 4:39 AMharvester01:~ # whoami
root
harvester01:~ # echo "10de 2204" > /sys/bus/pci/drivers/vfio-pci/new_id
-bash: echo: write error: File exists
great-bear-19718
04/27/2023, 4:39 AMrhythmic-painter-76998
04/27/2023, 4:39 AMharvester01:~ # echo "0000:01:00.0" > /sys/bus/pci/drivers/vfio-pci/bind
-bash: echo: write error: Invalid argument
great-bear-19718
04/27/2023, 4:42 AMecho "0000:01:00.0" > /sys/bus/pci/drivers/vfio-pci/unbind
rhythmic-painter-76998
04/27/2023, 4:43 AMharvester01:~ # echo "0000:01:00.0" > /sys/bus/pci/drivers/vfio-pci/unbind
-bash: echo: write error: No such device
great-bear-19718
04/27/2023, 4:49 AMlspci | grep -i nvidia
?rhythmic-painter-76998
04/27/2023, 4:52 AMharvester01:~ # lspci | grep -i nvidia
01:00.0 VGA compatible controller: NVIDIA Corporation GA102 [GeForce RTX 3090] (rev a1)
01:00.1 Audio device: NVIDIA Corporation GA102 High Definition Audio Controller (rev a1)
great-bear-19718
04/27/2023, 4:53 AMNVIDIA Corporation GA102 High Definition Audio Controller
as wellrhythmic-painter-76998
04/27/2023, 4:53 AMtime="2023-04-27T04:55:42Z" level=info msg="Reconciling PCI Devices list"
time="2023-04-27T04:55:54Z" level=info msg="Adding harvester01-000001000 to KubeVirt list of permitted devices"
time="2023-04-27T04:55:54Z" level=info msg="Enabling passthrough for PDC: harvester01-000001000"
time="2023-04-27T04:55:54Z" level=info msg="Binding device harvester01-000001000 [10de 2204] to vfio-pci"
time="2023-04-27T04:55:54Z" level=info msg="Binding device 0000:01:00.0 vfio-pci"
time="2023-04-27T04:55:54Z" level=error msg="error syncing 'harvester01-000001000': handler PCIDeviceClaimReconcile: error writing to bind file: write /sys/bus/pci/drivers/vfio-pci/bind: invalid argument, requeuing"
time="2023-04-27T04:55:56Z" level=info msg="Adding harvester01-000001001 to KubeVirt list of permitted devices"
time="2023-04-27T04:55:56Z" level=info msg="Enabling passthrough for PDC: harvester01-000001001"
time="2023-04-27T04:55:56Z" level=info msg="Binding device harvester01-000001001 [10de 1aef] to vfio-pci"
time="2023-04-27T04:55:56Z" level=info msg="Binding device 0000:01:00.1 vfio-pci"
time="2023-04-27T04:55:56Z" level=error msg="error syncing 'harvester01-000001001': handler PCIDeviceClaimReconcile: error writing to bind file: write /sys/bus/pci/drivers/vfio-pci/bind: invalid argument, requeuing"
great-bear-19718
04/27/2023, 4:56 AMls -alrt /sys/bus/pci/devices/0000:01:00.0
rhythmic-painter-76998
04/27/2023, 4:58 AMharvester01:~ # ls -alrt /sys/bus/pci/devices/0000:01:00.0
lrwxrwxrwx 1 root root 0 Apr 27 01:58 /sys/bus/pci/devices/0000:01:00.0 -> ../../../devices/pci0000:00/0000:00:01.0/0000:01:00.0
great-bear-19718
04/27/2023, 4:59 AMls -lart ls -alrt /sys/bus/pci/devices/0000:01:00.0/
rhythmic-painter-76998
04/27/2023, 5:01 AMharvester01:~ # ls -alrt /sys/bus/pci/devices/0000:01:00.0/
total 0
-r--r--r-- 1 root root 4096 Apr 27 01:58 waiting_for_supplier
-r--r--r-- 1 root root 4096 Apr 27 01:58 vendor
-rw-r--r-- 1 root root 4096 Apr 27 01:58 uevent
-r--r--r-- 1 root root 4096 Apr 27 01:58 subsystem_device
lrwxrwxrwx 1 root root 0 Apr 27 01:58 subsystem -> ../../../../bus/pci
-rw------- 1 root root 524288 Apr 27 01:58 rom
-r--r--r-- 1 root root 4096 Apr 27 01:58 revision
-rw------- 1 root root 128 Apr 27 01:58 resource5
-rw------- 1 root root 33554432 Apr 27 01:58 resource3_wc
-rw------- 1 root root 33554432 Apr 27 01:58 resource3
-rw------- 1 root root 268435456 Apr 27 01:58 resource1_wc
-rw------- 1 root root 268435456 Apr 27 01:58 resource1
-rw------- 1 root root 16777216 Apr 27 01:58 resource0
-r--r--r-- 1 root root 4096 Apr 27 01:58 resource
-rw-r--r-- 1 root root 4096 Apr 27 01:58 reset_method
--w------- 1 root root 4096 Apr 27 01:58 reset
--w------- 1 root root 4096 Apr 27 01:58 rescan
--w--w---- 1 root root 4096 Apr 27 01:58 remove
-r--r--r-- 1 root root 4096 Apr 27 01:58 power_state
drwxr-xr-x 2 root root 0 Apr 27 01:58 power
-rw-r--r-- 1 root root 4096 Apr 27 01:58 numa_node
-rw-r--r-- 1 root root 4096 Apr 27 01:58 msi_bus
-r--r--r-- 1 root root 4096 Apr 27 01:58 modalias
-r--r--r-- 1 root root 4096 Apr 27 01:58 max_link_width
-r--r--r-- 1 root root 4096 Apr 27 01:58 max_link_speed
-r--r--r-- 1 root root 4096 Apr 27 01:58 local_cpus
-r--r--r-- 1 root root 4096 Apr 27 01:58 local_cpulist
drwxr-xr-x 2 root root 0 Apr 27 01:58 link
-r--r--r-- 1 root root 4096 Apr 27 01:58 irq
lrwxrwxrwx 1 root root 0 Apr 27 01:58 firmware_node -> ../../../LNXSYSTM:00/LNXSYBUS:00/PNP0A08:00/device:00/device:01
-rw-r--r-- 1 root root 4096 Apr 27 01:58 enable
-rw-r--r-- 1 root root 4096 Apr 27 01:58 driver_override
-r--r--r-- 1 root root 4096 Apr 27 01:58 dma_mask_bits
-r--r--r-- 1 root root 4096 Apr 27 01:58 device
-rw-r--r-- 1 root root 4096 Apr 27 01:58 d3cold_allowed
-r--r--r-- 1 root root 4096 Apr 27 01:58 current_link_width
-r--r--r-- 1 root root 4096 Apr 27 01:58 current_link_speed
lrwxrwxrwx 1 root root 0 Apr 27 01:58 consumer:pci:0000:01:00.1 -> ../../../virtual/devlink/pci:0000:01:00.0--pci:0000:01:00.1
-r--r--r-- 1 root root 4096 Apr 27 01:58 consistent_dma_mask_bits
-rw-r--r-- 1 root root 4096 Apr 27 01:58 config
-r--r--r-- 1 root root 4096 Apr 27 01:58 class
-rw-r--r-- 1 root root 4096 Apr 27 01:58 broken_parity_status
-r--r--r-- 1 root root 4096 Apr 27 01:58 boot_vga
-r--r--r-- 1 root root 4096 Apr 27 01:58 ari_enabled
-r--r--r-- 1 root root 4096 Apr 27 01:58 aer_dev_nonfatal
-r--r--r-- 1 root root 4096 Apr 27 01:58 aer_dev_fatal
-r--r--r-- 1 root root 4096 Apr 27 01:58 aer_dev_correctable
drwxr-xr-x 12 root root 0 Apr 27 01:58 ..
drwxr-xr-x 4 root root 0 Apr 27 01:58 .
-r--r--r-- 1 root root 4096 Apr 27 05:00 subsystem_vendor
great-bear-19718
04/27/2023, 5:01 AMrhythmic-painter-76998
04/27/2023, 5:02 AMgreat-bear-19718
04/27/2023, 5:16 AMrhythmic-painter-76998
04/27/2023, 5:22 AM/sys/bus/pci/drivers/vfio-pci/bind: invalid argument, requeuing"
time="2023-04-27T05:22:25Z" level=info msg="Adding harvester01-000001000 to KubeVirt list of permitted devices"
time="2023-04-27T05:22:25Z" level=info msg="Enabling passthrough for PDC: harvester01-000001000"
time="2023-04-27T05:22:25Z" level=info msg="Binding device harvester01-000001000 [10de 2204] to vfio-pci"
time="2023-04-27T05:22:25Z" level=info msg="Binding device 0000:01:00.0 vfio-pci"
time="2023-04-27T05:22:25Z" level=error msg="error syncing 'harvester01-000001000': handler PCIDeviceClaimReconcile: error writing to bind file: write /sys/bus/pci/drivers/vfio-pci/bind: invalid argument, requeuing"
time="2023-04-27T05:22:27Z" level=info msg="Adding harvester01-000001001 to KubeVirt list of permitted devices"
time="2023-04-27T05:22:27Z" level=info msg="Enabling passthrough for PDC: harvester01-000001001"
time="2023-04-27T05:22:27Z" level=info msg="Binding device harvester01-000001001 [10de 1aef] to vfio-pci"
time="2023-04-27T05:22:27Z" level=info msg="Binding device 0000:01:00.1 vfio-pci"
time="2023-04-27T05:22:27Z" level=error msg="error syncing 'harve
great-bear-19718
04/27/2023, 5:49 AMrhythmic-painter-76998
04/27/2023, 5:52 AMharvester01:/home/rancher # dmesg | grep IOMMU
[ 0.023360] DMAR: IOMMU enabled
is this enough ?great-bear-19718
04/27/2023, 5:53 AMrhythmic-painter-76998
04/27/2023, 5:55 AMgreat-bear-19718
04/27/2023, 5:56 AMrhythmic-painter-76998
04/27/2023, 5:59 AMapiVersion: <http://devices.harvesterhci.io/v1beta1|devices.harvesterhci.io/v1beta1>
kind: PCIDevice
metadata:
annotations:
<http://harvesterhci.io/pcideviceDriver|harvesterhci.io/pcideviceDriver>: ""
creationTimestamp: "2023-04-26T02:07:48Z"
generation: 1
labels:
nodename: harvester01
name: harvester01-000001000
resourceVersion: "22811"
uid: febacf0f-8ddd-4ad9-b44e-ae4c06114643
spec: {}
status:
address: "0000:01:00.0"
classId: "0300"
description: 'VGA compatible controller: NVIDIA Corporation GA102 [GeForce RTX 3090]'
deviceId: "2204"
iommuGroup: ""
nodeName: harvester01
resourceName: <http://nvidia.com/GA102_GEFORCE_RTX_3090|nvidia.com/GA102_GEFORCE_RTX_3090>
vendorId: 10de
iommuGroup
is emptygreat-bear-19718
04/27/2023, 6:09 AMrhythmic-painter-76998
04/27/2023, 6:11 AMgreat-bear-19718
04/27/2023, 6:14 AMvt-d
is enabled in the biosrhythmic-painter-76998
04/27/2023, 6:29 AMgreat-bear-19718
04/27/2023, 6:30 AMVT-d
in the biosapiVersion: <http://devices.harvesterhci.io/v1beta1|devices.harvesterhci.io/v1beta1>
kind: PCIDevice
metadata:
annotations:
<http://harvesterhci.io/pcideviceDriver|harvesterhci.io/pcideviceDriver>: ixgbe
creationTimestamp: "2023-04-14T03:23:29Z"
generation: 1
labels:
nodename: harvester-ldgh9
name: harvester-ldgh9-000004000
resourceVersion: "38802"
uid: 502ab495-414d-45fa-b5b8-0e86fa92e899
spec: {}
status:
address: "0000:04:00.0"
classId: "0200"
description: 'Ethernet controller: Intel Corporation 82599ES 10-Gigabit SFI/SFP+
Network Connection'
deviceId: 10fb
iommuGroup: "36"
kernelDriverInUse: ixgbe
nodeName: harvester-ldgh9
resourceName: <http://intel.com/82599ES_10GIGABIT_SFI_SFP_NETWORK_CONNECTION|intel.com/82599ES_10GIGABIT_SFI_SFP_NETWORK_CONNECTION>
vendorId: "8086"
apiVersion: <http://devices.harvesterhci.io/v1beta1|devices.harvesterhci.io/v1beta1>
kind: PCIDevice
metadata:
annotations:
<http://harvesterhci.io/pcideviceDriver|harvesterhci.io/pcideviceDriver>: ixgbe
creationTimestamp: "2023-04-27T06:31:03Z"
generation: 1
labels:
nodename: harvester-ldgh9
name: harvester-ldgh9-000004000
resourceVersion: "21133767"
uid: 989bb8bf-f6e8-424d-be70-f110e251c3b8
spec: {}
status:
address: "0000:04:00.0"
classId: "0200"
description: 'Ethernet controller: Intel Corporation 82599ES 10-Gigabit SFI/SFP+
Network Connection'
deviceId: 10fb
iommuGroup: ""
kernelDriverInUse: ixgbe
nodeName: harvester-ldgh9
resourceName: <http://intel.com/82599ES_10GIGABIT_SFI_SFP_NETWORK_CONNECTION|intel.com/82599ES_10GIGABIT_SFI_SFP_NETWORK_CONNECTION>
vendorId: "8086"
rhythmic-painter-76998
04/27/2023, 6:35 AMgreat-bear-19718
04/27/2023, 6:35 AMharvester-pcidevices-controller-2tpxr agent time="2023-04-27T06:35:20Z" level=info msg="Adding harvester-ldgh9-000004000 to KubeVirt list of permitted devices"
harvester-pcidevices-controller-2tpxr agent time="2023-04-27T06:35:20Z" level=info msg="Enabling passthrough for PDC: harvester-ldgh9-000004000"
harvester-pcidevices-controller-2tpxr agent time="2023-04-27T06:35:20Z" level=info msg="Binding device harvester-ldgh9-000004000 [8086 10fb] to vfio-pci"
harvester-pcidevices-controller-2tpxr agent time="2023-04-27T06:35:20Z" level=info msg="Binding device 0000:04:00.0 vfio-pci"
harvester-pcidevices-controller-2tpxr agent time="2023-04-27T06:35:20Z" level=error msg="error syncing 'harvester-ldgh9-000004000': handler PCIDeviceClaimReconcile: error writing to bind file: write /sys/bus/pci/drivers/vfio-pci/bind: invalid argument, requeuing"
rhythmic-painter-76998
04/27/2023, 6:35 AMgreat-bear-19718
04/27/2023, 6:57 AM