https://rancher.com/ logo
Join Slack
Powered by
This message was deleted.
# harvester
a
This message was deleted.
g
The workarounds are: A) unbind nvme manually:
Copy code
for d in 0000:c2:00.0 0000:c3:00.0 0000:c4:00.0 0000:c5:00.0; do
  echo $d > /sys/bus/pci/drivers/nvme/unbind
done
B) or patch longhorn node and change the driver to aio
g
is it possible to have a support bundle when this error occurs.. I use nvme's with lhv2 myself and have never seen such issues
g
I will DM you
looks like SPDK script is run:
Copy code
May 08 06:52:51.695770 harvester-0 spdk: /usr/src/spdk/scripts/setup.sh  (done)
May 08 06:52:51.725765 harvester-0 spdk: /usr/src/spdk/scripts/setup.sh unbind 0000:c3:00.0 (start)
May 08 06:52:52.986779 harvester-0 kernel: nvme nvme0: pci function 0000:c3:00.0
May 08 06:52:52.993767 harvester-0 kernel: nvme nvme0: D3 entry latency set to 8 seconds
May 08 06:52:53.016772 harvester-0 kernel: nvme nvme0: 32/0/0 default/read/poll queues
and still, the nvme driver is in use
Copy code
rancher@harvester-0:~> lspci -nnk -s c3:00.0
c3:00.0 Non-Volatile memory controller [0108]: Samsung Electronics Co Ltd NVMe SSD Controller SM981/PM981/PM983 [144d:a808]
        Subsystem: Samsung Electronics Co Ltd SSD 970 EVO/PRO [144d:a801]
        Kernel driver in use: nvme
        Kernel modules: nvme
I think this may be the cause:
Copy code
IOMMU group: 2
Devices in group:
  0000:c0:03.0
    c0:03.0 Host bridge [0600]: Advanced Micro Devices, Inc. [AMD] Starship/Matisse PCIe Dummy Host Bridge [1022:1482]
  0000:c0:03.1
    c0:03.1 PCI bridge [0604]: Advanced Micro Devices, Inc. [AMD] Starship/Matisse GPP Bridge [1022:1483]
        Subsystem: Advanced Micro Devices, Inc. [AMD] Device [1022:1453]
        Kernel driver in use: pcieport
  0000:c0:03.2
    c0:03.2 PCI bridge [0604]: Advanced Micro Devices, Inc. [AMD] Starship/Matisse GPP Bridge [1022:1483]
        Subsystem: Advanced Micro Devices, Inc. [AMD] Device [1022:1453]
        Kernel driver in use: pcieport
  0000:c0:03.3
    c0:03.3 PCI bridge [0604]: Advanced Micro Devices, Inc. [AMD] Starship/Matisse GPP Bridge [1022:1483]
        Subsystem: Advanced Micro Devices, Inc. [AMD] Device [1022:1453]
        Kernel driver in use: pcieport
  0000:c0:03.4
    c0:03.4 PCI bridge [0604]: Advanced Micro Devices, Inc. [AMD] Starship/Matisse GPP Bridge [1022:1483]
        Subsystem: Advanced Micro Devices, Inc. [AMD] Device [1022:1453]
        Kernel driver in use: pcieport
  0000:c3:00.0
    c3:00.0 Non-Volatile memory controller [0108]: Samsung Electronics Co Ltd NVMe SSD Controller SM981/PM981/PM983 [144d:a808]
        Subsystem: Samsung Electronics Co Ltd SSD 970 EVO/PRO [144d:a801]
        Kernel driver in use: nvme
        Kernel modules: nvme
  0000:c4:00.0
    c4:00.0 Non-Volatile memory controller [0108]: Samsung Electronics Co Ltd NVMe SSD Controller SM981/PM981/PM983 [144d:a808]
        Subsystem: Samsung Electronics Co Ltd SSD 970 EVO/PRO [144d:a801]
        Kernel driver in use: nvme
        Kernel modules: nvme
There are PCI bridges installed on the same IOMMU group.
in that case, Harvester should use AIO mode
I guess that by manually unbinding the driver, we may work around the SPDK limitation of initializing the device.
g
ideally it should not happen, I have not been able to check myself but had requested another team member to check the support bundle
i use lh v2 with nvme disks for all our internal dev/qa workloads and we have never experienced this issue
and i am still running 1.7.1 in one of my clusters
g
One more thing that I've noticed is that spdk setup.sh process is blocked
I see it when I run 
ps aux
I don't know if it's related, but unbind may be blocked
15 Views
Open in Slack
PreviousNext
Powered by