Bug#1086520: linux-image-6.11.2-amd64: makes opensm fail to start

[Francesco Poli (wintermute)]

Package: src:linux
Version: 6.11.2-1
Severity: important
X-Debbugs-Cc: ***@paranoici.org

Hello,
I encountered a major issue on an HPC cluster head node, as soon as
I upgraded the Linux kernel from version 6.10.11-1 to version 6.11.2-1 .

The issue is that the head node runs OpenSM (InfiniBand subnet manager),
which is needed for the Infiniband network to work.
As soon as I reboot the head node with kernel 6.11.2-1 (or 6.11.4-1),
OpenSM fails to start.
If I reboot with the previous kernel version 6.10.11-1, everything
works fine.

The symptoms are described in bug [#1085300], filed against package
opensm.

[#1085300]: <https://bugs.debian.org/1085300>

Now I am not sure what's going on.

Is there any important change in the Linux kernel that OpenSM needs
to adapt for?
Or is this a bug in the newer Linux kernel version (that needs to
be fixed there)?

I filed this bug report against the Debian Linux kernel, in order
to warn other users about this issue, and in order to ask the Debian
Kernel Team to investigate the issue and/or to forward the bug report
to the relevant upstream Linux kernel maintainers.

Please do not reassign to package opensm with the intention of
merging with bug [#1085300], unless you know for sure that the
issue is in opensm and you know how to fix it.


Please help, I would very much like to run the head node with
an up-to-date kernel!

Thanks for your time and patience.


-- Package-specific info:
** Kernel log: boot messages should be attached

** Model information
sys_vendor: Supermicro
product_name: AS -2014S-TR
product_version: 0123456789
chassis_vendor: Supermicro
chassis_version: 0123456789
bios_vendor: American Megatrends Inc.
bios_version: 2.9
board_vendor: Supermicro
board_name: H12SSL-i
board_version: 1.10

** PCI devices:
00:00.0 Host bridge [0600]: Advanced Micro Devices, Inc. [AMD] Starship/Matisse Root Complex [1022:1480]
Subsystem: Super Micro Computer Inc H12SSL-i [15d9:1b95]
Control: I/O- Mem- BusMaster- SpecCycle- MemWINV- VGASnoop- ParErr- Stepping- SERR- FastB2B- DisINTx-
Status: Cap- 66MHz- UDF- FastB2B- ParErr- DEVSEL=fast >TAbort- <TAbort- <MAbort- >SERR- <PERR- INTx-

00:00.2 IOMMU [0806]: Advanced Micro Devices, Inc. [AMD] Milan IOMMU [1022:164f] (rev 01)
Subsystem: Advanced Micro Devices, Inc. [AMD] Milan IOMMU [1022:164f]
Control: I/O- Mem- BusMaster- SpecCycle- MemWINV- VGASnoop- ParErr+ Stepping- SERR- FastB2B- DisINTx-
Status: Cap+ 66MHz- UDF- FastB2B- ParErr- DEVSEL=fast >TAbort- <TAbort- <MAbort- >SERR- <PERR- INTx-
Capabilities: <access denied>

00:01.0 Host bridge [0600]: Advanced Micro Devices, Inc. [AMD] Starship/Matisse PCIe Dummy Host Bridge [1022:1482]
Control: I/O- Mem- BusMaster- SpecCycle- MemWINV- VGASnoop- ParErr- Stepping- SERR- FastB2B- DisINTx-
Status: Cap- 66MHz- UDF- FastB2B- ParErr- DEVSEL=fast >TAbort- <TAbort- <MAbort- >SERR- <PERR- INTx-
IOMMU group: 58

00:02.0 Host bridge [0600]: Advanced Micro Devices, Inc. [AMD] Starship/Matisse PCIe Dummy Host Bridge [1022:1482]
Control: I/O- Mem- BusMaster- SpecCycle- MemWINV- VGASnoop- ParErr- Stepping- SERR- FastB2B- DisINTx-
Status: Cap- 66MHz- UDF- FastB2B- ParErr- DEVSEL=fast >TAbort- <TAbort- <MAbort- >SERR- <PERR- INTx-
IOMMU group: 59

00:03.0 Host bridge [0600]: Advanced Micro Devices, Inc. [AMD] Starship/Matisse PCIe Dummy Host Bridge [1022:1482]
Control: I/O- Mem- BusMaster- SpecCycle- MemWINV- VGASnoop- ParErr- Stepping- SERR- FastB2B- DisINTx-
Status: Cap- 66MHz- UDF- FastB2B- ParErr- DEVSEL=fast >TAbort- <TAbort- <MAbort- >SERR- <PERR- INTx-
IOMMU group: 60

00:04.0 Host bridge [0600]: Advanced Micro Devices, Inc. [AMD] Starship/Matisse PCIe Dummy Host Bridge [1022:1482]
Control: I/O- Mem- BusMaster- SpecCycle- MemWINV- VGASnoop- ParErr- Stepping- SERR- FastB2B- DisINTx-
Status: Cap- 66MHz- UDF- FastB2B- ParErr- DEVSEL=fast >TAbort- <TAbort- <MAbort- >SERR- <PERR- INTx-
IOMMU group: 61

00:05.0 Host bridge [0600]: Advanced Micro Devices, Inc. [AMD] Starship/Matisse PCIe Dummy Host Bridge [1022:1482]
Control: I/O- Mem- BusMaster- SpecCycle- MemWINV- VGASnoop- ParErr- Stepping- SERR- FastB2B- DisINTx-
Status: Cap- 66MHz- UDF- FastB2B- ParErr- DEVSEL=fast >TAbort- <TAbort- <MAbort- >SERR- <PERR- INTx-
IOMMU group: 62

00:07.0 Host bridge [0600]: Advanced Micro Devices, Inc. [AMD] Starship/Matisse PCIe Dummy Host Bridge [1022:1482]
Control: I/O- Mem- BusMaster- SpecCycle- MemWINV- VGASnoop- ParErr- Stepping- SERR- FastB2B- DisINTx-
Status: Cap- 66MHz- UDF- FastB2B- ParErr- DEVSEL=fast >TAbort- <TAbort- <MAbort- >SERR- <PERR- INTx-
IOMMU group: 63

00:07.1 PCI bridge [0604]: Advanced Micro Devices, Inc. [AMD] Starship/Matisse Internal PCIe GPP Bridge 0 to bus[E:B] [1022:1484] (prog-if 00 [Normal decode])
Subsystem: Advanced Micro Devices, Inc. [AMD] Starship/Matisse Internal PCIe GPP Bridge 0 to bus[E:B] [1022:1484]
Control: I/O+ Mem+ BusMaster+ SpecCycle- MemWINV- VGASnoop- ParErr- Stepping- SERR- FastB2B- DisINTx+
Status: Cap+ 66MHz- UDF- FastB2B- ParErr- DEVSEL=fast >TAbort- <TAbort- <MAbort- >SERR- <PERR- INTx-
Latency: 0, Cache Line Size: 64 bytes
Interrupt: pin A routed to IRQ 49
IOMMU group: 64
Bus: primary=00, secondary=01, subordinate=01, sec-latency=0
I/O behind bridge: [disabled] [32-bit]
Memory behind bridge: f8200000-f82fffff [size=1M] [32-bit]
Prefetchable memory behind bridge: [disabled] [64-bit]
Secondary status: 66MHz- FastB2B- ParErr- DEVSEL=fast >TAbort- <TAbort- <MAbort- <SERR- <PERR-
BridgeCtl: Parity- SERR+ NoISA- VGA- VGA16+ MAbort- >Reset- FastB2B-
PriDiscTmr- SecDiscTmr- DiscTmrStat- DiscTmrSERREn-
Capabilities: <access denied>
Kernel driver in use: pcieport

00:08.0 Host bridge [0600]: Advanced Micro Devices, Inc. [AMD] Starship/Matisse PCIe Dummy Host Bridge [1022:1482]
Control: I/O- Mem- BusMaster- SpecCycle- MemWINV- VGASnoop- ParErr- Stepping- SERR- FastB2B- DisINTx-
Status: Cap- 66MHz- UDF- FastB2B- ParErr- DEVSEL=fast >TAbort- <TAbort- <MAbort- >SERR- <PERR- INTx-
IOMMU group: 65

00:08.1 PCI bridge [0604]: Advanced Micro Devices, Inc. [AMD] Starship/Matisse Internal PCIe GPP Bridge 0 to bus[E:B] [1022:1484] (prog-if 00 [Normal decode])
Subsystem: Advanced Micro Devices, Inc. [AMD] Starship/Matisse Internal PCIe GPP Bridge 0 to bus[E:B] [1022:1484]
Control: I/O+ Mem+ BusMaster+ SpecCycle- MemWINV- VGASnoop- ParErr- Stepping- SERR- FastB2B- DisINTx+
Status: Cap+ 66MHz- UDF- FastB2B- ParErr- DEVSEL=fast >TAbort- <TAbort- <MAbort- >SERR- <PERR- INTx-
Latency: 0, Cache Line Size: 64 bytes
Interrupt: pin A routed to IRQ 50
IOMMU group: 66
Bus: primary=00, secondary=02, subordinate=02, sec-latency=0
I/O behind bridge: [disabled] [32-bit]
Memory behind bridge: f8000000-f81fffff [size=2M] [32-bit]
Prefetchable memory behind bridge: [disabled] [64-bit]
Secondary status: 66MHz- FastB2B- ParErr- DEVSEL=fast >TAbort- <TAbort- <MAbort- <SERR- <PERR-
BridgeCtl: Parity- SERR+ NoISA- VGA- VGA16+ MAbort- >Reset- FastB2B-
PriDiscTmr- SecDiscTmr- DiscTmrStat- DiscTmrSERREn-
Capabilities: <access denied>
Kernel driver in use: pcieport

00:14.0 SMBus [0c05]: Advanced Micro Devices, Inc. [AMD] FCH SMBus Controller [1022:790b] (rev 61)
Subsystem: Super Micro Computer Inc H12SSL-i [15d9:790b]
Control: I/O+ Mem+ BusMaster- SpecCycle- MemWINV- VGASnoop- ParErr- Stepping- SERR- FastB2B- DisINTx+
Status: Cap- 66MHz+ UDF- FastB2B- ParErr- DEVSEL=medium >TAbort- <TAbort- <MAbort- >SERR- <PERR- INTx-
IOMMU group: 67
Kernel driver in use: piix4_smbus
Kernel modules: i2c_piix4, sp5100_tco

00:14.3 ISA bridge [0601]: Advanced Micro Devices, Inc. [AMD] FCH LPC Bridge [1022:790e] (rev 51)
Subsystem: Super Micro Computer Inc H12SSL-i [15d9:790e]
Control: I/O+ Mem+ BusMaster+ SpecCycle+ MemWINV- VGASnoop- ParErr+ Stepping- SERR+ FastB2B- DisINTx-
Status: Cap- 66MHz+ UDF- FastB2B- ParErr- DEVSEL=medium >TAbort- <TAbort- <MAbort- >SERR- <PERR- INTx-
Latency: 0
IOMMU group: 67

00:18.0 Host bridge [0600]: Advanced Micro Devices, Inc. [AMD] Milan Data Fabric; Function 0 [1022:1650]
Control: I/O- Mem- BusMaster- SpecCycle- MemWINV- VGASnoop- ParErr- Stepping- SERR- FastB2B- DisINTx-
Status: Cap- 66MHz- UDF- FastB2B- ParErr- DEVSEL=fast >TAbort- <TAbort- <MAbort- >SERR- <PERR- INTx-
IOMMU group: 68

00:18.1 Host bridge [0600]: Advanced Micro Devices, Inc. [AMD] Milan Data Fabric; Function 1 [1022:1651]
Control: I/O- Mem- BusMaster- SpecCycle- MemWINV- VGASnoop- ParErr- Stepping- SERR- FastB2B- DisINTx-
Status: Cap- 66MHz- UDF- FastB2B- ParErr- DEVSEL=fast >TAbort- <TAbort- <MAbort- >SERR- <PERR- INTx-
IOMMU group: 68

00:18.2 Host bridge [0600]: Advanced Micro Devices, Inc. [AMD] Milan Data Fabric; Function 2 [1022:1652]
Control: I/O- Mem- BusMaster- SpecCycle- MemWINV- VGASnoop- ParErr- Stepping- SERR- FastB2B- DisINTx-
Status: Cap- 66MHz- UDF- FastB2B- ParErr- DEVSEL=fast >TAbort- <TAbort- <MAbort- >SERR- <PERR- INTx-
IOMMU group: 68

00:18.3 Host bridge [0600]: Advanced Micro Devices, Inc. [AMD] Milan Data Fabric; Function 3 [1022:1653]
Control: I/O- Mem- BusMaster- SpecCycle- MemWINV- VGASnoop- ParErr- Stepping- SERR- FastB2B- DisINTx-
Status: Cap- 66MHz- UDF- FastB2B- ParErr- DEVSEL=fast >TAbort- <TAbort- <MAbort- >SERR- <PERR- INTx-
IOMMU group: 68
Kernel driver in use: k10temp
Kernel modules: k10temp

00:18.4 Host bridge [0600]: Advanced Micro Devices, Inc. [AMD] Milan Data Fabric; Function 4 [1022:1654]
Control: I/O- Mem- BusMaster- SpecCycle- MemWINV- VGASnoop- ParErr- Stepping- SERR- FastB2B- DisINTx-
Status: Cap- 66MHz- UDF- FastB2B- ParErr- DEVSEL=fast >TAbort- <TAbort- <MAbort- >SERR- <PERR- INTx-
IOMMU group: 68

00:18.5 Host bridge [0600]: Advanced Micro Devices, Inc. [AMD] Milan Data Fabric; Function 5 [1022:1655]
Control: I/O- Mem- BusMaster- SpecCycle- MemWINV- VGASnoop- ParErr- Stepping- SERR- FastB2B- DisINTx-
Status: Cap- 66MHz- UDF- FastB2B- ParErr- DEVSEL=fast >TAbort- <TAbort- <MAbort- >SERR- <PERR- INTx-
IOMMU group: 68

00:18.6 Host bridge [0600]: Advanced Micro Devices, Inc. [AMD] Milan Data Fabric; Function 6 [1022:1656]
Control: I/O- Mem- BusMaster- SpecCycle- MemWINV- VGASnoop- ParErr- Stepping- SERR- FastB2B- DisINTx-
Status: Cap- 66MHz- UDF- FastB2B- ParErr- DEVSEL=fast >TAbort- <TAbort- <MAbort- >SERR- <PERR- INTx-
IOMMU group: 68

00:18.7 Host bridge [0600]: Advanced Micro Devices, Inc. [AMD] Milan Data Fabric; Function 7 [1022:1657]
Control: I/O- Mem- BusMaster- SpecCycle- MemWINV- VGASnoop- ParErr- Stepping- SERR- FastB2B- DisINTx-
Status: Cap- 66MHz- UDF- FastB2B- ParErr- DEVSEL=fast >TAbort- <TAbort- <MAbort- >SERR- <PERR- INTx-
IOMMU group: 68

01:00.0 Non-Essential Instrumentation [1300]: Advanced Micro Devices, Inc. [AMD] Starship/Matisse PCIe Dummy Function [1022:148a]
Subsystem: Advanced Micro Devices, Inc. [AMD] Starship/Matisse PCIe Dummy Function [1022:148a]
Control: I/O- Mem- BusMaster- SpecCycle- MemWINV- VGASnoop- ParErr- Stepping- SERR- FastB2B- DisINTx-
Status: Cap+ 66MHz- UDF- FastB2B- ParErr- DEVSEL=fast >TAbort- <TAbort- <MAbort- >SERR- <PERR- INTx-
IOMMU group: 69
Capabilities: <access denied>

01:00.2 Encryption controller [1080]: Advanced Micro Devices, Inc. [AMD] Starship/Matisse PTDMA [1022:1498]
Subsystem: Advanced Micro Devices, Inc. [AMD] Starship/Matisse PTDMA [1022:1498]
Control: I/O- Mem- BusMaster- SpecCycle- MemWINV- VGASnoop- ParErr- Stepping- SERR- FastB2B- DisINTx-
Status: Cap+ 66MHz- UDF- FastB2B- ParErr- DEVSEL=fast >TAbort- <TAbort- <MAbort- >SERR- <PERR- INTx-
Interrupt: pin B routed to IRQ 255
IOMMU group: 70
Region 2: Memory at f8200000 (32-bit, non-prefetchable) [disabled] [size=512K]
Region 5: Memory at f8280000 (32-bit, non-prefetchable) [disabled] [size=8K]
Capabilities: <access denied>

02:00.0 Non-Essential Instrumentation [1300]: Advanced Micro Devices, Inc. [AMD] Starship/Matisse Reserved SPP [1022:1485]
Subsystem: Advanced Micro Devices, Inc. [AMD] Starship/Matisse Reserved SPP [1022:1485]
Control: I/O- Mem- BusMaster- SpecCycle- MemWINV- VGASnoop- ParErr- Stepping- SERR- FastB2B- DisINTx-
Status: Cap+ 66MHz- UDF- FastB2B- ParErr- DEVSEL=fast >TAbort- <TAbort- <MAbort- >SERR- <PERR- INTx-
IOMMU group: 71
Capabilities: <access denied>

02:00.2 Encryption controller [1080]: Advanced Micro Devices, Inc. [AMD] Starship/Matisse PTDMA [1022:1498]
Subsystem: Advanced Micro Devices, Inc. [AMD] Starship/Matisse PTDMA [1022:1498]
Control: I/O- Mem- BusMaster- SpecCycle- MemWINV- VGASnoop- ParErr- Stepping- SERR- FastB2B- DisINTx-
Status: Cap+ 66MHz- UDF- FastB2B- ParErr- DEVSEL=fast >TAbort- <TAbort- <MAbort- >SERR- <PERR- INTx-
Interrupt: pin B routed to IRQ 255
IOMMU group: 72
Region 2: Memory at f8100000 (32-bit, non-prefetchable) [disabled] [size=512K]
Region 5: Memory at f8180000 (32-bit, non-prefetchable) [disabled] [size=8K]
Capabilities: <access denied>

02:00.3 USB controller [0c03]: Advanced Micro Devices, Inc. [AMD] Starship USB 3.0 Host Controller [1022:148c] (prog-if 30 [XHCI])
Subsystem: Super Micro Computer Inc H12SSL-i [15d9:145c]
Control: I/O+ Mem+ BusMaster+ SpecCycle- MemWINV- VGASnoop- ParErr- Stepping- SERR- FastB2B- DisINTx+
Status: Cap+ 66MHz- UDF- FastB2B- ParErr- DEVSEL=fast >TAbort- <TAbort- <MAbort- >SERR- <PERR- INTx-
Latency: 0, Cache Line Size: 64 bytes
Interrupt: pin C routed to IRQ 85
IOMMU group: 73
Region 0: Memory at f8000000 (64-bit, non-prefetchable) [size=1M]
Capabilities: <access denied>
Kernel driver in use: xhci_hcd
Kernel modules: xhci_pci

40:00.0 Host bridge [0600]: Advanced Micro Devices, Inc. [AMD] Starship/Matisse Root Complex [1022:1480]
Subsystem: Super Micro Computer Inc H12SSL-i [15d9:1b95]
Control: I/O- Mem- BusMaster- SpecCycle- MemWINV- VGASnoop- ParErr- Stepping- SERR- FastB2B- DisINTx-
Status: Cap- 66MHz- UDF- FastB2B- ParErr- DEVSEL=fast >TAbort- <TAbort- <MAbort- >SERR- <PERR- INTx-

40:00.2 IOMMU [0806]: Advanced Micro Devices, Inc. [AMD] Milan IOMMU [1022:164f] (rev 01)
Subsystem: Advanced Micro Devices, Inc. [AMD] Milan IOMMU [1022:164f]
Control: I/O- Mem- BusMaster- SpecCycle- MemWINV- VGASnoop- ParErr+ Stepping- SERR- FastB2B- DisINTx-
Status: Cap+ 66MHz- UDF- FastB2B- ParErr- DEVSEL=fast >TAbort- <TAbort- <MAbort- >SERR- <PERR- INTx-
Capabilities: <access denied>

40:01.0 Host bridge [0600]: Advanced Micro Devices, Inc. [AMD] Starship/Matisse PCIe Dummy Host Bridge [1022:1482]
Control: I/O- Mem- BusMaster- SpecCycle- MemWINV- VGASnoop- ParErr- Stepping- SERR- FastB2B- DisINTx-
Status: Cap- 66MHz- UDF- FastB2B- ParErr- DEVSEL=fast >TAbort- <TAbort- <MAbort- >SERR- <PERR- INTx-
IOMMU group: 31

40:02.0 Host bridge [0600]: Advanced Micro Devices, Inc. [AMD] Starship/Matisse PCIe Dummy Host Bridge [1022:1482]
Control: I/O- Mem- BusMaster- SpecCycle- MemWINV- VGASnoop- ParErr- Stepping- SERR- FastB2B- DisINTx-
Status: Cap- 66MHz- UDF- FastB2B- ParErr- DEVSEL=fast >TAbort- <TAbort- <MAbort- >SERR- <PERR- INTx-
IOMMU group: 32

40:03.0 Host bridge [0600]: Advanced Micro Devices, Inc. [AMD] Starship/Matisse PCIe Dummy Host Bridge [1022:1482]
Control: I/O- Mem- BusMaster- SpecCycle- MemWINV- VGASnoop- ParErr- Stepping- SERR- FastB2B- DisINTx-
Status: Cap- 66MHz- UDF- FastB2B- ParErr- DEVSEL=fast >TAbort- <TAbort- <MAbort- >SERR- <PERR- INTx-
IOMMU group: 33

40:03.3 PCI bridge [0604]: Advanced Micro Devices, Inc. [AMD] Starship/Matisse GPP Bridge [1022:1483] (prog-if 00 [Normal decode])
Subsystem: Advanced Micro Devices, Inc. [AMD] Device [1022:1453]
Control: I/O+ Mem+ BusMaster+ SpecCycle- MemWINV- VGASnoop- ParErr- Stepping- SERR- FastB2B- DisINTx+
Status: Cap+ 66MHz- UDF- FastB2B- ParErr- DEVSEL=fast >TAbort- <TAbort- <MAbort- >SERR- <PERR- INTx-
Latency: 0, Cache Line Size: 64 bytes
Interrupt: pin ? routed to IRQ 51
IOMMU group: 34
Bus: primary=40, secondary=41, subordinate=41, sec-latency=0
I/O behind bridge: [disabled] [32-bit]
Memory behind bridge: b3a00000-b3afffff [size=1M] [32-bit]
Prefetchable memory behind bridge: [disabled] [64-bit]
Secondary status: 66MHz- FastB2B- ParErr- DEVSEL=fast >TAbort- <TAbort- <MAbort- <SERR- <PERR-
BridgeCtl: Parity- SERR+ NoISA- VGA- VGA16+ MAbort- >Reset- FastB2B-
PriDiscTmr- SecDiscTmr- DiscTmrStat- DiscTmrSERREn-
Capabilities: <access denied>
Kernel driver in use: pcieport

40:03.4 PCI bridge [0604]: Advanced Micro Devices, Inc. [AMD] Starship/Matisse GPP Bridge [1022:1483] (prog-if 00 [Normal decode])
Subsystem: Advanced Micro Devices, Inc. [AMD] Device [1022:1453]
Control: I/O+ Mem+ BusMaster+ SpecCycle- MemWINV- VGASnoop- ParErr- Stepping- SERR- FastB2B- DisINTx+
Status: Cap+ 66MHz- UDF- FastB2B- ParErr- DEVSEL=fast >TAbort- <TAbort- <MAbort- >SERR- <PERR- INTx-
Latency: 0, Cache Line Size: 64 bytes
Interrupt: pin ? routed to IRQ 52
IOMMU group: 35
Bus: primary=40, secondary=42, subordinate=43, sec-latency=0
I/O behind bridge: 2000-2fff [size=4K] [16-bit]
Memory behind bridge: b2000000-b30fffff [size=17M] [32-bit]
Prefetchable memory behind bridge: [disabled] [64-bit]
Secondary status: 66MHz- FastB2B- ParErr- DEVSEL=fast >TAbort- <TAbort- <MAbort- <SERR- <PERR-
BridgeCtl: Parity- SERR+ NoISA- VGA+ VGA16+ MAbort- >Reset- FastB2B-
PriDiscTmr- SecDiscTmr- DiscTmrStat- DiscTmrSERREn-
Capabilities: <access denied>
Kernel driver in use: pcieport

40:03.5 PCI bridge [0604]: Advanced Micro Devices, Inc. [AMD] Starship/Matisse GPP Bridge [1022:1483] (prog-if 00 [Normal decode])
Subsystem: Advanced Micro Devices, Inc. [AMD] Device [1022:1453]
Control: I/O+ Mem+ BusMaster+ SpecCycle- MemWINV- VGASnoop- ParErr- Stepping- SERR- FastB2B- DisINTx+
Status: Cap+ 66MHz- UDF- FastB2B- ParErr- DEVSEL=fast >TAbort- <TAbort- <MAbort- >SERR- <PERR- INTx-
Latency: 0, Cache Line Size: 64 bytes
Interrupt: pin ? routed to IRQ 53
IOMMU group: 36
Bus: primary=40, secondary=44, subordinate=44, sec-latency=0
I/O behind bridge: [disabled] [32-bit]
Memory behind bridge: b3900000-b39fffff [size=1M] [32-bit]
Prefetchable memory behind bridge: [disabled] [64-bit]
Secondary status: 66MHz- FastB2B- ParErr- DEVSEL=fast >TAbort- <TAbort- <MAbort- <SERR- <PERR-
BridgeCtl: Parity- SERR+ NoISA- VGA- VGA16+ MAbort- >Reset- FastB2B-
PriDiscTmr- SecDiscTmr- DiscTmrStat- DiscTmrSERREn-
Capabilities: <access denied>
Kernel driver in use: pcieport

40:03.6 PCI bridge [0604]: Advanced Micro Devices, Inc. [AMD] Starship/Matisse GPP Bridge [1022:1483] (prog-if 00 [Normal decode])
Subsystem: Advanced Micro Devices, Inc. [AMD] Device [1022:1453]
Control: I/O+ Mem+ BusMaster+ SpecCycle- MemWINV- VGASnoop- ParErr- Stepping- SERR- FastB2B- DisINTx+
Status: Cap+ 66MHz- UDF- FastB2B- ParErr- DEVSEL=fast >TAbort- <TAbort- <MAbort- >SERR- <PERR- INTx-
Latency: 0, Cache Line Size: 64 bytes
Interrupt: pin ? routed to IRQ 54
IOMMU group: 37
Bus: primary=40, secondary=45, subordinate=45, sec-latency=0
I/O behind bridge: [disabled] [32-bit]
Memory behind bridge: b3800000-b38fffff [size=1M] [32-bit]
Prefetchable memory behind bridge: 28080100000-280801fffff [size=1M] [32-bit]
Secondary status: 66MHz- FastB2B- ParErr- DEVSEL=fast >TAbort- <TAbort- <MAbort- <SERR- <PERR-
BridgeCtl: Parity- SERR+ NoISA- VGA- VGA16+ MAbort- >Reset- FastB2B-
PriDiscTmr- SecDiscTmr- DiscTmrStat- DiscTmrSERREn-
Capabilities: <access denied>
Kernel driver in use: pcieport

40:04.0 Host bridge [0600]: Advanced Micro Devices, Inc. [AMD] Starship/Matisse PCIe Dummy Host Bridge [1022:1482]
Control: I/O- Mem- BusMaster- SpecCycle- MemWINV- VGASnoop- ParErr- Stepping- SERR- FastB2B- DisINTx-
Status: Cap- 66MHz- UDF- FastB2B- ParErr- DEVSEL=fast >TAbort- <TAbort- <MAbort- >SERR- <PERR- INTx-
IOMMU group: 38

40:05.0 Host bridge [0600]: Advanced Micro Devices, Inc. [AMD] Starship/Matisse PCIe Dummy Host Bridge [1022:1482]
Control: I/O- Mem- BusMaster- SpecCycle- MemWINV- VGASnoop- ParErr- Stepping- SERR- FastB2B- DisINTx-
Status: Cap- 66MHz- UDF- FastB2B- ParErr- DEVSEL=fast >TAbort- <TAbort- <MAbort- >SERR- <PERR- INTx-
IOMMU group: 39

40:07.0 Host bridge [0600]: Advanced Micro Devices, Inc. [AMD] Starship/Matisse PCIe Dummy Host Bridge [1022:1482]
Control: I/O- Mem- BusMaster- SpecCycle- MemWINV- VGASnoop- ParErr- Stepping- SERR- FastB2B- DisINTx-
Status: Cap- 66MHz- UDF- FastB2B- ParErr- DEVSEL=fast >TAbort- <TAbort- <MAbort- >SERR- <PERR- INTx-
IOMMU group: 40

40:07.1 PCI bridge [0604]: Advanced Micro Devices, Inc. [AMD] Starship/Matisse Internal PCIe GPP Bridge 0 to bus[E:B] [1022:1484] (prog-if 00 [Normal decode])
Subsystem: Advanced Micro Devices, Inc. [AMD] Starship/Matisse Internal PCIe GPP Bridge 0 to bus[E:B] [1022:1484]
Control: I/O+ Mem+ BusMaster+ SpecCycle- MemWINV- VGASnoop- ParErr- Stepping- SERR- FastB2B- DisINTx+
Status: Cap+ 66MHz- UDF- FastB2B- ParErr- DEVSEL=fast >TAbort- <TAbort- <MAbort- >SERR- <PERR- INTx-
Latency: 0, Cache Line Size: 64 bytes
Interrupt: pin A routed to IRQ 56
IOMMU group: 41
Bus: primary=40, secondary=46, subordinate=46, sec-latency=0
I/O behind bridge: [disabled] [32-bit]
Memory behind bridge: b3700000-b37fffff [size=1M] [32-bit]
Prefetchable memory behind bridge: [disabled] [64-bit]
Secondary status: 66MHz- FastB2B- ParErr- DEVSEL=fast >TAbort- <TAbort- <MAbort- <SERR- <PERR-
BridgeCtl: Parity- SERR+ NoISA- VGA- VGA16+ MAbort- >Reset- FastB2B-
PriDiscTmr- SecDiscTmr- DiscTmrStat- DiscTmrSERREn-
Capabilities: <access denied>
Kernel driver in use: pcieport

40:08.0 Host bridge [0600]: Advanced Micro Devices, Inc. [AMD] Starship/Matisse PCIe Dummy Host Bridge [1022:1482]
Control: I/O- Mem- BusMaster- SpecCycle- MemWINV- VGASnoop- ParErr- Stepping- SERR- FastB2B- DisINTx-
Status: Cap- 66MHz- UDF- FastB2B- ParErr- DEVSEL=fast >TAbort- <TAbort- <MAbort- >SERR- <PERR- INTx-
IOMMU group: 42

40:08.1 PCI bridge [0604]: Advanced Micro Devices, Inc. [AMD] Starship/Matisse Internal PCIe GPP Bridge 0 to bus[E:B] [1022:1484] (prog-if 00 [Normal decode])
Subsystem: Advanced Micro Devices, Inc. [AMD] Starship/Matisse Internal PCIe GPP Bridge 0 to bus[E:B] [1022:1484]
Control: I/O+ Mem+ BusMaster+ SpecCycle- MemWINV- VGASnoop- ParErr- Stepping- SERR- FastB2B- DisINTx+
Status: Cap+ 66MHz- UDF- FastB2B- ParErr- DEVSEL=fast >TAbort- <TAbort- <MAbort- >SERR- <PERR- INTx-
Latency: 0, Cache Line Size: 64 bytes
Interrupt: pin A routed to IRQ 57
IOMMU group: 43
Bus: primary=40, secondary=47, subordinate=47, sec-latency=0
I/O behind bridge: [disabled] [32-bit]
Memory behind bridge: b3200000-b34fffff [size=3M] [32-bit]
Prefetchable memory behind bridge: [disabled] [64-bit]
Secondary status: 66MHz- FastB2B- ParErr- DEVSEL=fast >TAbort- <TAbort- <MAbort- <SERR- <PERR-
BridgeCtl: Parity- SERR+ NoISA- VGA- VGA16+ MAbort- >Reset- FastB2B-
PriDiscTmr- SecDiscTmr- DiscTmrStat- DiscTmrSERREn-
Capabilities: <access denied>
Kernel driver in use: pcieport

40:08.2 PCI bridge [0604]: Advanced Micro Devices, Inc. [AMD] Starship/Matisse Internal PCIe GPP Bridge 0 to bus[E:B] [1022:1484] (prog-if 00 [Normal decode])
Subsystem: Advanced Micro Devices, Inc. [AMD] Starship/Matisse Internal PCIe GPP Bridge 0 to bus[E:B] [1022:1484]
Control: I/O+ Mem+ BusMaster+ SpecCycle- MemWINV- VGASnoop- ParErr- Stepping- SERR- FastB2B- DisINTx+
Status: Cap+ 66MHz- UDF- FastB2B- ParErr- DEVSEL=fast >TAbort- <TAbort- <MAbort- >SERR- <PERR- INTx-
Latency: 0, Cache Line Size: 64 bytes
Interrupt: pin A routed to IRQ 58
IOMMU group: 44
Bus: primary=40, secondary=48, subordinate=48, sec-latency=0
I/O behind bridge: [disabled] [32-bit]
Memory behind bridge: b3600000-b36fffff [size=1M] [32-bit]
Prefetchable memory behind bridge: [disabled] [64-bit]
Secondary status: 66MHz- FastB2B- ParErr- DEVSEL=fast >TAbort- <TAbort- <MAbort- <SERR- <PERR-
BridgeCtl: Parity- SERR+ NoISA- VGA- VGA16+ MAbort- >Reset- FastB2B-
PriDiscTmr- SecDiscTmr- DiscTmrStat- DiscTmrSERREn-
Capabilities: <access denied>
Kernel driver in use: pcieport

40:08.3 PCI bridge [0604]: Advanced Micro Devices, Inc. [AMD] Starship/Matisse Internal PCIe GPP Bridge 0 to bus[E:B] [1022:1484] (prog-if 00 [Normal decode])
Subsystem: Advanced Micro Devices, Inc. [AMD] Starship/Matisse Internal PCIe GPP Bridge 0 to bus[E:B] [1022:1484]
Control: I/O+ Mem+ BusMaster+ SpecCycle- MemWINV- VGASnoop- ParErr- Stepping- SERR- FastB2B- DisINTx+
Status: Cap+ 66MHz- UDF- FastB2B- ParErr- DEVSEL=fast >TAbort- <TAbort- <MAbort- >SERR- <PERR- INTx-
Latency: 0, Cache Line Size: 64 bytes
Interrupt: pin A routed to IRQ 59
IOMMU group: 45
Bus: primary=40, secondary=49, subordinate=49, sec-latency=0
I/O behind bridge: [disabled] [32-bit]
Memory behind bridge: b3500000-b35fffff [size=1M] [32-bit]
Prefetchable memory behind bridge: [disabled] [64-bit]
Secondary status: 66MHz- FastB2B- ParErr- DEVSEL=fast >TAbort- <TAbort- <MAbort- <SERR- <PERR-
BridgeCtl: Parity- SERR+ NoISA- VGA- VGA16+ MAbort- >Reset- FastB2B-
PriDiscTmr- SecDiscTmr- DiscTmrStat- DiscTmrSERREn-
Capabilities: <access denied>
Kernel driver in use: pcieport

41:00.0 USB controller [0c03]: ASMedia Technology Inc. ASM1042A USB 3.0 Host Controller [1b21:1142] (prog-if 30 [XHCI])
Subsystem: ASMedia Technology Inc. ASM1042A USB 3.0 Host Controller [1b21:1142]
Control: I/O- Mem+ BusMaster+ SpecCycle- MemWINV- VGASnoop- ParErr- Stepping- SERR- FastB2B- DisINTx+
Status: Cap+ 66MHz- UDF- FastB2B- ParErr- DEVSEL=fast >TAbort- <TAbort- <MAbort- >SERR- <PERR- INTx-
Latency: 0, Cache Line Size: 64 bytes
Interrupt: pin A routed to IRQ 94
IOMMU group: 46
Region 0: Memory at b3a00000 (64-bit, non-prefetchable) [size=32K]
Capabilities: <access denied>
Kernel driver in use: xhci_hcd
Kernel modules: xhci_pci

42:00.0 PCI bridge [0604]: ASPEED Technology, Inc. AST1150 PCI-to-PCI Bridge [1a03:1150] (rev 04) (prog-if 00 [Normal decode])
Subsystem: ASPEED Technology, Inc. AST1150 PCI-to-PCI Bridge [1a03:1150]
Control: I/O+ Mem+ BusMaster+ SpecCycle- MemWINV- VGASnoop- ParErr- Stepping- SERR- FastB2B- DisINTx-
Status: Cap+ 66MHz- UDF- FastB2B- ParErr- DEVSEL=fast >TAbort- <TAbort- <MAbort- >SERR- <PERR- INTx-
Latency: 0, Cache Line Size: 64 bytes
Interrupt: pin A routed to IRQ 164
IOMMU group: 47
Bus: primary=42, secondary=43, subordinate=43, sec-latency=32
I/O behind bridge: 2000-2fff [size=4K] [16-bit]
Memory behind bridge: b2000000-b30fffff [size=17M] [32-bit]
Prefetchable memory behind bridge: [disabled] [64-bit]
Secondary status: 66MHz+ FastB2B- ParErr- DEVSEL=medium >TAbort- <TAbort- <MAbort- <SERR- <PERR-
BridgeCtl: Parity- SERR+ NoISA- VGA+ VGA16+ MAbort- >Reset- FastB2B-
PriDiscTmr- SecDiscTmr- DiscTmrStat- DiscTmrSERREn-
Capabilities: <access denied>

43:00.0 VGA compatible controller [0300]: ASPEED Technology, Inc. ASPEED Graphics Family [1a03:2000] (rev 41) (prog-if 00 [VGA controller])
DeviceName: ASPEED Video AST2500
Subsystem: Super Micro Computer Inc H12SSL-i [15d9:1b95]
Control: I/O+ Mem+ BusMaster- SpecCycle- MemWINV- VGASnoop- ParErr- Stepping- SERR- FastB2B- DisINTx-
Status: Cap+ 66MHz- UDF- FastB2B- ParErr- DEVSEL=medium >TAbort- <TAbort- <MAbort- >SERR- <PERR- INTx-
Interrupt: pin A routed to IRQ 164
IOMMU group: 47
Region 0: Memory at b2000000 (32-bit, non-prefetchable) [size=16M]
Region 1: Memory at b3000000 (32-bit, non-prefetchable) [size=128K]
Region 2: I/O ports at 2000 [size=128]
Expansion ROM at 000c0000 [virtual] [disabled] [size=128K]
Capabilities: <access denied>
Kernel driver in use: ast
Kernel modules: ast

44:00.0 USB controller [0c03]: ASMedia Technology Inc. ASM1042A USB 3.0 Host Controller [1b21:1142] (prog-if 30 [XHCI])
Subsystem: ASMedia Technology Inc. ASM1042A USB 3.0 Host Controller [1b21:1142]
Control: I/O- Mem+ BusMaster+ SpecCycle- MemWINV- VGASnoop- ParErr- Stepping- SERR- FastB2B- DisINTx+
Status: Cap+ 66MHz- UDF- FastB2B- ParErr- DEVSEL=fast >TAbort- <TAbort- <MAbort- >SERR- <PERR- INTx-
Latency: 0, Cache Line Size: 64 bytes
Interrupt: pin A routed to IRQ 137
IOMMU group: 48
Region 0: Memory at b3900000 (64-bit, non-prefetchable) [size=32K]
Capabilities: <access denied>
Kernel driver in use: xhci_hcd
Kernel modules: xhci_pci

45:00.0 Ethernet controller [0200]: Broadcom Inc. and subsidiaries NetXtreme BCM5720 Gigabit Ethernet PCIe [14e4:165f]
DeviceName: Broadcom 1G Ethernet #1
Subsystem: Super Micro Computer Inc H12SSL-i [15d9:165f]
Control: I/O- Mem+ BusMaster+ SpecCycle- MemWINV- VGASnoop- ParErr- Stepping- SERR- FastB2B- DisINTx+
Status: Cap+ 66MHz- UDF- FastB2B- ParErr- DEVSEL=fast >TAbort- <TAbort- <MAbort- >SERR- <PERR- INTx-
Latency: 0, Cache Line Size: 64 bytes
Interrupt: pin A routed to IRQ 55
IOMMU group: 49
Region 0: Memory at 28080150000 (64-bit, prefetchable) [size=64K]
Region 2: Memory at 28080140000 (64-bit, prefetchable) [size=64K]
Region 4: Memory at 28080130000 (64-bit, prefetchable) [size=64K]
Expansion ROM at b3840000 [disabled] [size=256K]
Capabilities: <access denied>
Kernel driver in use: tg3
Kernel modules: tg3

45:00.1 Ethernet controller [0200]: Broadcom Inc. and subsidiaries NetXtreme BCM5720 Gigabit Ethernet PCIe [14e4:165f]
DeviceName: Broadcom 1G Ethernet #2
Subsystem: Super Micro Computer Inc H12SSL-i [15d9:165f]
Control: I/O- Mem+ BusMaster+ SpecCycle- MemWINV- VGASnoop- ParErr- Stepping- SERR- FastB2B- DisINTx+
Status: Cap+ 66MHz- UDF- FastB2B- ParErr- DEVSEL=fast >TAbort- <TAbort- <MAbort- >SERR- <PERR- INTx-
Latency: 0, Cache Line Size: 64 bytes
Interrupt: pin B routed to IRQ 25
IOMMU group: 49
Region 0: Memory at 28080120000 (64-bit, prefetchable) [size=64K]
Region 2: Memory at 28080110000 (64-bit, prefetchable) [size=64K]
Region 4: Memory at 28080100000 (64-bit, prefetchable) [size=64K]
Expansion ROM at b3800000 [disabled] [size=256K]
Capabilities: <access denied>
Kernel driver in use: tg3
Kernel modules: tg3

46:00.0 Non-Essential Instrumentation [1300]: Advanced Micro Devices, Inc. [AMD] Starship/Matisse PCIe Dummy Function [1022:148a]
Subsystem: Advanced Micro Devices, Inc. [AMD] Starship/Matisse PCIe Dummy Function [1022:148a]
Control: I/O- Mem- BusMaster- SpecCycle- MemWINV- VGASnoop- ParErr- Stepping- SERR- FastB2B- DisINTx-
Status: Cap+ 66MHz- UDF- FastB2B- ParErr- DEVSEL=fast >TAbort- <TAbort- <MAbort- >SERR- <PERR- INTx-
IOMMU group: 50
Capabilities: <access denied>

46:00.2 Encryption controller [1080]: Advanced Micro Devices, Inc. [AMD] Starship/Matisse PTDMA [1022:1498]
Subsystem: Advanced Micro Devices, Inc. [AMD] Starship/Matisse PTDMA [1022:1498]
Control: I/O- Mem- BusMaster- SpecCycle- MemWINV- VGASnoop- ParErr- Stepping- SERR- FastB2B- DisINTx-
Status: Cap+ 66MHz- UDF- FastB2B- ParErr- DEVSEL=fast >TAbort- <TAbort- <MAbort- >SERR- <PERR- INTx-
Interrupt: pin B routed to IRQ 255
IOMMU group: 51
Region 2: Memory at b3700000 (32-bit, non-prefetchable) [disabled] [size=512K]
Region 5: Memory at b3780000 (32-bit, non-prefetchable) [disabled] [size=8K]
Capabilities: <access denied>

47:00.0 Non-Essential Instrumentation [1300]: Advanced Micro Devices, Inc. [AMD] Starship/Matisse Reserved SPP [1022:1485]
Subsystem: Advanced Micro Devices, Inc. [AMD] Starship/Matisse Reserved SPP [1022:1485]
Control: I/O- Mem- BusMaster- SpecCycle- MemWINV- VGASnoop- ParErr- Stepping- SERR- FastB2B- DisINTx-
Status: Cap+ 66MHz- UDF- FastB2B- ParErr- DEVSEL=fast >TAbort- <TAbort- <MAbort- >SERR- <PERR- INTx-
IOMMU group: 52
Capabilities: <access denied>

47:00.1 Encryption controller [1080]: Advanced Micro Devices, Inc. [AMD] Starship/Matisse Cryptographic Coprocessor PSPCPP [1022:1486]
Subsystem: Advanced Micro Devices, Inc. [AMD] Starship/Matisse Cryptographic Coprocessor PSPCPP [1022:1486]
Control: I/O- Mem+ BusMaster+ SpecCycle- MemWINV- VGASnoop- ParErr- Stepping- SERR- FastB2B- DisINTx+
Status: Cap+ 66MHz- UDF- FastB2B- ParErr- DEVSEL=fast >TAbort- <TAbort- <MAbort- >SERR- <PERR- INTx-
Latency: 0, Cache Line Size: 64 bytes
Interrupt: pin A routed to IRQ 161
IOMMU group: 53
Region 2: Memory at b3300000 (32-bit, non-prefetchable) [size=1M]
Region 5: Memory at b3482000 (32-bit, non-prefetchable) [size=8K]
Capabilities: <access denied>
Kernel driver in use: ccp
Kernel modules: ccp

47:00.2 Encryption controller [1080]: Advanced Micro Devices, Inc. [AMD] Starship/Matisse PTDMA [1022:1498]
Subsystem: Advanced Micro Devices, Inc. [AMD] Starship/Matisse PTDMA [1022:1498]
Control: I/O- Mem- BusMaster- SpecCycle- MemWINV- VGASnoop- ParErr- Stepping- SERR- FastB2B- DisINTx-
Status: Cap+ 66MHz- UDF- FastB2B- ParErr- DEVSEL=fast >TAbort- <TAbort- <MAbort- >SERR- <PERR- INTx-
Interrupt: pin B routed to IRQ 255
IOMMU group: 54
Region 2: Memory at b3400000 (32-bit, non-prefetchable) [disabled] [size=512K]
Region 5: Memory at b3480000 (32-bit, non-prefetchable) [disabled] [size=8K]
Capabilities: <access denied>

47:00.3 USB controller [0c03]: Advanced Micro Devices, Inc. [AMD] Starship USB 3.0 Host Controller [1022:148c] (prog-if 30 [XHCI])
Subsystem: Super Micro Computer Inc H12SSL-i [15d9:145c]
Control: I/O+ Mem+ BusMaster+ SpecCycle- MemWINV- VGASnoop- ParErr- Stepping- SERR- FastB2B- DisINTx+
Status: Cap+ 66MHz- UDF- FastB2B- ParErr- DEVSEL=fast >TAbort- <TAbort- <MAbort- >SERR- <PERR- INTx-
Latency: 0, Cache Line Size: 64 bytes
Interrupt: pin C routed to IRQ 147
IOMMU group: 55
Region 0: Memory at b3200000 (64-bit, non-prefetchable) [size=1M]
Capabilities: <access denied>
Kernel driver in use: xhci_hcd
Kernel modules: xhci_pci

48:00.0 SATA controller [0106]: Advanced Micro Devices, Inc. [AMD] FCH SATA Controller [AHCI mode] [1022:7901] (rev 51) (prog-if 01 [AHCI 1.0])
Subsystem: Super Micro Computer Inc H12SSL-i [15d9:7901]
Control: I/O+ Mem+ BusMaster+ SpecCycle- MemWINV- VGASnoop- ParErr- Stepping- SERR- FastB2B- DisINTx+
Status: Cap+ 66MHz- UDF- FastB2B- ParErr- DEVSEL=fast >TAbort- <TAbort- <MAbort- >SERR- <PERR- INTx-
Latency: 0, Cache Line Size: 64 bytes
Interrupt: pin A routed to IRQ 96
IOMMU group: 56
Region 5: Memory at b3600000 (32-bit, non-prefetchable) [size=2K]
Capabilities: <access denied>
Kernel driver in use: ahci
Kernel modules: ahci

49:00.0 SATA controller [0106]: Advanced Micro Devices, Inc. [AMD] FCH SATA Controller [AHCI mode] [1022:7901] (rev 51) (prog-if 01 [AHCI 1.0])
Subsystem: Super Micro Computer Inc H12SSL-i [15d9:7901]
Control: I/O+ Mem+ BusMaster+ SpecCycle- MemWINV- VGASnoop- ParErr- Stepping- SERR- FastB2B- DisINTx+
Status: Cap+ 66MHz- UDF- FastB2B- ParErr- DEVSEL=fast >TAbort- <TAbort- <MAbort- >SERR- <PERR- INTx-
Latency: 0, Cache Line Size: 64 bytes
Interrupt: pin A routed to IRQ 113
IOMMU group: 57
Region 5: Memory at b3500000 (32-bit, non-prefetchable) [size=2K]
Capabilities: <access denied>
Kernel driver in use: ahci
Kernel modules: ahci

80:00.0 Host bridge [0600]: Advanced Micro Devices, Inc. [AMD] Starship/Matisse Root Complex [1022:1480]
Subsystem: Super Micro Computer Inc H12SSL-i [15d9:1b95]
Control: I/O- Mem- BusMaster- SpecCycle- MemWINV- VGASnoop- ParErr- Stepping- SERR- FastB2B- DisINTx-
Status: Cap- 66MHz- UDF- FastB2B- ParErr- DEVSEL=fast >TAbort- <TAbort- <MAbort- >SERR- <PERR- INTx-

80:00.2 IOMMU [0806]: Advanced Micro Devices, Inc. [AMD] Milan IOMMU [1022:164f] (rev 01)
Subsystem: Advanced Micro Devices, Inc. [AMD] Milan IOMMU [1022:164f]
Control: I/O- Mem- BusMaster- SpecCycle- MemWINV- VGASnoop- ParErr+ Stepping- SERR- FastB2B- DisINTx-
Status: Cap+ 66MHz- UDF- FastB2B- ParErr- DEVSEL=fast >TAbort- <TAbort- <MAbort- >SERR- <PERR- INTx-
Capabilities: <access denied>

80:01.0 Host bridge [0600]: Advanced Micro Devices, Inc. [AMD] Starship/Matisse PCIe Dummy Host Bridge [1022:1482]
Control: I/O- Mem- BusMaster- SpecCycle- MemWINV- VGASnoop- ParErr- Stepping- SERR- FastB2B- DisINTx-
Status: Cap- 66MHz- UDF- FastB2B- ParErr- DEVSEL=fast >TAbort- <TAbort- <MAbort- >SERR- <PERR- INTx-
IOMMU group: 15

80:01.1 PCI bridge [0604]: Advanced Micro Devices, Inc. [AMD] Starship/Matisse GPP Bridge [1022:1483] (prog-if 00 [Normal decode])
Subsystem: Advanced Micro Devices, Inc. [AMD] Device [1022:1453]
Control: I/O+ Mem+ BusMaster+ SpecCycle- MemWINV- VGASnoop- ParErr- Stepping- SERR- FastB2B- DisINTx+
Status: Cap+ 66MHz- UDF- FastB2B- ParErr- DEVSEL=fast >TAbort- <TAbort- <MAbort- >SERR- <PERR- INTx-
Latency: 0, Cache Line Size: 64 bytes
Interrupt: pin ? routed to IRQ 43
IOMMU group: 16
Bus: primary=80, secondary=81, subordinate=81, sec-latency=0
I/O behind bridge: [disabled] [32-bit]
Memory behind bridge: f2000000-f21fffff [size=2M] [32-bit]
Prefetchable memory behind bridge: 2004c000000-2004fffffff [size=64M] [32-bit]
Secondary status: 66MHz- FastB2B- ParErr- DEVSEL=fast >TAbort- <TAbort- <MAbort- <SERR- <PERR-
BridgeCtl: Parity- SERR+ NoISA- VGA- VGA16+ MAbort- >Reset- FastB2B-
PriDiscTmr- SecDiscTmr- DiscTmrStat- DiscTmrSERREn-
Capabilities: <access denied>
Kernel driver in use: pcieport

80:02.0 Host bridge [0600]: Advanced Micro Devices, Inc. [AMD] Starship/Matisse PCIe Dummy Host Bridge [1022:1482]
Control: I/O- Mem- BusMaster- SpecCycle- MemWINV- VGASnoop- ParErr- Stepping- SERR- FastB2B- DisINTx-
Status: Cap- 66MHz- UDF- FastB2B- ParErr- DEVSEL=fast >TAbort- <TAbort- <MAbort- >SERR- <PERR- INTx-
IOMMU group: 17

80:03.0 Host bridge [0600]: Advanced Micro Devices, Inc. [AMD] Starship/Matisse PCIe Dummy Host Bridge [1022:1482]
Control: I/O- Mem- BusMaster- SpecCycle- MemWINV- VGASnoop- ParErr- Stepping- SERR- FastB2B- DisINTx-
Status: Cap- 66MHz- UDF- FastB2B- ParErr- DEVSEL=fast >TAbort- <TAbort- <MAbort- >SERR- <PERR- INTx-
IOMMU group: 18

80:04.0 Host bridge [0600]: Advanced Micro Devices, Inc. [AMD] Starship/Matisse PCIe Dummy Host Bridge [1022:1482]
Control: I/O- Mem- BusMaster- SpecCycle- MemWINV- VGASnoop- ParErr- Stepping- SERR- FastB2B- DisINTx-
Status: Cap- 66MHz- UDF- FastB2B- ParErr- DEVSEL=fast >TAbort- <TAbort- <MAbort- >SERR- <PERR- INTx-
IOMMU group: 19

80:05.0 Host bridge [0600]: Advanced Micro Devices, Inc. [AMD] Starship/Matisse PCIe Dummy Host Bridge [1022:1482]
Control: I/O- Mem- BusMaster- SpecCycle- MemWINV- VGASnoop- ParErr- Stepping- SERR- FastB2B- DisINTx-
Status: Cap- 66MHz- UDF- FastB2B- ParErr- DEVSEL=fast >TAbort- <TAbort- <MAbort- >SERR- <PERR- INTx-
IOMMU group: 20

80:07.0 Host bridge [0600]: Advanced Micro Devices, Inc. [AMD] Starship/Matisse PCIe Dummy Host Bridge [1022:1482]
Control: I/O- Mem- BusMaster- SpecCycle- MemWINV- VGASnoop- ParErr- Stepping- SERR- FastB2B- DisINTx-
Status: Cap- 66MHz- UDF- FastB2B- ParErr- DEVSEL=fast >TAbort- <TAbort- <MAbort- >SERR- <PERR- INTx-
IOMMU group: 21

80:07.1 PCI bridge [0604]: Advanced Micro Devices, Inc. [AMD] Starship/Matisse Internal PCIe GPP Bridge 0 to bus[E:B] [1022:1484] (prog-if 00 [Normal decode])
Subsystem: Advanced Micro Devices, Inc. [AMD] Starship/Matisse Internal PCIe GPP Bridge 0 to bus[E:B] [1022:1484]
Control: I/O+ Mem+ BusMaster+ SpecCycle- MemWINV- VGASnoop- ParErr- Stepping- SERR- FastB2B- DisINTx+
Status: Cap+ 66MHz- UDF- FastB2B- ParErr- DEVSEL=fast >TAbort- <TAbort- <MAbort- >SERR- <PERR- INTx-
Latency: 0, Cache Line Size: 64 bytes
Interrupt: pin A routed to IRQ 45
IOMMU group: 22
Bus: primary=80, secondary=82, subordinate=82, sec-latency=0
I/O behind bridge: [disabled] [32-bit]
Memory behind bridge: f2300000-f23fffff [size=1M] [32-bit]
Prefetchable memory behind bridge: [disabled] [64-bit]
Secondary status: 66MHz- FastB2B- ParErr- DEVSEL=fast >TAbort- <TAbort- <MAbort- <SERR- <PERR-
BridgeCtl: Parity- SERR+ NoISA- VGA- VGA16+ MAbort- >Reset- FastB2B-
PriDiscTmr- SecDiscTmr- DiscTmrStat- DiscTmrSERREn-
Capabilities: <access denied>
Kernel driver in use: pcieport

80:08.0 Host bridge [0600]: Advanced Micro Devices, Inc. [AMD] Starship/Matisse PCIe Dummy Host Bridge [1022:1482]
Control: I/O- Mem- BusMaster- SpecCycle- MemWINV- VGASnoop- ParErr- Stepping- SERR- FastB2B- DisINTx-
Status: Cap- 66MHz- UDF- FastB2B- ParErr- DEVSEL=fast >TAbort- <TAbort- <MAbort- >SERR- <PERR- INTx-
IOMMU group: 23

80:08.1 PCI bridge [0604]: Advanced Micro Devices, Inc. [AMD] Starship/Matisse Internal PCIe GPP Bridge 0 to bus[E:B] [1022:1484] (prog-if 00 [Normal decode])
Subsystem: Advanced Micro Devices, Inc. [AMD] Starship/Matisse Internal PCIe GPP Bridge 0 to bus[E:B] [1022:1484]
Control: I/O+ Mem+ BusMaster+ SpecCycle- MemWINV- VGASnoop- ParErr- Stepping- SERR- FastB2B- DisINTx+
Status: Cap+ 66MHz- UDF- FastB2B- ParErr- DEVSEL=fast >TAbort- <TAbort- <MAbort- >SERR- <PERR- INTx-
Latency: 0, Cache Line Size: 64 bytes
Interrupt: pin A routed to IRQ 47
IOMMU group: 24
Bus: primary=80, secondary=83, subordinate=83, sec-latency=0
I/O behind bridge: [disabled] [32-bit]
Memory behind bridge: f2200000-f22fffff [size=1M] [32-bit]
Prefetchable memory behind bridge: [disabled] [64-bit]
Secondary status: 66MHz- FastB2B- ParErr- DEVSEL=fast >TAbort- <TAbort- <MAbort- <SERR- <PERR-
BridgeCtl: Parity- SERR+ NoISA- VGA- VGA16+ MAbort- >Reset- FastB2B-
PriDiscTmr- SecDiscTmr- DiscTmrStat- DiscTmrSERREn-
Capabilities: <access denied>
Kernel driver in use: pcieport

81:00.0 Infiniband controller [0207]: Mellanox Technologies MT28908 Family [ConnectX-6] [15b3:101b]
Subsystem: Mellanox Technologies Device [15b3:0008]
Control: I/O- Mem+ BusMaster+ SpecCycle- MemWINV- VGASnoop- ParErr- Stepping- SERR- FastB2B- DisINTx+
Status: Cap+ 66MHz- UDF- FastB2B- ParErr- DEVSEL=fast >TAbort- <TAbort- <MAbort- >SERR- <PERR- INTx-
Latency: 0, Cache Line Size: 64 bytes
Interrupt: pin A routed to IRQ 138
IOMMU group: 25
Region 0: Memory at 2004e000000 (64-bit, prefetchable) [size=32M]
Expansion ROM at f2100000 [disabled] [size=1M]
Capabilities: <access denied>
Kernel driver in use: mlx5_core
Kernel modules: mlx5_core

81:00.1 Infiniband controller [0207]: Mellanox Technologies MT28908 Family [ConnectX-6] [15b3:101b]
Subsystem: Mellanox Technologies Device [15b3:0008]
Control: I/O- Mem+ BusMaster+ SpecCycle- MemWINV- VGASnoop- ParErr- Stepping- SERR- FastB2B- DisINTx+
Status: Cap+ 66MHz- UDF- FastB2B- ParErr- DEVSEL=fast >TAbort- <TAbort- <MAbort- >SERR- <PERR- INTx-
Latency: 0, Cache Line Size: 64 bytes
Interrupt: pin B routed to IRQ 157
IOMMU group: 26
Region 0: Memory at 2004c000000 (64-bit, prefetchable) [size=32M]
Expansion ROM at f2000000 [disabled] [size=1M]
Capabilities: <access denied>
Kernel driver in use: mlx5_core
Kernel modules: mlx5_core

82:00.0 Non-Essential Instrumentation [1300]: Advanced Micro Devices, Inc. [AMD] Starship/Matisse PCIe Dummy Function [1022:148a]
Subsystem: Advanced Micro Devices, Inc. [AMD] Starship/Matisse PCIe Dummy Function [1022:148a]
Control: I/O- Mem- BusMaster- SpecCycle- MemWINV- VGASnoop- ParErr- Stepping- SERR- FastB2B- DisINTx-
Status: Cap+ 66MHz- UDF- FastB2B- ParErr- DEVSEL=fast >TAbort- <TAbort- <MAbort- >SERR- <PERR- INTx-
IOMMU group: 27
Capabilities: <access denied>

82:00.2 Encryption controller [1080]: Advanced Micro Devices, Inc. [AMD] Starship/Matisse PTDMA [1022:1498]
Subsystem: Advanced Micro Devices, Inc. [AMD] Starship/Matisse PTDMA [1022:1498]
Control: I/O- Mem- BusMaster- SpecCycle- MemWINV- VGASnoop- ParErr- Stepping- SERR- FastB2B- DisINTx-
Status: Cap+ 66MHz- UDF- FastB2B- ParErr- DEVSEL=fast >TAbort- <TAbort- <MAbort- >SERR- <PERR- INTx-
Interrupt: pin B routed to IRQ 255
IOMMU group: 28
Region 2: Memory at f2300000 (32-bit, non-prefetchable) [disabled] [size=512K]
Region 5: Memory at f2380000 (32-bit, non-prefetchable) [disabled] [size=8K]
Capabilities: <access denied>

83:00.0 Non-Essential Instrumentation [1300]: Advanced Micro Devices, Inc. [AMD] Starship/Matisse Reserved SPP [1022:1485]
Subsystem: Advanced Micro Devices, Inc. [AMD] Starship/Matisse Reserved SPP [1022:1485]
Control: I/O- Mem- BusMaster- SpecCycle- MemWINV- VGASnoop- ParErr- Stepping- SERR- FastB2B- DisINTx-
Status: Cap+ 66MHz- UDF- FastB2B- ParErr- DEVSEL=fast >TAbort- <TAbort- <MAbort- >SERR- <PERR- INTx-
IOMMU group: 29
Capabilities: <access denied>

83:00.2 Encryption controller [1080]: Advanced Micro Devices, Inc. [AMD] Starship/Matisse PTDMA [1022:1498]
Subsystem: Advanced Micro Devices, Inc. [AMD] Starship/Matisse PTDMA [1022:1498]
Control: I/O- Mem- BusMaster- SpecCycle- MemWINV- VGASnoop- ParErr- Stepping- SERR- FastB2B- DisINTx-
Status: Cap+ 66MHz- UDF- FastB2B- ParErr- DEVSEL=fast >TAbort- <TAbort- <MAbort- >SERR- <PERR- INTx-
Interrupt: pin B routed to IRQ 255
IOMMU group: 30
Region 2: Memory at f2200000 (32-bit, non-prefetchable) [disabled] [size=512K]
Region 5: Memory at f2280000 (32-bit, non-prefetchable) [disabled] [size=8K]
Capabilities: <access denied>

c0:00.0 Host bridge [0600]: Advanced Micro Devices, Inc. [AMD] Starship/Matisse Root Complex [1022:1480]
Subsystem: Super Micro Computer Inc H12SSL-i [15d9:1b95]
Control: I/O- Mem- BusMaster- SpecCycle- MemWINV- VGASnoop- ParErr- Stepping- SERR- FastB2B- DisINTx-
Status: Cap- 66MHz- UDF- FastB2B- ParErr- DEVSEL=fast >TAbort- <TAbort- <MAbort- >SERR- <PERR- INTx-

c0:00.2 IOMMU [0806]: Advanced Micro Devices, Inc. [AMD] Milan IOMMU [1022:164f] (rev 01)
Subsystem: Advanced Micro Devices, Inc. [AMD] Milan IOMMU [1022:164f]
Control: I/O- Mem- BusMaster- SpecCycle- MemWINV- VGASnoop- ParErr+ Stepping- SERR- FastB2B- DisINTx-
Status: Cap+ 66MHz- UDF- FastB2B- ParErr- DEVSEL=fast >TAbort- <TAbort- <MAbort- >SERR- <PERR- INTx-
Capabilities: <access denied>

c0:01.0 Host bridge [0600]: Advanced Micro Devices, Inc. [AMD] Starship/Matisse PCIe Dummy Host Bridge [1022:1482]
Control: I/O- Mem- BusMaster- SpecCycle- MemWINV- VGASnoop- ParErr- Stepping- SERR- FastB2B- DisINTx-
Status: Cap- 66MHz- UDF- FastB2B- ParErr- DEVSEL=fast >TAbort- <TAbort- <MAbort- >SERR- <PERR- INTx-
IOMMU group: 0

c0:01.1 PCI bridge [0604]: Advanced Micro Devices, Inc. [AMD] Starship/Matisse GPP Bridge [1022:1483] (prog-if 00 [Normal decode])
Subsystem: Advanced Micro Devices, Inc. [AMD] Device [1022:1453]
Control: I/O+ Mem+ BusMaster+ SpecCycle- MemWINV- VGASnoop- ParErr- Stepping- SERR- FastB2B- DisINTx+
Status: Cap+ 66MHz- UDF- FastB2B- ParErr- DEVSEL=fast >TAbort- <TAbort- <MAbort- >SERR- <PERR- INTx-
Latency: 0, Cache Line Size: 64 bytes
Interrupt: pin ? routed to IRQ 38
IOMMU group: 1
Bus: primary=c0, secondary=c1, subordinate=c1, sec-latency=0
I/O behind bridge: f000-ffff [size=4K] [16-bit]
Memory behind bridge: b8000000-b81fffff [size=2M] [32-bit]
Prefetchable memory behind bridge: 18020000000-180201fffff [size=2M] [32-bit]
Secondary status: 66MHz- FastB2B- ParErr- DEVSEL=fast >TAbort- <TAbort- <MAbort- <SERR- <PERR-
BridgeCtl: Parity- SERR+ NoISA- VGA- VGA16+ MAbort- >Reset- FastB2B-
PriDiscTmr- SecDiscTmr- DiscTmrStat- DiscTmrSERREn-
Capabilities: <access denied>
Kernel driver in use: pcieport

c0:02.0 Host bridge [0600]: Advanced Micro Devices, Inc. [AMD] Starship/Matisse PCIe Dummy Host Bridge [1022:1482]
Control: I/O- Mem- BusMaster- SpecCycle- MemWINV- VGASnoop- ParErr- Stepping- SERR- FastB2B- DisINTx-
Status: Cap- 66MHz- UDF- FastB2B- ParErr- DEVSEL=fast >TAbort- <TAbort- <MAbort- >SERR- <PERR- INTx-
IOMMU group: 2

c0:03.0 Host bridge [0600]: Advanced Micro Devices, Inc. [AMD] Starship/Matisse PCIe Dummy Host Bridge [1022:1482]
Control: I/O- Mem- BusMaster- SpecCycle- MemWINV- VGASnoop- ParErr- Stepping- SERR- FastB2B- DisINTx-
Status: Cap- 66MHz- UDF- FastB2B- ParErr- DEVSEL=fast >TAbort- <TAbort- <MAbort- >SERR- <PERR- INTx-
IOMMU group: 3

c0:04.0 Host bridge [0600]: Advanced Micro Devices, Inc. [AMD] Starship/Matisse PCIe Dummy Host Bridge [1022:1482]
Control: I/O- Mem- BusMaster- SpecCycle- MemWINV- VGASnoop- ParErr- Stepping- SERR- FastB2B- DisINTx-
Status: Cap- 66MHz- UDF- FastB2B- ParErr- DEVSEL=fast >TAbort- <TAbort- <MAbort- >SERR- <PERR- INTx-
IOMMU group: 4

c0:05.0 Host bridge [0600]: Advanced Micro Devices, Inc. [AMD] Starship/Matisse PCIe Dummy Host Bridge [1022:1482]
Control: I/O- Mem- BusMaster- SpecCycle- MemWINV- VGASnoop- ParErr- Stepping- SERR- FastB2B- DisINTx-
Status: Cap- 66MHz- UDF- FastB2B- ParErr- DEVSEL=fast >TAbort- <TAbort- <MAbort- >SERR- <PERR- INTx-
IOMMU group: 5

c0:07.0 Host bridge [0600]: Advanced Micro Devices, Inc. [AMD] Starship/Matisse PCIe Dummy Host Bridge [1022:1482]
Control: I/O- Mem- BusMaster- SpecCycle- MemWINV- VGASnoop- ParErr- Stepping- SERR- FastB2B- DisINTx-
Status: Cap- 66MHz- UDF- FastB2B- ParErr- DEVSEL=fast >TAbort- <TAbort- <MAbort- >SERR- <PERR- INTx-
IOMMU group: 6

c0:07.1 PCI bridge [0604]: Advanced Micro Devices, Inc. [AMD] Starship/Matisse Internal PCIe GPP Bridge 0 to bus[E:B] [1022:1484] (prog-if 00 [Normal decode])
Subsystem: Advanced Micro Devices, Inc. [AMD] Starship/Matisse Internal PCIe GPP Bridge 0 to bus[E:B] [1022:1484]
Control: I/O+ Mem+ BusMaster+ SpecCycle- MemWINV- VGASnoop- ParErr- Stepping- SERR- FastB2B- DisINTx+
Status: Cap+ 66MHz- UDF- FastB2B- ParErr- DEVSEL=fast >TAbort- <TAbort- <MAbort- >SERR- <PERR- INTx-
Latency: 0, Cache Line Size: 64 bytes
Interrupt: pin A routed to IRQ 40
IOMMU group: 7
Bus: primary=c0, secondary=c2, subordinate=c2, sec-latency=0
I/O behind bridge: [disabled] [32-bit]
Memory behind bridge: b8300000-b83fffff [size=1M] [32-bit]
Prefetchable memory behind bridge: [disabled] [64-bit]
Secondary status: 66MHz- FastB2B- ParErr- DEVSEL=fast >TAbort- <TAbort- <MAbort- <SERR- <PERR-
BridgeCtl: Parity- SERR+ NoISA- VGA- VGA16+ MAbort- >Reset- FastB2B-
PriDiscTmr- SecDiscTmr- DiscTmrStat- DiscTmrSERREn-
Capabilities: <access denied>
Kernel driver in use: pcieport

c0:08.0 Host bridge [0600]: Advanced Micro Devices, Inc. [AMD] Starship/Matisse PCIe Dummy Host Bridge [1022:1482]
Control: I/O- Mem- BusMaster- SpecCycle- MemWINV- VGASnoop- ParErr- Stepping- SERR- FastB2B- DisINTx-
Status: Cap- 66MHz- UDF- FastB2B- ParErr- DEVSEL=fast >TAbort- <TAbort- <MAbort- >SERR- <PERR- INTx-
IOMMU group: 8

c0:08.1 PCI bridge [0604]: Advanced Micro Devices, Inc. [AMD] Starship/Matisse Internal PCIe GPP Bridge 0 to bus[E:B] [1022:1484] (prog-if 00 [Normal decode])
Subsystem: Advanced Micro Devices, Inc. [AMD] Starship/Matisse Internal PCIe GPP Bridge 0 to bus[E:B] [1022:1484]
Control: I/O+ Mem+ BusMaster+ SpecCycle- MemWINV- VGASnoop- ParErr- Stepping- SERR- FastB2B- DisINTx+
Status: Cap+ 66MHz- UDF- FastB2B- ParErr- DEVSEL=fast >TAbort- <TAbort- <MAbort- >SERR- <PERR- INTx-
Latency: 0, Cache Line Size: 64 bytes
Interrupt: pin A routed to IRQ 42
IOMMU group: 9
Bus: primary=c0, secondary=c3, subordinate=c3, sec-latency=0
I/O behind bridge: [disabled] [32-bit]
Memory behind bridge: b8200000-b82fffff [size=1M] [32-bit]
Prefetchable memory behind bridge: [disabled] [64-bit]
Secondary status: 66MHz- FastB2B- ParErr- DEVSEL=fast >TAbort- <TAbort- <MAbort- <SERR- <PERR-
BridgeCtl: Parity- SERR+ NoISA- VGA- VGA16+ MAbort- >Reset- FastB2B-
PriDiscTmr- SecDiscTmr- DiscTmrStat- DiscTmrSERREn-
Capabilities: <access denied>
Kernel driver in use: pcieport

c1:00.0 RAID bus controller [0104]: Broadcom / LSI MegaRAID 12GSAS/PCIe Secure SAS39xx [1000:10e2]
Subsystem: Super Micro Computer Inc AOC-S3908L-H8iR RAID Adapter [15d9:1b66]
Control: I/O+ Mem+ BusMaster+ SpecCycle- MemWINV- VGASnoop- ParErr- Stepping- SERR- FastB2B- DisINTx+
Status: Cap+ 66MHz- UDF- FastB2B- ParErr- DEVSEL=fast >TAbort- <TAbort- <MAbort- >SERR- <PERR- INTx-
Latency: 0, Cache Line Size: 64 bytes
Interrupt: pin A routed to IRQ 60
IOMMU group: 10
Region 0: Memory at 18020100000 (64-bit, prefetchable) [size=1M]
Region 2: Memory at 18020000000 (64-bit, prefetchable) [size=1M]
Region 4: Memory at b8100000 (32-bit, non-prefetchable) [size=1M]
Region 5: I/O ports at f000 [size=256]
Expansion ROM at b8000000 [disabled] [size=1M]
Capabilities: <access denied>
Kernel driver in use: megaraid_sas
Kernel modules: megaraid_sas

c2:00.0 Non-Essential Instrumentation [1300]: Advanced Micro Devices, Inc. [AMD] Starship/Matisse PCIe Dummy Function [1022:148a]
Subsystem: Advanced Micro Devices, Inc. [AMD] Starship/Matisse PCIe Dummy Function [1022:148a]
Control: I/O- Mem- BusMaster- SpecCycle- MemWINV- VGASnoop- ParErr- Stepping- SERR- FastB2B- DisINTx-
Status: Cap+ 66MHz- UDF- FastB2B- ParErr- DEVSEL=fast >TAbort- <TAbort- <MAbort- >SERR- <PERR- INTx-
IOMMU group: 11
Capabilities: <access denied>

c2:00.2 Encryption controller [1080]: Advanced Micro Devices, Inc. [AMD] Starship/Matisse PTDMA [1022:1498]
Subsystem: Advanced Micro Devices, Inc. [AMD] Starship/Matisse PTDMA [1022:1498]
Control: I/O- Mem- BusMaster- SpecCycle- MemWINV- VGASnoop- ParErr- Stepping- SERR- FastB2B- DisINTx-
Status: Cap+ 66MHz- UDF- FastB2B- ParErr- DEVSEL=fast >TAbort- <TAbort- <MAbort- >SERR- <PERR- INTx-
Interrupt: pin B routed to IRQ 255
IOMMU group: 12
Region 2: Memory at b8300000 (32-bit, non-prefetchable) [disabled] [size=512K]
Region 5: Memory at b8380000 (32-bit, non-prefetchable) [disabled] [size=8K]
Capabilities: <access denied>

c3:00.0 Non-Essential Instrumentation [1300]: Advanced Micro Devices, Inc. [AMD] Starship/Matisse Reserved SPP [1022:1485]
Subsystem: Advanced Micro Devices, Inc. [AMD] Starship/Matisse Reserved SPP [1022:1485]
Control: I/O- Mem- BusMaster- SpecCycle- MemWINV- VGASnoop- ParErr- Stepping- SERR- FastB2B- DisINTx-
Status: Cap+ 66MHz- UDF- FastB2B- ParErr- DEVSEL=fast >TAbort- <TAbort- <MAbort- >SERR- <PERR- INTx-
IOMMU group: 13
Capabilities: <access denied>

c3:00.2 Encryption controller [1080]: Advanced Micro Devices, Inc. [AMD] Starship/Matisse PTDMA [1022:1498]
Subsystem: Advanced Micro Devices, Inc. [AMD] Starship/Matisse PTDMA [1022:1498]
Control: I/O- Mem- BusMaster- SpecCycle- MemWINV- VGASnoop- ParErr- Stepping- SERR- FastB2B- DisINTx-
Status: Cap+ 66MHz- UDF- FastB2B- ParErr- DEVSEL=fast >TAbort- <TAbort- <MAbort- >SERR- <PERR- INTx-
Interrupt: pin B routed to IRQ 255
IOMMU group: 14
Region 2: Memory at b8200000 (32-bit, non-prefetchable) [disabled] [size=512K]
Region 5: Memory at b8280000 (32-bit, non-prefetchable) [disabled] [size=8K]
Capabilities: <access denied>


** USB devices:
Bus 001 Device 001: ID 1d6b:0002 Linux Foundation 2.0 root hub
Bus 002 Device 001: ID 1d6b:0003 Linux Foundation 3.0 root hub
Bus 003 Device 001: ID 1d6b:0002 Linux Foundation 2.0 root hub
Bus 004 Device 001: ID 1d6b:0003 Linux Foundation 3.0 root hub
Bus 005 Device 001: ID 1d6b:0002 Linux Foundation 2.0 root hub
Bus 005 Device 002: ID 046d:c31c Logitech, Inc. Keyboard K120
Bus 006 Device 001: ID 1d6b:0003 Linux Foundation 3.0 root hub
Bus 007 Device 001: ID 1d6b:0002 Linux Foundation 2.0 root hub
Bus 007 Device 002: ID 1d6b:0107 Linux Foundation USB Virtual Hub
Bus 007 Device 003: ID 0b1f:03ee Insyde Software Corp. RNDIS/Ethernet Gadget
Bus 007 Device 004: ID 0557:9241 ATEN International Co., Ltd SMCI HID KM
Bus 008 Device 001: ID 1d6b:0003 Linux Foundation 3.0 root hub


-- System Information:
Debian Release: trixie/sid
APT prefers testing
APT policy: (500, 'testing')
Architecture: amd64 (x86_64)

Kernel: Linux 6.10.11-amd64 (SMP w/16 CPU threads; PREEMPT)
Kernel taint flags: TAINT_WARN
Locale: LANG=C, LC_CTYPE=en_US.utf8 (charmap=UTF-8), LANGUAGE not set
Shell: /bin/sh linked to /usr/bin/dash
Init: systemd (via /run/systemd/system)
LSM: AppArmor: enabled

Versions of packages linux-image-6.11.2-amd64 depends on:
ii initramfs-tools [linux-initramfs-tool] 0.145
ii kmod 33+20240816-2
ii linux-base 4.10.1

Versions of packages linux-image-6.11.2-amd64 recommends:
ii apparmor 3.1.7-1+b2

Versions of packages linux-image-6.11.2-amd64 suggests:
pn debian-kernel-handbook <none>
pn firmware-linux-free <none>
ii grub-efi-amd64 2.12-5
pn linux-doc-6.11 <none>

Versions of packages linux-image-6.11.2-amd64 is related to:
pn firmware-amd-graphics <none>
pn firmware-atheros <none>
pn firmware-bnx2 <none>
pn firmware-bnx2x <none>
pn firmware-brcm80211 <none>
pn firmware-cavium <none>
pn firmware-intel-sound <none>
pn firmware-intelwimax <none>
pn firmware-ipw2x00 <none>
pn firmware-ivtv <none>
pn firmware-iwlwifi <none>
pn firmware-libertas <none>
pn firmware-linux-nonfree <none>
ii firmware-misc-nonfree 20240709-2
pn firmware-myricom <none>
pn firmware-netxen <none>
pn firmware-qlogic <none>
pn firmware-realtek <none>
pn firmware-samsung <none>
pn firmware-siano <none>
pn firmware-ti-connectivity <none>
pn xen-hypervisor <none>

-- no debconf information

[Uwe Kleine-König]

Control: tag -1 + moreinfo
Control: forwarded -1 https://github.com/linux-rdma/opensm/issues/37

Hello,

Post by Francesco Poli (wintermute)
Package: src:linux
Version: 6.11.2-1
Severity: important
Hello,
I encountered a major issue on an HPC cluster head node, as soon as
I upgraded the Linux kernel from version 6.10.11-1 to version 6.11.2-1 .
The issue is that the head node runs OpenSM (InfiniBand subnet manager),
which is needed for the Infiniband network to work.
As soon as I reboot the head node with kernel 6.11.2-1 (or 6.11.4-1),
OpenSM fails to start.
If I reboot with the previous kernel version 6.10.11-1, everything
works fine.
The symptoms are described in bug [#1085300], filed against package
opensm.
[#1085300]: <https://bugs.debian.org/1085300>
Now I am not sure what's going on.
Is there any important change in the Linux kernel that OpenSM needs
to adapt for?
Or is this a bug in the newer Linux kernel version (that needs to
be fixed there)?
I filed this bug report against the Debian Linux kernel, in order
to warn other users about this issue, and in order to ask the Debian
Kernel Team to investigate the issue and/or to forward the bug report
to the relevant upstream Linux kernel maintainers.
Please do not reassign to package opensm with the intention of
merging with bug [#1085300], unless you know for sure that the
issue is in opensm and you know how to fix it.

Please do not report multiple bugs for the same issue. The right(er)
thing to do is to make use of "affects". Now there are three bug reports
(2 for Debian and one upstream) and someone being aware of only one (or
two) of them, might miss some action which results in duplicate work.

Post by Francesco Poli (wintermute)
Please help, I would very much like to run the head node with
an up-to-date kernel!

This is hard to act on without further input. Some questions to debug
this:

I guess the kernel provides a directory "/sys/class/infiniband_mad". Do
its contents look different on 6.10.x and 6.11.x?

Can you please bisect the problem? There are a few kernel versions that
were packaged for Debian (i.e. 6.11-1~exp1, 6.11~rc5-1~exp1,
6.11~rc4-1~exp1, 6.10.12-1). I would expect that 6.11~rc4-1~exp1 is the
oldest failing one. It would be great if you could bisect this further.
Something like the following on the working kernel:

git clone https://git.kernel.org/pub/scm/linux/kernel/git/torvalds/linux.git
cd linux
git checkout v6.10
cp /boot/config-6.10.11-1-$(uname -r) .config
make localmodconfig
cp .config arch/x86/configs/my_defconfig
make bindeb-pkg

This creates a debian kernel package that you can test. I would hope
this one to be "good".

(The following steps don't need to be done on the working kernel, this
is only critical for the localmodconfig step above.)

Then test 6.11:

git checkout v6.11
make my_defconfig bindeb-pkg

I would expect this one to produce a broken kernel package.

If you can confirm that (i.e. vanilla 6.10 works and 6.11 doesn't), do
the actual bisection:

git bisect start v6.11 v6.10

and in each to-be-tested version do:

make my_defconfig bindeb-pkg

and test the resulting kernel package. Depending on if that is good or
bad do:

git bisect good

or

git bisect bad

Note you don't need to test the versions that are suggested there. To
speed up, it might be beneficial to test v6.11-rc1~117 and v6.11-rc1~116
first. To do so just don't test the version that git-bisect proposes but
do:

git checkout v6.11-rc1~117
make my_defconfig bindeb-pkg
... test ...
git bisect ...

and then the same for v6.11-rc1~116.

Then report back the found first bad commit. If you have difficulties
following this instruction, feel free to contact me, e.g. in the
#debian-kernel irc channel.

Best regards
Uwe

[Francesco Poli]

On Mon, 11 Nov 2024 11:22:26 +0100 Uwe Kleine-König wrote:

[...]

Post by Francesco Poli (wintermute)
Hello,

Hi Uwe, thanks for your followup.
[...]

Post by Francesco Poli (wintermute)

Post by Francesco Poli (wintermute)
I filed this bug report against the Debian Linux kernel, in order
to warn other users about this issue, and in order to ask the Debian
Kernel Team to investigate the issue and/or to forward the bug report
to the relevant upstream Linux kernel maintainers.
Please do not reassign to package opensm with the intention of
merging with bug [#1085300], unless you know for sure that the
issue is in opensm and you know how to fix it.

Please do not report multiple bugs for the same issue. The right(er)
thing to do is to make use of "affects". Now there are three bug reports
(2 for Debian and one upstream) and someone being aware of only one (or
two) of them, might miss some action which results in duplicate work.

You are right, the "affects" field is the most appropriate means to
show that a bug report against a given package also affects other
packages.

However, in this case, the lack of replies from opensm maintainers made
me doubtful about the best possible course of action. Sorry about that.

Post by Francesco Poli (wintermute)

Post by Francesco Poli (wintermute)
Please help, I would very much like to run the head node with
an up-to-date kernel!

This is hard to act on without further input. Some questions to debug
I guess the kernel provides a directory "/sys/class/infiniband_mad". Do
its contents look different on 6.10.x and 6.11.x?

I will look into this as soon as I can reboot the cluster head node.

Post by Francesco Poli (wintermute)
Can you please bisect the problem?

[...]

I have to find a time window where I can perform multiple reboots,
which can result in a non-working InfiniBand network... It won't be
easy, since the cluster has entered production and users keep launching
jobs.

Anyway, what I have done so far is: I have tried and rebuilt a Linux
kernel image Debian package, following your instructions.
After some failed attempts (due to missing dependencies and/or required
tools), I think I succeeded, but I had to reply to a number of
questions during the procedure: I have always replied with the default
answer (by hitting [Enter]), I hope that was the right thing to do!

Before I go on and try to install the resulting Debian package, could
you please review the transcript of what I did (see the attached file)?

Please bear with me, some of the questions were really obscure to me
and I am not really familiar with the procedure: I think that the last
time I rebuilt a Linux kernel image Debian package was some 15 years
ago (I was still using the now-obsolete [kernel-package]!).

[kernel-package]: <https://tracker.debian.org/pkg/kernel-package>

Thanks for your time and for the help you are providing.

[Uwe Kleine-König]

Hello Francesco,

Post by Francesco Poli
[...]

Post by Francesco Poli (wintermute)
Hello,

Hi Uwe, thanks for your followup.
[...]

Post by Francesco Poli (wintermute)

Post by Francesco Poli (wintermute)
I filed this bug report against the Debian Linux kernel, in order
to warn other users about this issue, and in order to ask the Debian
Kernel Team to investigate the issue and/or to forward the bug report
to the relevant upstream Linux kernel maintainers.
Please do not reassign to package opensm with the intention of
merging with bug [#1085300], unless you know for sure that the
issue is in opensm and you know how to fix it.

Please do not report multiple bugs for the same issue. The right(er)
thing to do is to make use of "affects". Now there are three bug reports
(2 for Debian and one upstream) and someone being aware of only one (or
two) of them, might miss some action which results in duplicate work.

You are right, the "affects" field is the most appropriate means to
show that a bug report against a given package also affects other
packages.
However, in this case, the lack of replies from opensm maintainers made
me doubtful about the best possible course of action. Sorry about that.

Post by Francesco Poli (wintermute)

Post by Francesco Poli (wintermute)
Please help, I would very much like to run the head node with
an up-to-date kernel!

This is hard to act on without further input. Some questions to debug
I guess the kernel provides a directory "/sys/class/infiniband_mad". Do
its contents look different on 6.10.x and 6.11.x?

I will look into this as soon as I can reboot the cluster head node.

Post by Francesco Poli (wintermute)
Can you please bisect the problem?

[...]
I have to find a time window where I can perform multiple reboots,
which can result in a non-working InfiniBand network... It won't be
easy, since the cluster has entered production and users keep launching
jobs.
Anyway, what I have done so far is: I have tried and rebuilt a Linux
kernel image Debian package, following your instructions.
After some failed attempts (due to missing dependencies and/or required
tools), I think I succeeded, but I had to reply to a number of
questions during the procedure: I have always replied with the default
answer (by hitting [Enter]), I hope that was the right thing to do!

Yes, that sounds right. I wouldn't have expected that questions are
asked, but that's a problem with my expectations, not your following of
my instructions.

yes '' | make localmodconfig

would be the better recommendation it seems.

Post by Francesco Poli
Before I go on and try to install the resulting Debian package, could
you please review the transcript of what I did (see the attached file)?

Looks good. Probably the individual answers don't matter much and the
default should be fine. Just continue with my instructions and if the
resulting kernels boots and behave as the respective versions packaged
by Debian, everything is fine. Iff that fails, a more detailed review is
needed.

Best regards
Uwe

[Francesco Poli]

On Mon, 18 Nov 2024 09:58:03 +0100 Uwe Kleine-König wrote:

[...]
[...]

Post by Uwe Kleine-König

Post by Francesco Poli

Post by Uwe Kleine-König
I guess the kernel provides a directory "/sys/class/infiniband_mad". Do
its contents look different on 6.10.x and 6.11.x?

I will look into this as soon as I can reboot the cluster head node.

I looked into this, while testing the new Debian Linux kernel that has just migrated to testing (which, once again, makes opensm fail to start, just like other 6.11.x versions).

With a working kernel:

$ uname -v
#1 SMP PREEMPT_DYNAMIC Debian 6.10.11-1 (2024-09-22)
$ ls -altrF /sys/class/infiniband_mad/
total 0
lrwxrwxrwx 1 root root 0 Nov 4 15:58 umad0 -> ../../devices/pci0000:80/0000:80:01.1/0000:81:00.0/infiniband_mad/umad0/
lrwxrwxrwx 1 root root 0 Nov 4 15:58 umad1 -> ../../devices/pci0000:80/0000:80:01.1/0000:81:00.1/infiniband_mad/umad1/
lrwxrwxrwx 1 root root 0 Nov 11 15:54 issm1 -> ../../devices/pci0000:80/0000:80:01.1/0000:81:00.1/infiniband_mad/issm1/
lrwxrwxrwx 1 root root 0 Nov 11 15:54 issm0 -> ../../devices/pci0000:80/0000:80:01.1/0000:81:00.0/infiniband_mad/issm0/
drwxr-xr-x 2 root root 0 Nov 11 15:54 ./
drwxr-xr-x 72 root root 0 Nov 11 15:54 ../
-r--r--r-- 1 root root 4096 Nov 11 15:54 abi_version
$ cat /sys/class/infiniband_mad/abi_version
5

With a kernel that makes opensm fail to start:

$ uname -v
#1 SMP PREEMPT_DYNAMIC Debian 6.11.7-1 (2024-11-09)
$ ls -altrF /sys/class/infiniband_mad/
total 0
drwxr-xr-x 73 root root 0 Nov 18 09:41 ../
-r--r--r-- 1 root root 4096 Nov 18 09:41 abi_version
lrwxrwxrwx 1 root root 0 Nov 18 09:41 umad0 -> ../../devices/pci0000:80/0000:80:01.1/0000:81:00.0/infiniband_mad/umad0/
lrwxrwxrwx 1 root root 0 Nov 18 09:41 umad1 -> ../../devices/pci0000:80/0000:80:01.1/0000:81:00.1/infiniband_mad/umad1/
drwxr-xr-x 2 root root 0 Nov 18 09:43 ./
$ cat /sys/class/infiniband_mad/abi_version
5

As you can see, a couple of files (symlinks) are missing here...

Does this ring a bell?
Can you tell what's wrong, by just looking at this?
Or, at least, do you get some less vague idea of what's going on?

[...]

Post by Uwe Kleine-König

Post by Francesco Poli
Before I go on and try to install the resulting Debian package, could
you please review the transcript of what I did (see the attached file)?

Looks good. Probably the individual answers don't matter much and the
default should be fine. Just continue with my instructions and if the
resulting kernels boots and behave as the respective versions packaged
by Debian, everything is fine. Iff that fails, a more detailed review is
needed.

Thanks for confirming, I really hope I can find a time window, where I
can bisect...

[Uwe Kleine-König]

Hello Francesco,

[for the new-comers: This is about a regression in 6.11. Details
available at https://bugs.debian.org/1086520. The TL;DR; is that on
6.10.11 opensm works as expected, while it fails to start on 6.11.7.]

Post by Francesco Poli
[...]
[...]

Post by Francesco Poli

Post by Uwe Kleine-König
I guess the kernel provides a directory "/sys/class/infiniband_mad". Do
its contents look different on 6.10.x and 6.11.x?

I will look into this as soon as I can reboot the cluster head node.

I looked into this, while testing the new Debian Linux kernel that has
just migrated to testing (which, once again, makes opensm fail to
start, just like other 6.11.x versions).
$ uname -v
#1 SMP PREEMPT_DYNAMIC Debian 6.10.11-1 (2024-09-22)
$ ls -altrF /sys/class/infiniband_mad/
total 0
lrwxrwxrwx 1 root root 0 Nov 4 15:58 umad0 -> ../../devices/pci0000:80/0000:80:01.1/0000:81:00.0/infiniband_mad/umad0/
lrwxrwxrwx 1 root root 0 Nov 4 15:58 umad1 -> ../../devices/pci0000:80/0000:80:01.1/0000:81:00.1/infiniband_mad/umad1/
lrwxrwxrwx 1 root root 0 Nov 11 15:54 issm1 -> ../../devices/pci0000:80/0000:80:01.1/0000:81:00.1/infiniband_mad/issm1/
lrwxrwxrwx 1 root root 0 Nov 11 15:54 issm0 -> ../../devices/pci0000:80/0000:80:01.1/0000:81:00.0/infiniband_mad/issm0/
drwxr-xr-x 2 root root 0 Nov 11 15:54 ./
drwxr-xr-x 72 root root 0 Nov 11 15:54 ../
-r--r--r-- 1 root root 4096 Nov 11 15:54 abi_version
$ cat /sys/class/infiniband_mad/abi_version
5
$ uname -v
#1 SMP PREEMPT_DYNAMIC Debian 6.11.7-1 (2024-11-09)
$ ls -altrF /sys/class/infiniband_mad/
total 0
drwxr-xr-x 73 root root 0 Nov 18 09:41 ../
-r--r--r-- 1 root root 4096 Nov 18 09:41 abi_version
lrwxrwxrwx 1 root root 0 Nov 18 09:41 umad0 -> ../../devices/pci0000:80/0000:80:01.1/0000:81:00.0/infiniband_mad/umad0/
lrwxrwxrwx 1 root root 0 Nov 18 09:41 umad1 -> ../../devices/pci0000:80/0000:80:01.1/0000:81:00.1/infiniband_mad/umad1/
drwxr-xr-x 2 root root 0 Nov 18 09:43 ./
$ cat /sys/class/infiniband_mad/abi_version
5
As you can see, a couple of files (symlinks) are missing here...

It looks like the commit that is biting you is

https://git.kernel.org/linus/50660c5197f52b8137e223dc3ba8d43661179a1d

So if you bisect, try 50660c5197f52b8137e223dc3ba8d43661179a1d and its
parent 24943dcdc156cf294d97a36bf5c51168bf574c22 first.

I don't know about infiniband, but I'd say: Either your machine doesn't
have these issmX devices and opensm should cope with that, or these
issmX devices are available then
50660c5197f52b8137e223dc3ba8d43661179a1d is buggy.

Post by Francesco Poli
Does this ring a bell?

It doesn't for me, but maybe Mark Zhang or someone else among the new
recipients has an idea?

Best regards
Uwe

[Francesco Poli]

On Thu, 21 Nov 2024 11:04:13 +0100 Uwe Kleine-König wrote:

[...]

Post by Uwe Kleine-König
It looks like the commit that is biting you is
https://git.kernel.org/linus/50660c5197f52b8137e223dc3ba8d43661179a1d
So if you bisect, try 50660c5197f52b8137e223dc3ba8d43661179a1d and its
parent 24943dcdc156cf294d97a36bf5c51168bf574c22 first.

I started to bisect.

The first surprise is that 50660c5197f52b8137e223dc3ba8d43661179a1d is
good... :-o

$ git checkout 50660c5197f52b8137e223dc3ba8d43661179a1d
$ make -j 12 my_defconfig bindeb-pkg

[install and reboot with this kernel version]

# ls /sys/class/infiniband_mad/ -altrF
total 0
drwxr-xr-x 70 root root 0 Nov 25 12:05 ../
-r--r--r-- 1 root root 4096 Nov 25 12:05 abi_version
lrwxrwxrwx 1 root root 0 Nov 25 12:05 umad0 -> ../../devices/pci0000:80/0000:80:01.1/0000:81:00.0/infiniband_mad/umad0/
lrwxrwxrwx 1 root root 0 Nov 25 12:05 umad1 -> ../../devices/pci0000:80/0000:80:01.1/0000:81:00.1/infiniband_mad/umad1/
lrwxrwxrwx 1 root root 0 Nov 25 12:08 issm1 -> ../../devices/pci0000:80/0000:80:01.1/0000:81:00.1/infiniband_mad/issm1/
lrwxrwxrwx 1 root root 0 Nov 25 12:08 issm0 -> ../../devices/pci0000:80/0000:80:01.1/0000:81:00.0/infiniband_mad/issm0/
drwxr-xr-x 2 root root 0 Nov 25 12:08 ./

[InfiniBand works]

$ git bisect start
$ git bisect good
$ git checkout v6.11
$ make -j 12 my_defconfig bindeb-pkg

[install and reboot with this kernel version]

# ls /sys/class/infiniband_mad/ -altrF
total 0
drwxr-xr-x 70 root root 0 Nov 25 12:29 ../
-r--r--r-- 1 root root 4096 Nov 25 12:29 abi_version
lrwxrwxrwx 1 root root 0 Nov 25 12:29 umad0 -> ../../devices/pci0000:80/0000:80:01.1/0000:81:00.0/infiniband_mad/umad0/
lrwxrwxrwx 1 root root 0 Nov 25 12:29 umad1 -> ../../devices/pci0000:80/0000:80:01.1/0000:81:00.1/infiniband_mad/umad1/
drwxr-xr-x 2 root root 0 Nov 25 12:30 ./

[InfiniBand fails, because OpenSM fails to start]

$ git bisect bad
Bisecting: 7036 revisions left to test after this (roughly 13 steps)
[b3ce7a30847a54a7f96a35e609303d8afecd460b] Merge tag 'drm-next-2024-07-18' of https://gitlab.freedesktop.org/drm/kernel
$ make -j 12 my_defconfig bindeb-pkg


Woooha, 13 steps are a lot...

I went on until 10 steps are left:

[test b3ce7a30847a54a7f96a35e609303d8afecd460b]
$ git bisect good
Bisecting: 3385 revisions left to test after this (roughly 12 steps)
[fbc90c042cd1dc7258ebfebe6d226017e5b5ac8c] Merge tag 'mm-stable-2024-07-21-14-50' of git://git.kernel.org/pub/scm/linux/kernel/git/akpm/mm

[test fbc90c042cd1dc7258ebfebe6d226017e5b5ac8c]
$ git bisect bad
Bisecting: 1763 revisions left to test after this (roughly 11 steps)
[09ea8089abb5d851ce08a9b1a43706e42ef39db2] Merge tag 'staging-6.11-rc1' of git://git.kernel.org/pub/scm/linux/kernel/git/gregkh/staging

[test 09ea8089abb5d851ce08a9b1a43706e42ef39db2]
$ git bisect bad
Bisecting: 910 revisions left to test after this (roughly 10 steps)
[4305ca0087dd99c3c3e0e2ac8a228b7e53a21c78] Merge tag 'scsi-misc' of git://git.kernel.org/pub/scm/linux/kernel/git/jejb/scsi


Since I could not afford to keep the cluster out of service any longer
(each step takes at least 20 or 25 minutes: build + install + reboot +
check InfiniBand), I decided to return the cluster to service.

I will try to continue to bisect by testing the resulting kernels on a
compute node: there's no OpenSM there and it cannot run anyway, if
there's another OpenSM on the same InfiniBand network.
However, I can check whether those issm* symlinks are created in
/sys/class/infiniband_mad/
I really hope that this is enough to pinpoint the first bad
commit...

Any better ideas?

[Debian Bug Tracking System]

Post by Uwe Kleine-König
tag -1 + moreinfo

Bug #1086520 [src:linux] linux-image-6.11.2-amd64: makes opensm fail to start
Added tag(s) moreinfo.

Post by Uwe Kleine-König
forwarded -1 https://github.com/linux-rdma/opensm/issues/37

Bug #1086520 [src:linux] linux-image-6.11.2-amd64: makes opensm fail to start
Set Bug forwarded-to-address to 'https://github.com/linux-rdma/opensm/issues/37'.

[Francesco Poli]

On Tue, 26 Nov 2024 09:21:37 +0800 Mark Zhang wrote:

[...]

Yes looks like FW reports vport.num_plane > 0. What is your hw type and
FW version ("ethtool -i <netdev_of_the_ibdev>")? I don't think it
supports multiplane.

$ /sbin/ethtool -i ibp129s0f0
driver: mlx5_core[ib_ipoib]
version: 6.10.11-amd64
firmware-version: 20.40.1000 (MT_0000000224)
expansion-rom-version:
bus-info: 0000:81:00.0
supports-statistics: yes
supports-test: yes
supports-eeprom-access: no
supports-register-dump: no
supports-priv-flags: yes

Please note that I determined <netdev_of_the_ibdev> by looking at
the output of 'ibv_devices': I hope this is a correct way to answer
your question.

[Francesco Poli]

[...]

Post by Francesco Poli
I will try to continue to bisect by testing the resulting kernels on a
compute node: there's no OpenSM there and it cannot run anyway, if
there's another OpenSM on the same InfiniBand network.
However, I can check whether those issm* symlinks are created in
/sys/class/infiniband_mad/
I really hope that this is enough to pinpoint the first bad
commit...

Yes, these symlinks should be there. Your test scenario is correct one.

OK, I have completed the bisect on a compute node without OpenSM, by
looking at the issm* symlinks, as I said.

See below.

Post by Francesco Poli
Any better ideas?

I think that commit: 2a5db20fa532 ("RDMA/mlx5: Add support to multi-plane device and port")
is the one which is causing to troubles, which leads me to suspect FW.

[...]

Thanks to your guess about the possibly troublesome commit, the bisect was completed in a few steps:

$ git checkout 2a5db20fa532
$ make -j 12 my_defconfig bindeb-pkg

[install this version on a compute node test image and reboot
one compute node with that image: the InfiniBand network was
working for that node, that's no surprise, since OpenSM was running
on the head node, but no issm* symlink was created; please note
that, surprisingly, the Ethernet network was not working, I mean
that the Ethernet interfaces were not found by the kernel...]

***@node # ls -altrF /sys/class/infiniband_mad/
total 0
drwxr-xr-x 60 root root 0 Nov 26 17:06 ../
lrwxrwxrwx 1 root root 0 Nov 26 17:06 umad0 -> ../../devices/pci0000:00/0000:00:01.1/0000:01:00.0/infiniband_mad/umad0/
-r--r--r-- 1 root root 4096 Nov 26 17:06 abi_version
lrwxrwxrwx 1 root root 0 Nov 26 17:06 umad1 -> ../../devices/pci0000:00/0000:00:01.1/0000:01:00.1/infiniband_mad/umad1/
drwxr-xr-x 2 root root 0 Nov 26 17:08 ./

$ git bisect bad
Bisecting: 0 revisions left to test after this (roughly 0 steps)
[65528cfb21fdb68de8ae6dccae19af180d93e143] net/mlx5: mlx5_ifc update for multi-plane support
$ make -j 12 my_defconfig bindeb-pkg

[install this version on the compute node test image and reboot
one compute node with that image: the InfiniBand network again
working for that node, issm* symlinks were created;
Ethernet network again not working for that node...]

***@node # ls -altrF /sys/class/infiniband_mad/
total 0
drwxr-xr-x 60 root root 0 Nov 26 17:31 ../
lrwxrwxrwx 1 root root 0 Nov 26 17:31 umad0 -> ../../devices/pci0000:00/0000:00:01.1/0000:01:00.0/infiniband_mad/umad0/
-r--r--r-- 1 root root 4096 Nov 26 17:31 abi_version
lrwxrwxrwx 1 root root 0 Nov 26 17:31 umad1 -> ../../devices/pci0000:00/0000:00:01.1/0000:01:00.1/infiniband_mad/umad1/
lrwxrwxrwx 1 root root 0 Nov 26 17:36 issm1 -> ../../devices/pci0000:00/0000:00:01.1/0000:01:00.1/infiniband_mad/issm1/
lrwxrwxrwx 1 root root 0 Nov 26 17:36 issm0 -> ../../devices/pci0000:00/0000:00:01.1/0000:01:00.0/infiniband_mad/issm0/
drwxr-xr-x 2 root root 0 Nov 26 17:36 ./

$ git bisect good
2a5db20fa532198639671713c6213f96ff285b85 is the first bad commit
commit 2a5db20fa532198639671713c6213f96ff285b85
Author: Mark Zhang <***@nvidia.com>
Date: Sun Jun 16 19:08:35 2024 +0300

RDMA/mlx5: Add support to multi-plane device and port

When multi-plane is supported, a logical port, which is aggregation of
multiple physical plane ports, is exposed for data transmission.
Compared with a normal mlx5 IB port, this logical port supports all
functionalities except Subnet Management.

Signed-off-by: Mark Zhang <***@nvidia.com>
Link: https://lore.kernel.org/r/***@kernel.org
Signed-off-by: Leon Romanovsky <***@nvidia.com>

drivers/infiniband/hw/mlx5/main.c | 60 +++++++++++++++++++++----
drivers/infiniband/hw/mlx5/mlx5_ib.h | 2 +
drivers/net/ethernet/mellanox/mlx5/core/vport.c | 1 +
include/linux/mlx5/driver.h | 1 +
4 files changed, 55 insertions(+), 9 deletions(-)


In other words, bingo!, your guess looks correct, the first bad commit
is the one you mentioned.


Now, I will try to upgrade the firmware of the InfiniBand NICs, as you
suggested, and check whether this solves the issue with the recent
Linux kernel versions.

Please confirm that the procedure to be followed is the one described in
<https://docs.nvidia.com/networking/display/ubuntu2204/firmware+burning>

Thanks for your time and patience, and for all the help you are kindly
providing! :-)

[Debian Bug Tracking System]

Your message dated Wed, 4 Dec 2024 18:13:56 +0100
with message-id <***@paranoici.org>
and subject line Re: Bug#1086520: linux-image-6.11.2-amd64: makes opensm fail to start
has caused the Debian Bug report #1086520,
regarding linux-image-6.11.2-amd64: makes opensm fail to start
to be marked as done.

This means that you claim that the problem has been dealt with.
If this is not the case it is now your responsibility to reopen the
Bug report if necessary, and/or fix the problem forthwith.

(NB: If you are a system administrator and have no idea what this
message is talking about, this may indicate a serious mail system
misconfiguration somewhere. Please contact ***@bugs.debian.org
immediately.)