[openib-general] Low cost MBs compatible with Mellanox PCIE Cards

[Jason Gunthorpe]

On Wed, Feb 15, 2006 at 02:12:10PM +1300, Dave Watkins wrote:
> I've tried a few Asus boards with IB cards in their graphics card slots
> with no luck, the boards did post but the cards weren't visable to the
> drivers. Same cards in other machines were fine.

I just ran out an bought an Asus A8N-VM (~80$ CDN) and so far it seems
to work (0702 bios), at least the driver loads and lspci -vv reports a
x8 PCI-E link with the card in the x16 graphics slot. So that's promising.

Unfortunately the board's BIOS's ACPI tables are totally broken and
the BIOS assigns every interrupt source in the system
to IRQ 5 :< At least it does work in full APIC mode which means it
might be possible to get MSI working if the nvidia bridge isn't
broken.

Using stock Kernel 2.6.15.4, in-kernel mthca driver and a mhgs18-xt HCA.

03:00.0 InfiniBand: Mellanox Technologies MT25204 [InfiniHost III Lx HCA] (rev 20)
        Subsystem: Mellanox Technologies MT25204 [InfiniHost III Lx HCA]
        Control: I/O- Mem+ BusMaster+ SpecCycle- MemWINV- VGASnoop-
	ParErr- Stepping- SERR+ FastB2B-
        Status: Cap+ 66MHz- UDF- FastB2B- ParErr- DEVSEL=fast >TAbort-
	<TAbort- <MAbort- >SERR- <PERR-
        Latency: 0, Cache Line Size 10
        Interrupt: pin A routed to IRQ 5
        Region 0: Memory at faa00000 (64-bit, non-prefetchable)
	[size=1M]
        Region 2: Memory at bf000000 (64-bit, prefetchable) [size=8M]
        Capabilities: [40] Power Management version 2
                Flags: PMEClk- DSI- D1- D2- AuxCurrent=0mA
		PME(D0-,D1-,D2-,D3hot-,D3cold-)
                Status: D0 PME-Enable- DSel=0 DScale=0 PME-
        Capabilities: [48] Vital Product Data
        Capabilities: [90] Message Signalled Interrupts: 64bit+ Queue=0/5 Enable-
                Address: 0000000000000000  Data: 0000
        Capabilities: [84] MSI-X: Enable- Mask- TabSize=32
                Vector table: BAR=0 offset=00082000
                PBA: BAR=0 offset=00082200
        Capabilities: [60] Express Endpoint IRQ 0
                Device: Supported: MaxPayload 128 bytes, PhantFunc 0, ExtTag-
                Device: Latency L0s <64ns, L1 unlimited
                Device: AtnBtn- AtnInd- PwrInd-
                Device: Errors: Correctable- Non-Fatal- Fatal- Unsupported-
                Device: RlxdOrd- ExtTag- PhantFunc- AuxPwr- NoSnoop-
                Device: MaxPayload 128 bytes, MaxReadReq 4096 bytes
                Link: Supported Speed 2.5Gb/s, Width x8, ASPM L0s, Port 8
                Link: Latency L0s unlimited, L1 unlimited
                Link: ASPM Disabled RCB 64 bytes CommClk- ExtSynch-
                Link: Speed 2.5Gb/s, Width x8

> I suspect somewhere in the BIOS or chipset you can actually say that a
> certain PCI-E x16 slot is for "graphics" and no other cards are
> initialised, but that's only a guess.

I've never heard that a chipset has limited support to graphics or
IO. My understanding is that any incompatibility is soundly a BIOS
problem with enumeration and configuration of the bus segment. I
expect with the chipset specs available it would probably be possible
to make a linux PCI fixup to make systems that boot but won't enable
the device work by directly twiddling the PCI-E bridge in the
system. Basically the same way linux currently has fixups for BIOS
bugs in supporting PCI-PCI bridges.

Thanks,
Jason