[openib-general] Re: Tavor HCAs with openib

[Michael S. Tsirkin]

Quoting r. Manpreet Singh <manpreets7 at yahoo.com>:
> Subject: Tavor HCAs with openib
> 
> Hi,
> 
> I was wondering if the Mellanox PCI-X cards (with
> memory) are still supported in the openib stack.
> Although the code seems to include it in the PCI
> device id list, but I get the following error when I
> load ib_mthca:
> 
> ib_mthca 0000:04:00.0: Missing UAR, aborting
> 
> Details of the configuration:
> 
> Kernel version: 2.6.12.3.
> OpenIB kernel stack version: 3459 (from Sep 16).
> HCA card device ID: 0x5a44.
> 
> The following is the PCI configuration space dump of
> the HCA:
> 
> [root at driver5-linux linux-kernel]# lspci -s 04:00.00
> -xxx
> 04:00.0 InfiniBand: Mellanox Technology MT23108
> InfiniHost (rev a1)
> 00: b3 15 44 5a 53 01 30 02 a1 00 06 0c 08 40 00 00
> 10: 04 00 d0 df 00 00 00 00 0c 00 80 ff 0f 00 00 00
> 20: 0c 00 00 f0 0f 00 00 00 00 00 00 00 b3 15 44 5a
> 30: 00 00 00 00 40 00 00 00 00 00 00 00 0a 01 00 00
> 40: 11 50 1f 00 00 20 08 00 00 22 08 00 00 00 00 00
> 50: 03 60 ff 7f 11 11 00 00 00 00 00 00 00 00 00 00
> 60: 05 70 8a 00 00 00 00 00 00 00 00 00 00 00 00 00
> 70: 07 00 1c 00 00 04 e3 00 00 00 00 00 00 00 00 00
> 80: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
> 90: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
> a0: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
> b0: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
> c0: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
> d0: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
> e0: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
> f0: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
> 
> I'd appreciate any help/comments on this.
> 
> Thanks,
> Manpreet.

Yes.
10: 04 00 d0 df 00 00 00 00 0c 00 80 ff 0f 00 00 00
20: 0c 00 00 f0 0f 00 00 00 00 00 00 00 b3 15 44 5a
Region 0 (offset 0x10) is at dfd00000, but Regions 2 and 4 (0x18 and
0x20) got assigned to 0x0fff080000 and 0x0ff0000000.

It really looks like a BIOS/kernel issue.
Does linux actually support 64 bit hardware addresses?
pciutils seems not to like them:

lspci (rev 2.1.11) says:

~>./lspci -F ~/foot -vv
pcilib: 04:00.0 64-bit device address ignored.
pcilib: 04:00.0 64-bit device address ignored.
04:00.0 Class 0c06: 15b3:5a44 (rev a1)
        Subsystem: 15b3:5a44
        Control: I/O+ Mem+ BusMaster- SpecCycle- MemWINV+ VGASnoop-
ParErr+ Stepping- SERR+ FastB2B-
        Status: Cap+ 66Mhz+ UDF- FastB2B- ParErr- DEVSEL=medium >TAbort-
<TAbort- <MAbort- >SERR- <PERR-
        Interrupt: pin A routed to IRQ 10
        Region 0: Memory at dfd00000 (64-bit, non-prefetchable)
        Region 2: Memory at <ignored> (64-bit, prefetchable)
        Region 4: Memory at <ignored> (64-bit, prefetchable)
        Capabilities: [40] #11 [001f]
        Capabilities: [50] Vital Product Data
        Capabilities: [60] Message Signalled Interrupts: 64bit+
Queue=0/5 Enable-
                Address: 0000000000000000  Data: 0000
        Capabilities: [70] PCI-X non-bridge device.
                Command: DPERE- ERO- RBC=3 OST=1
                Status: Bus=0 Dev=0 Func=0 64bit- 133MHz- SCD- USC-,
		DC=simple, DMMRBC=0, DMOST=0, DMCRS=0, RSCEM-

Just to make sure, could you please look at the PCI bridge
that the device is behind with lspci -vv?

Need to check what happends to prefetchable memory behind bridge
base/limit values.

If region 2/4 values are outside this range, its a BIOS issue, otherwise
it might be a kernel issue.

-- 
MST