[ofa-general] Performance of MT25204 versus MT25208

[Chuck Hartley]

Bart -
I started a thread similar to this a while back about expected RDMA
performance after I measured low bandwidth similar to yours.  In our case,
we are using DDR and you apparently are using SDR, but we are getting
bandwidth almost exactly twice what you are getting.  That is: your SDR BW =
674 MB/s and our DDR BW = 1336 MB/s.  Our motherboards are SuperMicro (X7DBU
and X7DBT) using the same 5000P chipset as your board. They are dual Xeon
CPU boards. The HCA is the MT25204 also. Here is our output from lspci for
comparison:

0b:00.0 InfiniBand: Mellanox Technologies MT25204 [InfiniHost III Lx HCA]
(rev 20)
        Subsystem: Mellanox Technologies MT25204 [InfiniHost III Lx HCA]
        Control: I/O- Mem+ BusMaster+ SpecCycle- MemWINV- VGASnoop- ParErr+
Stepping- SERR+ FastB2B-
        Status: Cap+ 66MHz- UDF- FastB2B- ParErr- DEVSEL=fast >TAbort-
<TAbort- <MAbort- >SERR- <PERR-
        Latency: 0, Cache Line Size: 32 bytes
        Interrupt: pin A routed to IRQ 18
        Region 0: Memory at ca200000 (64-bit, non-prefetchable) [size=1M]
        Region 2: Memory at cb000000 (64-bit, prefetchable) [size=8M]
        Capabilities: [40] Power Management version 2
                Flags: PMEClk- DSI- D1- D2- AuxCurrent=0mA
PME(D0-,D1-,D2-,D3hot-,D3cold-)
                Status: D0 PME-Enable- DSel=0 DScale=0 PME-
        Capabilities: [48] Vital Product Data
        Capabilities: [90] Message Signalled Interrupts: 64bit+ Queue=0/5
Enable-
                Address: 0000000000000000  Data: 0000
        Capabilities: [84] MSI-X: Enable- Mask- TabSize=32
                Vector table: BAR=0 offset=00082000
                PBA: BAR=0 offset=00082200
        Capabilities: [60] Express Endpoint IRQ 0
                Device: Supported: MaxPayload 128 bytes, PhantFunc 0,
ExtTag+
                Device: Latency L0s <64ns, L1 unlimited
                Device: AtnBtn- AtnInd- PwrInd-
                Device: Errors: Correctable- Non-Fatal- Fatal- Unsupported-
                Device: RlxdOrd- ExtTag- PhantFunc- AuxPwr- NoSnoop-
                Device: MaxPayload 128 bytes, MaxReadReq 512 bytes
                Link: Supported Speed 2.5Gb/s, Width x8, ASPM L0s, Port 8
                Link: Latency L0s unlimited, L1 unlimited
                Link: ASPM Disabled RCB 64 bytes CommClk- ExtSynch-
                Link: Speed 2.5Gb/s, Width x8

There were some comments in the previous thread that the 5000P chipset is
limiting the BW we could achieve.  We have SuperMicro boards that have the
MT25204 onboard and others that are on HCA plugin cards, but they all show
the same level of performance (all using the 5000P chipset).  We did plug in
one of the cards to an x4 slot by mistake and the performance was chopped
off at the level you are seeing, but lspci correctly identified it as an x4
slot.  We were unsuccessful in finding any BIOS settings that would improve
these numbers.  Also note that we have the good MaxReadReq = 512 that Sagi
mentions.

Are both of your motherboards using the 5000P chipset? Are the lspci results
from the dual CPU board the comparable to the ones you included  above?
Maybe someone can identify some parameter common to different PCI
configurations that may be the source of the problem.

Chuck
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.openfabrics.org/pipermail/general/attachments/20080118/079cf76e/attachment.html>