[ofa-general] Performance of MT25204 versus MT25208

[Sagi Rotem]

This may be your problem:
MaxReadReq 128 bytes
U need a BIOS update , common value with good performance is 512. 
Alternatively u can force it using setpci but than system may be
unstable.

-----Original Message-----
From: Bart Van Assche [mailto:bart.vanassche at gmail.com] 
Sent: Friday, January 18, 2008 10:56 AM
To: Sagi Rotem
Cc: Openib-General
Subject: Re: [ofa-general] Performance of MT25204 versus MT25208

> On January 17, 2008 Bart Van Assche wrote:
> >
> > Via ib_rdma_bw and ib_rdma_lat I obtained the following performance
> > data:
> > * peak bandwith 674 MB/s / latency 3.07 microseconds between two 
> > uniprocessor systems with MT25204 HCA's.
> > * peak bandwith 935 MB/s / latency 3.36 microseconds between two 
> > dual processor systems with MT25208 HCA's.
> >
> > I'm wondering now whether the performance differences are due to the

> > HCA's or due to the motherboard ?
>
On Jan 17, 2008 5:13 PM, Sagi Rotem <Sagir at mellanox.co.il> wrote:
> Did you check that the PCIe slot you are using is x8

The motherboard is an Intel(R) Server Board S5000PAL
(http://www.intel.com/design/servers/boards/s5000pal/), which has both
x4 and x8 slots. I'm not sure the information from lspci can be trusted,
but lspci says it's a x8 slot:

# lspci -vv
...
08:00.0 InfiniBand: Mellanox Technologies MT25204 [InfiniHost III Lx
HCA] (rev a0)
        Subsystem: Mellanox Technologies MT25204 [InfiniHost III Lx HCA]
        Control: I/O- Mem+ BusMaster+ SpecCycle- MemWINV- VGASnoop-
ParErr- Stepping- SERR- FastB2B-
        Status: Cap+ 66MHz- UDF- FastB2B- ParErr- DEVSEL=fast >TAbort-
<TAbort- <MAbort- >SERR- <PERR-
        Latency: 0, Cache Line Size: 64 bytes
        Interrupt: pin A routed to IRQ 16
        Region 0: Memory at b9300000 (64-bit, non-prefetchable)
[size=1M]
        Region 2: Memory at b8000000 (64-bit, prefetchable) [size=8M]
        Capabilities: [40] Power Management version 2
                Flags: PMEClk- DSI- D1- D2- AuxCurrent=0mA
PME(D0-,D1-,D2-,D3hot-,D3cold-)
                Status: D0 PME-Enable- DSel=0 DScale=0 PME-
        Capabilities: [48] Vital Product Data
        Capabilities: [90] Message Signalled Interrupts: Mask- 64bit+
Queue=0/5 Enable-
                Address: 0000000000000000  Data: 0000
        Capabilities: [84] MSI-X: Enable- Mask- TabSize=32
                Vector table: BAR=0 offset=00082000
                PBA: BAR=0 offset=00082200
        Capabilities: [60] Express Endpoint IRQ 0
                Device: Supported: MaxPayload 128 bytes, PhantFunc 0,
ExtTag-
                Device: Latency L0s <64ns, L1 unlimited
                Device: AtnBtn- AtnInd- PwrInd-
                Device: Errors: Correctable+ Non-Fatal+ Fatal+
Unsupported-
                Device: RlxdOrd- ExtTag- PhantFunc- AuxPwr- NoSnoop-
                Device: MaxPayload 128 bytes, MaxReadReq 128 bytes
                Link: Supported Speed 2.5Gb/s, Width x8, ASPM L0s, Port
8
                Link: Latency L0s unlimited, L1 unlimited
                Link: ASPM Disabled RCB 64 bytes CommClk- ExtSynch-
                Link: Speed 2.5Gb/s, Width x8 ...

Bart.