[openib-general] Latency Problem with MT25204 HCAs

[Don.Dhondt at Bull.com]

We are seeing a latency problem that seems to be specific to the Mellanox 
MT25204 HCA.
We do not see the same problem with MT25208 HCAs running in MT23108 
compatibility mode.
The problem is demonstrated running multiple streams of ib_rdma_lat.

On the SDR MT25208 HCA:
                typical latency
1 stream        3.70 usec
2 streams       4.47 usec
4 streams       6.74 usec

On the DDR MT25204 HCA:
                typical latency
1 stream        3.03 usec
2 streams       7.36 usec
4 streams       22.4 usec

Can anyone explain this behavior?

We are running OFED 1.0 release on a pair of EM64T dual CPU nodes.

ibstat output:
CA 'mthca0'
        CA type: MT25208 (MT23108 compat mode)
        Number of ports: 2
        Firmware version: 4.7.400
        Hardware version: a0
        Node GUID: 0x0005ad0000035950
        System image GUID: 0x0005ad000100d050
        Port 1:
                State: Active
                Physical state: LinkUp
                Rate: 10
                Base lid: 21
                LMC: 0
                SM lid: 16
                Capability mask: 0x02500a68
                Port GUID: 0x0005ad0000035951
        Port 2:
                State: Down
                Physical state: Polling
                Rate: 10
                Base lid: 0
                LMC: 0
                SM lid: 0
                Capability mask: 0x02500a68
                Port GUID: 0x0005ad0000035952
CA 'mthca1'
        CA type: MT25204
        Number of ports: 1
        Firmware version: 1.0.800
        Hardware version: a0
        Node GUID: 0x0002c90200216e40
        System image GUID: 0x0002c90200216e43
        Port 1:
                State: Active
                Physical state: LinkUp
                Rate: 20
                Base lid: 22
                LMC: 0
                SM lid: 22
                Capability mask: 0x02500a6a
                Port GUID: 0x0002c90200216e41

-Don
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.openfabrics.org/pipermail/general/attachments/20060906/99c86f68/attachment.html>