[ewg] dissappointing IPoIB performance

[richard Croucher]

I've been undertaking some internal QA testing of the Mellanox CX3's.

An observation that I've seen for some time and is most likely to do
with the IPoIB implementation rather than the HCA is that the latency of
IPoIB is getting increasingly poor in comparison with the standard
kernel TCP/IP stack over 10g Ethernet.

If we look at the results for the CX3's running both 10G Ethernet and
40G InfiniBand, on same serve hardware,  I get the following median
latency with my test setup.  Results are with my own test program so are
only meaningful as a comparison with the other configurations running
the same test.

Running OFED 1.5.3 and RH 6.0

IPoIB (connected)   TCP 33.67 uS    (switchless)
IPoIB (datagram)    TCP 31.63 uS    (switchless)
IPoIB (connected)  UDP 24.78 uS    (switchless)
IPoIB (datagram)  UDP 24.28 uS     (switchless)
IPoIB (connected)  UDP 25.37 uS    (1 hop) between ports on same switch
IPoIB (connected)  TCP 34.48 uS     (1 hop)
10G Ethernet      UDP 24.04uS     (2 hops) across a LAG connected pair
of Ethernet switches
10G Ethernet      TCP  34.59 uS    (2 hops)

The Mellanox Ethernet drivers are tuned for low latency rather than
throughput, but I would have hoped that given the 4x extra bandwidth
available it would have helped the InfiniBand drivers outperform.

I've seen similar results for CX2 .    10G ethernet is increasingly
looking like the better option for low latency, particularly with the
current generation of low latency Ethernet switches.  Switchless
Ethernet has been better for some time than switchless InfiniBand, but
it now looks to be the case in switched environments as well.  I think
this reflects that there has been a lot of effort tweaking and tuning
TCP/IP over Ethernet and its low level drivers, with very little
activity on the IPoIB front.  Unless we see improvements here it will
get increasingly difficult to justify InfiniBand deployments.



--

Richard Croucher
www.informatix-sol.com
+44-7802-213901 

-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.openfabrics.org/pipermail/ewg/attachments/20120220/07b24f94/attachment.html>
-------------- next part --------------
A non-text attachment was scrubbed...
Name: smime.p7s
Type: application/x-pkcs7-signature
Size: 5147 bytes
Desc: not available
URL: <http://lists.openfabrics.org/pipermail/ewg/attachments/20120220/07b24f94/attachment.bin>