<html xmlns:v="urn:schemas-microsoft-com:vml" xmlns:o="urn:schemas-microsoft-com:office:office" xmlns:w="urn:schemas-microsoft-com:office:word" xmlns:m="http://schemas.microsoft.com/office/2004/12/omml" xmlns="http://www.w3.org/TR/REC-html40">
<head>
<meta http-equiv="Content-Type" content="text/html; charset=utf-8">
<meta name="Generator" content="Microsoft Word 14 (filtered medium)">
<style><!--
/* Font Definitions */
@font-face
{font-family:"Cambria Math";
panose-1:2 4 5 3 5 4 6 3 2 4;}
@font-face
{font-family:Calibri;
panose-1:2 15 5 2 2 2 4 3 2 4;}
@font-face
{font-family:Tahoma;
panose-1:2 11 6 4 3 5 4 4 2 4;}
/* Style Definitions */
p.MsoNormal, li.MsoNormal, div.MsoNormal
{margin:0in;
margin-bottom:.0001pt;
font-size:12.0pt;
font-family:"Times New Roman","serif";}
a:link, span.MsoHyperlink
{mso-style-priority:99;
color:blue;
text-decoration:underline;}
a:visited, span.MsoHyperlinkFollowed
{mso-style-priority:99;
color:purple;
text-decoration:underline;}
span.EmailStyle17
{mso-style-type:personal-reply;
font-family:"Calibri","sans-serif";
color:#1F497D;}
.MsoChpDefault
{mso-style-type:export-only;
font-size:10.0pt;}
@page WordSection1
{size:8.5in 11.0in;
margin:1.0in 1.0in 1.0in 1.0in;}
div.WordSection1
{page:WordSection1;}
--></style><!--[if gte mso 9]><xml>
<o:shapedefaults v:ext="edit" spidmax="1026" />
</xml><![endif]--><!--[if gte mso 9]><xml>
<o:shapelayout v:ext="edit">
<o:idmap v:ext="edit" data="1" />
</o:shapelayout></xml><![endif]-->
</head>
<body lang="EN-US" link="blue" vlink="purple">
<div class="WordSection1">
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:"Calibri","sans-serif";color:#1F497D">Richard,<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:"Calibri","sans-serif";color:#1F497D"><o:p> </o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:"Calibri","sans-serif";color:#1F497D">Critical missing is the setup information. What is the server, CPU etc. Can you please provide?<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:"Calibri","sans-serif";color:#1F497D"><o:p> </o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:"Calibri","sans-serif";color:#1F497D">Gilad<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:"Calibri","sans-serif";color:#1F497D"><o:p> </o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:"Calibri","sans-serif";color:#1F497D"><o:p> </o:p></span></p>
<div>
<div style="border:none;border-top:solid #B5C4DF 1.0pt;padding:3.0pt 0in 0in 0in">
<p class="MsoNormal"><b><span style="font-size:10.0pt;font-family:"Tahoma","sans-serif"">From:</span></b><span style="font-size:10.0pt;font-family:"Tahoma","sans-serif""> ewg-bounces@lists.openfabrics.org [mailto:ewg-bounces@lists.openfabrics.org]
<b>On Behalf Of </b>richard Croucher<br>
<b>Sent:</b> Monday, February 20, 2012 8:50 AM<br>
<b>To:</b> ewg@lists.openfabrics.org<br>
<b>Subject:</b> [ewg] dissappointing IPoIB performance<o:p></o:p></span></p>
</div>
</div>
<p class="MsoNormal"><o:p> </o:p></p>
<p class="MsoNormal" style="margin-bottom:12.0pt">I've been undertaking some internal QA testing of the Mellanox CX3's.<br>
<br>
An observation that I've seen for some time and is most likely to do with the IPoIB implementation rather than the HCA is that the latency of IPoIB is getting increasingly poor in comparison with the standard kernel TCP/IP stack over 10g Ethernet.<br>
<br>
If we look at the results for the CX3's running both 10G Ethernet and 40G InfiniBand, on same serve hardware, I get the following median latency with my test setup. Results are with my own test program so are only meaningful as a comparison with the other
configurations running the same test.<br>
<br>
Running OFED 1.5.3 and RH 6.0<br>
<br>
IPoIB (connected) TCP 33.67 uS (switchless)<br>
IPoIB (datagram) TCP 31.63 uS (switchless)<br>
IPoIB (connected) UDP 24.78 uS (switchless)<br>
IPoIB (datagram) UDP 24.28 uS (switchless)<br>
IPoIB (connected) UDP 25.37 uS (1 hop) between ports on same switch<br>
IPoIB (connected) TCP 34.48 uS (1 hop)<br>
10G Ethernet UDP 24.04uS (2 hops) across a LAG connected pair of Ethernet switches<br>
10G Ethernet TCP 34.59 uS (2 hops)<br>
<br>
The Mellanox Ethernet drivers are tuned for low latency rather than throughput, but I would have hoped that given the 4x extra bandwidth available it would have helped the InfiniBand drivers outperform.<br>
<br>
I've seen similar results for CX2 . 10G ethernet is increasingly looking like the better option for low latency, particularly with the current generation of low latency Ethernet switches. Switchless Ethernet has been better for some time than switchless
InfiniBand, but it now looks to be the case in switched environments as well. I think this reflects that there has been a lot of effort tweaking and tuning TCP/IP over Ethernet and its low level drivers, with very little activity on the IPoIB front. Unless
we see improvements here it will get increasingly difficult to justify InfiniBand deployments.<br>
<br>
<br>
<o:p></o:p></p>
<table class="MsoNormalTable" border="0" cellspacing="0" cellpadding="0" width="100%" style="width:100.0%">
<tbody>
<tr>
<td style="padding:0in 0in 0in 0in">
<p class="MsoNormal" style="margin-bottom:12.0pt">--<br>
<br>
Richard Croucher<br>
<a href="http://www.informatix-sol.com">www.informatix-sol.com</a><br>
+44-7802-213901 <o:p></o:p></p>
</td>
</tr>
</tbody>
</table>
<p class="MsoNormal"><o:p> </o:p></p>
</div>
</body>
</html>