<html xmlns:v="urn:schemas-microsoft-com:vml" xmlns:o="urn:schemas-microsoft-com:office:office" xmlns:w="urn:schemas-microsoft-com:office:word" xmlns:m="http://schemas.microsoft.com/office/2004/12/omml" xmlns="http://www.w3.org/TR/REC-html40">
<head>
<meta http-equiv=Content-Type content="text/html; charset=us-ascii">
<meta name=Generator content="Microsoft Word 12 (filtered medium)">
<style>
<!--
/* Font Definitions */
@font-face
{font-family:"Cambria Math";
panose-1:2 4 5 3 5 4 6 3 2 4;}
@font-face
{font-family:Calibri;
panose-1:2 15 5 2 2 2 4 3 2 4;}
@font-face
{font-family:Consolas;
panose-1:2 11 6 9 2 2 4 3 2 4;}
/* Style Definitions */
p.MsoNormal, li.MsoNormal, div.MsoNormal
{margin:0cm;
margin-bottom:.0001pt;
font-size:11.0pt;
font-family:"Calibri","sans-serif";}
a:link, span.MsoHyperlink
{mso-style-priority:99;
color:blue;
text-decoration:underline;}
a:visited, span.MsoHyperlinkFollowed
{mso-style-priority:99;
color:purple;
text-decoration:underline;}
p.MsoPlainText, li.MsoPlainText, div.MsoPlainText
{mso-style-priority:99;
mso-style-link:"Plain Text Char";
margin:0cm;
margin-bottom:.0001pt;
font-size:10.5pt;
font-family:Consolas;}
span.EmailStyle17
{mso-style-type:personal-compose;
font-family:"Calibri","sans-serif";
color:windowtext;}
span.PlainTextChar
{mso-style-name:"Plain Text Char";
mso-style-priority:99;
mso-style-link:"Plain Text";
font-family:Consolas;}
.MsoChpDefault
{mso-style-type:export-only;}
@page Section1
{size:612.0pt 792.0pt;
margin:72.0pt 72.0pt 72.0pt 72.0pt;}
div.Section1
{page:Section1;}
-->
</style>
<!--[if gte mso 9]><xml>
<o:shapedefaults v:ext="edit" spidmax="1026" />
</xml><![endif]--><!--[if gte mso 9]><xml>
<o:shapelayout v:ext="edit">
<o:idmap v:ext="edit" data="1" />
</o:shapelayout></xml><![endif]-->
</head>
<body lang=EN-US link=blue vlink=purple>
<div class=Section1>
<p class=MsoPlainText>I see very low qperf TCP performance of IPOIB on Centos
5.2 by default (RDMA performance is fine). <o:p></o:p></p>
<p class=MsoPlainText><o:p> </o:p></p>
<p class=MsoPlainText>Centos sets the maximum MTU for ib0 on boot
automatically. For CM centos sets the maximum MTU that is just slightly less
than the logical 65536 setting. Connected mode only seems to like nice round
MTU values (eg. 32768 or 2048) so the default setting always gives terrible
performance. <o:p></o:p></p>
<p class=MsoPlainText><o:p> </o:p></p>
<p class=MsoPlainText>Datagram mode seems to have no problem with odd MTU
values like 2044 (This seems to be the max in Datagram mode). <o:p></o:p></p>
<p class=MsoPlainText><o:p> </o:p></p>
<p class=MsoPlainText>The other thing I noticed was Centos warns that multicast
will be broken with MTUs above 2044, this seems to be true, I don’t need
it so disabled multicast on the Infiniband interface. <o:p></o:p></p>
<p class=MsoPlainText><o:p> </o:p></p>
<p class=MsoPlainText>Is the behaviour I’m seeing normal or is this some
kind of bug? (Either in OFED running on Centos 5.2 or QPerf) Results with
various settings below.<o:p></o:p></p>
<p class=MsoPlainText><o:p> </o:p></p>
<p class=MsoPlainText>Rob<o:p></o:p></p>
<p class=MsoPlainText><o:p> </o:p></p>
<p class=MsoPlainText>Default (Connected mode)<o:p></o:p></p>
<p class=MsoPlainText>[root@mrhappy OFED-1.3.1]# qperf 192.168.10.161 tcp_bw
tcp_lat<o:p></o:p></p>
<p class=MsoPlainText>tcp_bw:<o:p></o:p></p>
<p class=MsoPlainText> bw = 32.8 KB/sec<o:p></o:p></p>
<p class=MsoPlainText>tcp_lat:<o:p></o:p></p>
<p class=MsoPlainText> latency = 42.2 us<o:p></o:p></p>
<p class=MsoPlainText><o:p> </o:p></p>
<p class=MsoPlainText>Datagram Mode with 2044 MTU (Max for Datagram):<o:p></o:p></p>
<p class=MsoPlainText>[root@mrtickle OFED-1.3.1]# qperf 192.168.10.160 tcp_bw
tcp_lat<o:p></o:p></p>
<p class=MsoPlainText>tcp_bw:<o:p></o:p></p>
<p class=MsoPlainText> bw = 400 MB/sec<o:p></o:p></p>
<p class=MsoPlainText>tcp_lat:<o:p></o:p></p>
<p class=MsoPlainText> latency = 41.2 us<o:p></o:p></p>
<p class=MsoPlainText><o:p> </o:p></p>
<p class=MsoPlainText>Connected Mode with 32768 MTU:<o:p></o:p></p>
<p class=MsoPlainText>[root@mrtickle ~]# qperf 192.168.10.160 tcp_bw tcp_lat<o:p></o:p></p>
<p class=MsoPlainText>tcp_bw:<o:p></o:p></p>
<p class=MsoPlainText> bw = 866 MB/sec<o:p></o:p></p>
<p class=MsoPlainText>tcp_lat:<o:p></o:p></p>
<p class=MsoPlainText> latency = 30.8 us<o:p></o:p></p>
<p class=MsoNormal><o:p> </o:p></p>
</div>
</body>
<!--[object_id=#saq.co.uk#]--><FONT face=Tahoma size=2><FONT color=#0000ff>
<P class=MsoPlainText style="MARGIN: 0cm 0cm 0pt"><SPAN style="FONT-SIZE: 10pt; COLOR: black; FONT-FAMILY: 'Verdana','sans-serif'"><STRONG></STRONG></SPAN></P>
<P class=MsoPlainText style="MARGIN: 0cm 0cm 0pt"><SPAN style="FONT-SIZE: 10pt; COLOR: black; FONT-FAMILY: 'Verdana','sans-serif'"><STRONG></STRONG></SPAN> </P>
<P class=MsoPlainText style="MARGIN: 0cm 0cm 0pt"><SPAN style="FONT-SIZE: 10pt; COLOR: black; FONT-FAMILY: 'Verdana','sans-serif'"><STRONG>The SAQ Group</STRONG></SPAN></P>
<P class=MsoPlainText style="MARGIN: 0cm 0cm 0pt"><SPAN style="FONT-SIZE: 10pt; COLOR: black; FONT-FAMILY: 'Verdana','sans-serif'"><STRONG>Registered Office: <?xml:namespace prefix = st1 ns = "urn:schemas-microsoft-com:office:smarttags" /><st1:Street w:st="on"><st1:address w:st="on">18 Chapel Street, Petersfield, Hampshire GU32 3DZ</st1:address></st1:Street></STRONG><BR>SEMTEC Limited Trading as SAQ is Registered in <st1:country-region w:st="on">England</st1:country-region> & <st1:country-region w:st="on"><st1:place w:st="on">Wales</st1:place></st1:country-region><BR>Company Number: 06481952</SPAN></P>
<P class=MsoPlainText style="MARGIN: 0cm 0cm 0pt"><SPAN style="FONT-SIZE: 10pt; COLOR: black; FONT-FAMILY: 'Verdana','sans-serif'"><?xml:namespace prefix = o ns = "urn:schemas-microsoft-com:office:office" /><o:p> </o:p></SPAN></P>
<P class=MsoPlainText style="MARGIN: 0cm 0cm 0pt"><SPAN style="FONT-SIZE: 10pt; COLOR: black; FONT-FAMILY: 'Verdana','sans-serif'"><A href="http://www.saqnet.co.uk/">http://www.saqnet.co.uk</A> AS29219<o:p></o:p></SPAN></P>
<P class=MsoPlainText style="MARGIN: 0cm 0cm 0pt"><SPAN style="FONT-SIZE: 10pt; COLOR: black; FONT-FAMILY: 'Verdana','sans-serif'">SAQ Group Delivers high quality, honestly priced communication and I.T. services to UK Business.<o:p></o:p></SPAN></P>
<P class=MsoPlainText style="MARGIN: 0cm 0cm 0pt"><SPAN style="FONT-SIZE: 10pt; COLOR: black; FONT-FAMILY: 'Verdana','sans-serif'">DSL : Domains : Email : Hosting : CoLo : Servers : Racks : Transit : Backups : Managed Networks : Remote Support.</SPAN></P>
<P class=MsoPlainText style="MARGIN: 0cm 0cm 0pt"><SPAN style="FONT-SIZE: 10pt; COLOR: black; FONT-FAMILY: 'Verdana','sans-serif'"></SPAN> </P>
<P class=MsoPlainText style="MARGIN: 0cm 0cm 0pt"><SPAN style="FONT-SIZE: 10pt; COLOR: black; FONT-FAMILY: 'Verdana','sans-serif'">Find us in <A href="http://www.thebestof.co.uk/petersfield">http://www.thebestof.co.uk/petersfield</A></SPAN></P>
<P class=MsoPlainText style="MARGIN: 0cm 0cm 0pt"><SPAN style="FONT-SIZE: 10pt; COLOR: black; FONT-FAMILY: 'Verdana','sans-serif'"> </SPAN></FONT></FONT></P></html>