<html xmlns:v="urn:schemas-microsoft-com:vml" xmlns:o="urn:schemas-microsoft-com:office:office" xmlns:w="urn:schemas-microsoft-com:office:word" xmlns:m="http://schemas.microsoft.com/office/2004/12/omml" xmlns="http://www.w3.org/TR/REC-html40"><head><meta http-equiv=Content-Type content="text/html; charset=us-ascii"><meta name=Generator content="Microsoft Word 14 (filtered medium)"><style><!--
/* Font Definitions */
@font-face
{font-family:Wingdings;
panose-1:5 0 0 0 0 0 0 0 0 0;}
@font-face
{font-family:"Cambria Math";
panose-1:2 4 5 3 5 4 6 3 2 4;}
@font-face
{font-family:Calibri;
panose-1:2 15 5 2 2 2 4 3 2 4;}
@font-face
{font-family:Tahoma;
panose-1:2 11 6 4 3 5 4 4 2 4;}
/* Style Definitions */
p.MsoNormal, li.MsoNormal, div.MsoNormal
{margin:0in;
margin-bottom:.0001pt;
font-size:11.0pt;
font-family:"Calibri","sans-serif";}
a:link, span.MsoHyperlink
{mso-style-priority:99;
color:blue;
text-decoration:underline;}
a:visited, span.MsoHyperlinkFollowed
{mso-style-priority:99;
color:purple;
text-decoration:underline;}
p.MsoListParagraph, li.MsoListParagraph, div.MsoListParagraph
{mso-style-priority:34;
margin-top:0in;
margin-right:0in;
margin-bottom:0in;
margin-left:.5in;
margin-bottom:.0001pt;
font-size:11.0pt;
font-family:"Calibri","sans-serif";}
span.EmailStyle18
{mso-style-type:personal;
font-family:"Calibri","sans-serif";
color:windowtext;}
span.EmailStyle19
{mso-style-type:personal-reply;
font-family:"Times New Roman","serif";
color:windowtext;}
.MsoChpDefault
{mso-style-type:export-only;
font-size:10.0pt;}
@page WordSection1
{size:8.5in 11.0in;
margin:1.0in 1.0in 1.0in 1.0in;}
div.WordSection1
{page:WordSection1;}
/* List Definitions */
@list l0
{mso-list-id:218445561;
mso-list-type:hybrid;
mso-list-template-ids:-2134221482 -331042242 134807555 134807557 134807553 134807555 134807557 134807553 134807555 134807557;}
@list l0:level1
{mso-level-start-at:0;
mso-level-number-format:bullet;
mso-level-text:\F0B7;
mso-level-tab-stop:none;
mso-level-number-position:left;
text-indent:-.25in;
font-family:Symbol;
mso-fareast-font-family:"Times New Roman";
mso-bidi-font-family:Arial;}
@list l0:level2
{mso-level-number-format:bullet;
mso-level-text:o;
mso-level-tab-stop:none;
mso-level-number-position:left;
text-indent:-.25in;
font-family:"Courier New";}
@list l0:level3
{mso-level-number-format:bullet;
mso-level-text:\F0A7;
mso-level-tab-stop:none;
mso-level-number-position:left;
text-indent:-.25in;
font-family:Wingdings;}
@list l0:level4
{mso-level-number-format:bullet;
mso-level-text:\F0B7;
mso-level-tab-stop:none;
mso-level-number-position:left;
text-indent:-.25in;
font-family:Symbol;}
@list l0:level5
{mso-level-number-format:bullet;
mso-level-text:o;
mso-level-tab-stop:none;
mso-level-number-position:left;
text-indent:-.25in;
font-family:"Courier New";}
@list l0:level6
{mso-level-number-format:bullet;
mso-level-text:\F0A7;
mso-level-tab-stop:none;
mso-level-number-position:left;
text-indent:-.25in;
font-family:Wingdings;}
@list l0:level7
{mso-level-number-format:bullet;
mso-level-text:\F0B7;
mso-level-tab-stop:none;
mso-level-number-position:left;
text-indent:-.25in;
font-family:Symbol;}
@list l0:level8
{mso-level-number-format:bullet;
mso-level-text:o;
mso-level-tab-stop:none;
mso-level-number-position:left;
text-indent:-.25in;
font-family:"Courier New";}
@list l0:level9
{mso-level-number-format:bullet;
mso-level-text:\F0A7;
mso-level-tab-stop:none;
mso-level-number-position:left;
text-indent:-.25in;
font-family:Wingdings;}
ol
{margin-bottom:0in;}
ul
{margin-bottom:0in;}
--></style><!--[if gte mso 9]><xml>
<o:shapedefaults v:ext="edit" spidmax="1026" />
</xml><![endif]--><!--[if gte mso 9]><xml>
<o:shapelayout v:ext="edit">
<o:idmap v:ext="edit" data="1" />
</o:shapelayout></xml><![endif]--></head><body lang=EN-US link=blue vlink=purple><div class=WordSection1><p class=MsoNormal><span style='font-size:12.0pt;font-family:"Times New Roman","serif"'>Hi,<o:p></o:p></span></p><p class=MsoNormal><span style='font-size:12.0pt;font-family:"Times New Roman","serif"'><o:p> </o:p></span></p><p class=MsoNormal><span style='font-size:12.0pt;font-family:"Times New Roman","serif"'>Have you run ibdiagnet to verify that your link width and speed is what you expect on all links?<o:p></o:p></span></p><p class=MsoNormal><span style='font-size:12.0pt;font-family:"Times New Roman","serif"'><o:p> </o:p></span></p><p class=MsoNormal><span style='font-size:12.0pt;font-family:"Times New Roman","serif"'>Thanks<o:p></o:p></span></p><p class=MsoNormal><span style='font-size:12.0pt;font-family:"Times New Roman","serif"'><o:p> </o:p></span></p><p class=MsoNormal><span style='font-size:12.0pt;font-family:"Times New Roman","serif"'>Rupert Dance<o:p></o:p></span></p><p class=MsoNormal><span style='font-size:12.0pt;font-family:"Times New Roman","serif"'><o:p> </o:p></span></p><p class=MsoNormal><span style='font-size:12.0pt;font-family:"Times New Roman","serif"'>Software Forge<o:p></o:p></span></p><p class=MsoNormal><span style='font-size:12.0pt;font-family:"Times New Roman","serif"'><o:p> </o:p></span></p><div><div style='border:none;border-top:solid #B5C4DF 1.0pt;padding:3.0pt 0in 0in 0in'><p class=MsoNormal><b><span style='font-size:10.0pt;font-family:"Tahoma","sans-serif"'>From:</span></b><span style='font-size:10.0pt;font-family:"Tahoma","sans-serif"'> ewg-bounces@lists.openfabrics.org [mailto:ewg-bounces@lists.openfabrics.org] <b>On Behalf Of </b>koen_segers@computacenter.com<br><b>Sent:</b> Monday, October 17, 2011 3:22 AM<br><b>To:</b> ewg@lists.openfabrics.org<br><b>Subject:</b> [ewg] 200m cable results in slower rdma read performance? [ CC Anti-Virus checked ]<o:p></o:p></span></p></div></div><p class=MsoNormal><o:p> </o:p></p><p class=MsoNormal><span lang=EN-GB>Hi, <o:p></o:p></span></p><p class=MsoNormal><span lang=EN-GB><o:p> </o:p></span></p><p class=MsoNormal><span lang=EN-GB>In my test setup I have 3 servers of which 2 are residing in Datacenter1 and the other in Datacenter2.<o:p></o:p></span></p><p class=MsoNormal><span lang=EN-GB>If I do a rdma test between datacenters, I get a much lower performance than if I would do the test between servers residing in the same datacenter.<o:p></o:p></span></p><p class=MsoNormal><span lang=EN-GB><o:p> </o:p></span></p><p class=MsoNormal><span lang=EN-GB><o:p> </o:p></span></p><p class=MsoNormal><span lang=EN-GB>DC1: gpfsprod1n1, gpfsprod1n3<o:p></o:p></span></p><p class=MsoNormal><span lang=EN-GB>DC2: gpfsprod1n2<o:p></o:p></span></p><p class=MsoNormal><span lang=EN-GB><o:p> </o:p></span></p><p class=MsoNormal><span lang=EN-GB>08:54:48|root@gpfsprod1n1:~ 0 # qperf -t 5 cic-gpfsprod1n2 rc_rdma_write_bw<o:p></o:p></span></p><p class=MsoNormal><span lang=EN-GB>rc_rdma_write_bw:<o:p></o:p></span></p><p class=MsoNormal><span lang=EN-GB> bw = 1.9 GB/sec<o:p></o:p></span></p><p class=MsoNormal><span lang=EN-GB>08:54:59|root@gpfsprod1n1:~ 0 # qperf -t 5 cic-gpfsprod1n3 rc_rdma_write_bw<o:p></o:p></span></p><p class=MsoNormal><span lang=EN-GB>rc_rdma_write_bw:<o:p></o:p></span></p><p class=MsoNormal><span lang=EN-GB> bw = 3.39 GB/sec<o:p></o:p></span></p><p class=MsoNormal><span lang=EN-GB><o:p> </o:p></span></p><p class=MsoNormal><span lang=EN-GB style='font-size:10.0pt;font-family:"Arial","sans-serif";color:black;mso-fareast-language:EN-GB'><o:p> </o:p></span></p><p class=MsoNormal><span lang=EN-GB style='font-size:10.0pt;font-family:"Arial","sans-serif";color:black;mso-fareast-language:EN-GB'>The setup contains two pairs of edge switches (on each datacenter one pair) and two spine switches (each datacenter one), configured as a non blocking fat tree.<o:p></o:p></span></p><p class=MsoNormal><span lang=EN-GB style='font-size:10.0pt;font-family:"Arial","sans-serif";color:black;mso-fareast-language:EN-GB'>So:<o:p></o:p></span></p><p class=MsoNormal><span lang=EN-GB style='font-size:10.0pt;font-family:"Arial","sans-serif";color:black;mso-fareast-language:EN-GB'>The servers are connected to the edge switches.<o:p></o:p></span></p><p class=MsoNormal><span lang=EN-GB style='font-size:10.0pt;font-family:"Arial","sans-serif";color:black;mso-fareast-language:EN-GB'>The spine switches are connected to all edge switches.<o:p></o:p></span></p><p class=MsoNormal><span lang=EN-GB style='font-size:10.0pt;font-family:"Arial","sans-serif";color:black;mso-fareast-language:EN-GB'><o:p> </o:p></span></p><p class=MsoNormal><span lang=EN-GB style='font-size:10.0pt;font-family:"Arial","sans-serif";color:black;mso-fareast-language:EN-GB'>These are the cables we are using:<o:p></o:p></span></p><p class=MsoListParagraph style='text-indent:-.25in;mso-list:l0 level1 lfo2'><![if !supportLists]><span lang=EN-GB style='font-size:10.0pt;font-family:Symbol;color:black;mso-fareast-language:EN-GB'><span style='mso-list:Ignore'>·<span style='font:7.0pt "Times New Roman"'> </span></span></span><![endif]><span lang=EN-GB style='font-size:10.0pt;font-family:"Arial","sans-serif";color:black;mso-fareast-language:EN-GB'>Length 5m Vendor Name: WLGORE Code: QSFP+ Vendor PN: 498385-B24 Vendor Rev: D Vendor SN xxxx<o:p></o:p></span></p><p class=MsoListParagraph style='text-indent:-.25in;mso-list:l0 level1 lfo2'><![if !supportLists]><span lang=EN-GB style='font-size:10.0pt;font-family:Symbol;color:black;mso-fareast-language:EN-GB'><span style='mso-list:Ignore'>·<span style='font:7.0pt "Times New Roman"'> </span></span></span><![endif]><span lang=EN-GB style='font-size:10.0pt;font-family:"Arial","sans-serif";color:black;mso-fareast-language:EN-GB'>Length 200m Vendor Name: MOLEX Code: QSFP+ Vendor PN: 106410-1200 Vendor Rev: A Vendor SN xxxx<o:p></o:p></span></p><p class=MsoNormal><span lang=EN-GB style='font-size:10.0pt;font-family:"Arial","sans-serif";color:black;mso-fareast-language:EN-GB'><o:p> </o:p></span></p><p class=MsoNormal><span lang=EN-GB style='font-size:10.0pt;font-family:"Arial","sans-serif";color:black;mso-fareast-language:EN-GB'><o:p> </o:p></span></p><p class=MsoNormal><span lang=EN-GB style='font-size:10.0pt;font-family:"Arial","sans-serif";color:black;mso-fareast-language:EN-GB'>Can someone tell me why this is so? And maybe how I can solve this?<o:p></o:p></span></p><p class=MsoNormal><span lang=EN-GB style='font-size:10.0pt;font-family:"Arial","sans-serif";color:black;mso-fareast-language:EN-GB'><o:p> </o:p></span></p><p class=MsoNormal><span lang=EN-GB style='font-size:10.0pt;font-family:"Arial","sans-serif";color:black;mso-fareast-language:EN-GB'>Best regards,<o:p></o:p></span></p><p class=MsoNormal><span lang=EN-GB style='font-size:10.0pt;font-family:"Arial","sans-serif";color:black;mso-fareast-language:EN-GB'><o:p> </o:p></span></p><p class=MsoNormal><span lang=EN-GB style='font-size:10.0pt;font-family:"Arial","sans-serif";color:black;mso-fareast-language:EN-GB'><o:p> </o:p></span></p><p class=MsoNormal><b><span lang=EN-GB style='font-size:10.0pt;font-family:"Arial","sans-serif";color:black;mso-fareast-language:EN-GB'>Koen Segers<o:p></o:p></span></b></p><p class=MsoNormal><b><span lang=EN-GB style='font-size:10.0pt;font-family:"Arial","sans-serif";color:black;mso-fareast-language:EN-GB'>Enterprise Consultant<o:p></o:p></span></b></p><p class=MsoNormal><span lang=EN-GB style='font-size:10.0pt;font-family:"Arial","sans-serif";color:black;mso-fareast-language:EN-GB'> <o:p></o:p></span></p><p class=MsoNormal><b><span lang=EN-GB style='font-size:10.0pt;font-family:"Arial","sans-serif";color:#24387F;mso-fareast-language:EN-GB'>Computacenter<o:p></o:p></span></b></p><p class=MsoNormal><span lang=EN-GB style='font-size:10.0pt;font-family:"Arial","sans-serif";color:#009DDC;mso-fareast-language:EN-GB'>Services & Solutions<o:p></o:p></span></p><p class=MsoNormal><span lang=EN-GB style='font-size:10.0pt;font-family:"Arial","sans-serif";color:black;mso-fareast-language:EN-GB'> <o:p></o:p></span></p><p class=MsoNormal><span lang=NL style='font-size:10.0pt;font-family:"Arial","sans-serif";color:black;mso-fareast-language:EN-GB'>Ikaroslaan 31<o:p></o:p></span></p><p class=MsoNormal><span lang=NL style='font-size:10.0pt;font-family:"Arial","sans-serif";color:black;mso-fareast-language:EN-GB'>B-1930 Zaventem<o:p></o:p></span></p><p class=MsoNormal><span lang=NL style='font-size:10.0pt;font-family:"Arial","sans-serif";color:black;mso-fareast-language:EN-GB'>Belgium<o:p></o:p></span></p><p class=MsoNormal><span lang=NL style='font-size:10.0pt;font-family:"Arial","sans-serif";color:black;mso-fareast-language:EN-GB'> <o:p></o:p></span></p><p class=MsoNormal><span lang=NL style='font-size:10.0pt;font-family:"Arial","sans-serif";color:black;mso-fareast-language:EN-GB'>Tel: +32 2 704 94 67<o:p></o:p></span></p><p class=MsoNormal><span lang=EN-GB style='font-size:10.0pt;font-family:"Arial","sans-serif";color:black;mso-fareast-language:EN-GB'>Fax: +32 2 704 95 95<o:p></o:p></span></p><p class=MsoNormal><span lang=EN-GB style='font-size:10.0pt;font-family:"Arial","sans-serif";color:black;mso-fareast-language:EN-GB'>Mob: +32 497 909353<o:p></o:p></span></p><p class=MsoNormal><span lang=NL style='font-size:10.0pt;font-family:"Arial","sans-serif";color:black;mso-fareast-language:EN-GB'><a href="mailto:koen_segers@computacenter.com" target="_blank"><span lang=EN-GB style='color:#24387F'>koen_segers@computacenter.com</span></a></span><span lang=EN-GB style='font-size:10.0pt;font-family:"Arial","sans-serif";color:black;mso-fareast-language:EN-GB'><o:p></o:p></span></p><p class=MsoNormal><span lang=NL style='font-size:10.0pt;font-family:"Arial","sans-serif";color:black;mso-fareast-language:EN-GB'><a href="http://www.computacenter.com/benelux" target="_blank"><span style='color:#24387F'>www.computacenter.com/benelux</span></a><o:p></o:p></span></p><p class=MsoNormal><span lang=EN-GB><o:p> </o:p></span></p><p class=MsoNormal><span lang=EN-GB style='font-size:12.0pt;font-family:"Times New Roman","serif"'><br><br><br></span><span lang=EN-GB style='font-size:8.0pt;font-family:"Arial","sans-serif";color:black'>Visit us at <a href="http://www.computacenter.com/">http://www.computacenter.com/</a><br>Computacenter: Transforming IT service delivery.<br><br>========================== Disclaimer ==================================<br>The information in this email is confidential, and is intended solely for the addressee(s). If you are not the intended recipient of this email please let us know by reply and then delete it from your system; you should not copy this message or disclose its contents to anyone. Due to the integrity risk of sending emails over the Internet, Computacenter will accept no liability for any comments and / or attachments contained within this email.<br>========================== Disclaimer ==================================</span><span lang=EN-GB style='font-size:12.0pt;font-family:"Times New Roman","serif"'><o:p></o:p></span></p></div></body></html>