<html xmlns:v="urn:schemas-microsoft-com:vml" xmlns:o="urn:schemas-microsoft-com:office:office" xmlns:w="urn:schemas-microsoft-com:office:word" xmlns:m="http://schemas.microsoft.com/office/2004/12/omml" xmlns="http://www.w3.org/TR/REC-html40"><head><meta http-equiv=Content-Type content="text/html; charset=us-ascii"><meta name=Generator content="Microsoft Word 14 (filtered medium)"><style><!--
/* Font Definitions */
@font-face
{font-family:Wingdings;
panose-1:5 0 0 0 0 0 0 0 0 0;}
@font-face
{font-family:Wingdings;
panose-1:5 0 0 0 0 0 0 0 0 0;}
@font-face
{font-family:Calibri;
panose-1:2 15 5 2 2 2 4 3 2 4;}
/* Style Definitions */
p.MsoNormal, li.MsoNormal, div.MsoNormal
{margin:0cm;
margin-bottom:.0001pt;
font-size:11.0pt;
font-family:"Calibri","sans-serif";
mso-fareast-language:EN-US;}
a:link, span.MsoHyperlink
{mso-style-priority:99;
color:blue;
text-decoration:underline;}
a:visited, span.MsoHyperlinkFollowed
{mso-style-priority:99;
color:purple;
text-decoration:underline;}
p.MsoListParagraph, li.MsoListParagraph, div.MsoListParagraph
{mso-style-priority:34;
margin-top:0cm;
margin-right:0cm;
margin-bottom:0cm;
margin-left:36.0pt;
margin-bottom:.0001pt;
font-size:11.0pt;
font-family:"Calibri","sans-serif";
mso-fareast-language:EN-US;}
span.EmailStyle17
{mso-style-type:personal-compose;
font-family:"Calibri","sans-serif";
color:windowtext;}
.MsoChpDefault
{mso-style-type:export-only;
font-family:"Calibri","sans-serif";
mso-fareast-language:EN-US;}
@page WordSection1
{size:612.0pt 792.0pt;
margin:72.0pt 72.0pt 72.0pt 72.0pt;}
div.WordSection1
{page:WordSection1;}
/* List Definitions */
@list l0
{mso-list-id:218445561;
mso-list-type:hybrid;
mso-list-template-ids:-2134221482 -331042242 134807555 134807557 134807553 134807555 134807557 134807553 134807555 134807557;}
@list l0:level1
{mso-level-start-at:0;
mso-level-number-format:bullet;
mso-level-text:\F0B7;
mso-level-tab-stop:none;
mso-level-number-position:left;
text-indent:-18.0pt;
font-family:Symbol;
mso-fareast-font-family:"Times New Roman";
mso-bidi-font-family:Arial;}
@list l0:level2
{mso-level-number-format:bullet;
mso-level-text:o;
mso-level-tab-stop:none;
mso-level-number-position:left;
text-indent:-18.0pt;
font-family:"Courier New";}
@list l0:level3
{mso-level-number-format:bullet;
mso-level-text:\F0A7;
mso-level-tab-stop:none;
mso-level-number-position:left;
text-indent:-18.0pt;
font-family:Wingdings;}
@list l0:level4
{mso-level-number-format:bullet;
mso-level-text:\F0B7;
mso-level-tab-stop:none;
mso-level-number-position:left;
text-indent:-18.0pt;
font-family:Symbol;}
@list l0:level5
{mso-level-number-format:bullet;
mso-level-text:o;
mso-level-tab-stop:none;
mso-level-number-position:left;
text-indent:-18.0pt;
font-family:"Courier New";}
@list l0:level6
{mso-level-number-format:bullet;
mso-level-text:\F0A7;
mso-level-tab-stop:none;
mso-level-number-position:left;
text-indent:-18.0pt;
font-family:Wingdings;}
@list l0:level7
{mso-level-number-format:bullet;
mso-level-text:\F0B7;
mso-level-tab-stop:none;
mso-level-number-position:left;
text-indent:-18.0pt;
font-family:Symbol;}
@list l0:level8
{mso-level-number-format:bullet;
mso-level-text:o;
mso-level-tab-stop:none;
mso-level-number-position:left;
text-indent:-18.0pt;
font-family:"Courier New";}
@list l0:level9
{mso-level-number-format:bullet;
mso-level-text:\F0A7;
mso-level-tab-stop:none;
mso-level-number-position:left;
text-indent:-18.0pt;
font-family:Wingdings;}
ol
{margin-bottom:0cm;}
ul
{margin-bottom:0cm;}
--></style><!--[if gte mso 9]><xml>
<o:shapedefaults v:ext="edit" spidmax="1026" />
</xml><![endif]--><!--[if gte mso 9]><xml>
<o:shapelayout v:ext="edit">
<o:idmap v:ext="edit" data="1" />
</o:shapelayout></xml><![endif]--></head><body lang=EN-GB link=blue vlink=purple><div class=WordSection1><p class=MsoNormal>Hi, <o:p></o:p></p><p class=MsoNormal><o:p> </o:p></p><p class=MsoNormal>In my test setup I have 3 servers of which 2 are residing in Datacenter1 and the other in Datacenter2.<o:p></o:p></p><p class=MsoNormal>If I do a rdma test between datacenters, I get a much lower performance than if I would do the test between servers residing in the same datacenter.<o:p></o:p></p><p class=MsoNormal><o:p> </o:p></p><p class=MsoNormal><o:p> </o:p></p><p class=MsoNormal>DC1: gpfsprod1n1, gpfsprod1n3<o:p></o:p></p><p class=MsoNormal>DC2: gpfsprod1n2<o:p></o:p></p><p class=MsoNormal><o:p> </o:p></p><p class=MsoNormal>08:54:48|root@gpfsprod1n1:~ 0 # qperf -t 5 cic-gpfsprod1n2 rc_rdma_write_bw<o:p></o:p></p><p class=MsoNormal>rc_rdma_write_bw:<o:p></o:p></p><p class=MsoNormal> bw = 1.9 GB/sec<o:p></o:p></p><p class=MsoNormal>08:54:59|root@gpfsprod1n1:~ 0 # qperf -t 5 cic-gpfsprod1n3 rc_rdma_write_bw<o:p></o:p></p><p class=MsoNormal>rc_rdma_write_bw:<o:p></o:p></p><p class=MsoNormal> bw = 3.39 GB/sec<o:p></o:p></p><p class=MsoNormal><o:p> </o:p></p><p class=MsoNormal><span style='font-size:10.0pt;font-family:"Arial","sans-serif";color:black;mso-fareast-language:EN-GB'><o:p> </o:p></span></p><p class=MsoNormal><span style='font-size:10.0pt;font-family:"Arial","sans-serif";color:black;mso-fareast-language:EN-GB'>The setup contains two pairs of edge switches (on each datacenter one pair) and two spine switches (each datacenter one), configured as a non blocking fat tree.<o:p></o:p></span></p><p class=MsoNormal><span style='font-size:10.0pt;font-family:"Arial","sans-serif";color:black;mso-fareast-language:EN-GB'>So:<o:p></o:p></span></p><p class=MsoNormal><span style='font-size:10.0pt;font-family:"Arial","sans-serif";color:black;mso-fareast-language:EN-GB'>The servers are connected to the edge switches.<o:p></o:p></span></p><p class=MsoNormal><span style='font-size:10.0pt;font-family:"Arial","sans-serif";color:black;mso-fareast-language:EN-GB'>The spine switches are connected to all edge switches.<o:p></o:p></span></p><p class=MsoNormal><span style='font-size:10.0pt;font-family:"Arial","sans-serif";color:black;mso-fareast-language:EN-GB'><o:p> </o:p></span></p><p class=MsoNormal><span style='font-size:10.0pt;font-family:"Arial","sans-serif";color:black;mso-fareast-language:EN-GB'>These are the cables we are using:<o:p></o:p></span></p><p class=MsoListParagraph style='text-indent:-18.0pt;mso-list:l0 level1 lfo1'><![if !supportLists]><span style='font-size:10.0pt;font-family:Symbol;color:black;mso-fareast-language:EN-GB'><span style='mso-list:Ignore'>·<span style='font:7.0pt "Times New Roman"'> </span></span></span><![endif]><span style='font-size:10.0pt;font-family:"Arial","sans-serif";color:black;mso-fareast-language:EN-GB'>Length 5m Vendor Name: WLGORE Code: QSFP+ Vendor PN: 498385-B24 Vendor Rev: D Vendor SN xxxx<o:p></o:p></span></p><p class=MsoListParagraph style='text-indent:-18.0pt;mso-list:l0 level1 lfo1'><![if !supportLists]><span style='font-size:10.0pt;font-family:Symbol;color:black;mso-fareast-language:EN-GB'><span style='mso-list:Ignore'>·<span style='font:7.0pt "Times New Roman"'> </span></span></span><![endif]><span style='font-size:10.0pt;font-family:"Arial","sans-serif";color:black;mso-fareast-language:EN-GB'>Length 200m Vendor Name: MOLEX Code: QSFP+ Vendor PN: 106410-1200 Vendor Rev: A Vendor SN xxxx<o:p></o:p></span></p><p class=MsoNormal><span style='font-size:10.0pt;font-family:"Arial","sans-serif";color:black;mso-fareast-language:EN-GB'><o:p> </o:p></span></p><p class=MsoNormal><span style='font-size:10.0pt;font-family:"Arial","sans-serif";color:black;mso-fareast-language:EN-GB'><o:p> </o:p></span></p><p class=MsoNormal><span style='font-size:10.0pt;font-family:"Arial","sans-serif";color:black;mso-fareast-language:EN-GB'>Can someone tell me why this is so? And maybe how I can solve this?<o:p></o:p></span></p><p class=MsoNormal><span style='font-size:10.0pt;font-family:"Arial","sans-serif";color:black;mso-fareast-language:EN-GB'><o:p> </o:p></span></p><p class=MsoNormal><span style='font-size:10.0pt;font-family:"Arial","sans-serif";color:black;mso-fareast-language:EN-GB'>Best regards,<o:p></o:p></span></p><p class=MsoNormal><span style='font-size:10.0pt;font-family:"Arial","sans-serif";color:black;mso-fareast-language:EN-GB'><o:p> </o:p></span></p><p class=MsoNormal><span style='font-size:10.0pt;font-family:"Arial","sans-serif";color:black;mso-fareast-language:EN-GB'><o:p> </o:p></span></p><p class=MsoNormal><b><span style='font-size:10.0pt;font-family:"Arial","sans-serif";color:black;mso-fareast-language:EN-GB'>Koen Segers<o:p></o:p></span></b></p><p class=MsoNormal><b><span style='font-size:10.0pt;font-family:"Arial","sans-serif";color:black;mso-fareast-language:EN-GB'>Enterprise Consultant<o:p></o:p></span></b></p><p class=MsoNormal><span style='font-size:10.0pt;font-family:"Arial","sans-serif";color:black;mso-fareast-language:EN-GB'> <o:p></o:p></span></p><p class=MsoNormal><b><span style='font-size:10.0pt;font-family:"Arial","sans-serif";color:#24387F;mso-fareast-language:EN-GB'>Computacenter<o:p></o:p></span></b></p><p class=MsoNormal><span style='font-size:10.0pt;font-family:"Arial","sans-serif";color:#009DDC;mso-fareast-language:EN-GB'>Services & Solutions<o:p></o:p></span></p><p class=MsoNormal><span style='font-size:10.0pt;font-family:"Arial","sans-serif";color:black;mso-fareast-language:EN-GB'> <o:p></o:p></span></p><p class=MsoNormal><span lang=NL style='font-size:10.0pt;font-family:"Arial","sans-serif";color:black;mso-fareast-language:EN-GB'>Ikaroslaan 31<o:p></o:p></span></p><p class=MsoNormal><span lang=NL style='font-size:10.0pt;font-family:"Arial","sans-serif";color:black;mso-fareast-language:EN-GB'>B-1930 Zaventem<o:p></o:p></span></p><p class=MsoNormal><span lang=NL style='font-size:10.0pt;font-family:"Arial","sans-serif";color:black;mso-fareast-language:EN-GB'>Belgium<o:p></o:p></span></p><p class=MsoNormal><span lang=NL style='font-size:10.0pt;font-family:"Arial","sans-serif";color:black;mso-fareast-language:EN-GB'> <o:p></o:p></span></p><p class=MsoNormal><span lang=NL style='font-size:10.0pt;font-family:"Arial","sans-serif";color:black;mso-fareast-language:EN-GB'>Tel: +32 2 704 94 67<o:p></o:p></span></p><p class=MsoNormal><span style='font-size:10.0pt;font-family:"Arial","sans-serif";color:black;mso-fareast-language:EN-GB'>Fax: +32 2 704 95 95<o:p></o:p></span></p><p class=MsoNormal><span style='font-size:10.0pt;font-family:"Arial","sans-serif";color:black;mso-fareast-language:EN-GB'>Mob: +32 497 909353<o:p></o:p></span></p><p class=MsoNormal><span lang=NL style='font-size:10.0pt;font-family:"Arial","sans-serif";color:black;mso-fareast-language:EN-GB'><a href="mailto:koen_segers@computacenter.com" target="_blank"><span lang=EN-GB style='color:#24387F'>koen_segers@computacenter.com</span></a></span><span style='font-size:10.0pt;font-family:"Arial","sans-serif";color:black;mso-fareast-language:EN-GB'><o:p></o:p></span></p><p class=MsoNormal><span lang=NL style='font-size:10.0pt;font-family:"Arial","sans-serif";color:black;mso-fareast-language:EN-GB'><a href="http://www.computacenter.com/benelux" target="_blank"><span style='color:#24387F'>www.computacenter.com/benelux</span></a><o:p></o:p></span></p><p class=MsoNormal><o:p> </o:p></p></div><br>
<br>
<br>
<span style="font-family:'Arial',sans-serif; font-size:8pt; color:#000000">Visit us at http://www.computacenter.com/<br>
Computacenter: Transforming IT service delivery.<br>
<br>
========================== Disclaimer ==================================<br>
The information in this email is confidential, and is intended solely for the addressee(s). If you are not the intended recipient of this email please let us know by reply and then delete it from your system; you should not copy this message or disclose its contents to anyone. Due to the integrity risk of sending emails over the Internet, Computacenter will accept no liability for any comments and / or attachments contained within this email.<br>
========================== Disclaimer ==================================</span><br>
</body></html>