<html xmlns:v="urn:schemas-microsoft-com:vml" xmlns:o="urn:schemas-microsoft-com:office:office" xmlns:w="urn:schemas-microsoft-com:office:word" xmlns:m="http://schemas.microsoft.com/office/2004/12/omml" xmlns="http://www.w3.org/TR/REC-html40"><head><meta http-equiv=Content-Type content="text/html; charset=us-ascii"><meta name=Generator content="Microsoft Word 14 (filtered medium)"><style><!--
/* Font Definitions */
@font-face
{font-family:Wingdings;
panose-1:5 0 0 0 0 0 0 0 0 0;}
@font-face
{font-family:"Cambria Math";
panose-1:2 4 5 3 5 4 6 3 2 4;}
@font-face
{font-family:Calibri;
panose-1:2 15 5 2 2 2 4 3 2 4;}
@font-face
{font-family:Tahoma;
panose-1:2 11 6 4 3 5 4 4 2 4;}
/* Style Definitions */
p.MsoNormal, li.MsoNormal, div.MsoNormal
{margin:0in;
margin-bottom:.0001pt;
font-size:11.0pt;
font-family:"Calibri","sans-serif";}
a:link, span.MsoHyperlink
{mso-style-priority:99;
color:blue;
text-decoration:underline;}
a:visited, span.MsoHyperlinkFollowed
{mso-style-priority:99;
color:purple;
text-decoration:underline;}
p.MsoAcetate, li.MsoAcetate, div.MsoAcetate
{mso-style-priority:99;
mso-style-link:"Balloon Text Char";
margin:0in;
margin-bottom:.0001pt;
font-size:8.0pt;
font-family:"Tahoma","sans-serif";}
p.MsoListParagraph, li.MsoListParagraph, div.MsoListParagraph
{mso-style-priority:34;
margin-top:0in;
margin-right:0in;
margin-bottom:0in;
margin-left:.5in;
margin-bottom:.0001pt;
font-size:11.0pt;
font-family:"Calibri","sans-serif";}
span.BalloonTextChar
{mso-style-name:"Balloon Text Char";
mso-style-priority:99;
mso-style-link:"Balloon Text";
font-family:"Tahoma","sans-serif";}
span.EmailStyle20
{mso-style-type:personal;
font-family:"Calibri","sans-serif";
color:windowtext;}
span.EmailStyle21
{mso-style-type:personal;
font-family:"Times New Roman","serif";
color:windowtext;}
span.EmailStyle22
{mso-style-type:personal;
font-family:"Calibri","sans-serif";
color:windowtext;}
span.EmailStyle23
{mso-style-type:personal-reply;
font-family:"Times New Roman","serif";
color:windowtext;}
.MsoChpDefault
{mso-style-type:export-only;
font-size:10.0pt;}
@page WordSection1
{size:8.5in 11.0in;
margin:1.0in 1.0in 1.0in 1.0in;}
div.WordSection1
{page:WordSection1;}
/* List Definitions */
@list l0
{mso-list-id:218445561;
mso-list-type:hybrid;
mso-list-template-ids:-2134221482 -331042242 134807555 134807557 134807553 134807555 134807557 134807553 134807555 134807557;}
@list l0:level1
{mso-level-start-at:0;
mso-level-number-format:bullet;
mso-level-text:\F0B7;
mso-level-tab-stop:none;
mso-level-number-position:left;
text-indent:-.25in;
font-family:Symbol;
mso-fareast-font-family:"Times New Roman";
mso-bidi-font-family:Arial;}
@list l0:level2
{mso-level-number-format:bullet;
mso-level-text:o;
mso-level-tab-stop:none;
mso-level-number-position:left;
text-indent:-.25in;
font-family:"Courier New";}
@list l0:level3
{mso-level-number-format:bullet;
mso-level-text:\F0A7;
mso-level-tab-stop:none;
mso-level-number-position:left;
text-indent:-.25in;
font-family:Wingdings;}
@list l0:level4
{mso-level-number-format:bullet;
mso-level-text:\F0B7;
mso-level-tab-stop:none;
mso-level-number-position:left;
text-indent:-.25in;
font-family:Symbol;}
@list l0:level5
{mso-level-number-format:bullet;
mso-level-text:o;
mso-level-tab-stop:none;
mso-level-number-position:left;
text-indent:-.25in;
font-family:"Courier New";}
@list l0:level6
{mso-level-number-format:bullet;
mso-level-text:\F0A7;
mso-level-tab-stop:none;
mso-level-number-position:left;
text-indent:-.25in;
font-family:Wingdings;}
@list l0:level7
{mso-level-number-format:bullet;
mso-level-text:\F0B7;
mso-level-tab-stop:none;
mso-level-number-position:left;
text-indent:-.25in;
font-family:Symbol;}
@list l0:level8
{mso-level-number-format:bullet;
mso-level-text:o;
mso-level-tab-stop:none;
mso-level-number-position:left;
text-indent:-.25in;
font-family:"Courier New";}
@list l0:level9
{mso-level-number-format:bullet;
mso-level-text:\F0A7;
mso-level-tab-stop:none;
mso-level-number-position:left;
text-indent:-.25in;
font-family:Wingdings;}
ol
{margin-bottom:0in;}
ul
{margin-bottom:0in;}
--></style><!--[if gte mso 9]><xml>
<o:shapedefaults v:ext="edit" spidmax="1026" />
</xml><![endif]--><!--[if gte mso 9]><xml>
<o:shapelayout v:ext="edit">
<o:idmap v:ext="edit" data="1" />
</o:shapelayout></xml><![endif]--></head><body lang=EN-US link=blue vlink=purple><div class=WordSection1><p class=MsoNormal><span style='font-size:12.0pt;font-family:"Times New Roman","serif"'>Koen,<o:p></o:p></span></p><p class=MsoNormal><span style='font-size:12.0pt;font-family:"Times New Roman","serif"'><o:p> </o:p></span></p><p class=MsoNormal><span style='font-size:12.0pt;font-family:"Times New Roman","serif"'>Can you try running ibdiagnet –P all=1 –ls 10 –lw 4x<o:p></o:p></span></p><p class=MsoNormal><span style='font-size:12.0pt;font-family:"Times New Roman","serif"'><o:p> </o:p></span></p><p class=MsoNormal><span style='font-size:12.0pt;font-family:"Times New Roman","serif"'>This will tell us if any links are not running at Link speed of 10 (QDR) and Link Width of 4x.<o:p></o:p></span></p><p class=MsoNormal><span style='font-size:12.0pt;font-family:"Times New Roman","serif"'><o:p> </o:p></span></p><p class=MsoNormal><span style='font-size:12.0pt;font-family:"Times New Roman","serif"'>You may also want to suggest an upgrade of OFED to 1.5.3.2 GA. There have been major improvements in the stack since 1.4.2. Also please be sure that you update the firmware in all hardware for the same reason.<o:p></o:p></span></p><p class=MsoNormal><span style='font-size:12.0pt;font-family:"Times New Roman","serif"'><o:p> </o:p></span></p><p class=MsoNormal><span style='font-size:12.0pt;font-family:"Times New Roman","serif"'>Thanks<o:p></o:p></span></p><p class=MsoNormal><span style='font-size:12.0pt;font-family:"Times New Roman","serif"'><o:p> </o:p></span></p><p class=MsoNormal><span style='font-size:12.0pt;font-family:"Times New Roman","serif"'>Rupert<o:p></o:p></span></p><p class=MsoNormal><span style='font-size:12.0pt;font-family:"Times New Roman","serif"'><o:p> </o:p></span></p><div><div style='border:none;border-top:solid #B5C4DF 1.0pt;padding:3.0pt 0in 0in 0in'><p class=MsoNormal><b><span style='font-size:10.0pt;font-family:"Tahoma","sans-serif"'>From:</span></b><span style='font-size:10.0pt;font-family:"Tahoma","sans-serif"'> koen_segers@computacenter.com [mailto:koen_segers@computacenter.com] <br><b>Sent:</b> Monday, October 17, 2011 9:22 AM<br><b>To:</b> rsdance@soft-forge.com; ewg@lists.openfabrics.org<br><b>Cc:</b> koen_segers@computacenter.com<br><b>Subject:</b> RE: [ewg] 200m cable results in slower rdma read performance? [ CC Anti-Virus checked ]<o:p></o:p></span></p></div></div><p class=MsoNormal><o:p> </o:p></p><p class=MsoNormal><span lang=EN-GB>Rupert,<o:p></o:p></span></p><p class=MsoNormal><span lang=EN-GB><o:p> </o:p></span></p><p class=MsoNormal><span lang=EN-GB>Thanks for replying. <o:p></o:p></span></p><p class=MsoNormal><span lang=EN-GB><o:p> </o:p></span></p><p class=MsoNormal><span lang=EN-GB>Below is the output of the ibdiagnet command.<o:p></o:p></span></p><p class=MsoNormal><span lang=EN-GB>I don’t see any issues here. Just tell me if you need more info.<o:p></o:p></span></p><p class=MsoNormal><span lang=EN-GB><o:p> </o:p></span></p><p class=MsoNormal><span lang=EN-GB>I forgot to mention that we are using the following switch version:<o:p></o:p></span></p><p class=MsoNormal><span lang=EN-GB>edgeprod1# version show<o:p></o:p></span></p><p class=MsoNormal><span lang=EN-GB> version: 3.6.0<o:p></o:p></span></p><p class=MsoNormal><span lang=EN-GB> date: Jun 07 2011 11:19:33 AM<o:p></o:p></span></p><p class=MsoNormal><span lang=EN-GB> build Id:857<o:p></o:p></span></p><p class=MsoNormal><span lang=EN-GB><o:p> </o:p></span></p><p class=MsoNormal><span lang=EN-GB>And the default SLES 11 SP1 ofed build: ofed-1.4.2-0.9.6<o:p></o:p></span></p><p class=MsoNormal><span lang=EN-GB><o:p> </o:p></span></p><p class=MsoNormal><span lang=EN-GB>Best regards,<o:p></o:p></span></p><p class=MsoNormal><span lang=EN-GB><o:p> </o:p></span></p><p class=MsoNormal><span lang=EN-GB>15:00:28|root@gpfsprod1n1:~ 0 # ibdiagnet<o:p></o:p></span></p><p class=MsoNormal><span lang=EN-GB>Loading IBDIAGNET from: /usr/lib64/ibdiagnet1.2<o:p></o:p></span></p><p class=MsoNormal><span lang=EN-GB>-W- Topology file is not specified.<o:p></o:p></span></p><p class=MsoNormal><span lang=EN-GB> Reports regarding cluster links will use direct routes.<o:p></o:p></span></p><p class=MsoNormal><span lang=EN-GB>Loading IBDM from: /usr/lib64/ibdm1.2<o:p></o:p></span></p><p class=MsoNormal><span lang=EN-GB>-W- A few ports of local device are up.<o:p></o:p></span></p><p class=MsoNormal><span lang=EN-GB> Since port-num was not specified (-p option), port 1 of device 1 will be<o:p></o:p></span></p><p class=MsoNormal><span lang=EN-GB> used as the local port.<o:p></o:p></span></p><p class=MsoNormal><span lang=EN-GB>-I- Discovering ... 39 nodes (6 Switches & 33 CA-s) discovered.<o:p></o:p></span></p><p class=MsoNormal><span lang=EN-GB><o:p> </o:p></span></p><p class=MsoNormal><span lang=EN-GB><o:p> </o:p></span></p><p class=MsoNormal><span lang=EN-GB>-I---------------------------------------------------<o:p></o:p></span></p><p class=MsoNormal><span lang=EN-GB>-I- Bad Guids/LIDs Info<o:p></o:p></span></p><p class=MsoNormal><span lang=EN-GB>-I---------------------------------------------------<o:p></o:p></span></p><p class=MsoNormal><span lang=EN-GB>-I- No bad Guids were found<o:p></o:p></span></p><p class=MsoNormal><span lang=EN-GB><o:p> </o:p></span></p><p class=MsoNormal><span lang=EN-GB>-I---------------------------------------------------<o:p></o:p></span></p><p class=MsoNormal><span lang=EN-GB>-I- Links With Logical State = INIT<o:p></o:p></span></p><p class=MsoNormal><span lang=EN-GB>-I---------------------------------------------------<o:p></o:p></span></p><p class=MsoNormal><span lang=EN-GB>-I- No bad Links (with logical state = INIT) were found<o:p></o:p></span></p><p class=MsoNormal><span lang=EN-GB><o:p> </o:p></span></p><p class=MsoNormal><span lang=EN-GB>-I---------------------------------------------------<o:p></o:p></span></p><p class=MsoNormal><span lang=EN-GB>-I- PM Counters Info<o:p></o:p></span></p><p class=MsoNormal><span lang=EN-GB>-I---------------------------------------------------<o:p></o:p></span></p><p class=MsoNormal><span lang=EN-GB>-I- No illegal PM counters values were found<o:p></o:p></span></p><p class=MsoNormal><span lang=EN-GB><o:p> </o:p></span></p><p class=MsoNormal><span lang=EN-GB>-I---------------------------------------------------<o:p></o:p></span></p><p class=MsoNormal><span lang=EN-GB>-I- Fabric Partitions Report (see ibdiagnet.pkey for a full hosts list)<o:p></o:p></span></p><p class=MsoNormal><span lang=EN-GB>-I---------------------------------------------------<o:p></o:p></span></p><p class=MsoNormal><span lang=EN-GB>-I- PKey:0x7fff Hosts:65 full:65 partial:0<o:p></o:p></span></p><p class=MsoNormal><span lang=EN-GB><o:p> </o:p></span></p><p class=MsoNormal><span lang=EN-GB>-I---------------------------------------------------<o:p></o:p></span></p><p class=MsoNormal><span lang=EN-GB>-I- IPoIB Subnets Check<o:p></o:p></span></p><p class=MsoNormal><span lang=EN-GB>-I---------------------------------------------------<o:p></o:p></span></p><p class=MsoNormal><span lang=EN-GB>-I- Subnet: IPv4 PKey:0x7fff QKey:0x00000b1b MTU:2048Byte rate:10Gbps SL:0x00<o:p></o:p></span></p><p class=MsoNormal><span lang=EN-GB>-W- Suboptimal rate for group. Lowest member rate:40Gbps > group-rate:10Gbps<o:p></o:p></span></p><p class=MsoNormal><span lang=EN-GB><o:p> </o:p></span></p><p class=MsoNormal><span lang=EN-GB>-I---------------------------------------------------<o:p></o:p></span></p><p class=MsoNormal><span lang=EN-GB>-I- Bad Links Info<o:p></o:p></span></p><p class=MsoNormal><span lang=EN-GB>-I- No bad link were found<o:p></o:p></span></p><p class=MsoNormal><span lang=EN-GB>-I---------------------------------------------------<o:p></o:p></span></p><p class=MsoNormal><span lang=EN-GB>----------------------------------------------------------------<o:p></o:p></span></p><p class=MsoNormal><span lang=EN-GB>-I- Stages Status Report:<o:p></o:p></span></p><p class=MsoNormal><span lang=EN-GB> STAGE Errors Warnings<o:p></o:p></span></p><p class=MsoNormal><span lang=EN-GB> Bad GUIDs/LIDs Check 0 0<o:p></o:p></span></p><p class=MsoNormal><span lang=EN-GB> Link State Active Check 0 0<o:p></o:p></span></p><p class=MsoNormal><span lang=EN-GB> Performance Counters Report 0 0<o:p></o:p></span></p><p class=MsoNormal><span lang=EN-GB> Partitions Check 0 0<o:p></o:p></span></p><p class=MsoNormal><span lang=EN-GB> IPoIB Subnets Check 0 1<o:p></o:p></span></p><p class=MsoNormal><span lang=EN-GB><o:p> </o:p></span></p><p class=MsoNormal><span lang=EN-GB>Please see /tmp/ibdiagnet.log for complete log<o:p></o:p></span></p><p class=MsoNormal><span lang=EN-GB>----------------------------------------------------------------<o:p></o:p></span></p><p class=MsoNormal><span lang=EN-GB><o:p> </o:p></span></p><p class=MsoNormal><span lang=EN-GB>-I- Done. Run time was 5 seconds.<o:p></o:p></span></p><p class=MsoNormal><span lang=EN-GB><o:p> </o:p></span></p><div><p class=MsoNormal><span lang=EN-GB style='font-size:10.0pt;font-family:"Arial","sans-serif";color:black'><o:p> </o:p></span></p><p class=MsoNormal><span lang=EN-GB style='font-size:10.0pt;font-family:"Arial","sans-serif";color:black'>This type of info is given in ibdiagnet.lst:<o:p></o:p></span></p><p class=MsoNormal><span lang=EN-GB style='font-size:10.0pt;font-family:"Arial","sans-serif";color:black'><o:p> </o:p></span></p><p class=MsoNormal><span lang=EN-GB style='font-size:10.0pt;font-family:"Arial","sans-serif";color:black'>{ SW Ports:24 SystemGUID:0008f10500108fa9 NodeGUID:0008f10500108fa8 PortGUID:0008f10500108fa8 VenID:000008F1 DevID:5A5A0000 Rev:000000A1 {Voltaire<o:p></o:p></span></p><p class=MsoNormal><span lang=EN-GB style='font-size:10.0pt;font-family:"Arial","sans-serif";color:black'>4036 # edgeprod3} LID:0001 PN:05 } { CA Ports:02 SystemGUID:0002c903004ab175 NodeGUID:0002c903004ab172 PortGUID:0002c903004ab173 VenID:000002C9 D<o:p></o:p></span></p><p class=MsoNormal><span lang=EN-GB style='font-size:10.0pt;font-family:"Arial","sans-serif";color:black'>evID:673C0000 Rev:000000B0 { HCA-1} LID:001A PN:01 } PHY=4x LOG=ACT SPD=10<o:p></o:p></span></p><p class=MsoNormal><span lang=EN-GB style='font-size:10.0pt;font-family:"Arial","sans-serif";color:black'><o:p> </o:p></span></p><p class=MsoNormal><span lang=EN-GB style='font-size:10.0pt;font-family:"Arial","sans-serif";color:black'><o:p> </o:p></span></p><p class=MsoNormal><span lang=EN-GB style='font-size:10.0pt;font-family:"Arial","sans-serif";color:black'><o:p> </o:p></span></p><p class=MsoNormal><span lang=EN-GB style='font-size:10.0pt;font-family:"Arial","sans-serif";color:black'><o:p> </o:p></span></p><p class=MsoNormal><b><span lang=EN-GB style='font-size:10.0pt;font-family:"Arial","sans-serif";color:black'>Koen Segers<o:p></o:p></span></b></p><p class=MsoNormal><b><span lang=EN-GB style='font-size:10.0pt;font-family:"Arial","sans-serif";color:black'>Enterprise Consultant<o:p></o:p></span></b></p><p class=MsoNormal><span lang=EN-GB style='font-size:10.0pt;font-family:"Arial","sans-serif";color:black'> <o:p></o:p></span></p><p class=MsoNormal><b><span lang=EN-GB style='font-size:10.0pt;font-family:"Arial","sans-serif";color:#24387F'>Computacenter<o:p></o:p></span></b></p><p class=MsoNormal><span lang=EN-GB style='font-size:10.0pt;font-family:"Arial","sans-serif";color:#009DDC'>Services & Solutions<o:p></o:p></span></p><p class=MsoNormal><span lang=EN-GB style='font-size:10.0pt;font-family:"Arial","sans-serif";color:black'> <o:p></o:p></span></p><p class=MsoNormal><span lang=NL style='font-size:10.0pt;font-family:"Arial","sans-serif";color:black'>Ikaroslaan 31<o:p></o:p></span></p><p class=MsoNormal><span lang=NL style='font-size:10.0pt;font-family:"Arial","sans-serif";color:black'>B-1930 Zaventem<o:p></o:p></span></p><p class=MsoNormal><span lang=NL style='font-size:10.0pt;font-family:"Arial","sans-serif";color:black'>Belgium<o:p></o:p></span></p><p class=MsoNormal><span lang=NL style='font-size:10.0pt;font-family:"Arial","sans-serif";color:black'> <o:p></o:p></span></p><p class=MsoNormal><span lang=NL style='font-size:10.0pt;font-family:"Arial","sans-serif";color:black'>Tel: +32 2 704 94 67<o:p></o:p></span></p><p class=MsoNormal><span lang=EN-GB style='font-size:10.0pt;font-family:"Arial","sans-serif";color:black'>Fax: +32 2 704 95 95<o:p></o:p></span></p><p class=MsoNormal><span lang=EN-GB style='font-size:10.0pt;font-family:"Arial","sans-serif";color:black'>Mob: +32 497 909353<o:p></o:p></span></p><p class=MsoNormal><span lang=NL style='font-size:10.0pt;font-family:"Arial","sans-serif";color:black'><a href="mailto:koen_segers@computacenter.com" target="_blank"><span lang=EN-GB style='color:#24387F'>koen_segers@computacenter.com</span></a></span><span lang=EN-GB style='font-size:10.0pt;font-family:"Arial","sans-serif";color:black'><o:p></o:p></span></p><p class=MsoNormal><span lang=NL style='font-size:10.0pt;font-family:"Arial","sans-serif";color:black'><a href="http://www.computacenter.com/benelux" target="_blank"><span style='color:#24387F'>www.computacenter.com/benelux</span></a><o:p></o:p></span></p></div><p class=MsoNormal><span lang=EN-GB><o:p> </o:p></span></p><div><div style='border:none;border-top:solid #B5C4DF 1.0pt;padding:3.0pt 0in 0in 0in'><p class=MsoNormal><b><span style='font-size:10.0pt;font-family:"Tahoma","sans-serif"'>From:</span></b><span style='font-size:10.0pt;font-family:"Tahoma","sans-serif"'> Rupert Dance <<a href="mailto:rsdance@soft-forge.com">rsdance@soft-forge.com</a>> [mailto:Rupert Dance <<a href="mailto:rsdance@soft-forge.com">rsdance@soft-forge.com</a>>] <br><b>Sent:</b> 17 October 2011 13:46<br><b>To:</b> <<a href="mailto:koen_segers@computacenter.com">koen_segers@computacenter.com</a>>; <<a href="mailto:ewg@lists.openfabrics.org">ewg@lists.openfabrics.org</a>><br><b>Subject:</b> RE: [ewg] 200m cable results in slower rdma read performance? [ CC Anti-Virus checked ]<o:p></o:p></span></p></div></div><p class=MsoNormal><span lang=EN-GB><o:p> </o:p></span></p><p class=MsoNormal><span style='font-size:12.0pt;font-family:"Times New Roman","serif"'>Hi,<o:p></o:p></span></p><p class=MsoNormal><span style='font-size:12.0pt;font-family:"Times New Roman","serif"'><o:p> </o:p></span></p><p class=MsoNormal><span style='font-size:12.0pt;font-family:"Times New Roman","serif"'>Have you run ibdiagnet to verify that your link width and speed is what you expect on all links?<o:p></o:p></span></p><p class=MsoNormal><span style='font-size:12.0pt;font-family:"Times New Roman","serif"'><o:p> </o:p></span></p><p class=MsoNormal><span style='font-size:12.0pt;font-family:"Times New Roman","serif"'>Thanks<o:p></o:p></span></p><p class=MsoNormal><span style='font-size:12.0pt;font-family:"Times New Roman","serif"'><o:p> </o:p></span></p><p class=MsoNormal><span style='font-size:12.0pt;font-family:"Times New Roman","serif"'>Rupert Dance<o:p></o:p></span></p><p class=MsoNormal><span style='font-size:12.0pt;font-family:"Times New Roman","serif"'><o:p> </o:p></span></p><p class=MsoNormal><span style='font-size:12.0pt;font-family:"Times New Roman","serif"'>Software Forge<o:p></o:p></span></p><p class=MsoNormal><span style='font-size:12.0pt;font-family:"Times New Roman","serif"'><o:p> </o:p></span></p><div><div style='border:none;border-top:solid #B5C4DF 1.0pt;padding:3.0pt 0in 0in 0in'><p class=MsoNormal><b><span style='font-size:10.0pt;font-family:"Tahoma","sans-serif"'>From:</span></b><span style='font-size:10.0pt;font-family:"Tahoma","sans-serif"'> <a href="mailto:ewg-bounces@lists.openfabrics.org">ewg-bounces@lists.openfabrics.org</a> <a href="mailto:[mailto:ewg-bounces@lists.openfabrics.org]">[mailto:ewg-bounces@lists.openfabrics.org]</a> <b>On Behalf Of </b><a href="mailto:koen_segers@computacenter.com">koen_segers@computacenter.com</a><br><b>Sent:</b> Monday, October 17, 2011 3:22 AM<br><b>To:</b> <a href="mailto:ewg@lists.openfabrics.org">ewg@lists.openfabrics.org</a><br><b>Subject:</b> [ewg] 200m cable results in slower rdma read performance? [ CC Anti-Virus checked ]<o:p></o:p></span></p></div></div><p class=MsoNormal><o:p> </o:p></p><p class=MsoNormal><span lang=EN-GB>Hi, <o:p></o:p></span></p><p class=MsoNormal><span lang=EN-GB><o:p> </o:p></span></p><p class=MsoNormal><span lang=EN-GB>In my test setup I have 3 servers of which 2 are residing in Datacenter1 and the other in Datacenter2.<o:p></o:p></span></p><p class=MsoNormal><span lang=EN-GB>If I do a rdma test between datacenters, I get a much lower performance than if I would do the test between servers residing in the same datacenter.<o:p></o:p></span></p><p class=MsoNormal><span lang=EN-GB><o:p> </o:p></span></p><p class=MsoNormal><span lang=EN-GB><o:p> </o:p></span></p><p class=MsoNormal><span lang=EN-GB>DC1: gpfsprod1n1, gpfsprod1n3<o:p></o:p></span></p><p class=MsoNormal><span lang=EN-GB>DC2: gpfsprod1n2<o:p></o:p></span></p><p class=MsoNormal><span lang=EN-GB><o:p> </o:p></span></p><p class=MsoNormal><span lang=EN-GB>08:54:48|root@gpfsprod1n1:~ 0 # qperf -t 5 cic-gpfsprod1n2 rc_rdma_write_bw<o:p></o:p></span></p><p class=MsoNormal><span lang=EN-GB>rc_rdma_write_bw:<o:p></o:p></span></p><p class=MsoNormal><span lang=EN-GB> bw = 1.9 GB/sec<o:p></o:p></span></p><p class=MsoNormal><span lang=EN-GB>08:54:59|root@gpfsprod1n1:~ 0 # qperf -t 5 cic-gpfsprod1n3 rc_rdma_write_bw<o:p></o:p></span></p><p class=MsoNormal><span lang=EN-GB>rc_rdma_write_bw:<o:p></o:p></span></p><p class=MsoNormal><span lang=EN-GB> bw = 3.39 GB/sec<o:p></o:p></span></p><p class=MsoNormal><span lang=EN-GB><o:p> </o:p></span></p><p class=MsoNormal><span lang=EN-GB style='font-size:10.0pt;font-family:"Arial","sans-serif";color:black'><o:p> </o:p></span></p><p class=MsoNormal><span lang=EN-GB style='font-size:10.0pt;font-family:"Arial","sans-serif";color:black'>The setup contains two pairs of edge switches (on each datacenter one pair) and two spine switches (each datacenter one), configured as a non blocking fat tree.<o:p></o:p></span></p><p class=MsoNormal><span lang=EN-GB style='font-size:10.0pt;font-family:"Arial","sans-serif";color:black'>So:<o:p></o:p></span></p><p class=MsoNormal><span lang=EN-GB style='font-size:10.0pt;font-family:"Arial","sans-serif";color:black'>The servers are connected to the edge switches.<o:p></o:p></span></p><p class=MsoNormal><span lang=EN-GB style='font-size:10.0pt;font-family:"Arial","sans-serif";color:black'>The spine switches are connected to all edge switches.<o:p></o:p></span></p><p class=MsoNormal><span lang=EN-GB style='font-size:10.0pt;font-family:"Arial","sans-serif";color:black'><o:p> </o:p></span></p><p class=MsoNormal><span lang=EN-GB style='font-size:10.0pt;font-family:"Arial","sans-serif";color:black'>These are the cables we are using:<o:p></o:p></span></p><p class=MsoListParagraph style='text-indent:-.25in;mso-list:l0 level1 lfo2'><![if !supportLists]><span lang=EN-GB style='font-size:10.0pt;font-family:Symbol;color:black'><span style='mso-list:Ignore'>·<span style='font:7.0pt "Times New Roman"'> </span></span></span><![endif]><span lang=EN-GB style='font-size:10.0pt;font-family:"Arial","sans-serif";color:black'>Length 5m Vendor Name: WLGORE Code: QSFP+ Vendor PN: 498385-B24 Vendor Rev: D Vendor SN xxxx<o:p></o:p></span></p><p class=MsoListParagraph style='text-indent:-.25in;mso-list:l0 level1 lfo2'><![if !supportLists]><span lang=EN-GB style='font-size:10.0pt;font-family:Symbol;color:black'><span style='mso-list:Ignore'>·<span style='font:7.0pt "Times New Roman"'> </span></span></span><![endif]><span lang=EN-GB style='font-size:10.0pt;font-family:"Arial","sans-serif";color:black'>Length 200m Vendor Name: MOLEX Code: QSFP+ Vendor PN: 106410-1200 Vendor Rev: A Vendor SN xxxx<o:p></o:p></span></p><p class=MsoNormal><span lang=EN-GB style='font-size:10.0pt;font-family:"Arial","sans-serif";color:black'><o:p> </o:p></span></p><p class=MsoNormal><span lang=EN-GB style='font-size:10.0pt;font-family:"Arial","sans-serif";color:black'><o:p> </o:p></span></p><p class=MsoNormal><span lang=EN-GB style='font-size:10.0pt;font-family:"Arial","sans-serif";color:black'>Can someone tell me why this is so? And maybe how I can solve this?<o:p></o:p></span></p><p class=MsoNormal><span lang=EN-GB style='font-size:10.0pt;font-family:"Arial","sans-serif";color:black'><o:p> </o:p></span></p><p class=MsoNormal><span lang=EN-GB style='font-size:10.0pt;font-family:"Arial","sans-serif";color:black'>Best regards,<o:p></o:p></span></p><p class=MsoNormal><span lang=EN-GB style='font-size:10.0pt;font-family:"Arial","sans-serif";color:black'><o:p> </o:p></span></p><p class=MsoNormal><span lang=EN-GB style='font-size:10.0pt;font-family:"Arial","sans-serif";color:black'><o:p> </o:p></span></p><p class=MsoNormal><b><span lang=EN-GB style='font-size:10.0pt;font-family:"Arial","sans-serif";color:black'>Koen Segers<o:p></o:p></span></b></p><p class=MsoNormal><b><span lang=EN-GB style='font-size:10.0pt;font-family:"Arial","sans-serif";color:black'>Enterprise Consultant<o:p></o:p></span></b></p><p class=MsoNormal><span lang=EN-GB style='font-size:10.0pt;font-family:"Arial","sans-serif";color:black'> <o:p></o:p></span></p><p class=MsoNormal><b><span lang=EN-GB style='font-size:10.0pt;font-family:"Arial","sans-serif";color:#24387F'>Computacenter<o:p></o:p></span></b></p><p class=MsoNormal><span lang=EN-GB style='font-size:10.0pt;font-family:"Arial","sans-serif";color:#009DDC'>Services & Solutions<o:p></o:p></span></p><p class=MsoNormal><span lang=EN-GB style='font-size:10.0pt;font-family:"Arial","sans-serif";color:black'> <o:p></o:p></span></p><p class=MsoNormal><span lang=NL style='font-size:10.0pt;font-family:"Arial","sans-serif";color:black'>Ikaroslaan 31<o:p></o:p></span></p><p class=MsoNormal><span lang=NL style='font-size:10.0pt;font-family:"Arial","sans-serif";color:black'>B-1930 Zaventem<o:p></o:p></span></p><p class=MsoNormal><span lang=NL style='font-size:10.0pt;font-family:"Arial","sans-serif";color:black'>Belgium<o:p></o:p></span></p><p class=MsoNormal><span lang=NL style='font-size:10.0pt;font-family:"Arial","sans-serif";color:black'> <o:p></o:p></span></p><p class=MsoNormal><span lang=NL style='font-size:10.0pt;font-family:"Arial","sans-serif";color:black'>Tel: +32 2 704 94 67<o:p></o:p></span></p><p class=MsoNormal><span lang=EN-GB style='font-size:10.0pt;font-family:"Arial","sans-serif";color:black'>Fax: +32 2 704 95 95<o:p></o:p></span></p><p class=MsoNormal><span lang=EN-GB style='font-size:10.0pt;font-family:"Arial","sans-serif";color:black'>Mob: +32 497 909353<o:p></o:p></span></p><p class=MsoNormal><span lang=NL style='font-size:10.0pt;font-family:"Arial","sans-serif";color:black'><a href="mailto:koen_segers@computacenter.com" target="_blank"><span lang=EN-GB style='color:#24387F'>koen_segers@computacenter.com</span></a></span><span lang=EN-GB style='font-size:10.0pt;font-family:"Arial","sans-serif";color:black'><o:p></o:p></span></p><p class=MsoNormal><span lang=NL style='font-size:10.0pt;font-family:"Arial","sans-serif";color:black'><a href="http://www.computacenter.com/benelux" target="_blank"><span style='color:#24387F'>www.computacenter.com/benelux</span></a><o:p></o:p></span></p><p class=MsoNormal><span lang=EN-GB><o:p> </o:p></span></p><p class=MsoNormal><span lang=EN-GB style='font-size:12.0pt;font-family:"Times New Roman","serif"'><br><br><br></span><span lang=EN-GB style='font-size:8.0pt;font-family:"Arial","sans-serif";color:black'>Visit us at <a href="http://www.computacenter.com/">http://www.computacenter.com/</a><br>Computacenter: Transforming IT service delivery.<br><br>========================== Disclaimer ==================================<br>The information in this email is confidential, and is intended solely for the addressee(s). If you are not the intended recipient of this email please let us know by reply and then delete it from your system; you should not copy this message or disclose its contents to anyone. Due to the integrity risk of sending emails over the Internet, Computacenter will accept no liability for any comments and / or attachments contained within this email.<br>========================== Disclaimer ==================================</span><span lang=EN-GB style='font-size:12.0pt;font-family:"Times New Roman","serif"'><o:p></o:p></span></p><p class=MsoNormal><span lang=EN-GB style='font-size:12.0pt;font-family:"Times New Roman","serif"'><br><br><br></span><span lang=EN-GB style='font-size:8.0pt;font-family:"Arial","sans-serif";color:black'>Visit us at <a href="http://www.computacenter.com/">http://www.computacenter.com/</a><br>Computacenter: Transforming IT service delivery.<br><br>========================== Disclaimer ==================================<br>The information in this email is confidential, and is intended solely for the addressee(s). If you are not the intended recipient of this email please let us know by reply and then delete it from your system; you should not copy this message or disclose its contents to anyone. Due to the integrity risk of sending emails over the Internet, Computacenter will accept no liability for any comments and / or attachments contained within this email.<br>========================== Disclaimer ==================================</span><span lang=EN-GB style='font-size:12.0pt;font-family:"Times New Roman","serif"'><o:p></o:p></span></p></div></body></html>