<html xmlns:o="urn:schemas-microsoft-com:office:office" xmlns:w="urn:schemas-microsoft-com:office:word" xmlns:m="http://schemas.microsoft.com/office/2004/12/omml" xmlns="http://www.w3.org/TR/REC-html40"><head><meta http-equiv=Content-Type content="text/html; charset=utf-8"><meta name=Generator content="Microsoft Word 15 (filtered medium)"><style><!--
/* Font Definitions */
@font-face
{font-family:Wingdings;
panose-1:5 0 0 0 0 0 0 0 0 0;}
@font-face
{font-family:"Cambria Math";
panose-1:2 4 5 3 5 4 6 3 2 4;}
@font-face
{font-family:Calibri;
panose-1:2 15 5 2 2 2 4 3 2 4;}
/* Style Definitions */
p.MsoNormal, li.MsoNormal, div.MsoNormal
{margin:0in;
margin-bottom:.0001pt;
font-size:11.0pt;
font-family:"Calibri",sans-serif;}
a:link, span.MsoHyperlink
{mso-style-priority:99;
color:blue;
text-decoration:underline;}
a:visited, span.MsoHyperlinkFollowed
{mso-style-priority:99;
color:#954F72;
text-decoration:underline;}
p.MsoListParagraph, li.MsoListParagraph, div.MsoListParagraph
{mso-style-priority:34;
margin-top:0in;
margin-right:0in;
margin-bottom:0in;
margin-left:.5in;
margin-bottom:.0001pt;
font-size:11.0pt;
font-family:"Calibri",sans-serif;}
.MsoChpDefault
{mso-style-type:export-only;}
@page WordSection1
{size:8.5in 11.0in;
margin:1.0in 1.0in 1.0in 1.0in;}
div.WordSection1
{page:WordSection1;}
/* List Definitions */
@list l0
{mso-list-id:1849902373;
mso-list-type:hybrid;
mso-list-template-ids:-1325250114 -1 67567619 67567621 67567617 67567619 67567621 67567617 67567619 67567621;}
@list l0:level1
{mso-level-start-at:0;
mso-level-number-format:bullet;
mso-level-text:-;
mso-level-tab-stop:none;
mso-level-number-position:left;
margin-left:53.4pt;
text-indent:-.25in;
font-family:"Calibri",sans-serif;
mso-fareast-font-family:"Times New Roman";}
@list l0:level2
{mso-level-number-format:bullet;
mso-level-text:o;
mso-level-tab-stop:none;
mso-level-number-position:left;
margin-left:89.4pt;
text-indent:-.25in;
font-family:"Courier New";}
@list l0:level3
{mso-level-number-format:bullet;
mso-level-text:\F0A7;
mso-level-tab-stop:none;
mso-level-number-position:left;
margin-left:125.4pt;
text-indent:-.25in;
font-family:Wingdings;}
@list l0:level4
{mso-level-number-format:bullet;
mso-level-text:\F0B7;
mso-level-tab-stop:none;
mso-level-number-position:left;
margin-left:161.4pt;
text-indent:-.25in;
font-family:Symbol;}
@list l0:level5
{mso-level-number-format:bullet;
mso-level-text:o;
mso-level-tab-stop:none;
mso-level-number-position:left;
margin-left:197.4pt;
text-indent:-.25in;
font-family:"Courier New";}
@list l0:level6
{mso-level-number-format:bullet;
mso-level-text:\F0A7;
mso-level-tab-stop:none;
mso-level-number-position:left;
margin-left:233.4pt;
text-indent:-.25in;
font-family:Wingdings;}
@list l0:level7
{mso-level-number-format:bullet;
mso-level-text:\F0B7;
mso-level-tab-stop:none;
mso-level-number-position:left;
margin-left:269.4pt;
text-indent:-.25in;
font-family:Symbol;}
@list l0:level8
{mso-level-number-format:bullet;
mso-level-text:o;
mso-level-tab-stop:none;
mso-level-number-position:left;
margin-left:305.4pt;
text-indent:-.25in;
font-family:"Courier New";}
@list l0:level9
{mso-level-number-format:bullet;
mso-level-text:\F0A7;
mso-level-tab-stop:none;
mso-level-number-position:left;
margin-left:341.4pt;
text-indent:-.25in;
font-family:Wingdings;}
ol
{margin-bottom:0in;}
ul
{margin-bottom:0in;}
--></style></head><body lang=EN-US link=blue vlink="#954F72"><div class=WordSection1><p class=MsoNormal><span lang=EN-GB>Hi everyone,<o:p></o:p></span></p><p class=MsoNormal><span lang=EN-GB><o:p> </o:p></span></p><p class=MsoNormal><span lang=EN-GB>as some may remember I’m working on an any-to-any Benchmark for my Bachelor Thesis.<o:p></o:p></span></p><p class=MsoNormal><span lang=EN-GB>I’m now at a stage where I’m getting my first proper results and I’m a Little intrigued, yet I don’t quite 100% understand the results I am getting. <o:p></o:p></span></p><p class=MsoNormal><span lang=EN-GB><o:p> </o:p></span></p><p class=MsoNormal><span lang=EN-GB>A Little background Information:<o:p></o:p></span></p><p class=MsoNormal><span lang=EN-GB><o:p> </o:p></span></p><p class=MsoNormal><span lang=EN-GB>Hardware:<o:p></o:p></span></p><p class=MsoNormal><span lang=EN-GB> The network I‘m testing on utilizes four 36 port switches. With one central switch connecting the 3 others via 12 physical links each(see right side of ´network topology.jpg´) utilizing 4xQDR to PCIe 2x8 (~32Gb/s theoretical effective throughput per Port). Two of those switches branch out to 16-20 nodes each(pn01-40) which are the nodes I’m working with. <o:p></o:p></span></p><p class=MsoNormal><span lang=EN-GB><o:p> </o:p></span></p><p class=MsoNormal><span lang=EN-GB>Software:<o:p></o:p></span></p><p class=MsoNormal><span lang=EN-GB> The Benchmark I wrote utilizes one central node to synchronize Stages of all the other nodes(see ´Benchmarking_process_model.png´) divided into passive Server - active Client groups. between Clients and Servers NxM connected endpoints are established. <o:p></o:p></span></p><p class=MsoNormal><span lang=EN-GB><o:p> </o:p></span></p><p class=MsoNormal><span lang=EN-GB>Benchmarking workflow:<o:p></o:p></span></p><ul style='margin-top:0in' type=disc><li class=MsoListParagraph style='margin-left:17.4pt;mso-list:l0 level1 lfo1'><span lang=EN-GB>´connect´: all N clients connect to all M servers establishing connected endpoints. <o:p></o:p></span></li><li class=MsoListParagraph style='margin-left:17.4pt;mso-list:l0 level1 lfo1'><span lang=EN-GB>´start´ signal all clients queue a defined number of fi_write() per endpoint and once a completion arrives the respective endpoint is looked up and a new fi_write is enqueued for that endpoint.<o:p></o:p></span></li><li class=MsoListParagraph style='margin-left:17.4pt;mso-list:l0 level1 lfo1'><span lang=EN-GB>´checkpoint´ every ´dt´ seconds: Clients send a current snapshot of their completion count for each endpoint to the controller<o:p></o:p></span></li><li class=MsoListParagraph style='margin-left:17.4pt;mso-list:l0 level1 lfo1'><span lang=EN-GB>´stop´ after ´t´ seconds: Clients and Server disconnect. Clients send the final sum of completions for each endpoint to the Controller<o:p></o:p></span></li></ul><p class=MsoNormal><span lang=EN-GB><o:p> </o:p></span></p><p class=MsoNormal><span lang=EN-GB>From what I can tell my implementation should run completely asynchronous. Endpoints and Nodes should work independently and there should be no waiting for each other implementation wise, the only critical area is the endpoint lookup and requeueing of fi_write() which I made sure is quick enough and able to serve all endpoints in time. So the only synchronization that should happen is during ´connect´, ´start´ and ´stop/checkpoint´ signal.<o:p></o:p></span></p><p class=MsoNormal><span lang=EN-GB><o:p> </o:p></span></p><p class=MsoNormal><span lang=EN-GB>The results of this Setup:<o:p></o:p></span></p><ul style='margin-top:0in' type=disc><li class=MsoListParagraph style='margin-left:17.4pt;mso-list:l0 level1 lfo1'><b><span lang=EN-GB>at all times</span></b><span lang=EN-GB> between <b>each</b> endpoint <b>and</b> node we observe extremely similar completion counts with variations of only 0.01~0.0001%<o:p></o:p></span></li><li class=MsoListParagraph style='margin-left:17.4pt;mso-list:l0 level1 lfo1'><span lang=EN-GB>Good: when routing is optimal each node performs almost as good as if it was a 1x1 Benchmark maintaining vey acceptable throughput.<o:p></o:p></span></li><li class=MsoListParagraph style='margin-left:17.4pt;mso-list:l0 level1 lfo1'><span lang=EN-GB>Bad: if just two Nodes share the same port in the switches routing table <b>all</b> of the nodes and endpoints are slowed down. <o:p></o:p></span></li></ul><p class=MsoNormal><span lang=EN-GB><o:p> </o:p></span></p><p class=MsoNormal><span lang=EN-GB>Example:<o:p></o:p></span></p><p class=MsoNormal><span lang=EN-GB>compare heatmap-healthy vs heatmap defector. The only difference is that I swapped out pn16 in healthy for pn17 in defector. <o:p></o:p></span></p><p class=MsoNormal><span lang=EN-GB>ibdiag and ibcongest -C on 5x5 Setup revealed that pn17 and pn22 seem to share a physical link within the routing tables:<o:p></o:p></span></p><p class=MsoNormal><span lang=EN-GB>Stage1 with pn17: pn17 and pn22 appear to have halved bandwidth<o:p></o:p></span></p><p class=MsoNormal><span lang=EN-GB>Stage3 pn16 replaces pn17 -> full bandwidth for everyone<o:p></o:p></span></p><p class=MsoNormal style='margin-left:35.4pt'><span lang=EN-GB style='font-size:8.0pt;color:#767171;mso-style-textfill-fill-color:#767171;mso-style-textfill-fill-alpha:100.0%'># SRC, SLID, DST, DLID, BW [GB/s], AGG-BW [GB/s]<o:p></o:p></span></p><p class=MsoNormal style='margin-left:35.4pt'><span lang=EN-GB style='font-size:8.0pt;color:#767171;mso-style-textfill-fill-color:#767171;mso-style-textfill-fill-alpha:100.0%'>STAGE:1<o:p></o:p></span></p><p class=MsoNormal style='margin-left:35.4pt'><span lang=EN-GB style='font-size:8.0pt;color:#767171;mso-style-textfill-fill-color:#767171;mso-style-textfill-fill-alpha:100.0%'>pn04/U1/P1, 10, pn17/mlx4_0/P1, 21, 0.40, 0.40<o:p></o:p></span></p><p class=MsoNormal style='margin-left:35.4pt'><span lang=EN-GB style='font-size:8.0pt;color:#767171;mso-style-textfill-fill-color:#767171;mso-style-textfill-fill-alpha:100.0%'>pn04/U1/P1, 10, pn18/mlx4_0/P1, 20, 0.80, 1.20<o:p></o:p></span></p><p class=MsoNormal style='margin-left:35.4pt'><span lang=EN-GB style='font-size:8.0pt;color:#767171;mso-style-textfill-fill-color:#767171;mso-style-textfill-fill-alpha:100.0%'>pn04/U1/P1, 10, pn19/mlx4_0/P1, 25, 0.80, 2.00<o:p></o:p></span></p><p class=MsoNormal style='margin-left:35.4pt'><span lang=EN-GB style='font-size:8.0pt;color:#767171;mso-style-textfill-fill-color:#767171;mso-style-textfill-fill-alpha:100.0%'>pn04/U1/P1, 10, pn21/mlx4_0/P1, 31, 0.80, 2.80<o:p></o:p></span></p><p class=MsoNormal style='margin-left:35.4pt'><span lang=EN-GB style='font-size:8.0pt;color:#767171;mso-style-textfill-fill-color:#767171;mso-style-textfill-fill-alpha:100.0%'>pn04/U1/P1, 10, pn22/mlx4_0/P1, 33, 0.40, 3.20<o:p></o:p></span></p><p class=MsoNormal style='margin-left:35.4pt'><span lang=EN-GB style='font-size:8.0pt;color:#767171;mso-style-textfill-fill-color:#767171;mso-style-textfill-fill-alpha:100.0%'>pn05/mlx4_0/P1, 14, pn17/mlx4_0/P1, 21, 0.40, 3.60<o:p></o:p></span></p><p class=MsoNormal style='margin-left:35.4pt'><span lang=EN-GB style='font-size:8.0pt;color:#767171;mso-style-textfill-fill-color:#767171;mso-style-textfill-fill-alpha:100.0%'>pn05/mlx4_0/P1, 14, pn18/mlx4_0/P1, 20, 0.80, 4.40<o:p></o:p></span></p><p class=MsoNormal style='margin-left:35.4pt'><span lang=EN-GB style='font-size:8.0pt;color:#767171;mso-style-textfill-fill-color:#767171;mso-style-textfill-fill-alpha:100.0%'>pn05/mlx4_0/P1, 14, pn19/mlx4_0/P1, 25, 0.80, 5.20<o:p></o:p></span></p><p class=MsoNormal style='margin-left:35.4pt'><span lang=EN-GB style='font-size:8.0pt;color:#767171;mso-style-textfill-fill-color:#767171;mso-style-textfill-fill-alpha:100.0%'>pn05/mlx4_0/P1, 14, pn21/mlx4_0/P1, 31, 0.80, 6.00<o:p></o:p></span></p><p class=MsoNormal style='margin-left:35.4pt'><span lang=EN-GB style='font-size:8.0pt;color:#767171;mso-style-textfill-fill-color:#767171;mso-style-textfill-fill-alpha:100.0%'>pn05/mlx4_0/P1, 14, pn22/mlx4_0/P1, 33, 0.40, 6.40<o:p></o:p></span></p><p class=MsoNormal style='margin-left:35.4pt'><span lang=EN-GB style='font-size:8.0pt;color:#767171;mso-style-textfill-fill-color:#767171;mso-style-textfill-fill-alpha:100.0%'>pn06/mlx4_0/P1, 15, pn17/mlx4_0/P1, 21, 0.40, 6.80<o:p></o:p></span></p><p class=MsoNormal style='margin-left:35.4pt'><span lang=EN-GB style='font-size:8.0pt;color:#767171;mso-style-textfill-fill-color:#767171;mso-style-textfill-fill-alpha:100.0%'>pn06/mlx4_0/P1, 15, pn18/mlx4_0/P1, 20, 0.80, 7.60<o:p></o:p></span></p><p class=MsoNormal style='margin-left:35.4pt'><span lang=EN-GB style='font-size:8.0pt;color:#767171;mso-style-textfill-fill-color:#767171;mso-style-textfill-fill-alpha:100.0%'>pn06/mlx4_0/P1, 15, pn19/mlx4_0/P1, 25, 0.80, 8.40<o:p></o:p></span></p><p class=MsoNormal style='margin-left:35.4pt'><span lang=EN-GB style='font-size:8.0pt;color:#767171;mso-style-textfill-fill-color:#767171;mso-style-textfill-fill-alpha:100.0%'>pn06/mlx4_0/P1, 15, pn21/mlx4_0/P1, 31, 0.80, 9.20<o:p></o:p></span></p><p class=MsoNormal style='margin-left:35.4pt'><span lang=EN-GB style='font-size:8.0pt;color:#767171;mso-style-textfill-fill-color:#767171;mso-style-textfill-fill-alpha:100.0%'>pn06/mlx4_0/P1, 15, pn22/mlx4_0/P1, 33, 0.40, 9.60<o:p></o:p></span></p><p class=MsoNormal style='margin-left:35.4pt'><span lang=EN-GB style='font-size:8.0pt;color:#767171;mso-style-textfill-fill-color:#767171;mso-style-textfill-fill-alpha:100.0%'>pn07/mlx4_0/P1, 23, pn17/mlx4_0/P1, 21, 0.40, 10.00<o:p></o:p></span></p><p class=MsoNormal style='margin-left:35.4pt'><span lang=EN-GB style='font-size:8.0pt;color:#767171;mso-style-textfill-fill-color:#767171;mso-style-textfill-fill-alpha:100.0%'>pn07/mlx4_0/P1, 23, pn18/mlx4_0/P1, 20, 0.80, 10.80<o:p></o:p></span></p><p class=MsoNormal style='margin-left:35.4pt'><span lang=EN-GB style='font-size:8.0pt;color:#767171;mso-style-textfill-fill-color:#767171;mso-style-textfill-fill-alpha:100.0%'>pn07/mlx4_0/P1, 23, pn19/mlx4_0/P1, 25, 0.80, 11.60<o:p></o:p></span></p><p class=MsoNormal style='margin-left:35.4pt'><span lang=EN-GB style='font-size:8.0pt;color:#767171;mso-style-textfill-fill-color:#767171;mso-style-textfill-fill-alpha:100.0%'>pn07/mlx4_0/P1, 23, pn21/mlx4_0/P1, 31, 0.80, 12.40<o:p></o:p></span></p><p class=MsoNormal style='margin-left:35.4pt'><span lang=EN-GB style='font-size:8.0pt;color:#767171;mso-style-textfill-fill-color:#767171;mso-style-textfill-fill-alpha:100.0%'>pn07/mlx4_0/P1, 23, pn22/mlx4_0/P1, 33, 0.40, 12.80<o:p></o:p></span></p><p class=MsoNormal style='margin-left:35.4pt'><span lang=EN-GB style='font-size:8.0pt;color:#767171;mso-style-textfill-fill-color:#767171;mso-style-textfill-fill-alpha:100.0%'>pn08/mlx4_0/P1, 17, pn17/mlx4_0/P1, 21, 0.40, 13.20<o:p></o:p></span></p><p class=MsoNormal style='margin-left:35.4pt'><span lang=EN-GB style='font-size:8.0pt;color:#767171;mso-style-textfill-fill-color:#767171;mso-style-textfill-fill-alpha:100.0%'>pn08/mlx4_0/P1, 17, pn18/mlx4_0/P1, 20, 0.80, 14.00<o:p></o:p></span></p><p class=MsoNormal style='margin-left:35.4pt'><span lang=EN-GB style='font-size:8.0pt;color:#767171;mso-style-textfill-fill-color:#767171;mso-style-textfill-fill-alpha:100.0%'>pn08/mlx4_0/P1, 17, pn19/mlx4_0/P1, 25, 0.80, 14.80<o:p></o:p></span></p><p class=MsoNormal style='margin-left:35.4pt'><span lang=EN-GB style='font-size:8.0pt;color:#767171;mso-style-textfill-fill-color:#767171;mso-style-textfill-fill-alpha:100.0%'>pn08/mlx4_0/P1, 17, pn21/mlx4_0/P1, 31, 0.80, 15.60<o:p></o:p></span></p><p class=MsoNormal style='margin-left:35.4pt'><span lang=EN-GB style='font-size:8.0pt;color:#767171;mso-style-textfill-fill-color:#767171;mso-style-textfill-fill-alpha:100.0%'>pn08/mlx4_0/P1, 17, pn22/mlx4_0/P1, 33, 0.40, 16.00<o:p></o:p></span></p><p class=MsoNormal style='margin-left:35.4pt'><span lang=EN-GB style='font-size:8.0pt;color:#767171;mso-style-textfill-fill-color:#767171;mso-style-textfill-fill-alpha:100.0%'>STAGE:2<o:p></o:p></span></p><p class=MsoNormal style='margin-left:35.4pt'><span lang=EN-GB style='font-size:8.0pt;color:#767171;mso-style-textfill-fill-color:#767171;mso-style-textfill-fill-alpha:100.0%'>pn04/U1/P1, 10, pn17/mlx4_0/P1, 21, 4.00, 4.00<o:p></o:p></span></p><p class=MsoNormal style='margin-left:35.4pt'><span lang=EN-GB style='font-size:8.0pt;color:#767171;mso-style-textfill-fill-color:#767171;mso-style-textfill-fill-alpha:100.0%'>STAGE:3<o:p></o:p></span></p><p class=MsoNormal style='margin-left:35.4pt'><span lang=EN-GB style='font-size:8.0pt;color:#767171;mso-style-textfill-fill-color:#767171;mso-style-textfill-fill-alpha:100.0%'>pn04/U1/P1, 10, pn16/mlx4_0/P1, 19, 0.80, 0.80<o:p></o:p></span></p><p class=MsoNormal style='margin-left:35.4pt'><span lang=EN-GB style='font-size:8.0pt;color:#767171;mso-style-textfill-fill-color:#767171;mso-style-textfill-fill-alpha:100.0%'>pn04/U1/P1, 10, pn18/mlx4_0/P1, 20, 0.80, 1.60<o:p></o:p></span></p><p class=MsoNormal style='margin-left:35.4pt'><span lang=EN-GB style='font-size:8.0pt;color:#767171;mso-style-textfill-fill-color:#767171;mso-style-textfill-fill-alpha:100.0%'>pn04/U1/P1, 10, pn19/mlx4_0/P1, 25, 0.80, 2.40<o:p></o:p></span></p><p class=MsoNormal style='margin-left:35.4pt'><span lang=EN-GB style='font-size:8.0pt;color:#767171;mso-style-textfill-fill-color:#767171;mso-style-textfill-fill-alpha:100.0%'>pn04/U1/P1, 10, pn21/mlx4_0/P1, 31, 0.80, 3.20<o:p></o:p></span></p><p class=MsoNormal style='margin-left:35.4pt'><span lang=EN-GB style='font-size:8.0pt;color:#767171;mso-style-textfill-fill-color:#767171;mso-style-textfill-fill-alpha:100.0%'>pn04/U1/P1, 10, pn22/mlx4_0/P1, 33, 0.80, 4.00<o:p></o:p></span></p><p class=MsoNormal style='margin-left:35.4pt'><span lang=EN-GB style='font-size:8.0pt;color:#767171;mso-style-textfill-fill-color:#767171;mso-style-textfill-fill-alpha:100.0%'>pn05/mlx4_0/P1, 14, pn16/mlx4_0/P1, 19, 0.80, 4.80<o:p></o:p></span></p><p class=MsoNormal style='margin-left:35.4pt'><span lang=EN-GB style='font-size:8.0pt;color:#767171;mso-style-textfill-fill-color:#767171;mso-style-textfill-fill-alpha:100.0%'>pn05/mlx4_0/P1, 14, pn18/mlx4_0/P1, 20, 0.80, 5.60<o:p></o:p></span></p><p class=MsoNormal style='margin-left:35.4pt'><span lang=EN-GB style='font-size:8.0pt;color:#767171;mso-style-textfill-fill-color:#767171;mso-style-textfill-fill-alpha:100.0%'>pn05/mlx4_0/P1, 14, pn19/mlx4_0/P1, 25, 0.80, 6.40<o:p></o:p></span></p><p class=MsoNormal style='margin-left:35.4pt'><span lang=EN-GB style='font-size:8.0pt;color:#767171;mso-style-textfill-fill-color:#767171;mso-style-textfill-fill-alpha:100.0%'>pn05/mlx4_0/P1, 14, pn21/mlx4_0/P1, 31, 0.80, 7.20<o:p></o:p></span></p><p class=MsoNormal style='margin-left:35.4pt'><span lang=EN-GB style='font-size:8.0pt;color:#767171;mso-style-textfill-fill-color:#767171;mso-style-textfill-fill-alpha:100.0%'>pn05/mlx4_0/P1, 14, pn22/mlx4_0/P1, 33, 0.80, 8.00<o:p></o:p></span></p><p class=MsoNormal style='margin-left:35.4pt'><span lang=EN-GB style='font-size:8.0pt;color:#767171;mso-style-textfill-fill-color:#767171;mso-style-textfill-fill-alpha:100.0%'>pn06/mlx4_0/P1, 15, pn16/mlx4_0/P1, 19, 0.80, 8.80<o:p></o:p></span></p><p class=MsoNormal style='margin-left:35.4pt'><span lang=EN-GB style='font-size:8.0pt;color:#767171;mso-style-textfill-fill-color:#767171;mso-style-textfill-fill-alpha:100.0%'>pn06/mlx4_0/P1, 15, pn18/mlx4_0/P1, 20, 0.80, 9.60<o:p></o:p></span></p><p class=MsoNormal style='margin-left:35.4pt'><span lang=EN-GB style='font-size:8.0pt;color:#767171;mso-style-textfill-fill-color:#767171;mso-style-textfill-fill-alpha:100.0%'>pn06/mlx4_0/P1, 15, pn19/mlx4_0/P1, 25, 0.80, 10.40<o:p></o:p></span></p><p class=MsoNormal style='margin-left:35.4pt'><span lang=EN-GB style='font-size:8.0pt;color:#767171;mso-style-textfill-fill-color:#767171;mso-style-textfill-fill-alpha:100.0%'>pn06/mlx4_0/P1, 15, pn21/mlx4_0/P1, 31, 0.80, 11.20<o:p></o:p></span></p><p class=MsoNormal style='margin-left:35.4pt'><span lang=EN-GB style='font-size:8.0pt;color:#767171;mso-style-textfill-fill-color:#767171;mso-style-textfill-fill-alpha:100.0%'>pn06/mlx4_0/P1, 15, pn22/mlx4_0/P1, 33, 0.80, 12.00<o:p></o:p></span></p><p class=MsoNormal style='margin-left:35.4pt'><span lang=EN-GB style='font-size:8.0pt;color:#767171;mso-style-textfill-fill-color:#767171;mso-style-textfill-fill-alpha:100.0%'>pn07/mlx4_0/P1, 23, pn16/mlx4_0/P1, 19, 0.80, 12.80<o:p></o:p></span></p><p class=MsoNormal style='margin-left:35.4pt'><span lang=EN-GB style='font-size:8.0pt;color:#767171;mso-style-textfill-fill-color:#767171;mso-style-textfill-fill-alpha:100.0%'>pn07/mlx4_0/P1, 23, pn18/mlx4_0/P1, 20, 0.80, 13.60<o:p></o:p></span></p><p class=MsoNormal style='margin-left:35.4pt'><span lang=EN-GB style='font-size:8.0pt;color:#767171;mso-style-textfill-fill-color:#767171;mso-style-textfill-fill-alpha:100.0%'>pn07/mlx4_0/P1, 23, pn19/mlx4_0/P1, 25, 0.80, 14.40<o:p></o:p></span></p><p class=MsoNormal style='margin-left:35.4pt'><span lang=EN-GB style='font-size:8.0pt;color:#767171;mso-style-textfill-fill-color:#767171;mso-style-textfill-fill-alpha:100.0%'>pn07/mlx4_0/P1, 23, pn21/mlx4_0/P1, 31, 0.80, 15.20<o:p></o:p></span></p><p class=MsoNormal style='margin-left:35.4pt'><span lang=EN-GB style='font-size:8.0pt;color:#767171;mso-style-textfill-fill-color:#767171;mso-style-textfill-fill-alpha:100.0%'>pn07/mlx4_0/P1, 23, pn22/mlx4_0/P1, 33, 0.80, 16.00<o:p></o:p></span></p><p class=MsoNormal style='margin-left:35.4pt'><span lang=EN-GB style='font-size:8.0pt;color:#767171;mso-style-textfill-fill-color:#767171;mso-style-textfill-fill-alpha:100.0%'>pn08/mlx4_0/P1, 17, pn16/mlx4_0/P1, 19, 0.80, 16.80<o:p></o:p></span></p><p class=MsoNormal style='margin-left:35.4pt'><span lang=EN-GB style='font-size:8.0pt;color:#767171;mso-style-textfill-fill-color:#767171;mso-style-textfill-fill-alpha:100.0%'>pn08/mlx4_0/P1, 17, pn18/mlx4_0/P1, 20, 0.80, 17.60<o:p></o:p></span></p><p class=MsoNormal style='margin-left:35.4pt'><span lang=EN-GB style='font-size:8.0pt;color:#767171;mso-style-textfill-fill-color:#767171;mso-style-textfill-fill-alpha:100.0%'>pn08/mlx4_0/P1, 17, pn19/mlx4_0/P1, 25, 0.80, 18.40<o:p></o:p></span></p><p class=MsoNormal style='margin-left:35.4pt'><span lang=EN-GB style='font-size:8.0pt;color:#767171;mso-style-textfill-fill-color:#767171;mso-style-textfill-fill-alpha:100.0%'>pn08/mlx4_0/P1, 17, pn21/mlx4_0/P1, 31, 0.80, 19.20<o:p></o:p></span></p><p class=MsoNormal style='text-indent:35.4pt'><span lang=EN-GB style='font-size:8.0pt;color:#767171;mso-style-textfill-fill-color:#767171;mso-style-textfill-fill-alpha:100.0%'>pn08/mlx4_0/P1, 17, pn22/mlx4_0/P1, 33, 0.80, 20.00<o:p></o:p></span></p><p class=MsoNormal><o:p> </o:p></p><p class=MsoNormal>However ibcongest(no -C) on the same setup:</p><p class=MsoNormal><span lang=EN-GB>Stage1 all nodes appear to have halved bandwidth<o:p></o:p></span></p><p class=MsoNormal><span lang=EN-GB>Stage3 pn16 replaces pn17 -> full bandwidth for everyone<o:p></o:p></span></p><p class=MsoNormal><o:p> </o:p></p><p class=MsoNormal style='margin-left:35.4pt'><span style='font-size:8.0pt;color:#767171;mso-style-textfill-fill-color:#767171;mso-style-textfill-fill-alpha:100.0%'># SRC, SLID, DST, DLID, BW [GB/s], AGG-BW [GB/s]<o:p></o:p></span></p><p class=MsoNormal style='margin-left:35.4pt'><span style='font-size:8.0pt;color:#767171;mso-style-textfill-fill-color:#767171;mso-style-textfill-fill-alpha:100.0%'>STAGE:1<o:p></o:p></span></p><p class=MsoNormal style='margin-left:35.4pt'><span style='font-size:8.0pt;color:#767171;mso-style-textfill-fill-color:#767171;mso-style-textfill-fill-alpha:100.0%'>pn04/U1/P1, 10, pn17/mlx4_0/P1, 21, 0.40, 0.40<o:p></o:p></span></p><p class=MsoNormal style='margin-left:35.4pt'><span style='font-size:8.0pt;color:#767171;mso-style-textfill-fill-color:#767171;mso-style-textfill-fill-alpha:100.0%'>pn04/U1/P1, 10, pn18/mlx4_0/P1, 20, 0.40, 0.80<o:p></o:p></span></p><p class=MsoNormal style='margin-left:35.4pt'><span style='font-size:8.0pt;color:#767171;mso-style-textfill-fill-color:#767171;mso-style-textfill-fill-alpha:100.0%'>pn04/U1/P1, 10, pn19/mlx4_0/P1, 25, 0.40, 1.20<o:p></o:p></span></p><p class=MsoNormal style='margin-left:35.4pt'><span style='font-size:8.0pt;color:#767171;mso-style-textfill-fill-color:#767171;mso-style-textfill-fill-alpha:100.0%'>pn04/U1/P1, 10, pn21/mlx4_0/P1, 31, 0.40, 1.60<o:p></o:p></span></p><p class=MsoNormal style='margin-left:35.4pt'><span style='font-size:8.0pt;color:#767171;mso-style-textfill-fill-color:#767171;mso-style-textfill-fill-alpha:100.0%'>pn04/U1/P1, 10, pn22/mlx4_0/P1, 33, 0.40, 2.00<o:p></o:p></span></p><p class=MsoNormal style='margin-left:35.4pt'><span style='font-size:8.0pt;color:#767171;mso-style-textfill-fill-color:#767171;mso-style-textfill-fill-alpha:100.0%'>pn05/mlx4_0/P1, 14, pn17/mlx4_0/P1, 21, 0.40, 2.40<o:p></o:p></span></p><p class=MsoNormal style='margin-left:35.4pt'><span style='font-size:8.0pt;color:#767171;mso-style-textfill-fill-color:#767171;mso-style-textfill-fill-alpha:100.0%'>pn05/mlx4_0/P1, 14, pn18/mlx4_0/P1, 20, 0.40, 2.80<o:p></o:p></span></p><p class=MsoNormal style='margin-left:35.4pt'><span style='font-size:8.0pt;color:#767171;mso-style-textfill-fill-color:#767171;mso-style-textfill-fill-alpha:100.0%'>pn05/mlx4_0/P1, 14, pn19/mlx4_0/P1, 25, 0.40, 3.20<o:p></o:p></span></p><p class=MsoNormal style='margin-left:35.4pt'><span style='font-size:8.0pt;color:#767171;mso-style-textfill-fill-color:#767171;mso-style-textfill-fill-alpha:100.0%'>pn05/mlx4_0/P1, 14, pn21/mlx4_0/P1, 31, 0.40, 3.60<o:p></o:p></span></p><p class=MsoNormal style='margin-left:35.4pt'><span style='font-size:8.0pt;color:#767171;mso-style-textfill-fill-color:#767171;mso-style-textfill-fill-alpha:100.0%'>pn05/mlx4_0/P1, 14, pn22/mlx4_0/P1, 33, 0.40, 4.00<o:p></o:p></span></p><p class=MsoNormal style='margin-left:35.4pt'><span style='font-size:8.0pt;color:#767171;mso-style-textfill-fill-color:#767171;mso-style-textfill-fill-alpha:100.0%'>pn06/mlx4_0/P1, 15, pn17/mlx4_0/P1, 21, 0.40, 4.40<o:p></o:p></span></p><p class=MsoNormal style='margin-left:35.4pt'><span style='font-size:8.0pt;color:#767171;mso-style-textfill-fill-color:#767171;mso-style-textfill-fill-alpha:100.0%'>pn06/mlx4_0/P1, 15, pn18/mlx4_0/P1, 20, 0.40, 4.80<o:p></o:p></span></p><p class=MsoNormal style='margin-left:35.4pt'><span style='font-size:8.0pt;color:#767171;mso-style-textfill-fill-color:#767171;mso-style-textfill-fill-alpha:100.0%'>pn06/mlx4_0/P1, 15, pn19/mlx4_0/P1, 25, 0.40, 5.20<o:p></o:p></span></p><p class=MsoNormal style='margin-left:35.4pt'><span style='font-size:8.0pt;color:#767171;mso-style-textfill-fill-color:#767171;mso-style-textfill-fill-alpha:100.0%'>pn06/mlx4_0/P1, 15, pn21/mlx4_0/P1, 31, 0.40, 5.60<o:p></o:p></span></p><p class=MsoNormal style='margin-left:35.4pt'><span style='font-size:8.0pt;color:#767171;mso-style-textfill-fill-color:#767171;mso-style-textfill-fill-alpha:100.0%'>pn06/mlx4_0/P1, 15, pn22/mlx4_0/P1, 33, 0.40, 6.00<o:p></o:p></span></p><p class=MsoNormal style='margin-left:35.4pt'><span style='font-size:8.0pt;color:#767171;mso-style-textfill-fill-color:#767171;mso-style-textfill-fill-alpha:100.0%'>pn07/mlx4_0/P1, 23, pn17/mlx4_0/P1, 21, 0.40, 6.40<o:p></o:p></span></p><p class=MsoNormal style='margin-left:35.4pt'><span style='font-size:8.0pt;color:#767171;mso-style-textfill-fill-color:#767171;mso-style-textfill-fill-alpha:100.0%'>pn07/mlx4_0/P1, 23, pn18/mlx4_0/P1, 20, 0.40, 6.80<o:p></o:p></span></p><p class=MsoNormal style='margin-left:35.4pt'><span style='font-size:8.0pt;color:#767171;mso-style-textfill-fill-color:#767171;mso-style-textfill-fill-alpha:100.0%'>pn07/mlx4_0/P1, 23, pn19/mlx4_0/P1, 25, 0.40, 7.20<o:p></o:p></span></p><p class=MsoNormal style='margin-left:35.4pt'><span style='font-size:8.0pt;color:#767171;mso-style-textfill-fill-color:#767171;mso-style-textfill-fill-alpha:100.0%'>pn07/mlx4_0/P1, 23, pn21/mlx4_0/P1, 31, 0.40, 7.60<o:p></o:p></span></p><p class=MsoNormal style='margin-left:35.4pt'><span style='font-size:8.0pt;color:#767171;mso-style-textfill-fill-color:#767171;mso-style-textfill-fill-alpha:100.0%'>pn07/mlx4_0/P1, 23, pn22/mlx4_0/P1, 33, 0.40, 8.00<o:p></o:p></span></p><p class=MsoNormal style='margin-left:35.4pt'><span style='font-size:8.0pt;color:#767171;mso-style-textfill-fill-color:#767171;mso-style-textfill-fill-alpha:100.0%'>pn08/mlx4_0/P1, 17, pn17/mlx4_0/P1, 21, 0.40, 8.40<o:p></o:p></span></p><p class=MsoNormal style='margin-left:35.4pt'><span style='font-size:8.0pt;color:#767171;mso-style-textfill-fill-color:#767171;mso-style-textfill-fill-alpha:100.0%'>pn08/mlx4_0/P1, 17, pn18/mlx4_0/P1, 20, 0.40, 8.80<o:p></o:p></span></p><p class=MsoNormal style='margin-left:35.4pt'><span style='font-size:8.0pt;color:#767171;mso-style-textfill-fill-color:#767171;mso-style-textfill-fill-alpha:100.0%'>pn08/mlx4_0/P1, 17, pn19/mlx4_0/P1, 25, 0.40, 9.20<o:p></o:p></span></p><p class=MsoNormal style='margin-left:35.4pt'><span style='font-size:8.0pt;color:#767171;mso-style-textfill-fill-color:#767171;mso-style-textfill-fill-alpha:100.0%'>pn08/mlx4_0/P1, 17, pn21/mlx4_0/P1, 31, 0.40, 9.60<o:p></o:p></span></p><p class=MsoNormal style='margin-left:35.4pt'><span style='font-size:8.0pt;color:#767171;mso-style-textfill-fill-color:#767171;mso-style-textfill-fill-alpha:100.0%'>pn08/mlx4_0/P1, 17, pn22/mlx4_0/P1, 33, 0.40, 10.00<o:p></o:p></span></p><p class=MsoNormal style='margin-left:35.4pt'><span style='font-size:8.0pt;color:#767171;mso-style-textfill-fill-color:#767171;mso-style-textfill-fill-alpha:100.0%'>STAGE:2<o:p></o:p></span></p><p class=MsoNormal style='margin-left:35.4pt'><span style='font-size:8.0pt;color:#767171;mso-style-textfill-fill-color:#767171;mso-style-textfill-fill-alpha:100.0%'>pn04/U1/P1, 10, pn17/mlx4_0/P1, 21, 4.00, 4.00<o:p></o:p></span></p><p class=MsoNormal style='margin-left:35.4pt'><span style='font-size:8.0pt;color:#767171;mso-style-textfill-fill-color:#767171;mso-style-textfill-fill-alpha:100.0%'>STAGE:3<o:p></o:p></span></p><p class=MsoNormal style='margin-left:35.4pt'><span style='font-size:8.0pt;color:#767171;mso-style-textfill-fill-color:#767171;mso-style-textfill-fill-alpha:100.0%'>pn04/U1/P1, 10, pn16/mlx4_0/P1, 19, 0.80, 0.80<o:p></o:p></span></p><p class=MsoNormal style='margin-left:35.4pt'><span style='font-size:8.0pt;color:#767171;mso-style-textfill-fill-color:#767171;mso-style-textfill-fill-alpha:100.0%'>pn04/U1/P1, 10, pn18/mlx4_0/P1, 20, 0.80, 1.60<o:p></o:p></span></p><p class=MsoNormal style='margin-left:35.4pt'><span style='font-size:8.0pt;color:#767171;mso-style-textfill-fill-color:#767171;mso-style-textfill-fill-alpha:100.0%'>pn04/U1/P1, 10, pn19/mlx4_0/P1, 25, 0.80, 2.40<o:p></o:p></span></p><p class=MsoNormal style='margin-left:35.4pt'><span style='font-size:8.0pt;color:#767171;mso-style-textfill-fill-color:#767171;mso-style-textfill-fill-alpha:100.0%'>pn04/U1/P1, 10, pn21/mlx4_0/P1, 31, 0.80, 3.20<o:p></o:p></span></p><p class=MsoNormal style='margin-left:35.4pt'><span style='font-size:8.0pt;color:#767171;mso-style-textfill-fill-color:#767171;mso-style-textfill-fill-alpha:100.0%'>pn04/U1/P1, 10, pn22/mlx4_0/P1, 33, 0.80, 4.00<o:p></o:p></span></p><p class=MsoNormal style='margin-left:35.4pt'><span style='font-size:8.0pt;color:#767171;mso-style-textfill-fill-color:#767171;mso-style-textfill-fill-alpha:100.0%'>pn05/mlx4_0/P1, 14, pn16/mlx4_0/P1, 19, 0.80, 4.80<o:p></o:p></span></p><p class=MsoNormal style='margin-left:35.4pt'><span style='font-size:8.0pt;color:#767171;mso-style-textfill-fill-color:#767171;mso-style-textfill-fill-alpha:100.0%'>pn05/mlx4_0/P1, 14, pn18/mlx4_0/P1, 20, 0.80, 5.60<o:p></o:p></span></p><p class=MsoNormal style='margin-left:35.4pt'><span style='font-size:8.0pt;color:#767171;mso-style-textfill-fill-color:#767171;mso-style-textfill-fill-alpha:100.0%'>pn05/mlx4_0/P1, 14, pn19/mlx4_0/P1, 25, 0.80, 6.40<o:p></o:p></span></p><p class=MsoNormal style='margin-left:35.4pt'><span style='font-size:8.0pt;color:#767171;mso-style-textfill-fill-color:#767171;mso-style-textfill-fill-alpha:100.0%'>pn05/mlx4_0/P1, 14, pn21/mlx4_0/P1, 31, 0.80, 7.20<o:p></o:p></span></p><p class=MsoNormal style='margin-left:35.4pt'><span style='font-size:8.0pt;color:#767171;mso-style-textfill-fill-color:#767171;mso-style-textfill-fill-alpha:100.0%'>pn05/mlx4_0/P1, 14, pn22/mlx4_0/P1, 33, 0.80, 8.00<o:p></o:p></span></p><p class=MsoNormal style='margin-left:35.4pt'><span style='font-size:8.0pt;color:#767171;mso-style-textfill-fill-color:#767171;mso-style-textfill-fill-alpha:100.0%'>pn06/mlx4_0/P1, 15, pn16/mlx4_0/P1, 19, 0.80, 8.80<o:p></o:p></span></p><p class=MsoNormal style='margin-left:35.4pt'><span style='font-size:8.0pt;color:#767171;mso-style-textfill-fill-color:#767171;mso-style-textfill-fill-alpha:100.0%'>pn06/mlx4_0/P1, 15, pn18/mlx4_0/P1, 20, 0.80, 9.60<o:p></o:p></span></p><p class=MsoNormal style='margin-left:35.4pt'><span style='font-size:8.0pt;color:#767171;mso-style-textfill-fill-color:#767171;mso-style-textfill-fill-alpha:100.0%'>pn06/mlx4_0/P1, 15, pn19/mlx4_0/P1, 25, 0.80, 10.40<o:p></o:p></span></p><p class=MsoNormal style='margin-left:35.4pt'><span style='font-size:8.0pt;color:#767171;mso-style-textfill-fill-color:#767171;mso-style-textfill-fill-alpha:100.0%'>pn06/mlx4_0/P1, 15, pn21/mlx4_0/P1, 31, 0.80, 11.20<o:p></o:p></span></p><p class=MsoNormal style='margin-left:35.4pt'><span style='font-size:8.0pt;color:#767171;mso-style-textfill-fill-color:#767171;mso-style-textfill-fill-alpha:100.0%'>pn06/mlx4_0/P1, 15, pn22/mlx4_0/P1, 33, 0.80, 12.00<o:p></o:p></span></p><p class=MsoNormal style='margin-left:35.4pt'><span style='font-size:8.0pt;color:#767171;mso-style-textfill-fill-color:#767171;mso-style-textfill-fill-alpha:100.0%'>pn07/mlx4_0/P1, 23, pn16/mlx4_0/P1, 19, 0.80, 12.80<o:p></o:p></span></p><p class=MsoNormal style='margin-left:35.4pt'><span style='font-size:8.0pt;color:#767171;mso-style-textfill-fill-color:#767171;mso-style-textfill-fill-alpha:100.0%'>pn07/mlx4_0/P1, 23, pn18/mlx4_0/P1, 20, 0.80, 13.60<o:p></o:p></span></p><p class=MsoNormal style='margin-left:35.4pt'><span style='font-size:8.0pt;color:#767171;mso-style-textfill-fill-color:#767171;mso-style-textfill-fill-alpha:100.0%'>pn07/mlx4_0/P1, 23, pn19/mlx4_0/P1, 25, 0.80, 14.40<o:p></o:p></span></p><p class=MsoNormal style='margin-left:35.4pt'><span style='font-size:8.0pt;color:#767171;mso-style-textfill-fill-color:#767171;mso-style-textfill-fill-alpha:100.0%'>pn07/mlx4_0/P1, 23, pn21/mlx4_0/P1, 31, 0.80, 15.20<o:p></o:p></span></p><p class=MsoNormal style='margin-left:35.4pt'><span style='font-size:8.0pt;color:#767171;mso-style-textfill-fill-color:#767171;mso-style-textfill-fill-alpha:100.0%'>pn07/mlx4_0/P1, 23, pn22/mlx4_0/P1, 33, 0.80, 16.00<o:p></o:p></span></p><p class=MsoNormal style='margin-left:35.4pt'><span style='font-size:8.0pt;color:#767171;mso-style-textfill-fill-color:#767171;mso-style-textfill-fill-alpha:100.0%'>pn08/mlx4_0/P1, 17, pn16/mlx4_0/P1, 19, 0.80, 16.80<o:p></o:p></span></p><p class=MsoNormal style='margin-left:35.4pt'><span style='font-size:8.0pt;color:#767171;mso-style-textfill-fill-color:#767171;mso-style-textfill-fill-alpha:100.0%'>pn08/mlx4_0/P1, 17, pn18/mlx4_0/P1, 20, 0.80, 17.60<o:p></o:p></span></p><p class=MsoNormal style='margin-left:35.4pt'><span style='font-size:8.0pt;color:#767171;mso-style-textfill-fill-color:#767171;mso-style-textfill-fill-alpha:100.0%'>pn08/mlx4_0/P1, 17, pn19/mlx4_0/P1, 25, 0.80, 18.40<o:p></o:p></span></p><p class=MsoNormal style='margin-left:35.4pt'><span style='font-size:8.0pt;color:#767171;mso-style-textfill-fill-color:#767171;mso-style-textfill-fill-alpha:100.0%'>pn08/mlx4_0/P1, 17, pn21/mlx4_0/P1, 31, 0.80, 19.20<o:p></o:p></span></p><p class=MsoNormal style='margin-left:35.4pt'><span style='font-size:8.0pt;color:#767171;mso-style-textfill-fill-color:#767171;mso-style-textfill-fill-alpha:100.0%'>pn08/mlx4_0/P1, 17, pn22/mlx4_0/P1, 33, 0.80, 20.00</span></p><p class=MsoNormal><span lang=EN-GB><o:p> </o:p></span></p><p class=MsoNormal><span lang=EN-GB>They way I interpret these results is that there must be some form of underlying synchronization going on that I am not aware of and I have no idea whether this is the doing of libfabric or the InfiniBand protocol however since ibcongest without flow control appears to behave similarly(however not linear) to the benchmark I tend to assume that this is the doing of the InfiniBand protocol. So some questions arise: what is causing this synch? can I turn this synch off? Can I do it through libfabric? Is it possible to implement manual routing within libfabric? If anyone could share some insight on this issue I would be very grateful. <o:p></o:p></span></p><p class=MsoNormal><span lang=EN-GB><o:p> </o:p></span></p><p class=MsoNormal><span lang=EN-GB>regards,<o:p></o:p></span></p><p class=MsoNormal><span lang=EN-GB><o:p> </o:p></span></p><p class=MsoNormal><span lang=EN-GB>Andrew<o:p></o:p></span></p></div></body></html>