<html xmlns:v="urn:schemas-microsoft-com:vml" xmlns:o="urn:schemas-microsoft-com:office:office" xmlns:w="urn:schemas-microsoft-com:office:word" xmlns:m="http://schemas.microsoft.com/office/2004/12/omml" xmlns:mv="http://macVmlSchemaUri" xmlns="http://www.w3.org/TR/REC-html40">
<head>
<meta http-equiv="Content-Type" content="text/html; charset=utf-8">
<meta name="Title" content="">
<meta name="Keywords" content="">
<meta name="Generator" content="Microsoft Word 15 (filtered medium)">
<!--[if !mso]><style>v\:* {behavior:url(#default#VML);}
o\:* {behavior:url(#default#VML);}
w\:* {behavior:url(#default#VML);}
.shape {behavior:url(#default#VML);}
</style><![endif]--><style><!--
/* Font Definitions */
@font-face
{font-family:"Cambria Math";
panose-1:2 4 5 3 5 4 6 3 2 4;}
@font-face
{font-family:Calibri;
panose-1:2 15 5 2 2 2 4 3 2 4;}
/* Style Definitions */
p.MsoNormal, li.MsoNormal, div.MsoNormal
{margin:0in;
margin-bottom:.0001pt;
font-size:12.0pt;
font-family:Calibri;}
a:link, span.MsoHyperlink
{mso-style-priority:99;
color:#0563C1;
text-decoration:underline;}
a:visited, span.MsoHyperlinkFollowed
{mso-style-priority:99;
color:#954F72;
text-decoration:underline;}
span.EmailStyle17
{mso-style-type:personal-compose;
font-family:Calibri;
color:windowtext;}
span.msoIns
{mso-style-type:export-only;
mso-style-name:"";
text-decoration:underline;
color:teal;}
.MsoChpDefault
{mso-style-type:export-only;
font-family:Calibri;}
@page WordSection1
{size:8.5in 11.0in;
margin:1.0in 1.0in 1.0in 1.0in;}
div.WordSection1
{page:WordSection1;}
--></style>
</head>
<body bgcolor="white" lang="EN-US" link="#0563C1" vlink="#954F72">
<div class="WordSection1">
<p class="MsoNormal"><span style="font-size:11.0pt">OFI group;<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt"><o:p> </o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt">We work with HPC systems that deploy same but multiple network adapters (including Intel OmniPath and MLX infiniband adapters) on compute nodes.<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt">Over time, we encountered two issues which we believe can be addressed by OFI library.<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt"><o:p> </o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt">First, a number of MPI implementations assume homogenous SW/HW setup on all compute nodes. For example, assume nodes with 2 adapters and 2 separate networks. Some MPI implementations assume that network adapter
A resides on CPU socket 0 on all nodes and connect to network 0; and network adapter B resides on CPU socket 1 and connect to network 1. Unfortunately that is not always the case. There are systems where some nodes use adapter A to connect to network 0 and
others use adapter B to connect to network 0. Same for network 1, where we have mixed (crossed) adapters connected to same network. In such cases, MPII and lower layers cannot establish peer to peer connection. The best way to solve this is to use the
network subnet ID to establish connection between pairs. When there are multiple networks and subnetwork IDs, mpirun would specify a network ID (Platform MPI does this) and then the software can figure out from the subnet ID what adapter each node is using
to connect to such network. Instead of implementing this logic in each MPI, it would be great if OFI implements this logic since it is a one stop shop over all network devices and providers.<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt"><o:p> </o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt"><o:p> </o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt">Second, multirail support is a hit and miss across MPI implementations. Intel Omnipath PSM2 library actually did a great job here by implementing multirail support at the PSM2 level. This means all above
layers like MPI would get this functionality for free. Again, given that many MPI implementation can be built on top of OFI, It would be also great if OFI has multirail support.<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt"><o:p> </o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt">Thank you<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt"><o:p> </o:p></span></p>
<div>
<div>
<p class="MsoNormal"><span style="font-size:10.5pt;color:black"><o:p> </o:p></span></p>
</div>
<div>
<div>
<div>
<p class="MsoNormal"><span style="font-size:10.5pt;color:black">-- <o:p></o:p></span></p>
</div>
<div>
<p class="MsoNormal"><b><span style="font-size:10.5pt;color:black">Daniel Faraj</span></b><span style="font-size:10.5pt;color:black"><o:p></o:p></span></p>
</div>
<div>
<p class="MsoNormal"><span style="font-size:10.5pt;color:black">HPE Performance Engineering<o:p></o:p></span></p>
</div>
</div>
<div>
<p class="MsoNormal"><span style="font-size:10.5pt;color:black">651.683.7605 Office<o:p></o:p></span></p>
</div>
<div>
<p class="MsoNormal"><span style="font-size:10.5pt;color:black"><a href="mailto:daniel.faraj@hpe.com"><span style="color:blue">daniel.faraj@hpe.com</span></a><o:p></o:p></span></p>
</div>
</div>
<div>
<p class="MsoNormal"><span style="font-size:10.5pt;color:black"><o:p> </o:p></span></p>
</div>
<div>
<p class="MsoNormal"><span style="font-size:10.5pt;color:black"><img border="0" width="119" height="50" id="_x0000_i1025" src="cid:image001.png@01D2DA03.5C5447B0"></span><span style="font-size:10.5pt;color:black"><o:p></o:p></span></p>
</div>
<div>
<p class="MsoNormal"><span style="font-size:10.5pt;color:black"><o:p> </o:p></span></p>
</div>
</div>
<p class="MsoNormal"><o:p> </o:p></p>
</div>
</body>
</html>