<html>
<head>
<meta http-equiv="Content-Type" content="text/html; charset=us-ascii">
<style type="text/css" style="display:none;"> P {margin-top:0;margin-bottom:0;} </style>
</head>
<body dir="ltr">
<div style="font-family: Calibri, Arial, Helvetica, sans-serif; font-size: 12pt; color: rgb(0, 0, 0);">
<br>
</div>
<div style="font-family: Calibri, Arial, Helvetica, sans-serif; font-size: 12pt; color: rgb(0, 0, 0);">
Hi Arun, </div>
<div style="font-family: Calibri, Arial, Helvetica, sans-serif; font-size: 12pt; color: rgb(0, 0, 0);">
<br>
</div>
<div style="font-family: Calibri, Arial, Helvetica, sans-serif; font-size: 12pt; color: rgb(0, 0, 0);">
Thanks for asking.</div>
<div style="font-family: Calibri, Arial, Helvetica, sans-serif; font-size: 12pt; color: rgb(0, 0, 0);">
<br>
</div>
<div style="font-family: Calibri, Arial, Helvetica, sans-serif; font-size: 12pt; color: rgb(0, 0, 0);">
I am using Intel parallel studio 19.0.4.243. Using the "bt-mz.E.x" NASA benchmark (<a href="https://www.nas.nasa.gov/publications/npb.html" id="LPNoLP254908">https://www.nas.nasa.gov/publications/npb.html</a>), the job fails on 700 cores, but runs fine on
200 cores. The output is quite large, so I have gzipped it before attaching. I have seen repeatedly that when I get beyond a certain # of cores, the job fails. This happens for many other codes as well. This also does not happen with the Intel 2018
compilers.</div>
<div style="font-family: Calibri, Arial, Helvetica, sans-serif; font-size: 12pt; color: rgb(0, 0, 0);">
<br>
</div>
<div style="font-family: Calibri, Arial, Helvetica, sans-serif; font-size: 12pt; color: rgb(0, 0, 0);">
<span style="color: rgb(0, 0, 0); font-family: Calibri, Arial, Helvetica, sans-serif; font-size: 12pt;">BTW, I didn't let the 200 core run go to completion to reduce the output. It does run to completion with no problem.</span><br>
</div>
<div style="font-family: Calibri, Arial, Helvetica, sans-serif; font-size: 12pt; color: rgb(0, 0, 0);">
<br>
</div>
<div style="font-family: Calibri, Arial, Helvetica, sans-serif; font-size: 12pt; color: rgb(0, 0, 0);">
Let me know if you have any other questions.</div>
<div style="font-family: Calibri, Arial, Helvetica, sans-serif; font-size: 12pt; color: rgb(0, 0, 0);">
<br>
</div>
<div style="font-family: Calibri, Arial, Helvetica, sans-serif; font-size: 12pt; color: rgb(0, 0, 0);">
Regards, </div>
<div style="font-family: Calibri, Arial, Helvetica, sans-serif; font-size: 12pt; color: rgb(0, 0, 0);">
<br>
</div>
<div style="font-family: Calibri, Arial, Helvetica, sans-serif; font-size: 12pt; color: rgb(0, 0, 0);">
Eric</div>
<div style="font-family: Calibri, Arial, Helvetica, sans-serif; font-size: 12pt; color: rgb(0, 0, 0);">
<br>
</div>
<div style="font-family: Calibri, Arial, Helvetica, sans-serif; font-size: 12pt; color: rgb(0, 0, 0);">
<br>
</div>
<div style="font-family: Calibri, Arial, Helvetica, sans-serif; font-size: 12pt; color: rgb(0, 0, 0);">
<br>
</div>
<br>
<div id="Signature">
<div id="divtagdefaultwrapper" dir="ltr" style="font-size:12pt; color:#000000; font-family:Calibri,Helvetica,sans-serif">
<p style="margin-top: 0px; margin-bottom: 0px;"></p>
<div><br>
</div>
<div><span style="font-family:"Arial Black",Arial,sans-serif; font-size:9pt">-- </span></div>
<div><span style="font-family:"Arial Black",Arial,sans-serif; font-size:12pt">Eric J. Walter</span></div>
<div><br>
</div>
<div><span style="font-family:Corbel,Skia,sans-serif"><span style="font-size:9pt; font-family:"Arial Black",Arial,sans-serif">College of William and Mary </span><span style="font-size:9pt; font-family:"Courier New",monospace"></span></span></div>
<div><span style="font-family:"Arial Black",Arial,sans-serif; font-size:9pt">
<div style="font-family:Calibri,Helvetica,sans-serif,EmojiFont,"Apple Color Emoji","Segoe UI Emoji",NotoColorEmoji,"Segoe UI Symbol","Android Emoji",EmojiSymbols,EmojiFont,"Apple Color Emoji","Segoe UI Emoji",NotoColorEmoji,"Segoe UI Symbol","Android Emoji",EmojiSymbols; font-size:16px">
<span style="font-family:"Courier New",monospace; font-size:9pt"><span style="font-family:"Arial Black",Arial,sans-serif">
<div style="font-family:Calibri,Helvetica,sans-serif,EmojiFont,"Apple Color Emoji","Segoe UI Emoji",NotoColorEmoji,"Segoe UI Symbol","Android Emoji",EmojiSymbols,EmojiFont,"Apple Color Emoji","Segoe UI Emoji",NotoColorEmoji,"Segoe UI Symbol","Android Emoji",EmojiSymbols; font-size:16px">
<span style="font-family:"Arial Black",Arial,sans-serif; font-size:9pt">IT/High Performance Computing Group </span></div>
</span></span></div>
<div style="font-family:Calibri,Helvetica,sans-serif,EmojiFont,"Apple Color Emoji","Segoe UI Emoji",NotoColorEmoji,"Segoe UI Symbol","Android Emoji",EmojiSymbols,EmojiFont,"Apple Color Emoji","Segoe UI Emoji",NotoColorEmoji,"Segoe UI Symbol","Android Emoji",EmojiSymbols; font-size:16px">
<span style="font-family:"Courier New",monospace; font-size:9pt"><span style="font-family:"Arial Black",Arial,sans-serif">ISC 1271 </span></span></div>
</span></div>
<div><span style="font-family:"Arial Black",Arial,sans-serif; font-size:9pt">P.O. Box 8795</span><br>
</div>
<div><span style="font-family:"Courier New",monospace; font-size:9pt"><span style="font-family:"Courier New",monospace; font-size:14px"><span style="font-family:"Courier New",monospace; font-size:14px">
<div style="font-family:Calibri,Helvetica,sans-serif,EmojiFont,"Apple Color Emoji","Segoe UI Emoji",NotoColorEmoji,"Segoe UI Symbol","Android Emoji",EmojiSymbols,EmojiFont,"Apple Color Emoji","Segoe UI Emoji",NotoColorEmoji,"Segoe UI Symbol","Android Emoji",EmojiSymbols; font-size:16px">
<span style="font-family:"Arial Black",Arial,sans-serif; font-size:9pt">Williamsburg, VA 23187-8795</span></div>
</span></span></span></div>
<div><span style="font-family:"Arial Black",Arial,sans-serif; font-size:14px">email: ejwalt@wm.edu</span></div>
<div><span style="font-family:"Courier New",monospace; font-size:14px"></span><span style="font-family:"Arial Black",Arial,sans-serif; font-size:9pt">phone: (757) 221-1886</span><br>
</div>
<div><span style="font-family:"Arial Black",Arial,sans-serif; font-size:9pt">fax: (757) 221-1321</span></div>
<div><br>
</div>
<div><span style="font-family:"Courier New",monospace; font-size:9pt"></span></div>
<p style="margin-top: 0px; margin-bottom: 0px;"></p>
</div>
</div>
<div id="appendonsend"></div>
<hr style="display:inline-block;width:98%" tabindex="-1">
<div id="divRplyFwdMsg" dir="ltr"><font face="Calibri, sans-serif" style="font-size:11pt" color="#000000"><b>From:</b> Ilango, Arun <arun.ilango@intel.com><br>
<b>Sent:</b> Tuesday, September 10, 2019 2:43 PM<br>
<b>To:</b> Walter, Eric J <ejwalt@wm.edu>; libfabric-users@lists.openfabrics.org <libfabric-users@lists.openfabrics.org><br>
<b>Subject:</b> RE: libfabric/intel mpi with mlx5 and > 300 cores/ranks</font>
<div> </div>
</div>
<style>
<!--
@font-face
{font-family:"Cambria Math"}
@font-face
{font-family:Calibri}
@font-face
{font-family:"Arial Black"}
p.x_MsoNormal, li.x_MsoNormal, div.x_MsoNormal
{margin:0in;
margin-bottom:.0001pt;
font-size:11.0pt;
font-family:"Calibri",sans-serif}
a:link, span.x_MsoHyperlink
{color:#0563C1;
text-decoration:underline}
a:visited, span.x_MsoHyperlinkFollowed
{color:#954F72;
text-decoration:underline}
p.x_msonormal0, li.x_msonormal0, div.x_msonormal0
{margin:0in;
margin-bottom:.0001pt;
font-size:11.0pt;
font-family:"Calibri",sans-serif}
span.x_EmailStyle20
{font-family:"Calibri",sans-serif;
color:windowtext}
.x_MsoChpDefault
{font-size:10.0pt}
@page WordSection1
{margin:1.0in 1.0in 1.0in 1.0in}
div.x_WordSection1
{}
-->
</style>
<div lang="EN-US" link="#0563C1" vlink="#954F72">
<div class="x_WordSection1">
<p class="x_MsoNormal">Hi Eric,</p>
<p class="x_MsoNormal"> </p>
<p class="x_MsoNormal">Which version of Intel MPI 2019 are you using? Can you try running with FI_LOG_LEVEL=warn and share the logs?</p>
<p class="x_MsoNormal"> </p>
<p class="x_MsoNormal">Thanks,</p>
<p class="x_MsoNormal">Arun.</p>
<p class="x_MsoNormal"> </p>
<div style="border:none; border-left:solid blue 1.5pt; padding:0in 0in 0in 4.0pt">
<div>
<div style="border:none; border-top:solid #E1E1E1 1.0pt; padding:3.0pt 0in 0in 0in">
<p class="x_MsoNormal"><b>From:</b> Libfabric-users <libfabric-users-bounces@lists.openfabrics.org>
<b>On Behalf Of </b>Walter, Eric J<br>
<b>Sent:</b> Tuesday, August 06, 2019 10:48 AM<br>
<b>To:</b> libfabric-users@lists.openfabrics.org<br>
<b>Subject:</b> [libfabric-users] libfabric/intel mpi with mlx5 and > 300 cores/ranks</p>
</div>
</div>
<p class="x_MsoNormal"> </p>
<div>
<p class="x_MsoNormal"><span style="font-size:12.0pt; color:black"> </span></p>
</div>
<div id="x_Signature">
<div id="x_divtagdefaultwrapper">
<div>
<p class="x_MsoNormal"><span style="font-size:12.0pt; color:black">Hi, we are currently standing up a new cluster with Mellanox ConnectX-5 adapters. I have found that using openMPI, mvapich2, and intel2018-mpi, we can run MPI jobs on all 960 cores in the cluster,
however, using intel2019-mpi we can't get beyond ~300 mpi ranks. If we do, we get the following error for every rank:
</span></p>
<div>
<p class="x_MsoNormal"><span style="font-size:12.0pt; color:black"> </span></p>
</div>
<div>
<p class="x_MsoNormal"><span style="font-size:12.0pt; color:black">Abort(273768207) on node 650 (rank 650 in comm 0): Fatal error in PMPI_Comm_split: Other MPI error, error stack:
</span></p>
</div>
<div>
<p class="x_MsoNormal"><span style="font-size:12.0pt; color:black">PMPI_Comm_split(507)...................: MPI_Comm_split(MPI_COMM_WORLD, color=0, key=650, new_comm=0x7911e8) failed
</span></p>
</div>
<div>
<p class="x_MsoNormal"><span style="font-size:12.0pt; color:black">PMPI_Comm_split(489)...................:
</span></p>
</div>
<div>
<p class="x_MsoNormal"><span style="font-size:12.0pt; color:black">MPIR_Comm_split_impl(167)..............:
</span></p>
</div>
<div>
<p class="x_MsoNormal"><span style="font-size:12.0pt; color:black">MPIR_Allgather_intra_auto(145).........: Failure during collective
</span></p>
</div>
<div>
<p class="x_MsoNormal"><span style="font-size:12.0pt; color:black">MPIR_Allgather_intra_auto(141).........:
</span></p>
</div>
<div>
<p class="x_MsoNormal"><span style="font-size:12.0pt; color:black">MPIR_Allgather_intra_brucks(115).......:
</span></p>
</div>
<div>
<p class="x_MsoNormal"><span style="font-size:12.0pt; color:black">MPIC_Sendrecv(344).....................:
</span></p>
</div>
<div>
<p class="x_MsoNormal"><span style="font-size:12.0pt; color:black">MPID_Isend(662)........................:
</span></p>
</div>
<div>
<p class="x_MsoNormal"><span style="font-size:12.0pt; color:black">MPID_isend_unsafe(282).................:
</span></p>
</div>
<div>
<p class="x_MsoNormal"><span style="font-size:12.0pt; color:black">MPIDI_OFI_send_lightweight_request(106):
</span></p>
</div>
<div>
<p class="x_MsoNormal"><span style="font-size:12.0pt; color:black">(unknown)(): Other MPI error
</span></p>
</div>
<div>
<p class="x_MsoNormal"><span style="font-size:12.0pt; color:black">----------------------------------------------------------------------------------------------------------
</span></p>
</div>
<div>
<p class="x_MsoNormal"><span style="font-size:12.0pt; color:black">This is using the default FI_PROVIDER of ofi_rxm. If we switch to using "verbs", we can run all 960 cores, but tests show an order of magnitude increase in latency and much longer run times.
</span></p>
</div>
<div>
<p class="x_MsoNormal"><span style="font-size:12.0pt; color:black"> </span></p>
</div>
<div>
<p class="x_MsoNormal"><span style="font-size:12.0pt; color:black">We have tried installing our own libfabrics (from the git repo ; also we verified with verbose debugging that we are using this libfabrics) and this behavoir does not change</span></p>
</div>
<div>
<p class="x_MsoNormal"><span style="font-size:12.0pt; color:black"> </span></p>
</div>
<div>
<p class="x_MsoNormal"><span style="font-size:12.0pt; color:black">Is there anything I can change to allow all 960 cores using the default ofi_rxm provider? Or, is there a way to improve performance using the verbs provider?</span></p>
</div>
<div>
<p class="x_MsoNormal"><span style="font-size:12.0pt; color:black"> </span></p>
</div>
<div>
<p class="x_MsoNormal"><span style="font-size:12.0pt; color:black">For completeness:
</span></p>
</div>
<div>
<p class="x_MsoNormal"><span style="font-size:12.0pt; color:black">Using MLNX_OFED_LINUX-4.6-1.0.1.1-rhel7.6-x86_64 ofed
</span></p>
</div>
<div>
<p class="x_MsoNormal"><span style="font-size:12.0pt; color:black">CentOS 7.6.1810 (kernel = 3.10.0-957.21.3.el7.x86_64)
</span></p>
</div>
<div>
<p class="x_MsoNormal"><span style="font-size:12.0pt; color:black">Intel Parallel studio version 19.0.4.243
</span></p>
</div>
<div>
<p class="x_MsoNormal"><span style="font-size:12.0pt; color:black">Infiniband controller: Mellanox Technologies MT27800 Family [ConnectX-5]
</span></p>
</div>
<div>
<p class="x_MsoNormal"><span style="font-size:12.0pt; color:black"> </span></p>
</div>
<div>
<p class="x_MsoNormal"><span style="font-size:12.0pt; color:black"> </span></p>
</div>
<div>
<p class="x_MsoNormal"><span style="font-size:12.0pt; color:black">Thanks! </span>
</p>
</div>
<div>
<p class="x_MsoNormal"><span style="font-size:12.0pt; color:black"> </span></p>
</div>
<p class="x_MsoNormal"><span style="font-size:12.0pt; color:black">Eric</span></p>
</div>
<div>
<p class="x_MsoNormal"><span style="font-size:9.0pt; font-family:"Arial Black",sans-serif; color:black">-- </span><span style="font-size:12.0pt; color:black"></span></p>
</div>
<div>
<p class="x_MsoNormal"><span style="font-size:12.0pt; font-family:"Arial Black",sans-serif; color:black">Eric J. Walter</span><span style="font-size:12.0pt; color:black"></span></p>
</div>
<div>
<p class="x_MsoNormal"><span style="font-size:12.0pt; color:black"> </span></p>
</div>
<div>
<p class="x_MsoNormal"><span style="font-size:9.0pt; font-family:"Arial Black",sans-serif; color:black">College of William and Mary </span><span style="font-size:12.0pt; color:black"></span></p>
</div>
<div>
<div>
<div>
<p class="x_MsoNormal"><span style="font-size:9.0pt; font-family:"Arial Black",sans-serif; color:black">IT/High Performance Computing Group </span><span style="font-size:12.0pt; color:black"></span></p>
</div>
</div>
<div>
<p class="x_MsoNormal"><span style="font-size:9.0pt; font-family:"Arial Black",sans-serif; color:black">ISC 1271 </span><span style="font-size:12.0pt; color:black"></span></p>
</div>
</div>
<div>
<p class="x_MsoNormal"><span style="font-size:9.0pt; font-family:"Arial Black",sans-serif; color:black">P.O. Box 8795</span><span style="font-size:12.0pt; color:black"></span></p>
</div>
<div>
<div>
<p class="x_MsoNormal"><span style="font-size:9.0pt; font-family:"Arial Black",sans-serif; color:black">Williamsburg, VA 23187-8795</span><span style="font-size:12.0pt; color:black"></span></p>
</div>
</div>
<div>
<p class="x_MsoNormal"><span style="font-size:10.5pt; font-family:"Arial Black",sans-serif; color:black">email: <a href="mailto:ejwalt@wm.edu">ejwalt@wm.edu</a></span><span style="font-size:12.0pt; color:black"></span></p>
</div>
<div>
<p class="x_MsoNormal"><span style="font-size:9.0pt; font-family:"Arial Black",sans-serif; color:black">phone: (757) 221-1886</span><span style="font-size:12.0pt; color:black"></span></p>
</div>
<div>
<p class="x_MsoNormal"><span style="font-size:9.0pt; font-family:"Arial Black",sans-serif; color:black">fax: (757) 221-1321</span><span style="font-size:12.0pt; color:black"></span></p>
</div>
<div>
<p class="x_MsoNormal"><span style="font-size:12.0pt; color:black"> </span></p>
</div>
</div>
</div>
</div>
</div>
</div>
</body>
</html>