<html xmlns:v="urn:schemas-microsoft-com:vml" xmlns:o="urn:schemas-microsoft-com:office:office" xmlns:w="urn:schemas-microsoft-com:office:word" xmlns:m="http://schemas.microsoft.com/office/2004/12/omml" xmlns="http://www.w3.org/TR/REC-html40">
<head>
<meta http-equiv="Content-Type" content="text/html; charset=utf-8">
<meta name="Generator" content="Microsoft Word 15 (filtered medium)">
<style><!--
/* Font Definitions */
@font-face
{font-family:Helvetica;
panose-1:2 11 6 4 2 2 2 2 2 4;}
@font-face
{font-family:"Cambria Math";
panose-1:2 4 5 3 5 4 6 3 2 4;}
@font-face
{font-family:Calibri;
panose-1:2 15 5 2 2 2 4 3 2 4;}
@font-face
{font-family:"Lucida Console";
panose-1:2 11 6 9 4 5 4 2 2 4;}
/* Style Definitions */
p.MsoNormal, li.MsoNormal, div.MsoNormal
{margin:0cm;
margin-bottom:.0001pt;
font-size:12.0pt;
font-family:"Times New Roman",serif;}
a:link, span.MsoHyperlink
{mso-style-priority:99;
color:blue;
text-decoration:underline;}
a:visited, span.MsoHyperlinkFollowed
{mso-style-priority:99;
color:purple;
text-decoration:underline;}
span.EmailStyle17
{mso-style-type:personal-reply;
font-family:"Calibri",sans-serif;
color:#1F497D;}
.MsoChpDefault
{mso-style-type:export-only;
font-size:10.0pt;}
@page WordSection1
{size:612.0pt 792.0pt;
margin:72.0pt 72.0pt 72.0pt 72.0pt;}
div.WordSection1
{page:WordSection1;}
--></style><!--[if gte mso 9]><xml>
<o:shapedefaults v:ext="edit" spidmax="1026" />
</xml><![endif]--><!--[if gte mso 9]><xml>
<o:shapelayout v:ext="edit">
<o:idmap v:ext="edit" data="1" />
</o:shapelayout></xml><![endif]-->
</head>
<body lang="RU" link="blue" vlink="purple">
<div class="WordSection1">
<p class="MsoNormal"><span lang="EN-US" style="font-size:11.0pt;font-family:"Calibri",sans-serif;color:#1F497D;mso-fareast-language:EN-US">Hi Mohammed,<o:p></o:p></span></p>
<p class="MsoNormal"><span lang="EN-US" style="font-size:11.0pt;font-family:"Calibri",sans-serif;color:#1F497D;mso-fareast-language:EN-US"><o:p> </o:p></span></p>
<p class="MsoNormal"><span lang="EN-US" style="font-size:11.0pt;font-family:"Calibri",sans-serif;color:#1F497D;mso-fareast-language:EN-US">I don’t recommend to verbs provider with FI_EP_RDM EP type. We deprecating this provider since libfabric 1.7.<o:p></o:p></span></p>
<p class="MsoNormal"><span lang="EN-US" style="font-size:11.0pt;font-family:"Calibri",sans-serif;color:#1F497D;mso-fareast-language:EN-US">Now our default provider is RxM/verbs (verbs with FI_EP_MSG EP type) to run libfabric on IB/iWARP/RoCE verbs devices.<o:p></o:p></span></p>
<p class="MsoNormal"><span lang="EN-US" style="font-size:11.0pt;font-family:"Calibri",sans-serif;color:#1F497D;mso-fareast-language:EN-US">If you use libfabric 1.6.x, RxM/verbs should be the default provider.<o:p></o:p></span></p>
<p class="MsoNormal"><span lang="EN-US" style="font-size:11.0pt;font-family:"Calibri",sans-serif;color:#1F497D;mso-fareast-language:EN-US"><o:p> </o:p></span></p>
<p class="MsoNormal"><span lang="EN-US" style="font-size:11.0pt;font-family:"Calibri",sans-serif;color:#1F497D;mso-fareast-language:EN-US">Can you see the hangs when using libfabric’s master branch?<o:p></o:p></span></p>
<p class="MsoNormal"><span lang="EN-US" style="font-size:11.0pt;font-family:"Calibri",sans-serif;color:#1F497D;mso-fareast-language:EN-US"><o:p> </o:p></span></p>
<p class="MsoNormal"><span lang="EN-US" style="font-size:11.0pt;font-family:"Calibri",sans-serif;color:#1F497D;mso-fareast-language:EN-US">--<o:p></o:p></span></p>
<p class="MsoNormal"><span lang="EN-US" style="font-size:11.0pt;font-family:"Calibri",sans-serif;color:#1F497D;mso-fareast-language:EN-US">Dmitry<o:p></o:p></span></p>
<p class="MsoNormal"><a name="_MailEndCompose"><span lang="EN-US" style="mso-fareast-language:EN-US"><o:p> </o:p></span></a></p>
<div>
<div style="border:none;border-top:solid #E1E1E1 1.0pt;padding:3.0pt 0cm 0cm 0cm">
<p class="MsoNormal"><a name="_____replyseparator"></a><b><span lang="EN-US" style="font-size:11.0pt;font-family:"Calibri",sans-serif">From:</span></b><span lang="EN-US" style="font-size:11.0pt;font-family:"Calibri",sans-serif"> Mohammed Shaheen [mailto:m_shaheen1984@yahoo.com]
<br>
<b>Sent:</b> Thursday, November 22, 2018 4:54 PM<br>
<b>To:</b> Gladkov, Dmitry <dmitry.gladkov@intel.com>; Hefty, Sean <sean.hefty@intel.com>; libfabric-users@lists.openfabrics.org; ofiwg@lists.openfabrics.org; Ilango, Arun <arun.ilango@intel.com><br>
<b>Subject:</b> Re: [libfabric-users] intel mpi with libfabric<o:p></o:p></span></p>
</div>
</div>
<p class="MsoNormal"><o:p> </o:p></p>
<div>
<div>
<p class="MsoNormal"><span style="font-family:"Lucida Console"">Thanks Arun and Dmitry for your support.<o:p></o:p></span></p>
</div>
<div>
<p class="MsoNormal"><span style="font-family:"Lucida Console""><o:p> </o:p></span></p>
</div>
<div>
<p class="MsoNormal"><span style="font-family:"Lucida Console"">Well, I am building my own libfabric, and I export the right variables and source intel mpi with -ofi_internal=0. I figured out where the problem is:<o:p></o:p></span></p>
</div>
<div>
<p class="MsoNormal"><span style="font-family:"Lucida Console"">1. If libfabric is built for all providers, i.e. run ./configure without including and exluding providers, it will build ibverbs among others; however, the mpi test program will hang during execution.
<o:p></o:p></span></p>
</div>
<div>
<p class="MsoNormal"><span style="font-family:"Lucida Console"">2. If libfabric configured with only enabling ibverbs and setting all other providers, i.e. ./configure --enable-verbs=yes --enable-rxm=no --enable-rxd=no --enable-sockets=no --enable-tcp=no --enable-udp=no,
mpi test program will run through<o:p></o:p></span></p>
</div>
<div>
<p class="MsoNormal"><span style="font-family:"Lucida Console""><o:p> </o:p></span></p>
</div>
<div>
<p class="MsoNormal"><span style="font-family:"Lucida Console"">Another observation when I enable debug, --enable-debug, I get the aforementioned message (here it is again):<o:p></o:p></span></p>
</div>
<div>
<p class="MsoNormal"><span style="font-family:"Lucida Console"">prov/verbs/src/ep_rdm/verbs_rdm_cm.c:337: fi_ibv_rdm_process_addr_resolved: Assertion `id->verbs == ep->domain->verbs' failed.<o:p></o:p></span></p>
</div>
<div>
<p class="MsoNormal"><span style="font-family:"Lucida Console"">and the mpi test program runs through in case 2 above. I am not sure whether or not I should take this message seriously?<o:p></o:p></span></p>
</div>
<div>
<p class="MsoNormal"><span style="font-family:"Lucida Console""><o:p> </o:p></span></p>
</div>
<div>
<p class="MsoNormal"><span style="font-family:"Lucida Console"">I did not see any difference in the test mpi program behaviour if I build ibverbs as a DSO (--enable-verbs=dl) or as the default which I suppose would be part of libfabric (--enable-verbs=yes)
except in case of DSO, the FI_PROVIDER_PATH must be exported. However, worth mentioning as a bug (probably), when ibverbs (or any other provider I assume) is built as a DSO, the libfabric folder under which the provider DSOs are put has wrong permissions,
which means if you build libfabric as a root and use default installation folders (/usr/local/lib), your mpi program would not run through if you launch it as some other user.<o:p></o:p></span></p>
</div>
<div>
<p class="MsoNormal"><span style="font-family:"Lucida Console""><o:p> </o:p></span></p>
</div>
<div>
<p class="MsoNormal"><span style="font-family:"Lucida Console""><o:p> </o:p></span></p>
</div>
<div>
<p class="MsoNormal"><span style="font-family:"Lucida Console"">Regards,<o:p></o:p></span></p>
</div>
<div>
<p class="MsoNormal"><span style="font-family:"Lucida Console"">Mohammed<o:p></o:p></span></p>
</div>
<div>
<p class="MsoNormal"><span style="font-family:"Lucida Console""><o:p> </o:p></span></p>
</div>
</div>
<div id="yahoo_quoted_3340555604">
<div>
<div>
<p class="MsoNormal"><span style="font-size:10.0pt;font-family:"Helvetica",sans-serif;color:#26282A">Am Mittwoch, 21.
</span><span lang="EN-US" style="font-size:10.0pt;font-family:"Helvetica",sans-serif;color:#26282A">November 2018, 19:42:24 MEZ hat Ilango, Arun <</span><span style="font-size:10.0pt;font-family:"Helvetica",sans-serif;color:#26282A"><a href="mailto:arun.ilango@intel.com"><span lang="EN-US">arun.ilango@intel.com</span></a></span><span lang="EN-US" style="font-size:10.0pt;font-family:"Helvetica",sans-serif;color:#26282A">>
Folgendes geschrieben: <o:p></o:p></span></p>
</div>
<div>
<p class="MsoNormal"><span lang="EN-US" style="font-size:10.0pt;font-family:"Helvetica",sans-serif;color:#26282A"><o:p> </o:p></span></p>
</div>
<div>
<p class="MsoNormal"><span lang="EN-US" style="font-size:10.0pt;font-family:"Helvetica",sans-serif;color:#26282A"><o:p> </o:p></span></p>
</div>
<div>
<div>
<p class="MsoNormal"><span style="font-size:10.0pt;font-family:"Helvetica",sans-serif;color:#26282A">Mohammed,<br>
<br>
Just to add what Dmitry said, if you're using your own libfabric, please make sure it's the latest (i.e. v1.6.2). You can check the version by running fi_info --version.<br>
<br>
Other things to check:<br>
1. Make sure you have librdmacm package installed.<br>
2. Check if the IPoIB interface of the node has been configured with an IP address and is pingable from other nodes in the cluster.<br>
<br>
Thanks,<br>
Arun.<o:p></o:p></span></p>
<div id="yqtfd54283">
<p class="MsoNormal" style="margin-bottom:12.0pt"><span style="font-size:10.0pt;font-family:"Helvetica",sans-serif;color:#26282A"><br>
-----Original Message-----<br>
From: Gladkov, Dmitry <br>
Sent: Wednesday, November 21, 2018 10:31 AM<br>
To: Hefty, Sean <<a href="mailto:sean.hefty@intel.com">sean.hefty@intel.com</a>>; Mohammed Shaheen <<a href="mailto:m_shaheen1984@yahoo.com">m_shaheen1984@yahoo.com</a>>;
<a href="mailto:libfabric-users@lists.openfabrics.org">libfabric-users@lists.openfabrics.org</a>;
<a href="mailto:ofiwg@lists.openfabrics.org">ofiwg@lists.openfabrics.org</a><br>
Cc: Ilango, Arun <<a href="mailto:arun.ilango@intel.com">arun.ilango@intel.com</a>><br>
Subject: RE: [libfabric-users] intel mpi with libfabric<br>
<br>
Hi Mohammed,<br>
<br>
Do you use your own version of libfabirc?<br>
<br>
IMPI 2019 U1 uses its internal libfabric by default.<br>
If you use your libfabric, please, specify LD_LIBRABRY_PATH to your library and FI_PROVIDER_PATH to path to OFI DL providers (<ofi_install_dir>/lib/libfabric) if you use DL provider, or unset this variable (mpivars.sh sets it).<br>
<br>
--<br>
Dmitry<br>
<br>
-----Original Message-----<br>
From: Hefty, Sean<br>
Sent: Wednesday, November 21, 2018 8:52 PM<br>
To: Mohammed Shaheen <<a href="mailto:m_shaheen1984@yahoo.com">m_shaheen1984@yahoo.com</a>>;
<a href="mailto:libfabric-users@lists.openfabrics.org">libfabric-users@lists.openfabrics.org</a>;
<a href="mailto:ofiwg@lists.openfabrics.org">ofiwg@lists.openfabrics.org</a><br>
Cc: Ilango, Arun <<a href="mailto:arun.ilango@intel.com">arun.ilango@intel.com</a>>; Gladkov, Dmitry <<a href="mailto:dmitry.gladkov@intel.com">dmitry.gladkov@intel.com</a>><br>
Subject: RE: [libfabric-users] intel mpi with libfabric<br>
<br>
Copying ofiwg and key developers for this issue.<br>
<br>
- Sean<br>
<br>
> I get the following error running a small mpi test program using intel <br>
> mpi 2019 from intel parallel studio cluster edition update 1 (the<br>
> newest) on Mellanox FDR Cluster:<br>
> <br>
> <br>
> <br>
> test.e: prov/verbs/src/ep_rdm/verbs_rdm_cm.c:337:<br>
> fi_ibv_rdm_process_addr_resolved: Assertion `id->verbs == ep->domain-<br>
> >verbs' failed.<br>
> <br>
> <br>
> <br>
> The program hangs on this error message. I installed the newest <br>
> release of libfabric and configured it with only ibverbs support. I <br>
> used the inbox (sles 11 sp4 and sles 12 sp3) ibverbs and rdma <br>
> libraries. I also tried with mellanox ofed to no avail.<br>
> <br>
> <br>
> <br>
> <br>
> Any ideas how to go about it?<br>
> <br>
> <br>
> <br>
> <br>
> <br>
> Regards,<br>
> <br>
> Mohammed<o:p></o:p></span></p>
</div>
</div>
</div>
</div>
</div>
</div>
<p><br>--------------------------------------------------------------------<br>Joint Stock Company Intel A/O<br>Registered legal address: Krylatsky Hills Business Park, <br>17 Krylatskaya Str., Bldg 4, Moscow 121614, <br>Russian Federation</p><p>This e-mail and any attachments may contain confidential material for<br>the sole use of the intended recipient(s). Any review or distribution<br>by others is strictly prohibited. If you are not the intended<br>recipient, please contact the sender and delete all copies.</p>
</body>
</html>