<html><head></head><body><div class="ydpb7b0de12yahoo-style-wrap" style="font-family:lucida console, sans-serif;font-size:16px;"><div></div>
<div>Hi Dmitry,</div><div><br></div><div>I have tried with the development version from the master branch. I get the following errors while building the library (make)</div><div><span>prov/verbs/src/verbs_ep.c: In function 'fi_ibv_msg_xrc_ep_atomic_write':<br>prov/verbs/src/verbs_ep.c:1770: error: unknown field 'qp_type' specified in initializer<br>prov/verbs/src/verbs_ep.c: In function 'fi_ibv_msg_xrc_ep_atomic_writemsg':<br>prov/verbs/src/verbs_ep.c:1803: error: unknown field 'qp_type' specified in initializer<br>prov/verbs/src/verbs_ep.c: In function 'fi_ibv_msg_xrc_ep_atomic_readwrite':<br>prov/verbs/src/verbs_ep.c:1843: error: unknown field 'qp_type' specified in initializer<br>prov/verbs/src/verbs_ep.c: In function 'fi_ibv_msg_xrc_ep_atomic_readwritemsg':<br>prov/verbs/src/verbs_ep.c:1891: error: unknown field 'qp_type' specified in initializer<br>prov/verbs/src/verbs_ep.c: In function 'fi_ibv_msg_xrc_ep_atomic_compwrite':<br>prov/verbs/src/verbs_ep.c:1949: error: unknown field 'qp_type' specified in initializer<br>prov/verbs/src/verbs_ep.c: In function 'fi_ibv_msg_xrc_ep_atomic_compwritemsg':<br>prov/verbs/src/verbs_ep.c:1987: error: unknown field 'qp_type' specified in initializer<br>make[1]: *** [prov/verbs/src/src_libfabric_la-verbs_ep.lo] Error 1<br><br></span><div><br></div><div>I have the following ibverbs and rdma packages installed</div><div><span>libibverbs-1.1.4-0.16.37<br>libibverbs-32bit-1.1.4-0.16.37<br>libibverbs-devel-1.1.4-0.16.37<br>libibverbs-devel-32bit-1.1.4-0.16.37<br><br></span><div><span>librdmacm-devel-1.0.15-12.29<br>libmlx4-rdmav2-devel-1.0.1-15.30<br>libmthca-rdmav2-1.0.5-5.27.1<br>libnes-rdmav2-devel-1.1.3-0.11.1<br>libmthca-rdmav2-devel-1.0.5-5.27.1<br>librdmacm-1.0.15-12.29<br>libnes-rdmav2-1.1.3-0.11.1<br>libcxgb3-rdmav2-devel-1.2.5-0.8.1<br>libmlx4-rdmav2-1.0.1-15.30<br>libcxgb3-rdmav2-1.2.5-0.8.1<br><br></span><div><br></div><div>I think they are a bit old? I tried installing MLNX OFED but did not help either. Iget the following while building with MLNX OFED installed:</div><div><br></div><div><span>prov/rstream/src/rstream_ep.c: In function 'rstream_ep_open':<br>prov/rstream/src/rstream.h:145: warning: array subscript is below array bounds<br>prov/rstream/src/rstream_msg.c: In function 'rstream_process_cq':<br>prov/rstream/src/rstream_msg.c:138: warning: array subscript is below array bounds<br>prov/verbs/src/verbs_ep.c: In function 'fi_ibv_msg_xrc_ep_atomic_write':<br>prov/verbs/src/verbs_ep.c:1770: error: unknown field 'qp_type' specified in initializer<br>prov/verbs/src/verbs_ep.c: In function 'fi_ibv_msg_xrc_ep_atomic_writemsg':<br>prov/verbs/src/verbs_ep.c:1803: error: unknown field 'qp_type' specified in initializer<br>prov/verbs/src/verbs_ep.c: In function 'fi_ibv_msg_xrc_ep_atomic_readwrite':<br>prov/verbs/src/verbs_ep.c:1843: error: unknown field 'qp_type' specified in initializer<br>prov/verbs/src/verbs_ep.c: In function 'fi_ibv_msg_xrc_ep_atomic_readwritemsg':<br>prov/verbs/src/verbs_ep.c:1891: error: unknown field 'qp_type' specified in initializer<br>prov/verbs/src/verbs_ep.c: In function 'fi_ibv_msg_xrc_ep_atomic_compwrite':<br>prov/verbs/src/verbs_ep.c:1949: error: unknown field 'qp_type' specified in initializer<br>prov/verbs/src/verbs_ep.c: In function 'fi_ibv_msg_xrc_ep_atomic_compwritemsg':<br>prov/verbs/src/verbs_ep.c:1987: error: unknown field 'qp_type' specified in initializer<br>prov/verbs/src/verbs_msg.c: In function 'fi_ibv_msg_xrc_ep_sendmsg':<br>prov/verbs/src/verbs_msg.c:262: error: unknown field 'qp_type' specified in initializer<br>prov/verbs/src/verbs_msg.c:262: warning: initialization makes pointer from integer without a cast<br>prov/verbs/src/verbs_msg.c: In function 'fi_ibv_msg_xrc_ep_send':<br>prov/verbs/src/verbs_msg.c:285: error: unknown field 'qp_type' specified in initializer<br>prov/verbs/src/verbs_msg.c: In function 'fi_ibv_msg_xrc_ep_senddata':prov/verbs/src/verbs_rma.c: In function 'fi_ibv_msg_xrc_ep_rma_write':<br><br></span><br></div></div><div><br></div><div>Anything else I can do to go about these issues?</div><div><br></div><div>Regards,</div><div>Mohammed<br></div></div><div><br></div></div><div><br></div>
</div><div id="yahoo_quoted_4167743258" class="yahoo_quoted">
<div style="font-family:'Helvetica Neue', Helvetica, Arial, sans-serif;font-size:13px;color:#26282a;">
<div>
Am Donnerstag, 22. November 2018, 15:55:51 MEZ hat Mohammed Shaheen via Libfabric-users <libfabric-users@lists.openfabrics.org> Folgendes geschrieben:
</div>
<div><br></div>
<div><br></div>
<div><div id="yiv8884742860"><div><div class="yiv8884742860ydp4786efceyahoo-style-wrap" style="font-family:lucida console, sans-serif;font-size:16px;"><div></div>
<div>Hi Dmitry,</div><div><br clear="none"></div><div>I have only tried with the latest release 1.6.2. Ill check out the master branch 1.7 from github and let you know..</div><div><br clear="none"></div><div>Regards,</div><div>Mohammed <br clear="none"></div><div><br clear="none"></div>
</div><div class="yiv8884742860yahoo_quoted" id="yiv8884742860yahoo_quoted_3506775177">
<div style="font-family:'Helvetica Neue', Helvetica, Arial, sans-serif;font-size:13px;color:#26282a;">
<div>
Am Donnerstag, 22. November 2018, 15:24:53 MEZ hat Gladkov, Dmitry <dmitry.gladkov@intel.com> Folgendes geschrieben:
</div>
<div><br clear="none"></div>
<div><br clear="none"></div>
<div class="yiv8884742860yqt8709380944" id="yiv8884742860yqt02876"><div><div id="yiv8884742860"><style>#yiv8884742860 --
filtered {font-family:Helvetica;panose-1:2 11 6 4 2 2 2 2 2 4;}
#yiv8884742860 filtered {panose-1:2 4 5 3 5 4 6 3 2 4;}
#yiv8884742860 filtered {font-family:Calibri;panose-1:2 15 5 2 2 2 4 3 2 4;}
#yiv8884742860 filtered {panose-1:2 11 6 9 4 5 4 2 2 4;}
#yiv8884742860
p.yiv8884742860MsoNormal, #yiv8884742860 li.yiv8884742860MsoNormal, #yiv8884742860 div.yiv8884742860MsoNormal
{margin:0cm;margin-bottom:.0001pt;font-size:12.0pt;font-family:New serif;}
#yiv8884742860 a:link, #yiv8884742860 span.yiv8884742860MsoHyperlink
{color:blue;text-decoration:underline;}
#yiv8884742860 a:visited, #yiv8884742860 span.yiv8884742860MsoHyperlinkFollowed
{color:purple;text-decoration:underline;}
#yiv8884742860 span.yiv8884742860EmailStyle17
{font-family:sans-serif;color:#1F497D;}
#yiv8884742860 .yiv8884742860MsoChpDefault
{font-size:10.0pt;}
#yiv8884742860 filtered {margin:72.0pt 72.0pt 72.0pt 72.0pt;}
#yiv8884742860 div.yiv8884742860WordSection1
{}
#yiv8884742860 </style><div>
<div class="yiv8884742860WordSection1">
<p class="yiv8884742860MsoNormal"><span style="font-size:11.0pt;" lang="EN-US">Hi Mohammed,</span></p>
<p class="yiv8884742860MsoNormal"><span style="font-size:11.0pt;" lang="EN-US"> </span></p>
<p class="yiv8884742860MsoNormal"><span style="font-size:11.0pt;" lang="EN-US">I don’t recommend to verbs provider with FI_EP_RDM EP type. We deprecating this provider since libfabric 1.7.</span></p>
<p class="yiv8884742860MsoNormal"><span style="font-size:11.0pt;" lang="EN-US">Now our default provider is RxM/verbs (verbs with FI_EP_MSG EP type) to run libfabric on IB/iWARP/RoCE verbs devices.</span></p>
<p class="yiv8884742860MsoNormal"><span style="font-size:11.0pt;" lang="EN-US">If you use libfabric 1.6.x, RxM/verbs should be the default provider.</span></p>
<p class="yiv8884742860MsoNormal"><span style="font-size:11.0pt;" lang="EN-US"> </span></p>
<p class="yiv8884742860MsoNormal"><span style="font-size:11.0pt;" lang="EN-US">Can you see the hangs when using libfabric’s master branch?</span></p>
<p class="yiv8884742860MsoNormal"><span style="font-size:11.0pt;" lang="EN-US"> </span></p>
<p class="yiv8884742860MsoNormal"><span style="font-size:11.0pt;" lang="EN-US">--</span></p>
<p class="yiv8884742860MsoNormal"><span style="font-size:11.0pt;" lang="EN-US">Dmitry</span></p>
<p class="yiv8884742860MsoNormal"><a rel="nofollow" shape="rect" name="_MailEndCompose"><span style="" lang="EN-US"> </span></a></p>
<div class="yiv8884742860yqt0303214313" id="yiv8884742860yqt77904"><div>
<div style="border:none;border-top:solid #E1E1E1 1.0pt;padding:3.0pt 0cm 0cm 0cm;">
<p class="yiv8884742860MsoNormal"><a rel="nofollow" shape="rect" name="_____replyseparator"></a><b><span style="font-size:11.0pt;" lang="EN-US">From:</span></b><span style="font-size:11.0pt;" lang="EN-US"> Mohammed Shaheen [mailto:m_shaheen1984@yahoo.com]
<br clear="none">
<b>Sent:</b> Thursday, November 22, 2018 4:54 PM<br clear="none">
<b>To:</b> Gladkov, Dmitry <dmitry.gladkov@intel.com>; Hefty, Sean <sean.hefty@intel.com>; libfabric-users@lists.openfabrics.org; ofiwg@lists.openfabrics.org; Ilango, Arun <arun.ilango@intel.com><br clear="none">
<b>Subject:</b> Re: [libfabric-users] intel mpi with libfabric</span></p>
</div>
</div>
<p class="yiv8884742860MsoNormal"> </p>
<div>
<div>
<p class="yiv8884742860MsoNormal"><span style="">Thanks Arun and Dmitry for your support.</span></p>
</div>
<div>
<p class="yiv8884742860MsoNormal"><span style=""> </span></p>
</div>
<div>
<p class="yiv8884742860MsoNormal"><span style="">Well, I am building my own libfabric, and I export the right variables and source intel mpi with -ofi_internal=0. I figured out where the problem is:</span></p>
</div>
<div>
<p class="yiv8884742860MsoNormal"><span style="">1. If libfabric is built for all providers, i.e. run ./configure without including and exluding providers, it will build ibverbs among others; however, the mpi test program will hang during execution.
</span></p>
</div>
<div>
<p class="yiv8884742860MsoNormal"><span style="">2. If libfabric configured with only enabling ibverbs and setting all other providers, i.e. ./configure --enable-verbs=yes --enable-rxm=no --enable-rxd=no --enable-sockets=no --enable-tcp=no --enable-udp=no,
mpi test program will run through</span></p>
</div>
<div>
<p class="yiv8884742860MsoNormal"><span style=""> </span></p>
</div>
<div>
<p class="yiv8884742860MsoNormal"><span style="">Another observation when I enable debug, --enable-debug, I get the aforementioned message (here it is again):</span></p>
</div>
<div>
<p class="yiv8884742860MsoNormal"><span style="">prov/verbs/src/ep_rdm/verbs_rdm_cm.c:337: fi_ibv_rdm_process_addr_resolved: Assertion `id->verbs == ep->domain->verbs' failed.</span></p>
</div>
<div>
<p class="yiv8884742860MsoNormal"><span style="">and the mpi test program runs through in case 2 above. I am not sure whether or not I should take this message seriously?</span></p>
</div>
<div>
<p class="yiv8884742860MsoNormal"><span style=""> </span></p>
</div>
<div>
<p class="yiv8884742860MsoNormal"><span style="">I did not see any difference in the test mpi program behaviour if I build ibverbs as a DSO (--enable-verbs=dl) or as the default which I suppose would be part of libfabric (--enable-verbs=yes)
except in case of DSO, the FI_PROVIDER_PATH must be exported. However, worth mentioning as a bug (probably), when ibverbs (or any other provider I assume) is built as a DSO, the libfabric folder under which the provider DSOs are put has wrong permissions,
which means if you build libfabric as a root and use default installation folders (/usr/local/lib), your mpi program would not run through if you launch it as some other user.</span></p>
</div>
<div>
<p class="yiv8884742860MsoNormal"><span style=""> </span></p>
</div>
<div>
<p class="yiv8884742860MsoNormal"><span style=""> </span></p>
</div>
<div>
<p class="yiv8884742860MsoNormal"><span style="">Regards,</span></p>
</div>
<div>
<p class="yiv8884742860MsoNormal"><span style="">Mohammed</span></p>
</div>
<div>
<p class="yiv8884742860MsoNormal"><span style=""> </span></p>
</div>
</div>
<div id="yiv8884742860yahoo_quoted_3340555604">
<div>
<div>
<p class="yiv8884742860MsoNormal"><span style="font-size:10.0pt;">Am Mittwoch, 21.
</span><span style="font-size:10.0pt;" lang="EN-US">November 2018, 19:42:24 MEZ hat Ilango, Arun <</span><span style="font-size:10.0pt;"><a rel="nofollow" shape="rect" ymailto="mailto:arun.ilango@intel.com" target="_blank" href="mailto:arun.ilango@intel.com"><span lang="EN-US">arun.ilango@intel.com</span></a></span><span style="font-size:10.0pt;" lang="EN-US">>
Folgendes geschrieben: </span></p>
</div>
<div>
<p class="yiv8884742860MsoNormal"><span style="font-size:10.0pt;" lang="EN-US"> </span></p>
</div>
<div>
<p class="yiv8884742860MsoNormal"><span style="font-size:10.0pt;" lang="EN-US"> </span></p>
</div>
<div>
<div>
<p class="yiv8884742860MsoNormal"><span style="font-size:10.0pt;">Mohammed,<br clear="none">
<br clear="none">
Just to add what Dmitry said, if you're using your own libfabric, please make sure it's the latest (i.e. v1.6.2). You can check the version by running fi_info --version.<br clear="none">
<br clear="none">
Other things to check:<br clear="none">
1. Make sure you have librdmacm package installed.<br clear="none">
2. Check if the IPoIB interface of the node has been configured with an IP address and is pingable from other nodes in the cluster.<br clear="none">
<br clear="none">
Thanks,<br clear="none">
Arun.</span></p>
<div id="yiv8884742860yqtfd54283">
<p class="yiv8884742860MsoNormal" style="margin-bottom:12.0pt;"><span style="font-size:10.0pt;"><br clear="none">
-----Original Message-----<br clear="none">
From: Gladkov, Dmitry <br clear="none">
Sent: Wednesday, November 21, 2018 10:31 AM<br clear="none">
To: Hefty, Sean <<a rel="nofollow" shape="rect" ymailto="mailto:sean.hefty@intel.com" target="_blank" href="mailto:sean.hefty@intel.com">sean.hefty@intel.com</a>>; Mohammed Shaheen <<a rel="nofollow" shape="rect" ymailto="mailto:m_shaheen1984@yahoo.com" target="_blank" href="mailto:m_shaheen1984@yahoo.com">m_shaheen1984@yahoo.com</a>>;
<a rel="nofollow" shape="rect" ymailto="mailto:libfabric-users@lists.openfabrics.org" target="_blank" href="mailto:libfabric-users@lists.openfabrics.org">libfabric-users@lists.openfabrics.org</a>;
<a rel="nofollow" shape="rect" ymailto="mailto:ofiwg@lists.openfabrics.org" target="_blank" href="mailto:ofiwg@lists.openfabrics.org">ofiwg@lists.openfabrics.org</a><br clear="none">
Cc: Ilango, Arun <<a rel="nofollow" shape="rect" ymailto="mailto:arun.ilango@intel.com" target="_blank" href="mailto:arun.ilango@intel.com">arun.ilango@intel.com</a>><br clear="none">
Subject: RE: [libfabric-users] intel mpi with libfabric<br clear="none">
<br clear="none">
Hi Mohammed,<br clear="none">
<br clear="none">
Do you use your own version of libfabirc?<br clear="none">
<br clear="none">
IMPI 2019 U1 uses its internal libfabric by default.<br clear="none">
If you use your libfabric, please, specify LD_LIBRABRY_PATH to your library and FI_PROVIDER_PATH to path to OFI DL providers (<ofi_install_dir>/lib/libfabric) if you use DL provider, or unset this variable (mpivars.sh sets it).<br clear="none">
<br clear="none">
--<br clear="none">
Dmitry<br clear="none">
<br clear="none">
-----Original Message-----<br clear="none">
From: Hefty, Sean<br clear="none">
Sent: Wednesday, November 21, 2018 8:52 PM<br clear="none">
To: Mohammed Shaheen <<a rel="nofollow" shape="rect" ymailto="mailto:m_shaheen1984@yahoo.com" target="_blank" href="mailto:m_shaheen1984@yahoo.com">m_shaheen1984@yahoo.com</a>>;
<a rel="nofollow" shape="rect" ymailto="mailto:libfabric-users@lists.openfabrics.org" target="_blank" href="mailto:libfabric-users@lists.openfabrics.org">libfabric-users@lists.openfabrics.org</a>;
<a rel="nofollow" shape="rect" ymailto="mailto:ofiwg@lists.openfabrics.org" target="_blank" href="mailto:ofiwg@lists.openfabrics.org">ofiwg@lists.openfabrics.org</a><br clear="none">
Cc: Ilango, Arun <<a rel="nofollow" shape="rect" ymailto="mailto:arun.ilango@intel.com" target="_blank" href="mailto:arun.ilango@intel.com">arun.ilango@intel.com</a>>; Gladkov, Dmitry <<a rel="nofollow" shape="rect" ymailto="mailto:dmitry.gladkov@intel.com" target="_blank" href="mailto:dmitry.gladkov@intel.com">dmitry.gladkov@intel.com</a>><br clear="none">
Subject: RE: [libfabric-users] intel mpi with libfabric<br clear="none">
<br clear="none">
Copying ofiwg and key developers for this issue.<br clear="none">
<br clear="none">
- Sean<br clear="none">
<br clear="none">
> I get the following error running a small mpi test program using intel <br clear="none">
> mpi 2019 from intel parallel studio cluster edition update 1 (the<br clear="none">
> newest) on Mellanox FDR Cluster:<br clear="none">
> <br clear="none">
> <br clear="none">
> <br clear="none">
> test.e: prov/verbs/src/ep_rdm/verbs_rdm_cm.c:337:<br clear="none">
> fi_ibv_rdm_process_addr_resolved: Assertion `id->verbs == ep->domain-<br clear="none">
> >verbs' failed.<br clear="none">
> <br clear="none">
> <br clear="none">
> <br clear="none">
> The program hangs on this error message. I installed the newest <br clear="none">
> release of libfabric and configured it with only ibverbs support. I <br clear="none">
> used the inbox (sles 11 sp4 and sles 12 sp3) ibverbs and rdma <br clear="none">
> libraries. I also tried with mellanox ofed to no avail.<br clear="none">
> <br clear="none">
> <br clear="none">
> <br clear="none">
> <br clear="none">
> Any ideas how to go about it?<br clear="none">
> <br clear="none">
> <br clear="none">
> <br clear="none">
> <br clear="none">
> <br clear="none">
> Regards,<br clear="none">
> <br clear="none">
> Mohammed</span></p>
</div>
</div>
</div>
</div>
</div></div>
</div>
<p><br clear="none">--------------------------------------------------------------------<br clear="none">Joint Stock Company Intel A/O<br clear="none">Registered legal address: Krylatsky Hills Business Park, <br clear="none">17 Krylatskaya Str., Bldg 4, Moscow 121614, <br clear="none">Russian Federation</p><p>This e-mail and any attachments may contain confidential material for<br clear="none">the sole use of the intended recipient(s). Any review or distribution<br clear="none">by others is strictly prohibited. If you are not the intended<br clear="none">recipient, please contact the sender and delete all copies.</p>
</div></div></div></div>
</div>
</div></div></div><div class="yqt8709380944" id="yqt44554">_______________________________________________<br clear="none">Libfabric-users mailing list<br clear="none"><a shape="rect" ymailto="mailto:Libfabric-users@lists.openfabrics.org" href="mailto:Libfabric-users@lists.openfabrics.org">Libfabric-users@lists.openfabrics.org</a><br clear="none"><a shape="rect" href="https://lists.openfabrics.org/mailman/listinfo/libfabric-users" target="_blank">https://lists.openfabrics.org/mailman/listinfo/libfabric-users</a><br clear="none"></div></div>
</div>
</div></body></html>