<html xmlns:v="urn:schemas-microsoft-com:vml" xmlns:o="urn:schemas-microsoft-com:office:office" xmlns:w="urn:schemas-microsoft-com:office:word" xmlns:m="http://schemas.microsoft.com/office/2004/12/omml" xmlns="http://www.w3.org/TR/REC-html40">
<head>
<meta http-equiv="Content-Type" content="text/html; charset=us-ascii">
<meta name="Generator" content="Microsoft Word 15 (filtered medium)">
<style><!--
/* Font Definitions */
@font-face
{font-family:"Cambria Math";
panose-1:2 4 5 3 5 4 6 3 2 4;}
@font-face
{font-family:Calibri;
panose-1:2 15 5 2 2 2 4 3 2 4;}
/* Style Definitions */
p.MsoNormal, li.MsoNormal, div.MsoNormal
{margin:0in;
margin-bottom:.0001pt;
font-size:11.0pt;
font-family:"Calibri",sans-serif;}
a:link, span.MsoHyperlink
{mso-style-priority:99;
color:blue;
text-decoration:underline;}
a:visited, span.MsoHyperlinkFollowed
{mso-style-priority:99;
color:purple;
text-decoration:underline;}
p.msonormal0, li.msonormal0, div.msonormal0
{mso-style-name:msonormal;
mso-margin-top-alt:auto;
margin-right:0in;
mso-margin-bottom-alt:auto;
margin-left:0in;
font-size:11.0pt;
font-family:"Calibri",sans-serif;}
span.EmailStyle18
{mso-style-type:personal;
font-family:"Calibri",sans-serif;
color:windowtext;}
span.EmailStyle19
{mso-style-type:personal;
font-family:"Calibri",sans-serif;
color:windowtext;}
span.EmailStyle20
{mso-style-type:personal;
font-family:"Calibri",sans-serif;
color:#1F497D;}
span.EmailStyle21
{mso-style-type:personal;
font-family:"Calibri",sans-serif;
color:windowtext;}
span.EmailStyle22
{mso-style-type:personal-reply;
font-family:"Calibri",sans-serif;
color:#1F497D;}
.MsoChpDefault
{mso-style-type:export-only;
font-size:10.0pt;}
@page WordSection1
{size:8.5in 11.0in;
margin:1.0in 1.0in 1.0in 1.0in;}
div.WordSection1
{page:WordSection1;}
--></style><!--[if gte mso 9]><xml>
<o:shapedefaults v:ext="edit" spidmax="1026" />
</xml><![endif]--><!--[if gte mso 9]><xml>
<o:shapelayout v:ext="edit">
<o:idmap v:ext="edit" data="1" />
</o:shapelayout></xml><![endif]-->
</head>
<body lang="EN-US" link="blue" vlink="purple">
<div class="WordSection1">
<p class="MsoNormal"><span style="color:#1F497D">Hi John,<o:p></o:p></span></p>
<p class="MsoNormal"><span style="color:#1F497D"><o:p> </o:p></span></p>
<p class="MsoNormal"><span style="color:#1F497D">Please let us know if you see the hang with “tcp;ofi_rxm” as well.
<o:p></o:p></span></p>
<p class="MsoNormal"><span style="color:#1F497D"><o:p> </o:p></span></p>
<p class="MsoNormal"><a name="_MailEndCompose"><span style="color:#1F497D">Yes, the sockets provider is feature complete but the plan is to replace it with “tcp;ofi_rxm” or “udp;ofi_rxd” stack in the future once they have all features. Also, future performance
improvements and feature additions would target the above two stacks. I believe issues in the sockets provider are fixed on a severity basis but I’ll let Sean comment on that.<o:p></o:p></span></a></p>
<p class="MsoNormal"><span style="color:#1F497D"><o:p> </o:p></span></p>
<p class="MsoNormal">> What would you recommend as an example implementation<o:p></o:p></p>
<p class="MsoNormal">> for someone who wants to understand how to develop a provider?<o:p></o:p></p>
<p class="MsoNormal"><span style="color:#1F497D"><o:p> </o:p></span></p>
<p class="MsoNormal"><span style="color:#1F497D">What feature set are you looking to implement in your provider? Depending on that, you can look at one of existing providers for example: the tcp provider supports a FI_EP_MSG endpoint type and the udp provider
supports FI_EP_DGRAM endpoint type over sockets. They make use of common utility code available in prov/util/* to implement many libfabric features. ofi_rxm and ofi_rxd “utility providers” build FI_EP_RDM endpoint support on top of any provider that implements
FI_EP_MSG and FI_EP_DGRAM respectively.<o:p></o:p></span></p>
<p class="MsoNormal"><span style="color:#1F497D"><o:p> </o:p></span></p>
<p class="MsoNormal"><span style="color:#1F497D">There’s also a developer guide here:
</span><a href="https://github.com/ofiwg/ofi-guide/blob/master/OFIGuide.md">https://github.com/ofiwg/ofi-guide/blob/master/OFIGuide.md</a><span style="color:#1F497D"><o:p></o:p></span></p>
<p class="MsoNormal"><span style="color:#1F497D"><o:p> </o:p></span></p>
<p class="MsoNormal"><span style="color:#1F497D">Thanks,<o:p></o:p></span></p>
<p class="MsoNormal"><span style="color:#1F497D">Arun.<o:p></o:p></span></p>
<p class="MsoNormal"><span style="color:#1F497D"><o:p> </o:p></span></p>
<div style="border:none;border-left:solid blue 1.5pt;padding:0in 0in 0in 4.0pt">
<div>
<div style="border:none;border-top:solid #E1E1E1 1.0pt;padding:3.0pt 0in 0in 0in">
<p class="MsoNormal"><a name="_____replyseparator"></a><b>From:</b> Wilkes, John [mailto:John.Wilkes@amd.com]
<br>
<b>Sent:</b> Thursday, March 21, 2019 2:01 PM<br>
<b>To:</b> Ilango, Arun <arun.ilango@intel.com>; libfabric-users@lists.openfabrics.org<br>
<b>Cc:</b> Hefty, Sean <sean.hefty@intel.com>; Franques, Antonio <Antonio.Franques@amd.com>; Wilkes, John <John.Wilkes@amd.com><br>
<b>Subject:</b> RE: MPI_Barrier hang with sockets provider<o:p></o:p></p>
</div>
</div>
<p class="MsoNormal"><o:p> </o:p></p>
<p class="MsoNormal">Hi Arun,<o:p></o:p></p>
<p class="MsoNormal"><o:p> </o:p></p>
<p class="MsoNormal">Thank you! This was driving me crazy! I’d started out with ompi-3.0 and libfabric-1.3 running on a 4-node cluster of QEMU VMs and had no problems, but when I moved to real hardware, it hung. So I tried the latest ompi and libfabric and
had the same problem. OpenMPI-4.0.0 and libfabric-1.7.0 also hangs on the QEMU virutal cluster.<o:p></o:p></p>
<p class="MsoNormal"><o:p> </o:p></p>
<p class="MsoNormal">My observation is that libfabric is very complex with many options, almost like an operating system all by itself. I have found lots of great documentation about libfabric internals and theory of operation, but it’s difficult for the unitiated
to get started.<o:p></o:p></p>
<p class="MsoNormal"><o:p> </o:p></p>
<p class="MsoNormal">I was under the impression that the sockets provider implemented most or all of libfabric’s feature set and therefore is a good reference and starting point. Maybe that was true in the past, but these days not so much?<o:p></o:p></p>
<p class="MsoNormal"><o:p> </o:p></p>
<p class="MsoNormal">What would you recommend as an example implementation for someone who wants to understand how to develop a provider?<o:p></o:p></p>
<p class="MsoNormal"><o:p> </o:p></p>
<p class="MsoNormal">Thanks again for your help!<o:p></o:p></p>
<p class="MsoNormal"><o:p> </o:p></p>
<div>
<p class="MsoNormal">John<o:p></o:p></p>
</div>
<p class="MsoNormal"><o:p> </o:p></p>
<div>
<div style="border:none;border-top:solid #E1E1E1 1.0pt;padding:3.0pt 0in 0in 0in">
<p class="MsoNormal"><b>From:</b> Ilango, Arun <<a href="mailto:arun.ilango@intel.com">arun.ilango@intel.com</a>>
<br>
<b>Sent:</b> Thursday, March 21, 2019 11:23 AM<br>
<b>To:</b> Wilkes, John <<a href="mailto:John.Wilkes@amd.com">John.Wilkes@amd.com</a>>;
<a href="mailto:libfabric-users@lists.openfabrics.org">libfabric-users@lists.openfabrics.org</a><br>
<b>Cc:</b> Hefty, Sean <<a href="mailto:sean.hefty@intel.com">sean.hefty@intel.com</a>><br>
<b>Subject:</b> RE: MPI_Barrier hang with sockets provider<o:p></o:p></p>
</div>
</div>
<p class="MsoNormal"><o:p> </o:p></p>
<p class="MsoNormal"><span style="color:#1F497D">Hi John,<o:p></o:p></span></p>
<p class="MsoNormal"><span style="color:#1F497D"><o:p> </o:p></span></p>
<p class="MsoNormal"><span style="color:#1F497D">I’m not sure what’s going wrong with sockets provider but can you try your test with “tcp;ofi_rxm” provider stack? Sockets provider is currently in maintenance mode and “tcp;ofi_rxm” is preferred for running
apps over TCP sockets. Please let me know if you face any issues.<o:p></o:p></span></p>
<p class="MsoNormal"><span style="color:#1F497D"><o:p> </o:p></span></p>
<p class="MsoNormal"><span style="color:#1F497D">Your configure and mpirun command line seems fine to me.<o:p></o:p></span></p>
<p class="MsoNormal"><span style="color:#1F497D"><o:p> </o:p></span></p>
<p class="MsoNormal">> Is there a basic “how to” for configuring and running MPI with<o:p></o:p></p>
<p class="MsoNormal">> the libfabric sockets provider? Getting started with libfabric is not easy!<o:p></o:p></p>
<p class="MsoNormal"><span style="color:#1F497D"><o:p> </o:p></span></p>
<p class="MsoNormal"><span style="color:#1F497D">If it isn’t available already, adding a readme for running MPI with libfabric would be a good idea. Let me check. Are there any other issues you see when getting started with libfabric? There are various resources
available in the Readme on libfabric GitHub page (<a href="https://github.com/ofiwg/libfabric">https://github.com/ofiwg/libfabric</a>).<o:p></o:p></span></p>
<p class="MsoNormal"><span style="color:#1F497D"><o:p> </o:p></span></p>
<p class="MsoNormal"><span style="color:#1F497D">Thanks,<o:p></o:p></span></p>
<p class="MsoNormal"><span style="color:#1F497D">Arun.<o:p></o:p></span></p>
<p class="MsoNormal"><span style="color:#1F497D"><o:p> </o:p></span></p>
<div style="border:none;border-left:solid blue 1.5pt;padding:0in 0in 0in 4.0pt">
<div>
<div style="border:none;border-top:solid #E1E1E1 1.0pt;padding:3.0pt 0in 0in 0in">
<p class="MsoNormal"><b>From:</b> Libfabric-users [<a href="mailto:libfabric-users-bounces@lists.openfabrics.org">mailto:libfabric-users-bounces@lists.openfabrics.org</a>]
<b>On Behalf Of </b>Wilkes, John<br>
<b>Sent:</b> Thursday, March 21, 2019 7:03 AM<br>
<b>To:</b> <a href="mailto:libfabric-users@lists.openfabrics.org">libfabric-users@lists.openfabrics.org</a><br>
<b>Subject:</b> Re: [libfabric-users] MPI_Barrier hang with sockets provider<o:p></o:p></p>
</div>
</div>
<p class="MsoNormal"><o:p> </o:p></p>
<p class="MsoNormal">I have hardly any experience with libfabric, and I would greatly appreciate some suggestions!<o:p></o:p></p>
<p class="MsoNormal"><o:p> </o:p></p>
<p class="MsoNormal">Am I not configuring libfabric properly? Am I missing something on the mpirun command line?<o:p></o:p></p>
<p class="MsoNormal"><o:p> </o:p></p>
<p class="MsoNormal">Is there a basic “how to” for configuring and running MPI with the libfabric sockets provider? Getting started with libfabric is not easy!<o:p></o:p></p>
<p class="MsoNormal"><o:p> </o:p></p>
<div>
<p class="MsoNormal">John<o:p></o:p></p>
</div>
<p class="MsoNormal"><o:p> </o:p></p>
<div>
<div style="border:none;border-top:solid #E1E1E1 1.0pt;padding:3.0pt 0in 0in 0in">
<p class="MsoNormal"><b>From:</b> Wilkes, John <br>
<b>Sent:</b> Wednesday, March 20, 2019 8:16 AM<br>
<b>To:</b> <a href="mailto:libfabric-users@lists.openfabrics.org">libfabric-users@lists.openfabrics.org</a><br>
<b>Cc:</b> Wilkes, John <<a href="mailto:John.Wilkes@amd.com">John.Wilkes@amd.com</a>><br>
<b>Subject:</b> MPI_Barrier hang with sockets provider<o:p></o:p></p>
</div>
</div>
<p class="MsoNormal"><o:p> </o:p></p>
<p class="MsoNormal">When I run the XSBench proxy app on 4 nodes, it finishes successfully, but when I run it with the libfabric sockets provider, it hangs. After<o:p></o:p></p>
<p class="MsoNormal">the simulation is complete, there are calls to MPI_Barrier(), MPI_Reduce(), and MPI_Finalize().<o:p></o:p></p>
<p class="MsoNormal"><o:p> </o:p></p>
<p class="MsoNormal">command line:<o:p></o:p></p>
<p class="MsoNormal">$ mpirun -np 4 --map-by node --hostfile /nfs/mpi/etc/mpi-hostfile --mca mtl_ofi_provider_include sockets /nfs/software/proxy_apps/XSBench-14/src/XSBench -t 1 -s small<o:p></o:p></p>
<p class="MsoNormal"><o:p> </o:p></p>
<p class="MsoNormal">XSBench with the sockets provider runs to completion (does not hang) with -np 3.<o:p></o:p></p>
<p class="MsoNormal"><o:p> </o:p></p>
<p class="MsoNormal">OpenMPI-4.0.0<o:p></o:p></p>
<p class="MsoNormal">$ ./configure --prefix=/nfs/mpi --with-libfabric=/nfs/mpi --enable-orterun-prefix-by-default --disable-verbs-sshmem --without-verbs --enable-debug CFLAGS="-I/nfs/mpi/include -g -L/nfs/mpi/lib -ggdb -O0"<o:p></o:p></p>
<p class="MsoNormal"><o:p> </o:p></p>
<p class="MsoNormal">libfabric-1.7.0<o:p></o:p></p>
<p class="MsoNormal">$ ./configure --prefix=/nfs/mpi --enable-sockets=yes --enable-verbs=no --enable-debug=yes CFLAGS="-ggdb -O0"<o:p></o:p></p>
<p class="MsoNormal"><o:p> </o:p></p>
<p class="MsoNormal">A gdb stack trace on each node shows that node0 (where mpirun was run) is stuck in MPI_Reduce(). Node1 and node2 are in MPI_Finalize(), and node3 is in MPI_Barrier(). This is one example; the node that hangs in MPI_Barrier varies from
run to run.<o:p></o:p></p>
<p class="MsoNormal"><o:p> </o:p></p>
<p class="MsoNormal">Node3 stack trace:<o:p></o:p></p>
<p class="MsoNormal"><o:p> </o:p></p>
<p class="MsoNormal">#0 fi_gettime_ms<o:p></o:p></p>
<p class="MsoNormal">#1 sock_cq_sreadfrom<o:p></o:p></p>
<p class="MsoNormal">#2 sock_cq_readfrom<o:p></o:p></p>
<p class="MsoNormal">#3 sock_cq_read<o:p></o:p></p>
<p class="MsoNormal">#4 fi_cq_read<o:p></o:p></p>
<p class="MsoNormal">#5 ompi_mtl_ofi_progress<o:p></o:p></p>
<p class="MsoNormal">#6 ompi_mtl_ofi_progress_no_inline<o:p></o:p></p>
<p class="MsoNormal">#7 opal_progress<o:p></o:p></p>
<p class="MsoNormal">#8 ompi_request_wait_completion<o:p></o:p></p>
<p class="MsoNormal">#9 ompi_request_default_wait<o:p></o:p></p>
<p class="MsoNormal">#10 ompi_coll_base_sendrecv_zero<o:p></o:p></p>
<p class="MsoNormal">#11 ompi_coll_base_barrier_intra_recursivedoubling<o:p></o:p></p>
<p class="MsoNormal">#12 ompi_coll_tuned_barrier_intra_dec_fixed<o:p></o:p></p>
<p class="MsoNormal">#13 PMPI_Barrier<o:p></o:p></p>
<p class="MsoNormal">#14 print_results<o:p></o:p></p>
<p class="MsoNormal">#15 main<o:p></o:p></p>
<p class="MsoNormal"><o:p> </o:p></p>
<p class="MsoNormal">Node0 stack trace:<o:p></o:p></p>
<p class="MsoNormal"><o:p> </o:p></p>
<p class="MsoNormal">#0 ??<o:p></o:p></p>
<p class="MsoNormal">#1 gettimeofday<o:p></o:p></p>
<p class="MsoNormal">#2 fi_gettime_ms<o:p></o:p></p>
<p class="MsoNormal">#3 sock_cq_sreadfrom<o:p></o:p></p>
<p class="MsoNormal">#4 sock_cq_readfrom<o:p></o:p></p>
<p class="MsoNormal">#5 sock_cq_read<o:p></o:p></p>
<p class="MsoNormal">#6 fi_cq_read<o:p></o:p></p>
<p class="MsoNormal">#7 ompi_mtl_ofi_progress<o:p></o:p></p>
<p class="MsoNormal">#8 ompi_mtl_ofi_progress_no_inline<o:p></o:p></p>
<p class="MsoNormal">#9 opal_progress<o:p></o:p></p>
<p class="MsoNormal">#10 ompi_request_wait_completion<o:p></o:p></p>
<p class="MsoNormal">#11 mca_pml_cm_recv<o:p></o:p></p>
<p class="MsoNormal">#12 ompi_coll_base_reduce_intra_basic_linear<o:p></o:p></p>
<p class="MsoNormal">#13 ompi_coll_tuned_reduce_intra_dec_fixed<o:p></o:p></p>
<p class="MsoNormal">#14 PMPI_Reduce<o:p></o:p></p>
<p class="MsoNormal">#15 print_results<o:p></o:p></p>
<p class="MsoNormal">#16 main<o:p></o:p></p>
<p class="MsoNormal"><o:p> </o:p></p>
<p class="MsoNormal">Node1 stack trace:<o:p></o:p></p>
<p class="MsoNormal">#0 __GI___nanosleep<o:p></o:p></p>
<p class="MsoNormal">#1 usleep<o:p></o:p></p>
<p class="MsoNormal">#2 ompi_mpi_finalize<o:p></o:p></p>
<p class="MsoNormal">#3 PMPI_Finalize<o:p></o:p></p>
<p class="MsoNormal">#4 main<o:p></o:p></p>
<p class="MsoNormal"><o:p> </o:p></p>
<p class="MsoNormal">Node2 stack trace:<o:p></o:p></p>
<p class="MsoNormal">#0 __GI___nanosleep<o:p></o:p></p>
<p class="MsoNormal">#1 usleep<o:p></o:p></p>
<p class="MsoNormal">#2 ompi_mpi_finalize<o:p></o:p></p>
<p class="MsoNormal">#3 PMPI_Finalize<o:p></o:p></p>
<p class="MsoNormal">#4 main<o:p></o:p></p>
<p class="MsoNormal"><o:p> </o:p></p>
<p class="MsoNormal">-- <o:p></o:p></p>
<p class="MsoNormal">John Wilkes | <b><span style="color:#00B050">AMD Research</span></b> |
<a href="mailto:john.wilkes@amd.com">john.wilkes@amd.com</a> | office: +1 425.586.6412 (x26412)<o:p></o:p></p>
<p class="MsoNormal"><o:p> </o:p></p>
</div>
</div>
</div>
</body>
</html>