<div dir="ltr">Hi Sean - Sorry if it is evident, but I just wanted to clarify something. Since OFI has a PSM provider, can I assume that it works? I am asking because you said "OFI targets Omni Path".</div><br><div class="gmail_quote"><div dir="ltr">On Wed, Jun 20, 2018 at 12:08 PM Hefty, Sean <<a href="mailto:sean.hefty@intel.com">sean.hefty@intel.com</a>> wrote:<br></div><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex">







<div lang="EN-US" link="blue" vlink="purple">
<div class="m_-7123154727851264023WordSection1">
<p class="MsoNormal"><font size="2" color="#1f497d" face="Calibri"><span style="font-size:11.0pt;font-family:"Calibri",sans-serif;color:#1f497d">To clarify, I meant
<span class="m_-7123154727851264023SpellE">GASNet</span> to <span class="m_-7123154727851264023SpellE">TrueScale</span> through OFI.<span> 
</span><span class="m_-7123154727851264023SpellE">GASNet</span> should have a <span class="m_-7123154727851264023SpellE">psm</span>-conduit.<span> 
</span>OFI targets Omni-Path.<u></u><u></u></span></font></p>
<p class="MsoNormal"><font size="2" color="#1f497d" face="Calibri"><span style="font-size:11.0pt;font-family:"Calibri",sans-serif;color:#1f497d"><u></u> <u></u></span></font></p>
<p class="MsoNormal"><font size="2" color="#1f497d" face="Calibri"><span style="font-size:11.0pt;font-family:"Calibri",sans-serif;color:#1f497d">- Sean<u></u><u></u></span></font></p>
<p class="MsoNormal"><a name="m_-7123154727851264023__MailEndCompose"><font size="2" color="#1f497d" face="Calibri"><span style="font-size:11.0pt;font-family:"Calibri",sans-serif;color:#1f497d"><u></u> <u></u></span></font></a></p>
<p class="MsoNormal"><span><font size="2" color="#1f497d" face="Calibri"><span style="font-size:11.0pt;font-family:"Calibri",sans-serif;color:#1f497d"><u></u> <u></u></span></font></span></p>
<span></span>
<div style="border:none;border-left:solid blue 1.5pt;padding:0in 0in 0in 4.0pt">
<p class="MsoNormal"><a name="m_-7123154727851264023______replyseparator"></a><font size="2" color="#1f497d" face="Calibri"><span style="font-size:11.0pt;font-family:"Calibri",sans-serif;color:#1f497d">This is my understanding as well.<span> 
</span>I don’t believe <span class="m_-7123154727851264023SpellE">GASNet</span> is a target application for
<span class="m_-7123154727851264023SpellE">TrueScale</span> or PSM.<span>  </span>
The high performance interface for <span class="m_-7123154727851264023SpellE">TrueScale</span> is PSM, rather than verbs, and targets older MPI implementations.<u></u><u></u></span></font></p>
<p class="MsoNormal"><font size="2" color="#1f497d" face="Calibri"><span style="font-size:11.0pt;font-family:"Calibri",sans-serif;color:#1f497d"><u></u> <u></u></span></font></p>
<p class="MsoNormal"><font size="2" color="#1f497d" face="Calibri"><span style="font-size:11.0pt;font-family:"Calibri",sans-serif;color:#1f497d">- Sean<u></u><u></u></span></font></p>
<p class="MsoNormal"><font size="2" color="#1f497d" face="Calibri"><span style="font-size:11.0pt;font-family:"Calibri",sans-serif;color:#1f497d"><u></u> <u></u></span></font></p>
<p class="MsoNormal"><font size="2" color="#1f497d" face="Calibri"><span style="font-size:11.0pt;font-family:"Calibri",sans-serif;color:#1f497d"><u></u> <u></u></span></font></p>
<div style="border:none;border-left:solid blue 1.5pt;padding:0in 0in 0in 4.0pt">
<div>
<div style="border:none;border-top:solid #e1e1e1 1.0pt;padding:3.0pt 0in 0in 0in">
<p class="MsoNormal"><b><font size="2" face="Calibri"><span style="font-size:11.0pt;font-family:"Calibri",sans-serif;font-weight:bold">From:</span></font></b><font size="2" face="Calibri"><span style="font-size:11.0pt;font-family:"Calibri",sans-serif">
<span class="m_-7123154727851264023SpellE">ofiwg</span> [<a href="mailto:ofiwg-bounces@lists.openfabrics.org" target="_blank">mailto:ofiwg-bounces@lists.openfabrics.org</a>]
<b><span style="font-weight:bold">On Behalf Of </span></b>Sayan Ghosh<br>
<b><span style="font-weight:bold">Sent:</span></b> Wednesday, June 20, 2018 9:17 AM<br>
<b><span style="font-weight:bold">To:</span></b> <a href="mailto:ofiwg@lists.openfabrics.org" target="_blank">
ofiwg@lists.openfabrics.org</a><br>
<b><span style="font-weight:bold">Subject:</span></b> Re: [<span class="m_-7123154727851264023SpellE">ofiwg</span>]
<span class="m_-7123154727851264023SpellE">GASNet</span> with OFI on PSM provider<u></u><u></u></span></font></p>
</div>
</div>
<p class="MsoNormal"><font size="3" face="Times New Roman"><span style="font-size:12.0pt"><u></u> <u></u></span></font></p>
<div>
<p class="MsoNormal"><font size="3" face="Times New Roman"><span style="font-size:12.0pt">Just wanted to clarify that a discussion with
<span class="m_-7123154727851264023SpellE">GASNet</span> developers revealed that their PSM provider compatibility over OFI is only for PSM2 (since
<span class="m_-7123154727851264023SpellE">TrueScale</span> is an old technology).<u></u><u></u></span></font></p>
<div>
<p class="MsoNormal"><font size="3" face="Times New Roman"><span style="font-size:12.0pt"><a href="https://groups.google.com/a/lbl.gov/forum/#!topic/gasnet-users/1Y_mDmdcCDw" target="_blank">https://groups.google.com/a/lbl.gov/forum/#!topic/gasnet-users/1Y_mDmdcCDw</a> <u></u><u></u></span></font></p>
</div>
</div>
<p class="MsoNormal"><font size="3" face="Times New Roman"><span style="font-size:12.0pt"><u></u> <u></u></span></font></p>
<div>
<div>
<p class="MsoNormal"><font size="3" face="Times New Roman"><span style="font-size:12.0pt">On Tue, Jun 19, 2018 at 10:44 PM Sayan Ghosh <<a href="mailto:zsayanz@gmail.com" target="_blank">zsayanz@gmail.com</a>> wrote:<u></u><u></u></span></font></p>
</div>
<blockquote style="border:none;border-left:solid #cccccc 1.0pt;padding:0in 0in 0in 6.0pt;margin-left:4.8pt;margin-top:5.0pt;margin-right:0in;margin-bottom:5.0pt">
<div>
<div>
<p class="MsoNormal"><font size="3" face="Times New Roman"><span style="font-size:12.0pt">I am trying to install
<span class="m_-7123154727851264023SpellE">GASNet</span> 1.30.0 using OFI on an Intel <span class="m_-7123154727851264023SpellE">
TrueScale</span> network (PSM provider), and although OFI <span class="m_-7123154727851264023SpellE">fi_info</span> detects PSM provider (see attached output of
<span class="m_-7123154727851264023SpellE">fi_info</span>), when I mention it during <span class="m_-7123154727851264023SpellE">
GASNet</span> configure (config.log attached, please see line #10126), <span class="m_-7123154727851264023SpellE">
GASNet</span> warns:<u></u><u></u></span></font></p>
</div>
<div>
<p class="MsoNormal"><font size="3" face="Times New Roman"><span style="font-size:12.0pt"><u></u> <u></u></span></font></p>
</div>
<div>
<p class="MsoNormal"><font size="3" face="Times New Roman"><span style="font-size:12.0pt;background:yellow">WARNING: Invalid or unsupported OFI provider specified via the --with-<span class="m_-7123154727851264023SpellE">ofi</span>-provider flag.</span><u></u><u></u></font></p>
</div>
<div>
<p class="MsoNormal"><font size="3" face="Times New Roman"><span style="font-size:12.0pt;background:yellow">Defaulting to runtime provider feature detection for
<span class="m_-7123154727851264023SpellE">ofi</span>-conduit.</span><u></u><u></u></font></p>
</div>
<div>
<p class="MsoNormal"><font size="3" face="Times New Roman"><span style="font-size:12.0pt"> <br clear="all">
<u></u><u></u></span></font></p>
<div>
<p class="MsoNormal"><span class="m_-7123154727851264023SpellE"><font size="3" face="Times New Roman"><span style="font-size:12.0pt">GASNet</span></font></span> configuration:<u></u><u></u></p>
</div>
<div>
<p class="MsoNormal"><font size="3" face="Times New Roman"><span style="font-size:12.0pt"><u></u> <u></u></span></font></p>
</div>
<div>
<div>
<p class="MsoNormal"><font size="3" face="Times New Roman"><span style="font-size:12.0pt;background:yellow">CC=<span class="m_-7123154727851264023SpellE">icc</span> CXX=<span class="m_-7123154727851264023SpellE">icpc</span> MPI_CC=/home/<span class="m_-7123154727851264023SpellE">sghosh</span>/builds/mvapich222-intel-psm/bin/<span class="m_-7123154727851264023SpellE">mpicc</span>
 MPI_CFLAGS="-I/home/<span class="m_-7123154727851264023SpellE">sghosh</span>/builds/mvapich222-intel-psm/include" MPI_LIBS="-L/home/<span class="m_-7123154727851264023SpellE">sghosh</span>/builds/mvapich222-intel-psm/lib -<span class="m_-7123154727851264023SpellE">lpciaccess</span> -<span class="m_-7123154727851264023SpellE">lnuma</span> -lxml2"
 ./configure --prefix=/home/<span class="m_-7123154727851264023SpellE">sghosh</span>/builds/<span class="m_-7123154727851264023SpellE">gasnet-ofi</span> --enable-<span class="m_-7123154727851264023SpellE">ofi</span> --with-<span class="m_-7123154727851264023SpellE">ofi</span>-home=/home/<span class="m_-7123154727851264023SpellE">sghosh</span>/builds/libfabric --disable-<span class="m_-7123154727851264023SpellE">psm</span>
 --disable-<span class="m_-7123154727851264023SpellE">ibv</span> --with-<span class="m_-7123154727851264023SpellE">ofi</span>-<span class="m_-7123154727851264023SpellE">spawner</span>=<span class="m_-7123154727851264023SpellE">mpi</span> --with-<span class="m_-7123154727851264023SpellE">ofi</span>-provider=<span class="m_-7123154727851264023SpellE">psm</span></span><u></u><u></u></font></p>
</div>
</div>
<div>
<p class="MsoNormal"><font size="3" face="Times New Roman"><span style="font-size:12.0pt"><u></u> <u></u></span></font></p>
</div>
<div>
<p class="MsoNormal"><span class="m_-7123154727851264023SpellE"><font size="3" face="Times New Roman"><span style="font-size:12.0pt">GASNet</span></font></span> throws this warning during application execution:<u></u><u></u></p>
</div>
<div>
<p class="MsoNormal"><font size="3" face="Times New Roman"><span style="font-size:12.0pt"><u></u> <u></u></span></font></p>
</div>
<div>
<div>
<p class="MsoNormal"><font size="3" face="Times New Roman"><span style="font-size:12.0pt;background:yellow">WARNING: Using OFI provider (<span class="m_-7123154727851264023SpellE">verbs;ofi_rxm</span>), which has not been validated to provide</span><u></u><u></u></font></p>
</div>
<div>
<p class="MsoNormal"><font size="3" face="Times New Roman"><span style="font-size:12.0pt;background:yellow">WARNING: acceptable
<span class="m_-7123154727851264023SpellE">GASNet</span> performance. You should consider using a more</span><u></u><u></u></font></p>
</div>
<div>
<p class="MsoNormal"><font size="3" face="Times New Roman"><span style="font-size:12.0pt;background:yellow">WARNING: hardware-appropriate
<span class="m_-7123154727851264023SpellE">GASNet</span> conduit. See <span class="m_-7123154727851264023SpellE">ofi</span>-conduit/README.</span><u></u><u></u></font></p>
</div>
<div>
<p class="MsoNormal"><font size="3" face="Times New Roman"><span style="font-size:12.0pt;background:yellow">*** FATAL ERROR: The statically-determined value for GASNETC_OFI_HAS_MR_SCALABLE=1 does</span><u></u><u></u></font></p>
</div>
<div>
<p class="MsoNormal"><font size="3" face="Times New Roman"><span style="font-size:12.0pt;background:yellow">  not match the memory registration support that the (<span class="m_-7123154727851264023SpellE">verbs;ofi_rxm</span>) provider reported.</span><u></u><u></u></font></p>
</div>
<div>
<p class="MsoNormal"><font size="3" face="Times New Roman"><span style="font-size:12.0pt;background:yellow">  This could happen if a provider that previously only supported FI_MR_BASIC</span><u></u><u></u></font></p>
</div>
<div>
<p class="MsoNormal"><font size="3" face="Times New Roman"><span style="font-size:12.0pt;background:yellow">  added support for FI_MR_SCALABLE, or if the wrong provider was selected at runtime.</span><u></u><u></u></font></p>
</div>
<div>
<p class="MsoNormal"><font size="3" face="Times New Roman"><span style="font-size:12.0pt;background:yellow">  Use configure option --disable-<span class="m_-7123154727851264023SpellE">ofi</span>-<span class="m_-7123154727851264023SpellE">mr</span>-scalable to correct this.</span><u></u><u></u></font></p>
</div>
<div>
<p class="MsoNormal"><font size="3" face="Times New Roman"><span style="font-size:12.0pt"><u></u> <u></u></span></font></p>
</div>
</div>
<div>
<p class="MsoNormal"><font size="3" face="Times New Roman"><span style="font-size:12.0pt">Instead of picking up
<span class="m_-7123154727851264023SpellE">psm</span>, <span class="m_-7123154727851264023SpellE">GASNet</span> picks up verbs. I have reported this issue on
<span class="m_-7123154727851264023SpellE">GASNet</span> user list as well. Please let me know if I can try some options in building
<span class="m_-7123154727851264023SpellE">libfabrics</span> for PSM provider, that may fix this issue. <u></u><u></u></span></font></p>
</div>
<div>
<p class="MsoNormal"><font size="3" face="Times New Roman"><span style="font-size:12.0pt"><u></u> <u></u></span></font></p>
</div>
</div>
<p class="MsoNormal"><font size="3" face="Times New Roman"><span style="font-size:12.0pt">-- <u></u><u></u></span></font></p>
<div>
<p class="MsoNormal"><font size="3" face="Times New Roman"><span style="font-size:12.0pt">Thanks,<u></u><u></u></span></font></p>
<div>
<div>
<div>
<div>
<div>
<div>
<div>
<div>
<div>
<div>
<div>
<div>
<p class="MsoNormal"><font size="3" face="Times New Roman"><span style="font-size:12.0pt">S</span></font><font size="2"><span style="font-size:9.5pt">ayan | <a href="https://sg0.github.io/" target="_blank">https://sg0.github.io/</a></span></font><u></u><u></u></p>
</div>
</div>
</div>
</div>
</div>
</div>
</div>
</div>
</div>
</div>
</div>
</div>
</div>
</div>
</blockquote>
</div>
<p class="MsoNormal"><font size="3" face="Times New Roman"><span style="font-size:12.0pt"><br clear="all">
<u></u><u></u></span></font></p>
<div>
<p class="MsoNormal"><font size="3" face="Times New Roman"><span style="font-size:12.0pt"><u></u> <u></u></span></font></p>
</div>
<p class="MsoNormal"><font size="3" face="Times New Roman"><span style="font-size:12.0pt">--
<u></u><u></u></span></font></p>
<div>
<div>
<div>
<div>
<div>
<div>
<div>
<div>
<div>
<div>
<div>
<div>
<p class="MsoNormal"><font size="3" face="Times New Roman"><span style="font-size:12.0pt">S</span></font><font size="2"><span style="font-size:9.5pt">ayan | <a href="https://sg0.github.io/" target="_blank">https://sg0.github.io/</a></span></font><u></u><u></u></p>
</div>
</div>
</div>
</div>
</div>
</div>
</div>
</div>
</div>
</div>
</div>
</div>
</div>
</div>
</div>
</div>

</blockquote></div><br clear="all"><div><br></div>-- <br><div dir="ltr" class="gmail_signature" data-smartmail="gmail_signature"><div dir="ltr"><div><div dir="ltr"><div><div dir="ltr"><div><div dir="ltr"><div><div dir="ltr"><div><div dir="ltr"><span style="font-size:small">S</span><span style="font-size:12.8px">ayan | </span><span style="font-size:12.8px"><a href="https://sg0.github.io/" target="_blank">https://sg0.github.io/</a></span><br></div></div></div></div></div></div></div></div></div></div></div></div>