[ofiwg] GASNet with OFI on PSM provider

[Sayan Ghosh]

Thanks Sean, just to clarify, this question that I asked right now has
nothing to do with GASNet (I was trying to avoid a new email chain).
I was just wondering if I can build OFI with PSM provider, and use it with
MPICH. From your answer, it seems that I can.

On Fri, Jun 22, 2018 at 4:53 PM Hefty, Sean <sean.hefty at intel.com> wrote:

> OFI does work over PSM.  However, the psm and psm2 providers do not
> support the same OFI feature set.  You can see the differences here:
>
> https://github.com/ofiwg/libfabric/wiki/Provider-Feature-Matrix-master
>
> I'm not familiar enough with the GasNET implementation to know what
> feature it needs.
>
> I'm copying the developer who maintains both the psm and psm2 providers.
> He can provide more details on what differences there are between psm and
> psm2, and *might* know if GasNET should work over the psm provider.
> Unfortunately, the developer who ported GasNET to OFI is not available.
>
> - Sean
>
>
> > -----Original Message-----
> > From: Sayan Ghosh [mailto:zsayanz at gmail.com]
> > Sent: Friday, June 22, 2018 4:39 PM
> > To: Hefty, Sean <sean.hefty at intel.com>
> > Cc: ofiwg at lists.openfabrics.org
> > Subject: Re: [ofiwg] GASNet with OFI on PSM provider
> >
> > Hi Sean - Sorry if it is evident, but I just wanted to clarify
> > something. Since OFI has a PSM provider, can I assume that it works? I
> > am asking because you said "OFI targets Omni Path".
> >
> > On Wed, Jun 20, 2018 at 12:08 PM Hefty, Sean <sean.hefty at intel.com
> > <mailto:sean.hefty at intel.com> > wrote:
> >
> >
> >       To clarify, I meant GASNet to TrueScale through OFI.  GASNet
> > should have a psm-conduit.  OFI targets Omni-Path.
> >
> >
> >
> >       - Sean
> >
> >
> >
> >
> >
> >
> >
> >       This is my understanding as well.  I don’t believe GASNet is a
> > target application for TrueScale or PSM.  The high performance
> > interface for TrueScale is PSM, rather than verbs, and targets older
> > MPI implementations.
> >
> >
> >
> >       - Sean
> >
> >
> >
> >
> >
> >       From: ofiwg [mailto:ofiwg-bounces at lists.openfabrics.org] On
> > Behalf Of Sayan Ghosh
> >       Sent: Wednesday, June 20, 2018 9:17 AM
> >       To: ofiwg at lists.openfabrics.org
> > <mailto:ofiwg at lists.openfabrics.org>
> >       Subject: Re: [ofiwg] GASNet with OFI on PSM provider
> >
> >
> >
> >       Just wanted to clarify that a discussion with GASNet developers
> > revealed that their PSM provider compatibility over OFI is only for
> > PSM2 (since TrueScale is an old technology).
> >
> >       https://groups.google.com/a/lbl.gov/forum/#!topic/gasnet-
> > users/1Y_mDmdcCDw
> >
> >
> >
> >       On Tue, Jun 19, 2018 at 10:44 PM Sayan Ghosh <zsayanz at gmail.com
> > <mailto:zsayanz at gmail.com> > wrote:
> >
> >               I am trying to install GASNet 1.30.0 using OFI on an Intel
> > TrueScale network (PSM provider), and although OFI fi_info detects PSM
> > provider (see attached output of fi_info), when I mention it during
> > GASNet configure (config.log attached, please see line #10126), GASNet
> > warns:
> >
> >
> >
> >               WARNING: Invalid or unsupported OFI provider specified via
> > the --with-ofi-provider flag.
> >
> >               Defaulting to runtime provider feature detection for ofi-
> > conduit.
> >
> >
> >
> >
> >               GASNet configuration:
> >
> >
> >
> >               CC=icc CXX=icpc MPI_CC=/home/sghosh/builds/mvapich222-
> > intel-psm/bin/mpicc MPI_CFLAGS="-I/home/sghosh/builds/mvapich222-
> > intel-psm/include" MPI_LIBS="-L/home/sghosh/builds/mvapich222-intel-
> > psm/lib -lpciaccess -lnuma -lxml2" ./configure --
> > prefix=/home/sghosh/builds/gasnet-ofi --enable-ofi --with-ofi-
> > home=/home/sghosh/builds/libfabric --disable-psm --disable-ibv --with-
> > ofi-spawner=mpi --with-ofi-provider=psm
> >
> >
> >
> >               GASNet throws this warning during application execution:
> >
> >
> >
> >               WARNING: Using OFI provider (verbs;ofi_rxm), which has not
> > been validated to provide
> >
> >               WARNING: acceptable GASNet performance. You should
> > consider using a more
> >
> >               WARNING: hardware-appropriate GASNet conduit. See ofi-
> > conduit/README.
> >
> >               *** FATAL ERROR: The statically-determined value for
> > GASNETC_OFI_HAS_MR_SCALABLE=1 does
> >
> >                 not match the memory registration support that the
> > (verbs;ofi_rxm) provider reported.
> >
> >                 This could happen if a provider that previously only
> > supported FI_MR_BASIC
> >
> >                 added support for FI_MR_SCALABLE, or if the wrong
> > provider was selected at runtime.
> >
> >                 Use configure option --disable-ofi-mr-scalable to
> > correct this.
> >
> >
> >
> >               Instead of picking up psm, GASNet picks up verbs. I have
> > reported this issue on GASNet user list as well. Please let me know if
> > I can try some options in building libfabrics for PSM provider, that
> > may fix this issue.
> >
> >
> >
> >               --
> >
> >               Thanks,
> >
> >               Sayan | https://sg0.github.io/
> >
> >
> >
> >
> >
> >
> >       --
> >
> >       Sayan | https://sg0.github.io/
> >
> >
> >
> > --
> >
> > Sayan | https://sg0.github.io/
>
>

-- 
Sayan | https://sg0.github.io/
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.openfabrics.org/pipermail/ofiwg/attachments/20180622/af2f300e/attachment.html>