[ofiwg] GASNet with OFI on PSM provider

Mon Jun 25 13:14:37 PDT 2018

Yes, it should work.

From: Sayan Ghosh [mailto:zsayanz at gmail.com]
Sent: Monday, June 25, 2018 12:37 PM
To: Hefty, Sean <sean.hefty at intel.com>
Cc: ofiwg at lists.openfabrics.org; Xiong, Jianxin <jianxin.xiong at intel.com>
Subject: Re: [ofiwg] GASNet with OFI on PSM provider

I meant MPICH with OFI for TrueScale.

On Mon, Jun 25, 2018 at 12:35 PM Sayan Ghosh <zsayanz at gmail.com<mailto:zsayanz at gmail.com>> wrote:
Thanks, in my preliminary evaluation of MPICH with OFI, I don't see an issue.

On Mon, Jun 25, 2018 at 9:28 AM Hefty, Sean <sean.hefty at intel.com<mailto:sean.hefty at intel.com>> wrote:
Actually copying the psm/psm2 provider maintainer this time.  But, MPICH should work.

- Sean

> -----Original Message-----
> From: ofiwg [mailto:ofiwg-bounces at lists.openfabrics.org<mailto:ofiwg-bounces at lists.openfabrics.org>] On Behalf Of
> Hefty, Sean
> Sent: Friday, June 22, 2018 4:53 PM
> To: Sayan Ghosh <zsayanz at gmail.com<mailto:zsayanz at gmail.com>>
> Cc: ofiwg at lists.openfabrics.org<mailto:ofiwg at lists.openfabrics.org>
> Subject: Re: [ofiwg] GASNet with OFI on PSM provider
>
> OFI does work over PSM.  However, the psm and psm2 providers do not
> support the same OFI feature set.  You can see the differences here:
>
> https://github.com/ofiwg/libfabric/wiki/Provider-Feature-Matrix-master
>
> I'm not familiar enough with the GasNET implementation to know what
> feature it needs.
>
> I'm copying the developer who maintains both the psm and psm2
> providers.  He can provide more details on what differences there are
> between psm and psm2, and *might* know if GasNET should work over the
> psm provider.  Unfortunately, the developer who ported GasNET to OFI
> is not available.
>
> - Sean
>
>
> > -----Original Message-----
> > From: Sayan Ghosh [mailto:zsayanz at gmail.com<mailto:zsayanz at gmail.com>]
> > Sent: Friday, June 22, 2018 4:39 PM
> > To: Hefty, Sean <sean.hefty at intel.com<mailto:sean.hefty at intel.com>>
> > Cc: ofiwg at lists.openfabrics.org<mailto:ofiwg at lists.openfabrics.org>
> > Subject: Re: [ofiwg] GASNet with OFI on PSM provider
> >
> > Hi Sean - Sorry if it is evident, but I just wanted to clarify
> > something. Since OFI has a PSM provider, can I assume that it works?
> I
> > am asking because you said "OFI targets Omni Path".
> >
> > On Wed, Jun 20, 2018 at 12:08 PM Hefty, Sean <sean.hefty at intel.com<mailto:sean.hefty at intel.com>
> > <mailto:sean.hefty at intel.com<mailto:sean.hefty at intel.com>> > wrote:
> >
> >
> >     To clarify, I meant GASNet to TrueScale through OFI.  GASNet
> should
> > have a psm-conduit.  OFI targets Omni-Path.
> >
> >
> >
> >     - Sean
> >
> >
> >
> >
> >
> >
> >
> >     This is my understanding as well.  I don’t believe GASNet is a
> target
> > application for TrueScale or PSM.  The high performance interface
> for
> > TrueScale is PSM, rather than verbs, and targets older MPI
> > implementations.
> >
> >
> >
> >     - Sean
> >
> >
> >
> >
> >
> >     From: ofiwg [mailto:ofiwg-bounces at lists.openfabrics.org<mailto:ofiwg-bounces at lists.openfabrics.org>] On
> Behalf Of
> > Sayan Ghosh
> >     Sent: Wednesday, June 20, 2018 9:17 AM
> >     To: ofiwg at lists.openfabrics.org<mailto:ofiwg at lists.openfabrics.org>
> > <mailto:ofiwg at lists.openfabrics.org<mailto:ofiwg at lists.openfabrics.org>>
> >     Subject: Re: [ofiwg] GASNet with OFI on PSM provider
> >
> >
> >
> >     Just wanted to clarify that a discussion with GASNet developers
> > revealed that their PSM provider compatibility over OFI is only for
> > PSM2 (since TrueScale is an old technology).
> >
> >     https://groups.google.com/a/lbl.gov/forum/#!topic/gasnet-
> > users/1Y_mDmdcCDw
> >
> >
> >
> >     On Tue, Jun 19, 2018 at 10:44 PM Sayan Ghosh <zsayanz at gmail.com<mailto:zsayanz at gmail.com>
> > <mailto:zsayanz at gmail.com<mailto:zsayanz at gmail.com>> > wrote:
> >
> >             I am trying to install GASNet 1.30.0 using OFI on an Intel
> TrueScale
> > network (PSM provider), and although OFI fi_info detects PSM
> provider
> > (see attached output of fi_info), when I mention it during GASNet
> > configure (config.log attached, please see line #10126), GASNet
> > warns:
> >
> >
> >
> >             WARNING: Invalid or unsupported OFI provider specified via
> the
> > --with-ofi-provider flag.
> >
> >             Defaulting to runtime provider feature detection for ofi-
> conduit.
> >
> >
> >
> >
> >             GASNet configuration:
> >
> >
> >
> >             CC=icc CXX=icpc MPI_CC=/home/sghosh/builds/mvapich222-
> > intel-psm/bin/mpicc MPI_CFLAGS="-I/home/sghosh/builds/mvapich222-
> > intel-psm/include" MPI_LIBS="-L/home/sghosh/builds/mvapich222-intel-
> > psm/lib -lpciaccess -lnuma -lxml2" ./configure --
> > prefix=/home/sghosh/builds/gasnet-ofi --enable-ofi --with-ofi-
> > home=/home/sghosh/builds/libfabric --disable-psm --disable-ibv --
> with-
> > ofi-spawner=mpi --with-ofi-provider=psm
> >
> >
> >
> >             GASNet throws this warning during application execution:
> >
> >
> >
> >             WARNING: Using OFI provider (verbs;ofi_rxm), which has not
> been
> > validated to provide
> >
> >             WARNING: acceptable GASNet performance. You should
> consider using a
> > more
> >
> >             WARNING: hardware-appropriate GASNet conduit. See ofi-
> > conduit/README.
> >
> >             *** FATAL ERROR: The statically-determined value for
> > GASNETC_OFI_HAS_MR_SCALABLE=1 does
> >
> >               not match the memory registration support that the
> > (verbs;ofi_rxm) provider reported.
> >
> >               This could happen if a provider that previously only
> supported
> > FI_MR_BASIC
> >
> >               added support for FI_MR_SCALABLE, or if the wrong
> provider was
> > selected at runtime.
> >
> >               Use configure option --disable-ofi-mr-scalable to
> correct this.
> >
> >
> >
> >             Instead of picking up psm, GASNet picks up verbs. I have
> reported
> > this issue on GASNet user list as well. Please let me know if I can
> > try some options in building libfabrics for PSM provider, that may
> fix
> > this issue.
> >
> >
> >
> >             --
> >
> >             Thanks,
> >
> >             Sayan | https://sg0.github.io/
> >
> >
> >
> >
> >
> >
> >     --
> >
> >     Sayan | https://sg0.github.io/
> >
> >
> >
> > --
> >
> > Sayan | https://sg0.github.io/
>
> _______________________________________________
> ofiwg mailing list
> ofiwg at lists.openfabrics.org<mailto:ofiwg at lists.openfabrics.org>
> http://lists.openfabrics.org/mailman/listinfo/ofiwg

--
Sayan | https://sg0.github.io/

--
Sayan | https://sg0.github.io/
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.openfabrics.org/pipermail/ofiwg/attachments/20180625/7d36daef/attachment.html>