[ofiwg] GASNet with OFI on PSM provider
Sayan Ghosh
zsayanz at gmail.com
Mon Jun 25 12:36:35 PDT 2018
I meant MPICH with OFI for TrueScale.
On Mon, Jun 25, 2018 at 12:35 PM Sayan Ghosh <zsayanz at gmail.com> wrote:
> Thanks, in my preliminary evaluation of MPICH with OFI, I don't see an
> issue.
>
> On Mon, Jun 25, 2018 at 9:28 AM Hefty, Sean <sean.hefty at intel.com> wrote:
>
>> Actually copying the psm/psm2 provider maintainer this time. But, MPICH
>> should work.
>>
>> - Sean
>>
>> > -----Original Message-----
>> > From: ofiwg [mailto:ofiwg-bounces at lists.openfabrics.org] On Behalf Of
>> > Hefty, Sean
>> > Sent: Friday, June 22, 2018 4:53 PM
>> > To: Sayan Ghosh <zsayanz at gmail.com>
>> > Cc: ofiwg at lists.openfabrics.org
>> > Subject: Re: [ofiwg] GASNet with OFI on PSM provider
>> >
>> > OFI does work over PSM. However, the psm and psm2 providers do not
>> > support the same OFI feature set. You can see the differences here:
>> >
>> > https://github.com/ofiwg/libfabric/wiki/Provider-Feature-Matrix-master
>> >
>> > I'm not familiar enough with the GasNET implementation to know what
>> > feature it needs.
>> >
>> > I'm copying the developer who maintains both the psm and psm2
>> > providers. He can provide more details on what differences there are
>> > between psm and psm2, and *might* know if GasNET should work over the
>> > psm provider. Unfortunately, the developer who ported GasNET to OFI
>> > is not available.
>> >
>> > - Sean
>> >
>> >
>> > > -----Original Message-----
>> > > From: Sayan Ghosh [mailto:zsayanz at gmail.com]
>> > > Sent: Friday, June 22, 2018 4:39 PM
>> > > To: Hefty, Sean <sean.hefty at intel.com>
>> > > Cc: ofiwg at lists.openfabrics.org
>> > > Subject: Re: [ofiwg] GASNet with OFI on PSM provider
>> > >
>> > > Hi Sean - Sorry if it is evident, but I just wanted to clarify
>> > > something. Since OFI has a PSM provider, can I assume that it works?
>> > I
>> > > am asking because you said "OFI targets Omni Path".
>> > >
>> > > On Wed, Jun 20, 2018 at 12:08 PM Hefty, Sean <sean.hefty at intel.com
>> > > <mailto:sean.hefty at intel.com> > wrote:
>> > >
>> > >
>> > > To clarify, I meant GASNet to TrueScale through OFI. GASNet
>> > should
>> > > have a psm-conduit. OFI targets Omni-Path.
>> > >
>> > >
>> > >
>> > > - Sean
>> > >
>> > >
>> > >
>> > >
>> > >
>> > >
>> > >
>> > > This is my understanding as well. I don’t believe GASNet is a
>> > target
>> > > application for TrueScale or PSM. The high performance interface
>> > for
>> > > TrueScale is PSM, rather than verbs, and targets older MPI
>> > > implementations.
>> > >
>> > >
>> > >
>> > > - Sean
>> > >
>> > >
>> > >
>> > >
>> > >
>> > > From: ofiwg [mailto:ofiwg-bounces at lists.openfabrics.org] On
>> > Behalf Of
>> > > Sayan Ghosh
>> > > Sent: Wednesday, June 20, 2018 9:17 AM
>> > > To: ofiwg at lists.openfabrics.org
>> > > <mailto:ofiwg at lists.openfabrics.org>
>> > > Subject: Re: [ofiwg] GASNet with OFI on PSM provider
>> > >
>> > >
>> > >
>> > > Just wanted to clarify that a discussion with GASNet developers
>> > > revealed that their PSM provider compatibility over OFI is only for
>> > > PSM2 (since TrueScale is an old technology).
>> > >
>> > > https://groups.google.com/a/lbl.gov/forum/#!topic/gasnet-
>> > > users/1Y_mDmdcCDw
>> > >
>> > >
>> > >
>> > > On Tue, Jun 19, 2018 at 10:44 PM Sayan Ghosh <zsayanz at gmail.com
>> > > <mailto:zsayanz at gmail.com> > wrote:
>> > >
>> > > I am trying to install GASNet 1.30.0 using OFI on an Intel
>> > TrueScale
>> > > network (PSM provider), and although OFI fi_info detects PSM
>> > provider
>> > > (see attached output of fi_info), when I mention it during GASNet
>> > > configure (config.log attached, please see line #10126), GASNet
>> > > warns:
>> > >
>> > >
>> > >
>> > > WARNING: Invalid or unsupported OFI provider specified via
>> > the
>> > > --with-ofi-provider flag.
>> > >
>> > > Defaulting to runtime provider feature detection for ofi-
>> > conduit.
>> > >
>> > >
>> > >
>> > >
>> > > GASNet configuration:
>> > >
>> > >
>> > >
>> > > CC=icc CXX=icpc MPI_CC=/home/sghosh/builds/mvapich222-
>> > > intel-psm/bin/mpicc MPI_CFLAGS="-I/home/sghosh/builds/mvapich222-
>> > > intel-psm/include" MPI_LIBS="-L/home/sghosh/builds/mvapich222-intel-
>> > > psm/lib -lpciaccess -lnuma -lxml2" ./configure --
>> > > prefix=/home/sghosh/builds/gasnet-ofi --enable-ofi --with-ofi-
>> > > home=/home/sghosh/builds/libfabric --disable-psm --disable-ibv --
>> > with-
>> > > ofi-spawner=mpi --with-ofi-provider=psm
>> > >
>> > >
>> > >
>> > > GASNet throws this warning during application execution:
>> > >
>> > >
>> > >
>> > > WARNING: Using OFI provider (verbs;ofi_rxm), which has not
>> > been
>> > > validated to provide
>> > >
>> > > WARNING: acceptable GASNet performance. You should
>> > consider using a
>> > > more
>> > >
>> > > WARNING: hardware-appropriate GASNet conduit. See ofi-
>> > > conduit/README.
>> > >
>> > > *** FATAL ERROR: The statically-determined value for
>> > > GASNETC_OFI_HAS_MR_SCALABLE=1 does
>> > >
>> > > not match the memory registration support that the
>> > > (verbs;ofi_rxm) provider reported.
>> > >
>> > > This could happen if a provider that previously only
>> > supported
>> > > FI_MR_BASIC
>> > >
>> > > added support for FI_MR_SCALABLE, or if the wrong
>> > provider was
>> > > selected at runtime.
>> > >
>> > > Use configure option --disable-ofi-mr-scalable to
>> > correct this.
>> > >
>> > >
>> > >
>> > > Instead of picking up psm, GASNet picks up verbs. I have
>> > reported
>> > > this issue on GASNet user list as well. Please let me know if I can
>> > > try some options in building libfabrics for PSM provider, that may
>> > fix
>> > > this issue.
>> > >
>> > >
>> > >
>> > > --
>> > >
>> > > Thanks,
>> > >
>> > > Sayan | https://sg0.github.io/
>> > >
>> > >
>> > >
>> > >
>> > >
>> > >
>> > > --
>> > >
>> > > Sayan | https://sg0.github.io/
>> > >
>> > >
>> > >
>> > > --
>> > >
>> > > Sayan | https://sg0.github.io/
>> >
>> > _______________________________________________
>> > ofiwg mailing list
>> > ofiwg at lists.openfabrics.org
>> > http://lists.openfabrics.org/mailman/listinfo/ofiwg
>>
>
>
> --
> Sayan | https://sg0.github.io/
>
--
Sayan | https://sg0.github.io/
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.openfabrics.org/pipermail/ofiwg/attachments/20180625/ee52cf4e/attachment.html>
More information about the ofiwg
mailing list