[ofiwg] GASNet with OFI on PSM provider

[Hefty, Sean]

Actually copying the psm/psm2 provider maintainer this time.  But, MPICH should work.

- Sean

> -----Original Message-----
> From: ofiwg [mailto:ofiwg-bounces at lists.openfabrics.org] On Behalf Of
> Hefty, Sean
> Sent: Friday, June 22, 2018 4:53 PM
> To: Sayan Ghosh <zsayanz at gmail.com>
> Cc: ofiwg at lists.openfabrics.org
> Subject: Re: [ofiwg] GASNet with OFI on PSM provider
> 
> OFI does work over PSM.  However, the psm and psm2 providers do not
> support the same OFI feature set.  You can see the differences here:
> 
> https://github.com/ofiwg/libfabric/wiki/Provider-Feature-Matrix-master
> 
> I'm not familiar enough with the GasNET implementation to know what
> feature it needs.
> 
> I'm copying the developer who maintains both the psm and psm2
> providers.  He can provide more details on what differences there are
> between psm and psm2, and *might* know if GasNET should work over the
> psm provider.  Unfortunately, the developer who ported GasNET to OFI
> is not available.
> 
> - Sean
> 
> 
> > -----Original Message-----
> > From: Sayan Ghosh [mailto:zsayanz at gmail.com]
> > Sent: Friday, June 22, 2018 4:39 PM
> > To: Hefty, Sean <sean.hefty at intel.com>
> > Cc: ofiwg at lists.openfabrics.org
> > Subject: Re: [ofiwg] GASNet with OFI on PSM provider
> >
> > Hi Sean - Sorry if it is evident, but I just wanted to clarify
> > something. Since OFI has a PSM provider, can I assume that it works?
> I
> > am asking because you said "OFI targets Omni Path".
> >
> > On Wed, Jun 20, 2018 at 12:08 PM Hefty, Sean <sean.hefty at intel.com
> > <mailto:sean.hefty at intel.com> > wrote:
> >
> >
> > 	To clarify, I meant GASNet to TrueScale through OFI.  GASNet
> should
> > have a psm-conduit.  OFI targets Omni-Path.
> >
> >
> >
> > 	- Sean
> >
> >
> >
> >
> >
> >
> >
> > 	This is my understanding as well.  I don’t believe GASNet is a
> target
> > application for TrueScale or PSM.  The high performance interface
> for
> > TrueScale is PSM, rather than verbs, and targets older MPI
> > implementations.
> >
> >
> >
> > 	- Sean
> >
> >
> >
> >
> >
> > 	From: ofiwg [mailto:ofiwg-bounces at lists.openfabrics.org] On
> Behalf Of
> > Sayan Ghosh
> > 	Sent: Wednesday, June 20, 2018 9:17 AM
> > 	To: ofiwg at lists.openfabrics.org
> > <mailto:ofiwg at lists.openfabrics.org>
> > 	Subject: Re: [ofiwg] GASNet with OFI on PSM provider
> >
> >
> >
> > 	Just wanted to clarify that a discussion with GASNet developers
> > revealed that their PSM provider compatibility over OFI is only for
> > PSM2 (since TrueScale is an old technology).
> >
> > 	https://groups.google.com/a/lbl.gov/forum/#!topic/gasnet-
> > users/1Y_mDmdcCDw
> >
> >
> >
> > 	On Tue, Jun 19, 2018 at 10:44 PM Sayan Ghosh <zsayanz at gmail.com
> > <mailto:zsayanz at gmail.com> > wrote:
> >
> > 		I am trying to install GASNet 1.30.0 using OFI on an Intel
> TrueScale
> > network (PSM provider), and although OFI fi_info detects PSM
> provider
> > (see attached output of fi_info), when I mention it during GASNet
> > configure (config.log attached, please see line #10126), GASNet
> > warns:
> >
> >
> >
> > 		WARNING: Invalid or unsupported OFI provider specified via
> the
> > --with-ofi-provider flag.
> >
> > 		Defaulting to runtime provider feature detection for ofi-
> conduit.
> >
> >
> >
> >
> > 		GASNet configuration:
> >
> >
> >
> > 		CC=icc CXX=icpc MPI_CC=/home/sghosh/builds/mvapich222-
> > intel-psm/bin/mpicc MPI_CFLAGS="-I/home/sghosh/builds/mvapich222-
> > intel-psm/include" MPI_LIBS="-L/home/sghosh/builds/mvapich222-intel-
> > psm/lib -lpciaccess -lnuma -lxml2" ./configure --
> > prefix=/home/sghosh/builds/gasnet-ofi --enable-ofi --with-ofi-
> > home=/home/sghosh/builds/libfabric --disable-psm --disable-ibv --
> with-
> > ofi-spawner=mpi --with-ofi-provider=psm
> >
> >
> >
> > 		GASNet throws this warning during application execution:
> >
> >
> >
> > 		WARNING: Using OFI provider (verbs;ofi_rxm), which has not
> been
> > validated to provide
> >
> > 		WARNING: acceptable GASNet performance. You should
> consider using a
> > more
> >
> > 		WARNING: hardware-appropriate GASNet conduit. See ofi-
> > conduit/README.
> >
> > 		*** FATAL ERROR: The statically-determined value for
> > GASNETC_OFI_HAS_MR_SCALABLE=1 does
> >
> > 		  not match the memory registration support that the
> > (verbs;ofi_rxm) provider reported.
> >
> > 		  This could happen if a provider that previously only
> supported
> > FI_MR_BASIC
> >
> > 		  added support for FI_MR_SCALABLE, or if the wrong
> provider was
> > selected at runtime.
> >
> > 		  Use configure option --disable-ofi-mr-scalable to
> correct this.
> >
> >
> >
> > 		Instead of picking up psm, GASNet picks up verbs. I have
> reported
> > this issue on GASNet user list as well. Please let me know if I can
> > try some options in building libfabrics for PSM provider, that may
> fix
> > this issue.
> >
> >
> >
> > 		--
> >
> > 		Thanks,
> >
> > 		Sayan | https://sg0.github.io/
> >
> >
> >
> >
> >
> >
> > 	--
> >
> > 	Sayan | https://sg0.github.io/
> >
> >
> >
> > --
> >
> > Sayan | https://sg0.github.io/
> 
> _______________________________________________
> ofiwg mailing list
> ofiwg at lists.openfabrics.org
> http://lists.openfabrics.org/mailman/listinfo/ofiwg