[ofiwg] GASNet with OFI on PSM provider

[Hefty, Sean]

OFI does work over PSM.  However, the psm and psm2 providers do not support the same OFI feature set.  You can see the differences here:

https://github.com/ofiwg/libfabric/wiki/Provider-Feature-Matrix-master

I'm not familiar enough with the GasNET implementation to know what feature it needs.

I'm copying the developer who maintains both the psm and psm2 providers.  He can provide more details on what differences there are between psm and psm2, and *might* know if GasNET should work over the psm provider.  Unfortunately, the developer who ported GasNET to OFI is not available.

- Sean


> -----Original Message-----
> From: Sayan Ghosh [mailto:zsayanz at gmail.com]
> Sent: Friday, June 22, 2018 4:39 PM
> To: Hefty, Sean <sean.hefty at intel.com>
> Cc: ofiwg at lists.openfabrics.org
> Subject: Re: [ofiwg] GASNet with OFI on PSM provider
> 
> Hi Sean - Sorry if it is evident, but I just wanted to clarify
> something. Since OFI has a PSM provider, can I assume that it works? I
> am asking because you said "OFI targets Omni Path".
> 
> On Wed, Jun 20, 2018 at 12:08 PM Hefty, Sean <sean.hefty at intel.com
> <mailto:sean.hefty at intel.com> > wrote:
> 
> 
> 	To clarify, I meant GASNet to TrueScale through OFI.  GASNet
> should have a psm-conduit.  OFI targets Omni-Path.
> 
> 
> 
> 	- Sean
> 
> 
> 
> 
> 
> 
> 
> 	This is my understanding as well.  I don’t believe GASNet is a
> target application for TrueScale or PSM.  The high performance
> interface for TrueScale is PSM, rather than verbs, and targets older
> MPI implementations.
> 
> 
> 
> 	- Sean
> 
> 
> 
> 
> 
> 	From: ofiwg [mailto:ofiwg-bounces at lists.openfabrics.org] On
> Behalf Of Sayan Ghosh
> 	Sent: Wednesday, June 20, 2018 9:17 AM
> 	To: ofiwg at lists.openfabrics.org
> <mailto:ofiwg at lists.openfabrics.org>
> 	Subject: Re: [ofiwg] GASNet with OFI on PSM provider
> 
> 
> 
> 	Just wanted to clarify that a discussion with GASNet developers
> revealed that their PSM provider compatibility over OFI is only for
> PSM2 (since TrueScale is an old technology).
> 
> 	https://groups.google.com/a/lbl.gov/forum/#!topic/gasnet-
> users/1Y_mDmdcCDw
> 
> 
> 
> 	On Tue, Jun 19, 2018 at 10:44 PM Sayan Ghosh <zsayanz at gmail.com
> <mailto:zsayanz at gmail.com> > wrote:
> 
> 		I am trying to install GASNet 1.30.0 using OFI on an Intel
> TrueScale network (PSM provider), and although OFI fi_info detects PSM
> provider (see attached output of fi_info), when I mention it during
> GASNet configure (config.log attached, please see line #10126), GASNet
> warns:
> 
> 
> 
> 		WARNING: Invalid or unsupported OFI provider specified via
> the --with-ofi-provider flag.
> 
> 		Defaulting to runtime provider feature detection for ofi-
> conduit.
> 
> 
> 
> 
> 		GASNet configuration:
> 
> 
> 
> 		CC=icc CXX=icpc MPI_CC=/home/sghosh/builds/mvapich222-
> intel-psm/bin/mpicc MPI_CFLAGS="-I/home/sghosh/builds/mvapich222-
> intel-psm/include" MPI_LIBS="-L/home/sghosh/builds/mvapich222-intel-
> psm/lib -lpciaccess -lnuma -lxml2" ./configure --
> prefix=/home/sghosh/builds/gasnet-ofi --enable-ofi --with-ofi-
> home=/home/sghosh/builds/libfabric --disable-psm --disable-ibv --with-
> ofi-spawner=mpi --with-ofi-provider=psm
> 
> 
> 
> 		GASNet throws this warning during application execution:
> 
> 
> 
> 		WARNING: Using OFI provider (verbs;ofi_rxm), which has not
> been validated to provide
> 
> 		WARNING: acceptable GASNet performance. You should
> consider using a more
> 
> 		WARNING: hardware-appropriate GASNet conduit. See ofi-
> conduit/README.
> 
> 		*** FATAL ERROR: The statically-determined value for
> GASNETC_OFI_HAS_MR_SCALABLE=1 does
> 
> 		  not match the memory registration support that the
> (verbs;ofi_rxm) provider reported.
> 
> 		  This could happen if a provider that previously only
> supported FI_MR_BASIC
> 
> 		  added support for FI_MR_SCALABLE, or if the wrong
> provider was selected at runtime.
> 
> 		  Use configure option --disable-ofi-mr-scalable to
> correct this.
> 
> 
> 
> 		Instead of picking up psm, GASNet picks up verbs. I have
> reported this issue on GASNet user list as well. Please let me know if
> I can try some options in building libfabrics for PSM provider, that
> may fix this issue.
> 
> 
> 
> 		--
> 
> 		Thanks,
> 
> 		Sayan | https://sg0.github.io/
> 
> 
> 
> 
> 
> 
> 	--
> 
> 	Sayan | https://sg0.github.io/
> 
> 
> 
> --
> 
> Sayan | https://sg0.github.io/