[ofiwg] GASNet with OFI on PSM provider

[Hefty, Sean]

To clarify, I meant GASNet to TrueScale through OFI.  GASNet should have a psm-conduit.  OFI targets Omni-Path.

- Sean


This is my understanding as well.  I don’t believe GASNet is a target application for TrueScale or PSM.  The high performance interface for TrueScale is PSM, rather than verbs, and targets older MPI implementations.

- Sean


From: ofiwg [mailto:ofiwg-bounces at lists.openfabrics.org] On Behalf Of Sayan Ghosh
Sent: Wednesday, June 20, 2018 9:17 AM
To: ofiwg at lists.openfabrics.org<mailto:ofiwg at lists.openfabrics.org>
Subject: Re: [ofiwg] GASNet with OFI on PSM provider

Just wanted to clarify that a discussion with GASNet developers revealed that their PSM provider compatibility over OFI is only for PSM2 (since TrueScale is an old technology).
https://groups.google.com/a/lbl.gov/forum/#!topic/gasnet-users/1Y_mDmdcCDw

On Tue, Jun 19, 2018 at 10:44 PM Sayan Ghosh <zsayanz at gmail.com<mailto:zsayanz at gmail.com>> wrote:
I am trying to install GASNet 1.30.0 using OFI on an Intel TrueScale network (PSM provider), and although OFI fi_info detects PSM provider (see attached output of fi_info), when I mention it during GASNet configure (config.log attached, please see line #10126), GASNet warns:

WARNING: Invalid or unsupported OFI provider specified via the --with-ofi-provider flag.
Defaulting to runtime provider feature detection for ofi-conduit.

GASNet configuration:

CC=icc CXX=icpc MPI_CC=/home/sghosh/builds/mvapich222-intel-psm/bin/mpicc MPI_CFLAGS="-I/home/sghosh/builds/mvapich222-intel-psm/include" MPI_LIBS="-L/home/sghosh/builds/mvapich222-intel-psm/lib -lpciaccess -lnuma -lxml2" ./configure --prefix=/home/sghosh/builds/gasnet-ofi --enable-ofi --with-ofi-home=/home/sghosh/builds/libfabric --disable-psm --disable-ibv --with-ofi-spawner=mpi --with-ofi-provider=psm

GASNet throws this warning during application execution:

WARNING: Using OFI provider (verbs;ofi_rxm), which has not been validated to provide
WARNING: acceptable GASNet performance. You should consider using a more
WARNING: hardware-appropriate GASNet conduit. See ofi-conduit/README.
*** FATAL ERROR: The statically-determined value for GASNETC_OFI_HAS_MR_SCALABLE=1 does
  not match the memory registration support that the (verbs;ofi_rxm) provider reported.
  This could happen if a provider that previously only supported FI_MR_BASIC
  added support for FI_MR_SCALABLE, or if the wrong provider was selected at runtime.
  Use configure option --disable-ofi-mr-scalable to correct this.

Instead of picking up psm, GASNet picks up verbs. I have reported this issue on GASNet user list as well. Please let me know if I can try some options in building libfabrics for PSM provider, that may fix this issue.

--
Thanks,
Sayan | https://sg0.github.io/


--
Sayan | https://sg0.github.io/
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.openfabrics.org/pipermail/ofiwg/attachments/20180620/4a656976/attachment.html>