[libfabric-users] OFI in combination with MPI

[Peter Boyle]

Hi,

I’m interested in trying to minimally modify a scientific library

	www.github.com/paboyle/Grid/

to tackle obtaining good dual rail OPA performance on KNL
from a single process per node. 

The code is naturally hybrid OpenMP + MPI and 1 process per node minimises internal
communication in the node so would be the preferred us.

The application currently has (compile time selection) SHMEM and MPI transport layers, but it appears
that the MPI versions we have tried (OpenMPI, MVAPICH2, Intel MPI) have a big lock and no 
real multi-core concurrency from a single process.

Is there any wisdom about either

i) should SHMEM suffice to gain concurrency from single multithreaded process, in a way that MPI does not

ii) would it be better to drop to OFI (using the useful SHMEM tutorial on GitHub as an example)

iii) Can job invocation rely on mpirun for job load, and use MPI (or OpenSHMEM) for exchange of network address, but then

     a) safely fire up OFI endpoints and use them instead of MPI

     b) safely fire up OFI endpoints and use them alongside and as well as MPI

Advice appreciated.

Peter


-- 
The University of Edinburgh is a charitable body, registered in
Scotland, with registration number SC005336.