[openfabrics-ewg] Default and per-user MPI selections

[Jeff Squyres]

All --

I think we need to re-visit the which-MPI-to-put-in-the-PATH issue.  Others
have previously expressed [strong] opinions that no MPI should be in the
path; sysadmins and/or users can set their PATH/LD_LIBRARY_PATH however they
want.  I disagree with this for two reasons:

1. Editing shell startup files is trivial for IT professionals/developers,
but can be a daunting task for a chemical engineer who doesn't know/care how
the system works and just wants to run his/her HPC programs to get results.

2. The job of the OFED installer is to make it *easier* to run HPC
applications.  OFED does everything *except* put MPI in your path -- yet MPI
is the most highly visible protocol used in HPC.  It seems pretty weird (to
me) that we don't go the final few steps to provide a complete solution.

What do the Linux vendors will feel about installing software that is
intended for general use but is not put in the PATH/LD_LIBRARY_PATH?

I think that OFED should have the capability of setting a system-wide
default MPI and/or setting a per-user default MPI.  During installation,
OFED should ask the sysadmin if they want a system-wide default MPI, and if
so, which one.  And then do the Right magic to make it so.  I also think
that OFED should provide a per-user mechanism to override the system-default
MPI (something better than "edit your shell startup files and change your
PATH/LD_LIBRARY_PATH").

-----

Doug Ledford has proposed a solution -- that the MPI implementations use the
"alternatives" system (is "alternatives" also available in SLES?).  This
would require:

A) the MPIs to be installed in non-default locations (e.g., /opt/openmpi*
and /opt/mvapich*)
   --> This will require changing the builder scripts to allow different
prefix values for OFED and each MPI installation (e.g., /usr for OFED,
/opt/openmpi-gnu for Open MPI compiled by GNU, /opt/openmpi-intel for Open
MPI compiled by the Intel compilers, etc.).
   --> The installer should also prompt as to which MPI should be the
default for the system (if any).

B) use the "alternatives" mechanism to allow users to choose which MPI to
use as the system default (i.e., would show up in /usr/bin and the like)
   --> This will require changing the specfile for both Open MPI and
MVAPICH.  Both MPI implementations' specfiles must be done for this to be
meaningful.

Note that the "alternatives" system does not have the inherent ability to
allow users to choose which MPI to use; "alternatives" only seems to set a
system-wide default.  If a user wants to use a different MPI, they can
prepend their PATH / LD_LIBRARY_PATH / etc. to avoid the one that is found
in system default paths (e.g., /usr/bin).

-----

Another option is to use something like environment modules and the
"switcher" package from OSCAR to have both a system-wide default for MPI and
allow users to et which MPI they want to use (if they want to override the
system-wide default) without needing to edit their shell startup files.

The changes for using modules/switcher are similar to those for using
"alternatives" (i.e., both options will take work to implement).  There are
other systems available that can be used as well (softenv comes to mind; I
think there might be others, too).

-----

Comments?

(obviously, this is not for OFED 1.1 -- this would be a 1.2 issue)

-- 
Jeff Squyres
Server Virtualization Business Unit
Cisco Systems