[openfabrics-ewg] Default and per-user MPI selections

[Jeff Squyres]

On 8/29/06 9:32 AM, "Steffen Persvold" <sp at scali.com> wrote:

> What's so difficult with doing :
> 
> $ . /opt/scali/etc/scampivars.sh

That's fine for the local node, but it does not necessarily work for the
remote nodes when you mpirun in rsh/ssh environments (e.g., Open MPI and
MVAPICH2 require the ability to find helper applications on remote nodes).
 
> In the HPC community, people also like to use "modules" :
> 
> $ module load mpi/scali

Right -- this is the environment modules/switcher solution that I mentioned.
Environment modules are great, but they are also a local-only solution.  The
"switcher" package that I mentioned is a trivial persistence layer on top of
environment modules.
 
>> -----Original Message-----
>> From: openfabrics-ewg-bounces at openib.org [mailto:openfabrics-ewg-
>> bounces at openib.org] On Behalf Of Jeff Squyres
>> Sent: Tuesday, August 29, 2006 9:14 AM
>> To: Open Fabrics
>> Subject: [openfabrics-ewg] Default and per-user MPI selections
>> 
>> All --
>> 
>> I think we need to re-visit the which-MPI-to-put-in-the-PATH issue.
>> Others
>> have previously expressed [strong] opinions that no MPI should be in
>> the
>> path; sysadmins and/or users can set their PATH/LD_LIBRARY_PATH however
>> they
>> want.  I disagree with this for two reasons:
>> 
>> 1. Editing shell startup files is trivial for IT
>> professionals/developers,
>> but can be a daunting task for a chemical engineer who doesn't
>> know/care how
>> the system works and just wants to run his/her HPC programs to get
>> results.
>> 
>> 2. The job of the OFED installer is to make it *easier* to run HPC
>> applications.  OFED does everything *except* put MPI in your path --
>> yet MPI
>> is the most highly visible protocol used in HPC.  It seems pretty weird
>> (to
>> me) that we don't go the final few steps to provide a complete
>> solution.
>> 
>> What do the Linux vendors will feel about installing software that is
>> intended for general use but is not put in the PATH/LD_LIBRARY_PATH?
>> 
>> I think that OFED should have the capability of setting a system-wide
>> default MPI and/or setting a per-user default MPI.  During
>> installation,
>> OFED should ask the sysadmin if they want a system-wide default MPI,
>> and if
>> so, which one.  And then do the Right magic to make it so.  I also
>> think
>> that OFED should provide a per-user mechanism to override the system-
>> default
>> MPI (something better than "edit your shell startup files and change
>> your
>> PATH/LD_LIBRARY_PATH").
>> 
>> -----
>> 
>> Doug Ledford has proposed a solution -- that the MPI implementations
>> use the
>> "alternatives" system (is "alternatives" also available in SLES?).
>> This
>> would require:
>> 
>> A) the MPIs to be installed in non-default locations (e.g.,
>> /opt/openmpi*
>> and /opt/mvapich*)
>>    --> This will require changing the builder scripts to allow
>> different
>> prefix values for OFED and each MPI installation (e.g., /usr for OFED,
>> /opt/openmpi-gnu for Open MPI compiled by GNU, /opt/openmpi-intel for
>> Open
>> MPI compiled by the Intel compilers, etc.).
>>    --> The installer should also prompt as to which MPI should be the
>> default for the system (if any).
>> 
>> B) use the "alternatives" mechanism to allow users to choose which MPI
>> to
>> use as the system default (i.e., would show up in /usr/bin and the
>> like)
>>    --> This will require changing the specfile for both Open MPI and
>> MVAPICH.  Both MPI implementations' specfiles must be done for this to
>> be
>> meaningful.
>> 
>> Note that the "alternatives" system does not have the inherent ability
>> to
>> allow users to choose which MPI to use; "alternatives" only seems to
>> set a
>> system-wide default.  If a user wants to use a different MPI, they can
>> prepend their PATH / LD_LIBRARY_PATH / etc. to avoid the one that is
>> found
>> in system default paths (e.g., /usr/bin).
>> 
>> -----
>> 
>> Another option is to use something like environment modules and the
>> "switcher" package from OSCAR to have both a system-wide default for
>> MPI and
>> allow users to et which MPI they want to use (if they want to override
>> the
>> system-wide default) without needing to edit their shell startup files.
>> 
>> The changes for using modules/switcher are similar to those for using
>> "alternatives" (i.e., both options will take work to implement).  There
>> are
>> other systems available that can be used as well (softenv comes to
>> mind; I
>> think there might be others, too).
>> 
>> -----
>> 
>> Comments?
>> 
>> (obviously, this is not for OFED 1.1 -- this would be a 1.2 issue)
>> 
>> --
>> Jeff Squyres
>> Server Virtualization Business Unit
>> Cisco Systems
>> 
>> _______________________________________________
>> openfabrics-ewg mailing list
>> openfabrics-ewg at openib.org
>> http://openib.org/mailman/listinfo/openfabrics-ewg


-- 
Jeff Squyres
Server Virtualization Business Unit
Cisco Systems