[openfabrics-ewg] Default and per-user MPI selections

[Scott Weitzenkamp (sweitzen)]

(Jeff and I have had this discussion a few times offline...)

I don't disagree with you about PATH.  But LD_LIBRARY_PATH is an Open
MPI issue, as MVAPICH does not have the problem.

For your hypothetical chemical engineer, can't the person who
installed/verified OFED on their cluster also set up their account with
the right PATH?

Scott Weitzenkamp
SQA and Release Manager
Server Virtualization Business Unit
Cisco Systems
 

> -----Original Message-----
> From: openfabrics-ewg-bounces at openib.org 
> [mailto:openfabrics-ewg-bounces at openib.org] On Behalf Of Jeff 
> Squyres (jsquyres)
> Sent: Tuesday, August 29, 2006 6:14 AM
> To: Open Fabrics
> Subject: [openfabrics-ewg] Default and per-user MPI selections
> 
> All --
> 
> I think we need to re-visit the which-MPI-to-put-in-the-PATH 
> issue.  Others
> have previously expressed [strong] opinions that no MPI 
> should be in the
> path; sysadmins and/or users can set their 
> PATH/LD_LIBRARY_PATH however they
> want.  I disagree with this for two reasons:
> 
> 1. Editing shell startup files is trivial for IT 
> professionals/developers,
> but can be a daunting task for a chemical engineer who 
> doesn't know/care how
> the system works and just wants to run his/her HPC programs 
> to get results.
> 
> 2. The job of the OFED installer is to make it *easier* to run HPC
> applications.  OFED does everything *except* put MPI in your 
> path -- yet MPI
> is the most highly visible protocol used in HPC.  It seems 
> pretty weird (to
> me) that we don't go the final few steps to provide a 
> complete solution.
> 
> What do the Linux vendors will feel about installing software that is
> intended for general use but is not put in the PATH/LD_LIBRARY_PATH?
> 
> I think that OFED should have the capability of setting a system-wide
> default MPI and/or setting a per-user default MPI.  During 
> installation,
> OFED should ask the sysadmin if they want a system-wide 
> default MPI, and if
> so, which one.  And then do the Right magic to make it so.  I 
> also think
> that OFED should provide a per-user mechanism to override the 
> system-default
> MPI (something better than "edit your shell startup files and 
> change your
> PATH/LD_LIBRARY_PATH").
> 
> -----
> 
> Doug Ledford has proposed a solution -- that the MPI 
> implementations use the
> "alternatives" system (is "alternatives" also available in 
> SLES?).  This
> would require:
> 
> A) the MPIs to be installed in non-default locations (e.g., 
> /opt/openmpi*
> and /opt/mvapich*)
>    --> This will require changing the builder scripts to 
> allow different
> prefix values for OFED and each MPI installation (e.g., /usr for OFED,
> /opt/openmpi-gnu for Open MPI compiled by GNU, 
> /opt/openmpi-intel for Open
> MPI compiled by the Intel compilers, etc.).
>    --> The installer should also prompt as to which MPI should be the
> default for the system (if any).
> 
> B) use the "alternatives" mechanism to allow users to choose 
> which MPI to
> use as the system default (i.e., would show up in /usr/bin 
> and the like)
>    --> This will require changing the specfile for both Open MPI and
> MVAPICH.  Both MPI implementations' specfiles must be done 
> for this to be
> meaningful.
> 
> Note that the "alternatives" system does not have the 
> inherent ability to
> allow users to choose which MPI to use; "alternatives" only 
> seems to set a
> system-wide default.  If a user wants to use a different MPI, they can
> prepend their PATH / LD_LIBRARY_PATH / etc. to avoid the one 
> that is found
> in system default paths (e.g., /usr/bin).
> 
> -----
> 
> Another option is to use something like environment modules and the
> "switcher" package from OSCAR to have both a system-wide 
> default for MPI and
> allow users to et which MPI they want to use (if they want to 
> override the
> system-wide default) without needing to edit their shell 
> startup files.
> 
> The changes for using modules/switcher are similar to those for using
> "alternatives" (i.e., both options will take work to 
> implement).  There are
> other systems available that can be used as well (softenv 
> comes to mind; I
> think there might be others, too).
> 
> -----
> 
> Comments?
> 
> (obviously, this is not for OFED 1.1 -- this would be a 1.2 issue)
> 
> -- 
> Jeff Squyres
> Server Virtualization Business Unit
> Cisco Systems
> 
> _______________________________________________
> openfabrics-ewg mailing list
> openfabrics-ewg at openib.org
> http://openib.org/mailman/listinfo/openfabrics-ewg
>