[ofa-general] RFC: Do we wish to take MPI out of OFED?

[Jeff Squyres]

I'll state right up front that I'm in favor of removing MPI from  
OFED.  :-)


On Jun 2, 2009, at 3:30 PM, Tziporet Koren wrote:

> Here are some of the reasons for keeping MPI in or out of OFED.
>
> Main reasons to take MPI out of OFED:
> - MPI is not developed under OFA
> - Need to synchronize between different projects
>
   --> This point alone is hugely difficult.  OFED has been held up  
multiple times because of MPI release delays.  I would like to stress  
that the open source MPI implementations are entirely different  
software projects with minimal overlap of personnel between MPI and  
OFED.  Also, OFED is but one of the stacks that the MPI's support.

For example, many Open MPI members don't care about OpenFabrics at  
all.  It has sometimes been difficult to rationalize MPI schedule  
adjustments because of OFED.

> - Some customers prefer to install a different MPI version then the  
> one in OFED
>
- Also note that OFED only includes the open source MPI's; it's not a  
level playing field.

- Several of Open MPI's features are disabled in OFED builds (e.g.,  
scheduler / resource manager support), causing customers to re- 
download / re-build Open MPI anyway.  *** This is a fairly important  
point to us.


> Main reasons to keep MPI in OFED:
> - All participants test with the same MPI versions, and when  
> installing OFED it is ensured that MPI will work fine with this  
> version.
>
   --> I think we're all convinced that testing OFED with the various  
MPI implementations is a Good Thing -- nobody is suggesting that we  
remove that.  As discussed on the call, we can certainly all test the  
same version of the open source MPI's during OFED releases and include  
this information in the release notes ("OFED x.y.z was tested with  
Open MPI a.b.c.").

The Open MPI project would be happy to contribute the MPI Testing Tool  
(MTT; which itself is also open source) to OFED.  The MTT was designed  
for almost exactly this purpose: a large, distributed set of  
organizations that all need to test the same versions of MPI and  
report the results to a central location.  The MTT is not specific to  
Open MPI; it can be used with any MPI implementation.  The MTT is  
essentially an engine to download/install MPI implementations, run a  
set of MPI tests (e.g., IMB, OSU and others), and then report the  
results to a central database.

FWIW: several OFA members are using MTT internally for their own MPI  
testing.

> - Customers convenience in install (no need to go to more sites to  
> get MPI)
>
   --> My $0.02 is that this is a dubious point.  In installing a  
production HPC cluster, you're getting 20 pieces of software and  
installing them together.  So if you have to get OFED *and* MPI (i.e.,  
21 pieces of software), I think it makes little difference.  Indeed,  
as noted above, MPI implementations move at a different speed than  
OFED, so many customers go install their own MPI's anyway (or, per an  
above point, need to re-install MPI's to enable features that the OFED  
packaging disables).

I think the biggest issue related to this point is going to be  
customers asking "you *used* to bundle MPI, why don't do anymore?"

> - MPI is an important RDMA ULP and although it is not developed in  
> OFA it is widely used by OFED customers
>
   --> This point is actually used as a crutch by the OFA: why develop  
your own comprehensive performance testing / monitoring tools when you  
have MPI with all of its tools?  Removing MPI may help motivate the  
OFA to have better networking tools than those provided by a ULP.   
(before you flame, I admit that this is a minor point, but it is still  
worth noting -- I've always thought it strange that people use MPI to  
monitor and test their OF-based networks...)

-- 
Jeff Squyres
Cisco Systems