[ewg] Re: [ofa-general] OFED 1.2 Feb-26 meeting summary

[Jeff Squyres]

To be totally clear, there are three issues:

1. *NOT AN MPI ISSUE*: base location of the stack.  Doug has  
repeatedly mentioned that /usr/local/ofed is not good.  This is a  
group issue to decide.

2. *NOT AN MPI ISSUE*: how the RPMs are built is Bad(tm).  Not  
deleting the buildroot is Bad; munging %build into %install is  
Bad; ...etc.  This needs to change.  4 choices jump to mind:

    a. Keep the same scheme.  Ick.
    b. Install while we build (i.e., the normal way to build a pile  
of interdependent RPMs)
    c. Use chroot (Red Hat does this in their internal setup, for  
example)
    d. Only distribute binary RPMs for supported platforms; source is  
available for those who want it.

3. Doug's final point about allowing multiple MPI's to play  
harmoniously on a single system is obviously an MPI issue.  The /etc/ 
alternatives mechanism is not really good enough (IMHO) for this -- / 
etc/alternatives is about choosing one implementation and making  
everyone use it.  The problem is that when multiple MPI's are  
installed on a single system, people need all of them (some users  
prefer one over the other, but much more important, some apps are  
only certified with one MPI or another).  The mpi-selector tool we  
introduced in OFED 1.2 will likely be "good enough" for this purpose,  
but we can also work on integrating the /etc/alternatives stuff if  
desired, particularly for those who only need/want one MPI  
implementation.


On Feb 28, 2007, at 12:56 PM, Doug Ledford wrote:

> On Wed, 2007-02-28 at 16:10 +0200, Tziporet Koren wrote:
>>       * Improved RPM usage by the install will not be part of OFED
>>         1.2
>
> Since I first brought this up, you have added new libraries, iWARP
> support, etc.  These constitute new RPMs.  And, because you guys have
> been doing things contrary to standards like the file hierarchy  
> standard
> in the original RPMs, it's been carried forward to these new RPMs.   
> This
> is a snowball, and the longer you put off fixing it, the harder it  
> gets
> to change.  And not just in your RPMs either.  The longer you put off
> coming up with a reasonable standard for MPI library and executable  
> file
> locations, the longer customers will hand roll their own site specific
> setups, and the harder it will be to get them to switch over to the
> standard once you *do* implement it.  You may end up dooming Jeff to
> maintaining those custom file location hacks in the OpenMPI spec
> forever.
>
> Not to mention that interoperability is about more than one machine
> talking to another machine.  It's also about a customer's application
> building properly on different versions of the stack, without the
> customer needing to change all the include file locations and link
> parameters.  It's also about a customer being able to rest assured  
> that
> if they tried to install two conflicting copies of libibverbs, it  
> would
> in fact cause RPM to throw conflict errors (which it doesn't now  
> because
> your libibverbs is in /usr/local, where I'm not allowed to put  
> ours, so
> since the files are in different locations, rpm will happily let the
> user install both your libibverbs and my libibverbs without a  
> conflict,
> and a customer could waste large amounts of time trying to track  
> down a
> bug in one library only to find out their application is linking  
> against
> the other).
>
>>               * The RPM usage will be enhanced for the next (1.3)
>>                 release and we will decide on the correct way in
>>                 Sonoma.
>
>
>
> There's not really much to decide.  Either the stack is Linux File
> Hierarchy Standard compliant or it isn't.  The only leeway for  
> decisions
> allowed by the standard is on things like where in /etc to put the
> config files (since you guys are striving to be a generic RDMA stack,
> not just an IB stack, I would suggest that all RDMA related config  
> files
> go into /etc/rdma, and for those applications that can reasonably  
> be run
> absent RDMA technology, like OpenMPI, I would separate their config
> files off into either /etc or /etc/openmpi, ditto for the include
> directories, /usr/include/rdma for the generic non-IB specific stuff,
> and possibly /usr/include/rdma/infiniband for IB specific stuff, or  
> you
> could put the IB stuff under /usr/include/infiniband, either way).
>
> The biggest variation from the spec that needs to be dealt with is the
> need for multiple MPI installations, which is problematic if you just
> use generic locations as it stands today, but with a few modifications
> to the MPI stack it could be worked around.
>
>
> -- 
> Doug Ledford <dledford at redhat.com>
>               GPG KeyID: CFBFF194
>               http://people.redhat.com/dledford
>
> Infiniband specific RPMs available at
>               http://people.redhat.com/dledford/Infiniband
> _______________________________________________
> general mailing list
> general at lists.openfabrics.org
> http://lists.openfabrics.org/cgi-bin/mailman/listinfo/general
>
> To unsubscribe, please visit http://openib.org/mailman/listinfo/ 
> openib-general


-- 
Jeff Squyres
Server Virtualization Business Unit
Cisco Systems