[ewg] Re: [ofa-general] OFED 1.2 Feb-26 meeting summary

[Eric W. Biederman]

"Jeff Squyres" <jsquyres at cisco.com> writes:

> To be totally clear, there are three issues:
>
> 1. *NOT AN MPI ISSUE*: base location of the stack.  Doug has  repeatedly
> mentioned that /usr/local/ofed is not good.  This is a  group issue to decide.

If you are really supporting linux you have two choices:
/usr or /opt/ofed/  (assuming you register /opt/ofed with LANNA) anything else
is wrong by definition.

How hard is this to change?  If it isn't too bad this should probably be changed
as soon as possible.   ABI issues suck and should be fixed as soon as you can.
The install path and config file path is an ABI issue.

> 2. *NOT AN MPI ISSUE*: how the RPMs are built is Bad(tm).  Not  deleting the
> buildroot is Bad; munging %build into %install is  Bad; ...etc.  This needs to
> change.  4 choices jump to mind:
>
>     a. Keep the same scheme.  Ick.
>     b. Install while we build (i.e., the normal way to build a pile  of
> interdependent RPMs)
>     c. Use chroot (Red Hat does this in their internal setup, for  example)
>     d. Only distribute binary RPMs for supported platforms; source is available
> for those who want it.

e.  Give up and building RPM's and let the distributions do it.

I think e is the most common solution and what distributions are doing now.
The only problem with it is that ofed may be evolving to fast to reasonably
expect the distributions to keep up.

Of course the ideal build scenario looks something like:
for source in *.tgz ; do rpmbuild -tb $source ; rpm -i ? ; done

Where the source tarballs are have a spec file in them that can be used
to build an rpm.

Sorting this out needs to happen but likely is  something that ugly bits
can be lived with.

> 3. Doug's final point about allowing multiple MPI's to play  harmoniously on a
> single system is obviously an MPI issue.  The /etc/ alternatives mechanism is
> not really good enough (IMHO) for this -- / etc/alternatives is about choosing
> one implementation and making  everyone use it.  The problem is that when
> multiple MPI's are  installed on a single system, people need all of them (some
> users  prefer one over the other, but much more important, some apps are  only
> certified with one MPI or another).  The mpi-selector tool we  introduced in
> OFED 1.2 will likely be "good enough" for this purpose, but we can also work on
> integrating the /etc/alternatives stuff if  desired, particularly for those who
> only need/want one MPI  implementation.

Agreed.  There are a few other issues here as well.

There seems to be no agreement on a fortran ABI for linux.  Even little
things like f77 and g90 are incompatible.   Or was their an ABI agreement
recently and I missed it?

When providing MPI fortran bindings that is a problem because you need
to compile separately for each different installed compiler, possibly even
for different versions of the same compiler.

Which makes even an /opt solution not quite good enough because you need multiple
version of the same mpi built with different compilers.

Eric