[openib-general] MVAPICH2 SRPM and install file patches

[Shaun Rowland]

Scott Weitzenkamp (sweitzen) wrote:
> Shaun,
> 
> Thanks for doing this.
> 
> I see things like romio and shlibs configurable in the patch, what about
> other MVAPICH2 features like fault tolerance, multi rail, threads, and
> MPD?  How can configure them when I use install.sh to compile and
> install OFED?

Hi Scott. I had thought about this a little when I was testing with the
install/build scripts Vlad gave us. I would appreciate his input if I
get anything wrong here as well. From the perspective of the user
running the install.sh script, the MPI packages are essentially built
one way. You do get to pick the compiler(s) to use, but as for other
options - you would have to edit the build.sh function associated with
the desired package. I created a hack for the iwarp vs ib configuration
for MVAPICH2 because I needed to distinguish between the two (for
reasons I will outline at the end of this message).

Theoretically, you should be able to export the proper variables from
our make.mvapich2.* scripts before running the install.sh script, and
the features would be enabled. For instance, you could do:

export MULTI_THREAD=yes
./install.sh

This is not a good solution for installing OFED, but should work due to
not conflicting with anything else - at least that I am aware. I see
that I need to update the make.mvapich2.iwarp script to have the
multithreading option anyway as well, so it would not quite work 100%
right now.

As far as each feature you asked about:

* fault tolerance
      - this is controlled during the build process with $ENABLE_CKPT 
and requires $BLCR_HOME pointing to a BLCR installation.  This only 
works for single threaded builds without rdmacm support (the ib case 
only, essentially).

* multi rail
      - this is controlled by runtime environment variables after 
installation.

* threads
      - This is controlled by $MULTI_THREAD during the build process. 
As noted above, there's a restriction with fault tolerance.

* MPD
      - MPD is used by MVAPICH2 as it is based on MPICH2.

There are actually a number of options that could be chosen. I believe
from our side, it will be good for me to go ahead and put these in our
SRPM now. Our SRPM can be used outside of the OFED installation system
of course, and these should really be there. There are even other
"devices", like uDAPL.

I did the SRPM in the install/build script patches the way I did
because that seemed like a good set of options for how the OFED
installation system works. There's no framework or examples of asking
about features to build in an MPI package. I just quickly tacked on the
iwarp question and made up a new configuration variable for the
ofed.conf file, but it's not necessarily a good way to do it.

One possibility would be to create a shell function that sets various
build options for MPI packages. Variables could be set in this function
using some name convention, in our case perhaps MVAPICH2_OPT_<whatever>.
In such a function (probably one for each package, that seems to be the
convention), it would be easier to code all the exceptions for features
- if there are any. There are some in our case, as I've mentioned. This
configuration function could be called when the user is choosing to
install MVAPICH2.

This leads to a number of problems. Can the user select different
options for each of the compiler versions of the MPI package? I think
clearly the answer should be "no". Even as implemented now, you cannot
install the iwarp and ib version of MVAPICH2 at the same time during the
install process. You must choose one or the other. Being able to do
either would require one of two changes:

1. Having another level of installer system configuration where I could
selected the devices desired, and options for each device (by device
here, I mean uDAPL, IB, iWARP).

- or -

2. Make multiple RPM packages to fit into how the installer currently
interacts with SRPMs, prompts, etc.

I've only had a limitted time to investigate this, so what I have done
so far mostly fits with how the OFED install system does things with the
other packages - except for my iwarp vs ib question prompt. I think
there's potential for a lot of compilication here. A configuration
function for each package would be one possible way to contain that,
however I'd have to go back and check out how things work again to see
how something like that would fit in.

So, I will add these new feature options to our SRPM because they could
be used outside of the OFED installation system anyway, and we would
like that to be possible and give the ability to set these options.
However, I cannot say what would be best for the OFED installation
system. It might be better to just go with what we have now - more
"mainstream" builds, and let the user do their own build if they want to
highly customize or something. Otherwise, I've given one possible idea
from the perspective of someone who is new to the install system. Vlad,
do you have any opinion here? Do you see where I am coming from as far
as what kind of situation we are talking about with presenting options
for MPI package builds?

> I also didn't quite understand the ib-vs-iwarp configuration, I thought
> OFED 1.2 would support both.

There are 2 reasons our SRPM has to be told whether it is being built
for iWARP or IB:

1. We need to use -DRDMA_CM_RNIC during the build for iWARP (this is
actually done by invoking our make.mvapich2.iwarp script in the RPM build).

2. If the %auto_req macro is set to 0, then simple RPM names for the
install requirements are used:

Autoreq: 0
Requires: libibumad libibverbs [default]
Requires: libibumad libibverbs librdmacm [iWARP]

This is actually not done, but it is there as a possibility (Autoreq is
used right now I mean): Vlad, I was thinking that you might want to
change our function in build.sh to set auto_req to 0 instead of 1. I see
that is how MVAPICH is doing requires, instead of letting Autoreq do it.
I think it will work either way probably, but using --define 'auto_req
0' will probably cut down on some potential issues. I had set it to 1
because I saw in OFED 1.1 it seemed that this was how things worked.
-- 
Shaun Rowland	rowland at cse.ohio-state.edu
http://www.cse.ohio-state.edu/~rowland/