[ofa-general] OFED-1.1, mvapich and openmpi question

[Jeff Squyres]

On Mar 9, 2007, at 10:32 AM, Bala wrote:

>        Recently we have installed OFED-1.1 on our
> 16 node blade cluster, as we are new to IB we
> have the following queries
>
> 1. For openmpi we have used the "--mca btl openib,self
> ..." as command line option and it is
> running fine, now the question is how to make sure
> the it uses IB for mpi communication??

This parameter tells Open MPI to only use the "openib" and "self"  
plugins for MPI communications.  The "openib" plugin uses the  
OpenFabrics stack; the "self" plugin is used for loopback (send-to- 
self) communication.  If you want to use shared memory for on-host  
communication, add "sm" into the list:

   --mca btl sm,openib,self

See http://www.open-mpi.org/faq/ for more information.

> 2. Is there any tool available to find activity over
>   IB??

I think Hal answered that.

> 3. Is there any option available for "mvapich" to use
>   IB??

The MVAPICH guys will have to answer that.

> 4. we are using cpi.c sample is there any other
>   sample we can use for testing as well as benchmark??

There's lots of other benchmarks out there; I personally like NetPIPE  
as a ping-pong test, but OSU has some benchmarks as well.

> 5. shall we use this with SGE(Sun Grid Engine)
> scheduler??, anybody already using SGE for
> openmpi and mvapich??

Open MPI v1.2 debuts support for SGE, but also supports a variety of  
other schedulers (Torque, SLURM, ...etc.).

-- 
Jeff Squyres
Cisco Systems