[openib-general] mvapich-gen2 on 2 x 16 CPU SGI Altix 1330 cluster

[Dhabaleswar Panda]

Hi John,

> I just though you would like to know that I have now tested the
> Pallas benchmark on a two node SGI Altix 1330 cluster using OpenIB
> and mvapich-gen2. Each node had 16 CPU's. To do this I had to change
> SMPI_MAX_NUMLOCALNODES to be defined as 16 instead of the normal 4
> for the test.  I ran a 2x16 (32 total) CPU Pallas benchmark several
> times with no hang ups or errors.

Very glad to know that you are able to run Pallas with the above
parameter change. We had put that parameter for such multi-way SMP
systems (Altix) in mind.

> I'm wondering if there would be any more changes I would need to
> make for scaling to much larger systems. I do plan at some point in
> the near future to test this on a much larger system with a LOT more
> CPU's

We have some ideas on scaling mvapich on multi-way SMP systems (like
Altix).  Unfortunately, we neither have information on the details of
the memory hierarchy/organization of these systems nor access to such
systems to try out these ideas.

> The test was conducted using a "kernel.org" 2.6.14 kernel and an
> OpenIB svn gen2 release of 3926 using Voltaire HCA's and switch
>
> We will be demonstrating OpenIB and mvapich-gen2 mpi at
> Supercomputing 05 (running smaller jobs though because the 32 way
> jobs take so long to complete). We will also demo rdma_lat, rdma_bw
> and IpoIB.

Good to know that you will be having the demo at SC '05. I will stop
by your booth. If you will have some time, we can discuss about the
memory hierarchy/organization of such systems and how to optimize
mvapich on it.

> I can send you the pallas results if you are interested.

Yes, please send it to me. We will be happy to look at the results. 

Best Regards, 

DK

> Regards
> John
> 
> -- 
> John Partridge
> 
> Silicon Graphics Inc
> Tel:  651-683-3428
> Vnet: 233-3428
> E-Mail: johnip at sgi.com
>