[openib-general] Problems running MPI jobs with MVAPICH and MVAPICH2

[Don.Albert at Bull.com]

Weikuan,

> Just a couple of questions to check with you on these systems?
> 
> a) Did you have to build the mvapich libraries separately on these 
> machines? Related to this, are the directories: /usr/local, 
> /tmp/mvapich, nfs exported for sharing across different machines? If 
> you had to build the libraries separately, please provide the other set 
> of info from `jatoba'.

Both the kernel and the openib software was compiled separately on each 
machine.  The corresponding logs from 'jatoba' are attached below.  None 
of the directories are shared.  For compiling the "cpi.c" program, I 
compile it on each machine, but the directory structure is the same:  i.e. 
the "cpi" executable is under /home/ib/test/mpi/cpi/cpi on each machine.

> b) Could you provide some additional specifications of these two 
> machines? Kernel versions, linux distribution, HCA firmware and 
> versions, gen2 kernel and userspace versions? We could have asked you 
> earlier. But just came to think these might be relevant...

The machines are both EM64T (Intel(R) Xeon(TM) CPU 3.00GHz).  They are 
similar, but not identical (e.g. one has an E1000 ethernet card, the other 
an Alteon AceNic).  Both have Mellanox HCAs (MT25208 InfiniHost III Ex 
(Tavor compatibility mode) (rev a0)).  Firmware in the HCAs is v4.7.400.

The machines were originally installed with RHEL4, with 2.6.9-11.EL 
kernel.  The current kernel is 2.6.15.6, which I built with the openib 
modules and the MVAPICH code (svn revision 5685).
> 
> Looking ahead. We may need access into your systems to give it a shot, 
> if it is ever possible...
> 
Unfortunately, I doubt that will be possible. But I will be glad to run 
anything you need.

        -Don Albert-

Information from jatoba:

[root at jatoba mvapich-gen2]# mpicc -show -o cpi examples/basic/cpi.c
gcc -DUSE_STDARG -DHAVE_STDLIB_H=1 -DHAVE_STRING_H=1 -DHAVE_UNISTD_H=1 
-DHAVE_STDARG_H=1 -DUSE_STDARG=1 -DMALLOC_RET_VOID=1 -c 
examples/basic/cpi.c -I/usr/local/mvapich/include
gcc -DUSE_STDARG -DHAVE_STDLIB_H=1 -DHAVE_STRING_H=1 -DHAVE_UNISTD_H=1 
-DHAVE_STDARG_H=1 -DUSE_STDARG=1 -DMALLOC_RET_VOID=1 
-L/usr/local/mvapich/lib cpi.o -o cpi -lmpich -L/usr/local/lib 
-Wl,-rpath=/usr/local/lib -libverbs -lpthread



-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.openfabrics.org/pipermail/general/attachments/20060322/855ed199/attachment.html>
-------------- next part --------------
A non-text attachment was scrubbed...
Name: config.log
Type: application/octet-stream
Size: 6718 bytes
Desc: not available
URL: <http://lists.openfabrics.org/pipermail/general/attachments/20060322/855ed199/attachment.obj>
-------------- next part --------------
A non-text attachment was scrubbed...
Name: config.status
Type: application/octet-stream
Size: 20374 bytes
Desc: not available
URL: <http://lists.openfabrics.org/pipermail/general/attachments/20060322/855ed199/attachment-0001.obj>
-------------- next part --------------
A non-text attachment was scrubbed...
Name: config-mine.log
Type: application/octet-stream
Size: 18029 bytes
Desc: not available
URL: <http://lists.openfabrics.org/pipermail/general/attachments/20060322/855ed199/attachment-0002.obj>
-------------- next part --------------
A non-text attachment was scrubbed...
Name: install-mine.log
Type: application/octet-stream
Size: 1887 bytes
Desc: not available
URL: <http://lists.openfabrics.org/pipermail/general/attachments/20060322/855ed199/attachment-0003.obj>
-------------- next part --------------
A non-text attachment was scrubbed...
Name: make.mvapich.gen2
Type: application/octet-stream
Size: 2591 bytes
Desc: not available
URL: <http://lists.openfabrics.org/pipermail/general/attachments/20060322/855ed199/attachment-0004.obj>
-------------- next part --------------
A non-text attachment was scrubbed...
Name: make-mine.log
Type: application/octet-stream
Size: 293862 bytes
Desc: not available
URL: <http://lists.openfabrics.org/pipermail/general/attachments/20060322/855ed199/attachment-0005.obj>