<br><font size=2 face="sans-serif">Matthew, </font>
<br>
<br><font size=2 face="sans-serif">Thanks for the response. I can
well believe that the problem is some error in the setup of my environment.
I just can't figure out what as yet. As requested, some more
information is below on the two systems 'koa' and 'jatoba'.</font>
<br>
<br><font size=2><tt>> <br>
> Since the MPD daemons seem to have been started properly, as your
ability<br>
> to run non-MPI commands shows, I'm wondering if your $PATH may not
be set<br>
> properly when compiling the cpi application.<br>
> <br>
> Can you just verify by 'echo'ing the $PATH before compiling? Can you
also<br>
> provide us with other information about your environment?<br>
> <br>
> Also, when running MVAPICH2, can you try using the 'mpdtrace' command
to<br>
> verify that MPD has properly started on the requested nodes?<br>
> <br>
<br>
For the MVAPICH (MPI-1) case:</tt></font>
<br>
<br><font size=2><tt>On system 'koa':</tt></font>
<br>
<br><font size=2><tt>[root@koa ib]# cat .mv1
<-- using this file to set the PATH</tt></font>
<br><font size=2><tt>echo "setting up MVAPICH environment"</tt></font>
<br><font size=2><tt>export PATH=/usr/local/mvapich/bin:$PATH</tt></font>
<br><font size=2><tt>[root@koa ib]# . .mv1
<-- sourcing the .mv1 file</tt></font>
<br><font size=2><tt>setting up MVAPICH environment</tt></font>
<br><font size=2><tt>[root@koa ib]# echo $PATH <--
PATH after setup</tt></font>
<br><font size=2><tt>/usr/local/mvapich/bin:/usr/kerberos/sbin:/sbin:/usr/kerberos/bin:/usr/local/bin:/bin:/usr/bin:/usr/X11R6/bin</tt></font>
<br><font size=2><tt>[root@koa ib]# which mpicc <--
showing where the compiler script is located</tt></font>
<br><font size=2><tt>/usr/local/mvapich/bin/mpicc</tt></font>
<br><font size=2><tt>[root@koa ib]# mpd &
<-- starting mpd</tt></font>
<br><font size=2><tt>[1] 32649</tt></font>
<br><font size=2><tt>[root@koa ib]# mpdtrace
<-- with just mpd running on 'koa'</tt></font>
<br><font size=2><tt>mpdtrace: koa_53859: lhs=koa_53859 rhs=koa_53859
rhs2=koa_53859 gen=1</tt></font>
<br><font size=2><tt>[root@koa ib]# mpdtrace
<-- after starting mpd on the 'jatoba' system</tt></font>
<br><font size=2><tt>mpdtrace: koa_53859: lhs=jatoba_44789 rhs=jatoba_44789
rhs2=koa_53859 gen=1</tt></font>
<br><font size=2><tt>mpdtrace: jatoba_44789: lhs=koa_53859 rhs=koa_53859
rhs2=jatoba_44789 gen=1</tt></font>
<br><font size=2><tt>[root@koa ib]# /sbin/lsmod <--
showing that the various ib modules are loaded</tt></font>
<br><font size=2><tt>Module
Size Used by</tt></font>
<br><font size=2><tt>ib_sdp
93448 0</tt></font>
<br><font size=2><tt>ib_cm
33472 1 ib_sdp</tt></font>
<br><font size=2><tt>ib_ipoib
41480 0</tt></font>
<br><font size=2><tt>ib_sa
14652 2 ib_sdp,ib_ipoib</tt></font>
<br><font size=2><tt>ib_uverbs
39760 0</tt></font>
<br><font size=2><tt>ib_umad
15024 0</tt></font>
<br><font size=2><tt>ib_mthca
120608 0</tt></font>
<br><font size=2><tt>ib_mad
37816 4 ib_cm,ib_sa,ib_umad,ib_mthca</tt></font>
<br><font size=2><tt>ib_core
48640 8 ib_sdp,ib_cm,ib_ipoib,ib_sa,ib_uverbs,ib_umad,ib_mthca,ib_mad</tt></font>
<br>
<br>
<br><font size=2><tt>On system 'jatoba':</tt></font>
<br>
<br><font size=2><tt>[root@jatoba ib]# cat .mv1</tt></font>
<br><font size=2><tt>echo "setting up MVAPICH environment"</tt></font>
<br><font size=2><tt>export PATH=/usr/local/mvapich/bin:$PATH</tt></font>
<br><font size=2><tt>setting up MVAPICH environment</tt></font>
<br><font size=2><tt>[root@jatoba ib]# echo $PATH</tt></font>
<br><font size=2><tt>/usr/local/mvapich/bin:/usr/kerberos/sbin:/opt/gcc-3.3.3/bin:/usr/kerberos/bin:/usr/local/bin:/bin:/usr/bin:/usr/X11R6/bin:/usr/local/bin:/usr/java/bin:/usr/ant/bin:/home/ib/bin</tt></font>
<br><font size=2><tt>[root@jatoba ib]# which mpicc</tt></font>
<br><font size=2><tt>/usr/local/mvapich/bin/mpicc</tt></font>
<br><font size=2><tt>[root@jatoba ib]# mpd -h koa -p 53859 &
<-- starting mpd with parameters from 'koa'</tt></font>
<br><font size=2><tt>[1] 9442</tt></font>
<br><font size=2><tt>[root@jatoba ib]# mpdtrace
<-- after starting mpd</tt></font>
<br><font size=2><tt>mpdtrace: jatoba_44789: lhs=koa_53859 rhs=koa_53859
rhs2=jatoba_44789 gen=1</tt></font>
<br><font size=2><tt>mpdtrace: koa_53859: lhs=jatoba_44789 rhs=jatoba_44789
rhs2=koa_53859 gen=1</tt></font>
<br><font size=2><tt>[root@jatoba ib]# /sbin/lsmod
<-- showing that the various ib modules are loaded</tt></font>
<br><font size=2><tt>Module
Size Used by</tt></font>
<br><font size=2><tt>ib_sdp
93448 0</tt></font>
<br><font size=2><tt>ib_cm
33344 1 ib_sdp</tt></font>
<br><font size=2><tt>ib_ipoib
41480 0</tt></font>
<br><font size=2><tt>ib_sa
14524 2 ib_sdp,ib_ipoib</tt></font>
<br><font size=2><tt>ib_uverbs
39760 0</tt></font>
<br><font size=2><tt>ib_umad
15024 0</tt></font>
<br><font size=2><tt>ib_mthca
120608 0</tt></font>
<br><font size=2><tt>ib_mad
37816 4 ib_cm,ib_sa,ib_umad,ib_mthca</tt></font>
<br><font size=2><tt>ib_core
48640 8 ib_sdp,ib_cm,ib_ipoib,ib_sa,ib_uverbs,ib_umad,ib_mthca,ib_mad</tt></font>
<br>
<br>
<br>
<br><font size=2><tt>For the MVAPICH2 case:</tt></font>
<br>
<br><font size=2><tt>On 'koa':</tt></font>
<br>
<br><font size=2><tt>[root@koa ib]# cat .mv2</tt></font>
<br><font size=2><tt>echo "setting up MVAPICH2 environment"</tt></font>
<br><font size=2><tt>export PATH=/tmp/mvapich2/bin:$PATH</tt></font>
<br><font size=2><tt>[root@koa ib]# . .mv2</tt></font>
<br><font size=2><tt>setting up MVAPICH2 environment</tt></font>
<br><font size=2><tt>[root@koa ib]# echo $PATH</tt></font>
<br><font size=2><tt>/tmp/mvapich2/bin:/usr/kerberos/sbin:/sbin:/usr/kerberos/bin:/usr/local/bin:/bin:/usr/bin:/usr/X11R6/bin</tt></font>
<br><font size=2><tt>[root@koa ib]# which mpicc</tt></font>
<br><font size=2><tt>/tmp/mvapich2/bin/mpicc</tt></font>
<br><font size=2><tt>[root@koa ib]# mpdboot -n 2 -f /root/mpd.hosts</tt></font>
<br><font size=2><tt>[root@koa ib]# mpdtrace -l</tt></font>
<br><font size=2><tt>koa.az05.bull.com_53727</tt></font>
<br><font size=2><tt>jatoba.az05.bull.com_54528</tt></font>
<br>
<br><font size=2><tt>On 'jatoba':</tt></font>
<br>
<br><font size=2><tt>[jatoba] (ib) ib> su</tt></font>
<br><font size=2><tt>Password:</tt></font>
<br><font size=2><tt>[root@jatoba ib]# cat .mv2</tt></font>
<br><font size=2><tt>echo "setting up MPVAPICH2 environment"</tt></font>
<br><font size=2><tt>export PATH=/tmp/mvapich2/bin:$PATH</tt></font>
<br><font size=2><tt>[root@jatoba ib]# . .mv2</tt></font>
<br><font size=2><tt>setting up MPVAPICH2 environment</tt></font>
<br><font size=2><tt>[root@jatoba ib]# echo $PATH</tt></font>
<br><font size=2><tt>/tmp/mvapich2/bin:/usr/kerberos/sbin:/opt/gcc-3.3.3/bin:/usr/kerberos/bin:/usr/local/bin:/bin:/usr/bin:/usr/X11R6/bin:/usr/local/bin:/usr/java/bin:/usr/ant/bin:/home/ib/bin</tt></font>
<br><font size=2><tt>[root@jatoba ib]# which mpicc</tt></font>
<br><font size=2><tt>/tmp/mvapich2/bin/mpicc</tt></font>
<br><font size=2><tt>[root@jatoba ib]# mpdtrace -l</tt></font>
<br><font size=2><tt>jatoba.az05.bull.com_54528</tt></font>
<br><font size=2><tt>koa.az05.bull.com_53727</tt></font>
<br>
<br>
<br>