[openib-general] mvapich2 ofed 1.2 problem

[Steve Wise]

On Tue, 2007-02-13 at 09:21 -0800, Roland Dreier wrote:
>  > How do I tell?  Can I tell from the .so files?
> 
> ldd on the .so and the app would probably give you good info.
> 
> I'm pretty sure that mpicc must be linking against an libibverbs 1.0
> from somewhere.
> 
>  - R.

By the way, the problem also happens running over mthca/IB with
librdmacm.

mpicc has '-libverbs'
mpicc.conf has '-libverbs' too.


ldd output.  Looks like they are all linking to libibverbs.so.1.  Is
that correct?


[mpi at vic20 ~]$ ldd IMB_2.3/src/IMB-MPI1
        libmpich.so => /usr/local/ofed/mpi/gcc/mvapich2-0.9.8-3/lib/libmpich.so (0x00002b0d7cefb000)
        librdmacm.so => /usr/local/ofed/lib64/librdmacm.so (0x00002b0d7d1b3000)
        libibverbs.so.1 => /usr/local/ofed/lib64/libibverbs.so.1 (0x00002b0d7d2b8000)
        libibumad.so.1 => /usr/local/ofed/lib64/libibumad.so.1 (0x00002b0d7d3c3000)
        libpthread.so.0 => /lib64/tls/libpthread.so.0 (0x0000003e07000000)
        librt.so.1 => /lib64/tls/librt.so.1 (0x0000003e0ba00000)
        libc.so.6 => /lib64/tls/libc.so.6 (0x0000003e06500000)
        libsysfs.so.1 => /usr/lib64/libsysfs.so.1 (0x0000003e06a00000)
        libdl.so.2 => /lib64/libdl.so.2 (0x0000003e06300000)
        libibcommon.so.1 => /usr/local/ofed/lib64/libibcommon.so.1 (0x00002b0d7d4cf000)
        /lib64/ld-linux-x86-64.so.2 (0x0000003e06100000)

[mpi at vic20 ~]$ ldd /usr/local/ofed/mpi/gcc/mvapich2-0.9.8-3/lib/libmpich.so
        libc.so.6 => /lib64/tls/libc.so.6 (0x00002b6a6061d000)
        /lib64/ld-linux-x86-64.so.2 (0x0000555555554000)

[mpi at vic20 ~]$ ldd /usr/local/ofed/lib64/librdmacm.so
        libibverbs.so.1 => /usr/local/ofed/lib64/libibverbs.so.1 (0x00002b3ef50de000)
        libsysfs.so.1 => /usr/lib64/libsysfs.so.1 (0x00002b3ef51ea000)
        libc.so.6 => /lib64/tls/libc.so.6 (0x00002b3ef52f6000)
        libpthread.so.0 => /lib64/tls/libpthread.so.0 (0x00002b3ef552a000)
        libdl.so.2 => /lib64/libdl.so.2 (0x00002b3ef5640000)
        /lib64/ld-linux-x86-64.so.2 (0x0000555555554000)

[mpi at vic20 ~]$ ldd /usr/local/ofed/lib64/libcxgb3-rdmav2.so
        libibverbs.so.1 => /usr/local/ofed/lib64/libibverbs.so.1 (0x00002b83c160e000)
        libc.so.6 => /lib64/tls/libc.so.6 (0x00002b83c171a000)
        libpthread.so.0 => /lib64/tls/libpthread.so.0 (0x00002b83c194e000)
        libdl.so.2 => /lib64/libdl.so.2 (0x00002b83c1a63000)
        /lib64/ld-linux-x86-64.so.2 (0x0000555555554000)

[mpi at vic20 ~]$ ldd /usr/local/ofed/lib64/libcxgb3.so
        libibverbs.so.1 => /usr/local/ofed/lib64/libibverbs.so.1 (0x00002ac8e4920000)
        libc.so.6 => /lib64/tls/libc.so.6 (0x00002ac8e4a2c000)
        libpthread.so.0 => /lib64/tls/libpthread.so.0 (0x00002ac8e4c60000)
        libdl.so.2 => /lib64/libdl.so.2 (0x00002ac8e4d75000)
        /lib64/ld-linux-x86-64.so.2 (0x0000555555554000)