[openib-general] problems with lustre o2ib module & ofed

[Thierry Delaitre]

On Mon, 25 Sep 2006, Michael S. Tsirkin wrote:

> Quoting r. Thierry Delaitre <delaitt at cpc.wmin.ac.uk>:
> >
> > I've set the o2ib path to /usr/local/ofed/src/openib-1.1 as shown in the
> > lustre's configure line below. Lustre's configure script looks for a
> > driver/infiniband directory which only seems to exist under
> > /usr/local/ofed/src/openib-1.1
> >
> > ./configure --with-o2ib=/usr/local/ofed/src/openib-1.1/
> >
> > Thierry.
> >
> > > replace /usr/local/ofed with the prefix you specified.
>
> This looks wrong - openib-1.1 is the pristine sources.
> openib/include is the exported interface and is what you should use
> for dependent modules.
> No idea why would lustre need drivers/infiniband.
> Try creating a softlink:
>
> mkdir /usr/local/ofed/src/openib/drivers/infiniband
> ln -s /usr/local/ofed/src/openib/include /usr/local/ofed/src/openib/drivers/infiniband

I untarred lustre 1.5.95, compiled it (./configure
--with-o2ib=/usr/local/ofed/src/openib) . did a make install, depmod -a
and still get the following:

my modprobe.conf is the following

options lnet ip2nets="o2ib0 161.74.83.[0-255]"

lctl network up
LNET configure error 100: Network is down

ko2iblnd: disagrees about version of symbol ib_create_cq
ko2iblnd: Unknown symbol ib_create_cq
ko2iblnd: disagrees about version of symbol ib_dereg_mr
ko2iblnd: Unknown symbol ib_dereg_mr
ko2iblnd: disagrees about version of symbol ib_destroy_cq
ko2iblnd: Unknown symbol ib_destroy_cq
ko2iblnd: disagrees about version of symbol ib_get_dma_mr
ko2iblnd: Unknown symbol ib_get_dma_mr
ko2iblnd: disagrees about version of symbol ib_alloc_pd
ko2iblnd: Unknown symbol ib_alloc_pd
ko2iblnd: disagrees about version of symbol ib_modify_qp
ko2iblnd: Unknown symbol ib_modify_qp
ko2iblnd: disagrees about version of symbol ib_dealloc_pd
ko2iblnd: Unknown symbol ib_dealloc_pd
LustreError: 4177:0:(api-ni.c:1002:lnet_startup_lndnis()) Can't load LND
o2ib, module ko2iblnd, rc=256

lsmod | grep ib
libcfs                103060  1 lnet
ib_ucm                 19332  0
ib_addr                10756  1 rdma_cm
ib_cm                  31968  2 ib_ucm,rdma_cm
ib_ipoib               48400  0
ib_sa                  16652  3 rdma_cm,ib_cm,ib_ipoib
ib_uverbs              38312  2 rdma_ucm,ib_ucm
ib_umad                17968  0
ib_mthca              116240  0
ib_mad                 36116  4 ib_cm,ib_sa,ib_umad,ib_mthca
ib_core                49024  9
ib_ucm,rdma_cm,ib_cm,ib_ipoib,ib_sa,ib_uverbs,ib_umad,ib_mthca,ib_mad

nm /lib/modules/2.6.16.21-0.8-smp/kernel/drivers/infiniband/core/ib_core.ko | grep ib_alloc_pd
d5dcb698 A __crc_ib_alloc_pd
0000001c r __kcrctab_ib_alloc_pd
0000006a r __kstrtab_ib_alloc_pd
00000038 r __ksymtab_ib_alloc_pd
00000c65 T ib_alloc_pd

from lustre's config.log:

configure:6500: checking whether to enable OpenIB gen2 support
configure:6586: cp conftest.c build && make modules CC=gcc -f
/root/lustre-1.5.95/build/Makefile LUSTRE_LINUX
_CONFIG=/usr/src/linux/.config -o tmp_include_depends -o scripts -o include/config/MARKER -C /usr/src/linux EXTRA_CFLAGS=-Werror-implicit-function-declaration -g -I/root/lustre-1.5.95/lnet/include -I/root/lustre-1.5.95/lustre/include -I/usr/local/ofed/src/openib/include  M=/root/lustre-1.5.95/build
/root/lustre-1.5.95/build/conftest.c:42: warning: function declaration
isn't a prototype
/root/lustre-1.5.95/build/conftest.c: In function 'main':
/root/lustre-1.5.95/build/conftest.c:49: warning: unused variable 'rej_reason'
/root/lustre-1.5.95/build/conftest.c:48: warning: unused variable 'pool_fmr'
/root/lustre-1.5.95/build/conftest.c:47: warning: unused variable 'qp_attr'
/root/lustre-1.5.95/build/conftest.c:46: warning: unused variable 'device_attr'
/root/lustre-1.5.95/build/conftest.c:45: warning: unused variable 'conn_param'
WARNING: "rdma_create_id" [/root/lustre-1.5.95/build/conftest.ko] undefined!
configure:6589: $? = 0
configure:6591: test -s build/conftest.o
configure:6594: $? = 0
configure:6597: result: yes


Thierry.