[openib-general] problems with lustre o2ib module & ofed

[Thierry Delaitre]

On Mon, 25 Sep 2006, Michael S. Tsirkin wrote:

> Quoting r. Thierry Delaitre <delaitt at cpc.wmin.ac.uk>:
> > Subject: Re: problems with lustre o2ib module & ofed
> >
> >
> > On Mon, 25 Sep 2006, Or Gerlitz wrote:
> >
> > > Jack Morgenstein wrote:
> > > > Did you recompile Lustre following the installation of ofed-1.1?
> > > > I'm not familiar with the Lustre installation procedure (i.e., if it
> > > > gets compiled on the current host).  If yes, you probably merely need
> > > > to uninstall and reinstall Lustre o2ib.
> > >
> > > OK, can we state clearly what's the user needs to do with modules
> > > directly dependent on ofed symbols (eg Lustre's o2ib, NFSoRDMA, RDS and
> > > hopefully more to come).
> > >
> > > Is it recompile / uninstall / install ???
> >
> > The issue is about the installation of Lustre 1.5.95 o2ib with OFED-1.1rc6
> > for SLES10.
> >
> > ofed-1.1-rc6 compiles nicely as shown below. The ib kernel modules all
> > resides under /lib/modules/2.6.16.21-0.8-smp/kernel/drivers/infiniband/
> > and do match the ones compiled by ofed. I have tried these steps several
> > times.
> >
> > n32:~ # lsmod | grep ib
> > libcfs                103060  1 lnet
> > ib_ucm                 19332  0
> > ib_addr                10756  1 rdma_cm
> > ib_cm                  31968  2 ib_ucm,rdma_cm
> > ib_ipoib               48144  0
> > ib_sa                  16652  3 rdma_cm,ib_cm,ib_ipoib
> > ib_uverbs              38312  2 rdma_ucm,ib_ucm
> > ib_umad                17968  0
> > ib_mthca              116240  0
> > ib_mad                 36116  4 ib_cm,ib_sa,ib_umad,ib_mthca
> > ib_core                49024  9
> > ib_ucm,rdma_cm,ib_cm,ib_ipoib,ib_sa,ib_uverbs,ib_umad,ib_mthca,ib_mad
> >
> > I compiled lustre for the above kernel and ofed installation. I get the
> > following when doing a 'lctl network up' in lustre. I have modversion set
> > to on in the kernel. If i set it to 'n' then i get a null pointer
> > exception and the module crashes.
> >
> > ko2iblnd: disagrees about version of symbol ib_create_cq
> > ko2iblnd: Unknown symbol ib_create_cq
>
> don't know anything about lustre, but note you must
> point build to pick up headers from
> /usr/local/ofed/src/openib/include/
> *before* the built-in header includes.

I've set the o2ib path to /usr/local/ofed/src/openib-1.1 as shown in the
lustre's configure line below. Lustre's configure script looks for a
driver/infiniband directory which only seems to exist under
/usr/local/ofed/src/openib-1.1

./configure --with-o2ib=/usr/local/ofed/src/openib-1.1/

Thierry.

> replace /usr/local/ofed with the prefix you specified.

> --
> MST