[openib-general] [Lustre-discuss] Re: problems with lustre o2ib module & ofed
Jack Morgenstein
jackm at dev.mellanox.co.il
Tue Sep 26 07:07:37 PDT 2006
On Monday 25 September 2006 17:01, Thierry Delaitre wrote:
I noticed in the Lustre configure file the following
--with-linux=path set path to Linux source (default=/usr/src/linux)
Where does /usr/src/linux link to?
You might consider explicitly specifying the following options as well in the
Lustre ./configure step:
--with-linux=path set path to Linux source (default=/usr/src/linux)
--with-linux-obj=path set path to Linux objects dir (default=$LINUX)
--with-linux-config=path
set path to Linux .conf (default=$LINUX_OBJ/.config)
- Jack
>
> On Mon, 25 Sep 2006, Thierry Delaitre wrote:
>
> >
> > It seems that lustre puts its modules in /lib/modules/2.6.16.21-0.8-default
> > despite the fact that my kernel is 2.6.16.21-0.8-smp !
> >
> > uname -a
> > Linux n32 2.6.16.21-0.8-smp #4 SMP Sun Sep 24 08:47:30 BST 2006 i686 i686 i386 GNU/Linux
> >
> > make[3]: Nothing to be done for `install-exec-am'.
> > /bin/sh ../../mkinstalldirs /lib/modules/2.6.16.21-0.8-default/kernel/fs/lustre
> > /usr/bin/install -c -m 644 lquota.ko /lib/modules/2.6.16.21-0.8-default/kernel/fs/lustre/lquota
> >
> > I therefore ends up with a /lib/modules/2.6.16.21-0.8-smp and
> > /lib/modules/2.6.16.21-0.8-default
> >
> > i'm now searching why lustre thinks my kernel is 2.6.16.21-0.8-default and
> > not 2.6.16.21-0.8-smp
>
> I've updated the UTS_RELEASE string in
> /usr/src/linux-2.6.16.21-0.8/include/linux/version.h from default to smp
> and deleted my /lib/modules/
> lustre now installs in /lib/modules/2.6.16.21-0.8-smp/kernel along with
> ofed ib drivers. i recompiled the kernel, ofed and lustre and still gets
> this:
>
> ko2iblnd: disagrees about version of symbol ib_create_cq
> ko2iblnd: Unknown symbol ib_create_cq
> ko2iblnd: disagrees about version of symbol ib_dereg_mr
> ko2iblnd: Unknown symbol ib_dereg_mr
> ko2iblnd: disagrees about version of symbol ib_destroy_cq
> ko2iblnd: Unknown symbol ib_destroy_cq
> ko2iblnd: disagrees about version of symbol ib_get_dma_mr
> ko2iblnd: Unknown symbol ib_get_dma_mr
> ko2iblnd: disagrees about version of symbol ib_alloc_pd
> ko2iblnd: Unknown symbol ib_alloc_pd
> ko2iblnd: disagrees about version of symbol ib_modify_qp
> ko2iblnd: Unknown symbol ib_modify_qp
> ko2iblnd: disagrees about version of symbol ib_dealloc_pd
> ko2iblnd: Unknown symbol ib_dealloc_pd
> LustreError: 7430:0:(api-ni.c:1002:lnet_startup_lndnis()) Can't load LND
> o2ib, module ko2iblnd, rc=256
>
> nm /lib/modules/2.6.16.21-0.8-smp/kernel/drivers/infiniband/core/ib_core.ko | grep ib_create_cq
> 3cfe7afa A __crc_ib_create_cq
> 00000060 r __kcrctab_ib_create_cq
> 0000015f r __kstrtab_ib_create_cq
> 000000c0 r __ksymtab_ib_create_cq
> 00000d50 T ib_create_cq
>
> i'm a bit stuck!
>
> Thierry.
>
> > Thierry.
> >
> > On Mon, 25 Sep 2006, Thierry Delaitre wrote:
> >
> > >
> > > On Mon, 25 Sep 2006, Michael S. Tsirkin wrote:
> > >
> > > > Quoting r. Thierry Delaitre <delaitt at cpc.wmin.ac.uk>:
> > > > >
> > > > > I've set the o2ib path to /usr/local/ofed/src/openib-1.1 as shown in the
> > > > > lustre's configure line below. Lustre's configure script looks for a
> > > > > driver/infiniband directory which only seems to exist under
> > > > > /usr/local/ofed/src/openib-1.1
> > > > >
> > > > > ./configure --with-o2ib=/usr/local/ofed/src/openib-1.1/
> > > > >
> > > > > Thierry.
> > > > >
> > > > > > replace /usr/local/ofed with the prefix you specified.
> > > >
> > > > This looks wrong - openib-1.1 is the pristine sources.
> > > > openib/include is the exported interface and is what you should use
> > > > for dependent modules.
> > > > No idea why would lustre need drivers/infiniband.
> > > > Try creating a softlink:
> > > >
> > > > mkdir /usr/local/ofed/src/openib/drivers/infiniband
> > > > ln -s /usr/local/ofed/src/openib/include /usr/local/ofed/src/openib/drivers/infiniband
> > >
> > > I untarred lustre 1.5.95, compiled it (./configure
> > > --with-o2ib=/usr/local/ofed/src/openib) . did a make install, depmod -a
> > > and still get the following:
> > >
> > > my modprobe.conf is the following
> > >
> > > options lnet ip2nets="o2ib0 161.74.83.[0-255]"
> > >
> > > lctl network up
> > > LNET configure error 100: Network is down
> > >
> > > ko2iblnd: disagrees about version of symbol ib_create_cq
> > > ko2iblnd: Unknown symbol ib_create_cq
> > > ko2iblnd: disagrees about version of symbol ib_dereg_mr
> > > ko2iblnd: Unknown symbol ib_dereg_mr
> > > ko2iblnd: disagrees about version of symbol ib_destroy_cq
> > > ko2iblnd: Unknown symbol ib_destroy_cq
> > > ko2iblnd: disagrees about version of symbol ib_get_dma_mr
> > > ko2iblnd: Unknown symbol ib_get_dma_mr
> > > ko2iblnd: disagrees about version of symbol ib_alloc_pd
> > > ko2iblnd: Unknown symbol ib_alloc_pd
> > > ko2iblnd: disagrees about version of symbol ib_modify_qp
> > > ko2iblnd: Unknown symbol ib_modify_qp
> > > ko2iblnd: disagrees about version of symbol ib_dealloc_pd
> > > ko2iblnd: Unknown symbol ib_dealloc_pd
> > > LustreError: 4177:0:(api-ni.c:1002:lnet_startup_lndnis()) Can't load LND
> > > o2ib, module ko2iblnd, rc=256
> > >
> > > lsmod | grep ib
> > > libcfs 103060 1 lnet
> > > ib_ucm 19332 0
> > > ib_addr 10756 1 rdma_cm
> > > ib_cm 31968 2 ib_ucm,rdma_cm
> > > ib_ipoib 48400 0
> > > ib_sa 16652 3 rdma_cm,ib_cm,ib_ipoib
> > > ib_uverbs 38312 2 rdma_ucm,ib_ucm
> > > ib_umad 17968 0
> > > ib_mthca 116240 0
> > > ib_mad 36116 4 ib_cm,ib_sa,ib_umad,ib_mthca
> > > ib_core 49024 9
> > > ib_ucm,rdma_cm,ib_cm,ib_ipoib,ib_sa,ib_uverbs,ib_umad,ib_mthca,ib_mad
> > >
> > > nm /lib/modules/2.6.16.21-0.8-smp/kernel/drivers/infiniband/core/ib_core.ko | grep ib_alloc_pd
> > > d5dcb698 A __crc_ib_alloc_pd
> > > 0000001c r __kcrctab_ib_alloc_pd
> > > 0000006a r __kstrtab_ib_alloc_pd
> > > 00000038 r __ksymtab_ib_alloc_pd
> > > 00000c65 T ib_alloc_pd
> > >
> > > from lustre's config.log:
> > >
> > > configure:6500: checking whether to enable OpenIB gen2 support
> > > configure:6586: cp conftest.c build && make modules CC=gcc -f
> > > /root/lustre-1.5.95/build/Makefile LUSTRE_LINUX
> > > _CONFIG=/usr/src/linux/.config -o tmp_include_depends -o scripts -o include/config/MARKER -C /usr/src/linux EXTRA_CFLAGS=-Werror-implicit-function-declaration -g -I/root/lustre-1.5.95/lnet/include -I/root/lustre-1.5.95/lustre/include -I/usr/local/ofed/src/openib/include M=/root/lustre-1.5.95/build
> > > /root/lustre-1.5.95/build/conftest.c:42: warning: function declaration
> > > isn't a prototype
> > > /root/lustre-1.5.95/build/conftest.c: In function 'main':
> > > /root/lustre-1.5.95/build/conftest.c:49: warning: unused variable 'rej_reason'
> > > /root/lustre-1.5.95/build/conftest.c:48: warning: unused variable 'pool_fmr'
> > > /root/lustre-1.5.95/build/conftest.c:47: warning: unused variable 'qp_attr'
> > > /root/lustre-1.5.95/build/conftest.c:46: warning: unused variable 'device_attr'
> > > /root/lustre-1.5.95/build/conftest.c:45: warning: unused variable 'conn_param'
> > > WARNING: "rdma_create_id" [/root/lustre-1.5.95/build/conftest.ko] undefined!
> > > configure:6589: $? = 0
> > > configure:6591: test -s build/conftest.o
> > > configure:6594: $? = 0
> > > configure:6597: result: yes
> > >
> > >
> > > Thierry.
> > >
> > > _______________________________________________
> > > openib-general mailing list
> > > openib-general at openib.org
> > > http://openib.org/mailman/listinfo/openib-general
> > >
> > > To unsubscribe, please visit http://openib.org/mailman/listinfo/openib-general
> > >
> > >
> >
> > ----------------------------------------
> > Dr Thierry DELAITRE
> > Systems and Services Manager, CSCS
> > University of Westminster
> > 115 New Cavendish Street, London W1W 6UW
> >
> > Tel: 020 7911 5000 ext: 3586
> > Fax: 020 7911 5089
> > Mobile short dial code 1788
> >
> > http://www.cscs.wmin.ac.uk/~delaitt
> > ----------------------------------------
> >
> > This e-mail and its attachments are intended for the above named only
> > and may be confidential. If they have come to you in error you must
> > not copy or show them to anyone, nor should you take any action based
> > on them, other than to notify the error by replying to the sender.
> >
> > _______________________________________________
> > Lustre-discuss mailing list
> > Lustre-discuss at clusterfs.com
> > https://mail.clusterfs.com/mailman/listinfo/lustre-discuss
> >
> >
>
> ----------------------------------------
> Dr Thierry DELAITRE
> Systems and Services Manager, CSCS
> University of Westminster
> 115 New Cavendish Street, London W1W 6UW
>
> Tel: 020 7911 5000 ext: 3586
> Fax: 020 7911 5089
> Mobile short dial code 1788
>
> http://www.cscs.wmin.ac.uk/~delaitt
> ----------------------------------------
>
> This e-mail and its attachments are intended for the above named only
> and may be confidential. If they have come to you in error you must
> not copy or show them to anyone, nor should you take any action based
> on them, other than to notify the error by replying to the sender.
>
> _______________________________________________
> openib-general mailing list
> openib-general at openib.org
> http://openib.org/mailman/listinfo/openib-general
>
> To unsubscribe, please visit http://openib.org/mailman/listinfo/openib-general
>
More information about the general
mailing list