[ofa-general] Status of NFS-RDMA ?

[Pawel Dziekonski]

On Mon, 28 Jan 2008 at 10:14:22AM -0500, James Lentini wrote:
> 
> 
> On Sat, 26 Jan 2008, Pawel Dziekonski wrote:
> 
> > I pulled Tom's tree from new url and build a kernel. 
> 
> If you enabled support for INFINIBAND drivers (IB and iWARP support) 
> and NFS client/server support, the kernel should be ready to go (run 
> "grep RDMA /your_kernel_sources/.config" to confirm that 
> CONFIG_SUNRPC_XPRT_RDMA is either m or y).
> 
> NFS/RDMA doesn't require OFED be installed. OFED is a release of the 
> Linux kernel sources and some userspace libraries/tools. If you are 

> > then I downloaded OFED from 
> > http://www.mellanox.com/downloads/NFSoRDMA/OFED-1.2-NFS-RDMA.gz,
> 
> I don't know what the above URL contains. The latest code is in Tom 
> Tucker's tree (and now NFS server maintainer Bruce Fields tree). It is 


hi,

back to subject on a proper mailing list.

I have a >3 year experience with mellanox hardware and IBGold so I
basically know what OFED is all about. up to now i was only using
IBGold since IB drivers appeared in kernel pretty recently.

currently I have new hardware. I'm running Tom's kernel and already
did some MPI tests. SDP is not working, probably because sdp kernel
modules where not build. ;) I understand that those modules are only
available from ofa-kernel. please correct me if i'm wrong.

system is Scientic Linux 4.5, which is supposed to be a fully
compatible RH4 clone. hardware is Supermicro mobos with Mellanox
MT25204 and Flextronisc switch.

error log from ofa-kernel build:

> > make[1]: Entering directory `/usr/src/ib/xprt-switch-2.6'
> > test -e include/linux/autoconf.h -a -e include/config/auto.conf || (            \
> > echo;                                                           \
> > echo "  ERROR: Kernel configuration is invalid.";               \
> > echo "         include/linux/autoconf.h or include/config/auto.conf are missing.";      \
> > echo "         Run 'make oldconfig && make prepare' on kernel src to fix it.";  \
> > echo;                                                           \
> > /bin/false)
> > 
> > obviously, doing 'make oldconfig && make prepare' does not help. 
> > anyway, above mentioned files do exist:
> > 
> > # ls -la /usr/src/ib/xprt-switch-2.6/{include/linux/autoconf.h,include/config/auto.conf}
> > -rw-r--r--  1 root root 10156 Jan 25 17:42 /usr/src/ib/xprt-switch-2.6/include/config/auto.conf
> > -rw-r--r--  1 root root 14733 Jan 25 17:42 /usr/src/ib/xprt-switch-2.6/include/linux/autoconf.h
> > 
> > despite of above, compilation continues but fails with:
> > 
> >   gcc -Wp,-MD,/var/tmp/OFEDRPM/BUILD/ofa_kernel-1.2/drivers/infiniband/core/.mad.o.d  -nostdinc -isystem /usr/lib/gcc/x86_64-redhat-linux/3.4.6/include -D__KERNEL__ -I/var/tmp/OFEDRPM/BUILD/ofa_kernel-1.2/include  -I/var/tmp/OFEDRPM/BUILD/ofa_kernel-1.2/drivers/infiniband/include  -Iinclude    -include include/linux/autoconf.h  -include /var/tmp/OFEDRPM/BUILD/ofa_kernel-1.2/include/linux/autoconf.h   -Wall -Wundef -Wstrict-prototypes -Wno-trigraphs -fno-strict-aliasing -fno-common -Werror-implicit-function-declaration -Os   -m64 -mno-red-zone -mcmodel=kernel -pipe -Wno-sign-compare -fno-asynchronous-unwind-tables -funit-at-a-time -mno-sse -mno-mmx -mno-sse2 -mno-3dnow -maccumulate-outgoing-args -DCONFIG_AS_CFI=1 -DCONFIG_AS_CFI_SIGNAL_FRAME=1 -fomit-frame-pointer  -Wdeclaration-after-statement     -DMODULE -D"KBUILD_STR(s)=#s" -D"KBUILD_BASENAME=KBUILD_STR(mad)"  -D"KBUILD_MODNAME=KBUILD_STR(ib_mad)" -c -o /var/tmp/OFEDRPM/BUILD/ofa_kernel-1.2/drivers/infiniband/core/.tmp_mad.o /var/tmp/OFEDRPM/BUILD/ofa_kernel-1.2/drivers/infiniband/core/mad.c
> > /var/tmp/OFEDRPM/BUILD/ofa_kernel-1.2/drivers/infiniband/core/mad.c: In function `ib_mad_init_module':
> > /var/tmp/OFEDRPM/BUILD/ofa_kernel-1.2/drivers/infiniband/core/mad.c:2966: error: too many arguments to function `kmem_cache_create'
> > make[4]: *** [/var/tmp/OFEDRPM/BUILD/ofa_kernel-1.2/drivers/infiniband/core/mad.o] Error 1
> > make[3]: *** [/var/tmp/OFEDRPM/BUILD/ofa_kernel-1.2/drivers/infiniband/core] Error 2
> > make[2]: *** [/var/tmp/OFEDRPM/BUILD/ofa_kernel-1.2/drivers/infiniband] Error 2
> > make[1]: *** [_module_/var/tmp/OFEDRPM/BUILD/ofa_kernel-1.2] Error 2
> > make[1]: Leaving directory `/usr/src/ib/xprt-switch-2.6'
> > make: *** [kernel] Error 2
> > error: Bad exit status from /var/tmp/rpm-tmp.3877 (%install)

> > full log:
> > https://cefeid.wcss.wroc.pl/d/tmp/OFED.build.32122.log

thanks in advance for any help, P


-- 
Pawel Dziekonski <pawel.dziekonski at wcss.pl>
Wroclaw Centre for Networking & Supercomputing, HPC Department
Politechnika Wr., pl. Grunwaldzki 9, bud. D2/101, 50-377 Wroclaw, POLAND
phone: +48 71 3202043, fax: +48 71 3225797, http://www.wcss.wroc.pl