[ofa-general] Status of NFS-RDMA ?

[James Lentini]

On Sat, 26 Jan 2008, Pawel Dziekonski wrote:

> I'm still writing to this list because 
> nfs-rdma-devel at lists.sourceforge.net seems to be dead... :(

We'll investigate that.

> I pulled Tom's tree from new url and build a kernel. 

If you enabled support for INFINIBAND drivers (IB and iWARP support) 
and NFS client/server support, the kernel should be ready to go (run 
"grep RDMA /your_kernel_sources/.config" to confirm that 
CONFIG_SUNRPC_XPRT_RDMA is either m or y).

NFS/RDMA doesn't require OFED be installed. OFED is a release of the 
Linux kernel sources and some userspace libraries/tools. If you are 
using InfiniBand adapters, you'll need a subnet manager. OFED contains 
an sm called OpenSM. You can get OpenSM via OFED or you can download 
OpenSM separately. It is up to you.

> then I downloaded OFED from 
> http://www.mellanox.com/downloads/NFSoRDMA/OFED-1.2-NFS-RDMA.gz,

I don't know what the above URL contains. The latest code is in Tom 
Tucker's tree (and now NFS server maintainer Bruce Fields tree). It is 
being merged in 2.6.25. You'll be able to get the server code from a 
kernel.org rc for 2.6.25. In the meantime, you'll have to use the 
development repos to get the server.

> however ofa-kernel fails to build. whatever I do I always got in 
> ofa-kernel:
> 
> make[1]: Entering directory `/usr/src/ib/xprt-switch-2.6'
> test -e include/linux/autoconf.h -a -e include/config/auto.conf || (            \
> echo;                                                           \
> echo "  ERROR: Kernel configuration is invalid.";               \
> echo "         include/linux/autoconf.h or include/config/auto.conf are missing.";      \
> echo "         Run 'make oldconfig && make prepare' on kernel src to fix it.";  \
> echo;                                                           \
> /bin/false)
> 
> obviously, doing 'make oldconfig && make prepare' does not help. 
> anyway, above mentioned files do exist:
> 
> # ls -la /usr/src/ib/xprt-switch-2.6/{include/linux/autoconf.h,include/config/auto.conf}
> -rw-r--r--  1 root root 10156 Jan 25 17:42 /usr/src/ib/xprt-switch-2.6/include/config/auto.conf
> -rw-r--r--  1 root root 14733 Jan 25 17:42 /usr/src/ib/xprt-switch-2.6/include/linux/autoconf.h
> 
> despite of above, compilation continues but fails with:
> 
>   gcc -Wp,-MD,/var/tmp/OFEDRPM/BUILD/ofa_kernel-1.2/drivers/infiniband/core/.mad.o.d  -nostdinc -isystem /usr/lib/gcc/x86_64-redhat-linux/3.4.6/include -D__KERNEL__ -I/var/tmp/OFEDRPM/BUILD/ofa_kernel-1.2/include  -I/var/tmp/OFEDRPM/BUILD/ofa_kernel-1.2/drivers/infiniband/include  -Iinclude    -include include/linux/autoconf.h  -include /var/tmp/OFEDRPM/BUILD/ofa_kernel-1.2/include/linux/autoconf.h   -Wall -Wundef -Wstrict-prototypes -Wno-trigraphs -fno-strict-aliasing -fno-common -Werror-implicit-function-declaration -Os   -m64 -mno-red-zone -mcmodel=kernel -pipe -Wno-sign-compare -fno-asynchronous-unwind-tables -funit-at-a-time -mno-sse -mno-mmx -mno-sse2 -mno-3dnow -maccumulate-outgoing-args -DCONFIG_AS_CFI=1 -DCONFIG_AS_CFI_SIGNAL_FRAME=1 -fomit-frame-pointer  -Wdeclaration-after-statement     -DMODULE -D"KBUILD_STR(s)=#s" -D"KBUILD_BASENAME=KBUILD_STR(mad)"  -D"KBUILD_MODNAME=KBUILD_STR(ib_mad)" -c -o /var/tmp/OFEDRPM/BUILD/ofa_kernel-1.2/drivers/infiniband/core/.tmp_mad.o /var/tmp/OFEDRPM/BUILD/ofa_kernel-1.2/drivers/infiniband/core/mad.c
> /var/tmp/OFEDRPM/BUILD/ofa_kernel-1.2/drivers/infiniband/core/mad.c: In function `ib_mad_init_module':
> /var/tmp/OFEDRPM/BUILD/ofa_kernel-1.2/drivers/infiniband/core/mad.c:2966: error: too many arguments to function `kmem_cache_create'
> make[4]: *** [/var/tmp/OFEDRPM/BUILD/ofa_kernel-1.2/drivers/infiniband/core/mad.o] Error 1
> make[3]: *** [/var/tmp/OFEDRPM/BUILD/ofa_kernel-1.2/drivers/infiniband/core] Error 2
> make[2]: *** [/var/tmp/OFEDRPM/BUILD/ofa_kernel-1.2/drivers/infiniband] Error 2
> make[1]: *** [_module_/var/tmp/OFEDRPM/BUILD/ofa_kernel-1.2] Error 2
> make[1]: Leaving directory `/usr/src/ib/xprt-switch-2.6'
> make: *** [kernel] Error 2
> error: Bad exit status from /var/tmp/rpm-tmp.3877 (%install)
> 
> full log:
> https://cefeid.wcss.wroc.pl/d/tmp/OFED.build.32122.log
> 
> thanks in advance, P
> 
> -- 
> Pawel Dziekonski <pawel.dziekonski at wcss.pl>
> Wroclaw Centre for Networking & Supercomputing, HPC Department
> Politechnika Wr., pl. Grunwaldzki 9, bud. D2/101, 50-377 Wroclaw, POLAND
> phone: +48 71 3202043, fax: +48 71 3225797, http://www.wcss.wroc.pl
>