[nfs-rdma-devel] [ofa-general] Status of NFS-RDMA ?

Tue Jan 29 06:53:46 PST 2008

On Tue, 29 Jan 2008, Pawel Dziekonski wrote:

> 
> On Mon, 28 Jan 2008 at 10:14:22AM -0500, James Lentini wrote:
> > 
> > 
> > On Sat, 26 Jan 2008, Pawel Dziekonski wrote:
> > 
> > > I pulled Tom's tree from new url and build a kernel. 
> > 
> > If you enabled support for INFINIBAND drivers (IB and iWARP support) 
> > and NFS client/server support, the kernel should be ready to go (run 
> > "grep RDMA /your_kernel_sources/.config" to confirm that 
> > CONFIG_SUNRPC_XPRT_RDMA is either m or y).
> > 
> > NFS/RDMA doesn't require OFED be installed. OFED is a release of the 
> > Linux kernel sources and some userspace libraries/tools. If you are 
> 
> > > then I downloaded OFED from 
> > > http://www.mellanox.com/downloads/NFSoRDMA/OFED-1.2-NFS-RDMA.gz,
> > 
> > I don't know what the above URL contains. The latest code is in Tom 
> > Tucker's tree (and now NFS server maintainer Bruce Fields tree). It is 
> 
> 
> hi,
> 
> back to subject on a proper mailing list.
> 
> I have a >3 year experience with mellanox hardware and IBGold so I
> basically know what OFED is all about. up to now i was only using
> IBGold since IB drivers appeared in kernel pretty recently.

You'll want to use the mainline kernel's IB drivers for NFS/RDMA. 
We've been developing the NFS/RDMA software on the OpenFabrics (aka 
OpenIB) code since it was merged into 2.6.10 in Dec 2004.

> currently I have new hardware. I'm running Tom's kernel and already
> did some MPI tests. SDP is not working, probably because sdp kernel
> modules where not build. ;) I understand that those modules are only
> available from ofa-kernel. please correct me if i'm wrong.

Correct. SDP has never been submitted to mainline Linux.

> system is Scientic Linux 4.5, which is supposed to be a fully
> compatible RH4 clone. hardware is Supermicro mobos with Mellanox
> MT25204 and Flextronisc switch.
> 
> error log from ofa-kernel build:

Is your goal to build a kernel with an NFS/RDMA server? If so, the 
kernel sources from Tom Tucker's git tree are the ones you want, not 
the old OFED 1.2-based packages which are out of date.

Did you try setting up the NFS/RDMA server on the kernel used for your 
MPI tests above?

> > > make[1]: Entering directory `/usr/src/ib/xprt-switch-2.6'
> > > test -e include/linux/autoconf.h -a -e include/config/auto.conf || (            \
> > > echo;                                                           \
> > > echo "  ERROR: Kernel configuration is invalid.";               \
> > > echo "         include/linux/autoconf.h or include/config/auto.conf are missing.";      \
> > > echo "         Run 'make oldconfig && make prepare' on kernel src to fix it.";  \
> > > echo;                                                           \
> > > /bin/false)
> > > 
> > > obviously, doing 'make oldconfig && make prepare' does not help. 
> > > anyway, above mentioned files do exist:
> > > 
> > > # ls -la /usr/src/ib/xprt-switch-2.6/{include/linux/autoconf.h,include/config/auto.conf}
> > > -rw-r--r--  1 root root 10156 Jan 25 17:42 /usr/src/ib/xprt-switch-2.6/include/config/auto.conf
> > > -rw-r--r--  1 root root 14733 Jan 25 17:42 /usr/src/ib/xprt-switch-2.6/include/linux/autoconf.h
> > > 
> > > despite of above, compilation continues but fails with:
> > > 
> > >   gcc -Wp,-MD,/var/tmp/OFEDRPM/BUILD/ofa_kernel-1.2/drivers/infiniband/core/.mad.o.d  -nostdinc -isystem /usr/lib/gcc/x86_64-redhat-linux/3.4.6/include -D__KERNEL__ -I/var/tmp/OFEDRPM/BUILD/ofa_kernel-1.2/include  -I/var/tmp/OFEDRPM/BUILD/ofa_kernel-1.2/drivers/infiniband/include  -Iinclude    -include include/linux/autoconf.h  -include /var/tmp/OFEDRPM/BUILD/ofa_kernel-1.2/include/linux/autoconf.h   -Wall -Wundef -Wstrict-prototypes -Wno-trigraphs -fno-strict-aliasing -fno-common -Werror-implicit-function-declaration -Os   -m64 -mno-red-zone -mcmodel=kernel -pipe -Wno-sign-compare -fno-asynchronous-unwind-tables -funit-at-a-time -mno-sse -mno-mmx -mno-sse2 -mno-3dnow -maccumulate-outgoing-args -DCONFIG_AS_CFI=1 -DCONFIG_AS_CFI_SIGNAL_FRAME=1 -fomit-frame-pointer  -Wdeclaration-after-statement     -DMODULE -D"KBUILD_STR(s)=#s" -D"KBUILD_BASENAME=KBUILD_STR(mad)"  -D"KBUILD_MODNAME=KBUILD_STR(ib_mad)" -c -o /var/tmp/OFEDRPM/BUILD/ofa_kernel-1.2/drivers/infiniband/core/.!
 tmp
>  _mad.o /var/tmp/OFEDRPM/BUILD/ofa_kernel-1.2/drivers/infiniband/core/mad.c
> > > /var/tmp/OFEDRPM/BUILD/ofa_kernel-1.2/drivers/infiniband/core/mad.c: In function `ib_mad_init_module':
> > > /var/tmp/OFEDRPM/BUILD/ofa_kernel-1.2/drivers/infiniband/core/mad.c:2966: error: too many arguments to function `kmem_cache_create'
> > > make[4]: *** [/var/tmp/OFEDRPM/BUILD/ofa_kernel-1.2/drivers/infiniband/core/mad.o] Error 1
> > > make[3]: *** [/var/tmp/OFEDRPM/BUILD/ofa_kernel-1.2/drivers/infiniband/core] Error 2
> > > make[2]: *** [/var/tmp/OFEDRPM/BUILD/ofa_kernel-1.2/drivers/infiniband] Error 2
> > > make[1]: *** [_module_/var/tmp/OFEDRPM/BUILD/ofa_kernel-1.2] Error 2
> > > make[1]: Leaving directory `/usr/src/ib/xprt-switch-2.6'
> > > make: *** [kernel] Error 2
> > > error: Bad exit status from /var/tmp/rpm-tmp.3877 (%install)
> 
> > > full log:
> > > https://cefeid.wcss.wroc.pl/d/tmp/OFED.build.32122.log
> 
> thanks in advance for any help, P
> 
> 
> -- 
> Pawel Dziekonski <pawel.dziekonski at wcss.pl>
> Wroclaw Centre for Networking & Supercomputing, HPC Department
> Politechnika Wr., pl. Grunwaldzki 9, bud. D2/101, 50-377 Wroclaw, POLAND
> phone: +48 71 3202043, fax: +48 71 3225797, http://www.wcss.wroc.pl
> 
> -------------------------------------------------------------------------
> This SF.net email is sponsored by: Microsoft
> Defy all challenges. Microsoft(R) Visual Studio 2008.
> http://clk.atdmt.com/MRT/go/vse0120000070mrt/direct/01/
> _______________________________________________
> nfs-rdma-devel mailing list
> nfs-rdma-devel at lists.sourceforge.net
> https://lists.sourceforge.net/lists/listinfo/nfs-rdma-devel
> 