[ofa-general] Re: Status of NFS-RDMA ?
Krishna Kumar2
krkumar2 at in.ibm.com
Wed Jan 30 20:16:10 PST 2008
Hi Jeff & James,
Great. If you let me know when the bits are ready (I don't always read the
mailing list),
I should be able to get some testing done.
Thanks,
- KK
Jeff Becker <Jeffrey.C.Becker at nasa.gov> wrote on 01/30/2008 11:02:09 PM:
> Hi all.
>
> James Lentini wrote:
> > On Wed, 30 Jan 2008, Krishna Kumar2 wrote:
> >
> >
> >> Hi James,
> >>
> >> Since you had mentioned in an earlier email that NFS-RDMA server
> >> side will be present in OFED1.4,
> >>
> >
> > Actually, that was Tziporet.
> >
> >
> >> do you know if any port of the server code to OFED1.3 (when it comes
> >> out) will happen? Is there any effort for that, any work ongoing,
> >> any help required, etc?
> >>
> >
> > Jeff Becker had looked into this. We would definitely appreciate the
> > help.
> >
> I have set up a git tree for NFSoRDMA and succesfully merged it with,
> and built it on OFED 1.3-rcx. I'm currently doing the backports (SLES 10
> SP1 first). All this is in preparation for OFED 1.4, as that is when
> NFSoRDMA will be included in OFED. I think I have this
> patching/backporting stuff under control. However, my testing resources
> are limited. Thus depending on your platform, I might be able to point
> you at OFED 1.3 based bits for testing if/when they are ready. Thanks.
>
> -jeff
>
> > The NFS framework has changed significantly in several areas in recent
> > kernel releases. This has made backporting the NFS/RDMA code to older
> > kernels challenging.
> >
> > If you are interested in working on OFED1.3 support, let us know.
> >
> >
> >> I couldn't find the release time lines for OFED1.4, is there any
> >> link on openfabrics homepage?
> >>
> >
> > I'm not involved with the OFED1.4 planning. Tziporet, is there
> > information on this?
> >
> >
> >> Thanks,
> >>
> >> - KK
> >>
> >> general-bounces at lists.openfabrics.org wrote on 01/29/2008 08:23:46 PM:
> >>
> >>
> >>> On Tue, 29 Jan 2008, Pawel Dziekonski wrote:
> >>>
> >>>
> >>>> On Mon, 28 Jan 2008 at 10:14:22AM -0500, James Lentini wrote:
> >>>>
> >>>>> On Sat, 26 Jan 2008, Pawel Dziekonski wrote:
> >>>>>
> >>>>>
> >>>>>> I pulled Tom's tree from new url and build a kernel.
> >>>>>>
> >>>>> If you enabled support for INFINIBAND drivers (IB and iWARP
support)
> >>>>> and NFS client/server support, the kernel should be ready to go
(run
> >>>>> "grep RDMA /your_kernel_sources/.config" to confirm that
> >>>>> CONFIG_SUNRPC_XPRT_RDMA is either m or y).
> >>>>>
> >>>>> NFS/RDMA doesn't require OFED be installed. OFED is a release of
the
> >>>>> Linux kernel sources and some userspace libraries/tools. If you are
> >>>>>
> >>>>>> then I downloaded OFED from
> >>>>>> http://www.mellanox.com/downloads/NFSoRDMA/OFED-1.2-NFS-RDMA.gz,
> >>>>>>
> >>>>> I don't know what the above URL contains. The latest code is in Tom
> >>>>> Tucker's tree (and now NFS server maintainer Bruce Fields tree). It
> >>>>>
> >> is
> >>
> >>>> hi,
> >>>>
> >>>> back to subject on a proper mailing list.
> >>>>
> >>>> I have a >3 year experience with mellanox hardware and IBGold so I
> >>>> basically know what OFED is all about. up to now i was only using
> >>>> IBGold since IB drivers appeared in kernel pretty recently.
> >>>>
> >>> You'll want to use the mainline kernel's IB drivers for NFS/RDMA.
> >>> We've been developing the NFS/RDMA software on the OpenFabrics (aka
> >>> OpenIB) code since it was merged into 2.6.10 in Dec 2004.
> >>>
> >>>
> >>>> currently I have new hardware. I'm running Tom's kernel and already
> >>>> did some MPI tests. SDP is not working, probably because sdp kernel
> >>>> modules where not build. ;) I understand that those modules are only
> >>>> available from ofa-kernel. please correct me if i'm wrong.
> >>>>
> >>> Correct. SDP has never been submitted to mainline Linux.
> >>>
> >>>
> >>>> system is Scientic Linux 4.5, which is supposed to be a fully
> >>>> compatible RH4 clone. hardware is Supermicro mobos with Mellanox
> >>>> MT25204 and Flextronisc switch.
> >>>>
> >>>> error log from ofa-kernel build:
> >>>>
> >>> Is your goal to build a kernel with an NFS/RDMA server? If so, the
> >>> kernel sources from Tom Tucker's git tree are the ones you want, not
> >>> the old OFED 1.2-based packages which are out of date.
> >>>
> >>> Did you try setting up the NFS/RDMA server on the kernel used for
your
> >>> MPI tests above?
> >>>
> >>>
> >>>>>> make[1]: Entering directory `/usr/src/ib/xprt-switch-2.6'
> >>>>>> test -e include/linux/autoconf.h -a -e include/config/auto.conf ||
> >>>>>>
> >> ( \
> >>
> >>>>>> echo; \
> >>>>>> echo " ERROR: Kernel configuration is invalid."; \
> >>>>>> echo " include/linux/autoconf.h or
include/config/auto.conf
> >>>>>>
> >> are
> >>
> >>> missing."; \
> >>>
> >>>>>> echo " Run 'make oldconfig && make prepare' on kernel src
> >>>>>>
> >> to fix it."; \
> >>
> >>>>>> echo; \
> >>>>>> /bin/false)
> >>>>>>
> >>>>>> obviously, doing 'make oldconfig && make prepare' does not help.
> >>>>>> anyway, above mentioned files do exist:
> >>>>>>
> >>>>>> # ls -la /usr/src/ib/xprt-switch-2.6/{include/linux/autoconf.h,
> >>>>>>
> >>> include/config/auto.conf}
> >>>
> >>>>>> -rw-r--r-- 1 root root 10156 Jan 25 17:42
> >>>>>>
> >> /usr/src/ib/xprt-switch-2.
> >>
> >>> 6/include/config/auto.conf
> >>>
> >>>>>> -rw-r--r-- 1 root root 14733 Jan 25 17:42
> >>>>>>
> >> /usr/src/ib/xprt-switch-2.
> >>
> >>> 6/include/linux/autoconf.h
> >>>
> >>>>>> despite of above, compilation continues but fails with:
> >>>>>>
> >>>>>> gcc -Wp,-MD,/var/tmp/OFEDRPM/BUILD/ofa_kernel-1.
> >>>>>>
> >>> 2/drivers/infiniband/core/.mad.o.d -nostdinc -isystem
> >>>
> >> /usr/lib/gcc/x86_64-
> >>
> >>> redhat-linux/3.4.6/include -D__KERNEL__
> >>>
> >> -I/var/tmp/OFEDRPM/BUILD/ofa_kernel-1.
> >>
> >>> 2/include -I/var/tmp/OFEDRPM/BUILD/ofa_kernel-1.2
> >>>
> >> /drivers/infiniband/include
> >>
> >>> -Iinclude -include include/linux/autoconf.h -include
> >>> /var/tmp/OFEDRPM/BUILD/ofa_kernel-1.2/include/linux/autoconf.h
-Wall
> >>>
> >> -Wundef
> >>
> >>> -Wstrict-prototypes -Wno-trigraphs -fno-strict-aliasing -fno-common
> >>>
> >> -Werror-
> >>
> >>> implicit-function-declaration -Os -m64 -mno-red-zone
-mcmodel=kernel
> >>>
> >> -pipe -
> >>
> >>> Wno-sign-compare -fno-asynchronous-unwind-tables -funit-at-a-time
> >>>
> >> -mno-sse -
> >>
> >>> mno-mmx -mno-sse2 -mno-3dnow -maccumulate-outgoing-args
-DCONFIG_AS_CFI=1
> >>>
> >> -
> >>
> >>> DCONFIG_AS_CFI_SIGNAL_FRAME=1 -fomit-frame-pointer
-Wdeclaration-after-
> >>> statement -DMODULE -D"KBUILD_STR(s)=#s" -
> >>> D"KBUILD_BASENAME=KBUILD_STR(mad)"
-D"KBUILD_MODNAME=KBUILD_STR(ib_mad)"
> >>>
> >> -c -
> >>
> >>> o /var/tmp/OFEDRPM/BUILD/ofa_kernel-1.2/drivers/infiniband/core/.!
> >>> tmp
> >>>
> >>>> _mad.o /var/tmp/OFEDRPM/BUILD/ofa_kernel-1.2
> >>>>
> >> /drivers/infiniband/core/mad.c
> >>
> >>>>>> /var/tmp/OFEDRPM/BUILD/ofa_kernel-1.2
> >>>>>>
> >> /drivers/infiniband/core/mad.c: In
> >>
> >>> function `ib_mad_init_module':
> >>>
> >>>>>> /var/tmp/OFEDRPM/BUILD/ofa_kernel-1.2
> >>>>>>
> >> /drivers/infiniband/core/mad.c:
> >>
> >>> 2966: error: too many arguments to function `kmem_cache_create'
> >>>
> >>>>>> make[4]: *** [/var/tmp/OFEDRPM/BUILD/ofa_kernel-1.
> >>>>>>
> >>> 2/drivers/infiniband/core/mad.o] Error 1
> >>>
> >>>>>> make[3]: *** [/var/tmp/OFEDRPM/BUILD/ofa_kernel-1.
> >>>>>>
> >>> 2/drivers/infiniband/core] Error 2
> >>>
> >>>>>> make[2]: *** [/var/tmp/OFEDRPM/BUILD/ofa_kernel-1.2
> >>>>>>
> >> /drivers/infiniband] Error 2
> >>
> >>>>>> make[1]: *** [_module_/var/tmp/OFEDRPM/BUILD/ofa_kernel-1.2] Error
> >>>>>>
> >> 2
> >>
> >>>>>> make[1]: Leaving directory `/usr/src/ib/xprt-switch-2.6'
> >>>>>> make: *** [kernel] Error 2
> >>>>>> error: Bad exit status from /var/tmp/rpm-tmp.3877 (%install)
> >>>>>>
> >>>>>> full log:
> >>>>>> https://cefeid.wcss.wroc.pl/d/tmp/OFED.build.32122.log
> >>>>>>
> >>>> thanks in advance for any help, P
> >>>>
> >>>>
> >>>> --
> >>>> Pawel Dziekonski <pawel.dziekonski at wcss.pl>
> >>>> Wroclaw Centre for Networking & Supercomputing, HPC Department
> >>>> Politechnika Wr., pl. Grunwaldzki 9, bud. D2/101, 50-377 Wroclaw,
> >>>>
> >> POLAND
> >>
> >>>> phone: +48 71 3202043, fax: +48 71 3225797, http://www.wcss.wroc.pl
> >>>>
> >>>>
> >>>>
> >>
-------------------------------------------------------------------------
> >>
> >>>> This SF.net email is sponsored by: Microsoft
> >>>> Defy all challenges. Microsoft(R) Visual Studio 2008.
> >>>> http://clk.atdmt.com/MRT/go/vse0120000070mrt/direct/01/
> >>>> _______________________________________________
> >>>> nfs-rdma-devel mailing list
> >>>> nfs-rdma-devel at lists.sourceforge.net
> >>>> https://lists.sourceforge.net/lists/listinfo/nfs-rdma-devel
> >>>>
> >>>>
> >>> _______________________________________________
> >>> general mailing list
> >>> general at lists.openfabrics.org
> >>> http://lists.openfabrics.org/cgi-bin/mailman/listinfo/general
> >>>
> >>> To unsubscribe, please visit
> >>>
> >> http://openib.org/mailman/listinfo/openib-general
> >>
> >>
>
More information about the general
mailing list