[ofa-general] Re: Status of NFS-RDMA ?
James Lentini
jlentini at netapp.com
Thu Jan 31 07:50:35 PST 2008
Krishna,
If you would like to do some testing/development on NFS/RDMA, take a
look at the current NFS/RDMA code. There are instructions on were to
get it and how to set it up here:
http://nfs-rdma.sourceforge.net/Documents/README
I'm revising the instructions for 2.6.25. I'll be posting the new
version once the first 2.6.25-rc is released. We would appreciate
feedback in this area as well.
james
On Thu, 31 Jan 2008, Krishna Kumar2 wrote:
> Hi Jeff & James,
>
> Great. If you let me know when the bits are ready (I don't always read the
> mailing list),
> I should be able to get some testing done.
>
> Thanks,
>
> - KK
>
> Jeff Becker <Jeffrey.C.Becker at nasa.gov> wrote on 01/30/2008 11:02:09 PM:
>
> > Hi all.
> >
> > James Lentini wrote:
> > > On Wed, 30 Jan 2008, Krishna Kumar2 wrote:
> > >
> > >
> > >> Hi James,
> > >>
> > >> Since you had mentioned in an earlier email that NFS-RDMA server
> > >> side will be present in OFED1.4,
> > >>
> > >
> > > Actually, that was Tziporet.
> > >
> > >
> > >> do you know if any port of the server code to OFED1.3 (when it comes
> > >> out) will happen? Is there any effort for that, any work ongoing,
> > >> any help required, etc?
> > >>
> > >
> > > Jeff Becker had looked into this. We would definitely appreciate the
> > > help.
> > >
> > I have set up a git tree for NFSoRDMA and succesfully merged it with,
> > and built it on OFED 1.3-rcx. I'm currently doing the backports (SLES 10
> > SP1 first). All this is in preparation for OFED 1.4, as that is when
> > NFSoRDMA will be included in OFED. I think I have this
> > patching/backporting stuff under control. However, my testing resources
> > are limited. Thus depending on your platform, I might be able to point
> > you at OFED 1.3 based bits for testing if/when they are ready. Thanks.
> >
> > -jeff
> >
> > > The NFS framework has changed significantly in several areas in recent
> > > kernel releases. This has made backporting the NFS/RDMA code to older
> > > kernels challenging.
> > >
> > > If you are interested in working on OFED1.3 support, let us know.
> > >
> > >
> > >> I couldn't find the release time lines for OFED1.4, is there any
> > >> link on openfabrics homepage?
> > >>
> > >
> > > I'm not involved with the OFED1.4 planning. Tziporet, is there
> > > information on this?
> > >
> > >
> > >> Thanks,
> > >>
> > >> - KK
> > >>
> > >> general-bounces at lists.openfabrics.org wrote on 01/29/2008 08:23:46 PM:
> > >>
> > >>
> > >>> On Tue, 29 Jan 2008, Pawel Dziekonski wrote:
> > >>>
> > >>>
> > >>>> On Mon, 28 Jan 2008 at 10:14:22AM -0500, James Lentini wrote:
> > >>>>
> > >>>>> On Sat, 26 Jan 2008, Pawel Dziekonski wrote:
> > >>>>>
> > >>>>>
> > >>>>>> I pulled Tom's tree from new url and build a kernel.
> > >>>>>>
> > >>>>> If you enabled support for INFINIBAND drivers (IB and iWARP
> support)
> > >>>>> and NFS client/server support, the kernel should be ready to go
> (run
> > >>>>> "grep RDMA /your_kernel_sources/.config" to confirm that
> > >>>>> CONFIG_SUNRPC_XPRT_RDMA is either m or y).
> > >>>>>
> > >>>>> NFS/RDMA doesn't require OFED be installed. OFED is a release of
> the
> > >>>>> Linux kernel sources and some userspace libraries/tools. If you are
> > >>>>>
> > >>>>>> then I downloaded OFED from
> > >>>>>> http://www.mellanox.com/downloads/NFSoRDMA/OFED-1.2-NFS-RDMA.gz,
> > >>>>>>
> > >>>>> I don't know what the above URL contains. The latest code is in Tom
> > >>>>> Tucker's tree (and now NFS server maintainer Bruce Fields tree). It
> > >>>>>
> > >> is
> > >>
> > >>>> hi,
> > >>>>
> > >>>> back to subject on a proper mailing list.
> > >>>>
> > >>>> I have a >3 year experience with mellanox hardware and IBGold so I
> > >>>> basically know what OFED is all about. up to now i was only using
> > >>>> IBGold since IB drivers appeared in kernel pretty recently.
> > >>>>
> > >>> You'll want to use the mainline kernel's IB drivers for NFS/RDMA.
> > >>> We've been developing the NFS/RDMA software on the OpenFabrics (aka
> > >>> OpenIB) code since it was merged into 2.6.10 in Dec 2004.
> > >>>
> > >>>
> > >>>> currently I have new hardware. I'm running Tom's kernel and already
> > >>>> did some MPI tests. SDP is not working, probably because sdp kernel
> > >>>> modules where not build. ;) I understand that those modules are only
> > >>>> available from ofa-kernel. please correct me if i'm wrong.
> > >>>>
> > >>> Correct. SDP has never been submitted to mainline Linux.
> > >>>
> > >>>
> > >>>> system is Scientic Linux 4.5, which is supposed to be a fully
> > >>>> compatible RH4 clone. hardware is Supermicro mobos with Mellanox
> > >>>> MT25204 and Flextronisc switch.
> > >>>>
> > >>>> error log from ofa-kernel build:
> > >>>>
> > >>> Is your goal to build a kernel with an NFS/RDMA server? If so, the
> > >>> kernel sources from Tom Tucker's git tree are the ones you want, not
> > >>> the old OFED 1.2-based packages which are out of date.
> > >>>
> > >>> Did you try setting up the NFS/RDMA server on the kernel used for
> your
> > >>> MPI tests above?
> > >>>
> > >>>
> > >>>>>> make[1]: Entering directory `/usr/src/ib/xprt-switch-2.6'
> > >>>>>> test -e include/linux/autoconf.h -a -e include/config/auto.conf ||
> > >>>>>>
> > >> ( \
> > >>
> > >>>>>> echo; \
> > >>>>>> echo " ERROR: Kernel configuration is invalid."; \
> > >>>>>> echo " include/linux/autoconf.h or
> include/config/auto.conf
> > >>>>>>
> > >> are
> > >>
> > >>> missing."; \
> > >>>
> > >>>>>> echo " Run 'make oldconfig && make prepare' on kernel src
> > >>>>>>
> > >> to fix it."; \
> > >>
> > >>>>>> echo; \
> > >>>>>> /bin/false)
> > >>>>>>
> > >>>>>> obviously, doing 'make oldconfig && make prepare' does not help.
> > >>>>>> anyway, above mentioned files do exist:
> > >>>>>>
> > >>>>>> # ls -la /usr/src/ib/xprt-switch-2.6/{include/linux/autoconf.h,
> > >>>>>>
> > >>> include/config/auto.conf}
> > >>>
> > >>>>>> -rw-r--r-- 1 root root 10156 Jan 25 17:42
> > >>>>>>
> > >> /usr/src/ib/xprt-switch-2.
> > >>
> > >>> 6/include/config/auto.conf
> > >>>
> > >>>>>> -rw-r--r-- 1 root root 14733 Jan 25 17:42
> > >>>>>>
> > >> /usr/src/ib/xprt-switch-2.
> > >>
> > >>> 6/include/linux/autoconf.h
> > >>>
> > >>>>>> despite of above, compilation continues but fails with:
> > >>>>>>
> > >>>>>> gcc -Wp,-MD,/var/tmp/OFEDRPM/BUILD/ofa_kernel-1.
> > >>>>>>
> > >>> 2/drivers/infiniband/core/.mad.o.d -nostdinc -isystem
> > >>>
> > >> /usr/lib/gcc/x86_64-
> > >>
> > >>> redhat-linux/3.4.6/include -D__KERNEL__
> > >>>
> > >> -I/var/tmp/OFEDRPM/BUILD/ofa_kernel-1.
> > >>
> > >>> 2/include -I/var/tmp/OFEDRPM/BUILD/ofa_kernel-1.2
> > >>>
> > >> /drivers/infiniband/include
> > >>
> > >>> -Iinclude -include include/linux/autoconf.h -include
> > >>> /var/tmp/OFEDRPM/BUILD/ofa_kernel-1.2/include/linux/autoconf.h
> -Wall
> > >>>
> > >> -Wundef
> > >>
> > >>> -Wstrict-prototypes -Wno-trigraphs -fno-strict-aliasing -fno-common
> > >>>
> > >> -Werror-
> > >>
> > >>> implicit-function-declaration -Os -m64 -mno-red-zone
> -mcmodel=kernel
> > >>>
> > >> -pipe -
> > >>
> > >>> Wno-sign-compare -fno-asynchronous-unwind-tables -funit-at-a-time
> > >>>
> > >> -mno-sse -
> > >>
> > >>> mno-mmx -mno-sse2 -mno-3dnow -maccumulate-outgoing-args
> -DCONFIG_AS_CFI=1
> > >>>
> > >> -
> > >>
> > >>> DCONFIG_AS_CFI_SIGNAL_FRAME=1 -fomit-frame-pointer
> -Wdeclaration-after-
> > >>> statement -DMODULE -D"KBUILD_STR(s)=#s" -
> > >>> D"KBUILD_BASENAME=KBUILD_STR(mad)"
> -D"KBUILD_MODNAME=KBUILD_STR(ib_mad)"
> > >>>
> > >> -c -
> > >>
> > >>> o /var/tmp/OFEDRPM/BUILD/ofa_kernel-1.2/drivers/infiniband/core/.!
> > >>> tmp
> > >>>
> > >>>> _mad.o /var/tmp/OFEDRPM/BUILD/ofa_kernel-1.2
> > >>>>
> > >> /drivers/infiniband/core/mad.c
> > >>
> > >>>>>> /var/tmp/OFEDRPM/BUILD/ofa_kernel-1.2
> > >>>>>>
> > >> /drivers/infiniband/core/mad.c: In
> > >>
> > >>> function `ib_mad_init_module':
> > >>>
> > >>>>>> /var/tmp/OFEDRPM/BUILD/ofa_kernel-1.2
> > >>>>>>
> > >> /drivers/infiniband/core/mad.c:
> > >>
> > >>> 2966: error: too many arguments to function `kmem_cache_create'
> > >>>
> > >>>>>> make[4]: *** [/var/tmp/OFEDRPM/BUILD/ofa_kernel-1.
> > >>>>>>
> > >>> 2/drivers/infiniband/core/mad.o] Error 1
> > >>>
> > >>>>>> make[3]: *** [/var/tmp/OFEDRPM/BUILD/ofa_kernel-1.
> > >>>>>>
> > >>> 2/drivers/infiniband/core] Error 2
> > >>>
> > >>>>>> make[2]: *** [/var/tmp/OFEDRPM/BUILD/ofa_kernel-1.2
> > >>>>>>
> > >> /drivers/infiniband] Error 2
> > >>
> > >>>>>> make[1]: *** [_module_/var/tmp/OFEDRPM/BUILD/ofa_kernel-1.2] Error
> > >>>>>>
> > >> 2
> > >>
> > >>>>>> make[1]: Leaving directory `/usr/src/ib/xprt-switch-2.6'
> > >>>>>> make: *** [kernel] Error 2
> > >>>>>> error: Bad exit status from /var/tmp/rpm-tmp.3877 (%install)
> > >>>>>>
> > >>>>>> full log:
> > >>>>>> https://cefeid.wcss.wroc.pl/d/tmp/OFED.build.32122.log
> > >>>>>>
> > >>>> thanks in advance for any help, P
> > >>>>
> > >>>>
> > >>>> --
> > >>>> Pawel Dziekonski <pawel.dziekonski at wcss.pl>
> > >>>> Wroclaw Centre for Networking & Supercomputing, HPC Department
> > >>>> Politechnika Wr., pl. Grunwaldzki 9, bud. D2/101, 50-377 Wroclaw,
> > >>>>
> > >> POLAND
> > >>
> > >>>> phone: +48 71 3202043, fax: +48 71 3225797, http://www.wcss.wroc.pl
> > >>>>
> > >>>>
> > >>>>
> > >>
> -------------------------------------------------------------------------
> > >>
> > >>>> This SF.net email is sponsored by: Microsoft
> > >>>> Defy all challenges. Microsoft(R) Visual Studio 2008.
> > >>>> http://clk.atdmt.com/MRT/go/vse0120000070mrt/direct/01/
> > >>>> _______________________________________________
> > >>>> nfs-rdma-devel mailing list
> > >>>> nfs-rdma-devel at lists.sourceforge.net
> > >>>> https://lists.sourceforge.net/lists/listinfo/nfs-rdma-devel
> > >>>>
> > >>>>
> > >>> _______________________________________________
> > >>> general mailing list
> > >>> general at lists.openfabrics.org
> > >>> http://lists.openfabrics.org/cgi-bin/mailman/listinfo/general
> > >>>
> > >>> To unsubscribe, please visit
> > >>>
> > >> http://openib.org/mailman/listinfo/openib-general
> > >>
> > >>
> >
>
More information about the general
mailing list