[ofa-general] Re: Status of NFS-RDMA ?
Jeff Becker
Jeffrey.C.Becker at nasa.gov
Wed Jan 30 09:32:09 PST 2008
Hi all.
James Lentini wrote:
> On Wed, 30 Jan 2008, Krishna Kumar2 wrote:
>
>
>> Hi James,
>>
>> Since you had mentioned in an earlier email that NFS-RDMA server
>> side will be present in OFED1.4,
>>
>
> Actually, that was Tziporet.
>
>
>> do you know if any port of the server code to OFED1.3 (when it comes
>> out) will happen? Is there any effort for that, any work ongoing,
>> any help required, etc?
>>
>
> Jeff Becker had looked into this. We would definitely appreciate the
> help.
>
I have set up a git tree for NFSoRDMA and succesfully merged it with,
and built it on OFED 1.3-rcx. I'm currently doing the backports (SLES 10
SP1 first). All this is in preparation for OFED 1.4, as that is when
NFSoRDMA will be included in OFED. I think I have this
patching/backporting stuff under control. However, my testing resources
are limited. Thus depending on your platform, I might be able to point
you at OFED 1.3 based bits for testing if/when they are ready. Thanks.
-jeff
> The NFS framework has changed significantly in several areas in recent
> kernel releases. This has made backporting the NFS/RDMA code to older
> kernels challenging.
>
> If you are interested in working on OFED1.3 support, let us know.
>
>
>> I couldn't find the release time lines for OFED1.4, is there any
>> link on openfabrics homepage?
>>
>
> I'm not involved with the OFED1.4 planning. Tziporet, is there
> information on this?
>
>
>> Thanks,
>>
>> - KK
>>
>> general-bounces at lists.openfabrics.org wrote on 01/29/2008 08:23:46 PM:
>>
>>
>>> On Tue, 29 Jan 2008, Pawel Dziekonski wrote:
>>>
>>>
>>>> On Mon, 28 Jan 2008 at 10:14:22AM -0500, James Lentini wrote:
>>>>
>>>>> On Sat, 26 Jan 2008, Pawel Dziekonski wrote:
>>>>>
>>>>>
>>>>>> I pulled Tom's tree from new url and build a kernel.
>>>>>>
>>>>> If you enabled support for INFINIBAND drivers (IB and iWARP support)
>>>>> and NFS client/server support, the kernel should be ready to go (run
>>>>> "grep RDMA /your_kernel_sources/.config" to confirm that
>>>>> CONFIG_SUNRPC_XPRT_RDMA is either m or y).
>>>>>
>>>>> NFS/RDMA doesn't require OFED be installed. OFED is a release of the
>>>>> Linux kernel sources and some userspace libraries/tools. If you are
>>>>>
>>>>>> then I downloaded OFED from
>>>>>> http://www.mellanox.com/downloads/NFSoRDMA/OFED-1.2-NFS-RDMA.gz,
>>>>>>
>>>>> I don't know what the above URL contains. The latest code is in Tom
>>>>> Tucker's tree (and now NFS server maintainer Bruce Fields tree). It
>>>>>
>> is
>>
>>>> hi,
>>>>
>>>> back to subject on a proper mailing list.
>>>>
>>>> I have a >3 year experience with mellanox hardware and IBGold so I
>>>> basically know what OFED is all about. up to now i was only using
>>>> IBGold since IB drivers appeared in kernel pretty recently.
>>>>
>>> You'll want to use the mainline kernel's IB drivers for NFS/RDMA.
>>> We've been developing the NFS/RDMA software on the OpenFabrics (aka
>>> OpenIB) code since it was merged into 2.6.10 in Dec 2004.
>>>
>>>
>>>> currently I have new hardware. I'm running Tom's kernel and already
>>>> did some MPI tests. SDP is not working, probably because sdp kernel
>>>> modules where not build. ;) I understand that those modules are only
>>>> available from ofa-kernel. please correct me if i'm wrong.
>>>>
>>> Correct. SDP has never been submitted to mainline Linux.
>>>
>>>
>>>> system is Scientic Linux 4.5, which is supposed to be a fully
>>>> compatible RH4 clone. hardware is Supermicro mobos with Mellanox
>>>> MT25204 and Flextronisc switch.
>>>>
>>>> error log from ofa-kernel build:
>>>>
>>> Is your goal to build a kernel with an NFS/RDMA server? If so, the
>>> kernel sources from Tom Tucker's git tree are the ones you want, not
>>> the old OFED 1.2-based packages which are out of date.
>>>
>>> Did you try setting up the NFS/RDMA server on the kernel used for your
>>> MPI tests above?
>>>
>>>
>>>>>> make[1]: Entering directory `/usr/src/ib/xprt-switch-2.6'
>>>>>> test -e include/linux/autoconf.h -a -e include/config/auto.conf ||
>>>>>>
>> ( \
>>
>>>>>> echo; \
>>>>>> echo " ERROR: Kernel configuration is invalid."; \
>>>>>> echo " include/linux/autoconf.h or include/config/auto.conf
>>>>>>
>> are
>>
>>> missing."; \
>>>
>>>>>> echo " Run 'make oldconfig && make prepare' on kernel src
>>>>>>
>> to fix it."; \
>>
>>>>>> echo; \
>>>>>> /bin/false)
>>>>>>
>>>>>> obviously, doing 'make oldconfig && make prepare' does not help.
>>>>>> anyway, above mentioned files do exist:
>>>>>>
>>>>>> # ls -la /usr/src/ib/xprt-switch-2.6/{include/linux/autoconf.h,
>>>>>>
>>> include/config/auto.conf}
>>>
>>>>>> -rw-r--r-- 1 root root 10156 Jan 25 17:42
>>>>>>
>> /usr/src/ib/xprt-switch-2.
>>
>>> 6/include/config/auto.conf
>>>
>>>>>> -rw-r--r-- 1 root root 14733 Jan 25 17:42
>>>>>>
>> /usr/src/ib/xprt-switch-2.
>>
>>> 6/include/linux/autoconf.h
>>>
>>>>>> despite of above, compilation continues but fails with:
>>>>>>
>>>>>> gcc -Wp,-MD,/var/tmp/OFEDRPM/BUILD/ofa_kernel-1.
>>>>>>
>>> 2/drivers/infiniband/core/.mad.o.d -nostdinc -isystem
>>>
>> /usr/lib/gcc/x86_64-
>>
>>> redhat-linux/3.4.6/include -D__KERNEL__
>>>
>> -I/var/tmp/OFEDRPM/BUILD/ofa_kernel-1.
>>
>>> 2/include -I/var/tmp/OFEDRPM/BUILD/ofa_kernel-1.2
>>>
>> /drivers/infiniband/include
>>
>>> -Iinclude -include include/linux/autoconf.h -include
>>> /var/tmp/OFEDRPM/BUILD/ofa_kernel-1.2/include/linux/autoconf.h -Wall
>>>
>> -Wundef
>>
>>> -Wstrict-prototypes -Wno-trigraphs -fno-strict-aliasing -fno-common
>>>
>> -Werror-
>>
>>> implicit-function-declaration -Os -m64 -mno-red-zone -mcmodel=kernel
>>>
>> -pipe -
>>
>>> Wno-sign-compare -fno-asynchronous-unwind-tables -funit-at-a-time
>>>
>> -mno-sse -
>>
>>> mno-mmx -mno-sse2 -mno-3dnow -maccumulate-outgoing-args -DCONFIG_AS_CFI=1
>>>
>> -
>>
>>> DCONFIG_AS_CFI_SIGNAL_FRAME=1 -fomit-frame-pointer -Wdeclaration-after-
>>> statement -DMODULE -D"KBUILD_STR(s)=#s" -
>>> D"KBUILD_BASENAME=KBUILD_STR(mad)" -D"KBUILD_MODNAME=KBUILD_STR(ib_mad)"
>>>
>> -c -
>>
>>> o /var/tmp/OFEDRPM/BUILD/ofa_kernel-1.2/drivers/infiniband/core/.!
>>> tmp
>>>
>>>> _mad.o /var/tmp/OFEDRPM/BUILD/ofa_kernel-1.2
>>>>
>> /drivers/infiniband/core/mad.c
>>
>>>>>> /var/tmp/OFEDRPM/BUILD/ofa_kernel-1.2
>>>>>>
>> /drivers/infiniband/core/mad.c: In
>>
>>> function `ib_mad_init_module':
>>>
>>>>>> /var/tmp/OFEDRPM/BUILD/ofa_kernel-1.2
>>>>>>
>> /drivers/infiniband/core/mad.c:
>>
>>> 2966: error: too many arguments to function `kmem_cache_create'
>>>
>>>>>> make[4]: *** [/var/tmp/OFEDRPM/BUILD/ofa_kernel-1.
>>>>>>
>>> 2/drivers/infiniband/core/mad.o] Error 1
>>>
>>>>>> make[3]: *** [/var/tmp/OFEDRPM/BUILD/ofa_kernel-1.
>>>>>>
>>> 2/drivers/infiniband/core] Error 2
>>>
>>>>>> make[2]: *** [/var/tmp/OFEDRPM/BUILD/ofa_kernel-1.2
>>>>>>
>> /drivers/infiniband] Error 2
>>
>>>>>> make[1]: *** [_module_/var/tmp/OFEDRPM/BUILD/ofa_kernel-1.2] Error
>>>>>>
>> 2
>>
>>>>>> make[1]: Leaving directory `/usr/src/ib/xprt-switch-2.6'
>>>>>> make: *** [kernel] Error 2
>>>>>> error: Bad exit status from /var/tmp/rpm-tmp.3877 (%install)
>>>>>>
>>>>>> full log:
>>>>>> https://cefeid.wcss.wroc.pl/d/tmp/OFED.build.32122.log
>>>>>>
>>>> thanks in advance for any help, P
>>>>
>>>>
>>>> --
>>>> Pawel Dziekonski <pawel.dziekonski at wcss.pl>
>>>> Wroclaw Centre for Networking & Supercomputing, HPC Department
>>>> Politechnika Wr., pl. Grunwaldzki 9, bud. D2/101, 50-377 Wroclaw,
>>>>
>> POLAND
>>
>>>> phone: +48 71 3202043, fax: +48 71 3225797, http://www.wcss.wroc.pl
>>>>
>>>>
>>>>
>> -------------------------------------------------------------------------
>>
>>>> This SF.net email is sponsored by: Microsoft
>>>> Defy all challenges. Microsoft(R) Visual Studio 2008.
>>>> http://clk.atdmt.com/MRT/go/vse0120000070mrt/direct/01/
>>>> _______________________________________________
>>>> nfs-rdma-devel mailing list
>>>> nfs-rdma-devel at lists.sourceforge.net
>>>> https://lists.sourceforge.net/lists/listinfo/nfs-rdma-devel
>>>>
>>>>
>>> _______________________________________________
>>> general mailing list
>>> general at lists.openfabrics.org
>>> http://lists.openfabrics.org/cgi-bin/mailman/listinfo/general
>>>
>>> To unsubscribe, please visit
>>>
>> http://openib.org/mailman/listinfo/openib-general
>>
>>
More information about the general
mailing list