[ofa-general] NFS-RDMA (OFED1.4) with standard distributions ?

[Talpey, Thomas]

At 11:27 AM 11/10/2008, Ciesielski, Frederic (EMEA HPC&OSLO CC) wrote:
>That's great, thanks.
>
>I ran some tests with the 2.6.27 kernel as server and client, and 
>basically it works fine.
>
>I could not find yet any situation where NFS-RDMA would outperform 
>NFS/IPoIB, at least when you compare apples to apples (same clients, 
>same server, same protocol, and not just write to/read from the 
>caches), and it even seems to have severe performance issues for 
>reading with files larger than the memory size of the client and the server.
>Hopefully this will improve when more users will be able to give 
>valuable feedback...

I have a couple of questions, and perhaps suggestions as well.
First the questions...

- Have you tried with a 2.6.28-rc4 client and server at all? There are
a number of significant NFS/RDMA improvements queued in kernel.org,
especially around RDMA memory registration as well as RDMA operation
scheduling. We've seen some significant throughput improvement even
for basic tunings.

- What type of storage are you using at the server, and have you
attempted to tune the server at all? For example, if you are storage
(spindle) limited, no network tuning is likely to help and you should
address that first. Also, there are tunings such as nfsd thread count,
export options, and adapter choice that can make a large difference.

Bottom line, you should be able to reach multi-hundred-MB/sec of
read/write throughput with NFS/RDMA, but there may be issues on
specific systems, or perhaps with the OFED1.4 code, that need to
be accounted for. If possible, you may want to set expectations
based on mainline, then try to duplicate them in the OFED backport.
The current OFED NFS/RDMA support is still evolving, while we consider
the mainline kernel.org version to be rather solid.

Tom.

>
>Fred.
>
>-----Original Message-----
>From: Jeff Becker [mailto:Jeffrey.C.Becker at nasa.gov]
>Sent: Saturday, 08 November, 2008 22:35
>To: Ciesielski, Frederic (EMEA HPC&OSLO CC)
>Cc: general at lists.openfabrics.org
>Subject: Re: [ofa-general] NFS-RDMA (OFED1.4) with standard distributions ?
>
>Ciesielski, Frederic (EMEA HPC&OSLO CC) wrote:
>> Is there any chance that the new NFS-RDMA features coming with OFED
>> 1.4 work with standard and current distributions, like RHEL5, SLES10 ?
>Not yet, but I'm working on it. I intend for NFSRDMA to work on 2.6.27 
>and 2.6.26 for OFED 1.4. The RHEL5 and SLES10 backports will likely be 
>done for OFED 1.4.1. Thanks.
>
>-jeff
>
>> Did anybody test this, or would pretend it is supposed to work ?
>>
>> I mean without building a 2.6.27 or equivalent kernel on top of it,
>> keeping almost full support from the vendors.
>>
>> Enhanced kernel modules may not be sufficient to work around the
>> limitations of old kernels...
>>
>>
>>