[Openib-windows] NFS performance and general disk network exportadvice (Linux-Windows)

Thu Feb 9 14:58:04 PST 2006

Here is some help from my side. I'm not so familiar with NFS but I'll
try to help.

IPoIB works well between Linux and windows, so running some NFS client
should work well if both sides uses IPoIB.

If, and this is a big if, both the NFS client on windows and the Linux
sides are using (TCP) sockets, one can use SDP in order to communicate
with high bandwidth.

Please note that even if both are using sockets, there is a need to
verify if the windows side is using the sockets from the user mode or
from the kernel. Currently SDP can only work with sockets at the user
level and if the client is actually a driver we should either write a
connecting layer, or find another client.

SDP on windows currently reaches ~1250MB of bandwidth, so I believe that
you will get the BW you need.

Thanks
Tzachi

> -----Original Message-----
> From: openib-windows-bounces at openib.org 
> [mailto:openib-windows-bounces at openib.org] On Behalf Of Paul Baxter
> Sent: Thursday, February 09, 2006 10:18 PM
> To: openib-windows at openib.org
> Cc: openib-general at openib.org
> Subject: [Openib-windows] NFS performance and general disk 
> network exportadvice (Linux-Windows)
> 
> I'm looking to export a filesystem from each of four linux 
> 64bit boxes to a single Windows server 2003 64bit Ed.
> 
> Has anyone achieved this already using an IB transport? Can I 
> use NFS over IPoIB cross platform? i.e. do both ends support 
> a solution?
> 
> Is NFS over RDMA compatible with Windows (pretty sure the 
> answer is no to this one but love to be proven wrong). I've 
> attached Tom's announcement of the latest to the bottom of 
> this email. I don't think Windows has the RDMA abstraction (yet)?
> 
> Are windows IB drivers (Openib or Mellanox) compatible with 
> these options? 
> Do I layer Windows services for Unix on top of the Windows IB 
> drivers and IPoIB to achieve a cross platform NFS?
> 
> Has anyone done much in the way of NFS performance 
> comparisons of NFS over IPoIB in cross-platform situations vs 
> say Gigabit ethernet. Does it work :) What is large file 
> throughput and processor loading - I'm aiming for 150-200 
> MB/s on large files on 4x SDR IB (possibly DDR if we can fit 
> the bigger 144 port switch chassis into our rack layout for 
> 50-ish nodes).
> 
> Are there any alternatives to using NFS that may be better 
> and that would 'transparently' receive a performance boost 
> with IB compared with using a simple NFS/gigabit ethernet 
> solution. Must be fairly straightforward, ideally application 
> neutral (configure a drive and load/unload script for Linux 
> and it just happens) and compatible between Win2003 and Linux? 
> Alternatives using perhaps Samba on the Linux side?
> 
> My lack of knowledge of IB in the windows world has got me 
> concerned over whether this is actually achievable (easily).
> 
> I hope to be trying this once we get a Windows 2003 machine, 
> but hope someone can encourage me that its a breeze prior to 
> my coming unstuck in a month or so!
> 
> Some detail about the bit I do understand:
> 
> I will be using a patched Linux kernel (realtime preemption 
> patches ) but prefer not to apply/track too many kernel 
> patches as the kernel evolves. The NFS patches suggested by 
> Tom in his announcement below make me a little nervous.
> 
> The application will alternate between a real-time mode with 
> (probably) no NFS (or similar network exporting of the disk) 
> and an archiving mode where Linux will load relevant network 
> filesystem modules and let the windows machine read the disks.
> 
> The reason for this odd load/unload behaviour is because our 
> current experience with NFS has been that the driver is prone 
> to putting multi-millisecond glitches that have a habit of 
> upsetting (soft) real-time behaviour at the sorts of timing 
> latencies we're looking at (milliseond or two). NFS (and 
> network cards) do like to batch up work and then run these 
> from interrupt contexts. SoftIRQs help tremendously but don't 
> seem to be the complete answer.
> 
> Paul Baxter
> 
> Tom's announcement:
> > We have released an updated NFS/RDMA client for Linux at 
> the project's 
> > Sourceforge site:
> >
> > <http://sourceforge.net/projects/nfs-rdma/>
> >
> > 
> <http://sourceforge.net/project/showfiles.php?group_id=97628&package_i
> > d=178973>
> >
> > This release updates the RPC/RDMA support as follows:
> > Linux 2.6.15.2 supported
> > Integrates with RPC via 2.6.15 transport switch Employs OpenIB RDMA 
> > verbs API (not kDAPL) Dual BSD/GPL2 licensing
> >
> > There are no protocol changes in this release, it is 
> identical to the 
> > previous release (and the IETF draft) in this respect. The 
> client has 
> > been tested with NFSv3 and passes the Connectathon test suite.
> >
> > At present, the client requires some additional transport switch 
> > patches to be applied to the Linux kernel, these are available at 
> > Chuck Lever's patches page:
> > 
> <http://troy.citi.umich.edu/~cel/linux-2.6/2.6.15/release-notes.html>
> >
> > The related CITI NFS/RDMA server project is currently available for 
> > 2.6.14 from:
> >
> > <http://www.citi.umich.edu/projects/rdma/>
> >
> > 
> <http://www.citi.umich.edu/projects/rdma/patches/stage2/2.6.14.3-RPCRD
> > MA_stage2_2005-12-19.patch>
> >
> > This server is functional but only supports small RDMA inline data 
> > transfers, and a single request in flight. So, its performance is 
> > quite far from the potential. However, it is functional and is the 
> > server we pass Connectathon with!
> >
> > The server project is now being developed by Open Grid Computing, 
> > moving to the OpenIB common RDMA verbs API. We'll be making 
> updates to 
> > both client and server as they become available. There's a 
> lot more to 
> > do.
> >
> > We look forward to comments and feedback from the various standards 
> > and open source communities on this. Feel free to use the 
> mailing list 
> > on the sourceforge project site, or any of these lists (which we 
> > usually
> > monitor) but cc at least me and James Lentini (jlentini at netapp.com).
> >
> > Thanks,
> > Tom Talpey, for the various NFS/RDMA projects.
> 
> _______________________________________________
> openib-windows mailing list
> openib-windows at openib.org
> http://openib.org/mailman/listinfo/openib-windows
> 