[openib-general] File transfer performance options

[Cain, Brian (GE Healthcare)]

> -----Original Message-----
> From: openib-general-bounces at openib.org 
> [mailto:openib-general-bounces at openib.org] On Behalf Of Paul Baxter
> Sent: Wednesday, August 30, 2006 2:13 PM
> To: openib-general at openib.org
> Cc: openib-windows at openib.org
> Subject: [openib-general] File transfer performance options
> 
> We've been testing an application that archives large 
> quantities of data 
> from a Linux system onto a Windows-based server (64bit server 
> 2003 R2).
> 
> As part of the investigation into relatively modest transfer 
> speeds in the 
> win-linux configuration, we configured a Linux-Linux transfer 
> via IpoIB with 
> NFS layered on top (with ram disks to avoid physical disk issues)
> 
> [Whilst for a real Linux-Linux configuration I would look for 
> the RDMA over 
> NFS solution, this wouldn't translate to our eventual win-linux 
> inter-operable system.]
> 
> I was surprised that even on linux-linux I hit a wall of 
> 100MB/s (test notes 
> below). Are others doing better? I was hoping for 150MB/s - 200MB/s

I can report streaming write results (using SRP, not NFS/IPoIB) of
around 380MiB/s.  Right now we think that there's a disk or controller
bottleneck on the SRP target that's keeping us from getting up near
450-500 MiB/s or so.  Both the initiator and target are linux-based.  I
think I heard of someone here using a Windows initiator and getting
streaming write results similar to the 380MiB/s we're getting now.  I
guess there's quite a few differences in the scenarios we're describing,
so it's pretty far from apples to apples.  OBTW, in my experience,
ext[23] seriously hamper performance.  Try XFS or ReiserFS.  The numbers
above are all for XFS-formatted partitions.

Maybe you should make the test notes a little more detailed.  Doesn't
NFS have a bunch of performance knobs (TCP vs UDP, block sizes, etc)?

-Brian