[openib-general] File transfer performance options

[Tom Tucker]

Are you familiar with NFSoRDMA? We get 600+MB/s on read and about
150-200MB/s on write with an XFS filesystem on a stripped software raid
with four spindles. 

This is Linux <--> Linux. 


On Wed, 2006-08-30 at 20:13 +0100, Paul Baxter wrote:
> We've been testing an application that archives large quantities of data 
> from a Linux system onto a Windows-based server (64bit server 2003 R2).
> 
> As part of the investigation into relatively modest transfer speeds in the 
> win-linux configuration, we configured a Linux-Linux transfer via IpoIB with 
> NFS layered on top (with ram disks to avoid physical disk issues)
> 
> [Whilst for a real Linux-Linux configuration I would look for the RDMA over 
> NFS solution, this wouldn't translate to our eventual win-linux 
> inter-operable system.]
> 
> I was surprised that even on linux-linux I hit a wall of 100MB/s (test notes 
> below). Are others doing better? I was hoping for 150MB/s - 200MB/s
> 
> Does anyone have any hints on tweaking of an IPoIB/NFS solution to get 
> better throughput for large files (not so concerned about latency).
> 
> Are there any other inter-operable windows-linux solutions now? 
> (cross-platform NFS over RDMA or SRP initiator/target?)
> 
> Paul Baxter
> 
> -------------------
> Some testing notes:
> 
> The windows server remotely inspects the Linux filesystem and does a 'remote 
> read' of large files (typical testing 1-4GB file)
> 
> Using IPoIB/mthca and Win IB 1.2 - no particular tweaks i.e. 32 kB NFS block 
> size
> 
> Win-Linux
> a)  Using untweaked Linux NFS and built-in Windows NFS
> Transfer rate 65MB/s
> 
> b) Similar but using Samba on Linux and windows file sharing
> Transfer rate 90 MB/s
> 
> c) Repeat a) and b) using ram disks rather than physical disks (1GB 
> transfer)
> Confirmed similar transfer rates ie physical disks not limiting this
> 
>  Presentations on winIB noted that IPoIB has to snoop each packet , so I 
> repeated test c) in a Linux-Linux configuration expecting much better 
> results...
> 
> NFS performance over Ext2 formatted filesystem ~ 100MB/s (~ 73MB/s on Ext3 
> with default (journalling on?) options)
> Samba performance ~ 64MB/s
> 
> Next tried having recipient of large file, copy it to /dev/null rather than 
> to a local file system. Reported transfer at 145MB/s
> (We've also noted along the way that remote read and remote write
> 
> 
> _______________________________________________
> openib-general mailing list
> openib-general at openib.org
> http://openib.org/mailman/listinfo/openib-general
> 
> To unsubscribe, please visit http://openib.org/mailman/listinfo/openib-general
>