[openib-general] File transfer performance options

[Paul Baxter]

We've been testing an application that archives large quantities of data 
from a Linux system onto a Windows-based server (64bit server 2003 R2).

As part of the investigation into relatively modest transfer speeds in the 
win-linux configuration, we configured a Linux-Linux transfer via IpoIB with 
NFS layered on top (with ram disks to avoid physical disk issues)

[Whilst for a real Linux-Linux configuration I would look for the RDMA over 
NFS solution, this wouldn't translate to our eventual win-linux 
inter-operable system.]

I was surprised that even on linux-linux I hit a wall of 100MB/s (test notes 
below). Are others doing better? I was hoping for 150MB/s - 200MB/s

Does anyone have any hints on tweaking of an IPoIB/NFS solution to get 
better throughput for large files (not so concerned about latency).

Are there any other inter-operable windows-linux solutions now? 
(cross-platform NFS over RDMA or SRP initiator/target?)

Paul Baxter

-------------------
Some testing notes:

The windows server remotely inspects the Linux filesystem and does a 'remote 
read' of large files (typical testing 1-4GB file)

Using IPoIB/mthca and Win IB 1.2 - no particular tweaks i.e. 32 kB NFS block 
size

Win-Linux
a)  Using untweaked Linux NFS and built-in Windows NFS
Transfer rate 65MB/s

b) Similar but using Samba on Linux and windows file sharing
Transfer rate 90 MB/s

c) Repeat a) and b) using ram disks rather than physical disks (1GB 
transfer)
Confirmed similar transfer rates ie physical disks not limiting this

 Presentations on winIB noted that IPoIB has to snoop each packet , so I 
repeated test c) in a Linux-Linux configuration expecting much better 
results...

NFS performance over Ext2 formatted filesystem ~ 100MB/s (~ 73MB/s on Ext3 
with default (journalling on?) options)
Samba performance ~ 64MB/s

Next tried having recipient of large file, copy it to /dev/null rather than 
to a local file system. Reported transfer at 145MB/s
(We've also noted along the way that remote read and remote write