[openib-general] ip over ib throughtput

[Talpey, Thomas]

Hi Shirley - I had a chance to try with the tiny blocksizes but I'm afraid
the results aren't useful to estimate max throughput. The server I am
using runs out of CPU at about 33,600 IOPS for small I/Os (<=4KB), so
with 2000 byte reads, all I can get is about 65MB/sec. (I get 33MB/s
with 1KB, 120MB/s with 4KB, etc). And recall with NFS-default 32KB reads
I get 450MB/s. All these limits are due to this server's CPU at 100%. Time
to find a bigger server!

The good news is, performance is nice and flat right up until the server
hits the CPU wall. In fact, the more directio threads I run in parallel, the
lower the client overhead. With 50 threads issuing reads, I see as little as
0.5 interrupts per I/O!

Sorry I couldn't push more throughput using only small reads. I could trunk
the I/O to multiple servers, but I assume you're only interested in single-
stream results.

Tom.

At 11:11 PM 5/10/2006, Shirley Ma wrote:

>"Talpey, Thomas" <Thomas.Talpey at netapp.com> wrote on 05/10/2006 03:10:57 PM:
>> Sure, but I wonder why it's interesting. Nobody ever uses NFS in such
>> small blocksizes, and 2044 bytes would mean, say, 1800 bytes of payload.
>> What data are you looking for, throughput and overhead? Direct RDMA,
>> or inline?
>> 
>> Tom. 
>
>Throughput. I am wondering how much room IPoIB performance (throughput) can go. 
>
>Thanks 
>Shirley Ma
>IBM Linux Technology Center
>15300 SW Koll Parkway
>Beaverton, OR 97006-6063
>Phone(Fax): (503) 578-7638