[ofa-general] Chelsio T3: Aggregate Throughput

[Philip Frey1]

Hello,

we am currently looking into the scalability of the T3 in terms of
connections. We are using a 1-to-n scenario where the one server
has a chunk of data and n client that fetch this chunk over and over
again using RDMA reads (each 1MB in size).

The clients do that such that they get an average data rate of about
9Mbps each. Every second we connect a new client to the server
and see how far it goes.

What puzzles us now is that after about 800 clients, they do no longer
seem to receive much data.

The first interesting thing is that the aggregate throughput actually 
drops
(we expected it to stall). And the second interesting thing is that it 
does
so already at about 6.3Gbps which is just a bit more than half of what the
card can do. We do not experience this kind of situation when using
much less clients that RDMA read the data at a much higher data rate.

Is there any limitation on the RNIC that would give an explanation for 
this?

(Setup: T3 RNICs, OFED-1.4, 2.6.26 kernel, MTU=9000)

Many thanks for your advice,
 Philip

-- 
   Philip Frey 
   IBM Zurich Research Laboratory
   Saumerstrasse 4                                   |  Phone: +41 44 724 
8613
   CH-8803 Rueschlikon/Switzerland  |  Email: phf at zurich.ibm.com
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.openfabrics.org/pipermail/general/attachments/20090205/c8a1bbbf/attachment.html>