[ofa-general] Re: [PATCH 0/9 Rev3] Implement batching skb API and support in IPoIB

[David Miller]

From: Krishna Kumar2 <krkumar2 at in.ibm.com>
Date: Fri, 17 Aug 2007 11:36:03 +0530

> > I ran 3 iterations of 45 sec tests (total 1 hour 16 min, but I will
> > run a longer one tonight). The results are (results in KB/s, and %):
> 
> I ran a 8.5 hours run with no batching + another 8.5 hours run with
> batching (Buffer sizes: "32 128 512 4096 16384", Threads: "1 8 32",
> Each test run time: 3 minutes, Iterations to average: 5). TCP seems
> to get a small improvement.

Using 16K buffer size really isn't going to keep the pipe full enough
for TSO.  And realistically applications queue much more data at a
time.  Also, with smaller buffer sizes can have negative effects for
the dynamic receive and send buffer growth algorithm the kernel uses,
it might consider the connection application limited for too long.

I would really prefer to see numbers that use buffer sizes more in
line with the amount of data that is typically inflight on a 1G
connection on a local network.

Do a tcpdump during the height of the transfer to see about what this
value is.  When an ACK comes in, compare the sequence number it's
ACK'ing with the sequence number of the most recently sent frame.
The difference is approximately the pipe size at maximum congestion
window assuming a loss free local network.