[ofa-general] Re: [ewg] [PATCH] IB/ipoib: copy small SKBs in CM mode

[Eli Cohen]

On Fri, 2008-05-23 at 11:05 -0700, Roland Dreier wrote:
> > ---
>  > 
>  > When running netperf I see significant improvement when using this patch
>  > (BW Mbps):
>  > 
>  > with patch:
>  > sender		receiver
>  > 313		313
>  > 
>  > without the patch:
>  > 509		134
> 
> Any reason why we wouldn't want this info in the patch changelog?
Not really. If you think it should be there, I'll add it to the
changelog along with an explanation to the question bellow.

> 
> Can you explain why the sender gets dramatically slower with the patch?
> 
When using this patch, the overhead of the CPU for handling RX packets
is dramatically reduced. As a result, we do not experience RNR NACK
messages from the receiver which cause the connection to be closed and
reopened again; when the patch is not used, the receiver cannot handle
the packets fast enough so there is less time to post new buffers and
hence the mentioned RNR NACKs. So what happens is that the application,
e.g. netperf, *thinks* it posted a certain number of packets for
transmission but these packets are flushed and do not really get
transmitted. Since the connection gets opened and closed many times,
each time netperf gets the CPU time that otherwise would have been given
to the CPU to actually transmit the packtes. This can be verified when
looking at the port counters, the output of ifconfig and the oputput of
netperf (this is for the case without the patch):

tx packets
==========
port counter:	1,543,996
ifconfig:	1,581,426
netperf:	5,142,034

rx packtes
==========
netperf		1,1304,089
	
UDP UNIDIRECTIONAL SEND TEST from 0.0.0.0 (0.0.0.0) port 0 AF_INET to
14.4.3.178 (14.4.3.178) port 0 AF_INET
Socket  Message  Elapsed      Messages                
Size    Size     Time         Okay Errors   Throughput
bytes   bytes    secs            #      #   10^6bits/sec

114688     128   10.00     5142034      0     526.31
114688           10.00     1130489            115.71