[openib-general] Re: [PATCH] ipoib tune rx ring size

[Grant Grundler]

On Mon, Jun 20, 2005 at 11:23:30PM +0300, Michael S. Tsirkin wrote:
> Grant, are you willing to test the patch and post what effect if any this has
> on netperf?

Conclusion: RX_RING @ 64 degrades netperf by about 1%.

I'm certain my runs are CPU bound.
I'm not certain that's true for your config.
Can you post netperf output?

Original SVN r2577 code running netperf 2.4.0:
(Sorry stats are busted on this verion of netperf - I'm tracking down that bug)

/usr/local/bin/netperf -l 60 -H 10.0.0.30 -t TCP_STREAM -T 1 -c 1024 -C 1024 -- -m 131072 -s 131072 -S 131072
Recv   Send    Send                          Utilization       Service Demand
Socket Socket  Message  Elapsed              Send     Recv     Send    Recv
Size   Size    Size     Time     Throughput  local    remote   local   remote
bytes  bytes   bytes    secs.    10^6bits/s  % S      % S      us/KB   us/KB

249856 249856 131072    60.00      2728.02   34.45    42.64    2.069   2.561 

With IPOIB_RX_RING_SIZE at 64: 
249856 249856 131072    60.00      2702.37   34.39    42.46    2.085   2.574 
249856 249856 131072    60.00      2700.92   34.35    42.36    2.084   2.569 
249856 249856 131072    60.00      2701.02   34.61    42.36    2.100   2.569 

I'm satisfied the runs are fairly consistent.

More caveats:
Original run had CPU pinned to same CPU that was taking interrupts
for both ends of the connection.
I just happened to get the same CPU on "netperf" client side
and was pinning the netserver ("-T 1") side.
(If I didn't, I'd see ~3300-3500 Mb/s.)


However, binding the tasks to the non-interrupt CPU ("-T 0,0") results
in essentially no change in perf.

With RX_RING @ 128:
249856 249856 131072    60.00      3585.08   61.31    70.67    2.802   3.230 
249856 249856 131072    60.00      3577.81   60.28    70.74    2.760   3.239 
249856 249856 131072    60.00      3577.27   61.16    70.38    2.801   3.223 

With RX_RING @ 64:
249856 249856 131072    60.00      3590.58   60.94    70.48    2.781   3.216 
249856 249856 131072    60.00      3569.83   61.25    70.47    2.811   3.234 
249856 249856 131072    60.00      3564.18   60.32    70.48    2.773   3.240 


hth,
grant