[openib-general] Timeline of IPoIB performance

[Rick Jones]

Roland Dreier wrote:
>      > 2.6.12-rc5      in-kernel    1     405   <<<<<
>      > 2.6.12-rc4      in-kernel    1     470   <<<<<
> 
> I was optimistic when I saw this, because the changeover to git
> occurred with 2.6.12-rc2, so I thought I could use git bisect to track
> down exactly when the performance regression happened.
> 
> However, I haven't been able to get numbers that are stable enough to
> track this down.  I have two systems, both HP DL145s with dual Opteron
> 875s and two-port mem-free PCI Express HCAs.  I use MSI-X with the
> completion interrupt affinity set to CPU 0, and "taskset 2" to run
> netserver and netperf on CPU 1.
> 
> With default netperf parameters (just "-H otherguy") I get numbers
> between ~490 MB/sec and ~550 MB/sec for 2.6.12-rc4 and 2.6.12-rc5.
> The numbers are quite consistent between reboots, but if I reboot the
> system (even keeping the kernel identical), I see large performance
> changes.  Presumably something is happening like the cache coloring of
> some hot data structures changing semi-randomly depending on the
> timing of various initialations.

Which rev of netperf are you using, and areyou using the "confidence intervals" 
options (-i, -I)?  for a long time, the linux-unique behaviour of returning the 
overhead bytes for SO_[SND|RCV]BUF and them being 2X what one gives in 
setsockopt() gave netperf some trouble - the socket buffer would double in size 
each iteration on a confidence interval run.  Later netperf versions (late 2.3, 
and 2.4.X) have a kludge for this.

Slightly related to that, IIRC, the linux receiver code adjusts the advertised 
window as the connection goes along - how far the receive code opens the window 
may change from run to run - might that have an effect?  If there is a way to 
get the linux receiver to simply advertise the full window from the beginning 
that might help minimize the number of variables.

Are there large changes in service demand along with the large performance changes?

FWIW, on later netperfs the -T option should allow you to specify the CPU on 
which netperf and/or netserver run, although I've had some trouble reliably 
detecting the right sched_setaffinity syntax among the releases.

rick jones