[openib-general] [PATCH] rewrite perftest/README

[Grant Grundler]

On Thu, May 19, 2005 at 06:32:32AM +0100, Paul Baxter wrote:
> Grant, Michael
> 
> Great work!
> 
> I wanted to point out a recent thread on comp.arch discussing the merits of 
> using the geometric mean instead of the arithmetic mean as a basis for 
> sampling the timing of a population.

Paul,
thanks for pointing out the thread - I'm learning alot! :^)

> My statistics is a bit rusty, but 
> calculating and presenting GM might be useful a well?

After reading the original posting, I think the answer is NO.

Particularly, two comments in John Mashey's original posting:
| The GM is the correct mean for combining benchmark results intended
| to be a sample from a larger population of programs and intended to
| predict the performance distribution of other benchmarks, which after
| all, is what people want for generalized performance comparisons.
| This is true if the population follows a *lognormal* [described later]
| distribution ... and it turns out, many do so. 


rdma_lat is one benchmark - not a "combination of benchmark results".
If perftest ends up with more than two benchmarks, we can try to
look weighted geometric mean to get one number from running the whole
mess. But TBH, I have no interest or need to do that. Sounds like
statistical wanking off to me. I'm much more interested in comparing
differences of individual rdma_lat runs and sort out why the results
are different.


| If you know your workload, your benchamrks *are* the population,
| and you use algebra. 

We know pretty well what we are measuring with rmda_lat.
Well, Michael does. I only have a dangerously vague understanding. :^)

I know we have two sets of data: "warmup" and "runtime".
The first two or three measurements are related to warmup.
John Mashey's posting suggested keeping separate "groups of data" apart.
Ie. it was probably correct to not include "warmup" values in the
std deviation and arithmetic mean (avg).

Conclusion: The median value still looks like the right thing to report.

> http://groups.google.co.uk/group/comp.arch/browse_thread/thread/3f5a9ed1d79ed726/416e58b5e48c1715?q=geometric&rnum=6&hl=en#416e58b5e48c1715
> 
> [comp.arch, 'SPEC use of Geometric Mean', May 13th, 2005, John Mashey ] 

thanks again,
grant