[openib-general] Re: [PATCH] rdma_lat-09 and results

[Grant Grundler]

On Thu, Jun 02, 2005 at 08:29:34AM +0300, Michael S. Tsirkin wrote:
...
> I changed the timestamping strategy. I used to:
...
> This meant that tstamp instruction was out of the data path,
> while we did polling.
> On the negative side, although the average (and likely median) delta
> between tstamps was a reliable measurement of round trip time
> (since there was one tstamp each roundtrip),
> min/max values were not measuring anything reliably: if I start polling
> late, two tstamps can be closer than what the wire allows for.

ok.

...
> And now, on the plus side, the mix/max deltas are actually pessimistic about
> roundtrip times, on the minus side, we are calling tstamp on datapath,
> slowing it down. ~15 cycles is a bit high: of course tstamp
> needs to prevent instructions from being reordered across it, and so
> it should take on the order of the pipeline depth to perform, but then
> maybe its a microcode thing.

Yes, that makes sense now.
get_cycles() on ia64 is expected to cost 36 cycles and
we are reporting roughly 1/2 of that.

> I'm not against going back to the previous measurement, but we'd have to
> give up the min/max reporting since its an artefact.
> What do you say?

Overall, 15 cycles isn't a big deal when the total is around 7000.

I was looking for differences thinking it would be something
in the SW stack (libibverbs or libmthca). But changes in the
test are obviously going to show up too and that's ok if
they can be explained.

min/max seem like useful numbers if someone wants to get an idea
of best/worst case performance...but maybe something like
standard deviation is a more useful metric to extract.

We should seperate "start up" times from "streaming" times.
I will re-run ibv_pingpong and collect similar measure for
non-stream Request-Response traffic.

Two more ideas:
1) call get_cycles() twice in a row and substract the difference
   from each measurement *after* all measurements are taken.

2) report 1/2 the cost of get_cycles() and leave the measurements alone.

thanks,
grant