[openib-general] Re: user-mode verbs on Itanium

[Grant Grundler]

On Wed, May 11, 2005 at 10:21:38AM +0300, Michael S. Tsirkin wrote:
> You mean just sample the clock before the first post?

Yes.

> That would be easy, but lets look at what does it measure:
> 
> On the client, you just get the time it takes to perform the
> first post_send operation. If this is interesting to you, I agree,
> but lets take the timestamps around each post send operation to
> make it statistically relevant.
> Let me know - and need to think of a way to do this without
> affecting latency numbers.

Yes, I'd like to try that.
But a few more of my pending patches need to either be rejected
or (preferably) accepted.

Given the roundtrip time is in the 10000-20000 cycle range,
the additional cost of reading the ITC should be light weight
enough on most architectures. ISTR it's ~36 cycles on IA64.

We could do two get_cycle calls back-to-back at the start
and substract that from any measurements.

> On the server, you get that, plus
> the latency, plus time it takes the OS to wake the client from a blocking
> socket operation. It's the last component that isnt IB-related and that
> makes it a non-interesting.

I think it's interesting but doesn't have to be part of this test.

If we meaure it, people will be aware of it when they use rdma_lat.c
as an example for their own code.

> By the way, what do you use for statistics?

I normally use netperf which includes statistical-aware sampling.
I haven't used anything for this perf test. But the results look
fairly reliable.

> > ok. Any other theories on why it takes ~12x longer (4.7 vs 63usec)?

The wakeup is *probably* a big chunk of the 63usec...but I'm still just
guessing.

> Hard to tell for sure. That needs to be investigated.

Ok. I don't have the HW or time to chase that. I'm just hoping
that by getting the test to measure the right things, it will
provide the people who can, some ideas on what needs to be
investigated.

thanks,
grant