[openib-general] Re: user-mode verbs on Itanium

[Michael S. Tsirkin]

Quoting r. Grant Grundler <iod00d at hp.com>:
> Subject: Re: user-mode verbs on Itanium
> 
> On Tue, May 10, 2005 at 06:40:30PM +0300, Michael S. Tsirkin wrote:
> > > Yes, but I think I want the first measurement *before* post_send
> > > since it seems to measure something different that's part of the
> > > code path too.
> > 
> > I'd prefer all deltas to measure the same thing.
> 
> Ok. Can we report the startup times as part of the summary?

You mean just sample the clock before the first post?
That would be easy, but lets look at what does it measure:

On the client, you just get the time it takes to perform the
first post_send operation. If this is interesting to you, I agree,
but lets take the timestamps around each
post send operation to make it statistically relevant.
Let me know - and need to think of a way to do this without
affecting latency numbers.


On the server, you get that, plus
the latency, plus time it takes the OS to wake the client from a blocking
socket operation. It's the last component that isnt IB-related and that
makes it a non-interesting.

By the way, what do you use for statistics?

> 
> > > I suspect it's measuring the first "TLB" miss of HCA.
> > 
> > I dont think so.
> 
> ok. Any other theories on why it takes ~12x longer (4.7 vs 63usec)?
> 
> > You still see this cache warm-up with the new code.
> 
> I'll try running it again with code in SVN and see what I get.  BTW,
> which cache are you referring to here?
> 
> thanks,
> grant
> 

Hard to tell for sure. That needs to be investigated.

-- 
MST - Michael S. Tsirkin