[openib-general] Re: ibv_get_async_event

[Caitlin Bestler]

I'm not sure I follow the rationale as to why acking is needed.
You're requiring a solution to a problem that most apps and 
most devices do not have.

But anyway, if you *do* have an acked cq reap, you should 
take advantage of it by having the cq_poll return a *pointer*
to the work completion rather than copying it to a supplied
buffer. This avoids an unecessary copy whenever the work
completion will be processed immediately or transformed
into another format (which, between them, I believe account
for most applications).

The only problem with a cq_poll routine that returns a pointer
is that it *requires* an ack call -- but if you're going to
do it anyway you might as well get the benefit of a two stage
call.
 

> -----Original Message-----
> From: openib-general-bounces at openib.org 
> [mailto:openib-general-bounces at openib.org] On Behalf Of Roland Dreier
> Sent: Tuesday, September 06, 2005 2:23 PM
> To: Sean Hefty
> Cc: openib
> Subject: [openib-general] Re: ibv_get_async_event
> 
> I thought about this some more and I came to the conclusion 
> that Sean is right.  We should come up with something 
> race-free, even if an app is perverse enough to use multiple 
> threads to read CQ events.
> 
> I think the only way to do that is for the app to acknowledge 
> completion events, since a completion event could be read by 
> a thread that loses the CPU before returning to the app and 
> then delayed for arbitrarily long before the app sees the 
> event.  However, it is possible to amortize the locking cost 
> of acknowledging events by allowing the app to acknowledge 
> multiple events in a single call.
> 
> The API I came up with is the following:
> 
> 	/**
> 	 * ibv_ack_cq_events - Free an async event
> 	 * @cq: CQ to acknowledge events for
> 	 * @nevents: Number of events to acknowledge.
> 	 *
> 	 * All completion events which are returned by 
> ibv_get_cq_event() must
> 	 * be acknowledged.  ibv_destroy_cq() will wait for all 
> completion
> 	 * events to be acknowledged, so there should be a one-to-one
> 	 * correspondence between acks and successful gets.  An 
> application
> 	 * may accumulate multiple completion events and 
> acknowledge them in a
> 	 * single call by passing the number of events to ack 
> in @nevents.
> 	 */
> 	extern void ibv_ack_cq_events(struct ibv_cq *cq, 
> unsigned int nevents);
> 
> (I also renamed ibv_put_async_event() to ibv_ack_async_event() for
> symmetry)
> 
> I coded this up and did some unscientific measurements using 
> ibv_rc_pingpong (using CQ events with --size=1).  Even with a call to
> ibv_async_event() every time a CQ event is read, the cost is 
> too small to measure.  In other words, the variability from 
> run to run of my test drowns out the cost of the call to 
> ibv_ack_cq_events().
> 
> Patches to follow...
> 
>  - R.
> _______________________________________________
> openib-general mailing list
> openib-general at openib.org
> http://openib.org/mailman/listinfo/openib-general
> 
> To unsubscribe, please visit 
> http://openib.org/mailman/listinfo/openib-general
> 
>