[openib-general] Completion callback /teardown race

[Rimmer, Todd]

> From: Eric Barton
> Sent: Tuesday, September 19, 2006 2:14 PM
> To: openib-general at openib.org
> Subject: [openib-general] Completion callback /teardown race
> 
> 
> 
> All the CQ callback does is wake a thread to poll the queue.  This
> effectively
> keeps polling completions out of the CQ until it is empty. Then it
calls
> ib_req_notify_cq(cq, IB_CQ_NEXT_COMP) and ib_poll_cq() 1 more time.
> 
> If this last call to ib_poll_cq() finds something, it repeats the
whole
> process
> - but can I be guaranteed another CQ callback in this case or is it
> indeterminate?
> 
The recommended algorithm would be:

poll_cq until empty
ib_req_notify_cq
poll_cq until empty

Once ib_req_notify_cq is called, its possible for an additional callback
to race with the poll_cq's which follow.

There are some differences in HCA behaviour with regard to
ib_req_notify_cq.  Mellanox HCAs will provide a callback/interrupt if
the CQ is not empty at this point (in which case the poll_cq's after the
notify are optional).

However the behaviour defined in the IBTA spec indicates that
ib_req_notify_cq will cause a callback/interrupt only on the next CQE
which arrives, hence to be portable the poll_cq loop after
ib_req_notify_cq is necessary to cover any CQEs which arrived between
the prior poll and the ib_req_notify_cq.

Within a given callback invokation, there is no reason to call notify
more than once.

Todd Rimmer