[openib-general] Completion callback /teardown race

[Fabian Tillier]

On 9/19/06, Rimmer, Todd <trimmer at silverstorm.com> wrote:
> > From: Eric Barton
> > Sent: Tuesday, September 19, 2006 2:14 PM
> > To: openib-general at openib.org
> > Subject: [openib-general] Completion callback /teardown race
> >
> > All the CQ callback does is wake a thread to poll the queue.  This
> > effectively
> > keeps polling completions out of the CQ until it is empty. Then it
> > calls
> > ib_req_notify_cq(cq, IB_CQ_NEXT_COMP) and ib_poll_cq() 1 more time.
> >
> > If this last call to ib_poll_cq() finds something, it repeats the
> > whole process
> > - but can I be guaranteed another CQ callback in this case or is it
> > indeterminate?
> >
> The recommended algorithm would be:
>
> poll_cq until empty
> ib_req_notify_cq
> poll_cq until empty

Note that if you are going to poll after ib_req_notify_cq, you can
simplify the above algorithm and just do:

ib_req_notify_cq
poll_cq until empty

However, such an algorithm will result in extra CQ events on Mellanox
HCAs.  On HCAs where the new CQ event is only generated for new CQEs
it works just as well as the opposite, which works only on Mellanox
HCAs:

poll_cq until empty
ib_req_notify_cq

> There are some differences in HCA behaviour with regard to
> ib_req_notify_cq.  Mellanox HCAs will provide a callback/interrupt if
> the CQ is not empty at this point (in which case the poll_cq's after the
> notify are optional).
>
> However the behaviour defined in the IBTA spec indicates that
> ib_req_notify_cq will cause a callback/interrupt only on the next CQE
> which arrives, hence to be portable the poll_cq loop after
> ib_req_notify_cq is necessary to cover any CQEs which arrived between
> the prior poll and the ib_req_notify_cq.

I remember a while ago a mention that the behavior of the Mellanox
HCAs could be controlled in the firmware, so that they would follow
the IBTA spec defined behavior.

I don't know what the impact on performance would be if such a change
were made.  Perhaps someone from Mellanox can confirm/deny the HCAs
ability to implement the IBA spec behavior, and quantify the effects.

- Fab