[openib-general] Completion callback /teardown race

[Roland Dreier]

    Eric> If this last call to ib_poll_cq() finds something, it
    Eric> repeats the whole process - but can I be guaranteed another
    Eric> CQ callback in this case or is it indeterminate?

In general there is an unavoidable race, since you don't know whether
the new completion you find in the CQ was generated before or after
you requested notification.  So with the completion semantics as
defined in the IBA spec, you have the choice of two poisons:

 1) Don't poll after you request notification.  Then you run the risk
    of a completion being added after your last poll but before you
    requested notification.  If another completion never occurs, then
    you're stuck forever.

 2) Poll after you request notification.  Then you run the risk of
    having a completion added after your request for notification but
    before your final poll.  This means another completion event will
    be pending, but you will likely drain the CQ before you take the event.

However, Mellanox HCAs implement stronger semantics: they generate an
event if the CQ is not empty at the time notification is requested,
which closes the race between draining the CQ and requesting
notification.  This means *for Mellanox HCAs only* it is safe to do:

  completion_handler():
    poll CQ until empty
    request notification on CQ

with no additional poll after the request for notification.

I'll have more to say on this in the context of IPoIB and NAPI
shortly, since I've been thinking about this issue myself.

The ipath driver implements only the weaker semantics guaranteed by
the IBA spec -- ie an event is generated if a completion is added
after the request for notification.  And I don't know what ehca and
amso1100 implement to be honest.

(The Mellanox semantics are conforming though, since it's not
well-defined exactly when a completion is added to a CQ if no one looks...)

 - R.