[ofa-general] Requesting CQ notifications

[Roland Dreier]

 > Ok. The reason I asked was that I am noticing the latter behavior with
 > the mthca adaptor. If in the above code I poll for two CQEs instead of
 > one I get them both back and can handle them. However, if I poll for
 > just one and go back into the select, it doesn't return the
 > comp. channel file descriptor as readable so I never handle the second
 > CQE. I expected that this could be because I had called
 > ibv_req_notify_cq() after both CQEs had already arrived.

Something doesn't add up.  Mellanox adapters definitely will generate
an event after ibv_req_notify_cq() even if the CQE is already in the
CQ.  There's no window where you can miss an event.

 > In this case it brings up an interesting question. If the adaptor will
 > only generate an event for a "new" CQE added, how do you tell that
 > you've successfully polled all the CQEs that triggered the first
 > event? You could continuously call ibv_poll_cq() until it returned
 > zero, but this has the side effect of potentially starving other CQs
 > in the case of a high CQE rate.n

The simplest thing to do is to call ibv_req_notify_cq() before you
poll the CQ.  This leaves no window for missing events.  Or as you
say, you could poll the CQ until it was empty.  If you're worried
about starvation, then limit the # of polls you do, and if the CQ is
not empty, put it on a list of CQs to come back to again.

One other option would be to implement a "maybe missed event" hint
returned from ibv_req_notify_cq(), the way I did at the kernel level
as part of the IPoIB NAPI patches.  But that would be a longer term
thing that would require changing the user-kernel ABI and the
libibverbs API (and hence would have to be in libibverbs 1.2).

 - R.