[ofa-general] Re: Requesting CQ notifications

[Michael S. Tsirkin]

> Quoting Roland Dreier <rdreier at cisco.com>:
> Subject: Re: Requesting CQ notifications
> 
>  > Ok. The reason I asked was that I am noticing the latter behavior with
>  > the mthca adaptor. If in the above code I poll for two CQEs instead of
>  > one I get them both back and can handle them. However, if I poll for
>  > just one and go back into the select, it doesn't return the
>  > comp. channel file descriptor as readable so I never handle the second
>  > CQE. I expected that this could be because I had called
>  > ibv_req_notify_cq() after both CQEs had already arrived.
> 
> Something doesn't add up.  Mellanox adapters definitely will generate
> an event after ibv_req_notify_cq() even if the CQE is already in the
> CQ.  There's no window where you can miss an event.
> 
>  > In this case it brings up an interesting question. If the adaptor will
>  > only generate an event for a "new" CQE added, how do you tell that
>  > you've successfully polled all the CQEs that triggered the first
>  > event? You could continuously call ibv_poll_cq() until it returned
>  > zero, but this has the side effect of potentially starving other CQs
>  > in the case of a high CQE rate.n
> 
> The simplest thing to do is to call ibv_req_notify_cq() before you
> poll the CQ.  This leaves no window for missing events.  Or as you
> say, you could poll the CQ until it was empty.

There's no "Or", actually. Even if you call ibv_req_notify_cq
before polling, you still must drain the CQ of completions generated
before the event
(or queue for polling later using some alternative mechanism).

> If you're worried
> about starvation, then limit the # of polls you do, and if the CQ is
> not empty, put it on a list of CQs to come back to again.
> 
> One other option would be to implement a "maybe missed event" hint
> returned from ibv_req_notify_cq(), the way I did at the kernel level
> as part of the IPoIB NAPI patches.  But that would be a longer term
> thing that would require changing the user-kernel ABI and the
> libibverbs API (and hence would have to be in libibverbs 1.2).

Note that "maybe missed event"
only checks for completions that have arrived after event was generated
but before you did request for notification
(hardware does not need to "know" which completions did you poll).

And of course if missed event is reported, you'll still need to
requeue the CQ for polling. So not much is changed.

-- 
MST