[ewg] retry_cnt, rnr_retry and ibv_get_cq_event/ibv_req_notify_cq

[Albert Strasheim]

Hello all

On Mon, Apr 19, 2010 at 4:59 PM, Albert Strasheim <fullung at gmail.com> wrote:
> What still seems a bit tricky to me is exactly how many events to poll
> for while avoiding blocking in ibv_poll_cq.
> Will num_entries=size of CQ be sufficient? It seems to me that it
> could be, even in the case of multiple QPs, since if your CQ overflows
> you are in trouble anyway, but consider the following sequence:
> In initialization of CQ:
> completion channel fd -> nonblocking
> ibv_req_notify_cq
> then when WCs start happening:
> select/poll/whatever success
> ibv_get_cq_event
> // WCs arrive here while completion channel is disarmed
> ibv_req_notify_cq
> // WCs can arrive here, all of which will be also be picked up by the
> next ibv_poll_cq
> n = ibv_poll_cq(num_entries=size of CQ)
> select/poll/whatever success
> ibv_get_cq_event
> ibv_req_notify_cq
> n = ibv_poll_cq(num_entries=size of CQ)
> Now the last ibv_poll_cq blocks because we drained the CQ on the previous call.

Changing the order of the ibv_req_notify_cq and ibv_poll_cq won't help
either, since you might get your last event between the poll returning
and re-arming the completion channel, in which case you won't poll
again.

Regards

Albert