[ofw] How to block waiting for a CQE?

[Diego Guella]

----- Original Message ----- 
>From: "Tzachi Dar"
>Ib rearm CQ will fire if there is an event after the last polled cqe.
>That is, if there is a new cqe it will fire, even if it happens between
>the poll
>And the arm.
>As a result, the code will work, and you don't have to wait for the
>second event.

Oh. This is what I was missing.
I am using Mellanox hw now, so I shouldn't have problems.


>[All said is true for Mellanox HW. I don't know what happens for other
>hw vendors.]
In ib_al.h:
/****f* Access Layer/ib_rearm_cq
* NAME
* ib_rearm_cq
*
* DESCRIPTION
* This indicates that the completion queue should notify the client when
* the next completion is added.
*
* SYNOPSIS
*/

...

* NOTES
* This routine instructs the channel interface to invoke the completion
* handler when the next completion queue entry is added to this CQ.

(In fact, this is the behavior of ibv_req_notify_cq/ibv_get_cq_event on Linux).
So if some other vendor implemented ib_rearm_cq using the strict behavior in ib_al.h, I will run into the problem I outlined.
Same thing if Mellanox changes the behavior to conform to what is written in ib_al.h.

Please look at the modified pseudocode below:
-----
// Initialization
ib_status = ib_rearm_cq( ctx->scq, FALSE );

// Loop
while (1) {
 cl_status = cl_waitobj_wait_on( ctx->cq_waitobj, EVENT_NO_TIMEOUT, TRUE );
 ib_status = ib_rearm_cq( ctx->scq, FALSE );    // ib_rearm_cq moved before poll loop

 do {
  ib_status = ib_poll_cq(ctx->scq, &p_wc_free, &p_wc_done);
  if (ib_status == IB_SUCCESS)
   handle_stuff();
 } while (ib_status == IB_SUCCESS );

}
-----

This has the drawback that on Mellanox hw I will double the number of outer loops, but at least I have a spec-compliant behavior 
from my side.
What do you think about it? Do you see some problems in it?


Thanks,
Diego