[ofa-general] Requesting CQ notifications

[Mike Heffner]

Roland Dreier wrote:
>  > Is there a differentiation between multiple CQE's being in the CQ
>  > vs. CQE's being arriving into the CQ when using completion
>  > notifications?
>  > 
>  > For example, assume I have the following order of events:
>  > 
>  > 
>  > 	2 CQEs arrive
>  > 
>  > 	select() returns readable for comp. channel
>  > 
>  > 	ibv_get_cq_event() returns event
>  > 
>  > 	ibv_req_notify_cq(cq, 0)
>  > 
>  > 	ibv_poll_cq(cq, 1, &cqe) returns 1
>  > 
>  > 	ibv_ack_cq_events(cq, 1)
>  > 
>  > 
>  > Will the comp. channel receive another event for the second CQE even
>  > if it had arrived before ibv_req_notify_cq() was called?
> 
> This is really an ill-posed question: according to the semantics
> defined by the verbs spec, the presence or absence of the second CQE
> is not defined until you poll the CQ again.
> 
> In practice we can look at what real hardware does, and the answer is
> "it depends."  Some adapters (eg mthca, mlx4) will generate an event
> immediately if ibv_req_notify_cq() is called for a CQ that contains an
> unpolled CQE, while other adapters (eg ipath, ehca) will only generate
> an event when a CQE is added after the cal to ibv_req_notify_cq().


Ok. The reason I asked was that I am noticing the latter behavior with 
the mthca adaptor. If in the above code I poll for two CQEs instead of 
one I get them both back and can handle them. However, if I poll for 
just one and go back into the select, it doesn't return the comp. 
channel file descriptor as readable so I never handle the second CQE. I 
expected that this could be because I had called ibv_req_notify_cq() 
after both CQEs had already arrived.

In this case it brings up an interesting question. If the adaptor will 
only generate an event for a "new" CQE added, how do you tell that 
you've successfully polled all the CQEs that triggered the first event? 
You could continuously call ibv_poll_cq() until it returned zero, but 
this has the side effect of potentially starving other CQs in the case 
of a high CQE rate.


Mike

-- 

   Mike Heffner <mike.heffner at evergrid.com>
   EverGrid Software
   Blacksburg, VA USA

   Voice: (540) 443-3500 #603