[ofa-general] Requesting CQ notifications

[Roland Dreier]

 > Is there a differentiation between multiple CQE's being in the CQ
 > vs. CQE's being arriving into the CQ when using completion
 > notifications?
 > 
 > For example, assume I have the following order of events:
 > 
 > 
 > 	2 CQEs arrive
 > 
 > 	select() returns readable for comp. channel
 > 
 > 	ibv_get_cq_event() returns event
 > 
 > 	ibv_req_notify_cq(cq, 0)
 > 
 > 	ibv_poll_cq(cq, 1, &cqe) returns 1
 > 
 > 	ibv_ack_cq_events(cq, 1)
 > 
 > 
 > Will the comp. channel receive another event for the second CQE even
 > if it had arrived before ibv_req_notify_cq() was called?

This is really an ill-posed question: according to the semantics
defined by the verbs spec, the presence or absence of the second CQE
is not defined until you poll the CQ again.

In practice we can look at what real hardware does, and the answer is
"it depends."  Some adapters (eg mthca, mlx4) will generate an event
immediately if ibv_req_notify_cq() is called for a CQ that contains an
unpolled CQE, while other adapters (eg ipath, ehca) will only generate
an event when a CQE is added after the cal to ibv_req_notify_cq().

 - R.