[ofa-general] Managing work completions (libibverbs)

[Roland Dreier]

 > The man page states that ibv_ack_cq_events is mandatory, however, the
 > examples in perftest don't ack when in event mode.  Is this a bug in
 > the perftest programs or a bug in the man page?

I guess it would be a bug in the perftest programs, but the only need to
call ibv_ack_cq_events() is when destroying a CQ -- ibv_destroy_cq()
will wait until all CQ events are ACKed before returning.

 > Is it possible use epoll to block on struct ibv_comp_channel::fd then
 > use ibv_poll_cq to grab completions when epoll wakes up the process?
 > Then (I hope) it would be unnecessary to call
 > ibv_get/ack_cq_event(s).  Or is it necessary to call these functions
 > in place of ibv_poll_cq when a completion channel is used?

You can use epoll to get comp channel events, but you'll need to collect
the event with ibv_get_cq_event() to rearm things.  epoll tells you when
the fd becomes readable, but you'll need to actually read all the
events queued on the fd before waiting again.  The overhead of
ibv_get_cq_event() should not be too high compared to the overhead of
sleeping and getting woken up again by an interrupt, and you can always
amortize ibv_ack_cq_events() by just keeping a counter of the number of
events you read and only calling ibv_ack_cq_events() occasionally.

 - R.