[openib-general] Re: OpenIB documentation

[Roland Dreier]

    Christopher> For example, is there any way to have IBverbs invoke
    Christopher> a callback function when an item arrives on the
    Christopher> completion queue? The IBA spec mentions a "Completion
    Christopher> Event Handler"; is this what I'm looking for?  If so,
    Christopher> how do I use it in IBverbs?

No, there is no way to get a callback, because there's no sane way to
implement this without imposing overhead or a programming model on the
consumer.

The model used by libibverbs is somewhat different and I think more
flexible than what the IB verbs spec offers.  There is a new type of
object: completion channels, which are essentially file descriptors
(you can block on them, poll() them, use SIGIO, etc, etc).  When
a completion event is triggered, then an event is added to the channel
attached to the CQ that generated the event.  You should call
ibv_get_cq_event() on the channel to retrieve the CQ, and then poll
that CQ to collect CQ entries.

If you want, you could spawn a thread that blocks in
ibv_get_cq_event() (or poll(), or...) and then calls back an event
handler function.

    Christopher> One function I'm kind of confused with is
    Christopher> ibv_req_notify_cq(). Is the second parameter simply a
    Christopher> true / false value? This isn't very obvious since the
    Christopher> rc_pingpong.c example file passes a 0 value when it
    Christopher> requests CQ notification.

The "solicited_only" parameter is as described in the verbs spec -- if
it is non-zero, then only solicited completion entries will generate
an event.

    Christopher> Another confusing one is ibv_get_cq_event(). Am I
    Christopher> correct in assuming that this is a blocking function? 
    Christopher> That's the impression I get from the example codes
    Christopher> and from the fact that struct ibv_comp_channel
    Christopher> appears to merely consist of a file descriptor. My
    Christopher> impression is that it "wakes up" when an element
    Christopher> appears on the CQ, but that it requires the use of
    Christopher> ibv_poll_cq() to actually obtain that element. Is
    Christopher> this correct?

Pretty close.  As I described above, the ibv_get_cq_event() returns
completion events, but not every completion entry will generate an
event.

 - R.