[openib-general] Asynchronous Event notification and registration

[Roland Dreier]

    Kevin> People need to excuse all my annoying startup questions.  I
    Kevin> just notices that the user space ib verbs lib doesn't have
    Kevin> any functions to register completion or event handlers.  Is
    Kevin> the intent that the API will only support polling or are
    Kevin> the asynchronous event handling stuff in somebodies branch
    Kevin> someplace?

Everything you need should already be present.  Completion and
asynchronous events are delivered by reading special file descriptors
that are created as part of the device context returned from
ibv_open_device().  You don't need to do the read() yourself --
ibv_get_cq_event() and ibv_get_async_event() return the next available
completion or asynchronous event, respectively.

It is true that there is not a mechanism for receiving a callback when
when an event is ready.  However, this is by design because there is
not really a single, efficient mechanism for kernel-to-user function
calls, which works for all application architectures.  You are free to
detect that context->cq_fd[0] and/or context->async_fd have become
readable in whatever way fits your application best.

(Perhaps an API should be wrapped around these file descriptor members
of the context structure so that applications don't have to look
around directly in the struct)

Some possibilities for detecting events are:

 - Use select()/poll()/epoll_wait() to monitor the FDs, along with any
   other FDs of interest to your event loop.
 - Have threads that block in ibv_get_cq_event() and/or
   ibv_get_async_event().
 - Use SIGIO/POSIX realtime signals to detect events.

<http://www.kegel.com/c10k.html> may offer some inspiration.

 - R.