[Openib-windows] CQ event notification in user-mode

[Fab Tillier]

Folks,

Currently, there are two possible event completion models available in Windows -
OS event driven, and callback driven.

I'm working on a plan to eliminate all callbacks from the access layer so that
an application has full control of threading, as well as to support truly single
threaded applications.

My current thinking is to implement and API somewhat similar to the Gen2 Linux
ibv_get_cq_event.  The key difference is that the API wouldn't return the next
CQ event, but rather would target a specific CQ and would complete only when
that CQ gets an event or is destroyed.

The user would be able to select synchronous or asynchronous behavior, as well
as a mix, so a single thread could issue event requests for every CQ it uses
asynchronously, and use the various asynchronous I/O options available in
Windows to track completions of those requests.

Windows provides I/O completion ports which function similarly to CQs, but for
asynchronous I/O operations performed on various files.  Files are associated
with an I/O completion port using the CreateIoCompletionPort function, which
allows the user to pass in a context value unique to the file that is returned
for any I/O completion on that file in a call to GetQueuedCompletionStatus.

The question at hand, and where I would like community input, relates to usage
of the I/O completion key with respect to CQs.

Allowing the user to specify a key per CQ implies that each CQ has an underlying
file object, which may impact scalability if a user has many CQs.  Having a
single key for all completions on the other hand allows for a single CQ
completion file object (likely managed in the CA object), and the I/O request
would have to be used to provide per-CQ context using the OVERLAPPED structure
returned by GetQueuedCompletionStatus (provided when issuing the request).

Currently, IBAL uses an I/O completion port internally, with a pool of threads,
to service callbacks.  The key returned by GetQueuedCompletionStatus provides
the type of event, and the OVERLAPPED structure provides the specific context
within that type.  A single CQ completion file object would match this behavior.

I'm leaning towards a single CQ completion file per open CA, and thus no ability
for the user to assign individual keys per CQ when binding CQs to I/O completion
ports.

The other ramification of a single CQ completion file is that a user cannot mix
I/O completion port and asynchronous procedure call completion models within a
single instance of a CA.  I don't think this is a big deal though, as
applications are likely to use a consistent completion model.

So, single CQ file per CA, or per CQ file?

- Fab