[ofw] ibv_get_cq_event()

[Hefty, Sean]

> I observe ibv_get_cq_event() gets CQ event signal from COMP Channel
> thread. This results in 2 levels of signal hop for each CQ event. I find
> this latency is visible for small size ( < 4K) data transfers.
>  Has anyone seen this issue ?  Instead, is it possible for
> ibv_get_cq_event() get CQ event directly from hw driver to avoid this
> latency ?

Windows threading is painful...

I tried to implement the event handling without introducing threads, but found it too difficult to avoid all race conditions.  The reason for the threading is because Linux apps like to call select/poll on multiple fd's to retrieve asynchronous events and completions.  The comp_channel abstraction is a mechanism that was added to help make it easier to port the apps between Linux and Windows.  Calling poll on an fd is a OS specific operation, so this isn't something that can just be handled for the app.

So... is it possible for ibv_get_cq_event() to get the event directly from the driver?  Yes.  However, it is difficult to implement this while still supporting apps that wish to poll across multiple fd's.  I'm open to any ideas on how to improve the code such that both work.

- Sean