[ofw] RE: [PATCHv2] WinVerbs: Make QP modification asynchronous

[Sean Hefty]

>So I modified the test to create the QPs outside of the timed loop, and
>likewise destroy them after the timed loop.  So the test now times only the QP
>transitions to INIT, RTR, RTS, and ERROR.

perfect - I was going to see if you could do this :)

>User/Kernel: 1 process/2 processes
>Sync/Async: 3012/4400
>Async/Async: 7168/7050
>Sync/Sync: 3363/4902
>Async/Sync: 3289/4764

Kernel sync/async indicates a work item or not.  What exactly is the user
sync/async?  And how does it differ from this run:

>Sync: 1806
>Async: 1672

In either case, it looks like the work item decreases performance about 10% for
an application wanting synchronous operation.

>I suppose we could tell the kernel driver whether the call is intended to be
>synchronous or not so that the sync behavior would skip the work item, but
>still get the extra performance for the async case.

The fully async benefit seems fairly convincing that it'd be nice to enable it
some way, but it'd also be nice to keep the code simple.  I'm not sure that
exposing a level of optimization of whether to use a work item is desirable.

(My *current* thinking is that ModifyQp should be left alone, but add work items
to Connect and Accept.)

Does the OS complete IOCTLs using a separate pool of threads?

I wonder how hard it would be add an async_modify the HW drivers.  We have per
QP event callback handlers that could be invoked to indicate modify completed.

- Sean