[ofw] RE: NetworkDirect over WinVerbs

[Sean Hefty]

>You know what, I'll put together a patch that will move the QP modification to
>a work item context and prove that it doesn't complicate locking, destruction
>handling, or device removal.

Yes - that patch maintains the same locking as the current code.

>In the CM callback you have a WDFREQUEST, so you can do these exact steps.  The
>only issue is related to the CM callback being invoked at DISPATCH_LEVEL, and
>the QP Modify being a synchronous call.  That problem would go away if the HCA
>driver model didn't suck.  In the mean time you can allocate a work item and
>queue it for processing at passive level.  You still have the IRP outstanding,
>so the app context won't go away.

As long as being at dispatch doesn't require changing the synchronization in
other areas to using spinlocks, this sounds fine.

>STDMETHOD(DisconnectAndFlush)(
>        THIS_
>        __in IVWConnectQueuePair* pQp,
>        __in_opt OVERLAPPED* pOverlapped
>        ) PURE;
>
>There's no more dependency between the two objects in the kernel code than
>exists today with Connect and Accept.

I don't have an objection to this if it can be implemented cleanly.  The
difference between this call and Connect or Accept is that the QP object is
accessed from a different thread.

>You can control whether I/O requests complete to the IOCP or not by setting the
>lowest bit of the event handle in the OVERLAPPED structure that you specify in
>your overlapped operation.  From the GetQueuedCompletionStatus docs:

That is not the obvious place to document this...

>"Even if you have passed the function a file handle associated with a
>completion port and a valid OVERLAPPED structure, an application can prevent
>completion port notification. This is done by specifying a valid event handle
>for the hEvent member of the OVERLAPPED structure, and setting its low-order
>bit. A valid event handle whose low-order bit is set keeps I/O completion from
>being queued to the completion port."

So, the winverbs ND provider could implement this with:

Disconnect()
NotifyDisconnect(internal overlap)
wait on internal overlap using internal thread
Modify(users overlap)

Which doesn't seem that much different than scheduling Modify to a work item.
Maybe slightly less efficient, but I think it's doable.

- Sean