[ofw] RE: NetworkDirect over WinVerbs

[Sean Hefty]

>Yes, ND::Disconnect does the equivalent of
>WV::NotifyDisconnectAndWhenThatHappensModifyQpToError followed by
>WV::Disconnect.

nit: WV:Disconnect() is part of the initial operation, since it sends the
disconnection message.  Although, I'm still not sure that this is really what
ND:Disconnect() is supposed to do, which is part of the problem.

>It merges the NotifyDisconnect and the Modify calls into a single overlapped...

Consider the passive disconnect (winverbs):

1 NotifyDisconnect() is called sometime after connecting.
2 NotifyDisconnect() completes
3 User calls Disconnect() and Modify() in any order.

Now, consider the active disconnect:

1 User calls NotifyDisconnect() and Disconnect() in any order.
2 NotifyDisconnect() completes.
3 User calls Modify().
4 Modify completes.

In both cases, two overlapped operations complete, which seems the same as
calling the ND NotifyDisconnect and Disconnect calls.  One could argue that
Modify() may not be required.  A user may just want to release the QP.  (Receive
buffers could be tracked separately from the QP.  E.g. belong to a single buffer
allocation/registration.)  Additionally, winverbs always calls Modify() in the
context of the user's thread.

Besides avoiding assumptions about the implementation, this is huge when trying
to ensure proper synchronize.  Otherwise, you can end up trying to synchronize
CM upcalls with QP downcalls, like destruction, and device removal handling.
I'm *very* reluctant to changes in this part of the kernel code.  I do not want
to get into the QPs and CEPs referencing each other or adding complex
synchronization (that likely won't work).

ND seems to want ND:NotifyDisconnect to complete only when receiving a DREQ, and
ND:Disconnect to complete after modify QP completes after receiving a DREP or
DREQ timeout.

- Sean