[ofw] NetworkDirect over WinVerbs

[Fab Tillier]

Hi Folks,

I wanted to start a discussion about how resources would be mapped in order to provide NetworkDirect support over WinVerbs.

In NetworkDirect, the INDAdapter interface represents an underlying hardware device.  All further resources are created at this level.  This is different from WinVerbs in that IWVConnectEndpoint objects are created at the provider level and not the device level.  Similarly, the file object used for overlapped operations is at the IWVProvider level and not the INDAdapter level.  Thus, the INDAdapter object needs to map to an IWVProvider, a IWVDevice, as well as a IWVProtectionDomain (for Register/DeregisterMemory.)

The INDProvider object would be at a higher level, though I don't see how one gets a list of supported IP addresses from WinVerbs.

Beyond this, I believe the mappings are pretty straight forward:

INDListen -> IWVConnectEndpoint
INDConnector -> IWVConnectEndpoint
INDEndpoint -> IWVConnectQueuePair
INDCompletionQueue -> IWVCompletionQueue
INDMemoryWindow -> no mapping, MWs are emulated

The only obvious issue I see at this point are:
- that IWVDevice::RegisterMemory fills a user-provided WV_MEMORY_KEYS structure upon success.  This works fine on 64-bit systems as the ND_MR_HANDLE is 8 bytes and casting will do the trick, but will fail for 32-bit applications, where ND_MR_HANDLE is only 4 bytes.  Allocating a structure in RegisterMemory would leak that structure if RegisterMemory fails asynchronously to the user.  The LKey could be returned as the ND_MR_HANDLE, in which case there would need to be a way to get the RKey given an LKey which doesn't seem to exist at this point.
- that disconnection mappings are a bit funny.  IWVConnectEndpoint::NotifyDisconnect will return in error when the endpoint gets disconnected (STATUS_CONNECTION_DISCONNECTED), or when the DREQ times out (STATUS_TIMEOUT).  It seems a bit unnatural that a user calling Disconnect would cause a NotifyDisconnect request to timeout.  The INDConnector::Disconnect call looks like it should map first to a call to IWVConnectEndpoint::NotifyDisconnect, followed by a call to IWVConnectEndpoint::Disconnect.  One missing piece is that INDConnector::Disconnect is expected to flush all pending requests from the associated INDEndpoint when Disconnect completes (QP transition to ERROR).  The transition to error can't happen when the DREQ is sent only when it either times out or a DREP is received.

Thoughts/comments?
-Fab