[ofw] Latest WinVerbs presentation & .h files.

[Fab Tillier]

>-----Original Message-----
>From: Sean Hefty [mailto:sean.hefty at intel.com]
>Sent: Thursday, February 14, 2008 12:39 PM
>
>>>typedef struct _WV_QP_ATTRIBUTES
>>>{

<snip>

>>>        WV_ADDRESS_VECTOR       PrimaryAddressVector;
>>>        WV_ADDRESS_VECTOR       AlternateAddressVector;
>>>        BOOL                    SignalSends;
>>
>>Can this be changed once the QP is created?  If not, is there value in
>>returning it?
>
>If path migration is in use, these can change.

I mean just the 'SignalSends'.  Looking at the spec it's specified at creation time only and cannot be changed.

>>>
>>>    STDMETHOD(RegisterMemory)(
>>>        THIS_
>>>        __in_bcount(BufferLength) const VOID* pBuffer,
>>>        __in SIZE_T BufferLength
>>>        __in DWORD AccessFlags,
>>
>>Access flags on registrations are dumb.  You're trying to protect the app from
>>misusing a buffer within its own address space.
>
>You're also protecting against the remote peer from invalid access.  Although I
>don't see where I defined what values AccessFlags would have.

Ah, I'd forgotten about exposing the MR for remote access.  I was still thinking in terms of what Network Direct does - use MWs exclusively for remote access, and MRs exclusively for local access, which makes the programming model a little simpler, though it forces MW use so you can't exchange RKeys at connection time.

>>>        __in OVERLAPPED* pOverlapped,
>>>        __deref_out WV_MR_HANDLE* phMr,
>>>        __out UINT32* pLKey,
>>
>>You don't need the lkey - the MR handle is enough.
>
>The SGE takes the LKey, not the MR_HANDLE.

Why not make it take the MR handle?  Is the LKey used anywhere else that the MR handle couldn't be used?

>>
>>>        __out UINT32* pRKey
>>
>>This is going to be a bit hard to handle I think as an overlapped operation if
>>you have multiple output parameters.  The overlapped returns directly to the
>>user, so the get multiple output buffers you need to do a bunch of extra work
>>in the kernel to be able to access the buffer (like pinning it so that you can
>>write to it from DISPATCH_LEVEL).  You can't rely on the I/O manager doing all
>>this for you.
>
>A memory window bind operation can return the RKey immediately before the send
>is processed by the HW.  I don't see why this would be any different.  The
>LKey/RKey just aren't usable until the Register operation completes.

Ok, so your register IOCTL goes down to the kernel.  There, the memory is pinned down and the DMA mapping is obtained (all in the context of the app's thread in this case).  The kernel driver knows the LKey and RKey at this point, but the UVP and application do not.  The HCA driver now starts to program the DMA mapping to the HCA, and returns pending while the HW processes this.  When are the LKey and RKey written?  When the register call unwinds with STATUS_PENDING, but before it returns out of the IOCTL handler in the proxy (so that you do it while still in the app's thread context)?  This can be made to work, but it's somewhat complicated.

-Fab