[Openib-windows] Windows DMA model

[Fab Tillier]

> From: Leonid Keller [mailto:leonid at mellanox.co.il]
> Sent: Wednesday, January 18, 2006 5:08 AM
> 
> Some thoughts about DMA API.
> 
> I find it very uncomfortable: it imposes some way of work with
> unrequested implicit actions instead of providing a minimal number of
> tools to do the things as we'd like it.
> I believe, Infiniband is a *typical* solution in the sense, that any
> interconnect, providing large bandwidth and low latency will need to
> support scatter/gather and 64-bit addressing.

DMA support isn't broken at all for clients that perform mappings on a
per-request basis, which is adequate for most existing hardware.  The existing
DMA APIs support scatter/gather and 64-bit addressing just fine.  The problem
with IB is that of memory registration, which is unique to RDMA hardware.  The
DMA APIs where not designed to allow arbitrary buffers to be converted to common
buffers (which is really what we're looking for here).

> Therefore is worth to require additional/another API, that will
> eliminate these deficiences both with enabling (as far as possible)
> kernel bypass.

I don't think we can require anything like this, but we can probably make a
request for a future version of the OS.

> I believe it would enough only 5 functions (let's call them NewDmaApi):
> 
>    IoGetDmaAdapterEx(*Adapter, *Flags, *NmapRegisters,...)
>       Returns Adapter object and capability Flags: IsDoubleBuffering,
> IsNewDmaApi, IsDmaCacheble.
> 
>    IoMapBufToDmaSpace(Adapter, Mode, PcuNonCachable, DmaNonCachable,
> addr, size, ...);
>       Locks the memory, allocates mapping registers and makes them
> uncachable, if requested.
>       Returns an array of logical addresses and an opaque DmaBuffer
> object.
> 	Pay attention:
>          No need for memory allocation: there are malloc and
> ExAllocatePool(Paged).
>          No need for mapping to kernel space: there is
> MmProbeAndLockPages.
>          An error NOT_ENOUGH_REGISTERS is to be handled as a failure on
> memory allocation.
> 
>    IoUnmapBufFromDmaSpace(DmaBuffer)
> 	Undoes the previous.
> 
>    KeFlushPcuBuffers(DmaBuffer)
>       Flushes CPU cache to the memory buffer.
>       Used only once, if the buffer was mapped as PcuNonCachable.
> 
>    KeFlushDmaBuffers(DmaBuffer)
>       Flushes DMA cache to the memory buffer.
>       Not used at all, if the buffer was successfully mapped as
> DmaNonCachable.
> 
> With such API and cachable mapping registers we could work both in
> kernel and user without changing our model. It will also work in a guest
> OS.

I don't think we need that many APIs - two should suffice:

SCATTER_GATHER_LIST IoMapCommonBuffer( DMA_ADAPTER *pAdapter, VOID *pBuf, ULONG
Length, BOOLEAN CacheEnabled, HANDLE *phCommonBuffer );
VOID IoUnmapCommonBuffer( HANDLE hCommonBuffer );

The pBuf parameter to IoMakeCommonBuffer could be changed to be an MDL list,
which might make more sense.  The MDL would be for a probed and locked buffer,
or for non-paged pool.

> Am I missing something ?

The only issue is that we don't have access to the OS source, and can't just
modify it.  If we come up with a good API, then we might have some success in
having it added in a future version of the OS.  It's worth a shot, but we'll
probably need to implement with the current DMA APIs until a better API is
available.

It's worth discussing what we would want this API to look like, and we can bring
it up once we have a good idea of how we want it to work.

- Fab