[Openib-windows] Windows DMA model

[Fab Tillier]

Hi Jan,

> I noticed there are NO calls to the Windows OS function IoGetDmaAdapter (or
> the GetDmaAdapter function of a bus interface) in the whole Windows IB source
> tree.
> 
> Can somebody explain how DMA could be correct without this?

For kernel drivers like SRP and IPoIB, the port driver (StorPort and NDIS,
respectively) perform all the DMA mappings before calling the miniport.  SRP and
IPoIB then get the scatter gather lists through the port specific mechanisms -
StorPortGetScatterGatherList for SRP, and NDIS_PER_PACKET_INFO_FROM_PACKET for
ScatterGatherListPacketInfo in IPoIB.

The SMI/GSI currently just call MmGetPhysicalAddress for MAD buffers, and this
is due to change, either by allocating MADs from a common buffer or by doing DMA
mappings when the MADs get posted to the QP (likely the latter).  As an
optimization on the send side, all outgoing MADs could be sent inline avoiding
the need for DMA, however this still requires the HCA driver to properly map the
WQE ring, which gets to memory registration mappings.

Memory registrations don't do any DMA mappings at the moment, and that is a
deficiency that I hope to correct when we move to the new verbs model (I don't
know if you saw my RFC emails about that API or not).  This gets a bit weird
though as the Windows DMA APIs take a flag that indicates which way the DMA is
to occur, but don't support DMA in both directions through a single mapping.
Also, driver verifier will completely break DMA for user-mode as it forces
double buffering to check that DMA mappings are used properly.

There is some work that needs to happen to get MAD traffic to do proper DMA
mappings, but upper level protocols already do the right thing.

For the time being, since we're running on platforms where the CPU and bus
addresses are consistent, it hasn't been an issue.  I do want to see this
change, though.

- Fab