[Openib-windows] Windows DMA model

[Jan Bottorff]

Why oh why does my ctrl-v sometimes send my half written email
message... anyway...

>The SMI/GSI currently just call MmGetPhysicalAddress for MAD buffers,
and >this is due to change, either by allocating MADs from a common
buffer or by >doing DMA mappings when the MADs get posted to the QP
(likely the latter).  

So what your saying is currently, the Windows IB driver does NOT assure
correct bus to processor physical address translation, and does NOT
assure cache coherency on systems that need software help for
maintaining cache coherency.

I asked Microsoft about just calling MmGetPhysicalAddress for DMA and
they responded:

=======================
To summarize, your drivers will break on chipsets that need extra cache
coherence help and on virtualized systems where there is an I/O MMU. 
Neither of these is particularly common today, but they'll be much more
common in the near future.  The drivers will also break on non-x86
machines where DMA address don't equal CPU-relative physical address,
but those machines have become very uncommon in the last five years.

-- 
Jake Oshins
Windows Kernel Group
=======================

>I don't know if you saw my RFC emails about that API or not. 

I didn't see that, as I'm a very new member of the list. It sounds like
your saying the current low level interface API to things will be
changing in the future?

>Also, driver verifier will completely break DMA for user-mode as it
forces >double buffering to check that DMA mappings are used properly.

If you turn on DMA verification in driver verifier I believe it will
double buffer ALL correctly done DMA, to help find memory boundary
violations. This is also a check of the hardware.

Passing tests with driver verifier active will be required to obtain any
kind of WHQL driver certification. Commercial users will absolutely need
these certifications. 

>There is some work that needs to happen to get MAD traffic to do proper
DMA >mappings, but upper level protocols already do the right thing.

I can understand how higher level packet traffic from a kernel mode NDIS
based driver or buffers from a STORPORT based storage driver can have
the correct mapping already. It also sounds like other kernel drivers
that use the IBAL interface currently aren't assured DMA is correct.

>For the time being, since we're running on platforms where the CPU and
bus >addresses are consistent, it hasn't been an issue.

Microsoft seems to say it's more than just an address mapping issue;
it's also a cache coherency issue. I'm not surprised that it's desirable
to get software to help with cache coherency as the number of processor
cores grows, especially on AMD processor systems with essentially a NUMA
architecture. 

- Jan