[Openib-windows] Windows DMA model

[Jan Bottorff]

Hi,

My understanding is MmMapLockedPagesSpecifyCache was defined because
there was no way to assure the match between underlying cache properties
and what was set in the page table entries when creating a new mapping.
Someplace (a MS KB article?) this is explained in detail.

If you look in the DDK under MEMORY_CACHING_TYPE for the 3rd parameter
to MmMapLockedPagesSpecifyCache, is says "Processor translation buffers
cache virtual to physical address translations. These translation
buffers allow many virtual addresses to map a single physical address.
However, only one caching behavior is allowed for any given physical
address translation. Therefore, if a driver maps two different virtual
address ranges to the same physical address, it must ensure that it
specifies the same caching behavior for both. Otherwise, the processor
behavior is undefined with unpredictable system results."

This certainly sounds like if you have memory with a virtual address,
you can't call MmMapLockedPagesSpecifyCache to create an alternative
mapping with a different cache behavior. Pretty much all the allocation
calls return a virtual address to describe the allocation, so any call
to MmMapLockedPagesSpecifyCache needs to use a matching cache behavior.
The end result is you can't just arbitrarily turn some memory into
cached or uncached memory. The Intel processor manual also warns about
mismatched mappings being invalid. 

- Jan


> -----Original Message-----
> From: Fab Tillier [mailto:ftillier at silverstorm.com] 
> Sent: Monday, January 23, 2006 3:57 PM
> To: Jan Bottorff; Leonid Keller
> Cc: openib-windows at openib.org
> Subject: RE: [Openib-windows] Windows DMA model
> 
> > From: Jan Bottorff [mailto:jbottorff at xsigo.com]
> > Sent: Monday, January 23, 2006 3:48 PM
> > 
> > > In my understanding, a buffer, allocated by malloc, is 
> cachable, and 
> > > we want to MAKE it unchacable both for CPU and DMA. I 
> guess, that in 
> > > your understanding a Common Buffer is already a SUCH one (i.e. - 
> > > twice unchachable).
> > 
> > It may be impossible to do this. I believe at boot time memory is 
> > chopped up into regions and the processor MTRR registers are 
> > programmed with the memory properties (like caching or not) 
> for each 
> > region. Most of memory is in a region that is fully 
> > cacheable/prefetchable. Memory reserved for common memory 
> MAY be in a 
> > different region, although exactly what properties are set 
> will depend 
> > on things like the I/O bridge architecture.
> > 
> > There also are bits in the page tables that indicate caching, and 
> > these MUST match the properties of the underlying memory (allocated 
> > from some MTRR defined region). If the MTRR and page tables 
> don't have 
> > matching attributes, I believe the Intel processor manual describes 
> > processor behaivor as undefined.
> 
> Calls like MmMapLockedPagesSpecifyCache let you specify cache 
> behavior, so I would expect the OS to be able to handle 
> runtime cache behavior specification.
> 
> > It's not just a matter of dreaming up any API that we want, it's a 
> > matter of defining API's which are implementable on current, recent 
> > past, and future processors based on underlying hardware 
> constraints.
> 
> I totally agree.  From my queries into Microsoft, they are 
> aware of the deficiencies of the current DMA APIs for RDMA 
> and similar hardware, and are working on a solution.  I don't 
> know more than that, however.
> 
> - Fab 
> 
> 
>