[Openib-windows] A bit more info on DMA issues

[Jan Bottorff]

I saw this message on the NTDEV list and thought it was relevant to the
recent discussion here about getting DMA correct...

At Friday, December 02, 2005 6:25 PM, Jake Oshins from Microsoft said...


While the OS-level protocol is the same for devices which use ports and
devices which use memory-mapped registers, the bus-level protocol isn't.
Many PCI (and PCI-X, PCI-Express, etc.) chipsets have a less strict
memory ordering model for MMIO (memory mapped I/O) access.

If your interrupt is related to a DMA transaction, the interrupt may be
delivered to the processor before the data is coherent in main memory,
which means that the processor can read or write stale data in its ISR.
You need to be sure that your chipset's bridge buffers are flushed
before you count on the data in memory from the DMA operation. With
port-mapped devices, a write to the device to clear the interrupt
status/enable registers (to stop the level-triggered interrupt) will
usually cause the bridge buffers to flush. (I'd like to say "always" but
there are a few chipsets that don't quite get it right.) With
MMIO-mapped devices, you generally have to read back what you've written
to that register to cause the bridge buffers to flush, ensuring that the
DMA operation is now coherent in main memory. In practice, this often
doesn't matter as there is usually some other register read that will
occur in the ISR, or the associated DPC.

This problem is becoming more historical now, as many interrupts are
coming to the local APIC as busmaster transactions (see MSI and MSI-X in
the PCI specs). This automatically flushes the DMA transaction, since
the interrupt follows the DMA and it takes the same form, a memory
write.

Jake Oshins
Windows Kernel Group