[openib-general] [PATCH] mthca updates (2.6.8 dependent)

[Grant Grundler]

On Mon, Aug 30, 2004 at 09:10:35PM -0700, Roland Dreier wrote:
>     Grant> MSI is useful with one if the driver can avoid all MMIO
>     Grant> read in the normal interrupt case. MSI guarantees strong
>     Grant> ordering of DMA to host mem and delivery of the interrupt.
> 
> I agree that MSI messages are strongly ordered with respect to other
> writes coming from a device.  However, I couldn't find anything
> explicit in the PCI spec that says that the interrupt can't pass an
> earlier write to memory once the MSI hits the host bridge.

MSI transaction is a DMA write from the chipset perspective.
It is subject to the same ordering constraints as any other 
DMA write (or PCI bus transaction for that matter).

> In other
> words the interrupt message will definitely hit the chipset after any
> writes that were initiated before it, but I didn't see anywhere it's
> guaranteed that the CPU will see those writes before it sees the
> interrupt.

I think it assumes the interrupt controller (Local xAPIC) is
inside the cache coherency domain. ie once a DMA write hits
the coherency domain, ordering no longer is a (PCI) problem.
ie we know transactions targeting memory will reach the coherency
domain before MSI and thus will be visible to the CPU before
the MSI is delivered.

> The Intel E7500 Xeon chipset PCI bridge datasheet does say that an
> interrupt message causes the bridge to flush its write buffers to
> preserve precisely this ordering, but I don't know whether everyone
> followed or will follow this example (I'm sure we'll see many more MSI
> implementations on PCI Express).

Maybe the flush is required because of DMA write coalescing in the E7500
chipset?

Ie any DMA writes which can't be coalesced will cause this kind of
a flushing behavior. I don't really know since I don't how E7500
handles cache coherency.

And yes, I'm certain some chipsets get DMA write coalescing wrong.
Look at drivers/net/tg3.c and search for TG3_FLAG_MBOX_WRITE_REORDER
in tg3_get_invariants().

grant