[openib-general] [PATCH] mthca updates (2.6.8 dependent)

[Grant Grundler]

On Mon, Aug 16, 2004 at 09:27:13PM +0300, Dror Goldenberg wrote:
> In PCI/PCIX, the interrupt is a wire, so it is not guaranteed that by the
> time you got the interrupt, the EQE will be waiting in memory.
> This is because interrupt goes on a separate wire from HCA to interrupt
> controller, while data goes up the PCI bridges.
> Therefore it is required to perform a PIO read to flush all
> posted writes flying upstream.

Dror,
I'm pretty sure you understand the issues but are using confusing
terminology:
o posted write. CPU does not wait for completion of write to IO device
o PIO write. Programmed IO - CPU write to IO device. May or not be posted
	and typically depends on chipset and which "space" (MMIO vs I/O
	Port) is the target.
o PIO read.  Programmed IO - CPU read stalls until completion (may be MMIO
      or I/O port space).
o DMA write: IO Device write to host memory (aka upstream)
o DMA read: Device command to retrieve data from host memory (downstream)
o DMA read return: completion portion of DMA read command (upstream)

A PIO Read "flushes" inflight DMA writes from a CPU perspective
because the CPU stalls until the PIO read completes.

> In PCI-Express, the interrupt is a message, so it will work. The interrupt
> will just flush the data to the memory because it maintain ordering with
> posted writes upstream.

The MSI/MSI-X interrupt doesn't do anything.
The interrupt transaction is just another DMA Write and must follow the
PCI ordering rules like any other DMA write. The destination address is
just not a regular host memory location.

> In the current driver, since it's PCI and PCI-Express we
> don't do it. In the new mode for Arbel we may do it.
> When you do MSI/MSI-X, then architecturally it is guaranteed that by the
> time you get the interrupt, the data already waits for you in memory.

yes.

FYI, PARISC *only* support transaction based interrupts.
The CPU has no interrupt lines going to it.
On IA64, I believe the same is true (for PCI) because the "Local SAPIC"
is integrated into the CPU (same silicon).

hth,
grant