[openib-general] How about ib_send_page() ?

[Grant Grundler]

On Wed, May 18, 2005 at 01:38:48PM -0700, Jeff Carr wrote:
> Maybe. I'm not a PCI expert and have only used a PCI bus analyzer a few 
> times. I don't understand how disabling the interrupts would interfere 
> with DMA or add overhead.

PCI ordering rules force the PCI Bus controller to service MMIO reads
(used to flush MMIO writes) and it may be before a PCI DMA read
request has been fully satisfied. Ie disconnect and retry to get
the rest of the data.

If the driver only did MMIO writes, some chipsets allow DMA reads
to bypass MMIO writes (both are going in the same direction) until
the PCI device is satisfied or their latency timer has expired.

MMIO reads typically only use 4 bytes of 8 byte wide PCI bus
and similarly for any system the MMIO read might cross.
Ie transaction setup overhead and poor bus width utization
are reasons why people want to design cards that don't
need MMIO reads. Add the CPU stall time (1000+ cycles
per MMIO read) and it's very compelling.

MMIO Writes are less of a problem since the only real downside
is PCI bus utilization - not so much a problem with DMA flows.

> That condition strikes me as being chip dependent.
> So, perhaps the tg3 does that, but an IB HCA would be different. (?)

Yes. Libor just clarified in previous mail the mechanism IB can use.
I don't know how updates to those fields are communicated
to the kernel/HCA but I hope it doesn't involve MMIO reads.

grant