[openib-general] ip over ib throughtput

[Grant Grundler]

On Tue, Jan 11, 2005 at 01:41:00PM -0800, Roland Dreier wrote:
> I think a full MSI update for tg3 would need a separate MSI interrupt
> routine that takes advantage of all the additional non-shared and
> ordering guarantees that MSI provides.

That's the beauty of bcm57xx. The DMA/Intr ordering is handled by passing
tags back of forth - so the NIC knows which status block the host saw
on the current interrupt. It might result in a spurious interrupt
on occasion but very portable IMHO.

> This is what I did in mthca --
> normally, we pass mthca_interrupt() to request_irq, but in MSI-X mode,
> we pass mthca_msi_x_interrupt(), which doesn't do any mmio reads.

*nod*

> That's an interesting idea.  It is possible to assign interrupts to a
> particular CPU on i386/x86_64 -- one can do it from userspace by
> writing a mask with only one 1 in it into /proc/irq/N/smp_affinity.

Yes. If one can program the HW so that is enforced.  I'm not sure
IRQ redirection is tunable on older x86 chipsets that might also have PCI.
It is for anything modern that uses xAPIC (aka SAPIC for ia64/parisc).

I've already suggested to parisc-linux forum it might be worth making IRQ_PER_CPU
a compile time option instead of a run time option. Especially on arches where that
is the ONLY available behavior.

> In fact even if the mask allows more than one CPU, in a lot of systems
> the interrupt will always go to the same CPU unless someone changes
> the binding (either by the kernel kirqd or userspace IRQ balancer).

AFAIK: PPC, IA64, parisc, and alpha by design. Maybe others too.

IA64 *can* redirect interrupts a la x86 if "XPR" register is implemented in the chipset.
HP IA64 chipsets to date do not and that was intentional.

> However I guess there may be a race condition when an IRQ is being
> moved from one CPU to another.  How do ia64/ppc64 handle this?

Off hand I don't know.
I expect something like disable_irq()/change params/enable_irq() to DTRT.
disable_irq() will guarantee all "inflight" IRQs have been handled
and the respective handlers have exited.
As the name implies, disable_irq_nosync() does not provide that guarantee.

grant