[openib-general] ip over ib throughtput

[Roland Dreier]

    Grant> I expect SA_SHIRQ is harmless since the tg3 interrupt
    Grant> handler can't assume the MSI will be assigned and
    Grant> working. I don't want to add run time checks to tg3 for
    Grant> this (assuming it's obvious where to add them).  It might
    Grant> be better to document this in Documentation/MSI-HOWTO.txt.

I think a full MSI update for tg3 would need a separate MSI interrupt
routine that takes advantage of all the additional non-shared and
ordering guarantees that MSI provides.  This is what I did in mthca --
normally, we pass mthca_interrupt() to request_irq, but in MSI-X mode,
we pass mthca_msi_x_interrupt(), which doesn't do any mmio reads.

    Grant> BTW, I expect better perf if the platform code specifies
    Grant> IRQ_PER_CPU in the IRQ descriptor.  AFAICT, only ia64 and
    Grant> ppc64 do that. (parisc-linux just switched over to generic
    Grant> IRQ support...it will too).  Folks who care about AMD64
    Grant> performance should look into that as well - maybe it's not
    Grant> possible to direct interrupts to a specific CPU.

That's an interesting idea.  It is possible to assign interrupts to a
particular CPU on i386/x86_64 -- one can do it from userspace by
writing a mask with only one 1 in it into /proc/irq/N/smp_affinity.
In fact even if the mask allows more than one CPU, in a lot of systems
the interrupt will always go to the same CPU unless someone changes
the binding (either by the kernel kirqd or userspace IRQ balancer).

However I guess there may be a race condition when an IRQ is being
moved from one CPU to another.  How do ia64/ppc64 handle this?

 - R.