[openib-general] ip over ib throughtput

[Grant Grundler]

On Tue, Jan 11, 2005 at 10:40:49AM -0800, Roland Dreier wrote:
> That's a little strange.  It could either be the pci_enable_msi() or
> the MSGINT_MODE setting that needs to happen later.

I suspect it's the MSGINT_MODE that needs to happen later.

> What puzzles me
> is that there's a call to tg3_init_hw() even after the request_irq,
> which does a tg3_reset_hw() -- so why doesn't that mess up our
> MSGINT_MODE?

Good question. I really don't know. Only broadcom will know.
tg3_init_hw doesn't mess up the BARs or MSI address/data either.

Keep in mind a "reset" isn't a *full* HW reset (ie like cycling the power).
It's all under SW (onboard firmware) control.

| eth1: RXcsums[1] LinkChgREG[0] MIirq[0] ASF[0] Split[0] WireSpeed[1] TSOcap[0] 

Looks like TSO isn't enabled and I'm running a 5701 chip right now.
I will need to find out a firmware upload (e.g. TG3_TSO_SUPPORT != 0)
would mess up MSGINT_MODE as well.

> In any case I'm glad someone is finally looking at enabling MSI
> support in more drivers.  It would be really nice to get this upstream
> so that MSI support starts getting some real testing on lots of
> systems.

Yes - and I'd like to twiddle with tg3 a bit more to see if I can get it
operating with ZERO mmio reads like it was designed for. 

> One detail that you should add before sending this upstream is that
> SA_SHIRQ is not needed in MSI mode -- the driver can be sure that the
> interrupt is not shared.

I expect SA_SHIRQ is harmless since the tg3 interrupt handler can't assume
the MSI will be assigned and working. I don't want to add run time checks
to tg3 for this (assuming it's obvious where to add them).
It might be better to document this in Documentation/MSI-HOWTO.txt.

BTW, I expect better perf if the platform code specifies IRQ_PER_CPU in the IRQ descriptor.
AFAICT, only ia64 and ppc64 do that. (parisc-linux just switched over to generic
IRQ support...it will too).  Folks who care about AMD64 performance should look
into that as well - maybe it's not possible to direct interrupts to a specific CPU.

thanks,
grant