[openib-general] ip over ib throughtput

[Roland Dreier]

    Grant> Getting IRQ line routing wrong will show up for *any*
    Grant> device.  I'll agree this case is possibly an ACPI bug - but
    Grant> in an experimental release of bleeding edge kernel
    Grant> version. That's no surprise.  Normal users will (almost)
    Grant> never see this kind of failure.

It seems that the fact that Mellanox HCAs show up as a device with a
giant 128 MB BAR behind a bridge triggers all sorts of BIOS nastiness
(since of course no BIOS vendor would ever test such a thing!).  I've
definitely seen HCA interrupt routing fail on many otherwise-working
systems.  (Things have been gradually improving but problems still
crop up)

    Grant> Maybe that's a good reason to put the pci_enable_msi()
    Grant> right before the enable_irq() call path?  (and ditto for
    Grant> free_irq()/pci_disable_msi())

    Grant> Right now those two are not anywhere near each other and
    Grant> the MSI_HOWTO.txt is very clear there is a dependency - at
    Grant> least for MSI support. I still haven't studied details of
    Grant> MSI-X though I'm aware of some of the basic differences.  I
    Grant> can submit a patch after I'm done messing with MSI on
    Grant> tg3...sound ok?

There is a reason for the slightly awkward order.  It turns out that
there are some quirks with the Mellanox HCA and it's better not to
touch the PCI header after the SYS_EN firmware command is executed.
So the driver ends up having to set up MSI/MSI-X way before it's ready
to enable interrupts.

 - R.