[openib-general] Help with CONFIG_PCI_MSI in the kernel

[Jason Gunthorpe]

On Mon, Apr 03, 2006 at 09:36:00PM -0700, Grant Grundler wrote:
> On Mon, Apr 03, 2006 at 03:14:56PM -0700, Greg Lindahl wrote:
> > Red Hat has started turning off CONFIG_PCI_MSI in their kernels (FC5
> > and the latest FC4 update). I remember a while back there was a
> > discussion about how MSI made the Mellanox HCA run faster, can someone
> > please add some concrete details about this to the bug? Thanks.
 
> I also don't see why MSI support should be disabled.
> MSI should work fine on most of the platforms out there.
> For the cases where MSI does not work correctly and there
> is no workaround in place yet, Matthew Wilcox proposed a
> "platform" MSI disable patch. User can then disable
> MSI globally at boot time.

One thing that struck me as strange with the MSI code was that it just
unconditionally turns on without any checks that the system actually
supports it, so that it breaks in some cases doesn't seem to be too
surprising. MSI requires end device support and something in a bridge
to transform the MSI message into an APIC message, but the kernel
currently only looks for end device support.

Presumably only some subset of Intel PCI/PCI-X bridges support MSI
translation to an APIC message on the FSB, and we know on HT systems
that the MSI PCI capability block is required (and must be *enabled*)
in the PCI/PCI-X/PCIe bridge for there to be support.

IMHO what is needed is that if a device X wants MSI on then the path
from X to the root bridge must hit a MSI capable bridge[1]. White list
the known good intel chips, detect HT capability blocks, turn on MSI
for *every* device (ie 100% of PCIe devices), not just high
performance ones that specifically ask for it - then black list any
broken hardware ;>

In the long term as PCIe becomes dominant this is best since linux
becomes firmly in control of all interrupt routing and doesn't rely on
working APCI for IOAPIC tables..

It is hard to blame RedHat for turning it off by default when I know
some BIOS vendors are not enabling MSI translation in the HT
chipsets. I have some MBs that are broken like that and MSI only works
with a kernel patch that forces the MSI translation to be turned on.
Ideally the kernel would detect that and just refuse to do MSI.

[1: In this same way the current hack to disable MSI if an 8131 is
 detected is bogus, there are systems with 8131's and other nvidia
 bridges that support MSI on PCIe slots (via a nvidia bridge) but
 not on PCI-X (since the 8131 doesn't support it)]

Regards,
Jason