[openib-general] Help with CONFIG_PCI_MSI in the kernel

[Grant Grundler]

On Tue, Apr 04, 2006 at 01:02:29AM -0600, Jason Gunthorpe wrote:
> One thing that struck me as strange with the MSI code was that it just
> unconditionally turns on without any checks that the system actually
> supports it, so that it breaks in some cases doesn't seem to be too
> surprising.

I agree it's not surprising that some implementations are broken.

> MSI requires end device support and something in a bridge
> to transform the MSI message into an APIC message, but the kernel
> currently only looks for end device support.

APIC Message?
MSI is just a DMA-write from the card point of view.
So if PCI is working and DMA is working, MSI should work too.
The difference is routing of the transaction and the fact that
it's not targeting Host memory but some other part of the chipset.


> Presumably only some subset of Intel PCI/PCI-X bridges support MSI
> translation to an APIC message on the FSB, and we know on HT systems
> that the MSI PCI capability block is required (and must be *enabled*)
> in the PCI/PCI-X/PCIe bridge for there to be support.

Well, can't linux enable that block if it's present?
That isn't a reason to disable MSI for _all_ systems.

> IMHO what is needed is that if a device X wants MSI on then the path
> from X to the root bridge must hit a MSI capable bridge[1]. White list
> the known good intel chips, detect HT capability blocks, turn on MSI
> for *every* device (ie 100% of PCIe devices), not just high
> performance ones that specifically ask for it - then black list any
> broken hardware ;>

I'd prefer to just black list the broken ones or encourage people
to use "nomsi". Secondly, we've already had this discussion on
the linux-pci mailing list about turning it on for _ALL_ devices
that advertise MSI/MSI-X - it's a non-starter. Too many devices
have broken implementations (e.g. don't turn off Line IRQs).
The driver needs to ask for MSI since it's the only one
to know that the device can really handle MSI properly.

> In the long term as PCIe becomes dominant this is best since linux
> becomes firmly in control of all interrupt routing and doesn't rely on
> working APCI for IOAPIC tables..

Sorry...I can't agree.
Line based interrupt routing is dependent on firmware to give
us the IRQ->APIC routing tables, enough info to identify CPUs (ID/EID
info for Intel implementations) and program IO-xAPIC entries.
Essentially, MSI only needs the CPU Info so MSI transactions get
routed correctly. Then MSI/-X entries on the devices can be
programmed (essentially the same way an IO-xAPIC gets programmed).


> It is hard to blame RedHat for turning it off by default when I know
> some BIOS vendors are not enabling MSI translation in the HT
> chipsets. I have some MBs that are broken like that and MSI only works
> with a kernel patch that forces the MSI translation to be turned on.
> Ideally the kernel would detect that and just refuse to do MSI.

If it's "just a BIOS" issue, then can't AMD help linux turn MSI support on?

ie linux can bang some values into the chipset like we do for other types
of initialization when BIOS doesn't do it right.

> [1: In this same way the current hack to disable MSI if an 8131 is
>  detected is bogus, there are systems with 8131's and other nvidia
>  bridges that support MSI on PCIe slots (via a nvidia bridge) but
>  not on PCI-X (since the 8131 doesn't support it)]

Can you submit a patch with the right way to disable MSI for 8131 chips?

thanks,
grant

> 
> Regards,
> Jason