[openib-general] Help with CONFIG_PCI_MSI in the kernel

[Jason Gunthorpe]

On Tue, Apr 04, 2006 at 04:52:32PM -0700, Grant Grundler wrote:

> Yes, that's what I meant with "routing of the transaction".
> "interrupt message transaction" is not "transformed" - just routed.
> Page 19 uses the same language.

Yes, I think we are saying exactly the same thing - I was saying
'transformed' because I don't know the exact bus protocol used
on an Intel FSB. (Not knowing that I have no idea if it is a memory
write to the CPU or some kind of special APIC interrupt message or
whatever.)

> Mark Maul is attempting to fix since it just doesn't
> work for large SGI systems.

Thats sounds great!
 
> > I doubt all intel chipsets ever produced have this transformation.
> 
> I expect any Intel box that uses a LOCAL xAPIC _must_ support a PIB
> (and therefore routing of MSI).

Ok. I just remember long ago when APIC first came out there used to be
a dedicated out of band APIC bus that carried the vectors from the IO
APCIs directly. These days I really only know exact bus protocol
specifics about HT and AMD64..

[clip about fedora]

You are totally right of course WRT to fedora. I was just thinking
aloud about how to have MSI turned on and work for all supported
platforms more generally.

Certainly it sounds like all Intel EM64T systems should work and a
varient of my patch can be used to identify the AMD64 ones that don't.
(look for the HT MSI MAPPING cap on each bridge, if it is absent then
it is not supported, 8131 does not have a HT MSI MAPPING cap)

> This fits nicely with the patches that Mark Maule (SGI) has submitted
> to make the hardcoding of PIB an x86 thing. Sounds like we have
> another interested party in making the MSI support less Intel
> specific.

Yes, I think it would go well with HT MSI support.

> ah ok. But I was commenting on the dependency on firmware. It's still
> there since we don't know which devices need IRQ lines and which can
> live without them. The concept of "IRQ Lines" doesn't go away
> with PCI-e. PCI-e just virtualizes the concept into transactions
> that a parent bridge can convert into MSI transactions.
> The parent bridge in this case is essentially doing the same
> thing that an IO xAPIC (or SAPIC) would do.

Mm, can you clarify this? Your description matches my understanding of
PCIe legacy (8252 and IOAPIC style) interrupts. However with MSI at
the device the parent bridge isn't involved in vector selection so
the APCI IOAPIC linkage information is unused.

I only point this out because it seems to be a common problem that
some el-chepo motherboards have broken APCI and important things like HD
controllers don't work because their interrupt linkages are mangled
somehow. In a UP system alot of the other APCI information is not
critical (or as error prone..) so using MSI would be a nice
alternative way to get such a system to work 100%.

> Nice. Looks good to me.
> I'll bounce your mail linux-pci.
> I've cc'd linux-pci on this reply.

Two notes about it:
1) It doesn't force the base address to 0xfee00000 [this is the
   documented reset value however]
2) If used with Mark Maule's work, then the base address should be
   taken from the first bridge with a HT MSI Mapping cap, going along the
   bridge chain toward the host from the device that the MSI address is
   being generated for.

Thanks,
Jason