[openib-general] has anyone tried MSI-X on Opteron?

[Roland Dreier]

> If anyone has an nForce4 mobo, I would be _very_ interested to see the
> output of lspci -vvxxx, especially using lspci from the latest
> pciutils 2.1.99 (available from /pub/software/utils/pciutils/alpha on
> ftp.kernel.org).

For what it's worth at this point, I finally got an nForce4 mobo.  The
nForce4 PCI bridge does support MSI:

00:0b.0 PCI bridge: nVidia Corporation CK804 PCIE Bridge (rev a3)
(prog-if 00 [Normal decode])
	Control: I/O+ Mem+ BusMaster+ SpecCycle- MemWINV- VGASnoop- ParErr-
Stepping- SERR+ FastB2B-
	Status: Cap+ 66Mhz- UDF- FastB2B- ParErr- DEVSEL=fast >TAbort-
<TAbort- <MAbort- >SERR- <PERR-
	Latency: 0, Cache Line Size 08
	Bus: primary=00, secondary=04, subordinate=04, sec-latency=0
	I/O behind bridge: 0000f000-00000fff
	Memory behind bridge: fff00000-000fffff
	Prefetchable memory behind bridge: 00000000fff00000-0000000000000000
	Secondary status: 66Mhz- FastB2B- ParErr- DEVSEL=fast >TAbort-
<TAbort- <MAbort- <SERR- <PERR-
	BridgeCtl: Parity- SERR+ NoISA- VGA- MAbort- >Reset- FastB2B-
	Capabilities: [40] Power Management version 2
		Flags: PMEClk- DSI- D1- D2- AuxCurrent=0mA PME(D0+,D1+,D2+,D3hot+,D3cold+)
		Status: D0 PME-Enable- DSel=0 DScale=0 PME-
	Capabilities: [48] Message Signalled Interrupts: 64bit+ Queue=0/1 Enable-
		Address: 0000000000000000  Data: 0000
	Capabilities: [58] HyperTransport: MSI Mapping
	Capabilities: [80] Express Root Port (Slot+) IRQ 0
		Device: Supported: MaxPayload 128 bytes, PhantFunc 0, ExtTag-
		Device: Latency L0s <512ns, L1 <4us
		Device: Errors: Correctable- Non-Fatal- Fatal- Unsupported-
		Device: RlxdOrd+ ExtTag- PhantFunc- AuxPwr- NoSnoop+
		Device: MaxPayload 128 bytes, MaxReadReq 512 bytes
		Link: Supported Speed 2.5Gb/s, Width x2, ASPM L0s, Port 3
		Link: Latency L0s <512ns, L1 <4us
		Link: ASPM Disabled RCB 64 bytes CommClk- ExtSynch-
		Link: Speed 2.5Gb/s, Width x4 0x1041
		Slot: AtnBtn- PwrCtrl- MRL- AtnInd- PwrInd- HotPlug- Surpise-
		Slot: Number 8, PowerLimit 25.000000
		Slot: Enabled AtnBtn- PwrFlt- MRL- PresDet- CmdCplt- HPIrq-
		Slot: AttnInd Off, PwrInd On, Power-
		Root: Correctable- Non-Fatal- Fatal- PME-

and in fact the current Linux MSI-X code works fine.  On that system
with a PCIe HCA, I have the following in /proc/interrupts:

           CPU0
  0:    1463994    IO-APIC-edge  timer
  7:          2    IO-APIC-edge  parport0
  8:          0    IO-APIC-edge  rtc
  9:          0   IO-APIC-level  acpi
 15:      12942    IO-APIC-edge  ide1
 50:          0       PCI-MSI-X  ib_mthca (async)
 58:        321       PCI-MSI-X  ib_mthca (cmd)
 66:          0   IO-APIC-level  SysKonnect SK-98xx
177:        525   IO-APIC-level  libata, NVidia CK804
185:     188960   IO-APIC-level  libata, eth0
193:      90623   IO-APIC-level  nvidia
201:      27005   IO-APIC-level  ohci_hcd
209:          2   IO-APIC-level  ehci_hcd
217:          3   IO-APIC-level  ohci1394
233:          0       PCI-MSI-X  ib_mthca (comp)
NMI:        173
LOC:    1463664
ERR:          0
MIS:          0

(nothing connected to the HCA yet, so no interrupts other
than command completions)

 - R.