[openib-general] Ordering between PCI config space writes and MMIO reads?

[Matthew Wilcox]

On Wed, Nov 01, 2006 at 10:27:19AM -0600, John Partridge wrote:
> Sorry, but I find this change a bit puzzling. The problem is particular to
> the PPB on the HCA and not Altix.

That's not true; it's more likely on Altix, but it's not unique.  *any*
PCI-PCI bridge can reorder pci config reads and writes.  Apparently the
normal PCI host bridge implementation avoids this problem by blocking
until the completion comes back.  If you put a quad-port tulip card into
an Altix, you could experience the same problem (but it would be
massively unlikely.  You'd probably have to bring up three interfaces,
saturate them with traffic, then bring up the fourth to see it.  And
even then it would be rare).

> I can't see anywhere that a PCI Config 
> Write
> is required to block until completion, it is the driver and the HCA ,not the
> Altix hardware that requires the Config Write to have completed before we
> leave mthca_reset()

There's several places in the PCI midlayer that require the config write
to have completed before we do a config read.  The MWI code relies on
this to see if the device supports MWI.  If it gets out of order, we'll
think that the device doesn't support MWI when it thinks it's been told
to use MWI.  Data corruption could result.

> Changing pci_write_config_xxx() will change the behavior
> for ALL drivers and the possibility of breaking something else. The fix was
> very low risk in mthca_reset(), changing the PCI code to fix this is much
> more onerous.

I really don't think so.  At worst you'll be changing the timing.