[openib-general] Ordering between PCI config space writes and MMIO reads?

[John Partridge]

Matthew,

So, if I understand correctly, you are saying because we cannot guarantee
the "flush" a config write even by doing a config read of the same register
(because the PPB can re-order) we have to make sure we block or spin on the
config write completion at the lowest level of the config write ?

Thanks
John

Matthew Wilcox wrote:
> On Wed, Nov 01, 2006 at 10:27:19AM -0600, John Partridge wrote:
> 
>>Sorry, but I find this change a bit puzzling. The problem is particular to
>>the PPB on the HCA and not Altix.
> 
> 
> That's not true; it's more likely on Altix, but it's not unique.  *any*
> PCI-PCI bridge can reorder pci config reads and writes.  Apparently the
> normal PCI host bridge implementation avoids this problem by blocking
> until the completion comes back.  If you put a quad-port tulip card into
> an Altix, you could experience the same problem (but it would be
> massively unlikely.  You'd probably have to bring up three interfaces,
> saturate them with traffic, then bring up the fourth to see it.  And
> even then it would be rare).
> 
> 
>>I can't see anywhere that a PCI Config 
>>Write
>>is required to block until completion, it is the driver and the HCA ,not the
>>Altix hardware that requires the Config Write to have completed before we
>>leave mthca_reset()
> 
> 
> There's several places in the PCI midlayer that require the config write
> to have completed before we do a config read.  The MWI code relies on
> this to see if the device supports MWI.  If it gets out of order, we'll
> think that the device doesn't support MWI when it thinks it's been told
> to use MWI.  Data corruption could result.
> 
> 
>>Changing pci_write_config_xxx() will change the behavior
>>for ALL drivers and the possibility of breaking something else. The fix was
>>very low risk in mthca_reset(), changing the PCI code to fix this is much
>>more onerous.
> 
> 
> I really don't think so.  At worst you'll be changing the timing.


-- 
John Partridge

Silicon Graphics Inc
Tel:	651-683-3428
Vnet:	233-3428
E-Mail:	johnip at sgi.com