[openib-general] Ordering between PCI config space writes and MMIO reads?

[Jack Steiner]

On Tue, Oct 24, 2006 at 04:36:32PM -0600, Matthew Wilcox wrote:
> On Tue, Oct 24, 2006 at 02:51:30PM -0700, Roland Dreier wrote:
> >  > I think the right way to fix this is to ensure mmio write ordering in
> >  > the pci_write_config_*() implementations.  Like this.
> > 
> > I'm happy to fix this in the PCI core and not force drivers to worry
> > about this.
> > 
> > John, can you confirm that this patch fixes the issue for you?
> 
> Hang on.  I wasn't thinking clearly.  mmiowb() only ensures the write
> has got as far as the shub.  

I think mmiowb() should work on SN hardware. mmiowb() delays until shub
reports that all previously issued PIO writes have completed. 

The processor "mf.a" guarantees "platform acceptance" which on SN means
that shub has accepted the write - not that it has actually completed (or
even forwarded anywhere by shub). That makes "mf.a" more-or-less useless
on SN. However, shub has an additional MMR register (PIO_WRITE_COUNT) that
counts actual outstanding PIOs.  mmiob() delays until that count goes to
zero.

I'll check if there is any additional reordering that can occur AFTER the
PIO_WRITE_COUNT goes to zero.  If so, it would be at bus level - not in
shub or routers.


> There's no way to fix this in the pci core
> -- any PCI-PCI bridge can reorder the two.
> 
> This is only really a problem for setup (when we program the BARs), so
> it seems silly to enforce an ordering at any other time.  Reluctantly, I
> must disagree with Jeff -- drivers need to fix this.

-- jack