[openib-general] Ordering between PCI config space writes and MMIO reads?

[Jason Gunthorpe]

On Wed, Oct 25, 2006 at 08:18:59AM -0600, Matthew Wilcox wrote:

> PCI-PCI bridges are allowed to do it.  If you look in table E-1 of PCI
> 2.3, or table 8-3 of PCI-X 2.0, you'll see that a Posted Memory Write
> can pass a Delayed Write Request (or in PCI-X, a Memory Write can pass a
> Split Write Request).

Carefull here.. Only MMIO writes are of the posted variety. Non posted
transactions (config write, IO write, all reads) are not ever allowed
to pass a posted write, but they can be re-ordered.

Table 8-3 shows that Split Write vs Split Read are all Y/N
meaning they could be re-ordered with respect to each other.

The problem with mthca is exactly this, although the operations were
issued in-order by the bridges the end device (mthca) is free to
complete them in any order, and it choose to complete the MMIO read
before the config write.

>   In most PCI-X implementations, Split Requests are managed in separate
>   buffers from Split Completions, so Split Requests naturally pass Split
>   Completions. However, no deadlocks occur if Split Completions block
>   Split Requests.

Again, this is only for posted writes. All these rules are designed to
prevent the bus from deadlocking due to buffer starvation under
certain situations. [Basically split completions and posted writes are
given a seperate queue that can advance if the request queue is
stalled. Otherwise you can deadlock a bridge]

> So all this code that checks to see if a write had an effect is unsafe.
> I'm a little perturbed by this.  It means the only way to reliably

MMIO based code that does this is correct and reliable.. PIO code that
does this is only safe if the platform is waiting for the PIO write
completion before starting the PIO read.

PCI-X (pg 80) says this about non-posted transaction ordering requirements:
  As in convention PCI, if a requester requires one non-posted
  transaction to complete before another, it must not initiate the
  second transaction until the first one compeltes.

IMHO, all sane hardware implementations of config ops and PIO should
block the host bridge until the completion is generated by the end
device..I'm sure that most x86 platforms do this. (For instance I've
observed this kind of behavior with a Hyper Transport probe on
Opterons)

This is more than just worrying about ordering, it is about how to
engage a platform specific way to know that the completion has been
generated. The person who suggested polling the PIO_OUTSTANDING
register on SN2 seems to have the right idea (if that counts all
pending non-posted operations, not just PIO ones) :|

The risk of re-ordering is probably not so much in the bridges since
that would be a fairly strange thing to do - but it is very likely in
end-devices. This is especially true if the accesses are to different
internal resources!

Jason