[openib-general] Race in mthca_cmd_post()

[Jason Gunthorpe]

On Thu, Oct 19, 2006 at 08:48:44PM -0700, Roland Dreier wrote:
> OK, this is the crux of my confusion.  I always thought (and the PCI
> spec seems to say this too) that config writes are non-posted, which
> means that the Config Write cycle in your trace should block
> everything until it is completed.  Is that not true?  Or could this be
> a bug in the SAL for this platform or something like that?

The posted/non-posted write stuff in the spec really only means that a
split completion is generated for that transaction on the bus. There
is no bus-level requirement that the bus halt while an outstanding
split is pending. In fact, the PCI-X ordering rules in this case
actually would allow your config read and memory read to be re-ordered
by a the bridge (table 8-3). ``Split requests are permitted to be
blocked by or pass other split requests.''

Most implementations block the CPU on a non-posted write which
provides the necessary serialization, but Altix clearly didn't..

To properly correct this you need to have a barrier that synchronizes
with the split completion from the device. I'd even go so far as to
say that not doing this for config_write is unusual/useless behavior
and the barrier might be better part of the config_write primitive...

Jason