[openib-general] Race in mthca_cmd_post()

[John Partridge]

Roland,

I checked with our chief hardware engineer and here's what he had to say :-

	Yes, config writes are non-posted, but, no, it should not
	block everything.

	From the 1.0 version of the PCI-X spec, sec 8.4.4 Transaction
	Ordering and Passing Rules for Bridges, p 153

	"Split Requests are permitted to be blocked by or to pass
	other Split Requests. (These Split Transactions in PCI-X have
	the same requirements as Delayed Transactions in conventional
	PCI.)"

	(Appendix E of the version 2.3 PCI spec says the same thing
	about Delayed Transactions)

	And on p151 of the PCI-X spec:

	"If an initiator requires two Split Transactions to complete
	in order, the initiator must not issue the second request
	until the first Split Transaction completes."

	So in the failing example, the bridge chip is issuing the 2nd Split
	Transaction (the read) before the 1st (the config write) completes.
	From my hardware bias, it seems like the driver should be in
	charge of determining whether or not there's an ordering dependence
	and take the appropriate action. Otherwise, the hardware has no way
	of knowing whether or not there's an order requirement and will
	have to serialize everything. That's a negative impact on
	performance and kind of renders the spec-allowed option of passing
	or not passing meaningless.

Roland Dreier wrote:
>  > The posted/non-posted write stuff in the spec really only means that a
>  > split completion is generated for that transaction on the bus. There
>  > is no bus-level requirement that the bus halt while an outstanding
>  > split is pending. In fact, the PCI-X ordering rules in this case
>  > actually would allow your config read and memory read to be re-ordered
>  > by a the bridge (table 8-3). ``Split requests are permitted to be
>  > blocked by or pass other split requests.''
>  > 
>  > Most implementations block the CPU on a non-posted write which
>  > provides the necessary serialization, but Altix clearly didn't..
> 
> Thanks, I think I get it now.  It does seem like this behavior skirts
> right inside the boundary of what the PCI spec allows.  What was
> confusing me was the section:
> 
>     Non-posted transactions reach their ultimate destination before
>     completing at the originating device. The master cannot proceed
>     with any other work until the transaction has completed at the
>     ultimate destination (if a dependency exists).
> 
> I just saw "the master cannot proceed" but the first few times I read
> this, I didn't see the "(if a dependency exists)."  Since no
> dependency exists between the config write and the subsequent memory
> read, the master _can_ proceed in this case.  So I do agree that this
> patch looks correct and needed, although the Altix behavior is
> somewhat unusual.

So, do you need anything alse from me like an attached copy of the patch ?

Thanks
John

-- 
John Partridge

Silicon Graphics Inc
Tel:  651-683-3428
Vnet: 233-3428
E-Mail: johnip at sgi.com