[openib-general] [PATCH] roland-uverbs: possible race condition

[Michael Krause]

At 02:12 PM 1/31/2005, Grant Grundler wrote:
>On Mon, Jan 31, 2005 at 12:54:47PM -0800, Roland Dreier wrote:
> > I seem to remember someone quoting the PCI spec saying that this was
> > not allowed, but I can't find it right now.  Grant, do you happen to
> > know if legacy INTx interrupts are allowed to pass writes to memory
> > once they leave PCI Express and make it into the chipset?
>
>I don't know offhand and (sorry) can't look it up right now.
>Poke me again later this week (thurs) if you need a solid answer
>and no one else has provided one.
>
>I do expect ordering issues with the path between the device and
>the bridge chip that is handling legacy INTx support. This is basically
>the same DMA ordering issues we see today with Line based interrupts.
>It would make sense for the rest of the path to NOT have strict ordering
>given this hole.

Ordering matrix is documented within the PCIe base 1.0a specification.  The 
rules are fairly straight forward depending upon the capabilities being 
accessed.  Too much to type in here but to give you an idea:

- INTx are treated as writes from a PCIe transaction perspective.

- In general, a TC defines an ordering domain.  The TC to VC mapping is 
used to control how a transaction (aka TLP) is serviced on a per link 
basis.  Consistency in how a TC is mapped can be important if the 
application is operating across multiple TC. However, servers do not 
require multiple TC or VC (waste of resources) nor really does any 
application so one can design under the assumption that a single TC is 
sufficient.  All transactions should therefore have the same TC or you are 
headed for a bit of complexity and pain.

- Within a link, all TLP are processed on a given VC in the order they are 
presented.  They are always strongly ordered across a given link.  If an 
intermediate device such as a PCIe switch is present, it must obey the PCI 
ordering rules and avoid re-ordering where prohibited (e.g. a Read getting 
ahead of a Write which is a common flush the path technique).

- The PCIe specifications only define how a TLP is process to / from a RC 
to another endpoint or between endpoints.  They do not define how a RC is 
implemented or how it interacts with the memory subsystem.

- Weak ordering is allowed in PCIe just like for PCI / PCI-X (our objective 
was to enable compatible operations while minimizing the pain to migrate to 
a new interconnect technology).

- In general, all interrupts (line or MSI/MSI-X) should be strongly ordered 
relative to other write operations to avoid silent data corruption from 
occurring.  As such, an interrupt should not pass a memory write when being 
processed by the chipset.

If you have any other questions, I'm happy to try and answer or one can 
read the friendly specification for more specifics.

Mike 
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.openfabrics.org/pipermail/general/attachments/20050131/98882e90/attachment.html>