[openib-general] basic IB doubt

[Michael Krause]

At 10:45 AM 8/25/2006, Tom Tucker wrote:
>On Fri, 2006-08-25 at 12:51 -0400, Talpey, Thomas wrote:
> > At 12:40 PM 8/25/2006, Sean Hefty wrote:
> > >>    Thomas> How does an adapter guarantee that no bridges or other
> > >>    Thomas> intervening devices reorder their writes, or for that
> > >>    Thomas> matter flush them to memory at all!?
> > >>
> > >>That's a good point.  The HCA would have to do a read to flush the
> > >>posted writes, and I'm sure it's not doing that (since it would add
> > >>horrible latency for no good reason).
> > >>
> > >>I guess it's not safe to rely on ordering of RDMA writes after all.
> > >
> > >Couldn't the same point then be made that a CQ entry may come before 
> the data
> > >has been posted?
> >
> > When the CQ entry arrives, the context that polls it off the queue
> > must use the dma_sync_*() api to finalize any associated data
> > transactions (known by the uper layer).
> >
> > This is basic, and it's the reason that a completion is so important.
> > The completion, in and of itself, isn't what drives the synchronization.
> > It's the transfer of control to the processor.
>
>This is a giant rat hole.
>
>On a coherent cache architecture, the CQE write posted to the bus
>following the write of the last byte of data will NOT be seen by the
>processor prior to the last byte of data. That is, write ordering is
>preserved in bridges.
>
>The dma_sync_* API has to do with processor cache, not transaction
>ordering. In fact, per this argument at the time you called dma_sync_*,
>the processor may not have seen the reordered transaction yet, so what
>would it be syncing?
>
>Write ordering and read ordering/fence is preserved in intervening
>bridges. What you DON'T know is whether or not a write (which was posted
>and may be sitting in a bridge FIFO) has been flushed and/or propagated
>to memory at the time you submit the next write and/or interrupt the
>host.
>
>If you submit a READ following the write, however, per the PCI bus
>ordering rules you know that the data is in the target.
>
>Unless, of course, I'm wrong ... :-)

A PCI read following a write to the same address will result validate that 
all prior write transactions are flushed to host memory.  This is one way 
that people have used (albeit with a performance penalty) to verify that a 
transaction it out of the HCA / RNIC fault zone and therefore an 
acknowledgement to the source means the data is safe and one can survive 
the HCA / RNIC failing without falling into a non-deterministic 
state.    PCI writes are strongly ordered on any PCI technology 
offering.   Relaxed ordering needs to be taken into account w.r.t. writes 
vs. reads as well as read completions being weakly ordered as well.

Mike  
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.openfabrics.org/pipermail/general/attachments/20060825/d724d80e/attachment.html>