<html>
<body>
<font size=3>At 10:45 AM 8/25/2006, Tom Tucker wrote:<br>
<blockquote type=cite class=cite cite="">On Fri, 2006-08-25 at 12:51
-0400, Talpey, Thomas wrote: <br>
> At 12:40 PM 8/25/2006, Sean Hefty wrote:<br>
> >> Thomas> How does an adapter guarantee
that no bridges or other<br>
> >> Thomas> intervening devices reorder
their writes, or for that<br>
> >> Thomas> matter flush them to memory at
all!?<br>
> >><br>
> >>That's a good point. The HCA would have to do a read
to flush the<br>
> >>posted writes, and I'm sure it's not doing that (since it
would add<br>
> >>horrible latency for no good reason).<br>
> >><br>
> >>I guess it's not safe to rely on ordering of RDMA writes
after all.<br>
> ><br>
> >Couldn't the same point then be made that a CQ entry may come
before the data<br>
> >has been posted?<br>
> <br>
> When the CQ entry arrives, the context that polls it off the
queue<br>
> must use the dma_sync_*() api to finalize any associated data<br>
> transactions (known by the uper layer).<br>
> <br>
> This is basic, and it's the reason that a completion is so
important.<br>
> The completion, in and of itself, isn't what drives the
synchronization.<br>
> It's the transfer of control to the processor.<br><br>
This is a giant rat hole. <br><br>
On a coherent cache architecture, the CQE write posted to the bus<br>
following the write of the last byte of data will NOT be seen by the<br>
processor prior to the last byte of data. That is, write ordering is<br>
preserved in bridges.<br><br>
The dma_sync_* API has to do with processor cache, not transaction<br>
ordering. In fact, per this argument at the time you called
dma_sync_*,<br>
the processor may not have seen the reordered transaction yet, so
what<br>
would it be syncing?<br><br>
Write ordering and read ordering/fence is preserved in intervening<br>
bridges. What you DON'T know is whether or not a write (which was
posted<br>
and may be sitting in a bridge FIFO) has been flushed and/or
propagated<br>
to memory at the time you submit the next write and/or interrupt the<br>
host. <br><br>
If you submit a READ following the write, however, per the PCI bus<br>
ordering rules you know that the data is in the target. <br><br>
Unless, of course, I'm wrong ... :-)</font></blockquote><br>
A PCI read following a write to the same address will result validate
that all prior write transactions are flushed to host memory. This
is one way that people have used (albeit with a performance penalty) to
verify that a transaction it out of the HCA / RNIC fault zone and
therefore an acknowledgement to the source means the data is safe and one
can survive the HCA / RNIC failing without falling into a
non-deterministic state. PCI writes are strongly
ordered on any PCI technology offering. Relaxed ordering
needs to be taken into account w.r.t. writes vs. reads as well as read
completions being weakly ordered as well. <br><br>
Mike </body>
</html>