[openib-general] basic IB doubt

[Tom Tucker]

On Fri, 2006-08-25 at 12:51 -0400, Talpey, Thomas wrote: 
> At 12:40 PM 8/25/2006, Sean Hefty wrote:
> >>    Thomas> How does an adapter guarantee that no bridges or other
> >>    Thomas> intervening devices reorder their writes, or for that
> >>    Thomas> matter flush them to memory at all!?
> >>
> >>That's a good point.  The HCA would have to do a read to flush the
> >>posted writes, and I'm sure it's not doing that (since it would add
> >>horrible latency for no good reason).
> >>
> >>I guess it's not safe to rely on ordering of RDMA writes after all.
> >
> >Couldn't the same point then be made that a CQ entry may come before the data
> >has been posted?
> 
> When the CQ entry arrives, the context that polls it off the queue
> must use the dma_sync_*() api to finalize any associated data
> transactions (known by the uper layer).
> 
> This is basic, and it's the reason that a completion is so important.
> The completion, in and of itself, isn't what drives the synchronization.
> It's the transfer of control to the processor.

This is a giant rat hole. 

On a coherent cache architecture, the CQE write posted to the bus
following the write of the last byte of data will NOT be seen by the
processor prior to the last byte of data. That is, write ordering is
preserved in bridges.

The dma_sync_* API has to do with processor cache, not transaction
ordering. In fact, per this argument at the time you called dma_sync_*,
the processor may not have seen the reordered transaction yet, so what
would it be syncing?

Write ordering and read ordering/fence is preserved in intervening
bridges. What you DON'T know is whether or not a write (which was posted
and may be sitting in a bridge FIFO) has been flushed and/or propagated
to memory at the time you submit the next write and/or interrupt the
host. 

If you submit a READ following the write, however, per the PCI bus
ordering rules you know that the data is in the target.  

Unless, of course, I'm wrong ... :-)


> 
> Tom.
> 
> 
> _______________________________________________
> openib-general mailing list
> openib-general at openib.org
> http://openib.org/mailman/listinfo/openib-general
> 
> To unsubscribe, please visit http://openib.org/mailman/listinfo/openib-general
>