[openib-general] basic IB doubt

[Michael Krause]

At 09:50 AM 8/25/2006, Caitlin Bestler wrote:
>openib-general-bounces at openib.org wrote:
> >>    Thomas> How does an adapter guarantee that no bridges or other
> >>    Thomas> intervening devices reorder their writes, or for that
> >>    Thomas> matter flush them to memory at all!?
> >>
> >> That's a good point.  The HCA would have to do a read to flush the
> >> posted writes, and I'm sure it's not doing that (since it would add
> >> horrible latency for no good reason).
> >>
> >> I guess it's not safe to rely on ordering of RDMA writes after all.
> >
> > Couldn't the same point then be made that a CQ entry may come
> > before the data has been posted?
> >
>
>That's why both specs (IBTA and RDMAC) are very explicit that all
>prior messages are complete before the CQE is given to the user.
>
>It is up to the RDMA Device and/or its driver to guarantee this
>by whatever means are appropriate. An implementation that allows
>a CQE post to pass the data placement that it is reporting on the
>PCI bus is in error.
>
>The critical concept of the Work Completion is that it consolidates
>guarantees and notificatins. The implementation can do all sorts
>of strange things that it thinks optimize *before* the work completion,
>but at the time the work completion is delivered to the user everything
>is supposed to be as expected.

Caitlin's logic is correct and the basis for why these two specifications 
call out this issue.  And yes, Roland, one cannot rely upon RDMA Write 
ordering whether for IB or iWARP. iWARP specifically allows out of order 
delivery.  IB while providing in-order delivery due to its strong ordering 
protocol still has no guarantees when it comes to the memory controller and 
I/O technology being used.  Given not everything was expected to operate 
over PCI, we made sure that the specifications pointed out these issues so 
that software would be designed to accommodate all interconnect attach 
types and usage models.  We wanted to maximize the underlying 
implementation options while providing software with a consistent operating 
model to enable it to be simplified as well.

Mike 
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.openfabrics.org/pipermail/general/attachments/20060825/c832112e/attachment.html>