[openib-general] basic IB doubt

[Caitlin Bestler]

openib-general-bounces at openib.org wrote:
>>    Thomas> How does an adapter guarantee that no bridges or other
>>    Thomas> intervening devices reorder their writes, or for that
>>    Thomas> matter flush them to memory at all!?
>> 
>> That's a good point.  The HCA would have to do a read to flush the
>> posted writes, and I'm sure it's not doing that (since it would add
>> horrible latency for no good reason).
>> 
>> I guess it's not safe to rely on ordering of RDMA writes after all.
> 
> Couldn't the same point then be made that a CQ entry may come
> before the data has been posted?
> 

That's why both specs (IBTA and RDMAC) are very explicit that all
prior messages are complete before the CQE is given to the user.

It is up to the RDMA Device and/or its driver to guarantee this
by whatever means are appropriate. An implementation that allows
a CQE post to pass the data placement that it is reporting on the
PCI bus is in error.

The critical concept of the Work Completion is that it consolidates
guarantees and notificatins. The implementation can do all sorts
of strange things that it thinks optimize *before* the work completion,
but at the time the work completion is delivered to the user everything
is supposed to be as expected.