[openib-general] basic IB doubt

[Ralph Campbell]

On Wed, 2006-08-23 at 09:47 -0700, Caitlin Bestler wrote:
> openib-general-bounces at openib.org wrote:
> > Quoting r. john t <johnt1johnt2 at gmail.com>:
> >> Subject: basic IB doubt
> >> 
> >> Hi
> >> 
> >> I have a very basic doubt. Suppose Host A is doing RDMA write (say 8
> >> MB) to Host B. When data is copied into Host B's local
> > buffer, is it guaranteed that data will be copied starting
> > from the first location (first buffer address) to the last
> > location (last buffer address)? or it could be in any order?
> > 
> > Once B gets a completion (e.g. of a subsequent send), data in
> > its buffer matches that of A, byte for byte.
> 
> An excellent and concise answer. That is exactly what the application
> should rely upon, and nothing else. With iWARP this is very explicit,
> because portions of the message not only MAY be placed out of 
> order, they SHOULD be when packets have been re-ordered by the
> network. But for *any* RDMA adapter there is no guarantee on
> what order the adapter flushes things to host memory or particularly
> when old contents that may be cached are invalidated or updated.
> The role of the completion is to limit the frequency with which
> the RDMA adapter MUST guarantee coherency with application visible
> buffers. The completion not only indicates that the entire message
> was received, but that it has been entirely delivered to host memory.

Actually, A knows the data is in B's memory when A gets the completion
notice.  B can't rely on anything unless A uses the RDMA write with
immediate which puts a completion event in B's CQ.
Most applications on B ignore this requirement and test for the last
memory location being modified which usually works but doesn't
guarantee that all the data is in memory.