[ofa-general] ***SPAM*** Accessing RDMA Memory Locally

[Caitlin Bestler]

On Fri, Aug 15, 2008 at 7:11 AM, Dotan Barak <dotanba at gmail.com> wrote:

>>
>> Can I access that memory directly with pointers in the case of a node
>> accessing its own memory location?  Are local access RDMA operations
>> efficient?
>
> You can read the buffers (that remote nodes RDMA Write to) using
> memcpy/pointer access or any other way that you can use memory buffers in
> your process.
>
> After the RDMA Write ands and the data is placed in the remote buffer, one
> can treat the local buffer as IB wasn't used ....
>

After the appropriate completion is received the application can treat
the local buffer as though RDMA was not used. But the application
generally should not attempt to infer when the RDMA Write ends,
but rather rely on completions. Completions are simple. Figuring
out possible re-orderings, IB vs iWARP and device specific cache
interactions are tricky.

It's ultimately like have a co-processor. You can use the memory
as normal, but you should rely on the agreed upon handshake to
determine when it is safe to do so. For RDMA, those are the
completions.