[openib-general] basic IB doubt

[Caitlin Bestler]

openib-general-bounces at openib.org wrote:
> At 03:39 AM 8/26/2006, Gleb Natapov wrote:
>> On Fri, Aug 25, 2006 at 03:53:12PM -0400, Talpey, Thomas wrote:
>>> Flush (sync for_device) before posting.
>>> Invalidate (sync for_cpu) before processing.
>>> 
>> So, before touching the data that was RDMAed into the buffer
>> application should cache invalidate the buffer, is this even possible
>> from user space? (Not on x86, but it isn't needed there.)
> 
> Interesting you should mention that. :-) There isn't a user
> verb for dma_sync, there's only deregister.
> 
> The kernel can perform this for receive completions, and
> signaled RDMA Reads, but it can't do so for remote RDMA
> Writes. Only the upper layer knows where those went.
> 
> There are two practical solutions:
> 
> 1) (practical solution) user mappings must be fully
> consistent, within the capability of the hardware. Still,
> don't go depending on any specific ordering here.
> 
> 2) user must deregister any mapping before inspecting the
> result. I doubt any of them do this, for that reason anyway.
> 
> MO is that this will bite us in the a** some day. If anybody
> was running this code on the Sparc architecture it already would have.
> 

The consensus I have seen in other forums is that the RDMA device
is expected to be at least as coherent as another CPU at the point
when a completion is delivered.

That is, it is reasonable to expect an application to treat memory
updated by the RDMA device just as it would treat memory updated
by another processor under platform appropriate rules. But it is
unreasonable to expect that application to do *more* to access
the memory, other than actually waiting for a completion.
DAPL, IT-API and RNIC-PI all include support for an exception,
which was totally related to a specific bus architecture that
is no longer sold.