[openib-general] basic IB doubt

[Talpey, Thomas]

At 03:39 AM 8/26/2006, Gleb Natapov wrote:
>On Fri, Aug 25, 2006 at 03:53:12PM -0400, Talpey, Thomas wrote:
>> Flush (sync for_device) before posting.
>> Invalidate (sync for_cpu) before processing.
>> 
>So, before touching the data that was RDMAed into the buffer application
>should cache invalidate the buffer, is this even possible from user
>space? (Not on x86, but it isn't needed there.)

Interesting you should mention that. :-) There isn't a user verb for
dma_sync, there's only deregister.

The kernel can perform this for receive completions, and signaled
RDMA Reads, but it can't do so for remote RDMA Writes. Only the
upper layer knows where those went.

There are two practical solutions:

1) (practical solution) user mappings must be fully consistent,
within the capability of the hardware. Still, don't go depending
on any specific ordering here.

2) user must deregister any mapping before inspecting the result. I
doubt any of them do this, for that reason anyway.

MO is that this will bite us in the a** some day. If anybody was
running this code on the Sparc architecture it already would have.

Tom.