[openib-general] basic IB doubt

[glebn at voltaire.com]

On Mon, Aug 28, 2006 at 08:24:26AM -0400, Talpey, Thomas wrote:
> At 03:39 AM 8/26/2006, Gleb Natapov wrote:
> >On Fri, Aug 25, 2006 at 03:53:12PM -0400, Talpey, Thomas wrote:
> >> Flush (sync for_device) before posting.
> >> Invalidate (sync for_cpu) before processing.
> >> 
> >So, before touching the data that was RDMAed into the buffer application
> >should cache invalidate the buffer, is this even possible from user
> >space? (Not on x86, but it isn't needed there.)
> 
> Interesting you should mention that. :-) There isn't a user verb for
> dma_sync, there's only deregister.
> 
> The kernel can perform this for receive completions, and signaled
> RDMA Reads, but it can't do so for remote RDMA Writes. Only the
> upper layer knows where those went.
> 
I think that kernel gets interrupt about completion only if notification
is requested otherwise cqe directly placed in CQ buffer. By the way CQ
buffer also has to be cache-invalidated before each cq_poll.

> There are two practical solutions:
> 
> 1) (practical solution) user mappings must be fully consistent,
> within the capability of the hardware. Still, don't go depending
> on any specific ordering here.
Memory registration should work on any memory allocated for user
process. Can we change memory to be consistent after it was allocated?
Based on linux DMA api - no, because we only have
dma_alloc/free_consistent() and not something that gets existing pointer
as a parameter.

> 
> 2) user must deregister any mapping before inspecting the result. I
> doubt any of them do this, for that reason anyway.
> 
This may have big performance impact.

> MO is that this will bite us in the a** some day. If anybody was
> running this code on the Sparc architecture it already would have.
> 
AFAIK SUN runs MPI over UDAPL, but they have their own IB
implementation, so may be they handle all coherency issues in the UDAPL
itself.

--
			Gleb.