[openib-general] [RFC/BUG] libibverbs: DMA vs. CQ race

[glebn at voltaire.com]

On Mon, Jan 29, 2007 at 01:49:04PM -0800, Roland Dreier wrote:
> Even with that resolved this all seems rather unfortunate to me.  I
> don't like the idea of having the kernel keep all these buffers around
> and then have the userspace library have to map the right buffer.  It
> leads to awkwardness like the fact that mthca_resize_cq() seems to be
> totally screwed if ibv_cmd_resize_cq() fails for some reason -- it
> already munmap'ed the original buffer, and it can't map the new
> buffer, and so the CQ is dead with no chance to recover.
I looked through ehca driver and it looks as it is doing exactly this
"keep all these buffers around and then have the userspace library have
to map the right buffer". ehca doesn't support resize_cq though, but
lest say this issue will be also resolved will this approach be
acceptable. This is how ehca works after all, so we are not inventing
something new here.

> 
> The really strange thing about this is that this Altix
> coherent/consistent memory really isn't about the memory itself, but
> about the relationship of that memory with DMA elsewhere -- as I
> understand the code, doing dma_alloc_coherent() returns normal memory
> with a special DMA address that tells the system to flush other DMAs
> before doing DMA to the coherent region.  Which isn't really what most
> people understand coherent memory to be, but it has the magic property
> of making most drivers work.
Yes. It seems Altix abuses dma_alloc_coherent() for this.

> 
> So I'd really like a better solution, but I don't have one in mind
> unfortunately.  Maybe we can all meditate on this and try to come up
> with something cleaner -- I really hope there is a better way to
> handle this.
>
Another approach may be to add another verbs (or we can make ibv_reg_mr
do this with special flag) for coherent memory allocation. This verb
will allocate coherent memory in the kernel and mmap it from a user space.
Than cq will be created as usual by providing lkey to the create_cq
verb. The resize will work exactly like it works now i.e allocate new cq
buffer call resize_cq with new buffer's lkey, copy cqes, unregister old buffer.

--
			Gleb.