[openib-general] [RFC/BUG] libibverbs: DMA vs. CQ race

[glebn at voltaire.com]

On Wed, Jan 10, 2007 at 08:19:58AM -0800, Roland Dreier wrote:
>  > OK, but as far as I know (and I may be wrong here) low level driver
>  > doesn't have device node, but libibverbs has one, we don't want to
>  > create device node for mthca just to provide mmap() to userspace, or do
>  > we?
> 
> Not sure what you mean -- mthca already implements mmap() to provide
> access to doorbell pages.  It would be no problem to extend that to
> allow allocating QPs and CQs via mmap() if we're forced into that.
Right, haven't looked at the code for a long time :). So how can we tell
to mmap that we want to allocate coherent memory and not map doorbell?
Magic offset value?
> 
>  > There is no way to make memory coherent after allocation, it have to be
>  > allocated coherent, I think the API doesn't exists because it cannot be
>  > implemented on all arches, but it is worth asking mm people.
> 
> Not sure what you mean here either.  dma_alloc_coherent() already
> exists on all architectures, but that's not the end of the story -- if
> we remap the memory into userspace then we have to make sure that the
> page tables for userspace also make the memory coherent (eg uncached
> on non-cache-coherent architectures, or whatever the altix magic is to
> avoid ordering problems).  dma_alloc_coherent() only takes care of the
> kernel mapping.
> 
If I understand correctly (and if not SGI people please correct me) it
doesn't matter how memory is mapped into userspace, what is important
what address is passed to HCA to use for DMAing data into host memory.
It just so happens that dma_alloc_coherent() returns this special
address that causes flush of all outstanding writes in dma_handle.

So what we should do is allocate memory with dma_alloc_coherent() in
mthca mmap() function and save dma_handle in database indexed by
vma->vm_start. Remap pointer returned by the function into userspace.
When create_{qp,cq} is called later on, we should lookup corespondent
dma_handle using address provided by userspace (if not found - return an
error) and pass this address to HCA. Is this make sense? 
--
			Gleb.