[ofa-general] [RFC 0/1] libmthca: CQ/DMA race on Altix

[Muli Ben-Yehuda]

On Mon, Jul 16, 2007 at 09:52:53AM -0700, Roland Dreier wrote:

>  > This will be very painful and frankly I don't think the pain is
>  > justified. Can't you confine the changes to the IB layerr so that
>  > the mapping happens through dma_alloc_coherent if you need
>  > coherent/consistent memory rather than through dma_map_sg?
> 
> The memory being dealt with here is buffers that are only used by
> the device and userspace.  And the problem being solved is not
> really that the memory needs to be coherent -- it is just that on
> Altix, using coherent memory turns on another side effect that DMAs
> to that memory flush other in-flight DMAs to other memory.
> 
> So there are several reasons I don't like using dma_alloc_coherent()
> to allocate this memory, and then mapping it into userspace (rather
> than having userspace allocate it and then map it to the device, as
> these patches do):
> 
>  - dma_alloc_coherent() has to allocate kernel address space for
>    memory, and in this case the kernel will never touch the memory.
>    So this is pure waste, and on 32-bit system, these allocations
>    could easily fail since kernel address space is scarce.

But isn't this an Altix specific issue, which makes the 32-bit issue
moot? (I'm assuming the "fix" to use dma_alloc_coherent() is only
implemented for Altix, which is in-arguably ugly).

>  - The property being asked for is not really coherent memory but
>    rather "set the magic bit in the bus address so the Altix chipset
>    flushes other DMAs", and I think it would be cleaner to ask for
>    that explicitly rather than relying on the side effect of
>    coherent memory.

That makes sense. However I didn't quite understand if the above means
that you're ok with the patch posted, or prefer a different (third)
approach?

Cheers,
Muli