[ofa-general] [RFC 0/1] libmthca: CQ/DMA race on Altix

[akepner at sgi.com]

On Tue, Jul 17, 2007 at 12:40:25AM +0300, Muli Ben-Yehuda wrote:
> On Mon, Jul 16, 2007 at 09:52:53AM -0700, Roland Dreier wrote:
> ....
> > The memory being dealt with here is buffers that are only used by
> > the device and userspace.  And the problem being solved is not
> > really that the memory needs to be coherent -- it is just that on
> > Altix, using coherent memory turns on another side effect that DMAs
> > to that memory flush other in-flight DMAs to other memory.
> > 
> > So there are several reasons I don't like using dma_alloc_coherent()
> > to allocate this memory, and then mapping it into userspace (rather
> > than having userspace allocate it and then map it to the device, as
> > these patches do):
> > 
> >  - dma_alloc_coherent() has to allocate kernel address space for
> >    memory, and in this case the kernel will never touch the memory.
> >    So this is pure waste, and on 32-bit system, these allocations
> >    could easily fail since kernel address space is scarce.
> 
> But isn't this an Altix specific issue, which makes the 32-bit issue
> moot? (I'm assuming the "fix" to use dma_alloc_coherent() is only
> implemented for Altix, which is in-arguably ugly).
> 

I believe Roland was referring to an alternate solution to the 
problem. One that I did before, and which didn't involve changing 
the dma_map_sg() prototype. (Instead it allocated memory with 
dma_alloc_coherent() and then mmap()-ed it into user space. (See:
http://lists.openfabrics.org/pipermail/general/2007-January/032218.html 
)


> >  - The property being asked for is not really coherent memory but
> >    rather "set the magic bit in the bus address so the Altix chipset
> >    flushes other DMAs", and I think it would be cleaner to ask for
> >    that explicitly rather than relying on the side effect of
> >    coherent memory.
> 
> That makes sense. However I didn't quite understand if the above means
> that you're ok with the patch posted, or prefer a different (third)
> approach?
> 

Another patchset is imminent. This one won't change the dma_map_sg() 
prototype. It'll pass extra flags (only for IA64_SGI_SN2) in the upper 
bits of the direction argument. Otherwise it's similar to what I posted 
at the start of this thread.

-- 
Arthur