[ofa-general] [RFC 0/1] libmthca: CQ/DMA race on Altix

[Roland Dreier]

 > This will be very painful and frankly I don't think the pain is
 > justified. Can't you confine the changes to the IB layerr so that the
 > mapping happens through dma_alloc_coherent if you need
 > coherent/consistent memory rather than through dma_map_sg?

The memory being dealt with here is buffers that are only used by the
device and userspace.  And the problem being solved is not really that
the memory needs to be coherent -- it is just that on Altix, using
coherent memory turns on another side effect that DMAs to that memory
flush other in-flight DMAs to other memory.

So there are several reasons I don't like using dma_alloc_coherent()
to allocate this memory, and then mapping it into userspace (rather
than having userspace allocate it and then map it to the device, as
these patches do):

 - dma_alloc_coherent() has to allocate kernel address space for
   memory, and in this case the kernel will never touch the memory.
   So this is pure waste, and on 32-bit system, these allocations
   could easily fail since kernel address space is scarce.

 - The property being asked for is not really coherent memory but
   rather "set the magic bit in the bus address so the Altix chipset
   flushes other DMAs", and I think it would be cleaner to ask for
   that explicitly rather than relying on the side effect of coherent
   memory.

 > Also, this kind of thing should definitely be CC'd to lkml.

I agree on that.

 - R.