[openib-general] [RFC/BUG] libibverbs: DMA vs. CQ race

[glebn at voltaire.com]

On Wed, Jan 10, 2007 at 08:04:55AM -0800, Roland Dreier wrote:
>  > Can we get back to this problem? I understand that the way ehca does
>  > things is not acceptable, so is seems that ehca people will also have
>  > to rethink how CQ/QP memory is allocated. It would be a good idea to
>  > consolidate solution for mthca and ehca. mmap() special allocator can be 
>  > added to fd libibverbs uses for communication with the kernel and it can
>  > be used by all low level drivers (I don't know if such a way is
>  > acceptable for echa). The question is if this is a good idea to add
>  > something as general as coherent memory allocation for userspace (which
>  > is needed by other userspace drivers) to infiniband subsystem.
>  > 
>  > What do you think?
> 
> I think that's kind of overengineering things.  Allocating and mapping
> memory in an mmap method is just a few lines of code so we might as
> well let drivers do exactly what they want.
OK, but as far as I know (and I may be wrong here) low level driver
doesn't have device node, but libibverbs has one, we don't want to
create device node for mthca just to provide mmap() to userspace, or do
we?

> 
> I think the bigger problem is fixing the kernel so it's easier for
> drivers to specify the page attributes they want for memory mapped to
> userspace (eg uncached, magic altix "OK for DMA", write combining, etc)
> 
There is no way to make memory coherent after allocation, it have to be
allocated coherent, I think the API doesn't exists because it cannot be
implemented on all arches, but it is worth asking mm people.

--
			Gleb.