[openib-general] [RFC/BUG] libibverbs: DMA vs. CQ race

[glebn at voltaire.com]

Hi Roland,

On Sun, Dec 17, 2006 at 05:42:41PM +0200, glebn at voltaire.com wrote:
> On Wed, Dec 13, 2006 at 11:41:41PM -0800, Roland Dreier wrote:
> > Are there other possible ordering problems involving user memory (not
> > in a CQ or QP)?  Something like a CPU on node A writing to memory on
> > node B and then posting a work request that makes the HCA DMA from
> > that memory on node B, and having the work request doorbell reach the
> > HCA before the write to node B actually happens, so the HCA DMAs the
> > old contents of node B's memory?
> > 
> > I guess the only feasible solution to the problem you're pointing out
> > is to have libmthca use some special mmap()-based allocator for queues
> > so that the kernel can give it memory that has the special
> > dma_map_consistent treatment.
> Do you think this should be part of mthca or some general framework like uio
> which allows writing driver in userspace?
> Also another solution could be to do something similar to ehca. It
> allocates QP and CQ in the kernel and maps them into process address
> space.
> 
Can we get back to this problem? I understand that the way ehca does
things is not acceptable, so is seems that ehca people will also have
to rethink how CQ/QP memory is allocated. It would be a good idea to
consolidate solution for mthca and ehca. mmap() special allocator can be 
added to fd libibverbs uses for communication with the kernel and it can
be used by all low level drivers (I don't know if such a way is
acceptable for echa). The question is if this is a good idea to add
something as general as coherent memory allocation for userspace (which
is needed by other userspace drivers) to infiniband subsystem.

What do you think?

--
			Gleb.