[openib-general] [RFC/BUG] libibverbs: DMA vs. CQ race

[akepner at sgi.com]

On Wed, 13 Dec 2006, Roland Dreier wrote:

> Are there other possible ordering problems involving user memory (not
> in a CQ or QP)?  Something like a CPU on node A writing to memory on
> node B and then posting a work request that makes the HCA DMA from
> that memory on node B, and having the work request doorbell reach the
> HCA before the write to node B actually happens, so the HCA DMAs the
> old contents of node B's memory?

Well, this case could be handled with mb() operations (if
I understand you correctly). The type of race I had in mind
is between DMA operations and updates to data structures
shared between the host and HCA. But, yes, the example I
used was only one of the possiblilities of this type of race.

>
> I guess the only feasible solution to the problem you're pointing out
> is to have libmthca use some special mmap()-based allocator for queues
> so that the kernel can give it memory that has the special
> dma_map_consistent treatment.

That's an excellent idea. (And, now that you've mentioned it,
it's "obvious" ;-) I'll see what I can come up with using
this approach.

>
> Ugh.
>

Well stated.

-- 
Arthur