[openib-general] [RFC/BUG] libibverbs: DMA vs. CQ race

[akepner at sgi.com]

Thanks for having a look at this.

On Mon, 29 Jan 2007, Roland Dreier wrote:

> ....
> Well, first the changes to the userspace libmthca need to be such that
> new libmthca continues to work with old kernels....

Absolutely.

> .....
> The really strange thing about this is that this Altix
> coherent/consistent memory really isn't about the memory itself, but
> about the relationship of that memory with DMA elsewhere -- as I
> understand the code, doing dma_alloc_coherent() returns normal memory
> with a special DMA address that tells the system to flush other DMAs
> before doing DMA to the coherent region.  Which isn't really what most
> people understand coherent memory to be, but it has the magic property
> of making most drivers work.
> ....

I agree that this isn't a very elegant solution, but I don't
know of a better one.

Assuming that something along the lines of the previous patch
is used, we need to address userspace/kernel compatibility.

The existing abi versioning doesn't seem to be exactly what
we want to use, though, because we want to change a verb's
semantics to work around a bug. (Changing the abi_version
may be an inevitable result, though.)

How about adding "semantic flags" to the mthca_* commands
(mthca_create_cq, etc.)? Userspace could read the contents of
a new sysfs file which, if found, would indicate the flags
that the kernel understands. Then it could pass the flags, if
it chooses, to get the kernel to use the desired semantics.

Something like:

# cat /sys/class/infiniband_verbs/uverbs0/abi_flags
0000000000000001 [64 bits of flags]

where:

enum abi_flags {
         COHERENT_USER_CQ        = (1<<0),
         .....
};

Better/different ideas?

-- 
Arthur