[openib-general] basic IB doubt

[Jason Gunthorpe]

On Sun, Aug 27, 2006 at 03:30:56PM -0700, Roland Dreier wrote:
>     glebn> So, before touching the data that was RDMAed into the
>     glebn> buffer application should cache invalidate the buffer, is
>     glebn> this even possible from user space? (Not on x86, but it
>     glebn> isn't needed there.)
 
> Yes, on any architecture that is not cache-coherent with PCI DMA, some
> cache invalidation/flushing will be necessary.  And this probably
> won't be possible from userspace if the cache is physically tagged.
> (Are there any such architectures in real use, ie non-coherent with
> PCI and physically tagged cache?)

It depends on the arch if it is a problem or not.. Ie PPC Book-E
has 'dcba' which is available from user space. It operates on virtual
addresses and is a flush and invalidate combined. So it is safe,
but less effecient than the pure invalidate that the kernel has access
to. 

So long as cache ops that works on virtual addresses are present
it should be fine from userspace, but in some cases the necessary
sequence of cache ops can be quite elaborate and hardware dependent,
so a syscall, or at least a vdso function would be needed to support
eveything.

In my experience most real architectures that have this problem these
days are embedded targetted lower performance processors. If you are
in the embedded space and using IB hardware then presumably you care
about performance and will avoid such things. (Although long ago, this
wasn't a choice and I actually have built an embedded IB capable
system with non-coherent PCI.. It is a big pain, I don't recommend it.)

Jason