[ofa-general] Memory registration redux

[Jason Gunthorpe]

On Mon, May 11, 2009 at 02:23:58PM -0700, Caitlin Bestler wrote:
> On Thu, May 7, 2009 at 3:48 PM, Jason Gunthorpe
> <jgunthorpe at obsidianresearch.com> wrote:
> >
> > Right, I was only thinking of a new driver call that was along the
> > lines of update_mr_pages() that just updates the HCA's mapping with
> > new page table entires atomically. It really would be device
> > specific. If there is no call available then unregister/register +
> > printk log is a fair generic implementation.
> >
> > To be clear, what I'm thinking is that this would only be invoked if
> 
> Both the IBTA and RDMAC verbs were defined so that the meaning of
> L-Key/R-Key/STag + Address could not instantly change from "X" to
> "Y", only from "X" to NULL and then NULL to "Y".

Well, this is sort of a grey area, in one sense the meaning isn't
changing, just the underlying phyiscal memory is being moved around by
the OS.

The notion that the verbs refer to some sort of invisible underlying
VM object is nice for an implementation but pretty useless for
MPI..

> There are a lot of good reasons for this, especially for R-Keys or
> remotely accessible STags. It ensures that all operations that
> started when the translation was "X" are completed before any that
> will use the "Y" translation can commence. That is not something we
> want to accidentally undermine.

I'm not sure I see how this helps, synchronizing all this is the
responsibility of the application, if it wants to change the mapping
then it should be able to, and if it does so with poor timing then it
will have races and loose data <shrug>. As it stands today there are
already races where apps can loose data transfered after an unmap() or
transfer the wrong data after a mmap() so the current model is already
broken from that perspective.

Of course an update verb has to operate with similar ordering
guarantees to regsiter/unregister relative to the local work request
queue - that is to say if the verb is done out-of-line with the WR
queue then it must wait for the queue to flush before issuing the
update to the HCA - just like unregister - and then wait for the verb
to complete before returning to the app - just like register.

And we all wish for userspace FRMRs...

Jason