[ofa-general] Memory registration redux

[Roland Dreier]

 > Well, this conceptually doesn't seem hard. Go through all the pages in
 > the MR, if any have changed then pin the new page and replace the
 > pages physical address in the HCA's page table. Once done, synchronize
 > with the hardware, then run through again and un-pin and release all
 > the replaced pages.
 > 
 > Every HCA must have the necessary primitives for this to support
 > register and unregister...

No... every HCA just needs to support register and unregister.  It
doesn't have to support changing the mapping without full unregister and
reregister.

Also this requires potentially walking the page tables of the entire
process, checking to see if any mappings have changed.  We really want
to keep the information that the MMU notifiers give us, namely which
virtual address range is changing.

 > The mmu notifiers can optionally make note of the affected pages
 > during the callback to reduce the workload of the syscall.

This requires an unbounded amount of events to be queued up in the
kernel, naively.  (If we lose some events then we have to go back to the
full page table scan, which I don't think is feasible)

 > Only part I don't immediately see is how to trap creation of new VM
 > (ie mmap), mmu notifiers seem focused on invalidating, ie munmap()..

Why do we care?  The initial faulting in of mappings occurs when an MR
is created.

 - R.