[openib-general] Getting rid of pinned memory requirement

[Troy Benjegerdes]

The current InfiniBand model of using 'mlock()' to maintain a constant virtual
to physical mapping for registered memory pages is not going to work with NUMA
page migration and memory hotplug.

I want to get some discussion started on this list, and once we have an idea
what's feasable from the infiniband side, to bring up the discussion on
linux-kernel, and get the memory hotplug and numa page migration people
involved as well.

I think the following list covers the major points. Are there any big
"gotcha's" involved? 

* Add "registered" flag to linux/mm.h (VM_REGISTERED 0x01000000)

* Need to define a 'registered memory' api. Maybe using 'madvise()' ?

* Kernel needs to be able to unpin registered memory and shoot down cached
  mappings in network cards (treat IB/Iwarp cards like a TLB)

* Requires IB/Iwarp card to dispatch an interrupt on a mapping 'miss'

* This model allows applications to register more memory than physically
  exists, and the kernel manages what is actually pinned.

* Requires adding hooks in MM code to dispatch driver mapping shootdowns. (A
  per-VM area list of adapters to be notified for the mapping?) 


I know that having the card dispatch an interrupt on an incoming packet that's
not mapped is outside the spec. The alternative is that if the kernel wants to
move some memory around that's registered, it's got to have some way to either
kill the application, or tear down and re-establish all the QP's. I suppose
an alternative would be a "SIG_I_KILLED_YOUR_MAPPINGS" type signal to tell
the application (or library) that it needs to re-establish all it's pinned
memory might work.