[ofa-general] Re: New proposal for memory management

[Jason Gunthorpe]

> The problem is that MPI needs to be aware of the application doing
> the free() and unregister or flush its MR cache for that virtual
> address range. Of course it would be difficult for OpenMPI to have
> callbacks or hooks into every way memory could be allocated/freed
> that an application might use.

There are only three calls that affect the way VM memory maps to
physical and thus would invalidate the mr cache: mmap, munmap and brk.

Specifically what must be happening is the app registers memory, calls
munmap on it, then gets the same VA back from mmap and the kernel
level mr is still pointing to the original mmap:

 foo = mmap(...);
 ibv_reg_mr(mr,foo)
 munmap(foo..)
 mmap(...) == foo; // By chance due to VA randomization
 // Ooops, mr no longer matches proc/self/maps

Actually, maybe that is the simple answer here - have the kernel fixup
the mr before returning from the 2nd mmap. Then the cache in user
space is still correct to assume that VA XX is registered and working.

Removing entries from the registration cache would have to be done in
some other way (age?).

Jason