[ofa-general] New proposal for memory management

[Barrett, Brian W]

On 4/30/09 8:35 , "Pavel Shamis (Pasha)" <pashash at gmail.com> wrote:

> Barrett, Brian W wrote:
>> Jeff and I talked for a while today, and we're pretty sure that as long as
>> the byte set by the kernel notifier is written before the pages are returned
>> into the unallocated list, there isn't actually a race condition.  It does
>> mean that every time the page cache is searched, we also have to check the
>> byte (and likely take a cache miss), but that's not too evil.
>> 
>> However, there's still then the problem with the notifier concept of how the
>> kernel passes which pages were given back to the kernel.  It has to pass a
>> (potentially very large) amount of data back to the user, so the memory
>> ownership issues with kernel/user space are interesting.  It also has to
>> somewhat atomically prepare the list and undset the notifier byte, which is
>> also problematic.  But probably workable.
>>  
> It sounds like we will have another 5k lines of code in MPI that will
> try to resolve
> the kernel/user notification issue :-)
> IMHO, Lets avoid all these tricks and move the registration cache to kernel.

I don't disagree - this is a problem best solved in kernel space, preferably
using the approach Jeff originally proposed.  I think the complexity of
handling notifier callback will be fairly high, but it is still an
improvement over where we are today.  Today's user-space memory hooks need
to go - they cause too many problems for both the MPI and the application.

Brian

--
   Brian W. Barrett
   Dept. 1423: Scalable System Software
   Sandia National Laboratories