[ofa-general] Memory registration redux

[Pavel Shamis (Pasha)]

Sounds good for me,

Jeff Squyres wrote:
> Other MPI implementors -- what do you think of this scheme?
>
>
> On May 27, 2009, at 1:49 PM, Roland Dreier (rdreier) wrote:
>
>>
>>  > > /*
>>  > >  * If type field is INVAL, then user_cookie_counter holds the
>>  > >  * user_cookie for the region being reported; if the HINT flag 
>> is set
>>  > >  * then hint_start/hint_end hold the start and end of the 
>> mapping that
>>  > >  * was invalidated.  (If HINT is not set, then multiple events
>>  > >  * invalidated parts of the registered range and 
>> hint_start/hint_end
>>  > >  * should be ignored)
>>
>>  > I don't quite grok this.  Is the intent that HINT will only be set if
>>  > an *entire* hint_start/hint_end range is invalidated by a single
>>  > event?  I.e., if only part of the hint_start/hint_end range is
>>  > invalidated, you'll get the cookie back, but not what part of the
>>  > range is invalid (because assumedly the entire IBV registration is 
>> now
>>  > invalid anyway)?
>>
>> Basically, I just keep one hint_start/hint_end.  If multiple events hit
>> the same registration then I just give up and don't give you a hint.
>>
>>  > >  * If type is LAST, then the read operation has emptied the list of
>>  > >  * invalidated regions, and user_cookie_counter holds the value 
>> of the
>>  > >  * kernel's generation counter when the empty list occurred.  The
>>  > >  * other fields are not filled in for this event.
>>
>>  > Just to be clear -- we're supposed to keep reading events until we 
>> get
>>  > a LAST event?
>>
>> Yes, that's probably the sanest use case.
>>
>>  > 1. Will it increase by 1 each time a page (or set of pages?) is
>>  > removed from a user process?
>>
>> As it stands it increases by 1 every time there is an MMU notification,
>> even if that notification hits multiple registrations.  It wouldn't be
>> hard to change that to count the number of events generated if that
>> works better.
>>
>>  > 2. Does it change if pages are *added* to a user process?  I.e., does
>>  > the counter indicate *removals* or *changes* to the user process page
>>  > table?
>>
>> No, additions don't trigger any MMU notification -- that's inherent in
>> the design of the MMU notifiers stuff.  The idea is that you have a
>> "secondary MMU" and MMU notifications are the equivalent of TLB
>> shootdowns; the secondary MMU is responsible for populating itself on
>> faults etc.
>>
>>  > Is the *unm_counter value guaranteed to have been changed by the time
>>  > munmap() returns?
>>
>> Yes.
>>
>>  > Did you pick [2] here simply because you're only expecting an INVAL
>>  > and a LAST event in this specific example?  I'm assuming that we
>>  > should normally loop over reading until we get LAST, correct?
>>
>> Right.
>>
>>  > What happens if I register multiple regions with the same cookie 
>> value?
>>
>> You get in trouble -- I need to fix things to reject duplicated cookies
>> actually, because otherwise there's no way to unregister.
>>
>>  > Is a process responsible for guaranteeing that it umn_unregister()s
>>  > everything before exiting, or will all pending registrations be
>>  > cleaned up/unregistered/whatever when a process exits?
>>
>> The kernel cleans up of course to handle crashes etc.
>>
>>  - R.
>>
>
>