[ofa-general] Memory registration redux

[Jeff Squyres]

Other MPI implementors -- what do you think of this scheme?


On May 27, 2009, at 1:49 PM, Roland Dreier (rdreier) wrote:

>
>  > > /*
>  > >  * If type field is INVAL, then user_cookie_counter holds the
>  > >  * user_cookie for the region being reported; if the HINT flag  
> is set
>  > >  * then hint_start/hint_end hold the start and end of the  
> mapping that
>  > >  * was invalidated.  (If HINT is not set, then multiple events
>  > >  * invalidated parts of the registered range and hint_start/ 
> hint_end
>  > >  * should be ignored)
>
>  > I don't quite grok this.  Is the intent that HINT will only be  
> set if
>  > an *entire* hint_start/hint_end range is invalidated by a single
>  > event?  I.e., if only part of the hint_start/hint_end range is
>  > invalidated, you'll get the cookie back, but not what part of the
>  > range is invalid (because assumedly the entire IBV registration  
> is now
>  > invalid anyway)?
>
> Basically, I just keep one hint_start/hint_end.  If multiple events  
> hit
> the same registration then I just give up and don't give you a hint.
>
>  > >  * If type is LAST, then the read operation has emptied the  
> list of
>  > >  * invalidated regions, and user_cookie_counter holds the value  
> of the
>  > >  * kernel's generation counter when the empty list occurred.  The
>  > >  * other fields are not filled in for this event.
>
>  > Just to be clear -- we're supposed to keep reading events until  
> we get
>  > a LAST event?
>
> Yes, that's probably the sanest use case.
>
>  > 1. Will it increase by 1 each time a page (or set of pages?) is
>  > removed from a user process?
>
> As it stands it increases by 1 every time there is an MMU  
> notification,
> even if that notification hits multiple registrations.  It wouldn't be
> hard to change that to count the number of events generated if that
> works better.
>
>  > 2. Does it change if pages are *added* to a user process?  I.e.,  
> does
>  > the counter indicate *removals* or *changes* to the user process  
> page
>  > table?
>
> No, additions don't trigger any MMU notification -- that's inherent in
> the design of the MMU notifiers stuff.  The idea is that you have a
> "secondary MMU" and MMU notifications are the equivalent of TLB
> shootdowns; the secondary MMU is responsible for populating itself on
> faults etc.
>
>  > Is the *unm_counter value guaranteed to have been changed by the  
> time
>  > munmap() returns?
>
> Yes.
>
>  > Did you pick [2] here simply because you're only expecting an INVAL
>  > and a LAST event in this specific example?  I'm assuming that we
>  > should normally loop over reading until we get LAST, correct?
>
> Right.
>
>  > What happens if I register multiple regions with the same cookie  
> value?
>
> You get in trouble -- I need to fix things to reject duplicated  
> cookies
> actually, because otherwise there's no way to unregister.
>
>  > Is a process responsible for guaranteeing that it umn_unregister()s
>  > everything before exiting, or will all pending registrations be
>  > cleaned up/unregistered/whatever when a process exits?
>
> The kernel cleans up of course to handle crashes etc.
>
>  - R.
>


-- 
Jeff Squyres
Cisco Systems