[ofa-general] Memory registration redux

[Supalov, Alexander]

Hi,

Sorry, it's kind of difficult to deduce looking at this Q&A sequence what works how and when. Is it possible to create a brief and direct description of the proposed solution?

Best regards.

Alexander

-----Original Message-----
From: Jeff Squyres [mailto:jsquyres at cisco.com] 
Sent: Wednesday, May 27, 2009 9:03 PM
To: Roland Dreier (rdreier)
Cc: Pavel Shamis; Hans Westgaard Ry; Dontje; Lenny Verkhovsky; H??kon Bugge; Donald Kerr; OpenFabrics General; Supalov, Alexander
Subject: Re: [ofa-general] Memory registration redux

Other MPI implementors -- what do you think of this scheme?


On May 27, 2009, at 1:49 PM, Roland Dreier (rdreier) wrote:

>
>  > > /*
>  > >  * If type field is INVAL, then user_cookie_counter holds the
>  > >  * user_cookie for the region being reported; if the HINT flag  
> is set
>  > >  * then hint_start/hint_end hold the start and end of the  
> mapping that
>  > >  * was invalidated.  (If HINT is not set, then multiple events
>  > >  * invalidated parts of the registered range and hint_start/ 
> hint_end
>  > >  * should be ignored)
>
>  > I don't quite grok this.  Is the intent that HINT will only be  
> set if
>  > an *entire* hint_start/hint_end range is invalidated by a single
>  > event?  I.e., if only part of the hint_start/hint_end range is
>  > invalidated, you'll get the cookie back, but not what part of the
>  > range is invalid (because assumedly the entire IBV registration  
> is now
>  > invalid anyway)?
>
> Basically, I just keep one hint_start/hint_end.  If multiple events  
> hit
> the same registration then I just give up and don't give you a hint.
>
>  > >  * If type is LAST, then the read operation has emptied the  
> list of
>  > >  * invalidated regions, and user_cookie_counter holds the value  
> of the
>  > >  * kernel's generation counter when the empty list occurred.  The
>  > >  * other fields are not filled in for this event.
>
>  > Just to be clear -- we're supposed to keep reading events until  
> we get
>  > a LAST event?
>
> Yes, that's probably the sanest use case.
>
>  > 1. Will it increase by 1 each time a page (or set of pages?) is
>  > removed from a user process?
>
> As it stands it increases by 1 every time there is an MMU  
> notification,
> even if that notification hits multiple registrations.  It wouldn't be
> hard to change that to count the number of events generated if that
> works better.
>
>  > 2. Does it change if pages are *added* to a user process?  I.e.,  
> does
>  > the counter indicate *removals* or *changes* to the user process  
> page
>  > table?
>
> No, additions don't trigger any MMU notification -- that's inherent in
> the design of the MMU notifiers stuff.  The idea is that you have a
> "secondary MMU" and MMU notifications are the equivalent of TLB
> shootdowns; the secondary MMU is responsible for populating itself on
> faults etc.
>
>  > Is the *unm_counter value guaranteed to have been changed by the  
> time
>  > munmap() returns?
>
> Yes.
>
>  > Did you pick [2] here simply because you're only expecting an INVAL
>  > and a LAST event in this specific example?  I'm assuming that we
>  > should normally loop over reading until we get LAST, correct?
>
> Right.
>
>  > What happens if I register multiple regions with the same cookie  
> value?
>
> You get in trouble -- I need to fix things to reject duplicated  
> cookies
> actually, because otherwise there's no way to unregister.
>
>  > Is a process responsible for guaranteeing that it umn_unregister()s
>  > everything before exiting, or will all pending registrations be
>  > cleaned up/unregistered/whatever when a process exits?
>
> The kernel cleans up of course to handle crashes etc.
>
>  - R.
>


-- 
Jeff Squyres
Cisco Systems

---------------------------------------------------------------------
Intel GmbH
Dornacher Strasse 1
85622 Feldkirchen/Muenchen Germany
Sitz der Gesellschaft: Feldkirchen bei Muenchen
Geschaeftsfuehrer: Douglas Lusk, Peter Gleissner, Hannes Schwaderer
Registergericht: Muenchen HRB 47456 Ust.-IdNr.
VAT Registration No.: DE129385895
Citibank Frankfurt (BLZ 502 109 00) 600119052

This e-mail and any attachments may contain confidential material for
the sole use of the intended recipient(s). Any review or distribution
by others is strictly prohibited. If you are not the intended
recipient, please contact the sender and delete all copies.