[ofa-general] Memory registration redux

[Jeff Squyres]

On May 26, 2009, at 7:13 PM, Roland Dreier (rdreier) wrote:

> /*
>  * If type field is INVAL, then user_cookie_counter holds the
>  * user_cookie for the region being reported; if the HINT flag is set
>  * then hint_start/hint_end hold the start and end of the mapping that
>  * was invalidated.  (If HINT is not set, then multiple events
>  * invalidated parts of the registered range and hint_start/hint_end
>  * should be ignored)
>

I don't quite grok this.  Is the intent that HINT will only be set if  
an *entire* hint_start/hint_end range is invalidated by a single  
event?  I.e., if only part of the hint_start/hint_end range is  
invalidated, you'll get the cookie back, but not what part of the  
range is invalid (because assumedly the entire IBV registration is now  
invalid anyway)?

>  * If type is LAST, then the read operation has emptied the list of
>  * invalidated regions, and user_cookie_counter holds the value of the
>  * kernel's generation counter when the empty list occurred.  The
>  * other fields are not filled in for this event.
>

Just to be clear -- we're supposed to keep reading events until we get  
a LAST event?

>         if (*umn_counter != 0) {
>                 fprintf(stderr, "counter = %lld (expected 0)\n",  
> *umn_counter);
>                 return 1;
>         }
>

Some clarification questions about umn_counter:

1. Will it increase by 1 each time a page (or set of pages?) is  
removed from a user process?

2. Does it change if pages are *added* to a user process?  I.e., does  
the counter indicate *removals* or *changes* to the user process page  
table?

>         t = mmap(NULL, 3 * page_size, PROT_READ,
>                  MAP_PRIVATE | MAP_ANONYMOUS | MAP_POPULATE, -1, 0);
>
>         if (umn_register(t, 3 * page_size, 123))
>                 return 1;
>
>         munmap(t + page_size, page_size);
>
>         printf("ummunot events: %lld\n", *umn_counter);
>
>         if (*umn_counter > 0) {
>

Is the *unm_counter value guaranteed to have been changed by the time  
munmap() returns?

>                 struct ummunot_event ev[2];
>

Did you pick [2] here simply because you're only expecting an INVAL  
and a LAST event in this specific example?  I'm assuming that we  
should normally loop over reading until we get LAST, correct?

>
>                 int len;
>                 int i;
>
>                 len = read(umn_fd, &ev, sizeof ev);
>                 printf("read %d events (%d tot)\n", len / sizeof  
> ev[0], len);
>
>                 for (i = 0; i < len / sizeof ev[0]; ++i) {
>                         switch (ev[i].type) {
>                         case UMMUNOT_EVENT_TYPE_INVAL:
>                                 printf("[%3d]: inval cookie %lld\n",
>                                        i, ev[i].user_cookie_counter);
>                                 if (ev[i].flags &  
> UMMUNOT_EVENT_FLAG_HINT)
>                                         printf("  hint %llx...%lx\n",
>                                                ev[i].hint_start,  
> ev[i].hint_end);
>                                 break;
>                         case UMMUNOT_EVENT_TYPE_LAST:
>                                 printf("[%3d]: empty up to %lld\n",
>                                        i, ev[i].user_cookie_counter);
>                                 break;
>                         default:
>                                 printf("[%3d]: unknown event type %d 
> \n",
>                                        i, ev[i].type);
>                                 break;
>                         }
>                 }
>         }
>
>         umn_unregister(123);
>

What happens if I register multiple regions with the same cookie value?

Is a process responsible for guaranteeing that it umn_unregister()s  
everything before exiting, or will all pending registrations be  
cleaned up/unregistered/whatever when a process exits?

-- 
Jeff Squyres
Cisco Systems