[openib-general] Re: Question about pinning memory

[Roland Dreier]

    Pete> No, of course not.  Each event is currently 8 bytes: hca
    Pete> handle and memory handle, as supplied at registration time.
    Pete> The size of the queue is configurable, and only needs to be
    Pete> big enough to hold any forced deregistrations that happen
    Pete> from VM activity like sbrk(<0) and munmap() between MPI (or
    Pete> other library) calls.  (If you add the signaling mechanism
    Pete> proposed by Gleb, perhaps with a separate thread, it might
    Pete> be possible to drain the queue immediately as it approaches
    Pete> fullness.)

Hmm, thinking about this some more, it occurs to me that it might be
possible for the kernel and userspace to work together even more
efficiently.  Does the following make sense?

When memory is registered, userspace tells the kernel VM activity
monitor where its registration cache data structure for that
registration is.  If the kernel detects some VM activity that affects
that registration, then it can invalidate it directly in the userspace
cache data structure.

I guess there are two issues to resolve here:

 - Are there any races between the kernel marking something invalid
   and userspace marking it valid again?

 - How do we keep track of a process's registrations in the kernel so
   that we can efficiently find which (if any) registrations are
   affected by a VM operation?  (Maybe you've solved this already)

 - R.