[openib-general] Re: Question about pinning memory

[Gleb Natapov]

On Mon, Jul 25, 2005 at 04:36:53PM -0700, Roland Dreier wrote:
>     Pete> No, of course not.  Each event is currently 8 bytes: hca
>     Pete> handle and memory handle, as supplied at registration time.
>     Pete> The size of the queue is configurable, and only needs to be
>     Pete> big enough to hold any forced deregistrations that happen
>     Pete> from VM activity like sbrk(<0) and munmap() between MPI (or
>     Pete> other library) calls.  (If you add the signaling mechanism
>     Pete> proposed by Gleb, perhaps with a separate thread, it might
>     Pete> be possible to drain the queue immediately as it approaches
>     Pete> fullness.)
> 
> Hmm, thinking about this some more, it occurs to me that it might be
> possible for the kernel and userspace to work together even more
> efficiently.  Does the following make sense?
> 
> When memory is registered, userspace tells the kernel VM activity
> monitor where its registration cache data structure for that
> registration is.  If the kernel detects some VM activity that affects
> that registration, then it can invalidate it directly in the userspace
> cache data structure.
It may set 'not valid' flag in struct ibv_mr. The question is how can you do it and
not penalise application that don't use memory registration.

> 
> I guess there are two issues to resolve here:
> 
>  - Are there any races between the kernel marking something invalid
>    and userspace marking it valid again?
As I said before we don't cache across mman/munmap so something that 
became invalid cannot became valid again.

--
			Gleb.