[ofa-general] Memory registration redux

[Caitlin Bestler]

On Mon, May 18, 2009 at 9:24 AM, Jeff Squyres <jsquyres at cisco.com> wrote:
> On May 7, 2009, at 5:58 PM, Roland Dreier (rdreier) wrote:
>
>>  > Specifically: the actual dereg of 0x1000-0x3fff is blocked on also
>>  > releasing 0x2000-0x2fff.
>>
>> If everyone is doing this, how do you handle the case that Jason pointed
>> out, namely:
>>
>>  * you register 0x1000 ... 0x3fff
>>  * you want to register 0x2000 ... 0x2fff and have a cache hit
>>  * you finish up with 0x1000 ... 0x3fff
>>  * app does something (which is valid since you finished up with the
>>   bigger range) that invalidates mapping 0x3000 ... 0x3fff (eg free()
>>   that leads to munmap() or whatever), and your hooks tell you so.
>>  * app reallocates a mapping in 0x3000 ... 0x3fff
>>  * you want to re-register 0x1000 ... 0x3fff -- but it has to be marked
>>   both invalid and in-use in the cache at this point !?
>>
>
>
> Sorry; this mail slipped by me and I just saw it now.
>
> If this can actually happen -- that the mapping of 0x1000 ... 0x3fff can
> change even though it is still registered, then we're screwed -- we have no
> way of knowing that this is now invalid (Open MPI, at least -- can't speak
> for others).
>
> Is there a way to detect condition this in userspace?
>
How does 0x1000 to 0x3fff get registered as a single Memory Region?
If it is legitimate to free() 0x3000..0x3fff then how can there ever be a
legitimate reference to 0x1000..0x3fff? If there is no such single reference,
I don't see how a Memory Region is every created covering that range.

If the user creates the Memory Region, then they are responsible for not
free()ing a portion of it.

Would the MPI library ever create a single large memory region based on
two distinct Sends?