[openib-general] Question about pinning memory

[Timur Tabi]

Roland Dreier wrote:
> The first statement in this paragraph is false.  It's easy to strace a
> simple program that does something like
> 
> 	x = malloc(1000000);
> 	free(x);
> 
> and watch glibc do an mmap(... MAP_ANONYMOUS ...) followed by
> munmap().  Even for smaller allocations, glibc may use sbrk() to
> shrink the heap in free().  You can read about M_TRIM_THRESHOLD and so
> on in the mallopt() documentation.

I wouldn't rely on this at all.

libc is under no obligation to inform the kernel in any way when free() is called.  I just 
don't see any reliable way of having MPI or OpenIB or whatever get notified when an 
application "frees" a block of memory before it explicitely unregisters it.  There's 
nothing stopping an application from simply re-using a buffer for non-RDMA purposes 
whenever it wants.  I would rather err on the side of pinning too much memory.

I just don't see how this problem can be solved.  The application needs to specifically 
tell the library when it no longer wants a particular block of memory to be used for RDMA 
operations.  If it re-uses or deallocates such a block BEFORE telling the library, I 
consider that to be an application bug.  At that point, all we need to worry about is 
whether that buffer gets allocated to another process before deregistration, because that 
would be a security hole.

(Disclaimer: I just noticed this thread and I don't understanding yet everything that 
others have so far written.)

-- 
Timur Tabi
Staff Software Engineer
timur.tabi at ammasso.com

One thing a Southern boy will never say is,
"I don't think duct tape will fix it."
      -- Ed Smylie, NASA engineer for Apollo 13