[openib-general] Re: Question about pinning memory

[Michael S. Tsirkin]

Quoting Jeff Squyres <jsquyres at open-mpi.org>:
> [...]
>
> Since MPI usually does not have visibility 
> for memory that has been free'd (or sbrk'ed), the following scenario is 
> possible (and in our experience, likely):
> 
> - user calls malloc() and gets virtual address A back
> - user calls MPI_Send with A
> - MPI pins address A
> - MPI saves A in internal lookup tables
> - MPI_Send returns
> - user calls free(A)
> - user calls malloc() and gets virtual address A back, but now A points 
> to different physical memory than before
> - user calls MPI_Send with A
> - MPI thinks that A is pinned (because A is in MPI's internal tables), 
> and therefore doesn't pin it
> - MPI tries to send A with the IB API, and Bad Things occur (either an 
> error, or worse, bad data is sent)

Jeff, I dont think openib stack can currently help you with this problem.

I suggest using malloc_hook and friends to track memory allocations.
Avoid caching memory for anything that isnt allocated with malloc,
and you are guaranteed that your hook gets called when the memory is
freed.

Having said that - do you have some benchmarks to prove that
pinning is that expensive? Are you sure that all these internal table
lookups aren't even slower?
I cant promise, but given a microbenchmark, maybe something could be done to
improve the speed of this operation.

Thanks,

-- 
MST