[ofa-general] Re: New proposal for memory management

[Jason Gunthorpe]

On Thu, Apr 30, 2009 at 02:24:47PM -0400, Tom Talpey wrote:
> At 06:11 PM 4/29/2009, Barrett, Brian W wrote:
> >On 4/29/09 15:55 , "Jason Gunthorpe" <jgunthorpe at obsidianresearch.com>
> >wrote:
> >
> >>> The problem is that MPI needs to be aware of the application doing
> >>> the free() and unregister or flush its MR cache for that virtual
> >>> address range. Of course it would be difficult for OpenMPI to have
> >>> callbacks or hooks into every way memory could be allocated/freed
> >>> that an application might use.
> >> 
> >> There are only three calls that affect the way VM memory maps to
> >> physical and thus would invalidate the mr cache: mmap, munmap and brk.
> >
> >There's also System V shared memory, which at least one scientific code out
> >there uses.
> 
> Don't forget fork, vfork, clone and exec, and also don't forget any
> copy-on-write mappings that result. Oh, and those pesky stack pages.

verbs just plain can't inherit QP and MR across fork and all the
related. There is no way to split a QP and a MR across to processes
and have things still make sense. One process gets it, so there isn't
really a problem.

Stack pages are mostly fixed in VM address space, particularly if you 
give up MAP_GROWSDOWN which I think most threading libraries do these
days?

Anyway, it doesn't matter, if the kernel has a syscall to enable
registering all VM in a process then the kernel is perfectly able to
capture all the wierd cases and fix them up. The API to userspace is
very simple and sane.

> I think the point is that making any guarantees that memory remains fixed
> and present will inevitably lead to nontransparent API requirements on the
> applications. Been there, done that, got plenty of t-shirts. It's a hard road,
> because APIs are forever.

That's why we are here, the MPI spec says all memory in a process is
valid to use with a network operation - that is utterly incompatible
with verb's notion of memory registration.

Jason