[ofa-general] Re: New proposal for memory management

[Jason Gunthorpe]

On Thu, Apr 30, 2009 at 09:52:32AM -0400, Jeff Squyres wrote:
> I think Jason is the only one who is remaining at least somewhat on-topic 
> here.

Thanks, but I have no stake in this, it is just interesting :)

After reading all the postings, I think my idea to fix the verbs API
to not, essentially, corrupt an existing registration when the virtual
address space changes is the best bet. This slightly changes the
semantics of the verbs MR to refer to virtual address space within the
process, not the underlying object(s) that happen to be mapped there
when the registration is made.

The data corruption problem would be immediately solved by doing this
and no changes at all would be needed in any MPIs. Surely a good
thing?

MPIs can choose to continue to hook malloc/free/etc or not, it doesn't
really matter from a correctness perspective. Minimizating the amount
of VM that is registered is perhaps important, perhaps not, I suspect,
depending on the job.. Certainly there are fairly fast ways to do
periodic garbage collection on the registration cache (ie by
inspecting proc/self/maps, or the glibc free block list, or
whatever.).

Notifiers are going to be very troublesome, every time any sort of
synchronous to user space notifier has been proposed or implemented in
the kernel it has been a disaster. I would not hold out much hope for
this..

> While MPI is currently the biggest victim, this broken memory management 
> model is also an enormous roadblock for any other application or ULP to 
> write to verbs.

I'm not sure this is true.. The purpose built verbs apps I've worked
on don't have a problem like MPI, and managing the memory registration
was not hard. My main complaint would be the lack of FMR and FRMR in
userspace - mostly because of the performance of the registration
functions..

Jason