[ofa-general] New proposal for memory management

[Roland Dreier]

 > But whacky situations might occur in a multithreaded application where
 > one thread calls free() while another thread calls malloc(), gets the
 > same virtual address that was just free()d but has not yet been
 > unregistered in the kernel, so a subsequent ibv_post_send() may
 > succeed but be sending the wrong data.
 > 
 > Put simply: in a multi-threaded application, there's always the chance
 > that the notify won't get to the user-level process until after the
 > global notifier variable has been checked, right?  Or, putting it the
 > other way: is there any kind of notify system that could be used that
 > *can't* create a potential race condition in a multi-threaded user
 > application?

Without thinking too much about the proposal (except that it adds a lot
of new verb interfaces and a lot of kernel code, and therefore feels
like a hassle to me), I don't see how this race is solved by moving a
cache to the kernel.

If you have free()/malloc() of a buffer running in parallel with send
operations targeting the same buffer, then that seems like a buggy MPI
application.  Since free()/malloc() might not involve the kernel at all
(the userspace library might keep its own free list, etc) I don't see
how a registration cache in the kernel would help anyway.

Now, since free()/malloc() operations must be serialized with respect to
send/receive operations in userspace anyway, I don't see why a simpler
(and possibly more flexible/powerful) kernel notifier design can't
work -- if free() releases virtual memory back to the kernel, then the
kernel notifier will run before the free() call returns, so things
should work as planned.

 - R.