[ofa-general] Re: New proposal for memory management

[Jeff Squyres]

On Apr 30, 2009, at 6:22 PM, Jason Gunthorpe wrote:

> After reading all the postings, I think my idea to fix the verbs API
> to not, essentially, corrupt an existing registration when the virtual
> address space changes is the best bet. This slightly changes the
> semantics of the verbs MR to refer to virtual address space within the
> process, not the underlying object(s) that happen to be mapped there
> when the registration is made.
>

I'm not sure how this helps MPI -- our registration caches will still  
become invalid if the MPI app free()'s registered memory...?

MPI maintains a registration cache because registration is so  
expensive.  Even if the registration cache becomes "safely" invalid  
(e.g., you'll never get a scenario where one virtual address could  
have previously pointed to a different hardware address within the  
span of one process), it doesn't help.
>
> MPIs can choose to continue to hook malloc/free/etc or not, it doesn't
>

No no no!  We don't want to continue hooking this stuff.  The hooks  
are horrible, horrible, horrible -- there's real-world apps that break  
them.

> > While MPI is currently the biggest victim, this broken memory  
> management
> > model is also an enormous roadblock for any other application or  
> ULP to
> > write to verbs.
>
> I'm not sure this is true.. The purpose built verbs apps I've worked
> on don't have a problem like MPI, and managing the memory registration
> was not hard.


Ok, I'll back off slightly: if you want verbs to go mainstream, there  
will be many other ULPs / middleware libraries that have memory models  
like MPI's (that the upper layer is responsible for allocating/freeing  
message buffers).  Put differently: the TCP/sockets stack doesn't have  
this restriction; it will be extremely difficult to convert legions of  
sockets programmers to verbs if you effectively restrict large  
messages to only be allocated/freed by the network layer (kinda  
defeats the point of RDMA if you have to copy large messages, right?).

-- 
Jeff Squyres
Cisco Systems