[ofa-general] Re: New proposal for memory management

[Ralph Campbell]

On Tue, 2009-04-28 at 21:03 -0700, Jason Gunthorpe wrote:
> On Tue, Apr 28, 2009 at 09:10:35PM -0400, Jeff Squyres wrote:
> 
> > The conversation needs to start somewhere.  MPI is verbs' biggest  
> > customer; this is a major pain point for all of us.  Can't we fix it?   
> > Do you need something more than a specific use case and API proposal  
> > to start the conversation?  No one has money to travel travel; the bi- 
> > weekly EWG call is for discussing bugs.  What other vehicle do you  
> > suggest for this discussion?
> 
> I've often wondered, wouldn't it just be fine for MPI if the entire
> process address space is kept pinned, registered and consistent with
> the HCA? The process would opt in to this behavior during MPI
> startup. Similar in spirit to the all physical memory registration the
> kernel can do.
> 
> That seems like a much more straightfoward problem than trying to fit
> the fairly incompatible verbs registration API and the MPI API together.
> 
> Jason

This would be nice except that it is impossible.
The whole idea behind virtual memory is that the application uses
virtual addresses which are only temporarily mapped to physical
memory by the OS. If the OS allowed a user to lock the
virtual to physical mapping for all of physical memory, the OS
wouldn't be able to run any other processes, do I/O, etc.
The process can have a larger virtual address space than the
available physical memory so no fixed mapping is possible.
To prevent this, the application has limits imposed by the OS
(type "ulimit -l").

The kernel doesn't really have a "map all physical memory"
registration. This mapping basically tells the HCA that the
kernel will handle the mapping instead of the HCA driver.