[ofa-general] Re: New proposal for memory management

[Jeff Squyres]

I think Jason is the only one who is remaining at least somewhat on- 
topic here.

My goal for this thread was to open discussion about solving the  
broken memory management model for OpenFabrics.  Specifically, I'm  
looking for a software solution for today's hardware (both IB and  
iWARP).

While MPI is currently the biggest victim, this broken memory  
management model is also an enormous roadblock for any other  
application or ULP to write to verbs.

On-demand paging, while the desired end result sounds great, will  
definitely require new hardware and will likely be a very complex  
design conversation that takes a long, long time.  That's great if my  
proposal [finally] seriously inspires people to tackle this problem  
(and/or other hardware-assisted ideas), but I consider those to be  
longer-term solutions.

I'm looking for a) a much shorter-term solution that is b) workable on  
current hardware.

Thanks.


On Apr 29, 2009, at 6:44 PM, Jason Gunthorpe wrote:

> On Wed, Apr 29, 2009 at 03:28:00PM -0700, Ralph Campbell wrote:
>
> > Besides, mmap() only allocates a virtual address range in the user's
> > address space. It doesn't fault in all the pages into physical  
> memory.
> > That happens when the application tries to read or write memory in
> > the VA range of the mmap. The IB memory registrations need physical
> > addresses and it would be impractical to do this for every mmap or
> > brk.
>
> If your goal is to keep the mr consistent then you only need to fault
> and pin pages from the new mmap that intersect with pre-existing
> memory registrations.
>
> I chucked out 3 things to consider:
>  - Pin and register all process memory (no swap!)
>  - Keep the MR consistent by pinning and registering new mmaps
>    that intersect with pre-existing memory registrations
>  - Keep the MR consistent by preventing the kernel from returning
>    new mmaps that overlap existing memory registrations.
>
> Jason
>


-- 
Jeff Squyres
Cisco Systems