[ofa-general] New proposal for memory management

[Aaron Fabbri]

Pavel Shamis (Pasha <pashash <at> gmail.com> writes:
> 
> Aaron Fabbri (aafabbri) wrote:
> > 3. Rip out your registration cache.  Make malloc'd buffers go really
> > slow (register in fast path) and mpi_alloc_mem() buffers go really fast.
> > People will migrate.
> People will migrate to what ? (A) new malloc ? Or (B) other interconnect 
> platform
> that does not require from user to change his application in order to 
> get reasonable performance ?
> I'm not sure that people will chose (A) 
> 

Agreed.  As I said, getting all MPIs to agree would be the hard part.  Given 
that, I think a new malloc() is easier than porting to non-MPI middleware.

My point is this:

- Verbs works well for a number of applications (Roland and I have each written 
multiple, for example).

- IMHO, there is a problem with your API that should be fixed (the messaging 
layer needs to manage network buf allocation).  If you required mpi_alloc_mem, 
you would get rid of a whole layer of complicated crap.  It may not be feasible 
for you, but it is the right thing to do from an engineering perspective, 
right?

- "MPI" doesn't want to fix the problem, but instead is asking other people to 
make kernel changes for them and saying things like "verbs is broken".

I totally see your guys' problem and feel for you.  Either way it comes down to 
politics; getting some MPI-specific code into the Linux Kernel (fun?), or 
getting MPI users to have to change crusty old scientific code (very fun?).

Could you use the silent corruption problems as leverage to get MPI to move to 
mpi_malloc_mem?

Final point I want to make is that this is open source, so you can always try 
submitting some elite patches and get the changes you need.

Aaron