[ofa-general] New proposal for memory management

[Pavel Shamis (Pasha)]

> - Verbs works well for a number of applications (Roland and I have each written 
> multiple, for example).
>   
BTW it is not too much user-level native IB applications that I know.
Verbs works perfect for kernel level ULPs that actually hide all the 
complexity from user level.

> - IMHO, there is a problem with your API that should be fixed (the messaging 
> layer needs to manage network buf allocation).  If you required mpi_alloc_mem, 
> you would get rid of a whole layer of complicated crap.  It may not be feasible 
> for you, but it is the right thing to do from an engineering perspective, 
> right?
>   
 From engineering point maybe it is correct. From business point we have 
application that were
written and defined before OFA. The application worked well and continue 
to work well today with
other modern interconnects. MPI people want to use IB and asking from 
OFA people to help to resolve
problem that just can not be resolved on user/MPI level.

> - "MPI" doesn't want to fix the problem, but instead is asking other people to 
> make kernel changes for them and saying things like "verbs is broken".
>   
Maybe the best way is to change spec and push people to use 
mpi_alloc_mem, but it is long term solution.
We want to allow people to run the applications now.
> I totally see your guys' problem and feel for you.  Either way it comes down to 
> politics; getting some MPI-specific code into the Linux Kernel (fun?), or 
> getting MPI users to have to change crusty old scientific code (very fun?).
>   
BTW the registration cache code may be useful not only for MPI model. I 
definitely see other
HPC models were I would like to have kernel level registration cache.

It will be very difficult to push users to change their code, especially 
when you have other
interconnect that does not require from them any code changes.
>
> Final point I want to make is that this is open source, so you can always try 
> submitting some elite patches and get the changes you need.
>   
Before somebody will put any human resource on this project it will be 
good to know if the
concept of this solution will be accepted by OFA community and it is 
reason why we are discussing it here.

Thanks
Pasha