[ofa-general] New proposal for memory management

[Woodruff, Robert J]

Brian wrote,

>And Open Fabrics is the only "commodity" interfaces that makes implementers
>go through these pains.  Myrinet's MX, Cray's Portals, and Quadric's Tports
>all handle the issues either at the driver library or kernel module level. 

One important note is that in general, Myrinet, Quadrics, and even Portals 
were designed to primarily to run MPI, so it is not a surprise that their interfaces
map almost 1:1 to the MPI interfaces. Also, note that all of these use a tag-matching
capability, which also seems to map well to MPI.
RDMA/OFA verbs were designed to be a more general interface to
support lots of ULPs, networking (tcp/ip), storage, etc, not just MPI. 

That said, for hardware that does support these tag-matching capabilities, like
myrinet, Qlogic's HCA (i.e. PSM), OpenMX, and even quadrix, maybe OFA should have a 
generic tag-matching set of verbs that the MPIs could use instead of the 
RDMA verbs. The IHVs, like Qlogic, MX, and others that support tag-matching could
plug into this generic tag-matching infrastructure. The MPIs would then only have to
write one driver in MPI to support all these different IHVs that support tag-matching,
and that MPI driver would be a very simple one, since the tag-matching verbs 
would map almost 1:1 to the MPI interfaces, like MX or PSM do.

Heck, maybe we should even encourage the IBTA and iWARP associations to add tag-matching
as a feature to the next version of the IBTA and iWARP specs. If they did that,
it would make the MPI implementers life a lot easier. I would rather see that done,
then hack thousands of lines of memory registration caching code and stuff it into the 
kernel.