[ofa-general] New proposal for memory management

[Woodruff, Robert J]

Brian wrote,

>I think there are other problems with the verbs interface that would still
>make MPI implementers twitch (some of which are in the slides Jeff sent out
>to begin this discussion).  But I certainly wouldn't say no to a real set of
>tag matching primitives.  Of course, that opens a whole can of worms that
>I'm not sure OFED is ready to deal with.

>It also may or may not solve the memory registration problem.  If the memory
>in the matching verb still had to be registered, we haven't solved the
>problem that started this discussion.  So the verb would have to also handle
>memory registration, which seems to go against the general "OFA way". 

I think if we did such a thing, we could implement a set of tag-matching 
primitives (similar to MX or PSM) that are kind of a separate library 
from the OFA RDMA verbs, just like PSM for Qlogic is a separate library and
not part of the OFA verbs. Just like with MX and PSM, I think the registration
can be done my the tag-matching driver (like PSM or MX do) and
not require MPI to do it. Think of this as "the MPI tag-matching interface" library 
for OFA. 

However, this would only completely solve your problem and complexity of using the
OFA RDMA verbs if all the hardware vendors implemented tag-matching in their 
NICs. Seems like if they want to better support MPIs, that is what they
would do and then MPIs would only have to use the simple tag-matching
primitives and would not have to worry about things like memory registration
caches and such.

Anyway, I think it is an interesting idea worth perusing with the IHVs as the long
term solution to most of the issues that Jeff raised in Sonoma. 

woody