[OMPI devel] [ofa-general] OpenMPI and RDMA-CM

[Jeff Squyres]

On May 9, 2007, at 1:37 AM, Or Gerlitz wrote:

> Doing a bit of zoom out from the "how to make ofed's udapl work for  
> ompi" thread, my thinking is that the ompi udapl btl enablement is  
> actually only the first step, where for production/longterm/etc you  
> want to have an rdmacm btl.

I think this is a bit of a misunderstanding.  The "BTL" in Open MPI  
is a byte transfer layer; it is a point-to-point abstraction for  
moving bytes between two processes.  BTL components (read: plugins)  
are typically distinguished by the underlying protocols used.  For  
example, we have an RC verbs-based BTL and we have a separate uDAPL- 
based BTL.  Andrew is also working on a research-quality UD verbs- 
based BTL.

Hence, how a particular BTL component makes connections between  
process peers is really a side-effect of moving bytes around, and not  
the focus of the BTL.  So having a "rdmacm" BTL doesn't really make  
sense.  If both the RC and UD verbs-based BTLs someday use the RDMA  
CM for connections, we might abstract the connection management out  
to a common piece of code between the two.  But that's a different  
issue.  If we end up having a mixed BTL someday that uses both RC and  
UD, then the need for the common code may go away.  But that's in the  
future.

> Reasoning here is made of many arguments, among them the quickest i  
> can make are:
>
> A) it seems that ompi would want to use not only RC but rather also  
> UD multicast and unicast, which are not covered by udapl
>
> B) there's actually no real justification to maintain two APIs  
> (namely udapl vs libibvers/librdmacm), so down the road, only one  
> of them would survive (udapl is implemented ***over*** libibverbs/ 
> librdmacm so if the latteres dies same does udapl). Specifically, I  
> hear here and there that the OFED stack is now on its way to be  
> deployed all over the place, specifically in commercial Unix OSs  
> (which want modern! code that supports IPoIB-CM,RDS,SRP,iSER, etc  
> you named it) so eventually the rdmacm btl can be used also over  
> Solaris et al.

I think that's not quite the point.

1. A piece of history: the uDAPL BTL was originally developed by a  
grad student just as an excuse to learn the BTL interface and OMPI  
internals.  We already had an RC verbs-based BTL at the time.

2. When Sun joined Open MPI, they took over the development and  
maintenance of the uDAPL BTL because uDAPL is the only high  
performance stack on Solaris.

3. It's fine that Sun will someday support the same verbs interface  
that OFED does.  But *today*, they don't.  So for their current  
customers, they need to support uDAPL.  As such, we have done little/ 
no testing of uDAPL on OFED since Sun took over the uDAPL BTL -- all  
testing since that point has been on Solaris uDAPL.  All of our Linux/ 
OFED efforts have been on the verbs interface.

4. The Open MPI focus on uDAPL over OFED at the moment is simply to  
jump-start iWARP testing.  Both NetEffect and Chelsio have chimed in  
to say that they will do the RDMA CM work for Open MPI, but uDAPL can  
be used as a temporary workaround that can be used [effectively]  
immediately while they get up to speed on the Open MPI code base and  
do the RDMA CM work.

-- 
Jeff Squyres
Cisco Systems