[OMPI devel] [ofa-general] OpenMPI and RDMA-CM

[Steve Wise]

On Wed, 2007-05-09 at 08:37 +0300, Or Gerlitz wrote:
> Andrew Friedley wrote:
> > Jeff Squyres wrote:
> >>>> FWIW, yes, adding RDMA CM support has actually been on my to-do list
> >>>> for a while, but it keeps getting bumped by higher priority items.
> >>>> It would be *much* better if some iWARP companies got involved in
> >>>> Open MPI...
> 
> > Hmm I'm interested.  I've already done some work switching over to RDMA 
> > CM for some research stuff I've been doing; it's not publicly accessible 
> > w/o the 3rd party agreement.  I can help answer questions on what 
> > exactly needs to change, and do some testing.
> 
> Doing a bit of zoom out from the "how to make ofed's udapl work for 
> ompi" thread, my thinking is that the ompi udapl btl enablement is 
> actually only the first step, where for production/longterm/etc you want 
> to have an rdmacm btl. Reasoning here is made of many arguments, among 
> them the quickest i can make are:
> 
> A) it seems that ompi would want to use not only RC but rather also UD 
> multicast and unicast, which are not covered by udapl
> 
> B) there's actually no real justification to maintain two APIs (namely 
> udapl vs libibvers/librdmacm), so down the road, only one of them would 
> survive (udapl is implemented ***over*** libibverbs/librdmacm so if the 
> latteres dies same does udapl). Specifically, I hear here and there that 
> the OFED stack is now on its way to be deployed all over the place, 
> specifically in commercial Unix OSs (which want modern! code that 
> supports IPoIB-CM,RDS,SRP,iSER, etc you named it) so eventually the 
> rdmacm btl can be used also over Solaris et al.
> 

Agreed.  enabling udapl will get OMPI over iwarp immediately (and
hopefully in ofed-1.2).  Post ofed-1.2, I think OMPI _should_ create a
rdma-cm btl.  That's the plan...

Steve.