[ewg] bug 1918 - openmpi broken due to rdma-cm changes

[Jeff Squyres (jsquyres)]

Note that it is highly unlikely that we will release open mpi 1.4.2 in time for ofed 1.5.1. 

Also note that trying to bind rdma cm to all interface ip addresses was the way that we were advised by openfabrics to figure out which devices are rdma-capable. 

As such, it is highly desirable to get the fix transparently in rdmacm and preserve the old semantic. More specifically, it seems undesirable to change this semantic in a minor ofed point release. 

-jms
Sent from my PDA.  No type good.

----- Original Message -----
From: Steve Wise <swise at opengridcomputing.com>
To: Sean Hefty <sean.hefty at intel.com>
Cc: linux-rdma <linux-rdma at vger.kernel.org>; OpenFabrics EWG <ewg at openfabrics.org>; Jeff Squyres (jsquyres); Roland Dreier (rdreier)
Sent: Thu Feb 04 18:04:23 2010
Subject: Re: bug 1918 - openmpi broken due to rdma-cm changes

Sean Hefty wrote:
>> Well then the rdma-cm needs to know which devices support hw loopback.
>> Cuz on a T3-only system, no hwloop...
>>     
>
> The problem sounds like it's more than just whether 127.0.0.1 is usable.  That
> check may fix openmpi, but it sounds more like the app needs to know whether the
> device can actually support loopback, regardless of what addresses are used.  Is
> this correct?
>
> What would openmpi do if there were two addresses assigned to the T3 device?
>   

It would use them and might even create two connections.

> Does openmpi simply bypass RDMA for all connections on the local machine?
>
>   

OpenMPI can be run to use hw loopback if its available.  For T3 
clusters, OMPI is run in a mode to use shared memory for intra-node 
communications.


> Basically, I'm not sure that this is *just* an rdma_cm issue.  Although it
> definitely appears that some sort of change needs to be made to the rdma_cm.
>
>   

I think the OpenMPI rdmacm code needs to skip 127.0.0.1, in this 
particular case.  Prior to ofed-1.5.1, however, the bind would fail and 
thus OpenMPI would not advertise 127.0.0.1 to its peer.  I will work to 
get that change done.

But lets also add a device attribute so the rdmacm can know if a device 
supports loopback.   Clearly, if the rdma-cm allows binds to T3, 
loopback connections will fail at connect time.

Hey Roland, are you ok with a device attribute to indicate hw-loopback 
support?


Steve.


-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.openfabrics.org/pipermail/ewg/attachments/20100205/4b430d33/attachment.html>