<!DOCTYPE HTML PUBLIC "-//W3C//DTD HTML 3.2//EN">
<HTML>
<HEAD>
<META HTTP-EQUIV="Content-Type" CONTENT="text/html; charset=UTF-8">
<META NAME="Generator" CONTENT="MS Exchange Server version 6.5.7654.12">
<TITLE>Re: bug 1918 - openmpi broken due to rdma-cm changes</TITLE>
</HEAD>
<BODY>
<!-- Converted from text/plain format -->
<P><FONT SIZE=2>Note that it is highly unlikely that we will release open mpi 1.4.2 in time for ofed 1.5.1.<BR>
<BR>
Also note that trying to bind rdma cm to all interface ip addresses was the way that we were advised by openfabrics to figure out which devices are rdma-capable.<BR>
<BR>
As such, it is highly desirable to get the fix transparently in rdmacm and preserve the old semantic. More specifically, it seems undesirable to change this semantic in a minor ofed point release.<BR>
<BR>
-jms<BR>
Sent from my PDA. No type good.<BR>
<BR>
----- Original Message -----<BR>
From: Steve Wise <swise@opengridcomputing.com><BR>
To: Sean Hefty <sean.hefty@intel.com><BR>
Cc: linux-rdma <linux-rdma@vger.kernel.org>; OpenFabrics EWG <ewg@openfabrics.org>; Jeff Squyres (jsquyres); Roland Dreier (rdreier)<BR>
Sent: Thu Feb 04 18:04:23 2010<BR>
Subject: Re: bug 1918 - openmpi broken due to rdma-cm changes<BR>
<BR>
Sean Hefty wrote:<BR>
>> Well then the rdma-cm needs to know which devices support hw loopback.<BR>
>> Cuz on a T3-only system, no hwloop...<BR>
>> <BR>
><BR>
> The problem sounds like it's more than just whether 127.0.0.1 is usable. That<BR>
> check may fix openmpi, but it sounds more like the app needs to know whether the<BR>
> device can actually support loopback, regardless of what addresses are used. Is<BR>
> this correct?<BR>
><BR>
> What would openmpi do if there were two addresses assigned to the T3 device?<BR>
> <BR>
<BR>
It would use them and might even create two connections.<BR>
<BR>
> Does openmpi simply bypass RDMA for all connections on the local machine?<BR>
><BR>
> <BR>
<BR>
OpenMPI can be run to use hw loopback if its available. For T3<BR>
clusters, OMPI is run in a mode to use shared memory for intra-node<BR>
communications.<BR>
<BR>
<BR>
> Basically, I'm not sure that this is *just* an rdma_cm issue. Although it<BR>
> definitely appears that some sort of change needs to be made to the rdma_cm.<BR>
><BR>
> <BR>
<BR>
I think the OpenMPI rdmacm code needs to skip 127.0.0.1, in this<BR>
particular case. Prior to ofed-1.5.1, however, the bind would fail and<BR>
thus OpenMPI would not advertise 127.0.0.1 to its peer. I will work to<BR>
get that change done.<BR>
<BR>
But lets also add a device attribute so the rdmacm can know if a device<BR>
supports loopback. Clearly, if the rdma-cm allows binds to T3,<BR>
loopback connections will fail at connect time.<BR>
<BR>
Hey Roland, are you ok with a device attribute to indicate hw-loopback<BR>
support?<BR>
<BR>
<BR>
Steve.<BR>
<BR>
<BR>
</FONT>
</P>
</BODY>
</HTML>