[ewg] bug 1918 - openmpi broken due to rdma-cm changes

[Steve Wise]

Sean Hefty wrote:
>> Also note that trying to bind rdma cm to all interface ip addresses was the way
>> that we were advised by openfabrics to figure out which devices are rdma-
>> capable.
>>
>> As such, it is highly desirable to get the fix transparently in rdmacm and
>> preserve the old semantic. More specifically, it seems undesirable to change
>> this semantic in a minor ofed point release.
>>     
>
> I think the issue is larger than just the rdma_cm.
>
> First, it sounds like openmpi tries to bind to 127.0.0.1, which now works.  If
> opemmpi uses shared memory for connections on the same machine, I'm not sure why
> this is a problem, unless it is passing that address to another machine to use
> for a connection.  If this is the case, then that is a bug in openmpi.
>   

Yes, OpenMPI incorrectly advertises 127.0.0.1 as a valid address 
to-which the peer can connect. This needs to be fixed.


> Second, I still don't understand whether iwarp is limited to 'loopback'
> connections that are not bound to 127.0.0.1.  For instance, if the RDMA device
> is associated with 192.168.0.1, then can it handle a connection from 192.168.0.1
> <-> 192.168.0.1?  If it can't, then the rdma_cm can't help in this case when
> bind is called.  The failure has to come during connect, which sounds like the
> behavior that's seen today with 127.0.0.1.
>   

Its not iWARP specific.  A device may or may not support hw loopback.    
Now the IB spec mandates this support, but the iWARP spec doesn't.  
Ammasso and Chelsio T3 rnics do not support HW loopback.  They will fail 
if you try to connect to a local address.  The rdma-cm shouldn't allow 
binds to 127.0.0.1 for these devices since it 100% implies that the 
connection will require hw loopback for that device.

> So, while the rdma_cm can fail binds to 127.0.0.1 if the RDMA device doesn't
> support loopback, I'm still not sure how much of a fix this is.
>   

My concern is breaking an existing working OpenMPI in a point release 
because we changed semantics of the rdma-cm in an ofed point release...

BTW:  Was this change an artifact of rebasing ofed-1.5.1 on a new kernel 
version?

Steve.

> - Sean
>
> --
> To unsubscribe from this list: send the line "unsubscribe linux-rdma" in
> the body of a message to majordomo at vger.kernel.org
> More majordomo info at  http://vger.kernel.org/majordomo-info.html
>