[ofa-general] peer to peer connections support

[Sean Hefty]

> I wonder what will it take to implement peer to peer connection 
> establishment support in the IB CM (and RDMA-CM, ofcourse). Also do you 
> expect any API changes needed in order to support that?

Peer to peer connection was never fully implemented in the ib_cm.  I 
don't think it would be that hard to implement at that level, and it 
shouldn't require API changes.

Support at the rdma_cm level may require an API change.  There's no easy 
way for the rdma_cm to know if it should invoke the IB peer-to-peer 
connection model.  I'm not even sure how one peer would know the other 
peer's port number, unless well known ports are used on both sides.

> Reading through section 12.10.4, I was not sure to fully understand the 
> following: "The ServiceID implicitly defines whether the service is 
> client/server or peer to peer, but the application must inform the CM so 
> that the CM will handle the inbound REQ correctly."

My interpretation of this is:

An app uses a single service ID and connects using either peer to peer 
or client/server.  This implies that a service ID is either for peer to 
peer or client/server connections.

The CM needs to know the connection model selected by the app, to avoid 
matching an incoming REQ incorrectly.  For example, suppose an app tries 
to connect to SID 1, while another app on the same machine is listening 
on SID 1.  Does an received REQ with SID 1 match for a peer to peer 
connection, or client/server?  In this case, you can probably guess 
client/server correctly, but suppose the REQ is received before the 
server app starts.  The CM needs to know not to match the REQ with the 
client.

> Such support would be useful in symmetric schemes such as MPIs that open 
> connections on demand and more applications where each party can both 
> accept and initiate connections. For example, I understand that some 
> work is done now at the open mpi community to use the rdma-cm as a 
> possible channel for connection establishment.

I would need to better understand the expected usage model, like how the 
peers find each other, but this is something that could be added if needed.

- Sean