[ofa-general] peer to peer connections support

[Sean Hefty]

> Sorry if I wasn't clear, let me see if I understand you: with this
> different domain implementation, under both client/server the passive
> calls cm listen and the active call cm connect, where under peer/to/peer
> both sides call cm listen and later both sides may call cm connect or
> only one side, correct?

My thinking was that the peer to peer model would have both sides call 
connect only.  The peer to peer connection model only kicks in when both 
sides are in the REQ sent state.

> But it could set it to one as well... assuming my understanding above of
> the suggested implementation is correct, we can change the RDMA-CM API
> to let users specify on rdma_connect that they want peer to peer
> support, so such apps can issue rdma_listen call and later call
> rdma_connect with this bit set and they are done (or almost done... I
> guess there some more devil in the details here, isn't it?)

This was why I said that the IB CM API was fine, but the RDMA CM API 
would require changes.

> This makes the all peer to peer model useless, since an app can not make
> sure that connection occur at exactly the same time!

yep - (anyone can feel free to step in a set me straight on this...)

> the spec is that peer to peer model has the ability to handle also 
> connections that occur at exactly the same time but not only.

Peer to peer seems inherently racy to me.

> Under MPI each rank uses a different SID, so I think we are safe from 
> this problem.

Any peer to peer implementation should handle this case however.

- Sean