[openib-general] rdma_cm.h: comment nits.

[Tom Tucker]

On Wed, 2006-05-10 at 09:26 -0700, Sean Hefty wrote:
> Tom Tucker wrote:
> >>Its OK to call rdma_reject on active side as well, isn't it?
> > 
> > You'll get -EINVAL on iWARP if you do this....
> 
> For IB, rdma_reject can be called on the active side if the user is managing 
> their own QP states, or is SDP.  How does iWarp support userspace QPs?
> 

iWARP presented a challenge with the current model because a QP becomes
logically bound to a connection when the QP state --> RTS. In fact,
there is no notion of an RDMA connection independent of a QP. In RDMAC,
the model assumed was that you would establish a TCP connection,
potentially do some initial data exchange and then 'migrate' the
connection to RDMA mode. The migration mechanism was that you would
provide the connection id (socket fd or whatever) to the qp_modify -->
RTS. Between then and QP state --> ERROR, TERMINATE the QP == the
connection.

Sorry for the long diatribe, but I'm trying to set the stage for the
approach...

Well, we don't have this notion in the API, so what I did was perform
this logical qp_modify in rdma_connect and rdma_accept respectively.
This is done by passing the QPN down to the provider's connect/accept
verb in the conn_attr parameter. The iw_cm then adds a reference on the
QP by calling a special provider method for this purpose, and the
provider adds a reference on the cm_id by calling a method hung off the
iw_cm_id (there is no API in IW CM to add the reference because I didn't
want a circular modular dependency).

I know this sounds complicated, but since the QP can be destroyed before
the CM_ID and vice versa (e.g. a kill -9 to the process and clean up in
fd order), I had to takes these references and make them explicit. This
method, btw, replaces the state-implies-reference count approach of a
previous implementation.

In summary: 
- For IB, there is no explicit association between the CM ID and the QP
for user-mode apps, while for iWARP there is. 
- To make QP management consistent between kernel and user-mode an iWARP
provider service maps a QPN to a struct ib_qp *.
- Explicit reference API were added to allow the provider to reference
the cm_id and the IW CM to reference the qp.
- The provider/RNIC can change the QP state without direct action by the
app.

So... all that said, I could in fact support rdma_reject on an active
side connection. But this would effectively reduce to a QP --> ERROR and
I doubt this matches the semantics you're looking for.


> - Sean