[openib-general] design for communication established affiliated asynchronous event handling

[Sean Hefty]

>The cma/verbs consumer can't just ignore the event since its qp state is
>still RTR which means an attempt to tx replying the rx would fail.

In most cases, I would expect that the IB CM will eventually receive the RTU,
which will generate an event to the RDMA CM to transition the QP into RTS.  This
is why I think that the event can safely be ignored.  It does however mean that
a user cannot send on the QP until the user sees RDMA_CM_EVENT_ESTABLISHED.

>I suggest the following design: the CMA would replace the event handler
>provided with the qp_init_attr struct with a callback of its own and
>keep the original handler/context on a private structure.

This sounds like it would work.  I don't think that there are any events where
the additional delay would matter.

As an alternative, I don't think that there's any reason why the QP can't be
transition to RTS when the CM REP is sent.  A user just can't post to the send
queue until either an RDMA_CM_EVENT_ESTABLISHED, IB_EVENT_COMM_EST, or a
completion occurs on the QP.  (This doesn't change the fact that the IB CM still
needs to know that the connection has been established, or it risks putting the
connection into an error state if an RTU is never received.)

- Sean