[openib-general] design for communication established affiliated asynchronous event handling

[Rimmer, Todd]

> -----Original Message-----
> From: Michael S. Tsirkin
> Sent: Thursday, June 29, 2006 1:45 AM
>  
> Quoting r. Roland Dreier <rdreier at cisco.com>:
> > Subject: Re: design for communication established affiliated
> asynchronous event handling
> >
> > >I suggest the following design: the CMA would replace the event
handler
> > >provided with the qp_init_attr struct with a callback of its own
and
> > >keep the original handler/context on a private structure.
> >
> > This is probably fine.  There is one further situation where the
> > connection needs to be established, beyond RTU and the communication
> > established async event.  Namely, if a receive completion is polled.
> > Since async events are, well, asynchronous, there's no guarantee
that
> > the communication established event will be reported any time
soon...
> 
> How about user taking this into account and not arming the CQ /
> not polling it until the established event?

If the ULP is properly designed, the asynchronous-ness of the event (or
RTU for that matter) should not be an issue.

Per the IBTA CM state machine, the passive side upon sending the REP
should move its endpoint (the QP and the ULPs state machine) state to
Ready to Receive.  QPs in RTR can have send WQEs posted to them, however
they will not be sent until the QP is moved to RTS.

This means the ULP while in RTR can perform its normal receive
completion handling and even build and post send requests in response to
such received messages.  Such sends will be queued until the QP later
moves to RTS.

Most ULPs have some sort of application level flow control.  This may be
simply RNR NAK or it could be a credit system (such as SRP) or an
additional application initialization protocol (such as SDP).  Hence the
active side will generally perform limited sends (typically one) to the
passive side until it gets a response from the passive side (which won't
happen until the QP is in RTS).  Hence for a good ULP protocol, there is
no risk of overflowing the send Q while waiting to move to RTS.

The only thing the passive side ULP should not do until in RTS is any
sort of "periodic status messages which don't require active side
acknowledgement".  Since the RTS state could be delayed, the ULP should
not risk overflowing its send Q with such messages.  Most of the
standard ULP protocols (SDP, etc) do not have such messages or they
require ULP level protocol negotiation before they are activated.

Hence if this is all properly handled, the passive side's RTU/Async
Event handling sequence will merely move the QP to RTS and notify the
ULP.  The ULP will likely do very limited work for this notification
(perhaps just a state transition) as all the real work should have been
done before sending the REP.

The movement to RTS will enable the QP to start processing its Send Q
and everything will be good.

Taking this approach keeps the CM/CMA and ULP simpler in design and
merely allows the RTS/RTU/Async Event handling to be another event in a
state machine.

Todd Rimmer