[openib-general] posting send requests in RTR

[Rimmer, Todd]

> From: Caitlin Bestler [mailto:caitlinb at broadcom.com]
> 
> That assumes that there is any valid reason for an application to
> post send requests before the connection is established. While there
> is clearly a need to post receive work requests before the connection
> is established I cannot think of any reason why an application needs
> to pre-prime the send queue.
> 
> Putting unneeded complexity in the definition of a hardware service
> just invites more hardware dependencies and eventual hardware specific
> bugs that will complicate life for application developers. "Don't post
> until the connection is established" is very simple for *both* the
> application and the verbs provider.

Here is a real world example and how we uncovered this issue: Native IB
storage SRP Targets.

SRP Targets implement the passive side, after processing the REQ, they
send a REP.  However the target QP is in RTR.

The Srp client gets the REP, sends the RTU and announces to the OS that
a device is available.  The OS immediately begins issuing SCSI commands.

The target receives the SCSI commands (such as Test Unit Ready or
Inquiry) and wants to act on them immediately.  However if the command
has passed the RTU or the RTU is lost, the target is still in RTR.  If
the command was very simple, the target may want to answer the query
immediately by posting a send with the response.  If the RTU is totally
lost and async event processing is delayed, the target may even be able
to do some processing and still not have the QP in RTS when it has its
response ready.

While its possible to build an additional queuing point above the send
Q, such queuing points tend to impact performance and latency for high
performance protocols.

While this example was for SRP, its not unique to SRP.  Most protocols
(such as SRP and SDP) include a set of initialization or capability
query messages which the client may issue to the target immediately
after the client believes it has a connection.  Many of those
initialization messages are the types of things which the target can
answer immediately and may even choose to do so in its completion
handler.

Todd Rimmer