[openib-general] posting send requests in RTR

[Or Gerlitz]

Sean Hefty wrote:
>> That assumes that there is any valid reason for an application to
>> post send requests before the connection is established. While there
>> is clearly a need to post receive work requests before the connection
>> is established I cannot think of any reason why an application needs
>> to pre-prime the send queue.
> 
> It's not pre-priming the send queue.  An application could pull a completed
> receive work completion off of a CQ.  The receive may very well be a request
> that requires a response.  At this point, the connection is obviously
> established from the consumers viewpoint, but not necessarily by the viewpoint
> of the RDMA CM or IB CM.  The response must now be queued.
> 
> I believe that the problem can be limited under the following application
> conditions:
> 
> 1. The application uses the CQ with different QPs.
> 2. The application is on the passive (server) side of the connection.
> 3. The active (client) side sends a request to the server.
> 
> Even combined these conditions could easily occur.
> 
> IMO, the question is do we pass this problem to the applications to deal with,
> or try to handle transparently it under verbs.  If we try to handle it under
> verbs, can it be done in one place?  How much would such checks affects the
> performance of post send operations?  And how would immediate or other errors be
> handled when posting queued sends?
> 
> My personal take at the moment is to let the ULPs handle the problem.

My personal take is as of Sean, move it to the ULP, it would be nice to 
have some document educating ULP programmers about in-nature IB races 
such eg RX before ESTABLISHED, RX after DISCONNECTED whatever.

Its wrong (overdoing, asking for bugs, you named it) adding a 
requirement for hw drivers to be able to serve post TX-es while QP is in 
RTR, as Sean pointed how do you return an instant error while posting 
the queued TX in this case??? how do you flush the queued TX if the QP 
gets from RTR into ERROR state??? i guess more minds can produce more 
error schemes which are hell to take care of.

Or.