[openib-general] Re: Dual Sided RMPP Support as well as OpenSM Implications

[Sean Hefty]

Hal Rosenstock wrote:
>>This is my understanding of what needs to happen to support dual-sided RMPP.
>>
>>Node A sends an RMPP message.  This requires normal RMPP processing.
>>Node A sends an ACK of the final ACK (I'll call ACK2), giving a new window.
>>Node B receives ACKs.
>>Node B sends the response.  This requires normal RMPP processing.
>>
>> From the perspective of node A, the RMPP code only needs to know to send ACK2. 
> 
> There's more to the state machine in turning the direction around in
> terms of the sender becoming the receiver. I thought that this is the
> "harder" direction change.

Can you describe what more is needed that what's listed above?

In terms of compliance, if node A is not, but node B is;
Node B cannot send back the response until ACK2 is received.  Since node A does 
not understand dual-sided RMPP, it will not send ACK2.  Node B will never send 
the response.

If node A is, but node B is not:
Node A will send ACK2, which node B should drop.  Node B will send an RMPP 
message assuming an initial window size of 1.  If node A had set the window 
larger, it may delay the ACK of segment 1.  Node B will eventually timeout and 
resend segment 1.  Most likely, this will cause node A to ACK segment 1, which 
will update the window size at node B.

>>  It can do this based on the method, or per transaction if directed by the client.
> 
> Yes; I was thinking of class/method based approach for this.

Currently, only a MultiPathRecord query requires this.  Why not limit dual-sided 
RMPP to _only_ this request?  All other queries can just use an RMPP message one 
direction, followed by an RMPP message in the other direction.  Beyond 
MultiPathRecord queries, aren't we talking about vendor specific queries anyway?

Personally, I'd vote for removing dual-sided RMPP completely from the spec. 
It's of questionable benefit and complicates the implementation, but it's 
probably a little too late for that.  Couldn't we just keep from defining 
anything else as "dual-sided"?

>>Node B is more complex.  It must now wait for ACK2, using timeout and retries of 
>>ACK1 until ACK2 is received.  And the response that will be generated by the 
>>client must be delayed until that ACK2 is received.
> 
> 
> Yes but isn't much of this already needed for the normal termination
> case or is that part not implemented yet ?

No - ACK2 is a new message unique to dual-sided RMPP transfers (an ACK of an ACK).

>>  The only information from ACK2 that's needed when sending the 
>>response is NewWindowLast.  A client could be expected to give this back to the 
>>RMPP layer when sending the response.  (A client that lied about NewWindowLast 
>>should only lead to sending some packets that would be dropped, with the 
>>transaction aborted.)
> 
> 
> Good idea. That would eliminate the need for some context transfer from
> the receive side to the send side in the RMPP code itself.
> 
> Leaving the NWL up to the client could have the effect you mentioned but
> this is known to the RMPP core and hence we needn't rely on the client
> for this.

For the RMPP core to know what NWL is, it would need to track that information 
between receiving a request and the generation of the reply.  Since the RMPP 
code can't trust the client to generate a reply, it would also need some sort of 
timeout for how long the NWL information is valid.  Tracking NWL by the client 
seems trivial, whereas, it's substantially more complex for the RMPP core.

- Sean