[openib-general] Re: Dual Sided RMPP Support as well as OpenSM Implications

[Sean Hefty]

Hal Rosenstock wrote:
> It can be added but may require an API change and possibly an ABI
> change. It seems that user space code needs to both say and know whether
> dual sided RMPP is supported or not so all mixes of user space and
> kernel code could "work".

Do we really need to support these combinations?  Does anything use the GetMulti 
method today?  Is dual-RMPP used with anything other than MultiPathRecords?

This is my understanding of what needs to happen to support dual-sided RMPP.

Node A sends an RMPP message.  This requires normal RMPP processing.
Node A sends an ACK of the final ACK (I'll call ACK2), giving a new window.
Node B receives ACKs.
Node B sends the response.  This requires normal RMPP processing.

 From the perspective of node A, the RMPP code only needs to know to send ACK2. 
  It can do this based on the method, or per transaction if directed by the client.

Node B is more complex.  It must now wait for ACK2, using timeout and retries of 
ACK1 until ACK2 is received.  And the response that will be generated by the 
client must be delayed until that ACK2 is received.

For node B, it may be simpler to delay handing the request up to a client until 
ACK2 is received.  The only information from ACK2 that's needed when sending the 
response is NewWindowLast.  A client could be expected to give this back to the 
RMPP layer when sending the response.  (A client that lied about NewWindowLast 
should only lead to sending some packets that would be dropped, with the 
transaction aborted.)

So, if we always make the sender of an RMPP message specify NewWindowLast, with 
a default of 1 set when the MAD is allocated, then we can keep RMPP consistent. 
  And we'd only be left handling ACK2.

- Sean