[openib-general] Re: Dual Sided RMPP Support as well as OpenSM Implications

[Hal Rosenstock]

On Mon, 2006-04-10 at 13:33, Sean Hefty wrote:
> Hal Rosenstock wrote:
> >>This is my understanding of what needs to happen to support dual-sided RMPP.
> >>
> >>Node A sends an RMPP message.  This requires normal RMPP processing.
> >>Node A sends an ACK of the final ACK (I'll call ACK2), giving a new window.
> >>Node B receives ACKs.
> >>Node B sends the response.  This requires normal RMPP processing.
> >>
> >> From the perspective of node A, the RMPP code only needs to know to send ACK2. 
> > 
> > There's more to the state machine in turning the direction around in
> > terms of the sender becoming the receiver. I thought that this is the
> > "harder" direction change.
> 
> Can you describe what more is needed that what's listed above?

I was referring to comparing the direction switch flows (Figure 181 p.
791) requires more than switch to DS in Figure 179 p. 787).

> In terms of compliance, if node A is not, but node B is;

Is = DS and not is not DS, right ?

Just out of curiosity, where's the compliance for this ? What are you
referring to here ?

> Node B cannot send back the response until ACK2 is received.  Since node A does 
> not understand dual-sided RMPP, it will not send ACK2.  Node B will never send 
> the response.

Correct. It would time out. But wouldn't it be better if the transaction
were aborted with some explicit status for this ?

> If node A is, but node B is not:
> Node A will send ACK2, which node B should drop.

Yes, figure 179 for receiver termination flow (IsDS false direction)
shows that packet as discarded with an Abort (BadT) sent.

> Node B will send an RMPP message assuming an initial window size of 1.  
> If node A had set the window 
> larger, it may delay the ACK of segment 1.  Node B will eventually timeout and 
> resend segment 1.  Most likely, this will cause node A to ACK segment 1, which 
> will update the window size at node B.

I'm not following you on this part.

And you just made me realize that the dual sidedness may be more than
binary (based on your comment below on vendor class 2).

> >>  It can do this based on the method, or per transaction if directed by the client.
> > 
> > Yes; I was thinking of class/method based approach for this.
> 
> Currently, only a MultiPathRecord query requires this.  Why not limit dual-sided 
> RMPP to _only_ this request?  

That would work for now. One future issue would be vendor class 2 needs
here.

> All other queries can just use an RMPP message one 
> direction, followed by an RMPP message in the other direction.

I don't understand what you mean here. That's not the way it works from
my understanding. If both the request and response are RMPP messages,
isn't this dual sided ?

> Beyond MultiPathRecord queries, aren't we talking about vendor specific queries anyway?

Yes.

> Personally, I'd vote for removing dual-sided RMPP completely from the spec. 
> It's of questionable benefit and complicates the implementation, but it's 
> probably a little too late for that.  Couldn't we just keep from defining 
> anything else as "dual-sided"?

I think the issue is turning things around but I'm not positive. I was
wondering about this in a slightly different way: as to why the
direction switch ? My initial foray into this area was to do just what
you said: two single sided RMPP transfers in opposite direction. In my
simple test case, the request was short (1 MAD) but that could be
changed. I haven't figured out the reason for the turnaround ACK but I
know the people who architected this although most have left the group
and am quite confident that this wouldn't just be there if it weren't
needed. (I'll eat my words later if necessary :-)

> >>Node B is more complex.  It must now wait for ACK2, using timeout and retries of 
> >>ACK1 until ACK2 is received.  And the response that will be generated by the 
> >>client must be delayed until that ACK2 is received.
> > 
> > 
> > Yes but isn't much of this already needed for the normal termination
> > case or is that part not implemented yet ?
> 
> No - ACK2 is a new message unique to dual-sided RMPP transfers (an ACK of an ACK).

We're talking about Figure 179, right ? If so, most of that needs to be
there already down to the Type decision (without the ACK direction
implemented).

Yes, ACK2 is new but this doesn't seem like much to add. The delay of
the client response would also be "new" and that seems harder.

> >>  The only information from ACK2 that's needed when sending the 
> >>response is NewWindowLast.  A client could be expected to give this back to the 
> >>RMPP layer when sending the response.  (A client that lied about NewWindowLast 
> >>should only lead to sending some packets that would be dropped, with the 
> >>transaction aborted.)
> > 
> > 
> > Good idea. That would eliminate the need for some context transfer from
> > the receive side to the send side in the RMPP code itself.
> > 
> > Leaving the NWL up to the client could have the effect you mentioned but
> > this is known to the RMPP core and hence we needn't rely on the client
> > for this.
> 
> For the RMPP core to know what NWL is, it would need to track that information 
> between receiving a request and the generation of the reply.  Since the RMPP 
> code can't trust the client to generate a reply, it would also need some sort of 
> timeout for how long the NWL information is valid.  Tracking NWL by the client 
> seems trivial, whereas, it's substantially more complex for the RMPP core.

Yes, that's the tradeoff. I agree it's way simpler to rely on the client
for this than to implement it in the core.

-- Hal
> 
> - Sean