[ofa-general] Re: iWARP peer-to-peer CM proposal

[Steve Wise]

Comments in-line below...


Kanevsky, Arkady wrote:
> 
>     Group,
> 
> 
>     below is proposal on how to resolve peer-to-peer iWARP CM issue
>     discovered at interop event.
> 
> 
>     The main issue is that MPA spec (relevant portion of IETF RFC 5044
>     is below) require that
> 
> 
>     connection initiator send first message over the established connection.
> 
> 
>     Multiple MPI implementations and several other apps use peer-to-peer
>     model.
> 
> 
>     So rather then forcing all of them to do it on their own, which will
>     not help with
> 
> 
>     interop between different implementations, the goal is to extend
>     lower layers to provide it.
> 
> 
>      
> 
> 
>     Our first idea was to leave MPA protocol untouched and try to solve
>     this problem
> 
> 
>     in iw_cm. But there are too many complications to it. First, in
>     order to adhere to RFC5044
> 
> 
>     initiator must send first FPDU and responder process it. But since
>     the connection is already
> 
> 
>     established processing FPDU involves ULP on whose behalf the
>     connection is created.
> 
> 
>     So either initiator sends a message which generates completion on
>     responder CQ, thus visible
> 
> 
>     to ULP, or not. 



> In the later case, the only op which can do it is
>     RDMA one, which means
> 
> 
>     that responder somehow provided initiator S-tag which it can use.
>     So, this is an extension
> 
> 
>     to MPA, probably using private data. And that responder upon
>     receiving it destroy this S-tag.
> 
> 
>     In any case this is an extension of MPA.
> 


This stag exchange isn't needed if this RDMA op is a 0B READ.  The 
responder waits for that 0B read and only indicates the rdma connection 
is established to its ULP when it replies to the 0B read.  In this 
scenario, the responder/server side doesn't consume any CQ resources. 
But it would require an IRD of at least 1 to be configured on the QP. 
The initiator still requires an SQ entry, and possibly a CQ entry, for 
initiating the 0B read and handling completion.  But its perhaps a 
little less painful than doing a SEND/RECV exchange.  The read wr could 
be unsignaled so that it won't generate a CQE.  But it still consumes an 
SQ WR slot so the SQ would have to be sized to allow this extra WR. And 
I guess the CQ would also need to be sized accordingly in case the read 
failed.

> 
>     In the former, Send is used but this requires a buffer to be posted
>     to CQ. But since
> 
> 
>     the same CQ (or SharedCQ) can be used by other connections at the
>     same time it can cause
> 
> 
>     the responder CM posted buffer to be consumed by other connection.
>     This is not acceptable.
> 
> 
>      
> 
> 
>     So new we consider extension to MPA protocol.
> 
> 
>     The goal is to be completely backwards compatible to existing version 1.
> 
> 
>     In a nutshell, use a "flag" in the MPA request message which
>     indicates that
> 
> 
>     "ready to receive" message will be send by requestor upon receiving
> 
> 
>     MPA response message with connection acceptance.
> 
> 
>      
> 
> 
>     here are the changes to IETF RFC5044
> 
> 
>      
> 
> 
>     1. 0 1 2 3 0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8
>     9 0 1
>     +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ 0
>     | | + Key (16 bytes containing "MPA ID Req Frame") + 4 | (4D 50 41
>     20 49 44 20 52 65 71 20 46 72 61 6D 65) | + Or (16 bytes containing
>     "MPA ID Rep Frame") + 8 | (4D 50 41 20 49 44 20 52 65 70 20 46 72 61
>     6D 65) | + Or (16 bytes containing "MPA ID Rtr Frame") + 12 | (4D 50
>     41 20 49 44 20 52 74 52 20 46 72 61 6D 65) | +
>     +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ 16
>     |M|C|R|S| Res | Rev | PD_Length |
>     +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ |
>     | ~ ~ ~ Private Data ~ | | | +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ | |
>     +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
> 
> 
>      
> 
> 
>     2. S: indicator in the Req frame whether or not Requestor will send
>     Rtr frame.
> 
> 
>           In Req frame, if set to 1 then Rtr frame will be sent if responder
> 
> 
>         sends Rep frame with accept bit set. 0 indicate that Rtr frame
> 
> 
>         will not be sent.
> 
> 
>         In Rep frame, 0 means that Responder cannot support Rtr frame,
> 
> 
>         while 1 that it is and is waiting for it.
> 
> 
>         (While my preference is to handle this as MPA protocol version
>     matching rules,
> 
> 
>         proposed method will provide complete backwards compatibility)
> 
> 
>         Unused by Rtr frame. That is set to 0 in Rtr frame and ignored
> 
> 
>         by responder.
> 
> 
>      
> 
> 
>         All other bits M,C,R and remainder of Res treated as in MPA ver 1.
> 
> 
>        
> 
> 
>         Rtr frame adhere to C bit as specified in Rep frame
> 
> 

First, the RTR frame _must_ be an FPDU for this to work.  Thus it 
violates the DDP/RDMAP specs because it is an known DDP/RDMAP opcode.

Second, assuming the RTR frame is sent as an FPDU, then this won't work 
with existing RNIC HW.  The HW will post an async error because the 
incoming DDP/RDMAP opcode is unknown.

The only way I see that we can fix this for the existing rnic HW is to 
come up with some way to send a valid RDMAP message from the initiator 
to the responder under the covers -and- have the responder only indicate 
that the connection is established when that FPDU is received.

Chelsio cannot support this hack via a 0B write, but the could support a 
0B read or send/recv exchange.  But as you indicate, this is very 
painful and perhaps impossible to do without impacting the ULP and 
breaking verbs semantics.

(that's why we punted on this a year ago :)


Steve.