[ewg] Re: [ofa-general] Re: [PATCH] Request For Comments:

[Steve Wise]

 From RFC 5044, section 7.1.2 "Connection Startup Rules", Page 29:

4.  MPA Responder mode implementations MUST receive and validate at
       least one FPDU before sending any FPDUs or Markers.

       Note: This requirement is present to allow the Initiator time to
           get its receiver into Full Operation before an FPDU arrives,
           avoiding potential race conditions at the Initiator.  This
           was also subject to some debate in the work group before
           rough consensus was reached.  Eliminating this requirement
           would allow faster startup in some types of applications.
           However, that would also make certain implementations
           (particularly "dual stack") much harder.




Steve Wise wrote:
> Sean Hefty wrote:
>>> The requirement is mostly driven from the receiving side.  For cxgb3 it
>>> is anyway...
>>>     
>>
>> Maybe you can help me understand the spec here.  If we ignore this feature for a
>> minute, then the side that calls rdma_connect() must instead issue the first
>> 'send' request to the server.  Can the first 'send' be a 0B rdma write or read?
>>   
> According to the MPI IETF RFC, the initiator must send the first 
> FPDU.  That could be anything.  The spec leaves it up to the ULP.
>
>> Why wouldn't the target of that request not have to transition to connected?
>>
>>   
> I don't understand this question?  What does 'transition to connected' 
> mean?
>
> The requirement is that the responder (the side that issues the 
> rdma_accept in rdma-cma terms) _cannot_ send an FPDU until it first 
> receives one from the initiator.   How that is enforces is an 
> implementation detail.  The responder driver could hold off on the 
> ESTABLISHED event until it receives the first FPDU.  Or it could stall 
> SQ processing until the first FPDU is received yet still indicate that 
> the connection is ESTABLISHED.
>
>> Is the issue that there's no way for the receiving FW/driver to know that this
>> has occurred so that it can signal that the connection has been established?
>> I.e. a client that does this must signal the server that things are ready
>> through some out of band means.
>>
>>   
> I don't understand what you're getting at exactly. 
>
> The issue is that the server doesn't know when the client receives the 
> MPA Start Response and has successfully transitioned the connection 
> into RDMA mode.  IF the server sends an FPDU immediately following the 
> MPA Start Response (which is in streaming mode), then its possible for 
> that first FPDU to get passed up to the driver/ULP as streaming mode 
> data.  Which breaks everything.  Soooo, the spec says the server 
> cannot send an FPDU until it first receives one and thus _knows_ the 
> client is in RDMA mode (by virtue of the fact that the client sent and 
> FPDU).
>
>
>>> server sends MPA Start response with "lets do RTR and send me X"  where
>>> X could be 0B write, 0B read request or 0B send.
>>>     
>>
>> Are there any restrictions where a client may not be able to issue what the
>> server requests?  E.g. the hardware doesn't issue 0B writes.
>>
>>   
>
> Well I guess there could be.  The concensus within the iWARP vendors 
> at Reno was that 0B read would ok.  During the previous discussion on 
> this list shortly after Reno, issues where raised that we should allow 
> other types.
>
> We could make the MPA start request have more info than "I can do 
> RTR".   It could have "Here are the RTR msgs I can send".    Does that 
> help?
>
>
>
> Steve.
>
>