[openib-general] [PATCH 1/2] perftest: enhancement to rdma_bw to allow use of RDMA CM

[Pradipta Kumar Banerjee]

Michael S. Tsirkin wrote:
> Quoting r. Pradipta Kumar Banerjee <bpradip at in.ibm.com>:
>> Subject: Re: [openib-general] [PATCH 1/2] perftest: enhancement to rdma_bw to allow use of RDMA CM
>>
>> Michael S. Tsirkin wrote:
>>> Quoting r. Pradipta Kumar Banerjee <bpradip at in.ibm.com>:
>>>> IMO using rcq seems to be a generic and better solution.
>>> Hmm, I see. Need to document the message format then.
>>> We are only pasing the vaddr there, right?
>>>
>> Michael,
>>    Actually 'rcq' is being used for handling the 'start' and 'done' messages.
>> As for the lid, qpn, psn, rkey and vaddr, these gets exchanged as part of the 
>> rdma_listen/rdma_connect calls. See pp_server_connect and pp_client_connect.
>> OTH I tried testing rdma_bw on Ammasso iWARP without exchanging the 'start' and 
>> 'done' messages and it worked. I am not sure if this is the right thing to do. 
>> Maybe Steve can throw more light on this.
> 
> This makes sense. But why do we need the start message then?
> 
Hi Michael,
As per my understanding of the mpa/iwarp spec, here is the reason for the 
'start' message .

The receipt of RDMA_CM_EVENT_ESTABLISHED event by the user mode application, as 
part of the rdma_listen/rdma_accept/rdma_connect sequence, is an indication that 
the MPA startup sequence has successfully completed.

Taking the example of the cxgb3 driver - the driver generates the
IW_CM_EVENT_ESTABLISHED event and goes to the FPDU mode only after successful
completion of the MPA startup sequence. This event eventually gets passed to the
user as RDMA_CM_EVENT_ESTABLISHED. (function cm_conn_est_handler in core/iwcm.c,
function cma_iw_handler in core/cma.c, function established_upcall in
hw/cxgb3/iwch_cm.c)

Now, as per the MPA spec, the complete startup negotation is over only when the 
initiator sends the first FPDU frame and the responder receives it.

[
http://www.ietf.org/internet-drafts/draft-ietf-rddp-mpa-05.txt

7.1.2  Connection Startup Rules

  4.  MPA Responder mode implementations MUST receive and validate at
        least one FPDU before sending any FPDUs or Markers.

        Note: this requirement is present to allow the Initiator time to
            get its receiver into Full Operation before an FPDU arrives,
            avoiding potential race conditions at the Initiator.  This
            was also subject to some debate in the work group before
            rough consensus was reached.  Eliminating this requirement
            would allow faster startup in some types of applications.
            However, that would also make certain implementations
            (particularly "dual stack") much harder.
]

So the 'pp_send_start/pp_wait_for_start' actually takes care of the above 
requirement, initiator (client) sending a 'start message' as FPDU and the 
responder (server) receiving the same correctly.


____Client____                            ____Server____
                                     Waits for MPA Request Frame
Sends MPA Request Frame
Waits for incoming
MPA Reply Frame
                                     Receives MPA Request Frame
                                     Enables FPDU decoding
                                     (doesn't send any FPDUs)
Receives MPA Reply Frame

< The above communication sequence is taken care by the rdma_listen - 
rdma_accept/rdma_connect calls and subsequent generation of 
RDMA_CM_EVENT_ESTABLISHED event >

Send first FPDU
                                     Receive first FPDU

< The above communication sequence is taken care by  the 
pp_wait_for_start/pp_send_start calls >

Hope I have made myself clear without generating further confusions :-)

Thanks,
Pradipta Kumar.