[ofa-general] openfabrics CMA interfaces for iWARP

[Ramaswamy Tummala]

Thanks Steve for answering the questions.

 >> - It appears that RNIC should send IW_CM_EVENT_DISCONNECT event to CMA
 >> prior
 >>   to the start of closing or aborting the connection (except in the case
 >>   where the disconnect has been initiated by CMA itself, for example
 >> by CMA
 >>   calling modify_qp entry point of RNIC to move the QP state to
 >> CLOSING or
 >>   ERROR). Is this correct?
 >
 > I'm not sure I understand your question.

Basically, I am trying to understand when RNIC should send
IW_CM_EVENT_DISCONNECT event.

Thank you,
Ramaswamy.



Steve Wise wrote:
> 
> 
> Ramaswamy Tummala wrote:
>> I have a few questions about the openfabrics CMA interfaces for iWARP.
>> I'd appreciate if anyone could clarify them.
>>
>> - If RNIC's modify_qp() entry point is called to move the QP state to 
>> CLOSING or
>>   ERROR while there are some WQEs on SQ and RQ, does RNIC flush the 
>> incomplete
>>   WRs on the SQ or RQ? 
> 
> It is really up to the device, but if the rnic supports the RDMAC verbs 
> then it will.
> 
>> If so, does RNIC wait until the flush is complete
>>   before returning modify_qp() to the caller? If RNIC does not wait 
>> for the
>>   flush to complete how does the caller know when the flush is complete
>>   (so that caller can poll CQ to retrieve the CQ entries)?
> 
> This is up to the provider also.  I don't think the verbs specify that 
> the flush will be done by the time you return from modify_qp().  Its up 
> to the application to deal with knowing when the flush is done.
> 
>>
>>   [ Another possibility is, when RNIC's modify_qp() entry point called to
>>   move the QP state to CLOSING while there some WQEs on the SQ, the 
>> RNIC would
>>   internally move the QP state to ERROR. My question still is does RNIC
>>   wait until the flushing of incomplete WRs from SQ and RQ are done 
>> before
>>   returning modify_qp() to the caller even though it internally 
>> transitioned
>>   the QP state to ERROR. If RNIC does not wait for the flush to complete
>>   how does the caller know when the flush is complete? ]
>>
> 
> I think the answer is no, you cannot depend on the flush being complete 
> when you exit modify_qp...
> 
>> - If RNIC's modify_qp() entry point called to move the QP state to 
>> CLOSING,
>>   does RNIC just initiate LLP CLOSE and return to the caller?, or does 
>> it wait
>>   until LLP CLOSE is complete?.
> 
> It initiates the LLP CLOSE.  It does not wait for the close to complete.
> 
>>
>> - It appears that RNIC should send IW_CM_EVENT_DISCONNECT event to CMA 
>> prior
>>   to the start of closing or aborting the connection (except in the case
>>   where the disconnect has been initiated by CMA itself, for example 
>> by CMA
>>   calling modify_qp entry point of RNIC to move the QP state to 
>> CLOSING or
>>   ERROR). Is this correct?
> 
> I'm not sure I understand your question.
> 
>>
>> - It appears that RNIC should send IW_CM_EVENT_CLOSE event after the 
>> connection
>>   has been closed. Should this event be sent on both active and 
>> passive sides
>>   after the connection has been closed?
> 
> Yes.
> 
>>
>> - RNIC has add_ref(struct ib_qp *qp), and rem_ref(struct ib_qp *qp) entry
>>   points. What is the expected use of CMA calling these entry points? 
>> My general
>>   thinking is that CMA can increase the reference count on QP (i.e. 
>> add_ref)
>>   to prevent the QP from being destroyed by RNIC. But, it is the CMA that
>>   initiates destroying of QP by calling destroy_qp() entry point of RNIC.
>>   So, CMA could maintain the reference count for QP in its own private 
>> data
>>   (instead of calling RNIC's add_ref entry point) and not call
>>   destroy_qp() entry point of RNIC if the reference count is not zero.
> 
> The iWCM keeps the ref on the QP while the QP is directly associated 
> with a iw_cm_id.
> 
>>
>> - It appears that if RNIC's accept() entry point is called to accept an
>>   incoming connection, the RNIC, after successful processing of accept,
>>   would send IW_CM_EVENT_ESTABLISHED event to CMA. What event RNIC should
>>   send if the call to accept() succeeded, but later RNIC encountered some
>>   error in sending MPA reply message to the remote peer or some other 
>> error?
>>   In this case although the call to accept() succeeded, the connection 
>> could
>>   still be not be established. So the RNIC can not send
>>   IW_CM_EVENT_ESTABLISHED event.
> 
> It is ok to block in the provider until the connection and qp are bound 
> and in FPDU mode.  That's what the chelsio device does. So when the 
> ESTABLISHED event is posted, the MPA reply was sent and ACKed and the QP 
> /connection moved into FPDU mode.  Then any problems in the connection 
> would be posted as IW_CM_EVENT_CLOSE or IW_CM_EVENT_DISCONNECT.
> 
>>
>> - It appears that a client of CMA needs to call rdma_resolve_route() 
>> after
>>   a successful rdma_resolve_addr(). Any reason for the existence of two
>>   interfaces instead of one interface that combines the functionality of
>>   both the interfaces?
> 
> Its an infiniband requirement, I think.  iWARP doesn't do anything for 
> resolve route.  The "next hop route" in iWARP terms is actually 
> determined as part to address resolution...
> 
> Steve.