[ofa-general] openfabrics CMA interfaces for iWARP

[Steve Wise]

Ramaswamy Tummala wrote:
> I have a few questions about the openfabrics CMA interfaces for iWARP.
> I'd appreciate if anyone could clarify them.
> 
> - If RNIC's modify_qp() entry point is called to move the QP state to 
> CLOSING or
>   ERROR while there are some WQEs on SQ and RQ, does RNIC flush the 
> incomplete
>   WRs on the SQ or RQ? 

It is really up to the device, but if the rnic supports the RDMAC verbs 
then it will.

> If so, does RNIC wait until the flush is complete
>   before returning modify_qp() to the caller? If RNIC does not wait for the
>   flush to complete how does the caller know when the flush is complete
>   (so that caller can poll CQ to retrieve the CQ entries)?

This is up to the provider also.  I don't think the verbs specify that 
the flush will be done by the time you return from modify_qp().  Its up 
to the application to deal with knowing when the flush is done.

> 
>   [ Another possibility is, when RNIC's modify_qp() entry point called to
>   move the QP state to CLOSING while there some WQEs on the SQ, the RNIC 
> would
>   internally move the QP state to ERROR. My question still is does RNIC
>   wait until the flushing of incomplete WRs from SQ and RQ are done before
>   returning modify_qp() to the caller even though it internally 
> transitioned
>   the QP state to ERROR. If RNIC does not wait for the flush to complete
>   how does the caller know when the flush is complete? ]
> 

I think the answer is no, you cannot depend on the flush being complete 
when you exit modify_qp...

> - If RNIC's modify_qp() entry point called to move the QP state to CLOSING,
>   does RNIC just initiate LLP CLOSE and return to the caller?, or does 
> it wait
>   until LLP CLOSE is complete?.

It initiates the LLP CLOSE.  It does not wait for the close to complete.

> 
> - It appears that RNIC should send IW_CM_EVENT_DISCONNECT event to CMA 
> prior
>   to the start of closing or aborting the connection (except in the case
>   where the disconnect has been initiated by CMA itself, for example by CMA
>   calling modify_qp entry point of RNIC to move the QP state to CLOSING or
>   ERROR). Is this correct?

I'm not sure I understand your question.

> 
> - It appears that RNIC should send IW_CM_EVENT_CLOSE event after the 
> connection
>   has been closed. Should this event be sent on both active and passive 
> sides
>   after the connection has been closed?

Yes.

> 
> - RNIC has add_ref(struct ib_qp *qp), and rem_ref(struct ib_qp *qp) entry
>   points. What is the expected use of CMA calling these entry points? My 
> general
>   thinking is that CMA can increase the reference count on QP (i.e. 
> add_ref)
>   to prevent the QP from being destroyed by RNIC. But, it is the CMA that
>   initiates destroying of QP by calling destroy_qp() entry point of RNIC.
>   So, CMA could maintain the reference count for QP in its own private data
>   (instead of calling RNIC's add_ref entry point) and not call
>   destroy_qp() entry point of RNIC if the reference count is not zero.

The iWCM keeps the ref on the QP while the QP is directly associated 
with a iw_cm_id.

> 
> - It appears that if RNIC's accept() entry point is called to accept an
>   incoming connection, the RNIC, after successful processing of accept,
>   would send IW_CM_EVENT_ESTABLISHED event to CMA. What event RNIC should
>   send if the call to accept() succeeded, but later RNIC encountered some
>   error in sending MPA reply message to the remote peer or some other 
> error?
>   In this case although the call to accept() succeeded, the connection 
> could
>   still be not be established. So the RNIC can not send
>   IW_CM_EVENT_ESTABLISHED event.

It is ok to block in the provider until the connection and qp are bound 
and in FPDU mode.  That's what the chelsio device does. So when the 
ESTABLISHED event is posted, the MPA reply was sent and ACKed and the QP 
/connection moved into FPDU mode.  Then any problems in the connection 
would be posted as IW_CM_EVENT_CLOSE or IW_CM_EVENT_DISCONNECT.

> 
> - It appears that a client of CMA needs to call rdma_resolve_route() after
>   a successful rdma_resolve_addr(). Any reason for the existence of two
>   interfaces instead of one interface that combines the functionality of
>   both the interfaces?

Its an infiniband requirement, I think.  iWARP doesn't do anything for 
resolve route.  The "next hop route" in iWARP terms is actually 
determined as part to address resolution...

Steve.