[ofa-general] openfabrics CMA interfaces for iWARP

[Ramaswamy Tummala]

I have a few questions about the openfabrics CMA interfaces for iWARP.
I'd appreciate if anyone could clarify them.

- If RNIC's modify_qp() entry point is called to move the QP state to CLOSING or
   ERROR while there are some WQEs on SQ and RQ, does RNIC flush the incomplete
   WRs on the SQ or RQ? If so, does RNIC wait until the flush is complete
   before returning modify_qp() to the caller? If RNIC does not wait for the
   flush to complete how does the caller know when the flush is complete
   (so that caller can poll CQ to retrieve the CQ entries)?

   [ Another possibility is, when RNIC's modify_qp() entry point called to
   move the QP state to CLOSING while there some WQEs on the SQ, the RNIC would
   internally move the QP state to ERROR. My question still is does RNIC
   wait until the flushing of incomplete WRs from SQ and RQ are done before
   returning modify_qp() to the caller even though it internally transitioned
   the QP state to ERROR. If RNIC does not wait for the flush to complete
   how does the caller know when the flush is complete? ]

- If RNIC's modify_qp() entry point called to move the QP state to CLOSING,
   does RNIC just initiate LLP CLOSE and return to the caller?, or does it wait
   until LLP CLOSE is complete?.

- It appears that RNIC should send IW_CM_EVENT_DISCONNECT event to CMA prior
   to the start of closing or aborting the connection (except in the case
   where the disconnect has been initiated by CMA itself, for example by CMA
   calling modify_qp entry point of RNIC to move the QP state to CLOSING or
   ERROR). Is this correct?

- It appears that RNIC should send IW_CM_EVENT_CLOSE event after the connection
   has been closed. Should this event be sent on both active and passive sides
   after the connection has been closed?

- RNIC has add_ref(struct ib_qp *qp), and rem_ref(struct ib_qp *qp) entry
   points. What is the expected use of CMA calling these entry points? My general
   thinking is that CMA can increase the reference count on QP (i.e. add_ref)
   to prevent the QP from being destroyed by RNIC. But, it is the CMA that
   initiates destroying of QP by calling destroy_qp() entry point of RNIC.
   So, CMA could maintain the reference count for QP in its own private data
   (instead of calling RNIC's add_ref entry point) and not call
   destroy_qp() entry point of RNIC if the reference count is not zero.

- It appears that if RNIC's accept() entry point is called to accept an
   incoming connection, the RNIC, after successful processing of accept,
   would send IW_CM_EVENT_ESTABLISHED event to CMA. What event RNIC should
   send if the call to accept() succeeded, but later RNIC encountered some
   error in sending MPA reply message to the remote peer or some other error?
   In this case although the call to accept() succeeded, the connection could
   still be not be established. So the RNIC can not send
   IW_CM_EVENT_ESTABLISHED event.

- It appears that a client of CMA needs to call rdma_resolve_route() after
   a successful rdma_resolve_addr(). Any reason for the existence of two
   interfaces instead of one interface that combines the functionality of
   both the interfaces?

Thanks,
Ramaswamy.