[ofa-general] RDS flow control

[Or Gerlitz]

Roland Dreier wrote:
>  > Is there a way of changing the RNR retry count back to 0 after establishing
>  > the connection?
>
> Yes... quite complicated but possible.  Basically you have to transition
> to the QP to the "send queue drained" (SQD) state, change the rnr retry
> value in an SQD->SQD transition and then transition back to RTS. 

In case the RTS->SQD->SQD->RTS transition is not applicable or just for the sake of being aware to more solutions, I gave it some thought and its seems possible for you to build a protocol which uses exchange (through the private data carried by the CM messages) whether each side supports credit management, and based on that && HW support of the IB_DEVICE_RC_RNR_NAK_GEN device capability decide what value to place into the QP RNR retries.

On the passive side of the connection its trivial, since the rdma-cm uses the values you place into the conn_param parameters of rdma_accept.

On the active side, things are a bit more complex, but with some changes, I think you would be able to do it also in a different way than the SQD one: the RNR retries are set into the QP once its being moved to RTS (Ready-To-Send). So, if you managed to get the QP into your hands --before-- the RTU is sent (since this point in time is the last synchoronization step provided to you by the IB CM), you could set the RNR retries value accroding to info carried in the REP message sent by the passive (which you have posted in the private data to rdma_accept, etc).

This would be possible, if you enhance the rdma-cm to deliver RDMA_CM_EVENT_CONNECT_RESPONSE event also to IDs created with the PS_TCP port space (eg conditioned on some new field in conn_param) where today its supported only to PS_SDP ones.

Once this change is in place, you will get RDMA_CM_EVENT_CONNECT_RESPONSE event, decide what RNR retry value you want to use, and call rdma_accept providing this value (one more little change is needed here in cma.c), the rdma cm would override the value set by cm_init_qp_rts_attr, see cma_modify_qp_rts -> rdma_init_qp_attr -> ib_cm_init_qp_attr -> cm_init_qp_rts_attr

and you are done...

Or.