Fwd: [ofa-general] RDS flow control

[Andy Grover]

Thanks Or! This is exactly the kind of info I was looking for.

Regards -- Andy

Or Gerlitz wrote:
> ---------- Forwarded message ----------
> From: Or Gerlitz <ogerlitz at voltaire.com>
> Date: Wed, May 21, 2008 at 9:41 AM
> Subject: Re: [ofa-general] RDS flow control
> To: Olaf Kirch <olaf.kirch at oracle.com>
> Cc: Roland Dreier <rdreier at cisco.com>, rds-devel at oss.oracle.com,
> general at lists.openfabrics.org
> 
> 
> Roland Dreier wrote:
>>  > Is there a way of changing the RNR retry count back to 0 after
> establishing
>>  > the connection?
>>
>> Yes... quite complicated but possible.  Basically you have to transition
>> to the QP to the "send queue drained" (SQD) state, change the rnr retry
>> value in an SQD->SQD transition and then transition back to RTS.
> 
> In case the RTS->SQD->SQD->RTS transition is not applicable or just for the
> sake of being aware to more solutions, I gave it some thought and its seems
> possible for you to build a protocol which uses exchange (through the
> private data carried by the CM messages) whether each side supports credit
> management, and based on that && HW support of the IB_DEVICE_RC_RNR_NAK_GEN
> device capability decide what value to place into the QP RNR retries.
> 
> On the passive side of the connection its trivial, since the rdma-cm uses
> the values you place into the conn_param parameters of rdma_accept.
> 
> On the active side, things are a bit more complex, but with some changes, I
> think you would be able to do it also in a different way than the SQD one:
> the RNR retries are set into the QP once its being moved to RTS
> (Ready-To-Send). So, if you managed to get the QP into your hands --before--
> the RTU is sent (since this point in time is the last synchoronization step
> provided to you by the IB CM), you could set the RNR retries value accroding
> to info carried in the REP message sent by the passive (which you have
> posted in the private data to rdma_accept, etc).
> 
> This would be possible, if you enhance the rdma-cm to deliver
> RDMA_CM_EVENT_CONNECT_RESPONSE event also to IDs created with the PS_TCP
> port space (eg conditioned on some new field in conn_param) where today its
> supported only to PS_SDP ones.
> 
> Once this change is in place, you will get RDMA_CM_EVENT_CONNECT_RESPONSE
> event, decide what RNR retry value you want to use, and call rdma_accept
> providing this value (one more little change is needed here in cma.c), the
> rdma cm would override the value set by cm_init_qp_rts_attr, see
> cma_modify_qp_rts -> rdma_init_qp_attr -> ib_cm_init_qp_attr ->
> cm_init_qp_rts_attr
> 
> and you are done...
> 
> Or.
>