Fwd: [ofa-general] RDS flow control

[Or Gerlitz]

---------- Forwarded message ----------
From: Or Gerlitz <ogerlitz at voltaire.com>
Date: Wed, May 21, 2008 at 9:41 AM
Subject: Re: [ofa-general] RDS flow control
To: Olaf Kirch <olaf.kirch at oracle.com>
Cc: Roland Dreier <rdreier at cisco.com>, rds-devel at oss.oracle.com,
general at lists.openfabrics.org


Roland Dreier wrote:
>
>  > Is there a way of changing the RNR retry count back to 0 after
establishing
>  > the connection?
>
> Yes... quite complicated but possible.  Basically you have to transition
> to the QP to the "send queue drained" (SQD) state, change the rnr retry
> value in an SQD->SQD transition and then transition back to RTS.

In case the RTS->SQD->SQD->RTS transition is not applicable or just for the
sake of being aware to more solutions, I gave it some thought and its seems
possible for you to build a protocol which uses exchange (through the
private data carried by the CM messages) whether each side supports credit
management, and based on that && HW support of the IB_DEVICE_RC_RNR_NAK_GEN
device capability decide what value to place into the QP RNR retries.

On the passive side of the connection its trivial, since the rdma-cm uses
the values you place into the conn_param parameters of rdma_accept.

On the active side, things are a bit more complex, but with some changes, I
think you would be able to do it also in a different way than the SQD one:
the RNR retries are set into the QP once its being moved to RTS
(Ready-To-Send). So, if you managed to get the QP into your hands --before--
the RTU is sent (since this point in time is the last synchoronization step
provided to you by the IB CM), you could set the RNR retries value accroding
to info carried in the REP message sent by the passive (which you have
posted in the private data to rdma_accept, etc).

This would be possible, if you enhance the rdma-cm to deliver
RDMA_CM_EVENT_CONNECT_RESPONSE event also to IDs created with the PS_TCP
port space (eg conditioned on some new field in conn_param) where today its
supported only to PS_SDP ones.

Once this change is in place, you will get RDMA_CM_EVENT_CONNECT_RESPONSE
event, decide what RNR retry value you want to use, and call rdma_accept
providing this value (one more little change is needed here in cma.c), the
rdma cm would override the value set by cm_init_qp_rts_attr, see
cma_modify_qp_rts -> rdma_init_qp_attr -> ib_cm_init_qp_attr ->
cm_init_qp_rts_attr

and you are done...

Or.
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.openfabrics.org/pipermail/general/attachments/20090213/b94aca3f/attachment.html>