[ofa-general] RDS flow control

[Olaf Kirch]

On Wednesday 21 May 2008 08:37:54 Olaf Kirch wrote:
> Anyway, when I take the brief detour through SQD state, resetting the RNR
> retry count seems to work.

For those willing to test this, I committed the following patch to
branch future-20080519 on my git tree.

Olaf
-- 
Olaf Kirch  |  --- o --- Nous sommes du soleil we love when we play
okir at lst.de |    / | \   sol.dhoop.naytheet.ah kin.ir.samse.qurax
-----
commit 735bdc95be33db4f21051c0d50090bc128719d98
Author: Olaf Kirch <olaf.kirch at oracle.com>
Date:   Tue May 20 22:41:04 2008 -0700

    RDS: disable RNR retries when flow control is on
    
    When flow control is enabled on a connection, we don't need RNR
    retries. Turning them off allows us to detect potential bugs in
    the credit accounting more quickly.
    
    Signed-off-by: Olaf Kirch <olaf.kirch at oracle.com>

diff --git a/net/rds/ib_cm.c b/net/rds/ib_cm.c
index 20c888d..a49e394 100644
--- a/net/rds/ib_cm.c
+++ b/net/rds/ib_cm.c
@@ -99,21 +99,40 @@ static void rds_ib_connect_complete(struct rds_connection *conn, struct rdma_cm_
 
 	rds_ib_recv_refill(conn, GFP_KERNEL, GFP_HIGHUSER, 1);
 
-	/* Tune the RNR timeout. We use a rather low timeout, but
-	 * not the absolute minimum - this should be tunable.
+	/* Tune RNR behavior. Without flow control, we use a rather
+	 * low timeout, but not the absolute minimum - this should
+	 * be tunable.
 	 *
 	 * We already set the RNR retry count to 7 (which is the
-	 * smallest infinite number :-) above
+	 * smallest infinite number :-) above.
+	 * If flow control is off, we want to change this back to 0
+	 * so that we learn quickly when our credit accounting is
+	 * buggy.
 	 */
-	qp_attr.qp_state = IB_QPS_RTS;
-	qp_attr.min_rnr_timer = IB_RNR_TIMER_000_32;
-	ret = ib_modify_qp(ic->i_cm_id->qp, &qp_attr,
-				IB_QP_STATE | IB_QP_MIN_RNR_TIMER);
-	if (ret) {
-		printk(KERN_NOTICE "ib_modify_qp(IB_QP_MIN_RNR_TIMER, %u): err=%d\n",
-				qp_attr.min_rnr_timer, -ret);
+	if (ic->i_flowctl) {
+		/* It seems we have to take a brief detour through SQD state
+		 * in order to change the RNR retry count. */
+		qp_attr.qp_state = IB_QPS_SQD;
+		ret = ib_modify_qp(ic->i_cm_id->qp, &qp_attr, IB_QP_STATE);
+		if (ret)
+			printk(KERN_NOTICE "ib_modify_qp(IB_QP_STATE, SQD): err=%d\n", -ret);
+
+		qp_attr.rnr_retry = 0;
+		ret = ib_modify_qp(ic->i_cm_id->qp, &qp_attr, IB_QP_RNR_RETRY);
+		if (ret)
+			printk(KERN_NOTICE "ib_modify_qp(IB_QP_RNR_RETRY, 0): err=%d\n", -ret);
+	} else {
+		qp_attr.min_rnr_timer = IB_RNR_TIMER_000_32;
+		ret = ib_modify_qp(ic->i_cm_id->qp, &qp_attr, IB_QP_MIN_RNR_TIMER);
+		if (ret)
+			printk(KERN_NOTICE "ib_modify_qp(IB_QP_MIN_RNR_TIMER): err=%d\n", -ret);
 	}
 
+	qp_attr.qp_state = IB_QPS_RTS;
+	ret = ib_modify_qp(ic->i_cm_id->qp, &qp_attr, IB_QP_STATE);
+	if (ret)
+		printk(KERN_NOTICE "ib_modify_qp(IB_QP_STATE, RTS): err=%d\n", -ret);
+
 	/* update ib_device with this local ipaddr */
 	rds_ibdev = ib_get_client_data(ic->i_cm_id->device, &rds_ib_client);
 	ib_update_ipaddr_for_device(rds_ibdev, conn->c_laddr);