[ofa-general] performance drop for datagram mode with the new connectx FW

[Eli Cohen]

Or,

I think this problem was introduced with the patch
ipoib_0390_restore_cm_mtu.patch in ofed. This patch attempts to
set the MTU when changing to CM mode to the max defined for CM mode
(e.g. 65520). However the change was done by setting dev->mtu which is
not how it should be done since it will not call any function
registered to be notified on MTU change. The following patch solved the
problem for me and you're welcome to try if it works for you too.


Index: ofa_1_3_dev_kernel/drivers/infiniband/ulp/ipoib/ipoib_cm.c
===================================================================
--- ofa_1_3_dev_kernel.orig/drivers/infiniband/ulp/ipoib/ipoib_cm.c	2008-06-30 11:37:59.000000000 +0300
+++ ofa_1_3_dev_kernel/drivers/infiniband/ulp/ipoib/ipoib_cm.c	2008-06-30 18:53:31.000000000 +0300
@@ -1433,15 +1433,15 @@ static ssize_t set_mode(struct class_dev
 		if (ipoib_cm_max_mtu(dev) > priv->mcast_mtu)
 			ipoib_warn(priv, "mtu > %d will cause multicast packet drops.\n",
 				   priv->mcast_mtu);
-		dev->mtu = ipoib_cm_max_mtu(dev);
 
+		dev_set_mtu(dev, ipoib_cm_max_mtu(dev));
 		ipoib_flush_paths(dev);
 		return count;
 	}
 
 	if (!strcmp(buf, "datagram\n")) {
 		clear_bit(IPOIB_FLAG_ADMIN_CM, &priv->flags);
-		dev->mtu = min(priv->mcast_mtu, dev->mtu);
+		dev_set_mtu(dev, min(priv->mcast_mtu, dev->mtu));
 		ipoib_flush_paths(dev);
 
 		if (priv->ca->flags & IB_DEVICE_IP_CSUM)

You can "fix" this problem by setting the MTU of CM mode manually from
the shell after switching mode in which case you will not need to use
this patch.