[openib-general] Performance Degradation with OFED v. Voltaire(lustre)

[Michael S. Tsirkin]

> So after a bit more testing, setting the route path mtu to 1024 before
> the qp creation (rdma_create_qp()) seems sufficient.

OK, so the following fixes the tavor_quirk flag in cma to actually do something.
Could you please replace the patch cma_tavor_quirk.patch with this one,
set tavor_quirk option for cma module, and see if this works as expected?

Unpack OFED 1.1, copy the following to
OFED-1.1/openib-1.1/kernel_patches/fixes/cma_tavor_quirk.patch
removing the patch by the same name that is in OFED
(also remove xxx_cma_tavor_quirk.txt or other patches if you put them there)
and then pack OFED 1.1 and rebuild.


Thanks,

-----------------

Tavor systems get better performance with 1K MTU. Since there does
not seem to be any way to find out whether the remote system uses Tavor,
add an option to limit the MTU globally.

Signed-off-by: Michael S. Tsirkin <mst at mellanox.co.il>

---

diff --git a/drivers/infiniband/core/cma.c b/drivers/infiniband/core/cma.c
index 50150c8..261bf45 100644
--- a/drivers/infiniband/core/cma.c
+++ b/drivers/infiniband/core/cma.c
@@ -48,6 +48,10 @@ MODULE_AUTHOR("Sean Hefty");
 MODULE_DESCRIPTION("Generic RDMA CM Agent");
 MODULE_LICENSE("Dual BSD/GPL");
 
+static int tavor_quirk = 0;
+module_param_named(tavor_quirk, tavor_quirk, int, 0644);
+MODULE_PARM_DESC(tavor_quirk, "Tavor performance quirk: limit MTU to 1K if > 0");
+
 #define CMA_CM_RESPONSE_TIMEOUT 20
 #define CMA_MAX_CM_RETRIES 3
 
@@ -1138,6 +1142,7 @@ static int cma_query_ib_route(struct rdma_id_private *id_priv, int timeout_ms,
 {
 	struct rdma_dev_addr *addr = &id_priv->id.route.addr.dev_addr;
 	struct ib_sa_path_rec path_rec;
+	ib_sa_comp_mask mask;
 
 	memset(&path_rec, 0, sizeof path_rec);
 	ib_addr_get_sgid(addr, &path_rec.sgid);
@@ -1145,8 +1150,15 @@ static int cma_query_ib_route(struct rdma_id_private *id_priv, int timeout_ms,
 	path_rec.pkey = cpu_to_be16(ib_addr_get_pkey(addr));
 	path_rec.numb_path = 1;
 
+	if (tavor_quirk) {
+		path_rec.mtu_selector = IB_SA_LT;
+		path_rec.mtu = IB_MTU_2048;
+		mask = IB_SA_PATH_REC_MTU_SELECTOR | IB_SA_PATH_REC_MTU;
+	} else
+		mask = 0;
+
 	id_priv->query_id = ib_sa_path_rec_get(id_priv->id.device,
-				id_priv->id.port_num, &path_rec,
+				id_priv->id.port_num, &path_rec, mask |
 				IB_SA_PATH_REC_DGID | IB_SA_PATH_REC_SGID |
 				IB_SA_PATH_REC_PKEY | IB_SA_PATH_REC_NUMB_PATH,
 				timeout_ms, GFP_KERNEL,

-- 
MST