[openib-general] Performance Degradation with OFED v. Voltaire

[Michael S. Tsirkin]

> >> I guess one last thing to check would be the MTU being used for the RC
> >> connections.  Since this is PCI-X HW then the MTU should be 1024 for
> >> best throughput (instead of the max MTU of 2048).
> 
> > The MTU issue is described in the OFED release notes.
> > You must turn the Tavor work-around for it on in opensm.
> > This was introduced late in release cycle to it was deemed safer
> > to make it off by default.
> 
> Michael,
> 
> Let me see i follow you correct: a user must enable the tavor quirk in 
> the **openSM** ? what about the cma_tavor_quirk? and what about users 
> who want to use OFED with commercial/3rd party SMs ??? looking in the 
> OFED 1.1 docs it is mentioned that either way should work.

Right. But CMA quirk can only work if OFED CMA is initiating the connection
from Tavor (i.e. it can't handle Arbel->Tavor case).
Enabling this in Opensm solves the problem for all ULPs, and in all cases -
whether Tavor is active or passive side in the connection.
So fixing this in the SM is clearly the best solution.

Further, as you point out the cma quirk patch in OFED looks broken :(.

> Looking on kernel_patches/fixes/cma_tavor_quirk.patch of OFED 1.1 
> (below) the thing seems to me uncompleted as the 
> IB_SA_PATH_REC_MTU_SELECTOR and IB_SA_PATH_REC_MTU bits are not set in 
> the component mask of the path record query done by the cma, am i 
> missing something?
> 
> Or.

Correct, looks like that bit is missing.

> Tavor systems get better performance with 1K MTU. Since there does
> not seem to be any way to find out whether the remote system uses Tavor,
> add an option to limit the MTU globally.
> 
> Signed-off-by: Michael S. Tsirkin <mst at mellanox.co.il>
> 
> Index: linux-2.6.18-rc2-devel/drivers/infiniband/core/cma.c
> ===================================================================
> --- linux-2.6.18-rc2-devel.orig/drivers/infiniband/core/cma.c   2006-09-11 16:01:37.000000000 +0300
> +++ linux-2.6.18-rc2-devel/drivers/infiniband/core/cma.c        2006-09-13 18:51:45.000000000 +0300
> @@ -48,6 +48,10 @@ MODULE_AUTHOR("Sean Hefty");
>  MODULE_DESCRIPTION("Generic RDMA CM Agent");
>  MODULE_LICENSE("Dual BSD/GPL");
> 
> +static int tavor_quirk = 0;
> +module_param_named(tavor_quirk, tavor_quirk, int, 0644);
> +MODULE_PARM_DESC(tavor_quirk, "Tavor performance quirk: limit MTU to 1K if > 0");
> +
>  #define CMA_CM_RESPONSE_TIMEOUT 20
>  #define CMA_MAX_CM_RETRIES 3
> 
> @@ -1123,6 +1127,11 @@ static int cma_query_ib_route(struct rdm
>         path_rec.pkey = cpu_to_be16(ib_addr_get_pkey(addr));
>         path_rec.numb_path = 1;
> 
> +       if (tavor_quirk) {
> +               path_rec.mtu_selector = IB_SA_LT;
> +               path_rec.mtu = IB_MTU_2048;
> +       }
> +
>         id_priv->query_id = ib_sa_path_rec_get(id_priv->id.device,
>                                 id_priv->id.port_num, &path_rec,
>                                 IB_SA_PATH_REC_DGID | IB_SA_PATH_REC_SGID |

-- 
MST