[openib-general] Performance Degradation with OFED v. Voltaire

[Or Gerlitz]

Michael S. Tsirkin wrote:
>> I guess one last thing to check would be the MTU being used for the RC
>> connections.  Since this is PCI-X HW then the MTU should be 1024 for
>> best throughput (instead of the max MTU of 2048).

> The MTU issue is described in the OFED release notes.
> You must turn the Tavor work-around for it on in opensm.
> This was introduced late in release cycle to it was deemed safer
> to make it off by default.

Michael,

Let me see i follow you correct: a user must enable the tavor quirk in 
the **openSM** ? what about the cma_tavor_quirk? and what about users 
who want to use OFED with commercial/3rd party SMs ??? looking in the 
OFED 1.1 docs it is mentioned that either way should work.

Looking on kernel_patches/fixes/cma_tavor_quirk.patch of OFED 1.1 
(below) the thing seems to me uncompleted as the 
IB_SA_PATH_REC_MTU_SELECTOR and IB_SA_PATH_REC_MTU bits are not set in 
the component mask of the path record query done by the cma, am i 
missing something?

Or.

> Tavor systems get better performance with 1K MTU. Since there does
> not seem to be any way to find out whether the remote system uses Tavor,
> add an option to limit the MTU globally.
> 
> Signed-off-by: Michael S. Tsirkin <mst at mellanox.co.il>
> 
> Index: linux-2.6.18-rc2-devel/drivers/infiniband/core/cma.c
> ===================================================================
> --- linux-2.6.18-rc2-devel.orig/drivers/infiniband/core/cma.c   2006-09-11 16:01:37.000000000 +0300
> +++ linux-2.6.18-rc2-devel/drivers/infiniband/core/cma.c        2006-09-13 18:51:45.000000000 +0300
> @@ -48,6 +48,10 @@ MODULE_AUTHOR("Sean Hefty");
>  MODULE_DESCRIPTION("Generic RDMA CM Agent");
>  MODULE_LICENSE("Dual BSD/GPL");
> 
> +static int tavor_quirk = 0;
> +module_param_named(tavor_quirk, tavor_quirk, int, 0644);
> +MODULE_PARM_DESC(tavor_quirk, "Tavor performance quirk: limit MTU to 1K if > 0");
> +
>  #define CMA_CM_RESPONSE_TIMEOUT 20
>  #define CMA_MAX_CM_RETRIES 3
> 
> @@ -1123,6 +1127,11 @@ static int cma_query_ib_route(struct rdm
>         path_rec.pkey = cpu_to_be16(ib_addr_get_pkey(addr));
>         path_rec.numb_path = 1;
> 
> +       if (tavor_quirk) {
> +               path_rec.mtu_selector = IB_SA_LT;
> +               path_rec.mtu = IB_MTU_2048;
> +       }
> +
>         id_priv->query_id = ib_sa_path_rec_get(id_priv->id.device,
>                                 id_priv->id.port_num, &path_rec,
>                                 IB_SA_PATH_REC_DGID | IB_SA_PATH_REC_SGID |