[ofa-general] [PATCH RFC] ib_mthca: avoid recycling FMR R_Keys too soon

[Or Gerlitz]

Jack Morgenstein wrote:
> As long as the underlying mpt index is not played with,
> there is no requirement that the sequence bits start from 0. Its just
> sufficient to guarantee that the same (full) key not be allocated twice
> before performing an unmap/SYNC_TPT.

> Index: ofed_kernel/drivers/infiniband/hw/mthca/mthca_mr.c
> ===================================================================
> --- ofed_kernel.orig/drivers/infiniband/hw/mthca/mthca_mr.c 2008-02-21 10:32:50.000000000 +0200
> +++ ofed_kernel/drivers/infiniband/hw/mthca/mthca_mr.c 2008-02-21 12:22:54.393777000 +0200

> @@ -839,11 +839,6 @@ void mthca_arbel_fmr_unmap(struct mthca_
> if (!fmr->maps)
> return;
> - key = arbel_key_to_hw_index(fmr->ibmr.lkey);
> - key &= dev->limits.num_mpts - 1;
> - key = adjust_key(dev, key);
> - fmr->ibmr.lkey = fmr->ibmr.rkey = arbel_hw_index_to_key(key);
> -
> fmr->maps = 0;
> *(u8 *) fmr->mem.arbel.mpt = MTHCA_MPT_STATUS_SW;
> ==============================================================

> This can be done for mlx4 (mlx4_fmr_unmap) and tavor (mthca_tavor_fmr_unmap) as well.

As far as I understand under Sinai you must issue an adjust_key call 
when the key is about to wraparound, correct?

Or.

> commit 608d8268be392444f825b4fc8fc7c8b509627129
> Author: Michael S. Tsirkin <mst at dev.mellanox.co.il>
> Date:   Mon Apr 16 17:04:55 2007 +0300
> 
>     IB/mthca: Fix data corruption after FMR unmap on Sinai
> 
>     In mthca_arbel_fmr_unmap(), the high bits of the key are masked off.
>     This gets rid of the effect of adjust_key(), which makes sure that
>     bits 3 and 23 of the key are equal when the Sinai throughput
>     optimization is enabled, and so it may happen that an FMR will end up
>     with bits 3 and 23 in the key being different.  This causes data
>     corruption, because when enabling the throughput optimization, the
>     driver promises the HCA firmware that bits 3 and 23 of all memory keys
>     will always be equal.
> 
>     Fix by re-applying adjust_key() after masking the key.
> 
>     Thanks to Or Gerlitz for reproducing the problem, and Ariel Shahar for
>     help in debug.
> 
>     Signed-off-by: Michael S. Tsirkin <mst at dev.mellanox.co.il>
>     Signed-off-by: Roland Dreier <rolandd at cisco.com>