[ofa-general] [PATCH RFC] ib_mthca: avoid recycling FMR R_Keys too soon

[Jack Morgenstein]

As long as the underlying mpt index is not played with,
there is no requirement that the sequence bits start from 0.  Its just
sufficient to guarantee that the same (full) key not be allocated twice
before performing an unmap/SYNC_TPT.

Thus, there is no problem with the change that Olaf requests. The lines between
the #if 0 and #endif can just be entirely deleted:

Index: ofed_kernel/drivers/infiniband/hw/mthca/mthca_mr.c
===================================================================
--- ofed_kernel.orig/drivers/infiniband/hw/mthca/mthca_mr.c	2008-02-21 10:32:50.000000000 +0200
+++ ofed_kernel/drivers/infiniband/hw/mthca/mthca_mr.c	2008-02-21 12:22:54.393777000 +0200
@@ -839,11 +839,6 @@ void mthca_arbel_fmr_unmap(struct mthca_
 	if (!fmr->maps)
 		return;
 
-	key = arbel_key_to_hw_index(fmr->ibmr.lkey);
-	key &= dev->limits.num_mpts - 1;
-	key = adjust_key(dev, key);
-	fmr->ibmr.lkey = fmr->ibmr.rkey = arbel_hw_index_to_key(key);
-
 	fmr->maps = 0;
 
 	*(u8 *) fmr->mem.arbel.mpt = MTHCA_MPT_STATUS_SW;

==============================================================

This can be done for mlx4 (mlx4_fmr_unmap) and tavor (mthca_tavor_fmr_unmap) as well.

Jack


On Wednesday 20 February 2008 23:42, Olaf Kirch wrote:
> 
> From 5f56547be86ce91dac0eed0fd6112c2277c92250 Mon Sep 17 00:00:00 2001
> From: Olaf Kirch <olaf.kirch at oracle.com>
> Date: Wed, 20 Feb 2008 20:43:48 +0100
> Subject: [PATCH] ib_mthca: avoid recycling old FMR R_Keys too soon
> 
> When a FMR is unmapped, ib_mthca resets the map count to 0, and clears
> the upper part of the R_Key which is used as the sequence counter.
> 
> This poses a problem for RDS, which uses ib_fmr_unmap as a fence
> operation.  RDS assumes that after issuing an unmap, the old R_Keys
> will be invalid for a "reasonable" period of time. For instance, Oracle
> processes uses shared memory buffers allocated from a pool of buffers.
> When a process dies, we want to reclaim these buffers - but we must make sure
> there are no pending RDMA operations to/from those buffers.
> The only way to achieve that is by using unmap and sync the TPT.
> 
> However, when the sequence count is reset on unmap, there is a high
> likelihood that a new mapping will be given the same R_Key that was
> issued a few milliseconds ago.
> 
> To prevent this, we suggest to not reset the sequence count when
> unmapping a FMR.
> 
> This patch addresses the issue on Arbel HCAs only - someone with
> a better understanding of OFED driver internals may want to take this
> patch and do it properly :-)
> 
> Signed-off-by: Olaf Kirch <olaf.kirch at oracle.com>
> ---
>  drivers/infiniband/hw/mthca/mthca_mr.c |    5 +++++
>  1 files changed, 5 insertions(+), 0 deletions(-)
> 
> diff --git a/drivers/infiniband/hw/mthca/mthca_mr.c b/drivers/infiniband/hw/mthca/mthca_mr.c
> index aa6c70a..09740a1 100644
> --- a/drivers/infiniband/hw/mthca/mthca_mr.c
> +++ b/drivers/infiniband/hw/mthca/mthca_mr.c
> @@ -835,10 +835,15 @@ void mthca_arbel_fmr_unmap(struct mthca_dev *dev, struct mthca_fmr *fmr)
>  	if (!fmr->maps)
>  		return;
>  
> +	/* Do not clear the sequence count on unmap, so that we cycle
> +	 * through all sequence numbers before issuing the same R_Key
> +	 * again. */
> +#if 0
>  	key = arbel_key_to_hw_index(fmr->ibmr.lkey);
>  	key &= dev->limits.num_mpts - 1;
>  	key = adjust_key(dev, key);
>  	fmr->ibmr.lkey = fmr->ibmr.rkey = arbel_hw_index_to_key(key);
> +#endif
>  
>  	fmr->maps = 0;
>  
> -- 
> 1.5.4.rc3
> 
> 
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.openfabrics.org/pipermail/general/attachments/20080221/81011956/attachment.html>