<html><head><meta name="qrichtext" content="1" /></head><body style="font-size:12pt;font-family:Monospace">
<p><span style="font-style:italic">As long as the underlying mpt index is not played with,</span></p>
<p><span style="font-style:italic">there is no requirement that the sequence bits start from 0. Its just</span></p>
<p><span style="font-style:italic">sufficient to guarantee that the same (full) key not be allocated twice</span></p>
<p><span style="font-style:italic">before performing an unmap/SYNC_TPT.</span></p>
<p></p>
<p><span style="font-style:italic">Thus, there is no problem with the change that Olaf requests. The lines between</span></p>
<p><span style="font-style:italic">the #if 0 and #endif can just be entirely deleted:</span></p>
<p></p>
<p><span style="font-style:italic">Index: ofed_kernel/drivers/infiniband/hw/mthca/mthca_mr.c</span></p>
<p><span style="font-style:italic">===================================================================</span></p>
<p><span style="font-style:italic">--- ofed_kernel.orig/drivers/infiniband/hw/mthca/mthca_mr.c 2008-02-21 10:32:50.000000000 +0200</span></p>
<p><span style="font-style:italic">+++ ofed_kernel/drivers/infiniband/hw/mthca/mthca_mr.c 2008-02-21 12:22:54.393777000 +0200</span></p>
<p><span style="font-style:italic">@@ -839,11 +839,6 @@ void mthca_arbel_fmr_unmap(struct mthca_</span></p>
<p><span style="font-style:italic"> if (!fmr->maps)</span></p>
<p><span style="font-style:italic"> return;</span></p>
<p><span style="font-style:italic"> </span></p>
<p><span style="font-style:italic">- key = arbel_key_to_hw_index(fmr->ibmr.lkey);</span></p>
<p><span style="font-style:italic">- key &= dev->limits.num_mpts - 1;</span></p>
<p><span style="font-style:italic">- key = adjust_key(dev, key);</span></p>
<p><span style="font-style:italic">- fmr->ibmr.lkey = fmr->ibmr.rkey = arbel_hw_index_to_key(key);</span></p>
<p><span style="font-style:italic">-</span></p>
<p><span style="font-style:italic"> fmr->maps = 0;</span></p>
<p><span style="font-style:italic"> </span></p>
<p><span style="font-style:italic"> *(u8 *) fmr->mem.arbel.mpt = MTHCA_MPT_STATUS_SW;</span></p>
<p></p>
<p><span style="font-style:italic">==============================================================</span></p>
<p></p>
<p><span style="font-style:italic">This can be done for mlx4 (mlx4_fmr_unmap) and tavor (mthca_tavor_fmr_unmap) as well.</span></p>
<p></p>
<p><span style="font-style:italic">Jack</span></p>
<p></p>
<p></p>
<p>On Wednesday 20 February 2008 23:42, Olaf Kirch wrote:</p>
<p>> </p>
<p>> From 5f56547be86ce91dac0eed0fd6112c2277c92250 Mon Sep 17 00:00:00 2001</p>
<p>> From: Olaf Kirch <olaf.kirch@oracle.com></p>
<p>> Date: Wed, 20 Feb 2008 20:43:48 +0100</p>
<p>> Subject: [PATCH] ib_mthca: avoid recycling old FMR R_Keys too soon</p>
<p>> </p>
<p>> When a FMR is unmapped, ib_mthca resets the map count to 0, and clears</p>
<p>> the upper part of the R_Key which is used as the sequence counter.</p>
<p>> </p>
<p>> This poses a problem for RDS, which uses ib_fmr_unmap as a fence</p>
<p>> operation. RDS assumes that after issuing an unmap, the old R_Keys</p>
<p>> will be invalid for a "reasonable" period of time. For instance, Oracle</p>
<p>> processes uses shared memory buffers allocated from a pool of buffers.</p>
<p>> When a process dies, we want to reclaim these buffers - but we must make sure</p>
<p>> there are no pending RDMA operations to/from those buffers.</p>
<p>> The only way to achieve that is by using unmap and sync the TPT.</p>
<p>> </p>
<p>> However, when the sequence count is reset on unmap, there is a high</p>
<p>> likelihood that a new mapping will be given the same R_Key that was</p>
<p>> issued a few milliseconds ago.</p>
<p>> </p>
<p>> To prevent this, we suggest to not reset the sequence count when</p>
<p>> unmapping a FMR.</p>
<p>> </p>
<p>> This patch addresses the issue on Arbel HCAs only - someone with</p>
<p>> a better understanding of OFED driver internals may want to take this</p>
<p>> patch and do it properly :-)</p>
<p>> </p>
<p>> Signed-off-by: Olaf Kirch <olaf.kirch@oracle.com></p>
<p>> ---</p>
<p>> drivers/infiniband/hw/mthca/mthca_mr.c | 5 +++++</p>
<p>> 1 files changed, 5 insertions(+), 0 deletions(-)</p>
<p>> </p>
<p>> diff --git a/drivers/infiniband/hw/mthca/mthca_mr.c b/drivers/infiniband/hw/mthca/mthca_mr.c</p>
<p>> index aa6c70a..09740a1 100644</p>
<p>> --- a/drivers/infiniband/hw/mthca/mthca_mr.c</p>
<p>> +++ b/drivers/infiniband/hw/mthca/mthca_mr.c</p>
<p>> @@ -835,10 +835,15 @@ void mthca_arbel_fmr_unmap(struct mthca_dev *dev, struct mthca_fmr *fmr)</p>
<p>> if (!fmr->maps)</p>
<p>> return;</p>
<p>> </p>
<p>> + /* Do not clear the sequence count on unmap, so that we cycle</p>
<p>> + * through all sequence numbers before issuing the same R_Key</p>
<p>> + * again. */</p>
<p>> +#if 0</p>
<p>> key = arbel_key_to_hw_index(fmr->ibmr.lkey);</p>
<p>> key &= dev->limits.num_mpts - 1;</p>
<p>> key = adjust_key(dev, key);</p>
<p>> fmr->ibmr.lkey = fmr->ibmr.rkey = arbel_hw_index_to_key(key);</p>
<p>> +#endif</p>
<p>> </p>
<p>> fmr->maps = 0;</p>
<p>> </p>
<p>> -- </p>
<p>> 1.5.4.rc3</p>
<p>> </p>
<p>> </p>
<p></p>
</body></html>