[ofa-general] [PATCH] mlx4_ib: Fix MTT leakage in resize_cq

[Jack Morgenstein]

mlx4_ib: Fix MTT leakage in resize_cq.

MTTs associated with the old CQE buffer were not deallocated
(i.e., returned to the free pool).  As a result, the MTT free pool
was eventually completely emptied (for users of resize_cq).

Once the resize_cq command returns successfully from FW, FW no longer
accesses the old CQE buffer, so it is safe to deallocate the MTT entries
used by the old CQE buffer.

Finally, if the resize_cq command fails, the MTTs allocated for the new
CQEs buffer also need to be de-allocated.

Signed-off-by: Jack Morgenstein <jackm at dev.mellanox.co.il>

---

Roland,

Resize_cq is already in kernel 2.6.27, so I think this fix should go into a "latest stable"
kernel version (say, 2.6.27.8).  For heavy resize_cq users, such as MPI, this fix is
significant.

There is no need to call mlx4_mtt_cleanup separately for userspace and kernel-space paths --
this way the call is made in only one place. Actually, we cannot call mlx4_mtt_cleanup inside
a spinlock anyway, because it may sleep (it indirectly invokes mlx4_UNMAP_ICM).
There is also no need to protect the mlx4_mtt_cleanup call against parallel access (spinlock),
since the cleanup only is relevant to FW access to the buffer (MTT entries); the buffer itself
is still available (or not!) to the driver.

Finally, I save the mtt for cleanup because the switchover to the new buffer
may have already occurred (in mlx4_ib_poll_one) when the resize_cq call returns (in which
case cq->buf.mtt has already been overwritten with the new CQE buffer info).

diff --git a/drivers/infiniband/hw/mlx4/cq.c b/drivers/infiniband/hw/mlx4/cq.c
index d0866a3..20d6833 100644
--- a/drivers/infiniband/hw/mlx4/cq.c
+++ b/drivers/infiniband/hw/mlx4/cq.c
@@ -343,6 +343,7 @@ int mlx4_ib_resize_cq(struct ib_cq *ibcq, int entries, struct ib_udata *udata)
 {
 	struct mlx4_ib_dev *dev = to_mdev(ibcq->device);
 	struct mlx4_ib_cq *cq = to_mcq(ibcq);
+	struct mlx4_mtt mtt;
 	int outst_cqe;
 	int err;
 
@@ -376,10 +377,12 @@ int mlx4_ib_resize_cq(struct ib_cq *ibcq, int entries, struct ib_udata *udata)
 			goto out;
 	}
 
+	mtt = cq->buf.mtt;
 	err = mlx4_cq_resize(dev->dev, &cq->mcq, entries, &cq->resize_buf->buf.mtt);
 	if (err)
 		goto err_buf;
 
+	mlx4_mtt_cleanup(dev->dev, &mtt);
 	if (ibcq->uobject) {
 		cq->buf      = cq->resize_buf->buf;
 		cq->ibcq.cqe = cq->resize_buf->cqe;
@@ -406,6 +409,7 @@ int mlx4_ib_resize_cq(struct ib_cq *ibcq, int entries, struct ib_udata *udata)
 	goto out;
 
 err_buf:
+	mlx4_mtt_cleanup(dev->dev, &cq->resize_buf->buf.mtt);
 	if (!ibcq->uobject)
 		mlx4_ib_free_cq_buf(dev, &cq->resize_buf->buf,
 				    cq->resize_buf->cqe);
diff --git a/drivers/infiniband/hw/mlx4/mad.c b/drivers/infiniband/hw/mlx4/mad.c
diff --git a/drivers/infiniband/hw/mthca/mthca_mad.c b/drivers/infiniband/hw/mthca/mthca_mad.c