[ofa-general] [PATCH] mlx4: Do not allow ib userspace open while device is being removed

[Jack Morgenstein]

Userspace apps are supposed to release all ib device resources if
they receive a fatal async event (IBV_EVENT_DEVICE_FATAL).  However,
the app has no way of knowing when the device has come back up, except
to repeatedly attempt ibv_open_device() until it succeeds.

However, currently there is no protection against open succeeding when
the device is in the midst of the removal following the fatal event.
In this case, the open will succeed, but as a result the device waits
in the middle of its removal until the new app releases its ib resources
 -- and the new app will not do so, since the open succeeded at a point
following the fatal event generation.

This patch adds an "active" flag to the device. The active flag is set to
false (in the interrupt context) before the "fatal" event is generated,
so any subsequent ibv_dev_open() call to the device will fail until the
device comes back up, thus preventing the above deadlock.

Signed-off-by: Jack Morgenstein <jackm at dev.mellanox.co.il>

---

Roland,
For good measure, I also set the active flag to false at mlx4_ib_remove() -- to
give some measure of protection against opening a new userspace app while the driver
is in the process of being removed.  The main issue in this patch, though, is setting
the active flag to false before triggering the fatal event in catas.c.

This patch is for kernel 2.6.31. 

-Jack

diff --git a/drivers/infiniband/hw/mlx4/main.c b/drivers/infiniband/hw/mlx4/main.c
index ae3d759..571d231 100644
--- a/drivers/infiniband/hw/mlx4/main.c
+++ b/drivers/infiniband/hw/mlx4/main.c
@@ -342,6 +344,9 @@ static struct ib_ucontext *mlx4_ib_alloc_ucontext(struct ib_device *ibdev,
 	struct mlx4_ib_alloc_ucontext_resp resp;
 	int err;
 
+	if (!dev->dev->ib_active)
+		return ERR_PTR(-EAGAIN);
+
 	resp.qp_tab_size      = dev->dev->caps.num_qps;
 	resp.bf_reg_size      = dev->dev->caps.bf_reg_size;
 	resp.bf_regs_per_page = dev->dev->caps.bf_regs_per_page;
@@ -673,6 +686,8 @@ static void *mlx4_ib_add(struct mlx4_dev *dev)
 			goto err_reg;
 	}
 
+	dev->ib_active = 1;
+
 	return ibdev;
 
 err_reg:
@@ -698,6 +713,8 @@ static void mlx4_ib_remove(struct mlx4_dev *dev, void *ibdev_ptr)
 	struct mlx4_ib_dev *ibdev = ibdev_ptr;
 	int p;
 
+	dev->ib_active = 0;
+
 	mlx4_ib_mad_cleanup(ibdev);
 	ib_unregister_device(&ibdev->ib_dev);
 
diff --git a/drivers/net/mlx4/catas.c b/drivers/net/mlx4/catas.c
index aa9674b..7a23beb 100644
--- a/drivers/net/mlx4/catas.c
+++ b/drivers/net/mlx4/catas.c
@@ -69,6 +69,7 @@ static void poll_catas(unsigned long dev_ptr)
 	if (readl(priv->catas_err.map)) {
 		dump_err_buf(dev);
 
+		dev->ib_active = 0;
 		mlx4_dispatch_event(dev, MLX4_DEV_EVENT_CATASTROPHIC_ERROR, 0);
 
 		if (internal_err_reset) {
diff --git a/include/linux/mlx4/device.h b/include/linux/mlx4/device.h
index ce7cc6c..7f6d800 100644
--- a/include/linux/mlx4/device.h
+++ b/include/linux/mlx4/device.h
@@ -379,6 +379,7 @@ struct mlx4_dev {
 	struct radix_tree_root	qp_table_tree;
 	u32			rev_id;
 	char			board_id[MLX4_BOARD_ID_LEN];
+	u32			ib_active;
 };
 
 struct mlx4_init_port_param {