[ofa-general] [PATCH V3] mlx4: Do not allow ib userspace open following a fatal event

[Jack Morgenstein]

Userspace apps are supposed to release all ib device resources if
they receive a fatal async event (IBV_EVENT_DEVICE_FATAL).  However,
the app has no way of knowing when the device has come back up, except
to repeatedly attempt ibv_open_device() until it succeeds.

However, currently there is no protection against open succeeding when
the device is in the midst of the removal following the fatal event.
In this case, the open will succeed, but as a result the device waits
in the middle of its removal until the new app releases its ib resources
 -- and the new app will not do so, since the open succeeded at a point
following the fatal event generation.

This patch adds an "active" flag to the device. The active flag is set to
false (in the fatal event flow) before the "fatal" event is generated,
so any subsequent ibv_dev_open() call to the device will fail until the
device comes back up, thus preventing the above deadlock.

V2: move active flag from net to hw/mlx4, and use only for fatal event flow.
(per feedback from Roland).

V3: fixed checkpatch.pl warnings.

Signed-off-by: Jack Morgenstein <jackm at dev.mellanox.co.il>

---
Roland,
Sorry about the checkpatch.pl oversight.  No excuse, but that day I was particularly
rushed -- I left for the airport that evening with my family to go on vacation for 2 weeks.
I guess I cut some corners, and shouldn't have.

diff --git a/drivers/infiniband/hw/mlx4/main.c b/drivers/infiniband/hw/mlx4/main.c
index ae3d759..4effc19 100644
--- a/drivers/infiniband/hw/mlx4/main.c
+++ b/drivers/infiniband/hw/mlx4/main.c
@@ -342,6 +342,9 @@ static struct ib_ucontext *mlx4_ib_alloc_ucontext(struct ib_device *ibdev,
 	struct mlx4_ib_alloc_ucontext_resp resp;
 	int err;
 
+	if (!dev->ib_active)
+		return ERR_PTR(-EAGAIN);
+
 	resp.qp_tab_size      = dev->dev->caps.num_qps;
 	resp.bf_reg_size      = dev->dev->caps.bf_reg_size;
 	resp.bf_regs_per_page = dev->dev->caps.bf_regs_per_page;
@@ -673,6 +676,8 @@ static void *mlx4_ib_add(struct mlx4_dev *dev)
 			goto err_reg;
 	}
 
+	ibdev->ib_active = 1;
+
 	return ibdev;
 
 err_reg:
@@ -729,6 +734,7 @@ static void mlx4_ib_event(struct mlx4_dev *dev, void *ibdev_ptr,
 		break;
 
 	case MLX4_DEV_EVENT_CATASTROPHIC_ERROR:
+		ibdev->ib_active = 0;
 		ibev.event = IB_EVENT_DEVICE_FATAL;
 		break;
 
diff --git a/drivers/infiniband/hw/mlx4/mlx4_ib.h b/drivers/infiniband/hw/mlx4/mlx4_ib.h
index 8a7dd67..b22df97 100644
--- a/drivers/infiniband/hw/mlx4/mlx4_ib.h
+++ b/drivers/infiniband/hw/mlx4/mlx4_ib.h
@@ -175,6 +175,7 @@ struct mlx4_ib_dev {
 	spinlock_t		sm_lock;
 
 	struct mutex		cap_mask_mutex;
+	int			ib_active;
 };
 
 static inline struct mlx4_ib_dev *to_mdev(struct ib_device *ibdev)