[ofa-general] RE: [RFC PATCH 4/4] rdma/cma: implement RDMA_ALIGN_WITH_NETDEVICE ha mode

[Sean Hefty]

>+static void cma_ha_work_handler(struct work_struct *work)
>+{
>+	struct rdma_id_private *id_priv;
>+
>+	id_priv = container_of(work, struct rdma_id_private, ha_work);
>+	rdma_disconnect(&id_priv->id);
>+}

This will race with other user calls.  I've found it fairly difficult for the
rdma_cm to call back into its own API and avoid racing with the user trying to
destroy the cm_id.  None of the APIs are coded to allow calling them
simultaneously with destroy.

A better solution for this may be for the rdma_cm to simply notify the user that
the IP mapping for their RDMA device has changed.  The user can then disconnect,
with the appropriate synchronization, if they want their RDMA connection to
follow the IP address.  (If I understood correctly, the reason for this is to
allow failing back to a repaired port.)

>+static int cma_netdev_callback(struct notifier_block *self, unsigned long
>event,
>+	void *ctx)
>+{
>+	struct net_device *ndev = (struct net_device *)ctx;
>+	struct cma_device *cma_dev;
>+	struct rdma_id_private *id_priv;
>+	struct rdma_dev_addr *dev_addr;
>+
>+	if (dev_net(ndev) != &init_net)
>+		return NOTIFY_DONE;
>+
>+	if (event != NETDEV_BONDING_FAILOVER)
>+		return NOTIFY_DONE;
>+
>+	if (!(ndev->flags & IFF_MASTER) || !(ndev->priv_flags & IFF_BONDING))
>+		return NOTIFY_DONE;
>+
>+	list_for_each_entry(cma_dev, &dev_list, list)
>+		list_for_each_entry(id_priv, &cma_dev->id_list, list) {
>+			dev_addr = &id_priv->id.route.addr.dev_addr;
>+			if (!memcmp(dev_addr->src_netdev_name, ndev->name,
IFNAMSIZ)
>&&
>+				memcmp(dev_addr->src_dev_addr, ndev->dev_addr,
ndev-
>>addr_len))
>+					if (id_priv->ha_mode ==
>RDMA_ALIGN_WITH_NETDEVICE)
>+
schedule_work(&id_priv->ha_work);
>+		}
>+	return NOTIFY_DONE;
>+}

As Roland mentioned, this is racy in the areas he pointed out.  This will take
some thought to handle correctly.

- Sean