[ofa-general] [BUG report / PATCH] fix race in the core multicast management

[Or Gerlitz]

Hi Sean,

We see a problem related to the core multicast management which seems as a bug:

It is possible for the multicast consumer to call ib_sa_free_multicast() where
this leave request is queued to be later processed by the workqueue thread, and
then call ib_sa_join_multicast() which calls acquire_group() --before-- the leave
request was excecuted by the thread. So the lookup done by acquire_group() succeeds,
the code goes to the found: label and the group reference count climbs to (eg) 2.

Following that the leave work-element causes the thread to just dec the
reference count to 1 in release_group() and do nothing else, and the join
work-element causes the thread to return the cached address-handle attributes
to the consumer. So no sa query is being sent to the SA.

We saw the bug on a uni processor system running the ipath driver, where the
consumer is ipoib and the group being the IPv4 broadcast. When we take down
the link of the switch port connected to the device across the cable, ipoib
rushes to leave the group and then join it. On this system the join "crosses
the leave" and the SA does not take into account the node when computing the
multicast routing of the group --> the node does not get the broadcast traffic.

For now we have applied a work around which causes the multicast code to
call release_group() from ib_sa_free_multicast(). The workaround is
implemented by using the patch below which causes mcast_groups_lost()
to be called also when the port actually goes up, and set the group state
to MCAST_ERROR such that the call to release_group() is not deferred (ipoib
does leave/join for every event, namely both on link down and up).

Please let me know what is your thinking on this issue,

thanks!

Or.

From: Matty Kadosh <mattyk at voltaire.com>

Index: linux-2.6.23-rc5/drivers/infiniband/core/multicast.c
===================================================================
--- linux-2.6.23-rc5.orig/drivers/infiniband/core/multicast.c	2007-09-18 12:32:08.000000000 +0300
+++ linux-2.6.23-rc5/drivers/infiniband/core/multicast.c	2007-09-18 13:31:35.000000000 +0300
@@ -735,6 +735,7 @@ static void mcast_event_handler(struct i
 	dev = container_of(handler, struct mcast_device, event_handler);

 	switch (event->event) {
+	case IB_EVENT_PORT_ACTIVE:
 	case IB_EVENT_PORT_ERR:
 	case IB_EVENT_LID_CHANGE:
 	case IB_EVENT_SM_CHANGE: