[ofa-general] [BUG report / PATCH] fix race in the core multicast management

[Or Gerlitz]

Or Gerlitz wrote:
> Sean Hefty wrote:
>>>> node 1 <-> switch A <-> switch B <-> switch C <-> SA

>>> The host would only see port up/down events as of changes in the link
>>> state in the local port or in the port which is connected to it through
>>> the cable.

>> So, if you brought the link down/up between switches A & B, node 1
>> wouldn't receive any events, but it would be removed from the multicast
>> group?

> However, from the view point of the node, no port down is experienced.

I have tested the a node  <-> switch A <-> switch B (Voltaire SM/SA 
running here) scheme and possible problem you have pointed on does not 
happen:

First, when the A-B link goes down the node is removed from the 
multicast group qat the SA database. No event is being experienced by 
the node.

Second, when the link is brought back online, the SM discovers and 
configures the port. This causes bunch (six!) of events to be generated 
and IPoIB joins the multicast group (the broadcast in this case) and we 
are done. This join actually goes out to the SA from the multicast core 
code and the node is listed in the SA database for the group.

The node system I was using has: OFED 1.2 / mthca / device 25208 (Arbel 
memfull) / firmware 4.8.200

I understand that for this device the events are generated by the 
firmware and not by some filtering code that captures mad passed through 
the process_mad() verb (Sean, am I correct?).

I don't know exactly which events were generated by the firmware, since 
the IPoIB event handler does not print the event number, however, I am 
sure that PORT_ACTIVE and PKEY_CHANGE (for this one there is a different 
print which you can see below) were among them, and there is some chance 
that CLIENT_REREGISTER is not one of them.

Note that port active event for itself is not enough for all this to 
work, since the multicast code does not flush the entries and hence a 
join that follows will be possibly replied with cached attributes (as 
discussed earlier on this thread) and no query would be sent to the SA.

At the bottom line, with this device/firmware the problem does not 
happen, but there's a possible hole here if the IB spec does not require 
the SM to set the client re-register bit each time it discovers a node.

Below is the ipoib log after I have reconnected the cable (when I 
removed it no event was generated and local read of the port info, eg 
through ibv_devinfo, reported the port is UP...)

Or.

> ib0: Port state change event
> ib0: join completion for ff12:401b:ffff:0000:0000:0000:ffff:ffff (status -102)
> ib0: Flushing ib0
> ib0.f1f1: Not flushing - IPOIB_FLAG_INITIALIZED not set.
> ib0: flushing
> ib0: downing ib_dev
> ib0: stopping multicast thread
> ib0: flushing multicast list
> ib0: leaving MGID ff12:401b:ffff:0000:0000:0000:ffff:ffff
> ib0: deleting multicast group ff12:401b:ffff:0000:0000:0000:ffff:ffff
> ib0: starting multicast thread
> ib0: restarting multicast task
> ib0: stopping multicast thread
> ib0: starting multicast thread
> ib0: joining MGID ff12:401b:ffff:0000:0000:0000:ffff:ffff
> ib0: Port state change event
> ib0: Flushing ib0
> ib0.f1f1: Not flushing - IPOIB_FLAG_INITIALIZED not set.
> ib0: flushing
> ib0: downing ib_dev
> ib0: stopping multicast thread
> ib0: flushing multicast list
> ib0: deleting multicast group ff12:401b:ffff:0000:0000:0000:ffff:ffff
> ib0: starting multicast thread
> ib0: restarting multicast task
> ib0: stopping multicast thread
> ib0: starting multicast thread
> ib0: Port state change event
> ib0: joining MGID ff12:401b:ffff:0000:0000:0000:ffff:ffff
> ib0: Flushing ib0
> ib0.f1f1: Not flushing - IPOIB_FLAG_INITIALIZED not set.
> ib0: flushing
> ib0: downing ib_dev
> ib0: stopping multicast thread
> ib0: flushing multicast list
> ib0: deleting multicast group ff12:401b:ffff:0000:0000:0000:ffff:ffff
> ib0: starting multicast thread
> ib0: restarting multicast task
> ib0: stopping multicast thread
> ib0: starting multicast thread
> ib0: joining MGID ff12:401b:ffff:0000:0000:0000:ffff:ffff
> ib0: Port state change event
> ib0: Flushing ib0
> ib0.f1f1: Not flushing - IPOIB_FLAG_INITIALIZED not set.
> ib0: flushing
> ib0: downing ib_dev
> ib0: stopping multicast thread
> ib0: flushing multicast list
> ib0: deleting multicast group ff12:401b:ffff:0000:0000:0000:ffff:ffff
> ib0: starting multicast thread
> ib0: restarting multicast task
> ib0: stopping multicast thread
> ib0: starting multicast thread
> ib0: Port state change event
> ib0: joining MGID ff12:401b:ffff:0000:0000:0000:ffff:ffff
> ib0: Flushing ib0
> ib0.f1f1: Not flushing - IPOIB_FLAG_INITIALIZED not set.
> ib0: flushing
> ib0: downing ib_dev
> ib0: stopping multicast thread
> ib0: flushing multicast list
> ib0: deleting multicast group ff12:401b:ffff:0000:0000:0000:ffff:ffff
> ib0: starting multicast thread
> ib0: restarting multicast task
> ib0: stopping multicast thread
> ib0: starting multicast thread
> ib0: joining MGID ff12:401b:ffff:0000:0000:0000:ffff:ffff
> ib0: pkey change event on port:1
> ib0: Flushing ib0 and restarting it's QP
> ib0.f1f1: Not flushing - IPOIB_FLAG_INITIALIZED not set.
> ib0: Not flushing - pkey index not changed.
> ib0: join completion for ff12:401b:ffff:0000:0000:0000:ffff:ffff (status -110)
> ib0: multicast join failed for ff12:401b:ffff:0000:0000:0000:ffff:ffff, status -110
> ib0: joining MGID ff12:401b:ffff:0000:0000:0000:ffff:ffff
> ib0: join completion for ff12:401b:ffff:0000:0000:0000:ffff:ffff (status 0)
> ib0: Created ah 000001001f447640
> ib0: MGID ff12:401b:ffff:0000:0000:0000:ffff:ffff AV 000001001f447640, LID 0xc000, SL 0
> ib0: successfully joined all multicast groups