***SPAM*** Re: [ofa-general] opensm: bad multicast forwarding table entries

[Hal Rosenstock]

On Wed, Nov 12, 2008 at 5:18 PM,  <akepner at sgi.com> wrote:
>
> Here's a description of a problem we're seeing where multicast
> forwarding tables are apparently getting set up incorrectly. I'd
> appreciate any debug help from the opensm experts out there.
>
> On large clusters (>1000 nodes or so) we often see hundreds of errors
> from 'ibdiagnet -r' like the following (this is the simplest example
> I could find):
>
> -I- Multicast Group:0xC069 has:2 switches and:2 HCAs
> -E- Disconnected switch:S0800690000002e51/U1 in group:0xC069
> -E- Disconnected HCA:r4i2n10/U1

Is it really an error to have a multicast group like this ? I agree
it's not needed to route if there's only 1 member port.

Can you describe the scenario under which this occurs ? Are things
steady state or are there changes going on in the subnet ? Any errors
in the opensm log ?

> These have invariably been multicast groups associated with IPv6
> solicited node multicast addresses, e.g., in this case 'saquery -m'
> shows only a single member, "r5lead":
>
> MCMemberRecord member dump:
>                MGID....................0xff12601bffff0000 : 0x00000001ff26d289
>                Mlid....................0xC069
>                PortGid.................0xfe80000000000000 : 0x0002c9020026d289
>                ScopeState..............0x1
>                ProxyJoin...............0x0
>                NodeDescription.........r5lead HCA-1
>
> ibdiagnet shows that "r5lead" is connected to the switch with lid
> 1609, port 24:
>
> Switch  24 "S-0800690000002db4"         # "MT47396 Infiniscale-III Mellanox Technologies" base port 0 lid 1609 lmc 0
> [24]    "H-0002c9020026d288"[1](2c9020026d289)          # "r5lead HCA-1" lid 1576 4xDDR
>
> and the multicast forwarding table (from 'dump_mfts.sh') is consistent:
>
> Multicast mlids [0xc000-0xc3ff] of switch Lid 1609 guid 0x0800690000002db4 (MT47396 Infiniscale-III Mellanox Technologies):
>            0                   1                   2
>     Ports: 0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1 2 3 4
>  MLid
> ....
> 0xc069                                                      x
>
>
> So far, so good. But we also have r4i2n10, connected to the switch with
> lid 1533 port 7:
>
> switchguid=0x800690000002e50(800690000002e50)
> Switch  24 "S-0800690000002e50"         # "MT47396 Infiniscale-III Mellanox Technologies" base port 0 lid 1533 lmc 0
> ......
> [7]     "H-003048c2438a0000"[1](3048c2438a0001)                 # "r4i2n10 HCA-1" lid 771 4xDDR
>
> with this mft entry:
>
> Multicast mlids [0xc000-0xc3ff] of switch Lid 1533 guid 0x0800690000002e50 (MT47396 Infiniscale-III Mellanox Technologies):
>            0                   1                   2
>     Ports: 0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1 2 3 4
>  MLid
> .....
> 0xc069                    x
>
> Any idea why "r4i2n10", with PortGid fe80::3048c2438a0001 would have a
> mft entry for the multicast group with MGID ff12601bffff::1ff26d289?

The MFT entry is based on an MLID and not the MGID. What does saquery
-g show ? Does it show one or more than one MGID with an MLID of
0xc069 ? Also, does saquery -m 0xc069 show one member ?

I don't think OpenSM does this but if the multicast groups are
disjoint, the same MLID could be used for two different groups (MGIDs)
in different parts of the subnet.

Sasha is probably best to comment on what has changed in this area. Is
it possible to try this with the latest OpenSM to see if this has been
fixed ?

-- Hal

> Anyone else seen similar?
>
> --
> Arthur
>
> _______________________________________________
> general mailing list
> general at lists.openfabrics.org
> http://lists.openfabrics.org/cgi-bin/mailman/listinfo/general
>
> To unsubscribe, please visit http://openib.org/mailman/listinfo/openib-general
>