[ofa-general] SPAM Issues with multicast on bonded IPoIB interfaces

Sat Apr 18 09:57:14 PDT 2009

Hello,

I've run into some issues with IPoIB bonding when attempting multicast
communication. I'm not using the ofa-kernel package, but the modules in
2.6.27 kernel including the bonding module. For the most part, everything
has worked so far with a few minor adjustments for Debian/Ubuntu.

Any assistance or suggestions would be greatly appreciated. I've already
done extensive research on this issue through the mailing list archives, but
none of the previous suggestions fit or work.

Using:
SilverStorm 9024 managed switches
SuperMicro SuperServer 6015TW-TB
QLogic 7104-HCA-128LPX-DDR dual port IB cards
Ubuntu 8.10 (2.6.27-11-server)
Using OFED 1.4.0 (Guy Coats debian packages)

IPoIB Multicast not working for bonded interfaces.

The above works when tested against single IB ports, single Ethernet ports,
and bonded Ethernet ports, but not on bonded IB ports

I've tried disabling IPv6 by blacklisting ipv6 in /etc/modprobe.d/blacklist

Unicast TCP/IP seems to work just fine.

I've tried setting mode to datagram and changing the MTU so it is lower 2044

Use ib-bond script also added bond directly with

echo +bond3 > /sys/class/net/bonding_masters
echo 1 > /sys/class/net/bond3/bonding/mode
echo 100 > /sys/class/net/bond3/bonding/miimon
echo +ib0 > /sys/class/net/bond3/bonding/slaves
echo +ib1 > /sys/class/net/bond3/bonding/slaves
ifconfig bond3 192.168.47.100/24
route add -net 224.0.0.0/3 gw 192.168.47.100
socat STDIO UDP4-DATAGRAM:224.1.0.1:6666,bind=:6666,range=
192.168.47.0/24,ip-add-membership=224.1.0.1:192.168.47.100
(socat is used to send multicast traffic, it's configure as peer-peer)

echo +bond3 > /sys/class/net/bonding_masters
echo 1 > /sys/class/net/bond3/bonding/mode
echo 100 > /sys/class/net/bond3/bonding/miimon
echo +ib0 > /sys/class/net/bond3/bonding/slaves
echo +ib1 > /sys/class/net/bond3/bonding/slaves
ifconfig bond3 192.168.47.102/24
route add -net 224.0.0.0/3 gw 192.168.47.102
socat STDIO UDP4-DATAGRAM:224.1.0.1:6666,bind=:6666,range=
192.168.47.0/24,ip-add-membership=224.1.0.1:192.168.47.102

=== dmesg reports ===
[45784.497143] bonding: bond3 is being created...
[45784.498401] bonding: bond3: setting mode to active-backup (1).
[45784.498434] bonding: bond3: Setting MII monitoring interval to 100.
[45784.511087] bonding: bond3: doing slave updates when interface is down.
[45784.511095] bonding: bond3: Adding slave ib0.
[45784.511099] bonding bond3: master_dev is not up in bond_enslave
[45784.511100] bonding: bond3: Warning: enslaved VLAN challenged slave ib0.
Adding VLANs will be blocked as long as ib0 is part of bond bond3
[45784.511103] bonding: bond3: Warning: The first slave device specified
does not support setting the MAC address. Setting fail_over_mac to
active.<6>bonding: bond3: enslaving ib0 as a backup interface with a down
link.
[45784.557785] bonding: bond3: doing slave updates when interface is down.
[45784.557794] bonding: bond3: Adding slave ib1.
[45784.557797] bonding bond3: master_dev is not up in bond_enslave
[45784.557798] bonding: bond3: Warning: enslaved VLAN challenged slave ib1.
Adding VLANs will be blocked as long as ib1 is part of bond bond3
[45784.604880] bonding: bond3: enslaving ib1 as a backup interface with a
down link.
[45784.607960] ib0: mtu > 2044 will cause multicast packet drops.
[45784.609909] ib1: mtu > 2044 will cause multicast packet drops.
[45784.613484] ADDRCONF(NETDEV_UP): bond3: link is not ready
[45784.613498] bonding: bond3: link status definitely up for interface ib0.
[45784.613501] bonding: bond3: making interface ib0 the new active one.
[45784.613524] bonding: bond3: first active interface up!
[45784.613527] bonding: bond3: link status definitely up for interface ib1.
[45784.615315] ADDRCONF(NETDEV_CHANGE): bond3: link becomes ready
[45784.616065] ib0: multicast join failed for
0001:0000:0000:0000:0000:0000:0000:0000, status -22
[45784.616176] ib0: multicast join failed for
0001:0000:0000:0000:0000:0000:0000:0000, status -22
[45786.610476] ib0: multicast join failed for
0001:0000:0000:0000:0000:0000:0000:0000, status -22
[45794.610301] ib0: multicast join failed for
0001:0000:0000:0000:0000:0000:0000:0000, status -22
[45795.113757] bond3: no IPv6 routers present
[45810.610409] ib0: multicast join failed for
0001:0000:0000:0000:0000:0000:0000:0000, status -22
[45826.610284] ib0: multicast join failed for
0001:0000:0000:0000:0000:0000:0000:0000, status -22

=== my network config /etc/network/interfaces ===

auto lo
iface lo inet loopback

auto eth0
iface eth0 inet manual

auto eth1
iface eth1 inet manual

# auto eth0.45
# iface eth0.45 inet manual
# vlan_raw_device eth0
#
# auto eth1.45
# iface eth1.45 inet manual
# vlan_raw_device eth1
#
# auto eth0.46
# iface eth0.46 inet manual
# vlan_raw_device eth0
#
# auto eth1.46
# iface eth1.46 inet manual
# vlan_raw_device eth1

# auto bond0
# iface bond0 inet manual
# slaves eth0.45 eth1.45
# bond_miimon 100
# bond_mode active-backup
# pre-up ifup eth0.45 eth1.45
# post-down ifdown eth0.45 eth1.45

auto bond0
iface bond0 inet manual
slaves eth0 eth1
bond_miimon 100
bond_mode active-backup
pre-up ifup eth0 eth1
post-down ifdown eth0 eth1

# auto bond1
# iface bond1 inet manual
# slaves eth0.46 eth1.46
# bond_miimon 100
# bond_mode active-backup
# pre-up ifup eth0.46 eth1.46
# post-down ifdown eth0.46 eth1.46

auto vmbr0
iface vmbr0 inet static
address 192.168.45.100
netmask 255.255.255.0
network 192.168.45.0
gateway 192.168.45.7
broadcast 192.168.45.255
bridge_ports bond0
bridge_fd 9
bridge_hello 2
bridge_maxage 12
bridge_stp off

# auto vmbr1
# iface vmbr1 inet static
# address 192.168.46.100
# netmask 255.255.252.0
# network 192.168.46.0
# gateway 192.168.46.7
# broadcast 192.168.46.255
# bridge_ports bond1
# bridge_fd 9
# bridge_hello 2
# bridge_maxage 12
# bridge_stp off

auto ib0
iface ib0 inet manual
pre-up echo connected > /sys/class/net/$IFACE/mode
pre-up echo 65520 > /sys/class/net/$IFACE/mtu

auto ib1
iface ib1 inet manual
pre-up echo connected > /sys/class/net/$IFACE/mode
pre-up echo 65520 > /sys/class/net/$IFACE/mtu

auto bond3
iface bond3 inet manual
up ib-bond --bond-name bond3 --bond-ip 192.168.47.100/24 --slaves ib0,ib1
up echo 65520 > /sys/class/net/$IFACE/mtu
down ib-bond --bond-name bond3 --stop

=== ifconfig -a reports ===
bond0     Link encap:Ethernet  HWaddr 00:30:48:c6:76:1c
          inet6 addr: fe80::230:48ff:fec6:761c/64 Scope:Link
          UP BROADCAST RUNNING MASTER MULTICAST  MTU:1500  Metric:1
          RX packets:34656 errors:0 dropped:0 overruns:0 frame:0
          TX packets:6411 errors:0 dropped:0 overruns:0 carrier:0
          collisions:0 txqueuelen:0
          RX bytes:2604958 (2.6 MB)  TX bytes:3494145 (3.4 MB)

bond3     Link encap:UNSPEC  HWaddr
80-00-04-04-FE-80-00-00-00-00-00-00-00-00-00-00
          inet addr:192.168.47.102  Bcast:192.168.47.255  Mask:255.255.255.0
          inet6 addr: fe80::206:6a00:a000:f77d/64 Scope:Link
          UP BROADCAST RUNNING MASTER MULTICAST  MTU:65520  Metric:1
          RX packets:2 errors:0 dropped:0 overruns:0 frame:0
          TX packets:5 errors:0 dropped:130 overruns:0 carrier:0
          collisions:0 txqueuelen:0
          RX bytes:112 (112.0 B)  TX bytes:324 (324.0 B)

eth0      Link encap:Ethernet  HWaddr 00:30:48:c6:76:1c
          UP BROADCAST RUNNING SLAVE MULTICAST  MTU:1500  Metric:1
          RX packets:34656 errors:0 dropped:0 overruns:0 frame:0
          TX packets:6411 errors:0 dropped:0 overruns:0 carrier:0
          collisions:0 txqueuelen:1000
          RX bytes:2604958 (2.6 MB)  TX bytes:3494145 (3.4 MB)
          Memory:d8220000-d8240000

eth1      Link encap:Ethernet  HWaddr 00:30:48:c6:76:1c
          UP BROADCAST SLAVE MULTICAST  MTU:1500  Metric:1
          RX packets:0 errors:0 dropped:0 overruns:0 frame:0
          TX packets:0 errors:0 dropped:0 overruns:0 carrier:0
          collisions:0 txqueuelen:1000
          RX bytes:0 (0.0 B)  TX bytes:0 (0.0 B)
          Memory:d8260000-d8280000

ib0       Link encap:UNSPEC  HWaddr
80-00-04-04-FE-80-00-00-00-00-00-00-00-00-00-00
          UP BROADCAST RUNNING SLAVE MULTICAST  MTU:65520  Metric:1
          RX packets:0 errors:0 dropped:0 overruns:0 frame:0
          TX packets:2 errors:0 dropped:123 overruns:0 carrier:0
          collisions:0 txqueuelen:256
          RX bytes:0 (0.0 B)  TX bytes:136 (136.0 B)

ib1       Link encap:UNSPEC  HWaddr
80-00-04-05-FE-80-00-00-00-00-00-00-00-00-00-00
          UP BROADCAST RUNNING SLAVE MULTICAST  MTU:65520  Metric:1
          RX packets:2 errors:0 dropped:0 overruns:0 frame:0
          TX packets:3 errors:0 dropped:7 overruns:0 carrier:0
          collisions:0 txqueuelen:256
          RX bytes:112 (112.0 B)  TX bytes:188 (188.0 B)

lo        Link encap:Local Loopback
          inet addr:127.0.0.1  Mask:255.0.0.0
          inet6 addr: ::1/128 Scope:Host
          UP LOOPBACK RUNNING  MTU:16436  Metric:1
          RX packets:4 errors:0 dropped:0 overruns:0 frame:0
          TX packets:4 errors:0 dropped:0 overruns:0 carrier:0
          collisions:0 txqueuelen:0
          RX bytes:352 (352.0 B)  TX bytes:352 (352.0 B)

vmbr0     Link encap:Ethernet  HWaddr 00:30:48:c6:76:1c
          inet addr:192.168.45.102  Bcast:192.168.45.255  Mask:255.255.255.0
          inet6 addr: fe80::230:48ff:fec6:761c/64 Scope:Link
          UP BROADCAST RUNNING MULTICAST  MTU:1500  Metric:1
          RX packets:11605 errors:0 dropped:0 overruns:0 frame:0
          TX packets:5638 errors:0 dropped:0 overruns:0 carrier:0
          collisions:0 txqueuelen:0
          RX bytes:1057952 (1.0 MB)  TX bytes:3439987 (3.4 MB)

vnet0     Link encap:Ethernet  HWaddr 8a:c0:6b:6a:db:5a
          inet addr:192.168.122.1  Bcast:192.168.122.255  Mask:255.255.255.0
          inet6 addr: fe80::88c0:6bff:fe6a:db5a/64 Scope:Link
          UP BROADCAST RUNNING MULTICAST  MTU:1500  Metric:1
          RX packets:0 errors:0 dropped:0 overruns:0 frame:0
          TX packets:48 errors:0 dropped:0 overruns:0 carrier:0
          collisions:0 txqueuelen:0
          RX bytes:0 (0.0 B)  TX bytes:6274 (6.2 KB)

Best regards,
Dennis Portello
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.openfabrics.org/pipermail/general/attachments/20090418/a637c40e/attachment.html>

[ofa-general] ***SPAM*** Issues with multicast on bonded IPoIB interfaces

[ofa-general] SPAM Issues with multicast on bonded IPoIB interfaces