SPAM Re: [ofa-general] SPAM Issues with multicast on bonded IPoIB interfaces

Sat Apr 18 10:16:10 PDT 2009

More information...

I also wanted to note that I did a verbose tcpdump on the receiving end and
didn't see anything on ib0 or ib1 that looked out of the ordinary.

More info on the software stack...
I'm using Roland libibverbs packages available in the Ubuntu 8.10
repository.

Best Regards,
Dennis Portello

On Sat, Apr 18, 2009 at 12:57 PM, Dennis Portello <dennis.portello at gmail.com
> wrote:

> Hello,
>
> I've run into some issues with IPoIB bonding when attempting multicast
> communication. I'm not using the ofa-kernel package, but the modules in
> 2.6.27 kernel including the bonding module. For the most part, everything
> has worked so far with a few minor adjustments for Debian/Ubuntu.
>
> Any assistance or suggestions would be greatly appreciated. I've already
> done extensive research on this issue through the mailing list archives, but
> none of the previous suggestions fit or work.
>
> Using:
> SilverStorm 9024 managed switches
> SuperMicro SuperServer 6015TW-TB
> QLogic 7104-HCA-128LPX-DDR dual port IB cards
> Ubuntu 8.10 (2.6.27-11-server)
> Using OFED 1.4.0 (Guy Coats debian packages)
>
> IPoIB Multicast not working for bonded interfaces.
>
> The above works when tested against single IB ports, single Ethernet ports,
> and bonded Ethernet ports, but not on bonded IB ports
>
> I've tried disabling IPv6 by blacklisting ipv6 in /etc/modprobe.d/blacklist
>
> Unicast TCP/IP seems to work just fine.
>
> I've tried setting mode to datagram and changing the MTU so it is lower
> 2044
>
> Use ib-bond script also added bond directly with
>
> echo +bond3 > /sys/class/net/bonding_masters
> echo 1 > /sys/class/net/bond3/bonding/mode
> echo 100 > /sys/class/net/bond3/bonding/miimon
> echo +ib0 > /sys/class/net/bond3/bonding/slaves
> echo +ib1 > /sys/class/net/bond3/bonding/slaves
> ifconfig bond3 192.168.47.100/24
> route add -net 224.0.0.0/3 gw 192.168.47.100
> socat STDIO UDP4-DATAGRAM:224.1.0.1:6666,bind=:6666,range=
> 192.168.47.0/24,ip-add-membership=224.1.0.1:192.168.47.100
> (socat is used to send multicast traffic, it's configure as peer-peer)
>
> echo +bond3 > /sys/class/net/bonding_masters
> echo 1 > /sys/class/net/bond3/bonding/mode
> echo 100 > /sys/class/net/bond3/bonding/miimon
> echo +ib0 > /sys/class/net/bond3/bonding/slaves
> echo +ib1 > /sys/class/net/bond3/bonding/slaves
> ifconfig bond3 192.168.47.102/24
> route add -net 224.0.0.0/3 gw 192.168.47.102
> socat STDIO UDP4-DATAGRAM:224.1.0.1:6666,bind=:6666,range=
> 192.168.47.0/24,ip-add-membership=224.1.0.1:192.168.47.102
>
> === dmesg reports ===
> [45784.497143] bonding: bond3 is being created...
> [45784.498401] bonding: bond3: setting mode to active-backup (1).
> [45784.498434] bonding: bond3: Setting MII monitoring interval to 100.
> [45784.511087] bonding: bond3: doing slave updates when interface is down.
> [45784.511095] bonding: bond3: Adding slave ib0.
> [45784.511099] bonding bond3: master_dev is not up in bond_enslave
> [45784.511100] bonding: bond3: Warning: enslaved VLAN challenged slave ib0.
> Adding VLANs will be blocked as long as ib0 is part of bond bond3
> [45784.511103] bonding: bond3: Warning: The first slave device specified
> does not support setting the MAC address. Setting fail_over_mac to
> active.<6>bonding: bond3: enslaving ib0 as a backup interface with a down
> link.
> [45784.557785] bonding: bond3: doing slave updates when interface is down.
> [45784.557794] bonding: bond3: Adding slave ib1.
> [45784.557797] bonding bond3: master_dev is not up in bond_enslave
> [45784.557798] bonding: bond3: Warning: enslaved VLAN challenged slave ib1.
> Adding VLANs will be blocked as long as ib1 is part of bond bond3
> [45784.604880] bonding: bond3: enslaving ib1 as a backup interface with a
> down link.
> [45784.607960] ib0: mtu > 2044 will cause multicast packet drops.
> [45784.609909] ib1: mtu > 2044 will cause multicast packet drops.
> [45784.613484] ADDRCONF(NETDEV_UP): bond3: link is not ready
> [45784.613498] bonding: bond3: link status definitely up for interface ib0.
> [45784.613501] bonding: bond3: making interface ib0 the new active one.
> [45784.613524] bonding: bond3: first active interface up!
> [45784.613527] bonding: bond3: link status definitely up for interface ib1.
> [45784.615315] ADDRCONF(NETDEV_CHANGE): bond3: link becomes ready
> [45784.616065] ib0: multicast join failed for
> 0001:0000:0000:0000:0000:0000:0000:0000, status -22
> [45784.616176] ib0: multicast join failed for
> 0001:0000:0000:0000:0000:0000:0000:0000, status -22
> [45786.610476] ib0: multicast join failed for
> 0001:0000:0000:0000:0000:0000:0000:0000, status -22
> [45794.610301] ib0: multicast join failed for
> 0001:0000:0000:0000:0000:0000:0000:0000, status -22
> [45795.113757] bond3: no IPv6 routers present
> [45810.610409] ib0: multicast join failed for
> 0001:0000:0000:0000:0000:0000:0000:0000, status -22
> [45826.610284] ib0: multicast join failed for
> 0001:0000:0000:0000:0000:0000:0000:0000, status -22
>
> === my network config /etc/network/interfaces ===
>
> auto lo
> iface lo inet loopback
>
> auto eth0
> iface eth0 inet manual
>
> auto eth1
> iface eth1 inet manual
>
> # auto eth0.45
> # iface eth0.45 inet manual
> # vlan_raw_device eth0
> #
> # auto eth1.45
> # iface eth1.45 inet manual
> # vlan_raw_device eth1
> #
> # auto eth0.46
> # iface eth0.46 inet manual
> # vlan_raw_device eth0
> #
> # auto eth1.46
> # iface eth1.46 inet manual
> # vlan_raw_device eth1
>
> # auto bond0
> # iface bond0 inet manual
> # slaves eth0.45 eth1.45
> # bond_miimon 100
> # bond_mode active-backup
> # pre-up ifup eth0.45 eth1.45
> # post-down ifdown eth0.45 eth1.45
>
> auto bond0
> iface bond0 inet manual
> slaves eth0 eth1
> bond_miimon 100
> bond_mode active-backup
> pre-up ifup eth0 eth1
> post-down ifdown eth0 eth1
>
> # auto bond1
> # iface bond1 inet manual
> # slaves eth0.46 eth1.46
> # bond_miimon 100
> # bond_mode active-backup
> # pre-up ifup eth0.46 eth1.46
> # post-down ifdown eth0.46 eth1.46
>
> auto vmbr0
> iface vmbr0 inet static
> address 192.168.45.100
> netmask 255.255.255.0
> network 192.168.45.0
> gateway 192.168.45.7
> broadcast 192.168.45.255
> bridge_ports bond0
> bridge_fd 9
> bridge_hello 2
> bridge_maxage 12
> bridge_stp off
>
> # auto vmbr1
> # iface vmbr1 inet static
> # address 192.168.46.100
> # netmask 255.255.252.0
> # network 192.168.46.0
> # gateway 192.168.46.7
> # broadcast 192.168.46.255
> # bridge_ports bond1
> # bridge_fd 9
> # bridge_hello 2
> # bridge_maxage 12
> # bridge_stp off
>
> auto ib0
> iface ib0 inet manual
> pre-up echo connected > /sys/class/net/$IFACE/mode
> pre-up echo 65520 > /sys/class/net/$IFACE/mtu
>
> auto ib1
> iface ib1 inet manual
> pre-up echo connected > /sys/class/net/$IFACE/mode
> pre-up echo 65520 > /sys/class/net/$IFACE/mtu
>
> auto bond3
> iface bond3 inet manual
> up ib-bond --bond-name bond3 --bond-ip 192.168.47.100/24 --slaves ib0,ib1
> up echo 65520 > /sys/class/net/$IFACE/mtu
> down ib-bond --bond-name bond3 --stop
>
> === ifconfig -a reports ===
> bond0     Link encap:Ethernet  HWaddr 00:30:48:c6:76:1c
>           inet6 addr: fe80::230:48ff:fec6:761c/64 Scope:Link
>           UP BROADCAST RUNNING MASTER MULTICAST  MTU:1500  Metric:1
>           RX packets:34656 errors:0 dropped:0 overruns:0 frame:0
>           TX packets:6411 errors:0 dropped:0 overruns:0 carrier:0
>           collisions:0 txqueuelen:0
>           RX bytes:2604958 (2.6 MB)  TX bytes:3494145 (3.4 MB)
>
> bond3     Link encap:UNSPEC  HWaddr
> 80-00-04-04-FE-80-00-00-00-00-00-00-00-00-00-00
>           inet addr:192.168.47.102  Bcast:192.168.47.255
> Mask:255.255.255.0
>           inet6 addr: fe80::206:6a00:a000:f77d/64 Scope:Link
>           UP BROADCAST RUNNING MASTER MULTICAST  MTU:65520  Metric:1
>           RX packets:2 errors:0 dropped:0 overruns:0 frame:0
>           TX packets:5 errors:0 dropped:130 overruns:0 carrier:0
>           collisions:0 txqueuelen:0
>           RX bytes:112 (112.0 B)  TX bytes:324 (324.0 B)
>
> eth0      Link encap:Ethernet  HWaddr 00:30:48:c6:76:1c
>           UP BROADCAST RUNNING SLAVE MULTICAST  MTU:1500  Metric:1
>           RX packets:34656 errors:0 dropped:0 overruns:0 frame:0
>           TX packets:6411 errors:0 dropped:0 overruns:0 carrier:0
>           collisions:0 txqueuelen:1000
>           RX bytes:2604958 (2.6 MB)  TX bytes:3494145 (3.4 MB)
>           Memory:d8220000-d8240000
>
> eth1      Link encap:Ethernet  HWaddr 00:30:48:c6:76:1c
>           UP BROADCAST SLAVE MULTICAST  MTU:1500  Metric:1
>           RX packets:0 errors:0 dropped:0 overruns:0 frame:0
>           TX packets:0 errors:0 dropped:0 overruns:0 carrier:0
>           collisions:0 txqueuelen:1000
>           RX bytes:0 (0.0 B)  TX bytes:0 (0.0 B)
>           Memory:d8260000-d8280000
>
> ib0       Link encap:UNSPEC  HWaddr
> 80-00-04-04-FE-80-00-00-00-00-00-00-00-00-00-00
>           UP BROADCAST RUNNING SLAVE MULTICAST  MTU:65520  Metric:1
>           RX packets:0 errors:0 dropped:0 overruns:0 frame:0
>           TX packets:2 errors:0 dropped:123 overruns:0 carrier:0
>           collisions:0 txqueuelen:256
>           RX bytes:0 (0.0 B)  TX bytes:136 (136.0 B)
>
> ib1       Link encap:UNSPEC  HWaddr
> 80-00-04-05-FE-80-00-00-00-00-00-00-00-00-00-00
>           UP BROADCAST RUNNING SLAVE MULTICAST  MTU:65520  Metric:1
>           RX packets:2 errors:0 dropped:0 overruns:0 frame:0
>           TX packets:3 errors:0 dropped:7 overruns:0 carrier:0
>           collisions:0 txqueuelen:256
>           RX bytes:112 (112.0 B)  TX bytes:188 (188.0 B)
>
> lo        Link encap:Local Loopback
>           inet addr:127.0.0.1  Mask:255.0.0.0
>           inet6 addr: ::1/128 Scope:Host
>           UP LOOPBACK RUNNING  MTU:16436  Metric:1
>           RX packets:4 errors:0 dropped:0 overruns:0 frame:0
>           TX packets:4 errors:0 dropped:0 overruns:0 carrier:0
>           collisions:0 txqueuelen:0
>           RX bytes:352 (352.0 B)  TX bytes:352 (352.0 B)
>
> vmbr0     Link encap:Ethernet  HWaddr 00:30:48:c6:76:1c
>           inet addr:192.168.45.102  Bcast:192.168.45.255
> Mask:255.255.255.0
>           inet6 addr: fe80::230:48ff:fec6:761c/64 Scope:Link
>           UP BROADCAST RUNNING MULTICAST  MTU:1500  Metric:1
>           RX packets:11605 errors:0 dropped:0 overruns:0 frame:0
>           TX packets:5638 errors:0 dropped:0 overruns:0 carrier:0
>           collisions:0 txqueuelen:0
>           RX bytes:1057952 (1.0 MB)  TX bytes:3439987 (3.4 MB)
>
> vnet0     Link encap:Ethernet  HWaddr 8a:c0:6b:6a:db:5a
>           inet addr:192.168.122.1  Bcast:192.168.122.255
> Mask:255.255.255.0
>           inet6 addr: fe80::88c0:6bff:fe6a:db5a/64 Scope:Link
>           UP BROADCAST RUNNING MULTICAST  MTU:1500  Metric:1
>           RX packets:0 errors:0 dropped:0 overruns:0 frame:0
>           TX packets:48 errors:0 dropped:0 overruns:0 carrier:0
>           collisions:0 txqueuelen:0
>           RX bytes:0 (0.0 B)  TX bytes:6274 (6.2 KB)
>
> Best regards,
> Dennis Portello
>
> _______________________________________________
> general mailing list
> general at lists.openfabrics.org
> http://lists.openfabrics.org/cgi-bin/mailman/listinfo/general
>
> To unsubscribe, please visit
> http://openib.org/mailman/listinfo/openib-general
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.openfabrics.org/pipermail/general/attachments/20090418/188aa985/attachment.html>

***SPAM*** Re: [ofa-general] ***SPAM*** Issues with multicast on bonded IPoIB interfaces

SPAM Re: [ofa-general] SPAM Issues with multicast on bonded IPoIB interfaces