More information...<br><br>I also wanted to note that I did a verbose tcpdump on the receiving end and didn't see anything on ib0 or ib1 that looked out of the ordinary.<br><br>More info on the software stack...<br>I'm using Roland libibverbs packages available in the Ubuntu 8.10 repository.<br>
<br>Best Regards,<br>Dennis Portello<br><br><div class="gmail_quote">On Sat, Apr 18, 2009 at 12:57 PM, Dennis Portello <span dir="ltr"><<a href="mailto:dennis.portello@gmail.com">dennis.portello@gmail.com</a>></span> wrote:<br>
<blockquote class="gmail_quote" style="border-left: 1px solid rgb(204, 204, 204); margin: 0pt 0pt 0pt 0.8ex; padding-left: 1ex;">Hello,<br><br>I've run into some issues with IPoIB bonding when attempting multicast communication. I'm not using the ofa-kernel package, but the modules in 2.6.27 kernel including the bonding module. For the most part, everything has worked so far with a few minor adjustments for Debian/Ubuntu.<br>
<br>Any assistance or suggestions would be greatly appreciated. I've already done extensive research on this issue through the mailing list archives, but none of the previous suggestions fit or work.<br><br>Using:<br>
SilverStorm 9024 managed switches<br>SuperMicro SuperServer 6015TW-TB<br>QLogic 7104-HCA-128LPX-DDR dual port IB cards<br>Ubuntu 8.10 (2.6.27-11-server)<br>Using OFED 1.4.0 (Guy Coats debian packages)<br><br>IPoIB Multicast not working for bonded interfaces.<br>
<br>The above works when tested against single IB ports, single Ethernet ports, and bonded Ethernet ports, but not on bonded IB ports<br><br>I've tried disabling IPv6 by blacklisting ipv6 in /etc/modprobe.d/blacklist<br>
<br>Unicast TCP/IP seems to work just fine.<br><br>I've tried setting mode to datagram and changing the MTU so it is lower 2044<br><br>Use ib-bond script also added bond directly with <br><br>echo +bond3 > /sys/class/net/bonding_masters<br>
echo 1 > /sys/class/net/bond3/bonding/mode<br>echo 100 > /sys/class/net/bond3/bonding/miimon<br>echo +ib0 > /sys/class/net/bond3/bonding/slaves<br>echo +ib1 > /sys/class/net/bond3/bonding/slaves<br>ifconfig bond3 <a href="http://192.168.47.100/24" target="_blank">192.168.47.100/24</a><br>
route add -net <a href="http://224.0.0.0/3" target="_blank">224.0.0.0/3</a> gw 192.168.47.100<br>socat STDIO UDP4-DATAGRAM:<a href="http://224.1.0.1:6666" target="_blank">224.1.0.1:6666</a>,bind=:6666,range=<a href="http://192.168.47.0/24,ip-add-membership=224.1.0.1:192.168.47.100" target="_blank">192.168.47.0/24,ip-add-membership=224.1.0.1:192.168.47.100</a><br>
(socat is used to send multicast traffic, it's configure as peer-peer)<br><br>echo +bond3 > /sys/class/net/bonding_masters<br>echo 1 > /sys/class/net/bond3/bonding/mode<br>echo 100 > /sys/class/net/bond3/bonding/miimon<br>
echo +ib0 > /sys/class/net/bond3/bonding/slaves<br>echo +ib1 > /sys/class/net/bond3/bonding/slaves<br>ifconfig bond3 <a href="http://192.168.47.102/24" target="_blank">192.168.47.102/24</a><br>route add -net <a href="http://224.0.0.0/3" target="_blank">224.0.0.0/3</a> gw 192.168.47.102<br>
socat STDIO UDP4-DATAGRAM:<a href="http://224.1.0.1:6666" target="_blank">224.1.0.1:6666</a>,bind=:6666,range=<a href="http://192.168.47.0/24,ip-add-membership=224.1.0.1:192.168.47.102" target="_blank">192.168.47.0/24,ip-add-membership=224.1.0.1:192.168.47.102</a><br>
<br>=== dmesg reports ===<br>[45784.497143] bonding: bond3 is being created...<br>[45784.498401] bonding: bond3: setting mode to active-backup (1).<br>[45784.498434] bonding: bond3: Setting MII monitoring interval to 100.<br>
[45784.511087] bonding: bond3: doing slave updates when interface is down.<br>[45784.511095] bonding: bond3: Adding slave ib0.<br>[45784.511099] bonding bond3: master_dev is not up in bond_enslave<br>[45784.511100] bonding: bond3: Warning: enslaved VLAN challenged slave ib0. Adding VLANs will be blocked as long as ib0 is part of bond bond3<br>
[45784.511103] bonding: bond3: Warning: The first slave device specified does not support setting the MAC address. Setting fail_over_mac to active.<6>bonding: bond3: enslaving ib0 as a backup interface with a down link.<br>
[45784.557785] bonding: bond3: doing slave updates when interface is down.<br>[45784.557794] bonding: bond3: Adding slave ib1.<br>[45784.557797] bonding bond3: master_dev is not up in bond_enslave<br>[45784.557798] bonding: bond3: Warning: enslaved VLAN challenged slave ib1. Adding VLANs will be blocked as long as ib1 is part of bond bond3<br>
[45784.604880] bonding: bond3: enslaving ib1 as a backup interface with a down link.<br>[45784.607960] ib0: mtu > 2044 will cause multicast packet drops.<br>[45784.609909] ib1: mtu > 2044 will cause multicast packet drops.<br>
[45784.613484] ADDRCONF(NETDEV_UP): bond3: link is not ready<br>[45784.613498] bonding: bond3: link status definitely up for interface ib0.<br>[45784.613501] bonding: bond3: making interface ib0 the new active one.<br>[45784.613524] bonding: bond3: first active interface up!<br>
[45784.613527] bonding: bond3: link status definitely up for interface ib1.<br>[45784.615315] ADDRCONF(NETDEV_CHANGE): bond3: link becomes ready<br>[45784.616065] ib0: multicast join failed for 0001:0000:0000:0000:0000:0000:0000:0000, status -22<br>
[45784.616176] ib0: multicast join failed for 0001:0000:0000:0000:0000:0000:0000:0000, status -22<br>[45786.610476] ib0: multicast join failed for 0001:0000:0000:0000:0000:0000:0000:0000, status -22<br>[45794.610301] ib0: multicast join failed for 0001:0000:0000:0000:0000:0000:0000:0000, status -22<br>
[45795.113757] bond3: no IPv6 routers present<br>[45810.610409] ib0: multicast join failed for 0001:0000:0000:0000:0000:0000:0000:0000, status -22<br>[45826.610284] ib0: multicast join failed for 0001:0000:0000:0000:0000:0000:0000:0000, status -22<br>
<br>=== my network config /etc/network/interfaces ===<br><br>auto lo<br>iface lo inet loopback<br><br>auto eth0<br>iface eth0 inet manual<br><br>auto eth1<br>iface eth1 inet manual<br><br># auto eth0.45<br># iface eth0.45 inet manual<br>
# vlan_raw_device eth0<br># <br># auto eth1.45<br># iface eth1.45 inet manual<br># vlan_raw_device eth1<br># <br># auto eth0.46<br># iface eth0.46 inet manual<br># vlan_raw_device eth0<br># <br># auto eth1.46<br># iface eth1.46 inet manual<br>
# vlan_raw_device eth1<br><br># auto bond0<br># iface bond0 inet manual<br># slaves eth0.45 eth1.45<br># bond_miimon 100<br># bond_mode active-backup<br># pre-up ifup eth0.45 eth1.45<br># post-down ifdown eth0.45 eth1.45<br>
<br>auto bond0<br>iface bond0 inet manual<br>slaves eth0 eth1<br>bond_miimon 100<br>bond_mode active-backup<br>pre-up ifup eth0 eth1<br>post-down ifdown eth0 eth1<br><br># auto bond1<br># iface bond1 inet manual<br># slaves eth0.46 eth1.46<br>
# bond_miimon 100<br># bond_mode active-backup<br># pre-up ifup eth0.46 eth1.46<br># post-down ifdown eth0.46 eth1.46<br><br>auto vmbr0<br>iface vmbr0 inet static<br>address 192.168.45.100<br>netmask 255.255.255.0<br>network 192.168.45.0<br>
gateway 192.168.45.7<br>broadcast 192.168.45.255<br>bridge_ports bond0<br>bridge_fd 9<br>bridge_hello 2<br>bridge_maxage 12<br>bridge_stp off<br><br># auto vmbr1<br># iface vmbr1 inet static<br># address 192.168.46.100<br>
# netmask 255.255.252.0<br># network 192.168.46.0<br># gateway 192.168.46.7<br># broadcast 192.168.46.255<br># bridge_ports bond1<br># bridge_fd 9<br># bridge_hello 2<br># bridge_maxage 12<br># bridge_stp off<br><br>auto ib0<br>
iface ib0 inet manual<br>pre-up echo connected > /sys/class/net/$IFACE/mode<br>pre-up echo 65520 > /sys/class/net/$IFACE/mtu<br><br>auto ib1<br>iface ib1 inet manual<br>pre-up echo connected > /sys/class/net/$IFACE/mode<br>
pre-up echo 65520 > /sys/class/net/$IFACE/mtu<br><br>auto bond3<br>iface bond3 inet manual<br>up ib-bond --bond-name bond3 --bond-ip <a href="http://192.168.47.100/24" target="_blank">192.168.47.100/24</a> --slaves ib0,ib1<br>
up echo 65520 > /sys/class/net/$IFACE/mtu<br>
down ib-bond --bond-name bond3 --stop<br><br>=== ifconfig -a reports ===<br>bond0 Link encap:Ethernet HWaddr 00:30:48:c6:76:1c <br> inet6 addr: fe80::230:48ff:fec6:761c/64 Scope:Link<br> UP BROADCAST RUNNING MASTER MULTICAST MTU:1500 Metric:1<br>
RX packets:34656 errors:0 dropped:0 overruns:0 frame:0<br> TX packets:6411 errors:0 dropped:0 overruns:0 carrier:0<br> collisions:0 txqueuelen:0 <br> RX bytes:2604958 (2.6 MB) TX bytes:3494145 (3.4 MB)<br>
<br>bond3 Link encap:UNSPEC HWaddr 80-00-04-04-FE-80-00-00-00-00-00-00-00-00-00-00 <br> inet addr:192.168.47.102 Bcast:192.168.47.255 Mask:255.255.255.0<br> inet6 addr: fe80::206:6a00:a000:f77d/64 Scope:Link<br>
UP BROADCAST RUNNING MASTER MULTICAST MTU:65520 Metric:1<br> RX packets:2 errors:0 dropped:0 overruns:0 frame:0<br> TX packets:5 errors:0 dropped:130 overruns:0 carrier:0<br> collisions:0 txqueuelen:0 <br>
RX bytes:112 (112.0 B) TX bytes:324 (324.0 B)<br><br>eth0 Link encap:Ethernet HWaddr 00:30:48:c6:76:1c <br> UP BROADCAST RUNNING SLAVE MULTICAST MTU:1500 Metric:1<br> RX packets:34656 errors:0 dropped:0 overruns:0 frame:0<br>
TX packets:6411 errors:0 dropped:0 overruns:0 carrier:0<br> collisions:0 txqueuelen:1000 <br> RX bytes:2604958 (2.6 MB) TX bytes:3494145 (3.4 MB)<br> Memory:d8220000-d8240000 <br><br>
eth1 Link encap:Ethernet HWaddr 00:30:48:c6:76:1c <br> UP BROADCAST SLAVE MULTICAST MTU:1500 Metric:1<br> RX packets:0 errors:0 dropped:0 overruns:0 frame:0<br> TX packets:0 errors:0 dropped:0 overruns:0 carrier:0<br>
collisions:0 txqueuelen:1000 <br> RX bytes:0 (0.0 B) TX bytes:0 (0.0 B)<br> Memory:d8260000-d8280000 <br><br>ib0 Link encap:UNSPEC HWaddr 80-00-04-04-FE-80-00-00-00-00-00-00-00-00-00-00 <br>
UP BROADCAST RUNNING SLAVE MULTICAST MTU:65520 Metric:1<br> RX packets:0 errors:0 dropped:0 overruns:0 frame:0<br> TX packets:2 errors:0 dropped:123 overruns:0 carrier:0<br> collisions:0 txqueuelen:256 <br>
RX bytes:0 (0.0 B) TX bytes:136 (136.0 B)<br><br>ib1 Link encap:UNSPEC HWaddr 80-00-04-05-FE-80-00-00-00-00-00-00-00-00-00-00 <br> UP BROADCAST RUNNING SLAVE MULTICAST MTU:65520 Metric:1<br>
RX packets:2 errors:0 dropped:0 overruns:0 frame:0<br>
TX packets:3 errors:0 dropped:7 overruns:0 carrier:0<br> collisions:0 txqueuelen:256 <br> RX bytes:112 (112.0 B) TX bytes:188 (188.0 B)<br><br>lo Link encap:Local Loopback <br> inet addr:127.0.0.1 Mask:255.0.0.0<br>
inet6 addr: ::1/128 Scope:Host<br> UP LOOPBACK RUNNING MTU:16436 Metric:1<br> RX packets:4 errors:0 dropped:0 overruns:0 frame:0<br> TX packets:4 errors:0 dropped:0 overruns:0 carrier:0<br>
collisions:0 txqueuelen:0 <br> RX bytes:352 (352.0 B) TX bytes:352 (352.0 B)<br><br>vmbr0 Link encap:Ethernet HWaddr 00:30:48:c6:76:1c <br> inet addr:192.168.45.102 Bcast:192.168.45.255 Mask:255.255.255.0<br>
inet6 addr: fe80::230:48ff:fec6:761c/64 Scope:Link<br> UP BROADCAST RUNNING MULTICAST MTU:1500 Metric:1<br> RX packets:11605 errors:0 dropped:0 overruns:0 frame:0<br> TX packets:5638 errors:0 dropped:0 overruns:0 carrier:0<br>
collisions:0 txqueuelen:0 <br> RX bytes:1057952 (1.0 MB) TX bytes:3439987 (3.4 MB)<br><br>vnet0 Link encap:Ethernet HWaddr 8a:c0:6b:6a:db:5a <br> inet addr:192.168.122.1 Bcast:192.168.122.255 Mask:255.255.255.0<br>
inet6 addr: fe80::88c0:6bff:fe6a:db5a/64 Scope:Link<br> UP BROADCAST RUNNING MULTICAST MTU:1500 Metric:1<br> RX packets:0 errors:0 dropped:0 overruns:0 frame:0<br> TX packets:48 errors:0 dropped:0 overruns:0 carrier:0<br>
collisions:0 txqueuelen:0 <br> RX bytes:0 (0.0 B) TX bytes:6274 (6.2 KB)<br> <br>Best regards,<br><font color="#888888">Dennis Portello<br>
</font><br>_______________________________________________<br>
general mailing list<br>
<a href="mailto:general@lists.openfabrics.org">general@lists.openfabrics.org</a><br>
<a href="http://lists.openfabrics.org/cgi-bin/mailman/listinfo/general" target="_blank">http://lists.openfabrics.org/cgi-bin/mailman/listinfo/general</a><br>
<br>
To unsubscribe, please visit <a href="http://openib.org/mailman/listinfo/openib-general" target="_blank">http://openib.org/mailman/listinfo/openib-general</a><br></blockquote></div><br>