Hello Or,<br><br>Thanks for the reply.<br><br>I enabled debug and these are the results of my test. (below)<br><br>First off, I ran this same test on bonded ethernet and on a single IB interface with success.<br><br>sudo route add -net <a href="http://224.0.0.0/3">224.0.0.0/3</a> gw 192.168.47.102<br>
socat STDIO UDP4-DATAGRAM:<a href="http://224.1.0.1:6666">224.1.0.1:6666</a>,bind=:6666,range=<a href="http://192.168.47.0/24,ip-add-membership=224.1.0.1:192.168.47.102">192.168.47.0/24,ip-add-membership=224.1.0.1:192.168.47.102</a><br>
<br>sudo route add -net <a href="http://224.0.0.0/3">224.0.0.0/3</a> gw 192.168.47.100<br>socat STDIO UDP4-DATAGRAM:<a href="http://224.1.0.1:6666">224.1.0.1:6666</a>,bind=:6666,range=<a href="http://192.168.47.0/24,ip-add-membership=224.1.0.1:192.168.47.100">192.168.47.0/24,ip-add-membership=224.1.0.1:192.168.47.100</a><br>
<br>socat sets up a peer-peer multicast communication, the expected results are echoed data on the sending end and data on the receiving end.<br><br>When attempting this test with bonded IB interfaces, I only get get the echoed data on the sending end and nothing on the recieving end.<br>
<br>here are the results from dmesg<br>[ 859.128720] bonding: bond3 is being created...<br>[ 859.129468] bonding: bond3: setting mode to active-backup (1).<br>[ 859.129501] bonding: bond3: Setting MII monitoring interval to 100.<br>
[ 859.141557] bonding: bond3: doing slave updates when interface is down.<br>[ 859.141563] bonding: bond3: Adding slave ib0.<br>[ 859.141566] bonding bond3: master_dev is not up in bond_enslave<br>[ 859.141567] bonding: bond3: Warning: enslaved VLAN challenged slave ib0. Adding VLANs will be blocked as long as ib0 is part of bond bond3<br>
[ 859.141570] bonding: bond3: Warning: The first slave device specified does not support setting the MAC address. Setting fail_over_mac to active.<7>ib0: bringing up interface<br>[ 859.182437] ib0: starting multicast thread<br>
[ 859.182568] ib0: joining MGID ff12:401b:ffff:0000:0000:0000:ffff:ffff<br>[ 859.182580] ib0: restarting multicast task<br>[ 859.182583] ib0: stopping multicast thread<br>[ 859.182586] ib0: adding multicast entry for mgid ff12:401b:ffff:0000:0000:0000:0000:0001<br>
[ 859.182589] ib0: starting multicast thread<br>[ 859.182739] ib0: join completion for ff12:401b:ffff:0000:0000:0000:ffff:ffff (status 0)<br>[ 859.182951] ib0: Created ah ffff8804379e8680<br>[ 859.182954] ib0: MGID ff12:401b:ffff:0000:0000:0000:ffff:ffff AV ffff8804379e8680, LID 0xc000, SL 0<br>
[ 859.183088] ib0: joining MGID ff12:401b:ffff:0000:0000:0000:0000:0001<br>[ 859.183222] ib0: join completion for ff12:401b:ffff:0000:0000:0000:0000:0001 (status 0)<br>[ 859.183354] ib0: Created ah ffff8804389a9880<br>
[ 859.183359] ib0: MGID ff12:401b:ffff:0000:0000:0000:0000:0001 AV ffff8804389a9880, LID 0xc001, SL 0<br>[ 859.184369] bonding: bond3: enslaving ib0 as a backup interface with a down link.<br>[ 859.186365] ib0: successfully joined all multicast groups<br>
[ 859.186385] ib0: restarting multicast task<br>[ 859.186386] ib0: stopping multicast thread<br>[ 859.186389] ib0: starting multicast thread<br>[ 859.186500] ib0: successfully joined all multicast groups<br>[ 859.188608] bonding: bond3: doing slave updates when interface is down.<br>
[ 859.188613] bonding: bond3: Adding slave ib1.<br>[ 859.188615] bonding bond3: master_dev is not up in bond_enslave<br>[ 859.188617] bonding: bond3: Warning: enslaved VLAN challenged slave ib1. Adding VLANs will be blocked as long as ib1 is part of bond bond3<br>
[ 859.221889] ib1: bringing up interface<br>[ 859.222359] ib1: starting multicast thread<br>[ 859.222483] ib1: joining MGID ff12:401b:ffff:0000:0000:0000:ffff:ffff<br>[ 859.222494] ib1: restarting multicast task<br>[ 859.222498] ib1: stopping multicast thread<br>
[ 859.222500] ib1: adding multicast entry for mgid ff12:401b:ffff:0000:0000:0000:0000:0001<br>[ 859.222503] ib1: starting multicast thread<br>[ 859.224240] bonding: bond3: enslaving ib1 as a backup interface with a down link.<br>
[ 859.224634] ib1: join completion for ff12:401b:ffff:0000:0000:0000:ffff:ffff (status 0)<br>[ 859.224837] ib1: Created ah ffff880436cc8400<br>[ 859.224841] ib1: MGID ff12:401b:ffff:0000:0000:0000:ffff:ffff AV ffff880436cc8400, LID 0xc000, SL 0<br>
[ 859.224968] ib1: joining MGID ff12:401b:ffff:0000:0000:0000:0000:0001<br>[ 859.225099] ib1: join completion for ff12:401b:ffff:0000:0000:0000:0000:0001 (status 0)<br>[ 859.225223] ib1: Created ah ffff88043840fec0<br>
[ 859.225228] ib1: MGID ff12:401b:ffff:0000:0000:0000:0000:0001 AV ffff88043840fec0, LID 0xc001, SL 0<br>[ 859.226956] ib1: successfully joined all multicast groups<br>[ 859.226961] ib1: restarting multicast task<br>[ 859.226962] ib1: stopping multicast thread<br>
[ 859.226964] ib1: starting multicast thread<br>[ 859.227074] ib1: successfully joined all multicast groups<br>[ 859.228034] ib0: mtu > 2044 will cause multicast packet drops.<br>[ 859.229779] ib1: mtu > 2044 will cause multicast packet drops.<br>
[ 859.233134] ADDRCONF(NETDEV_UP): bond3: link is not ready<br>[ 859.233153] bonding: bond3: link status definitely up for interface ib0.<br>[ 859.233156] bonding: bond3: making interface ib0 the new active one.<br>[ 859.233167] ib0: restarting multicast task<br>
[ 859.233170] ib0: stopping multicast thread<br>[ 859.233172] ib0: adding multicast entry for mgid 0001:0000:0000:0000:0000:0000:0000:0000<br>[ 859.233175] ib0: starting multicast thread<br>[ 859.233178] bonding: bond3: first active interface up!<br>
[ 859.233180] bonding: bond3: link status definitely up for interface ib1.<br>[ 859.233289] ib0: joining MGID 0001:0000:0000:0000:0000:0000:0000:0000<br>[ 859.234904] ADDRCONF(NETDEV_CHANGE): bond3: link becomes ready<br>
[ 859.234944] ib0: restarting multicast task<br>[ 859.234948] ib0: stopping multicast thread<br>[ 859.234951] ib0: adding multicast entry for mgid ff12:601b:ffff:0000:0000:0001:ff00:f778<br>[ 859.234954] ib0: starting multicast thread<br>
[ 859.235069] ib0: joining MGID ff12:601b:ffff:0000:0000:0001:ff00:f778<br>[ 859.235090] ib0: join completion for 0001:0000:0000:0000:0000:0000:0000:0000 (status -22)<br>[ 859.235095] ib0: multicast join failed for 0001:0000:0000:0000:0000:0000:0000:0000, status -22<br>
[ 859.235162] ib0: restarting multicast task<br>[ 859.235163] ib0: stopping multicast thread<br>[ 859.235166] ib0: adding multicast entry for mgid ff12:401b:ffff:0000:0000:0000:0000:00fb<br>[ 859.235168] ib0: starting multicast thread<br>
[ 859.235200] ib0: join completion for ff12:601b:ffff:0000:0000:0001:ff00:f778 (status 0)<br>[ 859.235304] ib0: joining MGID 0001:0000:0000:0000:0000:0000:0000:0000<br>[ 859.235343] ib0: Created ah ffff88043a9b9440<br>
[ 859.235347] ib0: MGID ff12:601b:ffff:0000:0000:0001:ff00:f778 AV ffff88043a9b9440, LID 0xc002, SL 0<br>[ 859.235408] ib0: join completion for 0001:0000:0000:0000:0000:0000:0000:0000 (status -22)<br>[ 859.235412] ib0: multicast join failed for 0001:0000:0000:0000:0000:0000:0000:0000, status -22<br>
[ 859.235481] ib0: joining MGID 0001:0000:0000:0000:0000:0000:0000:0000<br>[ 859.235592] ib0: join completion for 0001:0000:0000:0000:0000:0000:0000:0000 (status -22)<br>[ 859.235596] ib0: multicast join failed for 0001:0000:0000:0000:0000:0000:0000:0000, status -22<br>
[ 859.260028] ib0: setting up send only multicast group for ff12:601b:ffff:0000:0000:0000:0000:0016<br>[ 859.260042] ib0: no multicast record for ff12:601b:ffff:0000:0000:0000:0000:0016, starting join<br>[ 859.260136] ib0: multicast join failed for ff12:601b:ffff:0000:0000:0000:0000:0016, status -22<br>
[ 859.263792] ib0: setting up send only multicast group for ff12:401b:ffff:0000:0000:0000:0000:0016<br>[ 859.263806] ib0: no multicast record for ff12:401b:ffff:0000:0000:0000:0000:0016, starting join<br>[ 859.263883] ib0: multicast join failed for ff12:401b:ffff:0000:0000:0000:0000:0016, status -22<br>
[ 860.600025] ib0: setting up send only multicast group for ff12:601b:ffff:0000:0000:0000:0000:0002<br>[ 860.600035] ib0: no multicast record for ff12:601b:ffff:0000:0000:0000:0000:0002, starting join<br>[ 860.600149] ib0: multicast join failed for ff12:601b:ffff:0000:0000:0000:0000:0002, status -22<br>
[ 863.230303] ib0: joining MGID 0001:0000:0000:0000:0000:0000:0000:0000<br>[ 863.230406] ib0: join completion for 0001:0000:0000:0000:0000:0000:0000:0000 (status -22)<br>[ 863.230411] ib0: multicast join failed for 0001:0000:0000:0000:0000:0000:0000:0000, status -22<br>
[ 864.600035] ib0: no multicast record for ff12:601b:ffff:0000:0000:0000:0000:0002, starting join<br>[ 864.600124] ib0: multicast join failed for ff12:601b:ffff:0000:0000:0000:0000:0002, status -22<br>[ 868.600034] ib0: no multicast record for ff12:601b:ffff:0000:0000:0000:0000:0002, starting join<br>
[ 868.600119] ib0: multicast join failed for ff12:601b:ffff:0000:0000:0000:0000:0002, status -22<br>[ 868.620031] ib0: no multicast record for ff12:401b:ffff:0000:0000:0000:0000:0016, starting join<br>[ 868.620112] ib0: multicast join failed for ff12:401b:ffff:0000:0000:0000:0000:0016, status -22<br>
[ 869.100039] ib0: no multicast record for ff12:601b:ffff:0000:0000:0000:0000:0016, starting join<br>[ 869.100124] ib0: multicast join failed for ff12:601b:ffff:0000:0000:0000:0000:0016, status -22<br>[ 869.600029] bond3: no IPv6 routers present<br>
[ 879.230231] ib0: joining MGID 0001:0000:0000:0000:0000:0000:0000:0000<br>[ 879.230349] ib0: join completion for 0001:0000:0000:0000:0000:0000:0000:0000 (status -22)<br>[ 879.230355] ib0: multicast join failed for 0001:0000:0000:0000:0000:0000:0000:0000, status -22<br>
[ 886.919993] ib0: restarting multicast task<br>[ 886.919997] ib0: stopping multicast thread<br>[ 886.920002] ib0: adding multicast entry for mgid ff12:401b:ffff:0000:0000:0000:0001:0001<br>[ 886.920005] ib0: starting multicast thread<br>
[ 886.920140] ib0: joining MGID 0001:0000:0000:0000:0000:0000:0000:0000<br>[ 886.920244] ib0: join completion for 0001:0000:0000:0000:0000:0000:0000:0000 (status -22)<br>[ 886.920248] ib0: multicast join failed for 0001:0000:0000:0000:0000:0000:0000:0000, status -22<br>
[ 886.934421] ib0: no multicast record for ff12:401b:ffff:0000:0000:0000:0000:0016, starting join<br>[ 886.934520] ib0: multicast join failed for ff12:401b:ffff:0000:0000:0000:0000:0016, status -22<br>[ 889.000014] ib0: no multicast record for ff12:401b:ffff:0000:0000:0000:0000:0016, starting join<br>
[ 889.000102] ib0: multicast join failed for ff12:401b:ffff:0000:0000:0000:0000:0016, status -22<br>[ 899.053269] ib0: restarting multicast task<br>[ 899.053273] ib0: stopping multicast thread<br>[ 899.053277] ib0: deleting multicast group ff12:401b:ffff:0000:0000:0000:0001:0001<br>
[ 899.053280] ib0: deleting multicast group ff12:401b:ffff:0000:0000:0000:0001:0001<br>[ 899.053285] ib0: starting multicast thread<br>[ 899.053430] ib0: joining MGID 0001:0000:0000:0000:0000:0000:0000:0000<br>[ 899.053540] ib0: join completion for 0001:0000:0000:0000:0000:0000:0000:0000 (status -22)<br>
[ 899.053544] ib0: multicast join failed for 0001:0000:0000:0000:0000:0000:0000:0000, status -22<br>[ 899.073152] ib0: no multicast record for ff12:401b:ffff:0000:0000:0000:0000:0016, starting join<br>[ 899.073241] ib0: multicast join failed for ff12:401b:ffff:0000:0000:0000:0000:0016, status -22<br>
[ 903.420017] ib0: no multicast record for ff12:401b:ffff:0000:0000:0000:0000:0016, starting join<br>[ 903.420100] ib0: multicast join failed for ff12:401b:ffff:0000:0000:0000:0000:0016, status -22<br><br><br>Thank you,<br>
Dennis P.<br><br><div class="gmail_quote">On Mon, Apr 20, 2009 at 8:17 AM, Or Gerlitz <span dir="ltr"><<a href="mailto:ogerlitz@voltaire.com">ogerlitz@voltaire.com</a>></span> wrote:<br><blockquote class="gmail_quote" style="border-left: 1px solid rgb(204, 204, 204); margin: 0pt 0pt 0pt 0.8ex; padding-left: 1ex;">
<div class="im">Dennis Portello wrote:<br>
> Regular TCP/IP unicast works, though dmesg is full of warning about<br>
> multicast failing. Multicast does not work at all.<br>
<br>
</div>Unicast IP relies on ARP and IPoIB ARPs use the broadcast multicast group, so<br>
IB multicast does work on your setup... to see what IB multicast groups are being<br>
joined by your IPoIB devices, you can use the ipoib debugfs entries<br>
<br>
$ mount -t debugfs none /sys/kernel/debug<br>
$ cat /sys/kernel/debug/ipoib/ibxxx_mcg<br>
<br>
see Documentation/infiniband/ipoib.txt for more info<br>
<font color="#888888"><br>
Or.<br>
</font></blockquote></div><br>