<!DOCTYPE HTML PUBLIC "-//W3C//DTD HTML 3.2//EN">
<HTML>
<HEAD>
<META HTTP-EQUIV="Content-Type" CONTENT="text/html; charset=US-ASCII">
<META NAME="Generator" CONTENT="MS Exchange Server version 5.5.2654.45">
<TITLE>RE: [openib-general] IPoIB still not working</TITLE>
</HEAD>
<BODY>
<P><FONT SIZE=2>Forgive me for not following the entire thread. </FONT>
<BR><FONT SIZE=2>But I did take a look at the log files:</FONT>
</P>
<P><FONT SIZE=2>The 64bit version have the following multicast activities:</FONT>
<BR><FONT SIZE=2>1. Port 0x0002c9010ad258f1 joining MLID 0xC000 -> success.</FONT>
<BR><FONT SIZE=2> Note that MLID 0xC000 is predefined (IPoIB).</FONT>
<BR><FONT SIZE=2> MGID....................0xff12401bffff0000 : 0x00000000ffffffff</FONT>
<BR><FONT SIZE=2> PortGid.................0xfe80000000000000 : 0x0002c9010ad258f1</FONT>
<BR><FONT SIZE=2> qkey....................0x0</FONT>
<BR><FONT SIZE=2> Mlid....................0x0</FONT>
<BR><FONT SIZE=2> ScopeState..............0x1</FONT>
<BR><FONT SIZE=2> Rate....................0x0</FONT>
<BR><FONT SIZE=2> Mtu.....................0x0</FONT>
</P>
<P><FONT SIZE=2>2. Port 0x0002c9010ad258f1 joining MLID 0xC000. (Again).</FONT>
<BR><FONT SIZE=2> MGID....................0xff12401bffff0000 : 0x00000000ffffffff</FONT>
<BR><FONT SIZE=2> PortGid.................0xfe80000000000000 : 0x0002c9010ad258f1</FONT>
<BR><FONT SIZE=2> qkey....................0x1B0B0000</FONT>
<BR><FONT SIZE=2> Mlid....................0xC000</FONT>
<BR><FONT SIZE=2> ScopeState..............0x11</FONT>
<BR><FONT SIZE=2> Rate....................0x3</FONT>
<BR><FONT SIZE=2> Mtu.....................0x4</FONT>
<BR><FONT SIZE=2> -> considered as an update to the scope state.</FONT>
</P>
<P><FONT SIZE=2>3. Request to join :</FONT>
<BR><FONT SIZE=2> MGID....................0xff12601bffff0000 : 0x0000000000000016</FONT>
<BR><FONT SIZE=2> PortGid.................0xfe80000000000000 : 0x0002c9010ad258f1</FONT>
<BR><FONT SIZE=2> qkey....................0x0</FONT>
<BR><FONT SIZE=2> Mlid....................0x0</FONT>
<BR><FONT SIZE=2> ScopeState..............0x1</FONT>
<BR><FONT SIZE=2> Rate....................0x0</FONT>
<BR><FONT SIZE=2> Mtu.....................0x0</FONT>
<BR><FONT SIZE=2>Results with - ERR 1B10: Provided Join State != FullMember - required for create.</FONT>
<BR><FONT SIZE=2>You can not create a group if you are not a full member.</FONT>
</P>
<P><FONT SIZE=2>4. A sequence of requests arrive to create MGRPs with several MGIDs:</FONT>
<BR><FONT SIZE=2>MGID 0xff12601bffff0000:0x0000000000000002</FONT>
<BR><FONT SIZE=2>MGID 0xff12601bffff0000:0x0000000000000016</FONT>
<BR><FONT SIZE=2>MGID 0xff12601bffff0000:0x00000001ffd258f1</FONT>
<BR><FONT SIZE=2>All fail due to the same join state issue.</FONT>
</P>
<P><FONT SIZE=2>Inspecting the 32bit version:</FONT>
<BR><FONT SIZE=2>I see only one request to join</FONT>
<BR><FONT SIZE=2>Port 0x0002c90107fc5be1 joining MLID 0xC000</FONT>
<BR><FONT SIZE=2>And it succeeds </FONT>
</P>
<P><FONT SIZE=2>Hope this helps. </FONT>
</P>
<P><FONT SIZE=2>Eitan Zahavi</FONT>
<BR><FONT SIZE=2>Design Technology Director</FONT>
<BR><FONT SIZE=2>Mellanox Technologies LTD</FONT>
<BR><FONT SIZE=2>Tel:+972-4-9097208</FONT>
<BR><FONT SIZE=2>Fax:+972-4-9593245</FONT>
<BR><FONT SIZE=2>P.O. Box 586 Yokneam 20692 ISRAEL</FONT>
</P>
<BR>
<P><FONT SIZE=2>-----Original Message-----</FONT>
<BR><FONT SIZE=2>From: Woodruff, Robert J [<A HREF="mailto:robert.j.woodruff@intel.com">mailto:robert.j.woodruff@intel.com</A>] </FONT>
<BR><FONT SIZE=2>Sent: Wednesday, December 08, 2004 3:12 AM</FONT>
<BR><FONT SIZE=2>To: Roland Dreier</FONT>
<BR><FONT SIZE=2>Cc: openib-general@openib.org</FONT>
<BR><FONT SIZE=2>Subject: RE: [openib-general] IPoIB still not working</FONT>
</P>
<P><FONT SIZE=2> </FONT>
<BR><FONT SIZE=2>Here are some log files.</FONT>
</P>
<P><FONT SIZE=2>First file, mcast-64.log is the /var/log/messages output </FONT>
<BR><FONT SIZE=2>from the patch you sent on the 64-bit system.</FONT>
</P>
<P><FONT SIZE=2>Next log files is the opensm log file </FONT>
<BR><FONT SIZE=2>osm-64bit.log</FONT>
</P>
<P><FONT SIZE=2>Next log file is the opensm log file when running the 32-node.</FONT>
<BR><FONT SIZE=2>osm-32-bit.log</FONT>
</P>
<BR>
<P><FONT SIZE=2>In the passing case, ipoib sends 2 MCM messages and opensm has no</FONT>
<BR><FONT SIZE=2>complaints.</FONT>
<BR><FONT SIZE=2>Search for MCMember Record in osm-32-bit.log</FONT>
</P>
<P><FONT SIZE=2>In the failing case, ipoib sends 2 MCM messages that look similar with</FONT>
<BR><FONT SIZE=2>no errors</FONT>
<BR><FONT SIZE=2>reported. However, in the failing case ipoib continues to send MCM</FONT>
<BR><FONT SIZE=2>messages</FONT>
<BR><FONT SIZE=2>that opensm rejects. In the failing case there are a couple of </FONT>
<BR><FONT SIZE=2>differences, first the MGID lower 32-bits appear to be 0xffffffff in the</FONT>
<BR><FONT SIZE=2>passing case and something else when it fails. </FONT>
<BR><FONT SIZE=2>Second, it appears that perhaps the opensm is rejecting the messages</FONT>
<BR><FONT SIZE=2>because</FONT>
<BR><FONT SIZE=2>of a bug where the scope and join fields are reversed when extracted</FONT>
<BR><FONT SIZE=2>from</FONT>
<BR><FONT SIZE=2>the mad. In the passing case, since the lower 32 bits of the mgid are</FONT>
<BR><FONT SIZE=2>0xfffffffff,</FONT>
<BR><FONT SIZE=2>you never get to the code that checks the join member. </FONT>
<BR><FONT SIZE=2>Someone that understands opensm should look at this, but Sean</FONT>
<BR><FONT SIZE=2>I think it may be wrong.</FONT>
</P>
<P><FONT SIZE=2>This however does not explain why in the failing case, ipoib continues</FONT>
<BR><FONT SIZE=2>to </FONT>
<BR><FONT SIZE=2>try to join the mcast group unless it is having difficulties after</FONT>
<BR><FONT SIZE=2>trying yo </FONT>
<BR><FONT SIZE=2>join he group and decides to re-try, with the subsequent re-tries to </FONT>
<BR><FONT SIZE=2>join being failed by opensm.</FONT>
</P>
</BODY>
</HTML>