<html xmlns:v="urn:schemas-microsoft-com:vml" xmlns:o="urn:schemas-microsoft-com:office:office" xmlns:w="urn:schemas-microsoft-com:office:word" xmlns:m="http://schemas.microsoft.com/office/2004/12/omml" xmlns="http://www.w3.org/TR/REC-html40">
<head>
<meta http-equiv=Content-Type content="text/html; charset=us-ascii">
<meta name=Generator content="Microsoft Word 12 (filtered medium)">
<style>
<!--
/* Font Definitions */
@font-face
{font-family:"Cambria Math";
panose-1:2 4 5 3 5 4 6 3 2 4;}
@font-face
{font-family:Calibri;
panose-1:2 15 5 2 2 2 4 3 2 4;}
@font-face
{font-family:Tahoma;
panose-1:2 11 6 4 3 5 4 4 2 4;}
@font-face
{font-family:Consolas;
panose-1:2 11 6 9 2 2 4 3 2 4;}
/* Style Definitions */
p.MsoNormal, li.MsoNormal, div.MsoNormal
{margin:0cm;
margin-bottom:.0001pt;
font-size:12.0pt;
font-family:"Times New Roman","serif";
color:black;}
a:link, span.MsoHyperlink
{mso-style-priority:99;
color:blue;
text-decoration:underline;}
a:visited, span.MsoHyperlinkFollowed
{mso-style-priority:99;
color:purple;
text-decoration:underline;}
pre
{mso-style-priority:99;
mso-style-link:"HTML Preformatted Char";
margin:0cm;
margin-bottom:.0001pt;
font-size:10.0pt;
font-family:"Courier New";
color:black;}
span.HTMLPreformattedChar
{mso-style-name:"HTML Preformatted Char";
mso-style-priority:99;
mso-style-link:"HTML Preformatted";
font-family:"Consolas","serif";
color:black;}
span.EmailStyle19
{mso-style-type:personal-reply;
font-family:"Calibri","sans-serif";
color:#1F497D;}
.MsoChpDefault
{mso-style-type:export-only;
font-size:10.0pt;}
@page WordSection1
{size:612.0pt 792.0pt;
margin:72.0pt 72.0pt 72.0pt 72.0pt;}
div.WordSection1
{page:WordSection1;}
-->
</style>
<!--[if gte mso 9]><xml>
<o:shapedefaults v:ext="edit" spidmax="1026" />
</xml><![endif]--><!--[if gte mso 9]><xml>
<o:shapelayout v:ext="edit">
<o:idmap v:ext="edit" data="1" />
</o:shapelayout></xml><![endif]-->
</head>
<body bgcolor=white lang=EN-GB link=blue vlink=purple>
<div class=WordSection1>
<p class=MsoNormal><span style='font-size:11.0pt;font-family:"Calibri","sans-serif";
color:#1F497D'>The InfiniBand fabric knows very little about IpoIB, it is
handled by the host OS stack, however it does need capabilities such as
multicast to work properly for ARP name resolution.<o:p></o:p></span></p>
<p class=MsoNormal><span style='font-size:11.0pt;font-family:"Calibri","sans-serif";
color:#1F497D'><o:p> </o:p></span></p>
<p class=MsoNormal><span style='font-size:11.0pt;font-family:"Calibri","sans-serif";
color:#1F497D'>The problem you describe sounds similar to a situation I
encountered running multiple, incompatible SM’s.<o:p></o:p></span></p>
<p class=MsoNormal><span style='font-size:11.0pt;font-family:"Calibri","sans-serif";
color:#1F497D'>Make sure you only have a single vendor SM. Whilst
the OFED SM build is fine, I have found many vendors hack their distro’s
so they either ignore or always win the SM election. Explicitly
disable all SM’s on all environments you don’t want to be running. Don’t
rely on SM priority across different implementations . I’d
recommend running openSM on Linux and disabling all others.<o:p></o:p></span></p>
<p class=MsoNormal><span style='font-size:11.0pt;font-family:"Calibri","sans-serif";
color:#1F497D'><o:p> </o:p></span></p>
<p class=MsoNormal><span style='font-size:11.0pt;font-family:"Calibri","sans-serif";
color:#1F497D'>Identifying that no SM running is easy, since the ports don’t
get LIDs, however when multiple SM’s are running it sort of works, since
the different SM’s discover which LIDS have been allocated when they scan
the fabric. The problem I saw was with multicast, each SM had its independent
and different view of the MC nodes and paths.<o:p></o:p></span></p>
<p class=MsoNormal><span style='font-size:11.0pt;font-family:"Calibri","sans-serif";
color:#1F497D'><o:p> </o:p></span></p>
<p class=MsoNormal><span style='font-size:11.0pt;font-family:"Calibri","sans-serif";
color:#1F497D'>Sun did their own InfiniBand stack implementation, including SM
and is completely independent of OFED. I used it a few years ago
and IPoIB interoperated fine with Linux.<o:p></o:p></span></p>
<p class=MsoNormal><span style='font-size:11.0pt;font-family:"Calibri","sans-serif";
color:#1F497D'><o:p> </o:p></span></p>
<p class=MsoNormal><span style='font-size:11.0pt;font-family:"Calibri","sans-serif";
color:#1F497D'>You don’t say which InfiniBand distro you are running on
windows.<o:p></o:p></span></p>
<p class=MsoNormal><span style='font-size:11.0pt;font-family:"Calibri","sans-serif";
color:#1F497D'><o:p> </o:p></span></p>
<p class=MsoNormal><span style='font-size:11.0pt;font-family:"Calibri","sans-serif";
color:#1F497D'>Linux IPoIB defaults to Connected mode. This is an
optional feature of IpoIB. You may want to try setting it to the
mandatory datagram mode in each of your environments. You can disable CM mode
in openib.conf on the linux nodes<o:p></o:p></span></p>
<p class=MsoNormal><span style='font-size:11.0pt;font-family:"Calibri","sans-serif";
color:#1F497D'><o:p> </o:p></span></p>
<div>
<div style='border:none;border-top:solid #B5C4DF 1.0pt;padding:3.0pt 0cm 0cm 0cm'>
<p class=MsoNormal><b><span lang=EN-US style='font-size:10.0pt;font-family:
"Tahoma","sans-serif";color:windowtext'>From:</span></b><span lang=EN-US
style='font-size:10.0pt;font-family:"Tahoma","sans-serif";color:windowtext'> ewg-bounces@lists.openfabrics.org
[mailto:ewg-bounces@lists.openfabrics.org] <b>On Behalf Of </b>Jeff Becker<br>
<b>Sent:</b> 29 June 2010 19:26<br>
<b>To:</b> Matt Breitbach; ewg@lists.openfabrics.org<br>
<b>Subject:</b> Re: [ewg] Infiniband Interoperability<o:p></o:p></span></p>
</div>
</div>
<p class=MsoNormal><o:p> </o:p></p>
<p class=MsoNormal><span style='font-size:13.5pt'>Hi Matt</span><span
style='font-size:18.0pt'><br>
<br>
</span>On 06/29/10 10:15, Matt Breitbach wrote: <o:p></o:p></p>
<pre>So I know that this message isn't about starting a new group. I've actually<o:p></o:p></pre><pre>tried to join one of the mailing lists but it failed to sign me up.<o:p></o:p></pre><pre> <o:p></o:p></pre>
<p class=MsoNormal><span style='font-size:13.5pt'>You probably tried to sign up
for <a href="mailto:general@lists.openfabrics.org">general@lists.openfabrics.org</a>,
which doesn't really exist anymore (although the archives are still there for
searching). It sounds like you should post your question on <a
href="mailto:ewg@lists.openfabrics.org">ewg@lists.openfabrics.org</a>. I'll
forward it for you.<br>
<br>
Jeff Becker<br>
OpenFabrics Server admin<br>
<br>
<br>
</span><o:p></o:p></p>
<pre><o:p> </o:p></pre><pre>I'm working on getting an InfiniBand setup working in a mixed environment of<o:p></o:p></pre><pre>Windows, Linux, and OpenSolaris. I'm having huge difficulties getting the<o:p></o:p></pre><pre>IB network to be stable. We're working with IPoIB mainly, and get drops<o:p></o:p></pre><pre>between the Windows and Linux/OpenSolaris systems.<o:p></o:p></pre><pre><o:p> </o:p></pre><pre>We had a professional take a look at our configuration, and he thought that<o:p></o:p></pre><pre>OpenSM was configured properly, and actually had the network stable for<o:p></o:p></pre><pre>about 2 days, but it degraded severely after that. We are now to the point<o:p></o:p></pre><pre>that most times the OpenSolaris box is unreachable from the Windows systems,<o:p></o:p></pre><pre>and sometimes from the Linux system (which is running OpenSM).<o:p></o:p></pre><pre><o:p> </o:p></pre><pre>Is there any direction that you would be able to point me in to get some<o:p></o:p></pre><pre>advice on this or some high-end consulting? We've invested nearly 3 months<o:p></o:p></pre><pre>into this project only to have a bladecenter with 4 IB Capable blades and a<o:p></o:p></pre><pre>SuperMicro 4U server that can't reliably communicate over the InfiniBand<o:p></o:p></pre><pre>network.<o:p></o:p></pre><pre><o:p> </o:p></pre><pre>-Matt Breitbach<o:p></o:p></pre><pre>zfsbuild.com (our storage writeup about OpenSolaris ZFS over InfiniBand)<o:p></o:p></pre><pre><o:p> </o:p></pre><pre><o:p> </o:p></pre><pre> <o:p></o:p></pre>
<p class=MsoNormal><o:p> </o:p></p>
</div>
</body>
</html>