<html><body><div style="font-family: times new roman, new york, times, serif; font-size: 12pt; color: #000000"><div>Ira,<br></div><div><br></div><div>I think i've made some progress.</div><div><br></div><div>In the past, I had the following when trying to run ibstat:</div><div><br></div><div><p style="margin: 0px; font-size: 12px; font-family: Monaco;" data-mce-style="margin: 0px; font-size: 12px; font-family: Monaco;">ibstat</p><p style="margin: 0px; font-size: 12px; font-family: Monaco;" data-mce-style="margin: 0px; font-size: 12px; font-family: Monaco;">CA 'qib0'</p><p style="margin: 0px; font-size: 12px; font-family: Monaco;" data-mce-style="margin: 0px; font-size: 12px; font-family: Monaco;"> CA type: InfiniPath_QMH7342</p><p style="margin: 0px; font-size: 12px; font-family: Monaco;" data-mce-style="margin: 0px; font-size: 12px; font-family: Monaco;"> Number of ports: 2</p><p style="margin: 0px; font-size: 12px; font-family: Monaco;" data-mce-style="margin: 0px; font-size: 12px; font-family: Monaco;"> Firmware version:</p><p style="margin: 0px; font-size: 12px; font-family: Monaco;" data-mce-style="margin: 0px; font-size: 12px; font-family: Monaco;"> Hardware version: 2</p><p style="margin: 0px; font-size: 12px; font-family: Monaco;" data-mce-style="margin: 0px; font-size: 12px; font-family: Monaco;"> Node GUID: 0x00117500007965b4</p><p style="margin: 0px; font-size: 12px; font-family: Monaco;" data-mce-style="margin: 0px; font-size: 12px; font-family: Monaco;"> System image GUID: 0x00117500007965b4</p><p style="margin: 0px; font-size: 12px; font-family: Monaco;" data-mce-style="margin: 0px; font-size: 12px; font-family: Monaco;"> Port 1:</p><p style="margin: 0px; font-size: 12px; font-family: Monaco;" data-mce-style="margin: 0px; font-size: 12px; font-family: Monaco;"> State: Down</p><p style="margin: 0px; font-size: 12px; font-family: Monaco;" data-mce-style="margin: 0px; font-size: 12px; font-family: Monaco;"> Physical state: Disabled</p><p style="margin: 0px; font-size: 12px; font-family: Monaco;" data-mce-style="margin: 0px; font-size: 12px; font-family: Monaco;"> Rate: 40</p><p style="margin: 0px; font-size: 12px; font-family: Monaco;" data-mce-style="margin: 0px; font-size: 12px; font-family: Monaco;"> Base lid: 65535</p><p style="margin: 0px; font-size: 12px; font-family: Monaco;" data-mce-style="margin: 0px; font-size: 12px; font-family: Monaco;"> LMC: 0</p><p style="margin: 0px; font-size: 12px; font-family: Monaco;" data-mce-style="margin: 0px; font-size: 12px; font-family: Monaco;"> SM lid: 65535</p><p style="margin: 0px; font-size: 12px; font-family: Monaco;" data-mce-style="margin: 0px; font-size: 12px; font-family: Monaco;"> Capability mask: 0x07600868</p><p style="margin: 0px; font-size: 12px; font-family: Monaco;" data-mce-style="margin: 0px; font-size: 12px; font-family: Monaco;"> Port GUID: 0x00117500007965b4</p><p style="margin: 0px; font-size: 12px; font-family: Monaco;" data-mce-style="margin: 0px; font-size: 12px; font-family: Monaco;"> Link layer: InfiniBand</p><p style="margin: 0px; font-size: 12px; font-family: Monaco;" data-mce-style="margin: 0px; font-size: 12px; font-family: Monaco;"> Port 2: </p><p style="margin: 0px; font-size: 12px; font-family: Monaco;" data-mce-style="margin: 0px; font-size: 12px; font-family: Monaco;"> State: Down</p><p style="margin: 0px; font-size: 12px; font-family: Monaco;" data-mce-style="margin: 0px; font-size: 12px; font-family: Monaco;"> Physical state: Disabled</p><p style="margin: 0px; font-size: 12px; font-family: Monaco;" data-mce-style="margin: 0px; font-size: 12px; font-family: Monaco;"> Rate: 40</p><p style="margin: 0px; font-size: 12px; font-family: Monaco;" data-mce-style="margin: 0px; font-size: 12px; font-family: Monaco;"> Base lid: 65535</p><p style="margin: 0px; font-size: 12px; font-family: Monaco;" data-mce-style="margin: 0px; font-size: 12px; font-family: Monaco;"> LMC: 0</p><p style="margin: 0px; font-size: 12px; font-family: Monaco;" data-mce-style="margin: 0px; font-size: 12px; font-family: Monaco;"> SM lid: 65535</p><p style="margin: 0px; font-size: 12px; font-family: Monaco;" data-mce-style="margin: 0px; font-size: 12px; font-family: Monaco;"> Capability mask: 0x07600868</p><p style="margin: 0px; font-size: 12px; font-family: Monaco;" data-mce-style="margin: 0px; font-size: 12px; font-family: Monaco;"> Port GUID: 0x00117500007965b5</p><p style="margin: 0px; font-size: 12px; font-family: Monaco;" data-mce-style="margin: 0px; font-size: 12px; font-family: Monaco;"> Link layer: InfiniBand</p><p style="margin: 0px; font-size: 12px; font-family: Monaco;" data-mce-style="margin: 0px; font-size: 12px; font-family: Monaco;"><br></p><p style="margin: 0px; font-size: 12px; font-family: Monaco;" data-mce-style="margin: 0px; font-size: 12px; font-family: Monaco;">as you can see, the physical state is shown as Disabled.</p><p style="margin: 0px; font-size: 12px; font-family: Monaco;" data-mce-style="margin: 0px; font-size: 12px; font-family: Monaco;"><br></p><p style="margin: 0px; font-size: 12px; font-family: Monaco;" data-mce-style="margin: 0px; font-size: 12px; font-family: Monaco;">This was the case for both Ubuntu 12.04 (using roland dreier pap) and 14.04 using stock infiniband tools.</p><p style="margin: 0px; font-size: 12px; font-family: Monaco;" data-mce-style="margin: 0px; font-size: 12px; font-family: Monaco;"><br></p><p style="margin: 0px; font-size: 12px; font-family: Monaco;" data-mce-style="margin: 0px; font-size: 12px; font-family: Monaco;">I couldn't figure out what's going on as the exact estup with Mellanox card works perfectly well.</p><p style="margin: 0px; font-size: 12px; font-family: Monaco;" data-mce-style="margin: 0px; font-size: 12px; font-family: Monaco;"><br></p><p style="margin: 0px; font-size: 12px; font-family: Monaco;" data-mce-style="margin: 0px; font-size: 12px; font-family: Monaco;">I was just looking at the startup scripts for the openibd script that comes with qlogic/intel drivers and i've noticed that the script starts a file called /etc/infiniband/truescale.cmds. I've copied the script across from a centos server and started it. And now I have a link shown as ACTIVE and I can see other nodes when running iblinkinfo.</p><p style="margin: 0px; font-size: 12px; font-family: Monaco;" data-mce-style="margin: 0px; font-size: 12px; font-family: Monaco;"><br></p><p style="margin: 0px; font-size: 12px; font-family: Monaco;" data-mce-style="margin: 0px; font-size: 12px; font-family: Monaco;">So, I will now try to get ipoib working.</p><p style="margin: 0px; font-size: 12px; font-family: Monaco;" data-mce-style="margin: 0px; font-size: 12px; font-family: Monaco;"><br></p><p style="margin: 0px; font-size: 12px; font-family: Monaco;" data-mce-style="margin: 0px; font-size: 12px; font-family: Monaco;">Cheers</p><p style="margin: 0px; font-size: 12px; font-family: Monaco;" data-mce-style="margin: 0px; font-size: 12px; font-family: Monaco;"><br></p><p style="margin: 0px; font-size: 12px; font-family: Monaco;" data-mce-style="margin: 0px; font-size: 12px; font-family: Monaco;"><br></p><p style="margin: 0px; font-size: 12px; font-family: Monaco;" data-mce-style="margin: 0px; font-size: 12px; font-family: Monaco;">--</p></div><div>Andrei Mikhailovsky<br>Director<br>Arhont Information Security<br><div><br></div>Web: http://www.arhont.com<br>http://www.wi-foo.com<br>Tel: +44 (0)870 4431337<br>Fax: +44 (0)208 429 3111<br>PGP: Key ID - 0x2B3438DE<br>PGP: Server - keyserver.pgp.com<br><div><br></div>DISCLAIMER<br><div><br></div>The information contained in this email is intended only for the use of the person(s) to whom it is addressed and may be confidential or contain legally privileged information. If you are not the intended recipient you are hereby notified that any perusal, use, distribution, copying or disclosure is strictly prohibited. If you have received this email in error please immediately advise us by return email at andrei@arhont.com and delete and purge the email and any attachments without making a copy.<br><span name="x"></span><br></div><div><br></div><hr id="zwchr"><div style="color:#000;font-weight:normal;font-style:normal;text-decoration:none;font-family:Helvetica,Arial,sans-serif;font-size:12pt;"><b>From: </b>"Ira Weiny" <ira.weiny@intel.com><br><b>To: </b>"Rupert Dance" <rsdance@soft-forge.com>, "Andrei Mikhailovsky" <andrei@arhont.com>, "Jack Hill" <jackhill@unc.edu><br><b>Cc: </b>"users@lists.openfabrics.org" <users@lists.openfabrics.org><br><b>Sent: </b>Tuesday, 15 July, 2014 3:37:46 PM<br><b>Subject: </b>RE: [Users] ofed and ubuntu server<br><div><br></div>
<style>
<!--
@font-face
{font-family:Helvetica;
panose-1:2 11 6 4 2 2 2 2 2 4;}
@font-face
{font-family:Helvetica;
panose-1:2 11 6 4 2 2 2 2 2 4;}
@font-face
{font-family:Calibri;
panose-1:2 15 5 2 2 2 4 3 2 4;}
@font-face
{font-family:Tahoma;
panose-1:2 11 6 4 3 5 4 4 2 4;}
p.MsoNormal, li.MsoNormal, div.MsoNormal
{margin:0in;
margin-bottom:.0001pt;
font-size:12.0pt;
font-family:"Times New Roman","serif";}
a:link, span.MsoHyperlink
{mso-style-priority:99;
color:blue;
text-decoration:underline;}
a:visited, span.MsoHyperlinkFollowed
{mso-style-priority:99;
color:purple;
text-decoration:underline;}
p.MsoAcetate, li.MsoAcetate, div.MsoAcetate
{mso-style-priority:99;
mso-style-link:"Balloon Text Char";
margin:0in;
margin-bottom:.0001pt;
font-size:8.0pt;
font-family:"Tahoma","sans-serif";}
span.EmailStyle17
{mso-style-type:personal-reply;
font-family:"Times New Roman","serif";
color:windowtext;}
span.BalloonTextChar
{mso-style-name:"Balloon Text Char";
mso-style-priority:99;
mso-style-link:"Balloon Text";
font-family:"Tahoma","sans-serif";}
.MsoChpDefault
{mso-style-type:export-only;
font-size:10.0pt;}
@page WordSection1
{size:8.5in 11.0in;
margin:1.0in 1.0in 1.0in 1.0in;}
div.WordSection1
{page:WordSection1;}
-->
</style>
<title></title>
Infinipath-PSM is not needed to get the link to go active. If the link is not active what is it? (Ibstat output)<br>
<br>
Could you provide a bit more info on your configuration?<br>
<br>
Versions of the following<br>
Kernel on each node<br>
Opensm<br>
Infinibad-diags<br>
<br>
Node information<br>
Sm node config (mlx or Qib)<br>
Any switches between the sm node and the Qib node<br>
Qib node config<br>
"Sminfo" output<br>
What does the sm log say about that node or the switch port it is connected to?<br>
<br>
Finally can you talk to the failing node with smpquery from the sm node?<br>
<br>
Ira<br>
<br>
<br>
-----Original Message-----<br>
<b>From: </b>Rupert Dance [<a href="mailto:rsdance@soft-forge.com" target="_blank">rsdance@soft-forge.com</a>]<br>
<b>Sent: </b>Tuesday, July 15, 2014 06:33 AM Pacific Standard Time<br>
<b>To: </b>'Andrei Mikhailovsky'; 'Jack Hill'<br>
<b>Cc: </b>users@lists.openfabrics.org<br>
<b>Subject: </b>Re: [Users] ofed and ubuntu server<br>
<br>
<div class="WordSection1">
<p class="MsoNormal">Andrei,</p>
<p class="MsoNormal"> </p>
<p class="MsoNormal">There is a known issue with infinipath-psm and later generations of the gcc compiler. Perhaps it is related to your issue. Please check out this OFED Bug 2475:</p>
<p class="MsoNormal"> </p>
<p class="MsoNormal"><a href="http://bugs.openfabrics.org/bugzilla/show_bug.cgi?id=2475" target="_blank">http://bugs.openfabrics.org/bugzilla/show_bug.cgi?id=2475</a></p>
<p class="MsoNormal"> </p>
<p class="MsoNormal">Thanks</p>
<p class="MsoNormal"> </p>
<p class="MsoNormal">Rupert</p>
<p class="MsoNormal"> </p>
<p class="MsoNormal"> </p>
<p class="MsoNormal"> </p>
<div>
<div style="border:none;border-top:solid #B5C4DF 1.0pt;padding:3.0pt 0in 0in 0in">
<p class="MsoNormal"><b><span style="font-size:10.0pt;font-family:"Tahoma","sans-serif"">From:</span></b>
<span style="font-size:10.0pt;font-family:"Tahoma","sans-serif"">users-bounces@lists.openfabrics.org [mailto:users-bounces@lists.openfabrics.org]
<b>On Behalf Of</b> Andrei Mikhailovsky<br>
<b>Sent:</b> Tuesday, July 15, 2014 7:14 AM<br>
<b>To:</b> Jack Hill<br>
<b>Cc:</b> users@lists.openfabrics.org<br>
<b>Subject:</b> Re: [Users] ofed and ubuntu server</span></p>
</div>
</div>
<p class="MsoNormal"> </p>
<div>
<div>
<p class="MsoNormal"><span style="color:black">Jack,</span></p>
</div>
<div>
<p class="MsoNormal"><span style="color:black"> </span></p>
</div>
<div>
<p class="MsoNormal"><span style="color:black">Thanks for your reply. I was having issues getting the Active link status on those adapters. I've tried a bunch of things, but nothing helped me. I think the issues was with infinipath-psm as I couldnt get it compiled.
Did you have any problems with compiling infinipath-psm? In a contrast, I've had absolutely no issues with the Mellanox gear, which worked like a charm )))</span></p>
</div>
<div>
<p class="MsoNormal"><span style="color:black"> </span></p>
</div>
<div>
<p class="MsoNormal"><span style="color:black">Andrei</span></p>
</div>
<div>
<p class="MsoNormal"><span style="color:black"> </span></p>
</div>
<div class="MsoNormal" align="center" style="text-align:center">
<hr size="2" width="100%" align="center" id="zwchr">
</div>
<div>
<p class="MsoNormal"><b><span style="font-family:"Helvetica","sans-serif";color:black">From:</span></b>
<span style="font-family:"Helvetica","sans-serif";color:black">"Jack Hill" <<a href="mailto:jackhill@unc.edu" target="_blank">jackhill@unc.edu</a>><br>
<b>To:</b> "Andrei Mikhailovsky" <<a href="mailto:andrei@arhont.com" target="_blank">andrei@arhont.com</a>><br>
<b>Cc:</b> <a href="mailto:users@lists.openfabrics.org" target="_blank">users@lists.openfabrics.org</a><br>
<b>Sent:</b> Monday, 14 July, 2014 3:00:13 PM<br>
<b>Subject:</b> Re: [Users] ofed and ubuntu server</span></p>
<div>
<p class="MsoNormal"><span style="font-family:"Helvetica","sans-serif";color:black"> </span></p>
</div>
<p class="MsoNormal"><span style="font-family:"Helvetica","sans-serif";color:black">On Mon, 14 Jul 2014, Andrei Mikhailovsky wrote:</span></p>
<div>
<p class="MsoNormal"><span style="font-family:"Helvetica","sans-serif";color:black"> </span></p>
</div>
<p class="MsoNormal"><span style="font-family:"Helvetica","sans-serif";color:black">> I was wondering if there is any progress in supporting Ubuntu servers 12.04 or 14.04? I am having<br>
> a few issues getting my Qlogic 7322 QDR to work on Ubuntu Servers. It works fine on red hat based<br>
> distros, but not on Ubuntu ((</span></p>
<div>
<p class="MsoNormal"><span style="font-family:"Helvetica","sans-serif";color:black"> </span></p>
</div>
<p class="MsoNormal"><span style="font-family:"Helvetica","sans-serif";color:black">Andrei,</span></p>
<div>
<p class="MsoNormal"><span style="font-family:"Helvetica","sans-serif";color:black"> </span></p>
</div>
<p class="MsoNormal"><span style="font-family:"Helvetica","sans-serif";color:black">What issues are you having? I've been using those adapters with Ubuntu<br>
12.04 with distribution supplied packages. I did install infinipath-psm<br>
from source to support some MPI operations.</span></p>
<div>
<p class="MsoNormal"><span style="font-family:"Helvetica","sans-serif";color:black"> </span></p>
</div>
<p class="MsoNormal"><span style="font-family:"Helvetica","sans-serif";color:black">Best,<br>
Jack</span></p>
</div>
<div>
<p class="MsoNormal"><span style="color:black"> </span></p>
</div>
</div>
</div>
</div><div><br></div></div></body></html>