<div dir="ltr"><div>Speed 4 is not valid as an enabled speed. It probably is speed 7 (SDR, DDR, or QDR) . Since qib is QDR (only), espeed shouldn't be used. QDR only is non standard which is why it says "IBA extension".</div><div> </div><div>As to why the link comes up and then goes down, there are a number of possible reasons. What does perfquery 29 28 say ? Are there some error counters increasing there ? If so, which ones ?</div></div><div class="gmail_extra"><br><div class="gmail_quote">On Tue, Oct 13, 2015 at 1:09 PM, German Anders <span dir="ltr"><<a href="mailto:ganders@despegar.com" target="_blank">ganders@despegar.com</a>></span> wrote:<br><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex"><div dir="ltr"><div class="gmail_default" style="color:rgb(11,83,148);font-family:monospace,monospace;font-size:small">Ok, some good news! I've find the way to get LinkUp on the blade port... but.. not for so long :S, it seems that after making the following changes the ports at the end come up...but then in a couple of minutes goes down again:<br><br>- Move the Encl IB SW to Bay7<br>- Move the Blade Mezz cards to slot 3<br><br>Then for ex:<br><br></div><div class="gmail_default" style="color:rgb(11,83,148);font-family:monospace,monospace;font-size:small">Blade IB SW - LID: 29<br>Blade 4 IB-SW-PORT 28 <br></div><div class="gmail_default" style="color:rgb(11,83,148);font-family:monospace,monospace;font-size:small"><br>1)<br><br># ibportstate -L 29 28<br>Switch PortInfo:<br># Port info: Lid 29 port 28<br>LinkState:.......................Down<br>PhysLinkState:...................Polling<br>Lid:.............................75<br>SMLid:...........................2328<span><br>LMC:.............................0<br>LinkWidthSupported:..............1X or 4X<br>LinkWidthEnabled:................1X or 4X<br>LinkWidthActive:.................4X<br>LinkSpeedSupported:..............2.5 Gbps or 5.0 Gbps or 10.0 Gbps<br>LinkSpeedEnabled:................2.5 Gbps or 5.0 Gbps or 10.0 Gbps<br></span>LinkSpeedActive:.................2.5 Gbps<br><br><br>2)<br><br># ibportstate -L 29 28 disable<br># ibportstate -L 29 28 speed 4<br># ibportstate -L 29 28 espeed 4<br># ibportstate -L 29 28 smlid 2<br># ibportstate -L 29 28 enable<br><br>3) <br><br># ibportstate -L 29 28<br>Switch PortInfo:<br># Port info: Lid 29 port 28<br>LinkState:.......................Active<br>PhysLinkState:...................LinkUp<br>Lid:.............................75<br>SMLid:...........................2328<span><br>LMC:.............................0<br>LinkWidthSupported:..............1X or 4X<br>LinkWidthEnabled:................1X or 4X<br>LinkWidthActive:.................4X<br>LinkSpeedSupported:..............2.5 Gbps or 5.0 Gbps or 10.0 Gbps<br>LinkSpeedEnabled:................2.5 Gbps or 5.0 Gbps or 10.0 Gbps<br>LinkSpeedActive:.................10.0 Gbps<br></span>Peer PortInfo:<br># Port info: Lid 29 DR path slid 4; dlid 65535; 0,28 port 1<br>LinkState:.......................Active<br>PhysLinkState:...................LinkUp<br>Lid:.............................30<span><br>SMLid:...........................2<br>LMC:.............................0<br>LinkWidthSupported:..............1X or 4X<br>LinkWidthEnabled:................1X or 4X<br>LinkWidthActive:.................4X<br></span>LinkSpeedSupported:..............10.0 Gbps (IBA extension)<br>LinkSpeedEnabled:................10.0 Gbps (IBA extension)<span><br>LinkSpeedActive:.................10.0 Gbps<br>Mkey:............................<not displayed><br>MkeyLeasePeriod:.................0<br>ProtectBits:.....................0<br><br><br></span><span># ibstat<br>CA 'qib0'<br> CA type: InfiniPath_QMH7342<br> Number of ports: 2<br> Firmware version: <br> Hardware version: 2<br> Node GUID: 0x0011750000791fec<br> System image GUID: 0x0011750000791fec<br> Port 1:<br></span> State: <b>Active</b><br> Physical state: <b>LinkUp</b><br> Rate: 40<br> Base lid: 30<br> LMC: 0<br> SM lid: 2<span><br> Capability mask: 0x0761086a<br> Port GUID: 0x0011750000791fec<br> Link layer: InfiniBand<br> Port 2:<br> State: Down<br> Physical state: Polling<br> Rate: 40<br></span> Base lid: 65535<br> LMC: 0<br> SM lid: 65535<span><br> Capability mask: 0x0761086a<br> Port GUID: 0x0011750000791fed<br> Link layer: InfiniBand<br><br><br></span></div><div class="gmail_default" style="color:rgb(11,83,148);font-family:monospace,monospace;font-size:small">a couple of minutes then...<br><br># ibportstate -L 29 28<br>Switch PortInfo:<br># Port info: Lid 29 port 28<br>LinkState:.......................Down<br>PhysLinkState:...................Polling<br>Lid:.............................75<br>SMLid:...........................2328<span><br>LMC:.............................0<br>LinkWidthSupported:..............1X or 4X<br>LinkWidthEnabled:................1X or 4X<br>LinkWidthActive:.................4X<br>LinkSpeedSupported:..............2.5 Gbps or 5.0 Gbps or 10.0 Gbps<br>LinkSpeedEnabled:................2.5 Gbps or 5.0 Gbps or 10.0 Gbps<br></span>LinkSpeedActive:.................2.5 Gbps<span class="HOEnZb"><font color="#888888"><br></font></span></div><span class="HOEnZb"><font color="#888888"><div class="gmail_default" style="color:rgb(11,83,148);font-family:monospace,monospace;font-size:small"><br></div></font></span><div class="gmail_extra"><span class="HOEnZb"><font color="#888888"><br clear="all"><div><div><div dir="ltr"><div><div dir="ltr"><span><div><div dir="ltr"><p><b><span style="color:rgb(68,68,68);font-family:Trebuchet MS,sans-serif;font-size:10pt"></span></b></p><span style="color:rgb(68,68,68)"><span style="font-family:trebuchet ms,sans-serif"><font size="2"><b>German</b></font></span></span><span style="color:rgb(68,68,68)"><span style="font-family:trebuchet ms,sans-serif"><font size="2"><font size="1"><a href="mailto:ganders@despegar.com" target="_blank"></a></font></font></span></span></div></div></span></div></div></div></div></div></font></span><div><div class="h5">
<br><div class="gmail_quote">2015-10-13 12:02 GMT-03:00 German Anders <span dir="ltr"><<a href="mailto:ganders@despegar.com" target="_blank">ganders@despegar.com</a>></span>:<br><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;padding-left:1ex;border-left-color:rgb(204,204,204);border-left-width:1px;border-left-style:solid"><div dir="ltr"><div style="color:rgb(11,83,148);font-family:monospace,monospace;font-size:small">Can anyone that's using the HP BLc 4X QDR IB Switch tell me in what interconnect bay is configured the switch? I've that in bay3 & mezz 1, I'll try to move to bay7 & mezz 3 and see if at least something changed or not, any useful info will be really appreciated.<br><br></div><div style="color:rgb(11,83,148);font-family:monospace,monospace;font-size:small">Cheers,<br><br></div><div class="gmail_extra"><br clear="all"><div><div><div dir="ltr"><div><div dir="ltr"><span><div><div dir="ltr"><p><b><span style="color:rgb(68,68,68);font-family:Trebuchet MS,sans-serif;font-size:10pt"></span></b></p><span style="color:rgb(68,68,68)"><span style="font-family:trebuchet ms,sans-serif"><font size="2"><b>German</b></font></span></span><span style="color:rgb(68,68,68)"><span style="font-family:trebuchet ms,sans-serif"><font size="2"><font size="1"><a href="mailto:ganders@despegar.com" target="_blank"></a></font></font></span></span></div></div></span></div></div></div></div></div><div><div>
<br><div class="gmail_quote">2015-10-13 10:51 GMT-03:00 German Anders <span dir="ltr"><<a href="mailto:ganders@despegar.com" target="_blank">ganders@despegar.com</a>></span>:<br><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;padding-left:1ex;border-left-color:rgb(204,204,204);border-left-width:1px;border-left-style:solid"><div dir="ltr"><div style="color:rgb(11,83,148);font-family:monospace,monospace;font-size:small">yes, but the switch does not come with an external interface... just the i2c port :S<span><font color="#888888"><br><br><br></font></span></div><div class="gmail_extra"><span><font color="#888888"><br clear="all"><div><div><div dir="ltr"><div><div dir="ltr"><span><div><div dir="ltr"><p><b><span style="color:rgb(68,68,68);font-family:Trebuchet MS,sans-serif;font-size:10pt"></span></b></p><span style="color:rgb(68,68,68)"><span style="font-family:trebuchet ms,sans-serif"><font size="2"><b>German</b></font></span></span><span style="color:rgb(68,68,68)"><span style="font-family:trebuchet ms,sans-serif"><font size="2"><font size="1"><a href="mailto:ganders@despegar.com" target="_blank"></a></font></font></span></span></div></div></span></div></div></div></div></div></font></span><div><div>
<br><div class="gmail_quote">2015-10-13 10:49 GMT-03:00 Hal Rosenstock <span dir="ltr"><<a href="mailto:hal.rosenstock@gmail.com" target="_blank">hal.rosenstock@gmail.com</a>></span>:<br><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;padding-left:1ex;border-left-color:rgb(204,204,204);border-left-width:1px;border-left-style:solid"><div dir="ltr"><div>I don't really know but I was wondering about whether your hardware supports this.</div><div> </div><div>HPOA_2 shows internal interface to OA is absent as is external Ethernet interface.</div></div><div><div><div class="gmail_extra"><br><div class="gmail_quote">On Tue, Oct 13, 2015 at 9:31 AM, German Anders <span dir="ltr"><<a href="mailto:ganders@despegar.com" target="_blank">ganders@despegar.com</a>></span> wrote:<br><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;padding-left:1ex;border-left-color:rgb(204,204,204);border-left-width:1px;border-left-style:solid"><div dir="ltr"><div style="color:rgb(11,83,148);font-family:monospace,monospace;font-size:small">when trying to assign the ip addr to the interconnect bay it seems that it does not 'apply' the change, and the ip addr is not displayed as 'configured' so there's no way to enter from outside. ideas?<span><font color="#888888"><br></font></span></div><div class="gmail_extra"><span><font color="#888888"><br clear="all"><div><div><div dir="ltr"><div><div dir="ltr"><span><div><div dir="ltr"><p><b><span style="color:rgb(68,68,68);font-family:Trebuchet MS,sans-serif;font-size:10pt"></span></b></p><span style="color:rgb(68,68,68)"><span style="font-family:trebuchet ms,sans-serif"><font size="2"><b>German</b></font></span></span><span style="color:rgb(68,68,68)"><span style="font-family:trebuchet ms,sans-serif"></span></span></div></div></span></div></div></div></div></div></font></span><div><div>
<br><div class="gmail_quote">2015-10-13 10:15 GMT-03:00 Hal Rosenstock <span dir="ltr"><<a href="mailto:hal.rosenstock@gmail.com" target="_blank">hal.rosenstock@gmail.com</a>></span>:<br><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;padding-left:1ex;border-left-color:rgb(204,204,204);border-left-width:1px;border-left-style:solid"><div dir="ltr">What about the 3 critical errors ? What are they ?</div><div><div><div class="gmail_extra"><br><div class="gmail_quote">On Tue, Oct 13, 2015 at 9:13 AM, German Anders <span dir="ltr"><<a href="mailto:ganders@despegar.com" target="_blank">ganders@despegar.com</a>></span> wrote:<br><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;padding-left:1ex;border-left-color:rgb(204,204,204);border-left-width:1px;border-left-style:solid"><div dir="ltr"><div style="color:rgb(11,83,148);font-family:monospace,monospace;font-size:small">I've try that and in fact I try to put the IP ADDR like this:<br><br><pre>SET EBIPA INTERCONNECT XX.XX.XX.XX XXX.XXX.XXX.XXX 3
SET EBIPA INTERCONNECT GATEWAY XX.XX.XX.XX 3
SET EBIPA INTERCONNECT DOMAIN "<a href="http://xxxxxx.net" target="_blank">xxxxxx.net</a>" 3
ADD EBIPA INTERCONNECT DNS 10.xx.xx.xx 3
ADD EBIPA INTERCONNECT DNS 10.xx.xx.xx 3
SET EBIPA INTERCONNECT NTP PRIMARY NONE 3
SET EBIPA INTERCONNECT NTP SECONDARY NONE 3
ENABLE EBIPA INTERCONNECT 3<br></pre><pre>SAVE EBIPA<br><br></pre><pre>But i'm not getting any ip response, also I've try many diff ip addr with no luck...if i put that ip to one of the blades it works fine, but not to the interconnect bay :( any other idea?<br><br></pre><pre>Cheers,</pre></div><div class="gmail_extra"><br clear="all"><div><div><div dir="ltr"><div><div dir="ltr"><span><div><div dir="ltr"><p><b><span style="color:rgb(68,68,68);font-family:Trebuchet MS,sans-serif;font-size:10pt"></span></b></p><span style="color:rgb(68,68,68)"><span style="font-family:trebuchet ms,sans-serif"><font size="2"><b>German</b></font></span></span><span style="color:rgb(68,68,68)"><span style="font-family:trebuchet ms,sans-serif"><font size="2"><font size="1"><a href="mailto:ganders@despegar.com" target="_blank"></a></font></font></span></span></div></div></span></div></div></div></div></div><div><div>
<br><div class="gmail_quote">2015-10-13 10:01 GMT-03:00 Hal Rosenstock <span dir="ltr"><<a href="mailto:hal.rosenstock@gmail.com" target="_blank">hal.rosenstock@gmail.com</a>></span>:<br><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;padding-left:1ex;border-left-color:rgb(204,204,204);border-left-width:1px;border-left-style:solid"><div dir="ltr"><div>Looks like there are 3 critical errors in system status. Did you look at these ?</div><div> </div><div>I don't know if you've seen this but there is some info on configuring the management IPs in <a href="http://h10032.www1.hp.com/ctg/Manual/c00814176.pdf" target="_blank">http://h10032.www1.hp.com/ctg/Manual/c00814176.pdf</a></div><div> </div><div>Have you looked at/tried the command line interface ?</div></div><div><div><div class="gmail_extra"><br><div class="gmail_quote">On Tue, Oct 13, 2015 at 8:28 AM, German Anders <span dir="ltr"><<a href="mailto:ganders@despegar.com" target="_blank">ganders@despegar.com</a>></span> wrote:<br><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;padding-left:1ex;border-left-color:rgb(204,204,204);border-left-width:1px;border-left-style:solid"><div dir="ltr"><div style="color:rgb(11,83,148);font-family:monospace,monospace;font-size:small">Hi Hal,<br><br></div><div style="color:rgb(11,83,148);font-family:monospace,monospace;font-size:small">It does not allow me to setup an IP ADDR to the Internal SW so I can't access from outside, except from the tools that I mentioned before, also it doesn't allow me to access through serial connection from inside the enclosure. I've attach some screen-shots about the connectivity.<span><font color="#888888"><br><br><br></font></span></div><div class="gmail_extra"><span><font color="#888888"><br clear="all"><div><div><div dir="ltr"><div><div dir="ltr"><span><div><div dir="ltr"><p><b><span style="color:rgb(68,68,68);font-family:Trebuchet MS,sans-serif;font-size:10pt"></span></b></p><span style="color:rgb(68,68,68)"><span style="font-family:trebuchet ms,sans-serif"><font size="2"><b>German</b></font></span></span><span style="color:rgb(68,68,68)"><span style="font-family:trebuchet ms,sans-serif"></span></span></div></div></span></div></div></div></div></div></font></span><div><div>
<br><div class="gmail_quote">2015-10-13 9:13 GMT-03:00 Hal Rosenstock <span dir="ltr"><<a href="mailto:hal.rosenstock@gmail.com" target="_blank">hal.rosenstock@gmail.com</a>></span>:<br><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;padding-left:1ex;border-left-color:rgb(204,204,204);border-left-width:1px;border-left-style:solid"><div dir="ltr"><div>Hi German,</div><div> </div><div>Are the cards in the correct bays and slots ?</div><div> </div><div>Do you have the <font face="Futura-Book" size="2"><font face="Futura-Book" size="2">HP Onboard Administrator tool ? What does it say about internal connectivity ?</font><span><font color="#888888"><p>-- Hal</p></font></span></font><p> </p></div></div><div><div><div class="gmail_extra"><br><div class="gmail_quote">On Tue, Oct 13, 2015 at 7:44 AM, German Anders <span dir="ltr"><<a href="mailto:ganders@despegar.com" target="_blank">ganders@despegar.com</a>></span> wrote:<br><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;padding-left:1ex;border-left-color:rgb(204,204,204);border-left-width:1px;border-left-style:solid"><div dir="ltr"><div style="color:rgb(11,83,148);font-family:monospace,monospace;font-size:small">Hi Ira,<br><br></div><div style="color:rgb(11,83,148);font-family:monospace,monospace;font-size:small">I've some HP documentation but it quite short, also it doesn't describe any 'config' or 'impl' steps in order to get the internal switch up and running. The version of SW that came with enclosure does not had any management module at all, so it depends on external management. During weekend I've found a way to upgrade the firmware of the HP switch with the following command (mlxburn -d lid-0x001D -fw fw-IS4.mlx), and the I've run (flint -d /dev/mst/SW_MT48438_0x2c902004b0918_lid-0x001D dc /home/ceph/HPIBSW.INI) and found the following inside that file:<br><br>[PS_INFO]<br>Name = 489184-B21<br>Description = HP BLc 4X QDR IB Switch<br><br>[ADAPTER]<br>PSID = HP_0100000009<br><br>(...)<br><br>[IB_TO_HW_MAP]<br>PORT1=14<br>PORT2=15<br>PORT3=16<br>PORT4=17<br>PORT5=18<br>PORT6=12<br>PORT7=11<br>PORT8=10<br>PORT9=9<br>PORT10=8<br>PORT11=7<br>PORT12=6<br>PORT13=5<br>PORT14=4<br>PORT15=3<br>PORT16=2<br>PORT17=20<br>PORT18=22<br>PORT19=24<br><br>PORT20=26<br>PORT21=28<br>PORT22=30<br>PORT23=35<br>PORT24=33<br>PORT25=21<br>PORT26=23<br>PORT27=25<br>PORT28=27<br>PORT29=29<br>PORT30=36<br>PORT31=34<br>PORT32=32<br>PORT33=1<br>PORT34=13<br>PORT35=19<br>PORT36=31<br><br>[unused_ports]<br>hw_port1_not_in_use=1<br>hw_port13_not_in_use=1<br>hw_port19_not_in_use=1<br>hw_port31_not_in_use=1<br><br>(...)<br><br></div><div style="color:rgb(11,83,148);font-family:monospace,monospace;font-size:small">I don't know if maybe there's some issue with the port mapping, anyone had used this kind of switch? <br><br></div><div style="color:rgb(11,83,148);font-family:monospace,monospace;font-size:small">The summary of the problem is correct, the connectivity between the IB network (MLNX switches/gw) and the HP IB switch is working since I was able to upgrade the firmare of the switch and get information about it. But, the connection between the mezzanine cards of the blades and the internal IB sw enclosure is not working at all. Note, that if I go to the OA administration of the enclosure I can see the 'green' ports mapping of each of the blades and the interconnection switch, so I'm guessing that it should be working.<br><br></div><div style="color:rgb(11,83,148);font-family:monospace,monospace;font-size:small">Regarding the questions:<span><br><br><span style="color:rgb(31,73,125);font-family:"Calibri","sans-serif";font-size:11pt"><span>1)<span style="font:7pt/normal "Times New Roman";font-size-adjust:none;font-stretch:normal">
</span></span></span><span style="color:rgb(31,73,125);font-family:"Calibri","sans-serif";font-size:11pt">What type of switch is in the HP chassis?<br><span style="color:rgb(255,153,0)"><br></span></span></span></div><span><div style="font-family:monospace,monospace;font-size:small"><span style="color:rgb(255,153,0)"><b><span style="font-family:"Calibri","sans-serif";font-size:11pt">QLogic HP BLc 4X QDR IB Switch<br></span></b></span></div></span><div style="font-family:monospace,monospace;font-size:small"><span style="color:rgb(255,153,0)"><b><span style="font-family:"Calibri","sans-serif";font-size:11pt">PSID = HP_0100000009<br></span></b></span></div><div style="font-family:monospace,monospace;font-size:small"><span style="color:rgb(255,153,0)"><b><span style="font-family:"Calibri","sans-serif";font-size:11pt">Image type: FS2<br></span></b></span></div><div style="font-family:monospace,monospace;font-size:small"><span style="color:rgb(255,153,0)"><b><span style="font-family:"Calibri","sans-serif";font-size:11pt">FW ver: 7.4.3000<br></span></b></span></div><div style="font-family:monospace,monospace;font-size:small"><span style="color:rgb(255,153,0)"><b><span style="font-family:"Calibri","sans-serif";font-size:11pt">Device ID: 48438<br></span></b></span></div><div style="color:rgb(11,83,148);font-family:monospace,monospace;font-size:small"><span style="color:rgb(31,73,125);font-family:"Calibri","sans-serif";font-size:11pt"><span style="color:rgb(255,153,0)"><b>GUI: 0002c902004b0918</b></span><br></span></div><div style="color:rgb(11,83,148);font-family:monospace,monospace;font-size:small"><span>
<p><span style="color:rgb(31,73,125);font-family:"Calibri","sans-serif";font-size:11pt"><span>2)<span style="font:7pt/normal "Times New Roman";font-size-adjust:none;font-stretch:normal">
</span></span></span><span style="color:rgb(31,73,125);font-family:"Calibri","sans-serif";font-size:11pt">Do you have console access or http access to that switch?</span></p></span><p><span style="color:rgb(31,73,125);font-family:"Calibri","sans-serif";font-size:11pt"><span style="color:rgb(255,153,0)"><b>No, since it didn't had any manage module mezzanine card inside the switch, it only come with a i2c port. But, i can have access through the mlxburn and flint tools from one host that's connected to the ib network (outside the enclosure).</b></span><br></span></p><span><p><span style="color:rgb(31,73,125);font-family:"Calibri","sans-serif";font-size:11pt"><span>3)<span style="font:7pt/normal "Times New Roman";font-size-adjust:none;font-stretch:normal">
</span></span></span><span style="color:rgb(31,73,125);font-family:"Calibri","sans-serif";font-size:11pt">Does that switch have an SM in it?</span></p></span><p><span style="color:rgb(31,73,125);font-family:"Calibri","sans-serif";font-size:11pt"><b><span style="color:rgb(255,153,0)">No</span></b><br></span></p><span>
<p><span style="color:rgb(31,73,125);font-family:"Calibri","sans-serif";font-size:11pt"><span>4)<span style="font:7pt/normal "Times New Roman";font-size-adjust:none;font-stretch:normal">
</span></span></span><span style="color:rgb(31,73,125);font-family:"Calibri","sans-serif";font-size:11pt">What version of the kernel are you running with the qib cards?</span></p>
<p style="margin-left:1in">
<span style="color:rgb(31,73,125);font-family:"Calibri","sans-serif";font-size:11pt"><span>a.<span style="font:7pt/normal "Times New Roman";font-size-adjust:none;font-stretch:normal">
</span></span></span><span style="color:rgb(31,73,125);font-family:"Calibri","sans-serif";font-size:11pt">I assume you are using the qib driver in that kernel.</span></p></span><p style="margin-left:1in"><span style="color:rgb(31,73,125);font-family:"Calibri","sans-serif";font-size:11pt"><span style="color:rgb(255,153,0)"><b>Ubuntu 14.04.3 LTS - kernel 3.18.20-031820-generic</b></span><br></span></p><span>
<p class="MsoNormal"><span style="color:rgb(31,73,125);font-family:"Calibri","sans-serif";font-size:11pt"> </span></p>
<p class="MsoNormal"><span style="color:rgb(31,73,125);font-family:"Calibri","sans-serif";font-size:11pt">At
some point Hal spoke of “LLR being a Mellanox thing” Was that to solve
the problem of connecting the “HP switch” to the Mellanox switch?</span></p><p class="MsoNormal"><span style="color:rgb(31,73,125);font-family:"Calibri","sans-serif";font-size:11pt"><br></span></p></span><p class="MsoNormal"><span style="color:rgb(255,153,0)"><b><span style="font-family:"Calibri","sans-serif";font-size:11pt">No, since LLR is only supported between mlnx devices, the ISL are up and working, since it's possible for me to query the switch<br></span></b></span></p><span><span style="color:rgb(255,153,0)"><b>
</b></span><p class="MsoNormal"><span style="color:rgb(31,73,125);font-family:"Calibri","sans-serif";font-size:11pt"> </span></p>
<p class="MsoNormal"><span style="color:rgb(31,73,125);font-family:"Calibri","sans-serif";font-size:11pt">I would like it if you could verify that the
</span></p>
<p class="MsoNormal"><span style="color:rgb(31,73,125);font-family:"Calibri","sans-serif";font-size:11pt"> </span></p>
<p class="MsoNormal"><span style="color:rgb(11,83,148);font-family:"Courier New"">/usr/sbin/truescale-serdes.cmds</span></p>
<p class="MsoNormal"><span style="color:rgb(11,83,148);font-family:"Courier New""> </span></p>
<p class="MsoNormal"><span style="color:rgb(11,83,148);font-family:"Courier New"">Is being run?</span></p><p class="MsoNormal"><br><span style="color:rgb(11,83,148);font-family:"Courier New""></span></p></span><p class="MsoNormal"><span style="color:rgb(255,153,0)"><b><span style="font-family:"Courier New"">When trying to run the command:</span></b></span></p><p class="MsoNormal"><span style="color:rgb(255,153,0)"><b><br><span style="font-family:"Courier New""></span></b></span></p><p class="MsoNormal"><span style="color:rgb(11,83,148);font-family:"Courier New""><span style="color:rgb(255,153,0)"><b># /usr/sbin/truescale-serdes.cmds<br>/usr/sbin/truescale-serdes.cmds: 100: /usr/sbin/truescale-serdes.cmds: Syntax error: "(" unexpected (expecting "}")</b></span><br></span></p><span><p class="MsoNormal"><span style="color:rgb(11,83,148);font-family:"Courier New""><br></span></p>
<p class="MsoNormal"><span style="color:rgb(11,83,148);font-family:"Courier New""> </span></p>
<span style="color:rgb(11,83,148);font-family:"Courier New"">Also what version of libipathverbs do you have?</span><br></span></div><div style="color:rgb(11,83,148);font-family:monospace,monospace;font-size:small"><br><span style="color:rgb(255,153,0)"><b># rpm -qa | grep libipathverbs<br>libipathverbs-1.3-1.x86_64<br><br></b></span><br></div><div style="color:rgb(11,83,148);font-family:monospace,monospace;font-size:small">Thanks in advance,<br><br></div><div style="color:rgb(11,83,148);font-family:monospace,monospace;font-size:small">Cheers,</div><div class="gmail_extra"><br clear="all"><div><div><div dir="ltr"><div><div dir="ltr"><span><div><div dir="ltr"><p><b><span style="color:rgb(68,68,68);font-family:Trebuchet MS,sans-serif;font-size:10pt"></span></b></p><span style="color:rgb(68,68,68)"><span style="font-family:trebuchet ms,sans-serif"><font size="2"><b>German<br><br></b></font></span></span></div></div></span></div></div></div></div></div><div><div><div class="gmail_quote">2015-10-13 2:14 GMT-03:00 Weiny, Ira <span dir="ltr"><<a href="mailto:ira.weiny@intel.com" target="_blank">ira.weiny@intel.com</a>></span>:<br><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;padding-left:1ex;border-left-color:rgb(204,204,204);border-left-width:1px;border-left-style:solid">
<div lang="EN-US" vlink="purple" link="blue">
<div>
<p class="MsoNormal"><span style="color:rgb(31,73,125);font-family:"Calibri","sans-serif";font-size:11pt">German,<u></u><u></u></span></p>
<p class="MsoNormal"><span style="color:rgb(31,73,125);font-family:"Calibri","sans-serif";font-size:11pt"><u></u> <u></u></span></p>
<p class="MsoNormal"><span style="color:rgb(31,73,125);font-family:"Calibri","sans-serif";font-size:11pt">Do you have any documentation on the HP blade system? And the switch which is in that system?<u></u><u></u></span></p>
<p class="MsoNormal"><span style="color:rgb(31,73,125);font-family:"Calibri","sans-serif";font-size:11pt"><u></u> <u></u></span></p>
<p class="MsoNormal"><span style="color:rgb(31,73,125);font-family:"Calibri","sans-serif";font-size:11pt">I have to admit I have not followed everything in this thread regarding your configuration but it seems like you have some mellanox switches connected into
an HP chassis which has both a switch and blades with qib (Truescale) cards.<u></u><u></u></span></p>
<p class="MsoNormal"><span style="color:rgb(31,73,125);font-family:"Calibri","sans-serif";font-size:11pt"><u></u> <u></u></span></p>
<p class="MsoNormal"><span style="color:rgb(31,73,125);font-family:"Calibri","sans-serif";font-size:11pt">The connection from the mellanox switch to the “HP chassis switch” is linkup (active) but the connections to the individual qib HCAs are not even linkup.<u></u><u></u></span></p>
<p class="MsoNormal"><span style="color:rgb(31,73,125);font-family:"Calibri","sans-serif";font-size:11pt"><u></u> <u></u></span></p>
<p class="MsoNormal"><span style="color:rgb(31,73,125);font-family:"Calibri","sans-serif";font-size:11pt">Is that a correct summary of the problem?<u></u><u></u></span></p>
<p class="MsoNormal"><span style="color:rgb(31,73,125);font-family:"Calibri","sans-serif";font-size:11pt"><u></u> <u></u></span></p>
<p class="MsoNormal"><span style="color:rgb(31,73,125);font-family:"Calibri","sans-serif";font-size:11pt">If so here are some questions:<u></u><u></u></span></p>
<p class="MsoNormal"><span style="color:rgb(31,73,125);font-family:"Calibri","sans-serif";font-size:11pt"><u></u> <u></u></span></p>
<p><u></u><span style="color:rgb(31,73,125);font-family:"Calibri","sans-serif";font-size:11pt"><span>1)<span style="font:7pt/normal "Times New Roman";font-size-adjust:none;font-stretch:normal">
</span></span></span><u></u><span style="color:rgb(31,73,125);font-family:"Calibri","sans-serif";font-size:11pt">What type of switch is in the HP chassis?<u></u><u></u></span></p>
<p><u></u><span style="color:rgb(31,73,125);font-family:"Calibri","sans-serif";font-size:11pt"><span>2)<span style="font:7pt/normal "Times New Roman";font-size-adjust:none;font-stretch:normal">
</span></span></span><u></u><span style="color:rgb(31,73,125);font-family:"Calibri","sans-serif";font-size:11pt">Do you have console access or http access to that switch?<u></u><u></u></span></p>
<p><u></u><span style="color:rgb(31,73,125);font-family:"Calibri","sans-serif";font-size:11pt"><span>3)<span style="font:7pt/normal "Times New Roman";font-size-adjust:none;font-stretch:normal">
</span></span></span><u></u><span style="color:rgb(31,73,125);font-family:"Calibri","sans-serif";font-size:11pt">Does that switch have an SM in it?<u></u><u></u></span></p>
<p><u></u><span style="color:rgb(31,73,125);font-family:"Calibri","sans-serif";font-size:11pt"><span>4)<span style="font:7pt/normal "Times New Roman";font-size-adjust:none;font-stretch:normal">
</span></span></span><u></u><span style="color:rgb(31,73,125);font-family:"Calibri","sans-serif";font-size:11pt">What version of the kernel are you running with the qib cards?<u></u><u></u></span></p>
<p style="margin-left:1in">
<u></u><span style="color:rgb(31,73,125);font-family:"Calibri","sans-serif";font-size:11pt"><span>a.<span style="font:7pt/normal "Times New Roman";font-size-adjust:none;font-stretch:normal">
</span></span></span><u></u><span style="color:rgb(31,73,125);font-family:"Calibri","sans-serif";font-size:11pt">I assume you are using the qib driver in that kernel.<u></u><u></u></span></p>
<p class="MsoNormal"><span style="color:rgb(31,73,125);font-family:"Calibri","sans-serif";font-size:11pt"><u></u> <u></u></span></p>
<p class="MsoNormal"><span style="color:rgb(31,73,125);font-family:"Calibri","sans-serif";font-size:11pt">At some point Hal spoke of “LLR being a Mellanox thing” Was that to solve the problem of connecting the “HP switch” to the Mellanox switch?<u></u><u></u></span></p>
<p class="MsoNormal"><span style="color:rgb(31,73,125);font-family:"Calibri","sans-serif";font-size:11pt"><u></u> <u></u></span></p>
<p class="MsoNormal"><span style="color:rgb(31,73,125);font-family:"Calibri","sans-serif";font-size:11pt">I would like it if you could verify that the
<u></u><u></u></span></p>
<p class="MsoNormal"><span style="color:rgb(31,73,125);font-family:"Calibri","sans-serif";font-size:11pt"><u></u> <u></u></span></p>
<p class="MsoNormal"><span style="color:rgb(11,83,148);font-family:"Courier New"">/usr/sbin/truescale-serdes.cmds<u></u><u></u></span></p>
<p class="MsoNormal"><span style="color:rgb(11,83,148);font-family:"Courier New""><u></u> <u></u></span></p>
<p class="MsoNormal"><span style="color:rgb(11,83,148);font-family:"Courier New"">Is being run?<u></u><u></u></span></p>
<p class="MsoNormal"><span style="color:rgb(11,83,148);font-family:"Courier New""><u></u> <u></u></span></p>
<p class="MsoNormal"><span style="color:rgb(11,83,148);font-family:"Courier New"">Also what version of libipathverbs do you have?<u></u><u></u></span></p>
<p class="MsoNormal"><span style="color:rgb(11,83,148);font-family:"Courier New""><u></u> <u></u></span></p>
<p class="MsoNormal"><span style="color:rgb(11,83,148);font-family:"Courier New"">Ira</span><span style="color:rgb(31,73,125);font-family:"Calibri","sans-serif";font-size:11pt"><u></u><u></u></span></p>
<p class="MsoNormal"><span style="color:rgb(31,73,125);font-family:"Calibri","sans-serif";font-size:11pt"><u></u> <u></u></span></p>
<div style="border-width:medium medium medium 1.5pt;border-style:none none none solid;border-color:currentColor currentColor currentColor blue;padding:0in 0in 0in 4pt">
<div>
<div style="border-width:1pt medium medium;border-style:solid none none;border-color:rgb(181,196,223) currentColor currentColor;padding:3pt 0in 0in">
<p class="MsoNormal"><b><span style="font-family:"Tahoma","sans-serif";font-size:10pt">From:</span></b><span style="font-family:"Tahoma","sans-serif";font-size:10pt"> <a href="mailto:users-bounces@lists.openfabrics.org" target="_blank">users-bounces@lists.openfabrics.org</a> [mailto:<a href="mailto:users-bounces@lists.openfabrics.org" target="_blank">users-bounces@lists.openfabrics.org</a>]
<b>On Behalf Of </b>Weiny, Ira<br>
<b>Sent:</b> Wednesday, October 07, 2015 1:31 PM<br>
<b>To:</b> Hal Rosenstock; German Anders</span></p><div><div><br>
<b>Cc:</b> <a href="mailto:users@lists.openfabrics.org" target="_blank">users@lists.openfabrics.org</a><br>
<b>Subject:</b> Re: [Users] IB topology config and polling state<u></u><u></u></div></div><p></p>
</div>
</div><div><div>
<p class="MsoNormal"><u></u> <u></u></p>
<p class="MsoNormal"><span style="color:rgb(31,73,125);font-family:"Calibri","sans-serif";font-size:11pt">Agree with Hal here.<u></u><u></u></span></p>
<p class="MsoNormal"><span style="color:rgb(31,73,125);font-family:"Calibri","sans-serif";font-size:11pt"><u></u> <u></u></span></p>
<p class="MsoNormal"><span style="color:rgb(31,73,125);font-family:"Calibri","sans-serif";font-size:11pt">I’m not familiar with those blades/switches. I’ll ask around.<u></u><u></u></span></p>
<p class="MsoNormal"><span style="color:rgb(31,73,125);font-family:"Calibri","sans-serif";font-size:11pt"><u></u> <u></u></span></p>
<p class="MsoNormal"><span style="color:rgb(31,73,125);font-family:"Calibri","sans-serif";font-size:11pt">Ira<u></u><u></u></span></p>
<p class="MsoNormal"><span style="color:rgb(31,73,125);font-family:"Calibri","sans-serif";font-size:11pt"><u></u> <u></u></span></p>
<div style="border-width:medium medium medium 1.5pt;border-style:none none none solid;border-color:currentColor currentColor currentColor blue;padding:0in 0in 0in 4pt">
<div>
<div style="border-width:1pt medium medium;border-style:solid none none;border-color:rgb(181,196,223) currentColor currentColor;padding:3pt 0in 0in">
<p class="MsoNormal"><b><span style="font-family:"Tahoma","sans-serif";font-size:10pt">From:</span></b><span style="font-family:"Tahoma","sans-serif";font-size:10pt"> Hal Rosenstock [<a href="mailto:hal.rosenstock@gmail.com" target="_blank">mailto:hal.rosenstock@gmail.com</a>]
<br>
<b>Sent:</b> Wednesday, October 07, 2015 1:26 PM<br>
<b>To:</b> German Anders<br>
<b>Cc:</b> Weiny, Ira; <a href="mailto:users@lists.openfabrics.org" target="_blank">users@lists.openfabrics.org</a><br>
<b>Subject:</b> Re: [Users] IB topology config and polling state<u></u><u></u></span></p>
</div>
</div>
<p class="MsoNormal"><u></u> <u></u></p>
<div>
<p class="MsoNormal">That's the gateway to the switch in the enclosure. It's the internal connectivity in the blade enclosure that's (physically) broken.<u></u><u></u></p>
</div>
<div>
<p class="MsoNormal"><u></u> <u></u></p>
<div>
<p class="MsoNormal">On Wed, Oct 7, 2015 at 4:24 PM, German Anders <<a href="mailto:ganders@despegar.com" target="_blank">ganders@despegar.com</a>> wrote:<u></u><u></u></p>
<div>
<div>
<p class="MsoNormal" style="margin-bottom:12pt"><span style="color:rgb(11,83,148);font-family:"Courier New"">cabled<u></u><u></u></span></p>
</div>
<div>
<p class="MsoNormal" style="margin-bottom:12pt"><span style="color:rgb(11,83,148);font-family:"Courier New"">the blade it's:<br>
<br>
vendid=0x2c9<br>
devid=0xbd36<br>
sysimgguid=0x2c902004b0918<br>
switchguid=0x2c902004b0918(2c902004b0918)<br>
Switch 32 "S-0002c902004b0918" # "Infiniscale-IV Mellanox Technologies" base port 0
</span><b><span style="color:red;font-family:"Courier New"">lid 29</span></b><span style="color:rgb(11,83,148);font-family:"Courier New""> lmc 0<br>
[1] "S-e41d2d030031e9c1"[9] # "MF0;GWIB01:SX6036G/U1" lid 24 4xQDR<u></u><u></u></span></p>
</div>
<div>
<p class="MsoNormal"><span style="color:rgb(136,136,136)"><br clear="all">
<span><u></u><u></u></span></span></p>
<div>
<div>
<div>
<div>
<div>
<div>
<div>
<p class="MsoNormal"><b><span style="color:rgb(68,68,68);font-family:"Trebuchet MS","sans-serif";font-size:10pt">German</span></b><u></u><u></u></p>
</div>
</div>
</div>
</div>
</div>
</div>
</div>
<div>
<div>
<p class="MsoNormal"><u></u> <u></u></p>
<div>
<p class="MsoNormal">2015-10-07 17:21 GMT-03:00 Hal Rosenstock <<a href="mailto:hal.rosenstock@gmail.com" target="_blank">hal.rosenstock@gmail.com</a>>:<u></u><u></u></p>
<div>
<p class="MsoNormal">What are those HCAs cabled to or is it internal to the blade enclosure ?<u></u><u></u></p>
</div>
<div>
<div>
<div>
<p class="MsoNormal"><u></u> <u></u></p>
<div>
<p class="MsoNormal">On Wed, Oct 7, 2015 at 3:24 PM, German Anders <<a href="mailto:ganders@despegar.com" target="_blank">ganders@despegar.com</a>> wrote:<u></u><u></u></p>
<div>
<div>
<p class="MsoNormal"><span style="color:rgb(11,83,148);font-family:"Courier New"">Yeah, is there any command that I can run in order to change the port state on the remote switch? I mean everything looks good but in the hp blades still getting:<u></u><u></u></span></p>
<div>
<div>
<p class="MsoNormal" style="margin-bottom:12pt"><span style="color:rgb(11,83,148);font-family:"Courier New""><br>
<br>
# ibstat<br>
CA 'qib0'<br>
CA type: InfiniPath_QMH7342<br>
Number of ports: 2<br>
Firmware version: <br>
Hardware version: 2<br>
Node GUID: 0x0011750000791fec<br>
System image GUID: 0x0011750000791fec<br>
Port 1:<br>
State: <b>Down</b><br>
Physical state: <b>Polling</b><br>
Rate: 40<br>
Base lid: 4660<br>
LMC: 0<br>
SM lid: 4660<br>
Capability mask: 0x0761086a<br>
Port GUID: 0x0011750000791fec<br>
Link layer: InfiniBand<br>
Port 2:<br>
State: <b>Down</b><br>
Physical state: <b>Polling</b><br>
Rate: 40<br>
Base lid: 4660<br>
LMC: 0<br>
SM lid: 4660<br>
Capability mask: 0x0761086a<br>
Port GUID: 0x0011750000791fed<br>
Link layer: InfiniBand<u></u><u></u></span></p>
</div>
</div>
</div>
<div>
<p class="MsoNormal" style="margin-bottom:12pt"><span style="color:rgb(11,83,148);font-family:"Courier New"">Also on working hosts I only see devices from the local network, but didn't see any of the blades hca connections.
<u></u><u></u></span></p>
</div>
<div>
<p class="MsoNormal"><span style="color:rgb(11,83,148);font-family:"Courier New""><u></u> <u></u></span></p>
</div>
<div>
<p class="MsoNormal"><span style="color:rgb(136,136,136)"><br clear="all">
<u></u><u></u></span></p>
<div>
<div>
<div>
<div>
<div>
<div>
<div>
<p class="MsoNormal"><b><span style="color:rgb(68,68,68);font-family:"Trebuchet MS","sans-serif";font-size:10pt">German</span></b><span style="color:rgb(136,136,136)"><u></u><u></u></span></p>
</div>
</div>
</div>
</div>
</div>
</div>
</div>
<div>
<div>
<p class="MsoNormal"><u></u> <u></u></p>
<div>
<p class="MsoNormal">2015-10-07 16:21 GMT-03:00 Hal Rosenstock <<a href="mailto:hal.rosenstock@gmail.com" target="_blank">hal.rosenstock@gmail.com</a>>:<u></u><u></u></p>
<div>
<div>
<p class="MsoNormal">The screen shot looks good :-) SM brought the link up to active.<u></u><u></u></p>
</div>
<div>
<p class="MsoNormal"> <u></u><u></u></p>
</div>
<div>
<p class="MsoNormal">Note that the ibportstate command you gave was for switch port 0 of the Mellanox IS-4 switch in the QLogic HP BLc 4X QDR IB Switch.<u></u><u></u></p>
</div>
</div>
<div>
<div>
<div>
<p class="MsoNormal"><u></u> <u></u></p>
<div>
<p class="MsoNormal">On Wed, Oct 7, 2015 at 3:06 PM, German Anders <<a href="mailto:ganders@despegar.com" target="_blank">ganders@despegar.com</a>> wrote:<u></u><u></u></p>
<div>
<div>
<p class="MsoNormal" style="margin-bottom:12pt"><span style="color:rgb(11,83,148);font-family:"Courier New"">Yes, find attached an screenshot of the port information (# 9) the one that makes the ISL to the QLogic HP BLc 4X QDR IB Switch, also from one of the hosts
that are connected to one of the SX6018F I can see the 'remote' HP IB SW:<u></u><u></u></span></p>
</div>
<div>
<p class="MsoNormal" style="margin-bottom:12pt"><span style="color:rgb(11,83,148);font-family:"Courier New"">#
<b>ibnodes</b><br>
<br>
(...)<br>
Switch : 0x0002c902004b0918 ports 32 "Infiniscale-IV Mellanox Technologies" base port 0
</span><b><span style="color:red;font-family:"Courier New"">lid 29</span></b><b><span style="color:rgb(11,83,148);font-family:"Courier New"">
</span></b><span style="color:rgb(11,83,148);font-family:"Courier New"">lmc 0<br>
Switch : 0xe41d2d030031e9c1 ports 37 "MF0;GWIB01:SX6036G/U1" enhanced port 0 lid 24 lmc 0<br>
(...)<br>
<br>
# <b>ibportstate -L 29 query</b><br>
Switch PortInfo:<br>
# Port info: Lid 29 port 0<br>
LinkState:.......................Active<br>
PhysLinkState:...................LinkUp<br>
Lid:.............................29<br>
SMLid:...........................2<br>
LMC:.............................0<br>
LinkWidthSupported:..............1X or 4X<br>
LinkWidthEnabled:................1X or 4X<br>
LinkWidthActive:.................4X<br>
LinkSpeedSupported:..............2.5 Gbps or 5.0 Gbps or 10.0 Gbps<br>
LinkSpeedEnabled:................2.5 Gbps or 5.0 Gbps or 10.0 Gbps<br>
LinkSpeedActive:.................10.0 Gbps<br>
Mkey:............................<not displayed><br>
MkeyLeasePeriod:.................0<br>
ProtectBits:.....................0<u></u><u></u></span></p>
</div>
<div>
<p class="MsoNormal" style="margin-bottom:12pt"><span style="color:rgb(11,83,148);font-family:"Courier New""><u></u> <u></u></span></p>
</div>
<div>
<p class="MsoNormal"><span style="color:rgb(136,136,136)"><br clear="all">
<u></u><u></u></span></p>
<div>
<div>
<div>
<div>
<div>
<div>
<div>
<p class="MsoNormal"><b><span style="color:rgb(68,68,68);font-family:"Trebuchet MS","sans-serif";font-size:10pt">German</span></b><span style="color:rgb(136,136,136)"><u></u><u></u></span></p>
</div>
</div>
</div>
</div>
</div>
</div>
</div>
<div>
<div>
<p class="MsoNormal"><u></u> <u></u></p>
<div>
<p class="MsoNormal">2015-10-07 16:00 GMT-03:00 Hal Rosenstock <<a href="mailto:hal.rosenstock@gmail.com" target="_blank">hal.rosenstock@gmail.com</a>>:<u></u><u></u></p>
<div>
<div>
<p class="MsoNormal">One more thing hopefully before playing with the low level phy settings:<u></u><u></u></p>
</div>
<div>
<p class="MsoNormal"> <u></u><u></u></p>
</div>
<div>
<p class="MsoNormal">Are you using known good cables ? Do you have FDR cables on the FDR <-> FDR links ? Cable lengths can matter as well.<u></u><u></u></p>
</div>
</div>
<div>
<div>
<div>
<p class="MsoNormal"><u></u> <u></u></p>
<div>
<p class="MsoNormal">On Wed, Oct 7, 2015 at 12:57 PM, Hal Rosenstock <<a href="mailto:hal.rosenstock@gmail.com" target="_blank">hal.rosenstock@gmail.com</a>> wrote:<u></u><u></u></p>
<div>
<div>
<p class="MsoNormal">Were the ports mapped to the phy profile shutdown when you changed this ?<u></u><u></u></p>
</div>
<div>
<p class="MsoNormal"> <u></u><u></u></p>
</div>
<div>
<p class="MsoNormal">LLR is a proprietary Mellanox mechanism.<u></u><u></u></p>
</div>
<div>
<p class="MsoNormal"> <u></u><u></u></p>
</div>
<div>
<p class="MsoNormal">You might want 2 different profiles: one for the interfaces connected to other gateway interfaces (which are FDR (and FDR-10) capable and the other for the interfaces connecting to QDR (the older equipment in your network). By configuring
the Switch-X interfaces to the appropriate possible speeds and disabling the proprietary mechanisms there, the link should not only come up but also this will occur faster than if FDR/FDR10 are enabled.<u></u><u></u></p>
</div>
<div>
<p class="MsoNormal"> <u></u><u></u></p>
</div>
<div>
<p class="MsoNormal">I suspect that due to the Switch-X configuration that the links to the switch(es) in the HP enclosures do not negotiate properly (as shown by down rather than LinkUp).<u></u><u></u></p>
</div>
<div>
<p class="MsoNormal"> <u></u><u></u></p>
</div>
<div>
<p class="MsoNormal">Once you get all your links to INIT, negotiation has occurred and then it's time for SM to bring links to active.<u></u><u></u></p>
</div>
<div>
<p class="MsoNormal"> <u></u><u></u></p>
</div>
<div>
<p class="MsoNormal">Since you have down links, the SM can't do anything about those.<u></u><u></u></p>
</div>
<div>
<p class="MsoNormal"> <u></u><u></u></p>
</div>
</div>
<div>
<div>
<div>
<p class="MsoNormal"><u></u> <u></u></p>
<div>
<p class="MsoNormal">On Wed, Oct 7, 2015 at 12:44 PM, German Anders <<a href="mailto:ganders@despegar.com" target="_blank">ganders@despegar.com</a>> wrote:<u></u><u></u></p>
<div>
<div>
<p class="MsoNormal" style="margin-bottom:12pt"><span style="color:rgb(11,83,148);font-family:"Courier New"">Anyone had any experience with HP BLc 4X QDR IB Switch?? I know that this kind of SW does not come with an embedded sm, but I don't know how to access any
mgmt at all on this particularly switch, I mean for example to setup speed or anything like that, is possible to access through the chassis?<u></u><u></u></span></p>
</div>
<div>
<p class="MsoNormal"><span style="color:rgb(136,136,136)"><br clear="all">
<u></u><u></u></span></p>
<div>
<div>
<div>
<div>
<div>
<div>
<div>
<p class="MsoNormal"><b><span style="color:rgb(68,68,68);font-family:"Trebuchet MS","sans-serif";font-size:10pt">German</span></b><span style="color:rgb(136,136,136)"><u></u><u></u></span></p>
</div>
</div>
</div>
</div>
</div>
</div>
</div>
<div>
<div>
<p class="MsoNormal"><u></u> <u></u></p>
<div>
<p class="MsoNormal">2015-10-07 13:19 GMT-03:00 German Anders <<a href="mailto:ganders@despegar.com" target="_blank">ganders@despegar.com</a>>:<u></u><u></u></p>
<div>
<div>
<p class="MsoNormal" style="margin-bottom:12pt"><span style="color:rgb(11,83,148);font-family:"Courier New"">I think so, but when trying to configured the phy-profile on the interface in order to negotiate on QDR it failed to map the profile:<br>
<br>
GWIB01 [proxy-ha-group: master] (config) # show phy-profile high-speed-ber <br>
<br>
Profile: high-speed-ber <br>
--------<br>
llr support ib-speed<br>
SDR: disable<br>
DDR: disable<br>
QDR: disable<br>
FDR10: enable-request<br>
FDR: enable-request<br>
<br>
GWIB01 [proxy-ha-group: master] (config) # show phy-profile hp-encl-isl <br>
<br>
Profile: hp-encl-isl <br>
--------<br>
llr support ib-speed<br>
SDR: disable<br>
DDR: disable<br>
QDR: enable<br>
FDR10: enable-request<br>
FDR: enable-request<br>
<br>
GWIB01 [proxy-ha-group: master] (config) # <br>
GWIB01 [proxy-ha-group: master] (config) # interface ib 1/9 phy-profile map hp-encl-isl
<br>
</span><b><span style="color:red;font-family:"Courier New"">% Cannot map profile hp-encl-isl to port: 1/9</span></b><span style="color:rgb(11,83,148);font-family:"Courier New""><u></u><u></u></span></p>
</div>
<div>
<p class="MsoNormal"><span style="color:rgb(136,136,136)"><br clear="all">
<u></u><u></u></span></p>
<div>
<div>
<div>
<div>
<div>
<div>
<div>
<p class="MsoNormal"><b><span style="color:rgb(68,68,68);font-family:"Trebuchet MS","sans-serif";font-size:10pt">German</span></b><span style="color:rgb(136,136,136)"><u></u><u></u></span></p>
</div>
</div>
</div>
</div>
</div>
</div>
</div>
<div>
<div>
<p class="MsoNormal"><u></u> <u></u></p>
<div>
<p class="MsoNormal">2015-10-07 13:17 GMT-03:00 Weiny, Ira <<a href="mailto:ira.weiny@intel.com" target="_blank">ira.weiny@intel.com</a>>:<u></u><u></u></p>
<div>
<div>
<p class="MsoNormal"><span style="color:rgb(31,73,125);font-family:"Calibri","sans-serif";font-size:11pt">The driver ‘qib’ is loading fine. As can be seen by the ibstat output. The ib_ipath is an older
card.</span><u></u><u></u></p>
<p class="MsoNormal"><span style="color:rgb(31,73,125);font-family:"Calibri","sans-serif";font-size:11pt"> </span><u></u><u></u></p>
<p class="MsoNormal"><span style="color:rgb(31,73,125);font-family:"Calibri","sans-serif";font-size:11pt">The problem is the link is not coming up to init. Like Hal said the link should transition to “link
up” without the SMs involvement.</span><u></u><u></u></p>
<p class="MsoNormal"><span style="color:rgb(31,73,125);font-family:"Calibri","sans-serif";font-size:11pt"> </span><u></u><u></u></p>
<p class="MsoNormal"><span style="color:rgb(31,73,125);font-family:"Calibri","sans-serif";font-size:11pt">I think you are on to something with the fact that it seems like your switch ports are not configured
to do QDR.</span><u></u><u></u></p>
<p class="MsoNormal"><span style="color:rgb(31,73,125);font-family:"Calibri","sans-serif";font-size:11pt"> </span><u></u><u></u></p>
<p class="MsoNormal"><span style="color:rgb(31,73,125);font-family:"Calibri","sans-serif";font-size:11pt">Ira</span><u></u><u></u></p>
<p class="MsoNormal"><span style="color:rgb(31,73,125);font-family:"Calibri","sans-serif";font-size:11pt"> </span><u></u><u></u></p>
<p class="MsoNormal"><span style="color:rgb(31,73,125);font-family:"Calibri","sans-serif";font-size:11pt"> </span><u></u><u></u></p>
<div style="border-width:medium medium medium 1.5pt;border-style:none none none solid;border-color:currentColor currentColor currentColor blue;padding:0in 0in 0in 4pt">
<div>
<div style="border-width:1pt medium medium;border-style:solid none none;border-color:currentColor;padding:3pt 0in 0in">
<p class="MsoNormal"><b><span style="font-family:"Tahoma","sans-serif";font-size:10pt">From:</span></b><span style="font-family:"Tahoma","sans-serif";font-size:10pt"> German Anders [mailto:<a href="mailto:ganders@despegar.com" target="_blank">ganders@despegar.com</a>]
<br>
<b>Sent:</b> Wednesday, October 07, 2015 9:05 AM<br>
<b>To:</b> Weiny, Ira<br>
<b>Cc:</b> Hal Rosenstock; <a href="mailto:users@lists.openfabrics.org" target="_blank">
users@lists.openfabrics.org</a></span><u></u><u></u></p>
<div>
<div>
<p class="MsoNormal"><br>
<b>Subject:</b> Re: [Users] IB topology config and polling state<u></u><u></u></p>
</div>
</div>
</div>
</div>
<div>
<div>
<p class="MsoNormal"> <u></u><u></u></p>
<div>
<div>
<p class="MsoNormal" style="margin-bottom:12pt"><span style="color:rgb(11,83,148);font-family:"Courier New"">Yes I've that file:<br>
<br>
/usr/sbin/truescale-serdes.cmds</span><u></u><u></u></p>
</div>
<div>
<p class="MsoNormal" style="margin-bottom:12pt"><span style="color:rgb(11,83,148);font-family:"Courier New"">Also I've done the install of libipathverbs:</span><u></u><u></u></p>
</div>
<div>
<p class="MsoNormal" style="margin-bottom:12pt"><span style="color:rgb(11,83,148);font-family:"Courier New""># apt-get install libipathverbs-dev</span><u></u><u></u></p>
</div>
<div>
<p class="MsoNormal" style="margin-bottom:12pt"><span style="color:rgb(11,83,148);font-family:"Courier New"">But I try to load the ib_ipath module but I'm getting the following error msg:<br>
<br>
# modprobe ib_ipath<br>
modprobe: ERROR: could not insert 'ib_ipath': Device or resource busy</span><u></u><u></u></p>
</div>
<div>
<p class="MsoNormal"><br clear="all">
<u></u><u></u></p>
<div>
<div>
<div>
<div>
<div>
<div>
<div>
<p class="MsoNormal"><b><span style="color:rgb(68,68,68);font-family:"Trebuchet MS","sans-serif";font-size:10pt">German</span></b><u></u><u></u></p>
</div>
</div>
</div>
</div>
</div>
</div>
</div>
<p class="MsoNormal"> <u></u><u></u></p>
<div>
<p class="MsoNormal">2015-10-07 12:54 GMT-03:00 Weiny, Ira <<a href="mailto:ira.weiny@intel.com" target="_blank">ira.weiny@intel.com</a>>:<u></u><u></u></p>
<div>
<div>
<p class="MsoNormal"><span style="color:rgb(31,73,125);font-family:"Calibri","sans-serif";font-size:11pt">There are a few issues for routing in that diagram but the links should come up.</span><u></u><u></u></p>
<p class="MsoNormal"><span style="color:rgb(31,73,125);font-family:"Calibri","sans-serif";font-size:11pt"> </span><u></u><u></u></p>
<p class="MsoNormal"><span style="color:rgb(31,73,125);font-family:"Calibri","sans-serif";font-size:11pt">I assume there is some backplane between the blade servers and the switch in that chassis?</span><u></u><u></u></p>
<p class="MsoNormal"><span style="color:rgb(31,73,125);font-family:"Calibri","sans-serif";font-size:11pt"> </span><u></u><u></u></p>
<p class="MsoNormal"><span style="color:rgb(31,73,125);font-family:"Calibri","sans-serif";font-size:11pt">Have you gotten libipathverbs installed?</span><u></u><u></u></p>
<p class="MsoNormal"><span style="color:rgb(31,73,125);font-family:"Calibri","sans-serif";font-size:11pt"> </span><u></u><u></u></p>
<p class="MsoNormal"><span style="color:rgb(31,73,125);font-family:"Calibri","sans-serif";font-size:11pt">In ipathverbs there is a serdes tuning script.</span><u></u><u></u></p>
<p class="MsoNormal"><span style="color:rgb(31,73,125);font-family:"Calibri","sans-serif";font-size:11pt"> </span><u></u><u></u></p>
<p class="MsoNormal"><span style="color:rgb(31,73,125);font-family:"Calibri","sans-serif";font-size:11pt"><a href="https://github.com/01org/libipathverbs/blob/master/truescale-serdes.cmds" target="_blank">https://github.com/01org/libipathverbs/blob/master/truescale-serdes.cmds</a></span><u></u><u></u></p>
<p class="MsoNormal"><span style="color:rgb(31,73,125);font-family:"Calibri","sans-serif";font-size:11pt"> </span><u></u><u></u></p>
<p class="MsoNormal"><span style="color:rgb(31,73,125);font-family:"Calibri","sans-serif";font-size:11pt">Does your libipathverbs include that file? If not try the latest from github.</span><u></u><u></u></p>
<p class="MsoNormal"><span style="color:rgb(31,73,125);font-family:"Calibri","sans-serif";font-size:11pt"> </span><u></u><u></u></p>
<p class="MsoNormal"><span style="color:rgb(31,73,125);font-family:"Calibri","sans-serif";font-size:11pt">Ira</span><u></u><u></u></p>
<p class="MsoNormal"><span style="color:rgb(31,73,125);font-family:"Calibri","sans-serif";font-size:11pt"> </span><u></u><u></u></p>
<p class="MsoNormal"><span style="color:rgb(31,73,125);font-family:"Calibri","sans-serif";font-size:11pt"> </span><u></u><u></u></p>
<div style="border-width:medium medium medium 1.5pt;border-style:none none none solid;border-color:currentColor currentColor currentColor blue;padding:0in 0in 0in 4pt">
<div>
<div style="border-width:1pt medium medium;border-style:solid none none;border-color:currentColor;padding:3pt 0in 0in">
<p class="MsoNormal"><b><span style="font-family:"Tahoma","sans-serif";font-size:10pt">From:</span></b><span style="font-family:"Tahoma","sans-serif";font-size:10pt">
<a href="mailto:users-bounces@lists.openfabrics.org" target="_blank">users-bounces@lists.openfabrics.org</a> [mailto:<a href="mailto:users-bounces@lists.openfabrics.org" target="_blank">users-bounces@lists.openfabrics.org</a>]
<b>On Behalf Of </b>German Anders<br>
<b>Sent:</b> Wednesday, October 07, 2015 8:41 AM<br>
<b>To:</b> Hal Rosenstock<br>
<b>Cc:</b> <a href="mailto:users@lists.openfabrics.org" target="_blank">users@lists.openfabrics.org</a><br>
<b>Subject:</b> Re: [Users] IB topology config and polling state</span><u></u><u></u></p>
</div>
</div>
<div>
<div>
<p class="MsoNormal"> <u></u><u></u></p>
<div>
<div>
<p class="MsoNormal" style="margin-bottom:12pt"><span style="color:rgb(11,83,148);font-family:"Courier New"">Hi Hal,</span><u></u><u></u></p>
</div>
<div>
<p class="MsoNormal" style="margin-bottom:12pt"><span style="color:rgb(11,83,148);font-family:"Courier New"">Thanks for the reply, I've attach a pdf with the diagram topology, I don't know if this is the best way to go or if there's another
way to connect and setup the IB network, tips and suggestions will be very appreciated, also the mezzanine cards are already installed on the blade hosts:</span><u></u><u></u></p>
</div>
<div>
<p class="MsoNormal" style="margin-bottom:12pt"><span style="color:rgb(11,83,148);font-family:"Courier New""># lspci<br>
(...)<br>
41:00.0 InfiniBand: QLogic Corp. IBA7322 QDR InfiniBand HCA (rev 02)</span><u></u><u></u></p>
</div>
<div>
<p class="MsoNormal"><span style="color:rgb(11,83,148);font-family:"Courier New""> </span><u></u><u></u></p>
</div>
<div>
<p class="MsoNormal" style="margin-bottom:12pt"><span style="color:rgb(11,83,148);font-family:"Courier New"">Thanks in advance,</span><u></u><u></u></p>
</div>
<div>
<p class="MsoNormal" style="margin-bottom:12pt"><span style="color:rgb(11,83,148);font-family:"Courier New"">Cheers,</span><u></u><u></u></p>
</div>
<div>
<p class="MsoNormal"><br clear="all">
<u></u><u></u></p>
<div>
<div>
<div>
<div>
<div>
<div>
<div>
<p class="MsoNormal"><b><span style="color:rgb(68,68,68);font-family:"Trebuchet MS","sans-serif";font-size:10pt">German</span></b><u></u><u></u></p>
</div>
</div>
</div>
</div>
</div>
</div>
</div>
<p class="MsoNormal"> <u></u><u></u></p>
<div>
<p class="MsoNormal">2015-10-07 11:47 GMT-03:00 Hal Rosenstock <<a href="mailto:hal.rosenstock@gmail.com" target="_blank">hal.rosenstock@gmail.com</a>>:<u></u><u></u></p>
<div>
<div>
<p class="MsoNormal">Hi again German,<u></u><u></u></p>
</div>
<div>
<p class="MsoNormal"> <u></u><u></u></p>
</div>
<div>
<p class="MsoNormal">Looks like you made some progress from yesterday as the qib ports are now Polling rather than Disabled.<u></u><u></u></p>
</div>
<div>
<p class="MsoNormal"> <u></u><u></u></p>
</div>
<div>
<p class="MsoNormal">But since they are Down, do you have them cabled to a switch ? That should bring the links up and the port state will be Init. That is the "starting" point.<u></u><u></u></p>
</div>
<div>
<p class="MsoNormal"> <u></u><u></u></p>
</div>
<div>
<p class="MsoNormal">You will also then need to be running SM to bring the ports up to Active.<u></u><u></u></p>
</div>
<div>
<p class="MsoNormal"> <u></u><u></u></p>
</div>
<div>
<p class="MsoNormal">-- Hal<u></u><u></u></p>
</div>
</div>
<div>
<p class="MsoNormal"> <u></u><u></u></p>
<div>
<div>
<div>
<p class="MsoNormal">On Wed, Oct 7, 2015 at 10:37 AM, German Anders <<a href="mailto:ganders@despegar.com" target="_blank">ganders@despegar.com</a>> wrote:<u></u><u></u></p>
</div>
</div>
<blockquote style="border-width:medium medium medium 1pt;border-style:none none none solid;border-color:currentColor currentColor currentColor rgb(204,204,204);margin:5pt 0in 5pt 4.8pt;padding:0in 0in 0in 6pt">
<div>
<div>
<div>
<div>
<p class="MsoNormal" style="margin-bottom:12pt"><span style="color:rgb(11,83,148);font-family:"Courier New"">Hi all,</span><u></u><u></u></p>
</div>
<div>
<p class="MsoNormal" style="margin-bottom:12pt"><span style="color:rgb(11,83,148);font-family:"Courier New"">I don't know if this is the mailist list for this kind of topic but I'm really new to IB and I've just install two SX6036G gateways
connected to each other through two ISL ports, then I've configured a proxy-arp between both nodes (sm is disable on both gw's):<br>
<br>
GWIB01 [proxy-ha-group: master] (config) # show proxy-arp ha <br>
<br>
Load balancing algorithm: ib-base-ip<br>
Number of Proxy-Arp interfaces: 1<br>
<br>
Proxy-ARP VIP<br>
=============<br>
Pra-group name: proxy-ha-group<br>
HA VIP address: 10.xx.xx.xx/xx<br>
<br>
Active nodes:<br>
ID State IP <br>
--------------------------------------------------------------<br>
GWIB01 master 10.xx.xx.xx1 <br>
GWIB02 standby 10.xx.xx.xx2 </span><u></u><u></u></p>
</div>
<div>
<p class="MsoNormal" style="margin-bottom:12pt"><span style="color:rgb(11,83,148);font-family:"Courier New"">Then I setup two SX6018F switches (<b>SWIB01</b> and
<b>SWIB02</b>), one connected to GWIB01 and the other connected to GWIB02. The SM is configured locally on both SWIB01 & SWIB02 switches. So far so good, after this config I setup a commodity server with a MLNX IB ADPT FDR to the SWIB01 & SWIB02 switches, config
the drivers, etc and then get it up & running fine. <br>
<br>
Finally I've setup a HP Enclosure with an internal IB SW (then connect port 1 of the internal SW to GWIB01 - link is up but LLR status is inactive), install one of the blades and I see the following:<br>
<br>
# ibstat<br>
CA 'qib0'<br>
CA type: InfiniPath_QMH7342<br>
Number of ports: 2<br>
Firmware version: <br>
Hardware version: 2<br>
Node GUID: 0x0011750000791fec<br>
System image GUID: 0x0011750000791fec<br>
Port 1:<br>
State: Down<br>
Physical state: Polling<br>
Rate: 40<br>
Base lid: 4660<br>
LMC: 0<br>
SM lid: 4660<br>
Capability mask: 0x0761086a<br>
Port GUID: 0x0011750000791fec<br>
Link layer: InfiniBand<br>
Port 2:<br>
State: Down<br>
Physical state: Polling<br>
Rate: 40<br>
Base lid: 4660<br>
LMC: 0<br>
SM lid: 4660<br>
Capability mask: 0x0761086a<br>
Port GUID: 0x0011750000791fed<br>
Link layer: InfiniBand</span><u></u><u></u></p>
</div>
<div>
<p class="MsoNormal" style="margin-bottom:12pt"><span style="color:rgb(11,83,148);font-family:"Courier New"">So I was wondering if maybe the SM is not being recognized on the Blade system and that's why is not passing the Polling state, is
that possible? Or maybe is not possible to connect an ISL between the GW and the HP internal SW so that the sm is available or maybe the inactive LLR is causing this thing, any ideas? I
</span><span lang="EN" style="color:rgb(11,83,148);font-family:"Courier New"">thought about connecting the ISL of the HP IB SW to the SWIB01 or SWIB02 instead of the GW's but I don't have any available ports.</span><u></u><u></u></p>
</div>
<div>
<p class="MsoNormal" style="margin-bottom:12pt"><span lang="EN" style="color:rgb(11,83,148);font-family:"Courier New"">Thanks in advance,</span><u></u><u></u></p>
</div>
<div>
<p class="MsoNormal"><span lang="EN" style="color:rgb(11,83,148);font-family:"Courier New"">Cheers,</span><u></u><u></u></p>
</div>
<div>
<p class="MsoNormal"><span style="color:rgb(11,83,148);font-family:"Courier New""><br clear="all">
</span><u></u><u></u></p>
</div>
<div>
<div>
<div>
<div>
<div>
<div>
<div>
<p class="MsoNormal"><b><span style="color:rgb(68,68,68);font-family:"Trebuchet MS","sans-serif";font-size:10pt">German</span></b><u></u><u></u></p>
</div>
</div>
</div>
</div>
</div>
</div>
</div>
</div>
<p class="MsoNormal"> <u></u><u></u></p>
</div>
</div>
<p class="MsoNormal" style="margin-bottom:12pt">_______________________________________________<br>
Users mailing list<br>
<a href="mailto:Users@lists.openfabrics.org" target="_blank">Users@lists.openfabrics.org</a><br>
<a href="http://lists.openfabrics.org/mailman/listinfo/users" target="_blank">http://lists.openfabrics.org/mailman/listinfo/users</a><u></u><u></u></p>
</blockquote>
</div>
<p class="MsoNormal"> <u></u><u></u></p>
</div>
</div>
<p class="MsoNormal"> <u></u><u></u></p>
</div>
</div>
</div>
</div>
</div>
</div>
</div>
</div>
<p class="MsoNormal"> <u></u><u></u></p>
</div>
</div>
</div>
</div>
</div>
</div>
</div>
</div>
<p class="MsoNormal"><u></u> <u></u></p>
</div>
</div>
</div>
</div>
</div>
<p class="MsoNormal"><u></u> <u></u></p>
</div>
</div>
</div>
</div>
</div>
<p class="MsoNormal"><u></u> <u></u></p>
</div>
</div>
</div>
</div>
<p class="MsoNormal"><u></u> <u></u></p>
</div>
</div>
</div>
</div>
<p class="MsoNormal"><u></u> <u></u></p>
</div>
</div>
</div>
</div>
</div>
<p class="MsoNormal"><u></u> <u></u></p>
</div>
</div>
</div>
</div>
<p class="MsoNormal"><u></u> <u></u></p>
</div>
</div>
</div>
</div>
</div>
<p class="MsoNormal"><u></u> <u></u></p>
</div>
</div>
</div>
</div>
<p class="MsoNormal"><u></u> <u></u></p>
</div>
</div>
</div>
</div>
</div>
<p class="MsoNormal"><u></u> <u></u></p>
</div>
</div>
</div></div></div>
</div>
</div>
</blockquote></div><br></div></div></div></div>
</blockquote></div><br></div>
</div></div></blockquote></div><br></div></div></div></div>
</blockquote></div><br></div>
</div></div></blockquote></div><br></div></div></div></div>
</blockquote></div><br></div>
</div></div></blockquote></div><br></div></div></div></div>
</blockquote></div><br></div>
</div></div></blockquote></div><br></div></div></div></div>
</blockquote></div><br></div></div></div></div>
</blockquote></div><br></div></div></div></div>
</blockquote></div><br></div>