<br><br><div class="gmail_quote">On Tue, May 28, 2013 at 8:19 AM, Andrei Mikhailovsky <span dir="ltr"><<a href="mailto:andrei@arhont.com" target="_blank">andrei@arhont.com</a>></span> wrote:<br><blockquote style="margin:0px 0px 0px 0.8ex;padding-left:1ex;border-left-color:rgb(204,204,204);border-left-width:1px;border-left-style:solid" class="gmail_quote">
<div><div style="font-family:arial,helvetica,sans-serif;font-size:10pt">Hello guys,<br><br>I've just rebooted one of the server which last time took over 10 hours to get an ACTIVE port state. It has been 39 minutes since the reboot and so far, no link ((<br>
<br>I do see a rather large number of SymbolErrorCounter, which doesn't seems to change with the reset as you can see from the perfqery command below:<br><br>perfquery -r 2 18<br># Port counters: Lid 2 port 18 (CapMask: 0x500)<br>
PortSelect:......................18<br>CounterSelect:...................0x0000<br>SymbolErrorCounter:..............65535<br></div></div></blockquote><div> </div><div>That's a max'd out symbol errors counter and IB counters are sticky at max rather than rollover like IETF counters. To see if it's still changing, it needs to be reset. You can use -r option of perfquery to do this.</div>
<div> </div><blockquote style="margin:0px 0px 0px 0.8ex;padding-left:1ex;border-left-color:rgb(204,204,204);border-left-width:1px;border-left-style:solid" class="gmail_quote"><div><div style="font-family:arial,helvetica,sans-serif;font-size:10pt">
LinkErrorRecoveryCounter:........0<br>LinkDownedCounter:...............0<br>PortRcvErrors:...................0<br>PortRcvRemotePhysicalErrors:.....0<br>PortRcvSwitchRelayErrors:........0<br>PortXmitDiscards:................0<br>
PortXmitConstraintErrors:........0<br>PortRcvConstraintErrors:.........0<br>CounterSelect2:..................0x00<br>LocalLinkIntegrityErrors:........0<br>ExcessiveBufferOverrunErrors:....0<br>VL15Dropped:.....................0<br>
PortXmitData:....................0<br>PortRcvData:.....................0<br>PortXmitPkts:....................0<br>PortRcvPkts:.....................0<br><br>The card's State is shown as DOWN, but the Physical State changes from 2: Polling to 4: PortConfigurationTraining to 3: Disabled to 16: <unknown>.<br>
<br>OpenSM logs show the following:<br><br>May 28 12:31:06 097874 [26274700] 0x02 -> log_notice: Reporting Generic Notice type:3 num:67 (Mcast group deleted) from LID:1 GID:ff12:601b:ffff::202<br>May 28 12:31:06 098319 [27276700] 0x01 -> mcmr_rcv_join_mgrp: ERR 1B11: method = SubnAdmSet, scope_state = 0x1, component mask = 0x0000000000010083, expected comp mask = 0x00000000000130c7, MGID: ff12:601b:ffff::16 from port 0x001175000079669a (arh-cloud2 HCA-1)<br>
May 28 12:31:08 936842 [2DA83700] 0x02 -> log_notice: Reporting Generic Notice type:3 num:67 (Mcast group deleted) from LID:1 GID:ff12:601b:ffff::1:ff79:669a<br>May 28 12:31:15 955330 [2A27C700] 0x01 -> log_trap_info: Received Generic Notice type:1 num:128 (Link state change) Producer:2 (Switch) from LID:2 TID:0x0000008000000026<br>
May 28 12:31:15 955471 [2A27C700] 0x02 -> log_notice: Reporting Generic Notice type:1 num:128 (Link state change) from LID:2 GID:fe80::6:6a00:f000:24d<br>May 28 12:31:15 960718 [24A71700] 0x02 -> log_notice: Reporting Generic Notice type:3 num:65 (GID out of service) from LID:1 GID:fe80::11:7500:79:669a<br>
May 28 12:31:15 960860 [24A71700] 0x02 -> drop_mgr_remove_port: Removed port with GUID:0x001175000079669a LID range [3, 3] of node:arh-cloud2 HCA-1<br>May 28 12:31:15 960904 [24A71700] 0x01 -> osm_prtn_make_partitions: Partition configuration /etc/opensm/partitions.conf is not accessible (No such file or directory)<br>
<br>which seem to be entries corresponding to the server going to reboot. I do not see anything on the opensm side to indicate that the server (arh-cloud2) is trying to negotiate the link.<br></div></div></blockquote><div>
</div><div>So this is the link state change trap when that port goes DOWN but since the port never becomes LinkUp/Init, there's never an additional trap 128 from the switch for that.</div><div> </div><div>-- Hal</div>
<blockquote style="margin:0px 0px 0px 0.8ex;padding-left:1ex;border-left-color:rgb(204,204,204);border-left-width:1px;border-left-style:solid" class="gmail_quote"><div><div style="font-family:arial,helvetica,sans-serif;font-size:10pt">
<br>Andrei<br><hr><div style="font-family:Helvetica,Arial,sans-serif;font-size:12pt;font-style:normal;font-weight:normal;text-decoration:none"><b>From: </b>"Hal Rosenstock" <<a href="mailto:hal.rosenstock@gmail.com" target="_blank">hal.rosenstock@gmail.com</a>><br>
<b>To: </b>"Andrei Mikhailovsky" <<a href="mailto:andrei@arhont.com" target="_blank">andrei@arhont.com</a>><br><b>Cc: </b>"John Valdes" <<a href="mailto:valdes@anl.gov" target="_blank">valdes@anl.gov</a>>, <a href="mailto:users@lists.openfabrics.org" target="_blank">users@lists.openfabrics.org</a><br>
<b>Sent: </b>Sunday, 26 May, 2013 2:47:55 PM<div><div class="h5"><br><b>Subject: </b>Re: [Users] HP BLc QLogic 4X QDR IB Switch oddness<br><br><br><br><div class="gmail_quote">On Sun, May 26, 2013 at 9:09 AM, Andrei Mikhailovsky <span dir="ltr"><<a href="mailto:andrei@arhont.com" target="_blank">andrei@arhont.com</a>></span> wrote:<br>
<blockquote style="margin:0px 0px 0px 0.8ex;padding-left:1ex;border-left-color:rgb(204,204,204);border-left-width:1px;border-left-style:solid" class="gmail_quote">
<div><div style="font-family:arial,helvetica,sans-serif;font-size:10pt"><br><br><hr><div style="font-family:Helvetica,Arial,sans-serif;font-size:12pt;font-style:normal;font-weight:normal;text-decoration:none"><div>
<b>From: </b>"John Valdes" <<a href="mailto:valdes@anl.gov" target="_blank">valdes@anl.gov</a>><br><b>To: </b>"Andrei Mikhailovsky" <<a href="mailto:andrei@arhont.com" target="_blank">andrei@arhont.com</a>><br>
<b>Cc: </b><a href="mailto:users@lists.openfabrics.org" target="_blank">users@lists.openfabrics.org</a><br></div><b>Sent: </b>Saturday, 25 May, 2013 2:19:40 AM<div><br><b>Subject: </b>Re: [Users] HP BLc QLogic 4X QDR IB Switch oddness<br>
<br></div><div>Andrei Mikhailovsky wrote:<br>> John Valdes wrote:<br>> > What's the physical topology of the IB network between the blade <br>> > servers and the switch?<br>> <br>> AM: I am not really sure. The servers do have the IB mezzanine card and from what i've read it is a pci-e card. I am unsure how the blade servers are connected to the switch. I guess it's an internal HP/QLogic interconnect. <br>
<br>Found some docs on Intel's website at:<br><a href="http://www.intel.com/p/en_US/support/highlights/network/ts-fbs12100" target="_blank">http://www.intel.com/p/en_US/support/highlights/network/ts-fbs12100</a> <br>
>From that, it looks like the topology is very simple; the switch<br>
installs in a slot in the blade chassis, and it looks like it has 16<br>internal (through the backplane of the chassis) IB connections, one<br>to each blade server in the chassis, plus 16 external QSFP ports.<br><br><br>
</div>
AM: yeah, that pretty much sums up the switch<div><br><br><br><br>I was thinking maybe there was something odd in the topology that<br>was causing the subnet manager to fail to negotiate link properly w/<br>the blade servers. It doesn't sound like that's the case.<br>
<br><br></div>AM: I do not see any logs on the SM side when the port state changes. The only logs I see is when the port becomes Active, I see that in the logs and no errors before or prior to that</div><div style="font-family:Helvetica,Arial,sans-serif;font-size:12pt;font-style:normal;font-weight:normal;text-decoration:none">
</div></div></div></blockquote><div> </div><div>By port state, do you mean port state or port physical state ? Note that there's some relationship between the two but port physical state can change without port state changing. In the opensm log, you should see trap 128 when link (port) state changes. If the port/link is constantly (re)negotiating and doesn't get to LinkUp (port physical state)/Init (port state), you won't see this in the log. If port state is truly changing, you should see this trap in the opensm log.</div>
<div> </div><div>-- Hal<font size="1" face="ArialMT"><font size="1" face="ArialMT"></font></font></div><font size="1" face="ArialMT"><font size="1" face="ArialMT"></font></font><div> </div><blockquote style="margin:0px 0px 0px 0.8ex;padding-left:1ex;border-left-color:rgb(204,204,204);border-left-width:1px;border-left-style:solid" class="gmail_quote">
<div><div style="font-family:arial,helvetica,sans-serif;font-size:10pt"><div style="font-family:Helvetica,Arial,sans-serif;font-size:12pt;font-style:normal;font-weight:normal;text-decoration:none"> </div><br></div></div>
<br>
_______________________________________________<br>
Users mailing list<br>
<a href="mailto:Users@lists.openfabrics.org" target="_blank">Users@lists.openfabrics.org</a><br>
<a href="http://lists.openfabrics.org/cgi-bin/mailman/listinfo/users" target="_blank">http://lists.openfabrics.org/cgi-bin/mailman/listinfo/users</a><br>
<br></blockquote></div><br>
</div></div></div><br></div></div></blockquote></div><br>