<br><font size=2 face="sans-serif">Hi Ira,</font>
<br>
<br><font size=2 face="sans-serif">Here is the output of "iblinkinfo.pl
-R":</font>
<br>
<br><font size=2 face="sans-serif">++++++++++++++++++++++++++++++++++++++++++++++++</font>
<br><font size=2 face="sans-serif">[root@ib_manager ~]# iblinkinfo.pl -R</font>
<br><font size=2 face="sans-serif">Switch 0x00066a00d90009c1 InfiniCon
System InfinIO 9024 Lite:</font>
<br><font size=2 face="sans-serif"> 7 1[
] ==( 4X 2.5 Gbps Active / LinkUp)==>
6 1[ ] "MT2520</font>
<br><font size=2 face="sans-serif">4 InfiniHostLx Mellanox Technologies"
( )</font>
<br><font size=2 face="sans-serif"> 2[
] ==( 4X 2.5 Gbps Down / Polling)==>
[ ] "" ( )</font>
<br>
<br><font size=2 face="sans-serif"> 3[
] ==( 4X 2.5 Gbps Down / Polling)==>
[ ] "" ( )</font>
<br>
<br><font size=2 face="sans-serif"> 4[
] ==( 4X 2.5 Gbps Down / Polling)==>
[ ] "" ( )</font>
<br>
<br><font size=2 face="sans-serif"> 5[
] ==( 4X 2.5 Gbps Down / Polling)==>
[ ] "" ( )</font>
<br>
<br><font size=2 face="sans-serif"> 6[
] ==( 4X 2.5 Gbps Down / Polling)==>
[ ] "" ( )</font>
<br>
<br><font size=2 face="sans-serif"> 7[
] ==( 4X 2.5 Gbps Down / Polling)==>
[ ] "" ( )</font>
<br>
<br><font size=2 face="sans-serif"> 8[
] ==( 4X 2.5 Gbps Down / Polling)==>
[ ] "" ( )</font>
<br>
<br><font size=2 face="sans-serif"> 9[
] ==( 4X 2.5 Gbps Down / Polling)==>
[ ] "" ( )</font>
<br>
<br><font size=2 face="sans-serif"> 10[
] ==( 4X 2.5 Gbps Down / Polling)==>
[ ] "" ( )</font>
<br>
<br><font size=2 face="sans-serif"> 11[
] ==( 4X 2.5 Gbps Down / Polling)==>
[ ] "" ( )</font>
<br>
<br><font size=2 face="sans-serif"> 12[
] ==( 4X 2.5 Gbps Down / Polling)==>
[ ] "" ( )</font>
<br>
<br><font size=2 face="sans-serif"> 13[
] ==( 4X 2.5 Gbps Down / Polling)==>
[ ] "" ( )</font>
<br>
<br><font size=2 face="sans-serif"> 14[
] ==( 4X 2.5 Gbps Down / Polling)==>
[ ] "" ( )</font>
<br>
<br><font size=2 face="sans-serif"> 15[
] ==( 4X 2.5 Gbps Down / Polling)==>
[ ] "" ( )</font>
<br>
<br><font size=2 face="sans-serif"> 16[
] ==( 4X 2.5 Gbps Down / Polling)==>
[ ] "" ( )</font>
<br>
<br><font size=2 face="sans-serif"> 17[
] ==( 4X 2.5 Gbps Down / Polling)==>
[ ] "" ( )</font>
<br>
<br><font size=2 face="sans-serif"> 18[
] ==( 4X 2.5 Gbps Down / Polling)==>
[ ] "" ( )</font>
<br>
<br><font size=2 face="sans-serif"> 19[
] ==( 4X 2.5 Gbps Down / Polling)==>
[ ] "" ( )</font>
<br>
<br><font size=2 face="sans-serif"> 20[
] ==( 4X 2.5 Gbps Down / Polling)==>
[ ] "" ( )</font>
<br>
<br><font size=2 face="sans-serif"> 21[
] ==( 4X 2.5 Gbps Down / Polling)==>
[ ] "" ( )</font>
<br>
<br><font size=2 face="sans-serif"> 22[
] ==( 4X 2.5 Gbps Down / Polling)==>
[ ] "" ( )</font>
<br>
<br><font size=2 face="sans-serif"> 23[
] ==( 4X 2.5 Gbps Down / Polling)==>
[ ] "" ( )</font>
<br>
<br><font size=2 face="sans-serif"> 24[
] ==( 4X 2.5 Gbps Down / Polling)==>
[ ] "" ( )</font>
<br><font size=2 face="sans-serif">++++++++++++++++++++++++++++++++++++++++++++++++++++</font>
<br>
<br><font size=2 face="sans-serif">And the "ibstat" output:</font>
<br><font size=2 face="sans-serif">+++++++++++++++++++++++++++++++++++</font>
<br><font size=2 face="sans-serif">[root@ib_manager ~]# ibstat</font>
<br><font size=2 face="sans-serif">CA 'mthca0'</font>
<br><font size=2 face="sans-serif"> CA type:
MT25204</font>
<br><font size=2 face="sans-serif"> Number of
ports: 1</font>
<br><font size=2 face="sans-serif"> Firmware
version: 1.2.0</font>
<br><font size=2 face="sans-serif"> Hardware
version: a0</font>
<br><font size=2 face="sans-serif"> Node GUID:
0x0002c90200230784</font>
<br><font size=2 face="sans-serif"> System image
GUID: 0x0002c90200230787</font>
<br><font size=2 face="sans-serif"> Port 1:</font>
<br><font size=2 face="sans-serif">
State: Active</font>
<br><font size=2 face="sans-serif">
Physical state: LinkUp</font>
<br><font size=2 face="sans-serif">
Rate: 10</font>
<br><font size=2 face="sans-serif">
Base lid: 6</font>
<br><font size=2 face="sans-serif">
LMC: 0</font>
<br><font size=2 face="sans-serif">
SM lid: 6</font>
<br><font size=2 face="sans-serif">
Capability mask: 0x02500a6a</font>
<br><font size=2 face="sans-serif">
Port GUID: 0x0002c90200230785</font>
<br><font size=2 face="sans-serif">++++++++++++++++++++++++++++++++++++++++++++++</font>
<br>
<br><font size=2 face="sans-serif">The reset command I am using is "ibportstate
7 1 reset", I also tried "ibportstate -D 0,1 1 reset", and
it fails with the same result.</font>
<br>
<br><font size=2 face="sans-serif">Thanks!</font>
<br><font size=2 face="sans-serif">Yicheng Jia<br>
<br>
</font>
<br>
<br>
<br>
<table width=100%>
<tr valign=top>
<td width=40%><font size=1 face="sans-serif"><b>Ira Weiny <weiny2@llnl.gov></b>
</font>
<p><font size=1 face="sans-serif">04/16/2009 07:53 PM</font>
<td width=59%>
<table width=100%>
<tr valign=top>
<td>
<div align=right><font size=1 face="sans-serif">To</font></div>
<td><font size=1 face="sans-serif">Hal Rosenstock <hal.rosenstock@gmail.com></font>
<tr valign=top>
<td>
<div align=right><font size=1 face="sans-serif">cc</font></div>
<td><font size=1 face="sans-serif">Yicheng Jia <YJia@tmriusa.com>,
general@lists.openfabrics.org</font>
<tr valign=top>
<td>
<div align=right><font size=1 face="sans-serif">Subject</font></div>
<td><font size=1 face="sans-serif">Re: ***SPAM*** Re: [ofa-general] link
width problem of Qlogic 9024 unmanaged switch</font></table>
<br>
<table>
<tr valign=top>
<td>
<td></table>
<br></table>
<br>
<br>
<br><tt><font size=2>Yicheng,<br>
<br>
I am hoping your system is small when I ask; could you send the output
from: "iblinkinfo.pl -R" When everything is up and running.<br>
<br>
Also ibstat from the node you are attempting the reset on. As well
as the reset command you are using?<br>
<br>
Thanks,<br>
Ira<br>
<br>
On Thu, 16 Apr 2009 19:12:00 -0400<br>
Hal Rosenstock <hal.rosenstock@gmail.com> wrote:<br>
<br>
> On Thu, Apr 16, 2009 at 6:06 PM, Yicheng Jia <YJia@tmriusa.com>
wrote:<br>
> ><br>
> >> There's a race condition here that I was asking about. If
the link<br>
> >> initialization takes too long and doesn't complete (gets
to init)<br>
> >> prior to the enable trying to be sent to the switch, then
you could<br>
> >> see these results but since it's DOWN until reboot it's something<br>
> >> different.<br>
> ><br>
> > I did the "reset" when ports on both side of the link
are in INIT state and<br>
> > LinkUp phys state.<br>
> ><br>
> >> If the disable/wait/enable worked that would've been another
story.<br>
> ><br>
> > It fails too. Both ports go to DOWN after disable is issued and
never come<br>
> > back. How long am I supposed to wait?<br>
> <br>
> Ideally you would see init before doing the enable but sounds like<br>
> that's not occuring. Either you need low level debug to see why the<br>
> link does not initialize at that point or get support from your<br>
> CA/switch vendor(s). What's your CA device ?<br>
> <br>
> -- Hal<br>
> <br>
> > Thanks!<br>
> > Yicheng Jia<br>
> ><br>
> ><br>
> ><br>
> ><br>
> > Hal Rosenstock <hal.rosenstock@gmail.com><br>
> ><br>
> > 04/16/2009 03:26 PM<br>
> ><br>
> > To<br>
> > Yicheng Jia <YJia@tmriusa.com><br>
> > cc<br>
> > Nicolas Morey-Chaisemartin <devel-ofed@morey-chaisemartin.com>,<br>
> > general@lists.openfabrics.org<br>
> > Subject<br>
> > Re: [ofa-general] link width problem of Qlogic 9024 unmanaged
switch<br>
> ><br>
> ><br>
> ><br>
> ><br>
> > On Thu, Apr 16, 2009 at 3:47 PM, Yicheng Jia <YJia@tmriusa.com>
wrote:<br>
> >><br>
> >>> Are you resetting the switch from the peer HCA port or
some other port<br>
> >>> ? That's what Nicolas asked but I might have missed the
answer.<br>
> >><br>
> >> Yes, I am trying to reset from the peer HCA port. Is anything
wrong with<br>
> >> this?<br>
> ><br>
> > There's a race condition here that I was asking about. If the
link<br>
> > initialization takes too long and doesn't complete (gets to init)<br>
> > prior to the enable trying to be sent to the switch, then you
could<br>
> > see these results but since it's DOWN until reboot it's something<br>
> > different.<br>
> ><br>
> >>> Also, try disable (wait) and then enable and see if that
works.<br>
> >><br>
> >> It remains the same, the switch port is DOWN forever. No
SMP massage could<br>
> >> get to the switch port.<br>
> ><br>
> > Right; in down, the SMP can't be sent.<br>
> ><br>
> >>> If I recall correctly, you had those links which are
taking a long time<br>
> >>> to<br>
> >>> initialize. If the link stays down forever after disable,
this won't<br>
> >>> work but I want to be sure.<br>
> >><br>
> >> This is seperate issue.<br>
> ><br>
> > Since the link stays down yes. If the disable/wait/enable worked
that<br>
> > would've been another story.<br>
> ><br>
> > -- Hal<br>
> ><br>
> >> The "reset" command is tested on a single port
HCA<br>
> >> directly connected with Qlogic siwth. The HCA is plugged
into a Linux<br>
> >> machine. It is the simplest test environment.<br>
> >><br>
> >> Thanks!<br>
> >> Yicheng Jia<br>
> >><br>
> >><br>
> >><br>
> >><br>
> >> Hal Rosenstock <hal.rosenstock@gmail.com><br>
> >><br>
> >> 04/16/2009 02:29 PM<br>
> >><br>
> >> To<br>
> >> Yicheng Jia <YJia@tmriusa.com><br>
> >> cc<br>
> >> Nicolas Morey-Chaisemartin <devel-ofed@morey-chaisemartin.com>,<br>
> >> general@lists.openfabrics.org<br>
> >> Subject<br>
> >> Re: [ofa-general] link width problem of Qlogic 9024 unmanaged
switch<br>
> >><br>
> >><br>
> >><br>
> >><br>
> >> On Thu, Apr 16, 2009 at 3:20 PM, Hal Rosenstock<br>
> >> <hal.rosenstock@gmail.com> wrote:<br>
> >>> On Thu, Apr 16, 2009 at 3:18 PM, Yicheng Jia <YJia@tmriusa.com>
wrote:<br>
> >>>><br>
> >>>> They both are POLLING before "reset".<br>
> >>><br>
> >>> Then they _should_ come back to INIT.<br>
> >>><br>
> >>> What does the local LDDS value say after reset ? Any
way to get the<br>
> >>> switch port LDDS value ?<br>
> >><br>
> >> Are you resetting the switch from the peer HCA port or some
other port<br>
> >> ? That's what Nicolas asked but I might have missed the answer.<br>
> >><br>
> >> Also, try disable (wait) and then enable and see if that
works. If I<br>
> >> recall correctly, you had those links which are taking a
long time to<br>
> >> initialize. If the link stays down forever after disable,
this won't<br>
> >> work but I want to be sure.<br>
> >><br>
> >> -- Hal<br>
> >><br>
> >>> -- Hal<br>
> >>><br>
> >>>> Thanks!<br>
> >>>> Yicheng Jia<br>
> >>>><br>
> >>>><br>
> >>>><br>
> >>>><br>
> >>>> Hal Rosenstock <hal.rosenstock@gmail.com><br>
> >>>><br>
> >>>> 04/16/2009 01:53 PM<br>
> >>>><br>
> >>>> To<br>
> >>>> Yicheng Jia <YJia@tmriusa.com><br>
> >>>> cc<br>
> >>>> Nicolas Morey-Chaisemartin <devel-ofed@morey-chaisemartin.com>,<br>
> >>>> general@lists.openfabrics.org<br>
> >>>> Subject<br>
> >>>> Re: [ofa-general] link width problem of Qlogic 9024
unmanaged switch<br>
> >>>><br>
> >>>><br>
> >>>><br>
> >>>><br>
> >>>> On Thu, Apr 16, 2009 at 11:12 AM, Yicheng Jia <YJia@tmriusa.com>
wrote:<br>
> >>>>><br>
> >>>>> Hi Nicolas,<br>
> >>>>><br>
> >>>>> After this "reset" command, both ports
are DOWN forever, I can only get<br>
> >>>>> portinfo from local port.<br>
> >>>>><br>
> >>>>> I am sure that the port that has been reset is
not the local port,<br>
> >>>>> otherwise<br>
> >>>>> it will prompt "node type not switch"
error.<br>
> >>>>><br>
> >>>>> I tried to enable this switch port from another
port and brought it to<br>
> >>>>> POLLING state, but as long as I use "reset",
both ports are DOWN.<br>
> >>>><br>
> >>>> What are the peer port's LinkDownDefaultStates ?
Sounds like one or<br>
> >>>> more must be Sleeping rather than Polling for some
reason.<br>
> >>>><br>
> >>>> -- Hal<br>
> >>>><br>
> >>>>> Thanks!<br>
> >>>>><br>
> >>>>> Yicheng Jia<br>
> >>>>><br>
> >>>>><br>
> >>>>><br>
> >>>>><br>
> >>>>> Nicolas Morey-Chaisemartin <devel-ofed@morey-chaisemartin.com><br>
> >>>>><br>
> >>>>> 04/16/2009 12:43 AM<br>
> >>>>><br>
> >>>>> To<br>
> >>>>> Yicheng Jia <YJia@tmriusa.com><br>
> >>>>> cc<br>
> >>>>> general@lists.openfabrics.org<br>
> >>>>> Subject<br>
> >>>>> Re: [ofa-general] link width problem of Qlogic
9024 unmanaged switch<br>
> >>>>><br>
> >>>>><br>
> >>>>><br>
> >>>>><br>
> >>>>> By any chances have you not reset the port you're
on?<br>
> >>>>> Have you tried using another node to enable the
port again?<br>
> >>>>><br>
> >>>>> Nicolas<br>
> >>>>><br>
> >>>>> Le 16/04/2009 00:45, Yicheng Jia a écrit :<br>
> >>>>>><br>
> >>>>>> Hello Randy,<br>
> >>>>>><br>
> >>>>>> I am trying to run "ibportstate reset"
to reset the switch port on the<br>
> >>>>>> other side in order to get 4x link. However
I get the following error:<br>
> >>>>>> ibwarn: [19660] mad_rpc: _do_madrpc failed;
dport (Lid 7)<br>
> >>>>>> ibportstate: iberror: failed: smp set portinfo
failed<br>
> >>>>>><br>
> >>>>>> And the port status change to DOWN after
this. Have you ever tried to<br>
> >>>>>> run "ibportstate" to reset the
switch port?<br>
> >>>>>><br>
> >>>>>> Thanks!<br>
> >>>>>><br>
> >>>>>> Yicheng Jia<br>
> >>>>>><br>
> >>>>>><br>
> >>>>>><br>
> >>>>>><br>
> >>>>>><br>
> >>>>>> ------------------------------<br>
> >>>>>><br>
> >>>>>> Message: 2<br>
> >>>>>> Date: Wed, 4 Mar 2009 18:39:54 -0600<br>
> >>>>>> From: Randy Halverson <randy.halverson@qlogic.com><br>
> >>>>>> Subject: [ofa-general] link width problem
of Qlogic 9024 unmanaged<br>
> >>>>>> switch<br>
> >>>>>> To: "'general@lists.openfabrics.org'"
<general@lists.openfabrics.org><br>
> >>>>>> Message-ID:<br>
> >>>>>> <88EC963376E93B4DB0F2A69D932F786903D89CAF@MNEXMB2.qlogic.org><br>
> >>>>>> Content-Type: text/plain; charset="us-ascii"<br>
> >>>>>><br>
> >>>>>> Hello Yicheng,<br>
> >>>>>><br>
> >>>>>> After checking internally, this appears to
be a known problem with<br>
> >>>>>> older<br>
> >>>>>> firmware for the 9024FC switches.<br>
> >>>>>><br>
> >>>>>> It appears that you or another person at
'tmriusa.com' has recently<br>
> >>>>>> opened a case with QLogic Tech Support for
this issue. Please continue<br>
> >>>>>> to work with QLogic Tech Support on firmware
upgrade resolution since<br>
> >>>>>> you probably don't have our FastFabric Tools
to manage the 9024FC<br>
> >>>>>> switches..<br>
> >>>>>><br>
> >>>>>> Regards,<br>
> >>>>>><br>
> >>>>>> Randy<br>
> >>>>>> Technical Support<br>
> >>>>>> QLogic Corporation<br>
> >>>>>> -------------- next part --------------<br>
> >>>>>><br>
> >>>>>><br>
> >>>>>><br>
> >>>>>><br>
> >>>>>><br>
> >>>>>><br>
> >>>>>> _____________________________________________________________________________<br>
> >>>>>> Scanned by IBM Email Security Management
Services powered by<br>
> >>>>>> MessageLabs. For more information please
visit http:// www. ers.ibm.com<br>
> >>>>>><br>
> >>>>>><br>
> >>>>>><br>
> >>>>>><br>
> >>>>>> _____________________________________________________________________________<br>
> >>>>>> <http:// www. ers.ibm.com/><br>
> >>>>>><br>
> >>>>>><br>
> >>>>>><br>
> >>>>>><br>
> >>>>>><br>
> >>>>>> _____________________________________________________________________________<br>
> >>>>>> Scanned by IBM Email Security Management
Services powered by<br>
> >>>>>> MessageLabs. For more information please
visit http:// www. ers.ibm.com<br>
> >>>>>><br>
> >>>>>><br>
> >>>>>><br>
> >>>>>><br>
> >>>>>> _____________________________________________________________________________<br>
> >>>>>><br>
> >>>>>><br>
> >>>>>><br>
> >>>>>><br>
> >>>>>> ------------------------------------------------------------------------<br>
> >>>>>><br>
> >>>>>> _______________________________________________<br>
> >>>>>> general mailing list<br>
> >>>>>> general@lists.openfabrics.org<br>
> >>>>>> http:// lists.openfabrics.org/cgi-bin/mailman/listinfo/general<br>
> >>>>>><br>
> >>>>>> To unsubscribe, please visit<br>
> >>>>>> http:// openib.org/mailman/listinfo/openib-general<br>
> >>>>><br>
> >>>>><br>
> >>>>><br>
> >>>>><br>
> >>>>><br>
> >>>>> _____________________________________________________________________________<br>
> >>>>> Scanned by IBM Email Security Management Services
powered by<br>
> >>>>> MessageLabs.<br>
> >>>>> For more information please visit http:// www.
ers.ibm.com<br>
> >>>>><br>
> >>>>><br>
> >>>>><br>
> >>>>> _____________________________________________________________________________<br>
> >>>>><br>
> >>>>><br>
> >>>>><br>
> >>>>><br>
> >>>>><br>
> >>>>> _____________________________________________________________________________<br>
> >>>>> Scanned by IBM Email Security Management Services
powered by<br>
> >>>>> MessageLabs.<br>
> >>>>> For more information please visit http:// www.
ers.ibm.com<br>
> >>>>><br>
> >>>>><br>
> >>>>><br>
> >>>>> _____________________________________________________________________________<br>
> >>>>><br>
> >>>>> _______________________________________________<br>
> >>>>> general mailing list<br>
> >>>>> general@lists.openfabrics.org<br>
> >>>>> http:// lists.openfabrics.org/cgi-bin/mailman/listinfo/general<br>
> >>>>><br>
> >>>>> To unsubscribe, please visit<br>
> >>>>> http:// openib.org/mailman/listinfo/openib-general<br>
> >>>>><br>
> >>>><br>
> >>>><br>
> >>>><br>
> >>>> _____________________________________________________________________________<br>
> >>>> Scanned by IBM Email Security Management Services
powered by<br>
> >>>> MessageLabs.<br>
> >>>> For more information please visit http:// www. ers.ibm.com<br>
> >>>><br>
> >>>><br>
> >>>> _____________________________________________________________________________<br>
> >>>><br>
> >>>><br>
> >>>><br>
> >>>><br>
> >>>> _____________________________________________________________________________<br>
> >>>> Scanned by IBM Email Security Management Services
powered by<br>
> >>>> MessageLabs.<br>
> >>>> For more information please visit http:// www. ers.ibm.com<br>
> >>>><br>
> >>>><br>
> >>>> _____________________________________________________________________________<br>
> >>>><br>
> >>><br>
> >><br>
> >><br>
> >> _____________________________________________________________________________<br>
> >> Scanned by IBM Email Security Management Services powered
by MessageLabs.<br>
> >> For more information please visit http:// www. ers.ibm.com<br>
> >><br>
> >> _____________________________________________________________________________<br>
> >><br>
> >><br>
> >><br>
> >> _____________________________________________________________________________<br>
> >> Scanned by IBM Email Security Management Services powered
by MessageLabs.<br>
> >> For more information please visit http:// www. ers.ibm.com<br>
> >><br>
> >> _____________________________________________________________________________<br>
> >><br>
> ><br>
> > _____________________________________________________________________________<br>
> > Scanned by IBM Email Security Management Services powered by
MessageLabs.<br>
> > For more information please visit http:// www. ers.ibm.com<br>
> > _____________________________________________________________________________<br>
> ><br>
> ><br>
> > _____________________________________________________________________________<br>
> > Scanned by IBM Email Security Management Services powered by
MessageLabs.<br>
> > For more information please visit http:// www. ers.ibm.com<br>
> > _____________________________________________________________________________<br>
> ><br>
> _______________________________________________<br>
> general mailing list<br>
> general@lists.openfabrics.org<br>
> http:// lists.openfabrics.org/cgi-bin/mailman/listinfo/general<br>
> <br>
> To unsubscribe, please visit http:// openib.org/mailman/listinfo/openib-general<br>
> <br>
<br>
<br>
-- <br>
Ira Weiny<br>
Math Programmer/Computer Scientist<br>
Lawrence Livermore National Lab<br>
weiny2@llnl.gov<br>
<br>
_____________________________________________________________________________<br>
Scanned by IBM Email Security Management Services powered by MessageLabs.
For more information please visit </font></tt><a href=http://www.ers.ibm.com/><tt><font size=2>http://www.ers.ibm.com<br>
_____________________________________________________________________________<br>
</font></tt></a>
<br>
<BR>
_____________________________________________________________________________<BR>
Scanned by IBM Email Security Management Services powered by MessageLabs. For more information please visit http://www.ers.ibm.com<BR>
_____________________________________________________________________________<BR>