<font size=2 face="sans-serif">Hal,</font>
<br>
<br><font size=2 face="sans-serif">Thank you very much for the support,
I am the same person from the gmail account so I will respond through here.</font>
<br>
<br><font size=2 face="sans-serif">Attached is a picture of the switch
serial number:</font>
<br>
<p>
<br>
<br><font size=2 face="sans-serif">I am indeed using OFED 1.5.4-rc3. My
experiment consists of a 7 server system which I reboot via a script over
and over again. Technically speaking the switch is not being powered off
or physically rebooted. My server system is what is being rebooted. I am
running OpenSM on one of the 7 servers. This means I'm constantly shutting
down and rebooting OpenSM. I am running OpenSM on QNX but we have not had
this problem until we decided to upgrade to this switch.</font>
<br>
<br><font size=2 face="sans-serif">The problem is that every 1 out of 15
of this remote reboots OpenSM stalls or times out because stats->qp0_mads_outstanding
did not reach zero. Please excuse my ignorance as I'm relatively new at
this but how do I verify if it is a timeout problem vs a stall?</font>
<br>
<br><font size=2 face="sans-serif">You also mentioned that you'd like to
see the Verbose output of openSM; however, when I run in Verbose mode I
don't see the problem. It appears as if the verbose output stalls enough
time to give the switch time to do what ever it needs to do and hence not
have the problem occur. But this is the last I see when the problem occurs:</font>
<br>
<br>
<br>
<br><font size=2 face="sans-serif">-------------------------------------------------</font>
<br><font size=2 face="sans-serif">OpenSM 3.3.12</font>
<br><font size=2 face="sans-serif">Command Line Arguments:</font>
<br><font size=2 face="sans-serif"> Log file max size is 5 MBytes</font>
<br><font size=2 face="sans-serif"> Log File: /tmp/opensm.log</font>
<br><font size=2 face="sans-serif">-------------------------------------------------</font>
<br><font size=2 face="sans-serif">OpenSM 3.3.12</font>
<br>
<br><font size=2 face="sans-serif">Entering DISCOVERING state</font>
<br>
<br><font size=2 face="sans-serif">Using default GUID 0x2c9020023277d</font>
<br>
<br>
<br>
<br><font size=2 face="sans-serif">The problem occurs in function osm_vl15intf.c
-> vl15_poller in the else statement.</font>
<br>
<br><font size=2 face="sans-serif">if (p_madw != (osm_madw_t *) cl_qlist_end(p_fifo))
{</font>
<br><font size=2 face="sans-serif"> OSM_LOG(p_vl->p_log,
OSM_LOG_DEBUG,</font>
<br><font size=2 face="sans-serif"> "Servicing
p_madw = %p\n", p_madw);</font>
<br><font size=2 face="sans-serif"> if
(osm_log_is_active(p_vl->p_log, OSM_LOG_FRAMES))</font>
<br><font size=2 face="sans-serif"> osm_dump_dr_smp(p_vl->p_log,</font>
<br><font size=2 face="sans-serif"> osm_madw_get_smp_ptr(p_madw),</font>
<br><font size=2 face="sans-serif"> OSM_LOG_FRAMES);</font>
<br>
<br><font size=2 face="sans-serif"> vl15_send_mad(p_vl,
p_madw);</font>
<br><font size=2 face="sans-serif">} else</font>
<br><font size=2 face="sans-serif"> /*</font>
<br><font size=2 face="sans-serif">
The VL15 FIFO is empty, so we have nothing left to do.</font>
<br><font size=2 face="sans-serif"> */</font>
<br><font size=2 face="sans-serif"> status
= cl_event_wait_on(&p_vl->signal,</font>
<br><font size=2 face="sans-serif">
EVENT_NO_TIMEOUT, TRUE);</font>
<br>
<br><font size=2 face="sans-serif">It won't move forward from the cl_event_wait_on
in this line of code. However, there are other locations such as wait_for_pending_transactions
in the do_sweep function that won't move forward from. But I believe this
to be a side effect of the problem I'm mentioning.</font>
<br>
<br><font size=2 face="sans-serif">When you mention what is my timeout,
I'm guessing you refer to max_smps_timeout which is used in the second
while loop within vl15_poller? For this setting I am using the default
which is defined in osm_subnet.c as:</font>
<br>
<br><font size=2 face="sans-serif">p_opt->transaction_timeout = OSM_DEFAULT_TRANS_TIMEOUT_MILLISEC;</font>
<br><font size=2 face="sans-serif"> p_opt->transaction_retries
= OSM_DEFAULT_RETRY_COUNT;</font>
<br><font size=2 face="sans-serif"> p_opt->max_smps_timeout
= 1000 * p_opt->transaction_timeout *p_opt->transaction_retries;</font>
<br>
<br><font size=2 face="sans-serif">Would you explain to me what are the
advantages or disadvantages of OSM_DEFAULT_SMP_MAX_ON_WIRE? Does this parameter
change my bandwidth performance at all?</font>
<br>
<br><font size=2 face="sans-serif">I noticed that when using the default
setting of 4 I get into the else of the above if statement when there are
4 qp0_mads_outstanding. I noticed that if I change OSM_DEFAULT_SMP_MAX_ON_WIRE
to 1 I don't get the failure I'm mentioning at all. Partly (I think) because
I don't enter the else in the if statement until there is 1 qp0_mads_outstanding.</font>
<br>
<br><font size=2 face="sans-serif">I hope this explains the problem well
enough and it may be a time out problem but I'd like to understand why
the problem is occurring.</font>
<br><font size=2 face="sans-serif">Thank you very much,</font>
<br>
<br><font size=2 face="sans-serif">Hector Abrach</font>
<br>
<br>
<br>
<table width=100%>
<tr valign=top>
<td><font size=1 color=#5f5f5f face="sans-serif">From:</font>
<td><font size=1 face="sans-serif">Hal Rosenstock <hal@dev.mellanox.co.il></font>
<tr valign=top>
<td><font size=1 color=#5f5f5f face="sans-serif">To:</font>
<td><font size=1 face="sans-serif">Hector Abrach <HAbrach@TMRIUSA.COM></font>
<tr>
<td valign=top><font size=1 color=#5f5f5f face="sans-serif">Cc:</font>
<td><font size=1 face="sans-serif">ewg@lists.openfabrics.org</font>
<tr valign=top>
<td><font size=1 color=#5f5f5f face="sans-serif">Date:</font>
<td><font size=1 face="sans-serif">12/14/2011 08:03 AM</font>
<tr valign=top>
<td><font size=1 color=#5f5f5f face="sans-serif">Subject:</font>
<td><font size=1 face="sans-serif">Re: [ewg] OpenSM 1.5.4 Boot Problem</font></table>
<br>
<hr noshade>
<br>
<br>
<br><tt><font size=2>Hi,<br>
<br>
On 12/13/2011 2:35 PM, Hector Abrach wrote:<br>
> Hello,<br>
> <br>
> I have a boot problem with OpenSM<br>
<br>
Are you saying the switch is booted rather than OpenSM ?<br>
<br>
What is the OpenSM running on and in what environment ?<br>
<br>
> the problem occurs seldomly and<br>
> started to ocur when we started using a new Mellanox MT1118X03342
switch.<br>
> The problem occurs during the discovery phase within state_mgr_sweep_hop_1.<br>
> <br>
> However, I discovered that the actual location is because the<br>
> qp0_mads_outsanding stalls at 1 occasionally.<br>
<br>
Is it stuck or after timeout/retry does this get updated properly ?<br>
<br>
> Within file osm_vl15intf.c in function vl15_poller it checks at the<br>
> rfifo and if the qlist still has items it applies function vl15_send_mad<br>
> which later on triggers the signal.<br>
> With the current default setting of 4 for OSM_DEFAULT_SMP_MAX_ON_WIRE
I<br>
> noticed that cl_qlist_end reaches zero before<br>
> stats->qp0_mads_outstanding does. This causes a stall in<br>
> cl_event_wait_on. The rfifo always reaches 0 when there are 4<br>
> qp0_mads_outstanding however when it fails it always fails when there
is<br>
> 1 qp0_mad_outstanding.<br>
<br>
Is some (request) SMP that OpenSM sent timing out (not being responded
to) ?<br>
<br>
> Have you seen this failure? By the way, I see this failure once every
15<br>
> reboots approximately.<br>
> <br>
> I discovered that changing OSM_DEFAULT_SMP_MAX_ON_WIRE to 1 fixes
the<br>
> problem.<br>
<br>
What do you mean exactly by fixes the problem ? I'm not sure I<br>
understand what the problem is yet.<br>
<br>
-- Hal<br>
<br>
> My guess is that there is a race condition when the switch sends 4
SMPs<br>
> in parallel. Also, this failure only appears to occur at reboot. Another<br>
> solution which is not acceptable is when I add a delay in the process<br>
> the failure goes away. This as if the switch needed more time to do<br>
> something.<br>
> <br>
> I would really appreciate your help and insight.<br>
> Thank you<br>
> <br>
> Hector Abrach<br>
> ______________________________________________________________________<br>
> This email has been scanned by the Symantec Email Security.cloud service.<br>
> For more information please visit </font></tt><a href=http://www.symanteccloud.com/><tt><font size=2>http://www.symanteccloud.com</font></tt></a><tt><font size=2><br>
> ______________________________________________________________________<br>
> <br>
> <br>
> _______________________________________________<br>
> ewg mailing list<br>
> ewg@lists.openfabrics.org<br>
> </font></tt><a href="http://lists.openfabrics.org/cgi-bin/mailman/listinfo/ewg"><tt><font size=2>http://lists.openfabrics.org/cgi-bin/mailman/listinfo/ewg</font></tt></a><tt><font size=2><br>
<br>
<br>
______________________________________________________________________<br>
This email has been scanned by the Symantec Email Security.cloud service.<br>
For more information please visit </font></tt><a href=http://www.symanteccloud.com/><tt><font size=2>http://www.symanteccloud.com</font></tt></a><tt><font size=2><br>
______________________________________________________________________<br>
</font></tt>
<br>
<br clear="both">
______________________________________________________________________<BR>
This email has been scanned by the Symantec Email Security.cloud service.<BR>
For more information please visit http://www.symanteccloud.com<BR>
______________________________________________________________________<BR>