<!DOCTYPE HTML PUBLIC "-//W3C//DTD HTML 4.0 Transitional//EN">
<HTML><HEAD>
<META http-equiv=Content-Type content="text/html; charset=us-ascii">
<META content="MSHTML 6.00.2900.3243" name=GENERATOR></HEAD>
<BODY>
<DIV><FONT face=Arial color=#0000ff size=2><SPAN class=923312618-25032009>>
<SPAN class=542434816-25032009><FONT face=Arial><FONT size=+0><FONT
color=#000000 size=2><SPAN class=599521018-25032009>BTW, will you be submitting
patches similar to mlx4 for the mthca driver in the near
future?</SPAN></FONT></FONT></FONT></SPAN></SPAN></FONT></DIV>
<DIV><FONT face=Arial color=#0000ff size=2><SPAN class=923312618-25032009><SPAN
class=542434816-25032009><FONT face=Arial><FONT size=+0><FONT color=#000000
size=2><SPAN class=599521018-25032009>Which ones do you mean
?</SPAN></FONT></FONT></FONT></SPAN></SPAN></FONT></DIV><BR>
<BLOCKQUOTE dir=ltr
style="PADDING-LEFT: 5px; MARGIN-LEFT: 5px; BORDER-LEFT: #0000ff 2px solid; MARGIN-RIGHT: 0px">
<DIV class=OutlookMessageHeader lang=en-us dir=ltr align=left>
<HR tabIndex=-1>
<FONT face=Tahoma size=2><B>From:</B> Smith, Stan
[mailto:stan.smith@intel.com] <BR><B>Sent:</B> Wednesday, March 25, 2009 8:17
PM<BR><B>To:</B> Leonid Keller; ofw@lists.openfabrics.org<BR><B>Subject:</B>
RE: [ofw][patch][ibal] make IBBUS start up synchronous.<BR></FONT><BR></DIV>
<DIV></DIV>
<DIV>
<HR tabIndex=-1>
<FONT face=Tahoma size=2><B>From:</B> Leonid Keller
[mailto:leonid@mellanox.co.il] <BR><B>Sent:</B> Wednesday, March 25, 2009
10:04 AM<BR><B>To:</B> Smith, Stan;
ofw@lists.openfabrics.org<BR><B>Subject:</B> RE: [ofw][patch][ibal] make IBBUS
start up synchronous.<BR></FONT><BR></DIV>
<DIV></DIV>
<DIV><SPAN class=542434816-25032009><FONT face=Arial color=#0000ff size=2>And
what timeout would you set ? 3 seconds ?</FONT></SPAN></DIV>
<DIV><SPAN class=542434816-25032009><FONT face=Arial><FONT color=#0000ff><FONT
size=2>How does one understand 3 seconds is too long or too short? :)<SPAN
class=599521018-25032009> </SPAN></FONT></FONT></FONT></SPAN></DIV>
<DIV><SPAN class=542434816-25032009><FONT face=Arial><FONT color=#0000ff><FONT
size=2><SPAN
class=599521018-25032009></SPAN></FONT></FONT></FONT></SPAN> </DIV>
<DIV><SPAN class=542434816-25032009><FONT face=Arial><FONT size=+0><FONT
size=2><SPAN class=599521018-25032009>I think the 3 seconds is based on
experience with a specific HCA
implementation.</SPAN></FONT></FONT></FONT></SPAN></DIV>
<DIV><SPAN class=542434816-25032009><FONT face=Arial><FONT size=+0><FONT
size=2><SPAN class=599521018-25032009>What if the next generation HCA or
another vendors HCA takes 5
seconds?</SPAN></FONT></FONT></FONT></SPAN></DIV>
<DIV><SPAN class=542434816-25032009><FONT face=Arial><FONT size=+0><FONT
size=2><SPAN class=599521018-25032009>You have far more experience in this
area than I, so I'm good with your
solution.</SPAN></FONT></FONT></FONT></SPAN></DIV>
<DIV><SPAN class=542434816-25032009><FONT face=Arial><FONT size=+0><FONT
size=2><SPAN class=599521018-25032009>BTW, will you be submitting patches
similar to mlx4 for the mthca driver in the near
future?</SPAN></FONT></FONT></FONT></SPAN></DIV>
<DIV><SPAN class=542434816-25032009><FONT face=Arial><FONT size=+0><FONT
size=2><SPAN
class=599521018-25032009></SPAN></FONT></FONT></FONT></SPAN> </DIV>
<DIV><SPAN class=542434816-25032009><FONT face=Arial><FONT size=+0><FONT
size=2><SPAN
class=599521018-25032009>stan.</SPAN></FONT></FONT></FONT></SPAN></DIV>
<DIV><SPAN class=542434816-25032009><FONT face=Arial><FONT size=+0><FONT
size=2><SPAN
class=599521018-25032009></SPAN></FONT></FONT></FONT></SPAN> </DIV>
<DIV><SPAN class=542434816-25032009><FONT face=Arial><FONT color=#0000ff><FONT
size=2><SPAN
class=599521018-25032009></SPAN></FONT></FONT></FONT></SPAN> </DIV>
<DIV><SPAN class=542434816-25032009><FONT face=Arial color=#0000ff size=2>The
expected time of PnP handling 300-400 msec, so i think, 3 seconds has to be
enough.</FONT></SPAN></DIV>
<DIV><SPAN class=542434816-25032009><FONT face=Arial color=#0000ff size=2>One
can change the fix, returning error, when after 3 seconds reference counter
was not decreased.</FONT></SPAN></DIV>
<DIV><SPAN class=542434816-25032009><FONT face=Arial color=#0000ff
size=2></FONT></SPAN> </DIV>
<DIV><SPAN class=542434816-25032009><FONT face=Arial color=#0000ff size=2>You
are right, one could reach the same goal using events.</FONT></SPAN></DIV>
<DIV><SPAN class=542434816-25032009><FONT face=Arial color=#0000ff size=2>For
example, add an event to CI_CA object, which is a part of the event structure,
and post the event at the end of the callback
(__pnp_process_add_ca).</FONT></SPAN></DIV>
<DIV><SPAN class=542434816-25032009><FONT face=Arial color=#0000ff
size=2></FONT></SPAN> </DIV><BR>
<BLOCKQUOTE dir=ltr
style="PADDING-LEFT: 5px; MARGIN-LEFT: 5px; BORDER-LEFT: #0000ff 2px solid; MARGIN-RIGHT: 0px">
<DIV class=OutlookMessageHeader lang=en-us dir=ltr align=left>
<HR tabIndex=-1>
<FONT face=Tahoma size=2><B>From:</B> Smith, Stan
[mailto:stan.smith@intel.com] <BR><B>Sent:</B> Wednesday, March 25, 2009
6:23 PM<BR><B>To:</B> Leonid Keller;
ofw@lists.openfabrics.org<BR><B>Subject:</B> RE: [ofw][patch][ibal] make
IBBUS start up synchronous.<BR></FONT><BR></DIV>
<DIV></DIV>
<DIV><SPAN class=850291116-25032009><FONT face=Arial color=#0000ff
size=2>Hello Leonid,</FONT></SPAN></DIV>
<DIV><SPAN class=850291116-25032009><FONT face=Arial color=#0000ff
size=2> A small question. If the ADD_CA event does indeed signal
completion of HCA startup, why not have fdo_start() wait on an HCA guid
specific event (time-out enabled escape hatch) where the ADD_CA
event callback would then trigger the HCA guid specific event thus
completing fdo_start()?</FONT></SPAN></DIV>
<DIV><SPAN class=850291116-25032009><FONT face=Arial color=#0000ff size=2>If
the event wait times out, then fdo_start() returns an
error.</FONT></SPAN></DIV>
<DIV><SPAN class=850291116-25032009><FONT face=Arial color=#0000ff
size=2></FONT></SPAN> </DIV>
<DIV><SPAN class=850291116-25032009><FONT face=Arial color=#0000ff
size=2>Bottom-line: over the years I have found waiting in spin loops to be
problematic. How does one understand 3 seconds is too long or too
short?</FONT></SPAN></DIV>
<DIV><SPAN class=850291116-25032009><FONT face=Arial color=#0000ff
size=2></FONT></SPAN> </DIV>
<DIV><SPAN class=850291116-25032009><FONT face=Arial color=#0000ff
size=2>What am I not understanding about this synchronization
problem?</FONT></SPAN></DIV>
<DIV><SPAN class=850291116-25032009><FONT face=Arial color=#0000ff
size=2></FONT></SPAN> </DIV>
<DIV><SPAN class=850291116-25032009><FONT face=Arial color=#0000ff
size=2>thanks,</FONT></SPAN></DIV>
<DIV><SPAN class=850291116-25032009><FONT face=Arial color=#0000ff
size=2></FONT></SPAN> </DIV>
<DIV><SPAN class=850291116-25032009><FONT face=Arial color=#0000ff
size=2>stan.</FONT></SPAN></DIV>
<DIV><SPAN class=850291116-25032009><FONT face=Arial color=#0000ff
size=2></FONT></SPAN> </DIV>
<DIV><SPAN class=850291116-25032009><FONT face=Arial color=#0000ff
size=2></FONT></SPAN> </DIV><BR>
<DIV class=OutlookMessageHeader lang=en-us dir=ltr align=left>
<HR tabIndex=-1>
<FONT face=Tahoma size=2><B>From:</B> ofw-bounces@lists.openfabrics.org
[mailto:ofw-bounces@lists.openfabrics.org] <B>On Behalf Of </B>Leonid
Keller<BR><B>Sent:</B> Wednesday, March 25, 2009 7:02 AM<BR><B>To:</B>
ofw@lists.openfabrics.org<BR><B>Subject:</B> [ofw][patch][ibal] make IBBUS
start up synchronous.<BR></FONT><BR></DIV>
<DIV></DIV>
<DIV><FONT face=Arial size=2><SPAN class=114255013-25032009>IBAL start up
has by design two stages.</SPAN></FONT></DIV>
<DIV><FONT face=Arial size=2><SPAN class=114255013-25032009> -
initialization and preparing for getting IBAL PnP events (synchronous on
fdo_start);</SPAN></FONT></DIV>
<DIV><FONT face=Arial size=2><SPAN class=114255013-25032009> -
completing the start up upon receiving ADD_CA event (asynchronous, after
fdo_start).</SPAN></FONT></DIV>
<DIV><FONT face=Arial size=2><SPAN
class=114255013-25032009></SPAN></FONT> </DIV>
<DIV><FONT face=Arial size=2><SPAN class=114255013-25032009>It means, that
IBBUS driver notifies the completion of the driver start operation before it
really has finished it.</SPAN></FONT></DIV>
<DIV><FONT face=Arial size=2><SPAN class=114255013-25032009>It causes races
in quick HCA disable/enable sequences.</SPAN></FONT></DIV>
<DIV><FONT face=Arial size=2><SPAN
class=114255013-25032009></SPAN></FONT> </DIV>
<DIV><FONT face=Arial size=2><SPAN class=114255013-25032009>This patch makes
IBBUS start up synchronous by adding a wait in the first stage till the end
of the second stage.</SPAN></FONT></DIV>
<DIV><FONT face=Arial size=2><SPAN
class=114255013-25032009></SPAN></FONT> </DIV>
<DIV><FONT face=Arial size=2></FONT> </DIV>
<DIV><FONT face=Arial size=2>Index:
core/al/kernel/al_pnp.c<BR>===================================================================<BR>---
core/al/kernel/al_pnp.c (revision 2055)<BR>+++
core/al/kernel/al_pnp.c (working copy)<BR>@@ -1152,6 +1152,7
@@<BR> cl_status_t cl_status;<BR> al_pnp_ca_event_t *p_event_rec;<BR> ib_ca_attr_t *p_old_ca_attr;<BR>+ uint32_t ref_cnt,
prev_cnt, i;<BR> <BR> AL_ENTER( AL_DBG_PNP
);<BR> <BR>@@ -1232,9 +1233,18
@@<BR> }<BR> <BR> /* Queue the event to the async
processing manager. */<BR>- ref_al_obj( &gp_pnp->obj
);<BR>+ ref_cnt = ref_al_obj( &gp_pnp->obj
);<BR> cl_async_proc_queue( gp_async_pnp_mgr,
&p_event_rec->async_item );<BR> <BR>+ /* wait up to 3
seconds for the end of event propagation <BR>+ It is needed for
enabling quick HCA disable/enable scenarios. */<BR>+ prev_cnt = ref_cnt
- 1;<BR>+ for ( i = 0; i < 30; ++i ) {<BR>+ if (ref_cnt
<=
prev_cnt)<BR>+ break;<BR>+ cl_thread_suspend(100);<BR>+ }<BR>+<BR> AL_EXIT(
AL_DBG_PNP );<BR> return
IB_SUCCESS;<BR> }<BR></FONT></DIV></BLOCKQUOTE></BLOCKQUOTE></BODY></HTML>