<!DOCTYPE HTML PUBLIC "-//W3C//DTD HTML 4.0 Transitional//EN">
<HTML xmlns="http://www.w3.org/TR/REC-html40" XMLNS:Z =
"urn:schemas-microsoft-com:" xmlns:ex12m =
"http://schemas.microsoft.com/exchange/services/2006/messages" xmlns:ex12t =
"http://schemas.microsoft.com/exchange/services/2006/types" xmlns:mrels =
"http://schemas.openxmlformats.org/package/2006/relationships" xmlns:m =
"http://schemas.microsoft.com/office/2004/12/omml" xmlns:mver =
"http://schemas.openxmlformats.org/markup-compatibility/2006" xmlns:wf =
"http://schemas.microsoft.com/sharepoint/soap/workflow/" xmlns:udcxf =
"http://schemas.microsoft.com/data/udc/xmlfile" xmlns:xsi =
"http://www.w3.org/2001/XMLSchema-instance" xmlns:sps =
"http://schemas.microsoft.com/sharepoint/soap/" xmlns:sp =
"http://schemas.microsoft.com/sharepoint/" xmlns:ec =
"http://www.w3.org/2001/04/xmlenc#" xmlns:sub =
"http://schemas.microsoft.com/sharepoint/soap/2002/1/alerts/" xmlns:xsd =
"http://www.w3.org/2001/XMLSchema" xmlns:udc =
"http://schemas.microsoft.com/data/udc" xmlns:dsp =
"http://schemas.microsoft.com/sharepoint/dsp" xmlns:ds =
"http://www.w3.org/2000/09/xmldsig#" xmlns:dir =
"http://schemas.microsoft.com/sharepoint/soap/directory/" xmlns:ois =
"http://schemas.microsoft.com/sharepoint/soap/ois/" xmlns:x2 =
"http://schemas.microsoft.com/office/excel/2003/xml" XMLNS:D = "DAV:" xmlns:q =
"http://schemas.xmlsoap.org/soap/envelope/" xmlns:html =
"http://www.w3.org/TR/REC-html40" xmlns:oa =
"urn:schemas-microsoft-com:office:activation" xmlns:c =
"urn:schemas-microsoft-com:office:component:spreadsheet" xmlns:ss =
"urn:schemas-microsoft-com:office:spreadsheet" xmlns:b =
"urn:schemas-microsoft-com:office:publisher" xmlns:z = "#RowsetSchema" xmlns:rs
= "urn:schemas-microsoft-com:rowset" xmlns:s =
"uuid:BDC6E3F0-6DA3-11d1-A2A3-00AA00C14882" xmlns:dt =
"uuid:C2F41010-65B3-11d1-A29F-00AA00C14882" xmlns:a =
"urn:schemas-microsoft-com:office:access" xmlns:p =
"urn:schemas-microsoft-com:office:powerpoint" xmlns:x =
"urn:schemas-microsoft-com:office:excel" xmlns:w =
"urn:schemas-microsoft-com:office:word" xmlns:o =
"urn:schemas-microsoft-com:office:office" xmlns:v =
"urn:schemas-microsoft-com:vml"><HEAD><!--[if !mso]>
<META http-equiv=Content-Type content="text/html; charset=us-ascii">
<STYLE>v\:* {
BEHAVIOR: url(#default#VML)
}
o\:* {
BEHAVIOR: url(#default#VML)
}
w\:* {
BEHAVIOR: url(#default#VML)
}
.shape {
BEHAVIOR: url(#default#VML)
}
</STYLE>
<![endif]-->
<STYLE>@font-face {
font-family: Cambria Math;
}
@font-face {
font-family: Calibri;
}
@font-face {
font-family: Tahoma;
}
@font-face {
font-family: Courier New (Hebrew);
}
@page Section1 {size: 8.5in 11.0in; margin: 1.0in 1.0in 1.0in 1.0in; }
P.MsoNormal {
FONT-SIZE: 12pt; MARGIN: 0in 0in 0pt; FONT-FAMILY: "Times New Roman","serif"
}
LI.MsoNormal {
FONT-SIZE: 12pt; MARGIN: 0in 0in 0pt; FONT-FAMILY: "Times New Roman","serif"
}
DIV.MsoNormal {
FONT-SIZE: 12pt; MARGIN: 0in 0in 0pt; FONT-FAMILY: "Times New Roman","serif"
}
A:link {
COLOR: blue; TEXT-DECORATION: underline; mso-style-priority: 99
}
SPAN.MsoHyperlink {
COLOR: blue; TEXT-DECORATION: underline; mso-style-priority: 99
}
A:visited {
COLOR: purple; TEXT-DECORATION: underline; mso-style-priority: 99
}
SPAN.MsoHyperlinkFollowed {
COLOR: purple; TEXT-DECORATION: underline; mso-style-priority: 99
}
P {
FONT-SIZE: 12pt; MARGIN-LEFT: 0in; MARGIN-RIGHT: 0in; FONT-FAMILY: "Times New Roman","serif"; mso-style-priority: 99; mso-margin-top-alt: auto; mso-margin-bottom-alt: auto
}
SPAN.EmailStyle18 {
COLOR: #1f497d; FONT-FAMILY: "Calibri","sans-serif"; mso-style-type: personal-reply
}
.MsoChpDefault {
FONT-SIZE: 10pt; mso-style-type: export-only
}
DIV.Section1 {
page: Section1
}
</STYLE>
<!--[if gte mso 9]><xml>
<o:shapedefaults v:ext="edit" spidmax="1026" />
</xml><![endif]--><!--[if gte mso 9]><xml>
<o:shapelayout v:ext="edit">
<o:idmap v:ext="edit" data="1" />
</o:shapelayout></xml><![endif]-->
<META content="MSHTML 6.00.6000.16587" name=GENERATOR></HEAD>
<BODY lang=EN-US vLink=purple link=blue>
<DIV><SPAN class=847111007-29062008><FONT face=Arial color=#0000ff size=2>Hi
Fab,</FONT></SPAN></DIV>
<DIV><SPAN class=847111007-29062008><FONT face=Arial color=#0000ff
size=2></FONT></SPAN> </DIV>
<DIV><SPAN class=847111007-29062008><FONT face=Arial color=#0000ff size=2>Please
note that the general problem that exists here is harder.</FONT></SPAN></DIV>
<DIV><SPAN class=847111007-29062008><FONT face=Arial color=#0000ff size=2>When
running in a DPC, there are two main problems: 1) other DPCs won't run 2)
user mode threads that were running when the DPC has started will not get time
to execute.</FONT></SPAN></DIV>
<DIV><SPAN class=847111007-29062008><FONT face=Arial color=#0000ff
size=2></FONT></SPAN> </DIV>
<DIV><SPAN class=847111007-29062008><FONT face=Arial color=#0000ff size=2>The
first problem blocks the computer from responding to other hardware events. For
example this can mean that the mouse DPC is not getting it's time to
run or that a different NIC is not running. This problem can be solved by
stopping your DPC and queuing another DPC.</FONT></SPAN></DIV>
<DIV><SPAN class=847111007-29062008><FONT face=Arial color=#0000ff
size=2></FONT></SPAN> </DIV>
<DIV><SPAN class=847111007-29062008><FONT face=Arial color=#0000ff size=2>The
second problem is somewhat harder to understand, we have seen this happening
with a tcp benchmark that didn't reach wire speed. While analyzing this issue we
have reached to a conclusion that the application can not post recv buffer since
the thread that was posting the recive was stopped by a DPC that was bringing
more packets. Please also note that this machine had 8 cores, so the application
did have a place to run.</FONT></SPAN></DIV>
<DIV><SPAN class=847111007-29062008><FONT face=Arial color=#0000ff
size=2></FONT></SPAN> </DIV>
<DIV><SPAN class=847111007-29062008><FONT face=Arial color=#0000ff size=2>In any
case, in the following days I'll make a patch to the mlx4 driver that will allow
the different user to crate their own EQ and will give them a much better
control of what they are doing.</FONT></SPAN></DIV>
<DIV><SPAN class=847111007-29062008><FONT face=Arial color=#0000ff
size=2></FONT></SPAN> </DIV>
<DIV><SPAN class=847111007-29062008><FONT face=Arial color=#0000ff size=2>I also
agree that all ULPS should make sure not to stay in DPC for a long time. I
believe that IPOIB already does so.</FONT></SPAN></DIV>
<DIV><SPAN class=847111007-29062008><FONT face=Arial color=#0000ff
size=2></FONT></SPAN> </DIV>
<DIV><SPAN class=847111007-29062008><FONT face=Arial color=#0000ff
size=2>Thanks</FONT></SPAN></DIV>
<DIV><SPAN class=847111007-29062008><FONT face=Arial color=#0000ff
size=2>Tzachi</FONT></SPAN></DIV><BR>
<BLOCKQUOTE dir=ltr
style="PADDING-LEFT: 5px; MARGIN-LEFT: 5px; BORDER-LEFT: #0000ff 2px solid; MARGIN-RIGHT: 0px">
<DIV class=OutlookMessageHeader lang=en-us dir=ltr align=left>
<HR tabIndex=-1>
<FONT face=Tahoma size=2><B>From:</B> Fab Tillier
[mailto:ftillier@windows.microsoft.com] <BR><B>Sent:</B> Thursday, June 26,
2008 7:35 PM<BR><B>To:</B> Alex Naslednikov; ofw@lists.openfabrics.org; Tzachi
Dar<BR><B>Subject:</B> RE: Bugfix for IPOIB failure<BR></FONT><BR></DIV>
<DIV></DIV>
<DIV class=Section1>
<P class=MsoNormal><SPAN style="FONT-SIZE: 11pt; FONT-FAMILY: " color:#1F497D?
Calibri?,?sans-serif?;>Note that all kernel ULPs should do something along
these lines – SRP, IPoIB, VNIC, as well as internal IBAL services—should all
limit how much time they spend processing completions and requeue their DPC if
they exceed it.<o:p></o:p></SPAN></P>
<P class=MsoNormal><SPAN style="FONT-SIZE: 11pt; FONT-FAMILY: " color:#1F497D?
Calibri?,?sans-serif?;><o:p> </o:p></SPAN></P>
<P class=MsoNormal><SPAN style="FONT-SIZE: 11pt; FONT-FAMILY: " color:#1F497D?
Calibri?,?sans-serif?;>The natural progression of this idea is that a ULP
would provide the DPC object to queue in response to a CQ event, rather than
callbacks.<o:p></o:p></SPAN></P>
<P class=MsoNormal><SPAN style="FONT-SIZE: 11pt; FONT-FAMILY: " color:#1F497D?
Calibri?,?sans-serif?;><o:p> </o:p></SPAN></P>
<P class=MsoNormal><SPAN style="FONT-SIZE: 11pt; FONT-FAMILY: " color:#1F497D?
Calibri?,?sans-serif?;>-Fab<o:p></o:p></SPAN></P>
<P class=MsoNormal><SPAN style="FONT-SIZE: 11pt; FONT-FAMILY: " color:#1F497D?
Calibri?,?sans-serif?;><o:p> </o:p></SPAN></P>
<DIV
style="BORDER-RIGHT: medium none; PADDING-RIGHT: 0in; BORDER-TOP: medium none; PADDING-LEFT: 4pt; PADDING-BOTTOM: 0in; BORDER-LEFT: blue 1.5pt solid; PADDING-TOP: 0in; BORDER-BOTTOM: medium none">
<DIV>
<DIV
style="BORDER-RIGHT: medium none; PADDING-RIGHT: 0in; BORDER-TOP: #b5c4df 1pt solid; PADDING-LEFT: 0in; PADDING-BOTTOM: 0in; BORDER-LEFT: medium none; PADDING-TOP: 3pt; BORDER-BOTTOM: medium none">
<P class=MsoNormal><B><SPAN style="FONT-SIZE: 10pt; FONT-FAMILY: "
Tahoma?,?sans-serif??>From:</SPAN></B><SPAN
style="FONT-SIZE: 10pt; FONT-FAMILY: " Tahoma?,?sans-serif??>
ofw-bounces@lists.openfabrics.org [mailto:ofw-bounces@lists.openfabrics.org]
<B>On Behalf Of </B>Alex Naslednikov<BR><B>Sent:</B> Thursday, June 26, 2008
5:01 AM<BR><B>To:</B> ofw@lists.openfabrics.org; Tzachi Dar<BR><B>Subject:</B>
[ofw] Bugfix for IPOIB failure<o:p></o:p></SPAN></P></DIV></DIV>
<P class=MsoNormal><o:p> </o:p></P>
<DIV>
<P class=MsoNormal> <o:p></o:p></P></DIV>
<P class=MsoNormal><o:p> </o:p></P>
<DIV class=MsoNormal style="TEXT-ALIGN: center" align=center>
<HR align=center width="100%" SIZE=2>
</DIV>
<DIV>
<P><SPAN style="FONT-SIZE: 10pt; FONT-FAMILY: " Arial?,?sans-serif??>Hi
all,</SPAN><SPAN style="FONT-FAMILY: "
Arial?,?sans-serif??><o:p></o:p></SPAN></P>
<P><SPAN style="FONT-SIZE: 10pt; FONT-FAMILY: " Arial?,?sans-serif??>The
original problem was IPoIB failure (link up/down) during the operation
of 'heavy' applications.</SPAN><SPAN style="FONT-FAMILY: "
Arial?,?sans-serif??><o:p></o:p></SPAN></P>
<DIV>
<P class=MsoNormal><SPAN style="FONT-SIZE: 10pt; FONT-FAMILY: "
Arial?,?sans-serif??>After our investigation, we found when one execute
application with heavy load on the HCA, driver always have non-empty CQs, so
it continues to work with its DPCs and not allows to other DPC to be
performed.</SPAN><SPAN style="FONT-FAMILY: "
Arial?,?sans-serif??><o:p></o:p></SPAN></P></DIV>
<DIV>
<P class=MsoNormal><SPAN style="FONT-FAMILY: "
Arial?,?sans-serif??> <o:p></o:p></SPAN></P></DIV>
<DIV>
<P class=MsoNormal><SPAN style="FONT-SIZE: 10pt; FONT-FAMILY: "
Arial?,?sans-serif??>The solution</SPAN><SPAN style="FONT-FAMILY: "
Arial?,?sans-serif??><o:p></o:p></SPAN></P></DIV>
<DIV>
<P class=MsoNormal><SPAN style="FONT-SIZE: 10pt; FONT-FAMILY: "
Arial?,?sans-serif??>Based on an anologous solution for mthca driver, driver
has to enable other than its own DPC to be performed. It's possible to count
certain amount of time that driver spent on DPC handling, than exit, thus
allowing other DPC to run :</SPAN><SPAN style="FONT-FAMILY: "
Arial?,?sans-serif??><o:p></o:p></SPAN></P></DIV>
<P><SPAN style="FONT-FAMILY: "
Arial?,?sans-serif??> <o:p></o:p></SPAN></P>
<P><SPAN style="FONT-FAMILY: "
Arial?,?sans-serif??> <o:p></o:p></SPAN></P>
<P><SPAN style="FONT-SIZE: 10pt; FONT-FAMILY: " Arial?,?sans-serif??>Index:
hw/mlx4/kernel/bus/net/eq.c</SPAN><SPAN style="FONT-FAMILY: "
Arial?,?sans-serif??><o:p></o:p></SPAN></P>
<P><SPAN style="FONT-SIZE: 10pt; FONT-FAMILY: " (Hebrew)?,?serif?? New
Courier>===================================================================</SPAN><SPAN
style="FONT-FAMILY: " (Hebrew)?,?serif?? New Courier><o:p></o:p></SPAN></P>
<P><SPAN style="FONT-SIZE: 10pt; FONT-FAMILY: " (Hebrew)?,?serif?? New
Courier>--- </SPAN><SPAN
style="FONT-SIZE: 10pt">hw/mlx4/kernel/bus/net/eq.c</SPAN><SPAN
style="FONT-SIZE: 10pt; FONT-FAMILY: " (Hebrew)?,?serif?? New Courier>
(</SPAN><SPAN style="FONT-SIZE: 10pt">revision 2634</SPAN><SPAN
style="FONT-SIZE: 10pt; FONT-FAMILY: " (Hebrew)?,?serif?? New
Courier>)</SPAN><SPAN style="FONT-FAMILY: " (Hebrew)?,?serif?? New
Courier><o:p></o:p></SPAN></P>
<P><SPAN style="FONT-SIZE: 10pt; FONT-FAMILY: " (Hebrew)?,?serif?? New
Courier>+++ </SPAN><SPAN
style="FONT-SIZE: 10pt">hw/mlx4/kernel/bus/net/eq.c</SPAN><SPAN
style="FONT-SIZE: 10pt; FONT-FAMILY: " (Hebrew)?,?serif?? New Courier>
(</SPAN><SPAN style="FONT-SIZE: 10pt">revision 2635</SPAN><SPAN
style="FONT-SIZE: 10pt; FONT-FAMILY: " (Hebrew)?,?serif?? New
Courier>)</SPAN><SPAN style="FONT-FAMILY: " (Hebrew)?,?serif?? New
Courier><o:p></o:p></SPAN></P>
<P><SPAN style="FONT-SIZE: 10pt; FONT-FAMILY: " (Hebrew)?,?serif?? New
Courier>@@ -160,6 +160,9 @@</SPAN><SPAN style="FONT-FAMILY: "
(Hebrew)?,?serif?? New Courier><o:p></o:p></SPAN></P>
<P><SPAN style="FONT-SIZE: 10pt">int cqn</SPAN><SPAN
style="FONT-SIZE: 10pt; FONT-FAMILY: " (Hebrew)?,?serif?? New
Courier>;</SPAN><SPAN style="FONT-FAMILY: " (Hebrew)?,?serif?? New
Courier><o:p></o:p></SPAN></P>
<P><SPAN style="FONT-SIZE: 10pt">int eqes_found = 0</SPAN><SPAN
style="FONT-SIZE: 10pt; FONT-FAMILY: " (Hebrew)?,?serif?? New
Courier>;</SPAN><SPAN style="FONT-FAMILY: " (Hebrew)?,?serif?? New
Courier><o:p></o:p></SPAN></P>
<P><SPAN style="FONT-SIZE: 10pt">int set_ci = 0</SPAN><SPAN
style="FONT-SIZE: 10pt; FONT-FAMILY: " (Hebrew)?,?serif?? New
Courier>;</SPAN><SPAN style="FONT-FAMILY: " (Hebrew)?,?serif?? New
Courier><o:p></o:p></SPAN></P>
<P><SPAN style="FONT-SIZE: 10pt; FONT-FAMILY: " (Hebrew)?,?serif?? New
Courier>+ </SPAN><SPAN style="FONT-SIZE: 10pt">static const uint32_t
cDpcMaxTime = 10000; //max time to spend in a while loop</SPAN><SPAN
style="FONT-FAMILY: " Arial?,?sans-serif??><o:p></o:p></SPAN></P>
<P><SPAN style="FONT-SIZE: 10pt; FONT-FAMILY: " (Hebrew)?,?serif?? New
Courier>+ </SPAN><SPAN style="FONT-FAMILY: " (Hebrew)?,?serif?? New
Courier><o:p></o:p></SPAN></P>
<P><SPAN style="FONT-SIZE: 10pt; FONT-FAMILY: " (Hebrew)?,?serif?? New
Courier>+ </SPAN><SPAN style="FONT-SIZE: 10pt">uint64_t start =
cl_get_time_stamp</SPAN><SPAN style="FONT-SIZE: 10pt; FONT-FAMILY: "
(Hebrew)?,?serif?? New Courier>();</SPAN><SPAN style="FONT-FAMILY: "
(Hebrew)?,?serif?? New Courier><o:p></o:p></SPAN></P>
<P><SPAN style="FONT-SIZE: 10pt">while ((eqe = next_eqe_sw(eq</SPAN><SPAN
style="FONT-SIZE: 10pt; FONT-FAMILY: " (Hebrew)?,?serif?? New Courier>)))
{</SPAN><SPAN style="FONT-FAMILY: " (Hebrew)?,?serif?? New
Courier><o:p></o:p></SPAN></P>
<P><SPAN style="FONT-SIZE: 10pt; FONT-FAMILY: " (Hebrew)?,?serif?? New
Courier>/*</SPAN><SPAN style="FONT-FAMILY: " (Hebrew)?,?serif?? New
Courier><o:p></o:p></SPAN></P>
<P><SPAN style="FONT-SIZE: 10pt; FONT-FAMILY: " (Hebrew)?,?serif?? New
Courier>@@ -222,6 +225,7 @@</SPAN><SPAN style="FONT-FAMILY: "
(Hebrew)?,?serif?? New Courier><o:p></o:p></SPAN></P>
<P><SPAN style="FONT-SIZE: 10pt">default</SPAN><SPAN
style="FONT-SIZE: 10pt; FONT-FAMILY: " (Hebrew)?,?serif?? New
Courier>:</SPAN><SPAN style="FONT-FAMILY: " (Hebrew)?,?serif?? New
Courier><o:p></o:p></SPAN></P>
<P><SPAN style="FONT-SIZE: 10pt">mlx4_warn(dev, "Unhandled event %02x(%02x) on
EQ %d at index %u\n</SPAN><SPAN style="FONT-SIZE: 10pt; FONT-FAMILY: "
(Hebrew)?,?serif?? New Courier>",</SPAN><SPAN style="FONT-FAMILY: "
(Hebrew)?,?serif?? New Courier><o:p></o:p></SPAN></P>
<P><SPAN style="FONT-SIZE: 10pt">eqe->type, eqe->subtype, eq->eqn,
eq->cons_index</SPAN><SPAN style="FONT-SIZE: 10pt; FONT-FAMILY: "
(Hebrew)?,?serif?? New Courier>);</SPAN><SPAN style="FONT-FAMILY: "
(Hebrew)?,?serif?? New Courier><o:p></o:p></SPAN></P>
<P><SPAN style="FONT-SIZE: 10pt; FONT-FAMILY: " (Hebrew)?,?serif?? New
Courier>+ </SPAN><SPAN style="FONT-FAMILY: " (Hebrew)?,?serif?? New
Courier><o:p></o:p></SPAN></P>
<P><SPAN style="FONT-SIZE: 10pt">break</SPAN><SPAN
style="FONT-SIZE: 10pt; FONT-FAMILY: " (Hebrew)?,?serif?? New
Courier>;</SPAN><SPAN style="FONT-FAMILY: " (Hebrew)?,?serif?? New
Courier><o:p></o:p></SPAN></P>
<P><SPAN style="FONT-SIZE: 10pt; FONT-FAMILY: " (Hebrew)?,?serif?? New
Courier>};</SPAN><SPAN style="FONT-FAMILY: " (Hebrew)?,?serif?? New
Courier><o:p></o:p></SPAN></P>
<P><SPAN style="FONT-SIZE: 10pt; FONT-FAMILY: " (Hebrew)?,?serif?? New
Courier>@@ -244,6 +248,10 @@</SPAN><SPAN style="FONT-FAMILY: "
(Hebrew)?,?serif?? New Courier><o:p></o:p></SPAN></P>
<P><SPAN style="FONT-SIZE: 10pt">eq_set_ci(eq, 0</SPAN><SPAN
style="FONT-SIZE: 10pt; FONT-FAMILY: " (Hebrew)?,?serif?? New
Courier>);</SPAN><SPAN style="FONT-FAMILY: " (Hebrew)?,?serif?? New
Courier><o:p></o:p></SPAN></P>
<P><SPAN style="FONT-SIZE: 10pt">set_ci = 0</SPAN><SPAN
style="FONT-SIZE: 10pt; FONT-FAMILY: " (Hebrew)?,?serif?? New
Courier>;</SPAN><SPAN style="FONT-FAMILY: " (Hebrew)?,?serif?? New
Courier><o:p></o:p></SPAN></P>
<P><SPAN style="FONT-SIZE: 10pt; FONT-FAMILY: " (Hebrew)?,?serif?? New
Courier>}</SPAN><SPAN style="FONT-FAMILY: " (Hebrew)?,?serif?? New
Courier><o:p></o:p></SPAN></P>
<P><SPAN style="FONT-SIZE: 10pt; FONT-FAMILY: " (Hebrew)?,?serif?? New
Courier>+ </SPAN><SPAN style="FONT-FAMILY: " (Hebrew)?,?serif?? New
Courier><o:p></o:p></SPAN></P>
<P><SPAN style="FONT-SIZE: 10pt; FONT-FAMILY: " (Hebrew)?,?serif?? New
Courier>+ </SPAN><SPAN style="FONT-SIZE: 10pt">if (cl_get_time_stamp() - start
> cDpcMaxTime</SPAN><SPAN style="FONT-SIZE: 10pt; FONT-FAMILY: "
(Hebrew)?,?serif?? New Courier> ) {</SPAN><SPAN style="FONT-FAMILY: "
(Hebrew)?,?serif?? New Courier><o:p></o:p></SPAN></P>
<P><SPAN style="FONT-SIZE: 10pt; FONT-FAMILY: " (Hebrew)?,?serif?? New
Courier>+ </SPAN><SPAN style="FONT-SIZE: 10pt">break; //allow other DPCs as
well</SPAN><SPAN style="FONT-FAMILY: "
Arial?,?sans-serif??><o:p></o:p></SPAN></P>
<P><SPAN style="FONT-SIZE: 10pt; FONT-FAMILY: " (Hebrew)?,?serif?? New
Courier>+ }</SPAN><SPAN style="FONT-FAMILY: " (Hebrew)?,?serif?? New
Courier><o:p></o:p></SPAN></P>
<P><SPAN style="FONT-SIZE: 10pt; FONT-FAMILY: " (Hebrew)?,?serif?? New
Courier>}</SPAN><SPAN style="FONT-FAMILY: " (Hebrew)?,?serif?? New
Courier><o:p></o:p></SPAN></P>
<P><SPAN style="FONT-SIZE: 10pt">eq_set_ci(eq, 1</SPAN><SPAN
style="FONT-SIZE: 10pt; FONT-FAMILY: " (Hebrew)?,?serif?? New
Courier>);<o:p></o:p></SPAN></P></DIV></DIV></DIV></BLOCKQUOTE></BODY></HTML>