<html xmlns:v="urn:schemas-microsoft-com:vml" xmlns:o="urn:schemas-microsoft-com:office:office" xmlns:w="urn:schemas-microsoft-com:office:word" xmlns:x="urn:schemas-microsoft-com:office:excel" xmlns:p="urn:schemas-microsoft-com:office:powerpoint" xmlns:a="urn:schemas-microsoft-com:office:access" xmlns:dt="uuid:C2F41010-65B3-11d1-A29F-00AA00C14882" xmlns:s="uuid:BDC6E3F0-6DA3-11d1-A2A3-00AA00C14882" xmlns:rs="urn:schemas-microsoft-com:rowset" xmlns:z="#RowsetSchema" xmlns:b="urn:schemas-microsoft-com:office:publisher" xmlns:ss="urn:schemas-microsoft-com:office:spreadsheet" xmlns:c="urn:schemas-microsoft-com:office:component:spreadsheet" xmlns:oa="urn:schemas-microsoft-com:office:activation" xmlns:html="http://www.w3.org/TR/REC-html40" xmlns:q="http://schemas.xmlsoap.org/soap/envelope/" xmlns:D="DAV:" xmlns:x2="http://schemas.microsoft.com/office/excel/2003/xml" xmlns:ois="http://schemas.microsoft.com/sharepoint/soap/ois/" xmlns:dir="http://schemas.microsoft.com/sharepoint/soap/directory/" xmlns:ds="http://www.w3.org/2000/09/xmldsig#" xmlns:dsp="http://schemas.microsoft.com/sharepoint/dsp" xmlns:udc="http://schemas.microsoft.com/data/udc" xmlns:xsd="http://www.w3.org/2001/XMLSchema" xmlns:sub="http://schemas.microsoft.com/sharepoint/soap/2002/1/alerts/" xmlns:ec="http://www.w3.org/2001/04/xmlenc#" xmlns:sp="http://schemas.microsoft.com/sharepoint/" xmlns:sps="http://schemas.microsoft.com/sharepoint/soap/" xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance" xmlns:udcxf="http://schemas.microsoft.com/data/udc/xmlfile" xmlns:wf="http://schemas.microsoft.com/sharepoint/soap/workflow/" xmlns:mver="http://schemas.openxmlformats.org/markup-compatibility/2006" xmlns:m="http://schemas.microsoft.com/office/2004/12/omml" xmlns:mrels="http://schemas.openxmlformats.org/package/2006/relationships" xmlns:ex12t="http://schemas.microsoft.com/exchange/services/2006/types" xmlns:ex12m="http://schemas.microsoft.com/exchange/services/2006/messages" xmlns:Z="urn:schemas-microsoft-com:" xmlns="http://www.w3.org/TR/REC-html40">
<head>
<META HTTP-EQUIV="Content-Type" CONTENT="text/html; charset=us-ascii">
<meta name=Generator content="Microsoft Word 12 (filtered medium)">
<!--[if !mso]>
<style>
v\:* {behavior:url(#default#VML);}
o\:* {behavior:url(#default#VML);}
w\:* {behavior:url(#default#VML);}
.shape {behavior:url(#default#VML);}
</style>
<![endif]-->
<style>
<!--
/* Font Definitions */
@font-face
{font-family:"Cambria Math";
panose-1:2 4 5 3 5 4 6 3 2 4;}
@font-face
{font-family:Calibri;
panose-1:2 15 5 2 2 2 4 3 2 4;}
@font-face
{font-family:Tahoma;
panose-1:2 11 6 4 3 5 4 4 2 4;}
@font-face
{font-family:"Courier New \(Hebrew\)";
panose-1:0 0 0 0 0 0 0 0 0 0;}
/* Style Definitions */
p.MsoNormal, li.MsoNormal, div.MsoNormal
{margin:0in;
margin-bottom:.0001pt;
font-size:12.0pt;
font-family:"Times New Roman","serif";}
a:link, span.MsoHyperlink
{mso-style-priority:99;
color:blue;
text-decoration:underline;}
a:visited, span.MsoHyperlinkFollowed
{mso-style-priority:99;
color:purple;
text-decoration:underline;}
p
{mso-style-priority:99;
mso-margin-top-alt:auto;
margin-right:0in;
mso-margin-bottom-alt:auto;
margin-left:0in;
font-size:12.0pt;
font-family:"Times New Roman","serif";}
span.EmailStyle18
{mso-style-type:personal-reply;
font-family:"Calibri","sans-serif";
color:#1F497D;}
.MsoChpDefault
{mso-style-type:export-only;
font-size:10.0pt;}
@page Section1
{size:8.5in 11.0in;
margin:1.0in 1.0in 1.0in 1.0in;}
div.Section1
{page:Section1;}
-->
</style>
<!--[if gte mso 9]><xml>
<o:shapedefaults v:ext="edit" spidmax="1026" />
</xml><![endif]--><!--[if gte mso 9]><xml>
<o:shapelayout v:ext="edit">
<o:idmap v:ext="edit" data="1" />
</o:shapelayout></xml><![endif]-->
</head>
<body lang=EN-US link=blue vlink=purple>
<div class=Section1>
<p class=MsoNormal><span style='font-size:11.0pt;font-family:"Calibri","sans-serif";
color:#1F497D'>Note that all kernel ULPs should do something along these lines –
SRP, IPoIB, VNIC, as well as internal IBAL services—should all limit how much
time they spend processing completions and requeue their DPC if they exceed it.<o:p></o:p></span></p>
<p class=MsoNormal><span style='font-size:11.0pt;font-family:"Calibri","sans-serif";
color:#1F497D'><o:p> </o:p></span></p>
<p class=MsoNormal><span style='font-size:11.0pt;font-family:"Calibri","sans-serif";
color:#1F497D'>The natural progression of this idea is that a ULP would provide
the DPC object to queue in response to a CQ event, rather than callbacks.<o:p></o:p></span></p>
<p class=MsoNormal><span style='font-size:11.0pt;font-family:"Calibri","sans-serif";
color:#1F497D'><o:p> </o:p></span></p>
<p class=MsoNormal><span style='font-size:11.0pt;font-family:"Calibri","sans-serif";
color:#1F497D'>-Fab<o:p></o:p></span></p>
<p class=MsoNormal><span style='font-size:11.0pt;font-family:"Calibri","sans-serif";
color:#1F497D'><o:p> </o:p></span></p>
<div style='border:none;border-left:solid blue 1.5pt;padding:0in 0in 0in 4.0pt'>
<div>
<div style='border:none;border-top:solid #B5C4DF 1.0pt;padding:3.0pt 0in 0in 0in'>
<p class=MsoNormal><b><span style='font-size:10.0pt;font-family:"Tahoma","sans-serif"'>From:</span></b><span
style='font-size:10.0pt;font-family:"Tahoma","sans-serif"'>
ofw-bounces@lists.openfabrics.org [mailto:ofw-bounces@lists.openfabrics.org] <b>On
Behalf Of </b>Alex Naslednikov<br>
<b>Sent:</b> Thursday, June 26, 2008 5:01 AM<br>
<b>To:</b> ofw@lists.openfabrics.org; Tzachi Dar<br>
<b>Subject:</b> [ofw] Bugfix for IPOIB failure<o:p></o:p></span></p>
</div>
</div>
<p class=MsoNormal><o:p> </o:p></p>
<div>
<p class=MsoNormal> <o:p></o:p></p>
</div>
<p class=MsoNormal><o:p> </o:p></p>
<div class=MsoNormal align=center style='text-align:center'>
<hr size=2 width="100%" align=center>
</div>
<div>
<p><span style='font-size:10.0pt;font-family:"Arial","sans-serif"'>Hi all,</span><span
style='font-family:"Arial","sans-serif"'><o:p></o:p></span></p>
<p><span style='font-size:10.0pt;font-family:"Arial","sans-serif"'>The original
problem was IPoIB failure (link up/down) during the operation of 'heavy'
applications.</span><span style='font-family:"Arial","sans-serif"'><o:p></o:p></span></p>
<div>
<p class=MsoNormal><span style='font-size:10.0pt;font-family:"Arial","sans-serif"'>After
our investigation, we found when one execute application with heavy load
on the HCA, driver always have non-empty CQs, so it continues to work with its
DPCs and not allows to other DPC to be performed.</span><span style='font-family:
"Arial","sans-serif"'><o:p></o:p></span></p>
</div>
<div>
<p class=MsoNormal><span style='font-family:"Arial","sans-serif"'> <o:p></o:p></span></p>
</div>
<div>
<p class=MsoNormal><span style='font-size:10.0pt;font-family:"Arial","sans-serif"'>The
solution</span><span style='font-family:"Arial","sans-serif"'><o:p></o:p></span></p>
</div>
<div>
<p class=MsoNormal><span style='font-size:10.0pt;font-family:"Arial","sans-serif"'>Based
on an anologous solution for mthca driver, driver has to enable other than its
own DPC to be performed. It's possible to count certain amount of time that
driver spent on DPC handling, than exit, thus allowing other DPC to run :</span><span
style='font-family:"Arial","sans-serif"'><o:p></o:p></span></p>
</div>
<p><span style='font-family:"Arial","sans-serif"'> <o:p></o:p></span></p>
<p><span style='font-family:"Arial","sans-serif"'> <o:p></o:p></span></p>
<p><span style='font-size:10.0pt;font-family:"Arial","sans-serif"'>Index:
hw/mlx4/kernel/bus/net/eq.c</span><span style='font-family:"Arial","sans-serif"'><o:p></o:p></span></p>
<p><span style='font-size:10.0pt;font-family:"Courier New (Hebrew)","serif"'>===================================================================</span><span
style='font-family:"Courier New (Hebrew)","serif"'><o:p></o:p></span></p>
<p><span style='font-size:10.0pt;font-family:"Courier New (Hebrew)","serif"'>---
</span><span style='font-size:10.0pt'>hw/mlx4/kernel/bus/net/eq.c</span><span
style='font-size:10.0pt;font-family:"Courier New (Hebrew)","serif"'> (</span><span
style='font-size:10.0pt'>revision 2634</span><span style='font-size:10.0pt;
font-family:"Courier New (Hebrew)","serif"'>)</span><span style='font-family:
"Courier New (Hebrew)","serif"'><o:p></o:p></span></p>
<p><span style='font-size:10.0pt;font-family:"Courier New (Hebrew)","serif"'>+++
</span><span style='font-size:10.0pt'>hw/mlx4/kernel/bus/net/eq.c</span><span
style='font-size:10.0pt;font-family:"Courier New (Hebrew)","serif"'> (</span><span
style='font-size:10.0pt'>revision 2635</span><span style='font-size:10.0pt;
font-family:"Courier New (Hebrew)","serif"'>)</span><span style='font-family:
"Courier New (Hebrew)","serif"'><o:p></o:p></span></p>
<p><span style='font-size:10.0pt;font-family:"Courier New (Hebrew)","serif"'>@@
-160,6 +160,9 @@</span><span style='font-family:"Courier New (Hebrew)","serif"'><o:p></o:p></span></p>
<p><span style='font-size:10.0pt'>int cqn</span><span style='font-size:10.0pt;
font-family:"Courier New (Hebrew)","serif"'>;</span><span style='font-family:
"Courier New (Hebrew)","serif"'><o:p></o:p></span></p>
<p><span style='font-size:10.0pt'>int eqes_found = 0</span><span
style='font-size:10.0pt;font-family:"Courier New (Hebrew)","serif"'>;</span><span
style='font-family:"Courier New (Hebrew)","serif"'><o:p></o:p></span></p>
<p><span style='font-size:10.0pt'>int set_ci = 0</span><span style='font-size:
10.0pt;font-family:"Courier New (Hebrew)","serif"'>;</span><span
style='font-family:"Courier New (Hebrew)","serif"'><o:p></o:p></span></p>
<p><span style='font-size:10.0pt;font-family:"Courier New (Hebrew)","serif"'>+ </span><span
style='font-size:10.0pt'>static const uint32_t cDpcMaxTime = 10000; //max time
to spend in a while loop</span><span style='font-family:"Arial","sans-serif"'><o:p></o:p></span></p>
<p><span style='font-size:10.0pt;font-family:"Courier New (Hebrew)","serif"'>+ </span><span
style='font-family:"Courier New (Hebrew)","serif"'><o:p></o:p></span></p>
<p><span style='font-size:10.0pt;font-family:"Courier New (Hebrew)","serif"'>+ </span><span
style='font-size:10.0pt'>uint64_t start = cl_get_time_stamp</span><span
style='font-size:10.0pt;font-family:"Courier New (Hebrew)","serif"'>();</span><span
style='font-family:"Courier New (Hebrew)","serif"'><o:p></o:p></span></p>
<p><span style='font-size:10.0pt'>while ((eqe = next_eqe_sw(eq</span><span
style='font-size:10.0pt;font-family:"Courier New (Hebrew)","serif"'>))) {</span><span
style='font-family:"Courier New (Hebrew)","serif"'><o:p></o:p></span></p>
<p><span style='font-size:10.0pt;font-family:"Courier New (Hebrew)","serif"'>/*</span><span
style='font-family:"Courier New (Hebrew)","serif"'><o:p></o:p></span></p>
<p><span style='font-size:10.0pt;font-family:"Courier New (Hebrew)","serif"'>@@
-222,6 +225,7 @@</span><span style='font-family:"Courier New (Hebrew)","serif"'><o:p></o:p></span></p>
<p><span style='font-size:10.0pt'>default</span><span style='font-size:10.0pt;
font-family:"Courier New (Hebrew)","serif"'>:</span><span style='font-family:
"Courier New (Hebrew)","serif"'><o:p></o:p></span></p>
<p><span style='font-size:10.0pt'>mlx4_warn(dev, "Unhandled event
%02x(%02x) on EQ %d at index %u\n</span><span style='font-size:10.0pt;
font-family:"Courier New (Hebrew)","serif"'>",</span><span
style='font-family:"Courier New (Hebrew)","serif"'><o:p></o:p></span></p>
<p><span style='font-size:10.0pt'>eqe->type, eqe->subtype, eq->eqn,
eq->cons_index</span><span style='font-size:10.0pt;font-family:"Courier New (Hebrew)","serif"'>);</span><span
style='font-family:"Courier New (Hebrew)","serif"'><o:p></o:p></span></p>
<p><span style='font-size:10.0pt;font-family:"Courier New (Hebrew)","serif"'>+ </span><span
style='font-family:"Courier New (Hebrew)","serif"'><o:p></o:p></span></p>
<p><span style='font-size:10.0pt'>break</span><span style='font-size:10.0pt;
font-family:"Courier New (Hebrew)","serif"'>;</span><span style='font-family:
"Courier New (Hebrew)","serif"'><o:p></o:p></span></p>
<p><span style='font-size:10.0pt;font-family:"Courier New (Hebrew)","serif"'>};</span><span
style='font-family:"Courier New (Hebrew)","serif"'><o:p></o:p></span></p>
<p><span style='font-size:10.0pt;font-family:"Courier New (Hebrew)","serif"'>@@
-244,6 +248,10 @@</span><span style='font-family:"Courier New (Hebrew)","serif"'><o:p></o:p></span></p>
<p><span style='font-size:10.0pt'>eq_set_ci(eq, 0</span><span style='font-size:
10.0pt;font-family:"Courier New (Hebrew)","serif"'>);</span><span
style='font-family:"Courier New (Hebrew)","serif"'><o:p></o:p></span></p>
<p><span style='font-size:10.0pt'>set_ci = 0</span><span style='font-size:10.0pt;
font-family:"Courier New (Hebrew)","serif"'>;</span><span style='font-family:
"Courier New (Hebrew)","serif"'><o:p></o:p></span></p>
<p><span style='font-size:10.0pt;font-family:"Courier New (Hebrew)","serif"'>}</span><span
style='font-family:"Courier New (Hebrew)","serif"'><o:p></o:p></span></p>
<p><span style='font-size:10.0pt;font-family:"Courier New (Hebrew)","serif"'>+ </span><span
style='font-family:"Courier New (Hebrew)","serif"'><o:p></o:p></span></p>
<p><span style='font-size:10.0pt;font-family:"Courier New (Hebrew)","serif"'>+ </span><span
style='font-size:10.0pt'>if (cl_get_time_stamp() - start > cDpcMaxTime</span><span
style='font-size:10.0pt;font-family:"Courier New (Hebrew)","serif"'> ) {</span><span
style='font-family:"Courier New (Hebrew)","serif"'><o:p></o:p></span></p>
<p><span style='font-size:10.0pt;font-family:"Courier New (Hebrew)","serif"'>+ </span><span
style='font-size:10.0pt'>break; //allow other DPCs as well</span><span
style='font-family:"Arial","sans-serif"'><o:p></o:p></span></p>
<p><span style='font-size:10.0pt;font-family:"Courier New (Hebrew)","serif"'>+
}</span><span style='font-family:"Courier New (Hebrew)","serif"'><o:p></o:p></span></p>
<p><span style='font-size:10.0pt;font-family:"Courier New (Hebrew)","serif"'>}</span><span
style='font-family:"Courier New (Hebrew)","serif"'><o:p></o:p></span></p>
<p><span style='font-size:10.0pt'>eq_set_ci(eq, 1</span><span style='font-size:
10.0pt;font-family:"Courier New (Hebrew)","serif"'>);<o:p></o:p></span></p>
</div>
</div>
</div>
</body>
</html>