<!DOCTYPE HTML PUBLIC "-//W3C//DTD HTML 4.0 Transitional//EN">
<HTML xmlns="http://www.w3.org/TR/REC-html40" xmlns:ex12t =
"http://schemas.microsoft.com/exchange/services/2006/types" xmlns:m =
"http://schemas.microsoft.com/office/2004/12/omml" xmlns:mver =
"http://schemas.openxmlformats.org/markup-compatibility/2006" xmlns:wf =
"http://schemas.microsoft.com/sharepoint/soap/workflow/" xmlns:udcxf =
"http://schemas.microsoft.com/data/udc/xmlfile" xmlns:xsi =
"http://www.w3.org/2001/XMLSchema-instance" xmlns:sps =
"http://schemas.microsoft.com/sharepoint/soap/" xmlns:xsd =
"http://www.w3.org/2001/XMLSchema" xmlns:udc =
"http://schemas.microsoft.com/data/udc" xmlns:dsp =
"http://schemas.microsoft.com/sharepoint/dsp" xmlns:ds =
"http://www.w3.org/2000/09/xmldsig#" xmlns:dir =
"http://schemas.microsoft.com/sharepoint/soap/directory/" xmlns:ois =
"http://schemas.microsoft.com/sharepoint/soap/ois/" xmlns:x2 =
"http://schemas.microsoft.com/office/excel/2003/xml" XMLNS:D = "DAV:" xmlns:q =
"http://schemas.xmlsoap.org/soap/envelope/" xmlns:html =
"http://www.w3.org/TR/REC-html40" xmlns:oa =
"urn:schemas-microsoft-com:office:activation" xmlns:c =
"urn:schemas-microsoft-com:office:component:spreadsheet" xmlns:ss =
"urn:schemas-microsoft-com:office:spreadsheet" xmlns:b =
"urn:schemas-microsoft-com:office:publisher" xmlns:z = "#RowsetSchema" xmlns:rs
= "urn:schemas-microsoft-com:rowset" xmlns:s =
"uuid:BDC6E3F0-6DA3-11d1-A2A3-00AA00C14882" xmlns:dt =
"uuid:C2F41010-65B3-11d1-A29F-00AA00C14882" xmlns:a =
"urn:schemas-microsoft-com:office:access" xmlns:p =
"urn:schemas-microsoft-com:office:powerpoint" xmlns:x =
"urn:schemas-microsoft-com:office:excel" xmlns:w =
"urn:schemas-microsoft-com:office:word" xmlns:o =
"urn:schemas-microsoft-com:office:office" xmlns:v =
"urn:schemas-microsoft-com:vml"><HEAD>
<META http-equiv=Content-Type content="text/html; charset=us-ascii">
<STYLE>@font-face {
font-family: Wingdings;
}
@font-face {
font-family: Cambria Math;
}
@font-face {
font-family: Calibri;
}
@page Section1 {size: 8.5in 11.0in; margin: 1.0in 1.0in 1.0in 1.0in; }
P.MsoNormal {
FONT-SIZE: 11pt; MARGIN: 0in 0in 0pt; FONT-FAMILY: "Calibri","sans-serif"
}
LI.MsoNormal {
FONT-SIZE: 11pt; MARGIN: 0in 0in 0pt; FONT-FAMILY: "Calibri","sans-serif"
}
DIV.MsoNormal {
FONT-SIZE: 11pt; MARGIN: 0in 0in 0pt; FONT-FAMILY: "Calibri","sans-serif"
}
A:link {
COLOR: blue; TEXT-DECORATION: underline; mso-style-priority: 99
}
SPAN.MsoHyperlink {
COLOR: blue; TEXT-DECORATION: underline; mso-style-priority: 99
}
A:visited {
COLOR: purple; TEXT-DECORATION: underline; mso-style-priority: 99
}
SPAN.MsoHyperlinkFollowed {
COLOR: purple; TEXT-DECORATION: underline; mso-style-priority: 99
}
P.MsoListParagraph {
FONT-SIZE: 11pt; MARGIN: 0in 0in 0pt 0.5in; FONT-FAMILY: "Calibri","sans-serif"; mso-style-priority: 34
}
LI.MsoListParagraph {
FONT-SIZE: 11pt; MARGIN: 0in 0in 0pt 0.5in; FONT-FAMILY: "Calibri","sans-serif"; mso-style-priority: 34
}
DIV.MsoListParagraph {
FONT-SIZE: 11pt; MARGIN: 0in 0in 0pt 0.5in; FONT-FAMILY: "Calibri","sans-serif"; mso-style-priority: 34
}
SPAN.EmailStyle17 {
COLOR: windowtext; FONT-FAMILY: "Calibri","sans-serif"; mso-style-type: personal-compose
}
.MsoChpDefault {
mso-style-type: export-only
}
DIV.Section1 {
page: Section1
}
OL {
MARGIN-BOTTOM: 0in
}
UL {
MARGIN-BOTTOM: 0in
}
</STYLE>
<!--[if gte mso 9]><xml>
<o:shapedefaults v:ext="edit" spidmax="1026" />
</xml><![endif]--><!--[if gte mso 9]><xml>
<o:shapelayout v:ext="edit">
<o:idmap v:ext="edit" data="1" />
</o:shapelayout></xml><![endif]-->
<META content="MSHTML 6.00.6000.16414" name=GENERATOR></HEAD>
<BODY lang=EN-US vLink=purple link=blue>
<DIV><SPAN class=326405919-11042007><FONT face=Arial color=#0000ff size=2>See
below:</FONT></SPAN></DIV>
<DIV><SPAN class=326405919-11042007><FONT face=Arial color=#0000ff
size=2></FONT></SPAN> </DIV>
<DIV><SPAN class=326405919-11042007><FONT face=Arial color=#0000ff
size=2>Thanks</FONT></SPAN></DIV>
<DIV><SPAN class=326405919-11042007><FONT face=Arial color=#0000ff
size=2>Tzachi</FONT></SPAN></DIV><BR>
<BLOCKQUOTE dir=ltr
style="PADDING-LEFT: 5px; MARGIN-LEFT: 5px; BORDER-LEFT: #0000ff 2px solid; MARGIN-RIGHT: 0px">
<DIV class=OutlookMessageHeader lang=en-us dir=ltr align=left>
<HR tabIndex=-1>
<FONT face=Tahoma size=2><B>From:</B> ofw-bounces@lists.openfabrics.org
[mailto:ofw-bounces@lists.openfabrics.org] <B>On Behalf Of </B>Fab
Tillier<BR><B>Sent:</B> Wednesday, April 11, 2007 10:14 PM<BR><B>To:</B>
ofw@lists.openfabrics.org<BR><B>Cc:</B> Jeff Baxter; Xavier
Pillons<BR><B>Subject:</B> [ofw] MTHCA registry
parameters<BR></FONT><BR></DIV>
<DIV></DIV>
<DIV class=Section1>
<P class=MsoNormal>Hi folks,<o:p></o:p></P>
<P class=MsoNormal><o:p> </o:p></P>
<P class=MsoNormal>What do the following registry parameters for MTHCA
do?<o:p></o:p></P>
<P class=MsoNormal><o:p> </o:p></P>
<P class=MsoNormal>HKR,"Parameters","TunePci",%REG_DWORD%,0<o:p></o:p></P>
<P class=MsoListParagraph
style="TEXT-INDENT: -0.25in; mso-list: l0 level1 lfo1"><![if !supportLists]><SPAN
style="mso-list: Ignore">-<SPAN
style="FONT-WEIGHT: normal; FONT-SIZE: 7pt; LINE-HEIGHT: normal; FONT-STYLE: normal; FONT-VARIANT: normal"
Roman?? New Times>
</SPAN></SPAN><![endif]>What does PCI tuning do?<o:p></o:p></P>
<P class=MsoListParagraph
style="TEXT-INDENT: -0.25in; mso-list: l0 level1 lfo1"><![if !supportLists]><SPAN
style="mso-list: Ignore">-<SPAN
style="FONT-WEIGHT: normal; FONT-SIZE: 7pt; LINE-HEIGHT: normal; FONT-STYLE: normal; FONT-VARIANT: normal"
Roman?? New Times>
</SPAN></SPAN><![endif]>Why is it off by default?<o:p></o:p></P>
<P class=MsoListParagraph
style="TEXT-INDENT: -0.25in; mso-list: l0 level1 lfo1"><![if !supportLists]><SPAN
style="mso-list: Ignore">-<SPAN
style="FONT-WEIGHT: normal; FONT-SIZE: 7pt; LINE-HEIGHT: normal; FONT-STYLE: normal; FONT-VARIANT: normal"
Roman?? New Times>
</SPAN></SPAN><![endif]>Does turning it on result in higher bandwidth or lower
latency?<o:p></o:p></P>
<P class=MsoListParagraph
style="TEXT-INDENT: -0.25in; mso-list: l0 level1 lfo1"><![if !supportLists]><SPAN
style="mso-list: Ignore">-<SPAN
style="FONT-WEIGHT: normal; FONT-SIZE: 7pt; LINE-HEIGHT: normal; FONT-STYLE: normal; FONT-VARIANT: normal"
Roman?? New Times>
</SPAN></SPAN><![endif]>What are the risks in turning it on?<o:p></o:p></P>
<P class=MsoNormal><o:p> </o:p></P>
<P
class=MsoNormal>HKR,"Parameters","ProcessorAffinity",%REG_DWORD%,0<o:p></o:p></P><![if !supportLists]>
<P class=MsoListParagraph
style="TEXT-INDENT: -0.25in; mso-list: l0 level1 lfo1"><SPAN
style="mso-list: Ignore">-<SPAN
style="FONT-WEIGHT: normal; FONT-SIZE: 7pt; LINE-HEIGHT: normal; FONT-STYLE: normal; FONT-VARIANT: normal"
Roman?? New Times>
</SPAN></SPAN><![endif]>Is this ISR or DPC affinity, or both?<SPAN
class=326405919-11042007><FONT color=#0000ff size=2> </FONT></SPAN></P>
<P class=MsoListParagraph
style="TEXT-INDENT: -0.25in; mso-list: l0 level1 lfo1"><SPAN
class=326405919-11042007><FONT color=#0000ff size=2>This is for the ISR,
but it is also being forced by windows for the same DPC. on the same
processor. I have noticed that on some benchmarks this gives a 5%
improvement but on other it takes of up to 20%. So, one has to use it
carefully.</FONT></SPAN></P><![if !supportLists]>
<P class=MsoListParagraph
style="TEXT-INDENT: -0.25in; mso-list: l0 level1 lfo1"><SPAN
style="mso-list: Ignore">-<SPAN
style="FONT-WEIGHT: normal; FONT-SIZE: 7pt; LINE-HEIGHT: normal; FONT-STYLE: normal; FONT-VARIANT: normal"
Roman?? New Times>
</SPAN></SPAN><![endif]>Does this apply to kernel mode only?<SPAN
class=326405919-11042007><FONT color=#0000ff size=2> </FONT></SPAN></P>
<P class=MsoListParagraph
style="TEXT-INDENT: -0.25in; mso-list: l0 level1 lfo1"><SPAN
class=326405919-11042007><FONT color=#0000ff size=2>yes. Only for the ISR
and DPC that are created by mthca.</FONT> </SPAN><o:p></o:p></P>
<P class=MsoNormal><o:p> </o:p></P>
<P
class=MsoNormal>HKR,"Parameters","MaxDpcTimeUs",%REG_DWORD%,10000<o:p></o:p></P><![if !supportLists]>
<P class=MsoListParagraph
style="TEXT-INDENT: -0.25in; mso-list: l0 level1 lfo1"><SPAN
style="mso-list: Ignore">-<SPAN
style="FONT-WEIGHT: normal; FONT-SIZE: 7pt; LINE-HEIGHT: normal; FONT-STYLE: normal; FONT-VARIANT: normal"
Roman?? New Times>
</SPAN></SPAN><![endif]>What does this do?<SPAN class=326405919-11042007><FONT
color=#0000ff size=2> </FONT></SPAN></P>
<P class=MsoListParagraph
style="TEXT-INDENT: -0.25in; mso-list: l0 level1 lfo1"><SPAN
class=326405919-11042007><FONT color=#0000ff size=2>There have been rare cases
in which a DPC was running "for ever" (well not really for ever but for
30 seconds). This had the effect that the mouse was not moving and so were
other devices. this registry key is a limit for the time one DPC might be
running.</FONT> <FONT color=#0000ff size=2>If this limit is reached, we
stop the DPC and create another one at the end of DPC queue for that
processor.</FONT></SPAN></P><![if !supportLists]>
<P class=MsoListParagraph
style="TEXT-INDENT: -0.25in; mso-list: l0 level1 lfo1"><SPAN
style="mso-list: Ignore">-<SPAN
style="FONT-WEIGHT: normal; FONT-SIZE: 7pt; LINE-HEIGHT: normal; FONT-STYLE: normal; FONT-VARIANT: normal"
Roman?? New Times>
</SPAN></SPAN><![endif]>How does it affect performance when the value is
increased or lowered?<SPAN class=326405919-11042007><FONT color=#0000ff
size=2> </FONT></SPAN></P><![if !supportLists]>
<P class=MsoListParagraph
style="TEXT-INDENT: -0.25in; mso-list: l0 level1 lfo1"><SPAN
style="mso-list: Ignore">-<SPAN
style="FONT-WEIGHT: normal; FONT-SIZE: 7pt; LINE-HEIGHT: normal; FONT-STYLE: normal; FONT-VARIANT: normal"
Roman?? New Times>
</SPAN></SPAN><![endif]>Is this default value tuned for performance or system
responsiveness?<SPAN class=326405919-11042007><FONT color=#0000ff
size=2> </FONT></SPAN></P>
<P class=MsoListParagraph
style="TEXT-INDENT: -0.25in; mso-list: l0 level1 lfo1"><SPAN
class=326405919-11042007><SPAN class=326405919-11042007><FONT color=#0000ff
size=2>Usually, this has no effect at all as for most normal scenarios DPCs
end much faster than the default value. In the case that they don't
this means that by default every 10ms we stop handling the IB tasks
and doing other things. If other things are "normal" things that don't take a
lot of time (compared to 10 ms) (such as moving the mouse) than there is
no problem, the system is responsive and is also efficient. in the
case that things take more time then IB suffers but I guess that one has
to do the other things as well. One way to solve the problem is using the
ProcessorAffinity in order to lock the system to one processor and the other
on the other processor. A better way to solve this is use MSI (LH only) or
make sure that DPCs are being created on all processors and not only
one.</FONT></SPAN></SPAN></P>
<P class=MsoListParagraph
style="TEXT-INDENT: -0.25in; mso-list: l0 level1 lfo1"><SPAN
class=326405919-11042007><SPAN class=326405919-11042007><FONT color=#0000ff
size=2>[here is the scenario that we saw DPCs that were running for 30
seconds: The system was running ipoib stress both inbound and outbound.
What happens is that packets arrive on the networks, so there is a DPC that is
pooling, until there are no packets on the receive side. mean time there are
send completions so the send CQ creates an EQE. the DPC finds this EKE, and go
to handle the send CQ. All packets sent are handled and then there are more
packets on the receive CQ, so it creates another EQE. so the same DPC finds
the EQE and handles it. This goes on forever, there are packets on the send
side and while there are handled packets come on the receive side and so on.
They are all handled by the same DPC. This registry variable is responsible
for breaking the chain. Please note that from the remote side things look well
as ipoib packets are handled well. But as time pass, it is clear that
DPCS can not wait that long]</FONT></SPAN></SPAN></P>
<P class=MsoNormal><o:p> </o:p></P>
<P class=MsoNormal>Thanks!<o:p></o:p></P>
<P class=MsoNormal>-Fab<o:p></o:p></P></DIV></BLOCKQUOTE></BODY></HTML>