<html dir="ltr"><head>
<meta http-equiv="Content-Type" content="text/html; charset=Windows-1252">
<style>@font-face {
font-family: Calibri;
}
@font-face {
font-family: Tahoma;
}
@page WordSection1 {margin: 1.0in 1.0in 1.0in 1.0in; }
P.MsoNormal {
MARGIN: 0in 0in 0pt; FONT-FAMILY: "Calibri","sans-serif"; FONT-SIZE: 11pt
}
LI.MsoNormal {
MARGIN: 0in 0in 0pt; FONT-FAMILY: "Calibri","sans-serif"; FONT-SIZE: 11pt
}
DIV.MsoNormal {
MARGIN: 0in 0in 0pt; FONT-FAMILY: "Calibri","sans-serif"; FONT-SIZE: 11pt
}
A:link {
COLOR: blue; TEXT-DECORATION: underline
}
SPAN.MsoHyperlink {
COLOR: blue; TEXT-DECORATION: underline
}
A:visited {
COLOR: purple; TEXT-DECORATION: underline
}
SPAN.MsoHyperlinkFollowed {
COLOR: purple; TEXT-DECORATION: underline
}
P {
MARGIN: 0in 0in 0pt; FONT-FAMILY: "Times New Roman","serif"; FONT-SIZE: 12pt
}
P.MsoListParagraph {
MARGIN: 0in 0in 0pt 0.5in; FONT-FAMILY: "Calibri","sans-serif"; FONT-SIZE: 11pt
}
LI.MsoListParagraph {
MARGIN: 0in 0in 0pt 0.5in; FONT-FAMILY: "Calibri","sans-serif"; FONT-SIZE: 11pt
}
DIV.MsoListParagraph {
MARGIN: 0in 0in 0pt 0.5in; FONT-FAMILY: "Calibri","sans-serif"; FONT-SIZE: 11pt
}
P.msochpdefault {
MARGIN: 0in 0in 0pt; FONT-FAMILY: "Calibri","sans-serif"; FONT-SIZE: 10pt
}
LI.msochpdefault {
MARGIN: 0in 0in 0pt; FONT-FAMILY: "Calibri","sans-serif"; FONT-SIZE: 10pt
}
DIV.msochpdefault {
MARGIN: 0in 0in 0pt; FONT-FAMILY: "Calibri","sans-serif"; FONT-SIZE: 10pt
}
SPAN.emailstyle18 {
FONT-STYLE: normal; FONT-FAMILY: "Calibri","sans-serif"; COLOR: windowtext; FONT-WEIGHT: normal; TEXT-DECORATION: none
}
SPAN.emailstyle19 {
FONT-STYLE: normal; FONT-FAMILY: "Calibri","sans-serif"; COLOR: #1f497d; FONT-WEIGHT: normal; TEXT-DECORATION: none
}
SPAN.EmailStyle22 {
FONT-STYLE: normal; FONT-FAMILY: "Calibri","sans-serif"; COLOR: #1f497d; FONT-WEIGHT: normal; TEXT-DECORATION: none
}
SPAN.SpellE {
}
.msochpdefault {
FONT-FAMILY: "Calibri","sans-serif"; FONT-SIZE: 10pt
}
DIV.WordSection1 {
}
OL {
MARGIN-BOTTOM: 0in
}
UL {
MARGIN-BOTTOM: 0in
}
</style>
<meta name="GENERATOR" content="MSHTML 9.00.8112.16440">
<style id="owaTempEditStyle"></style><style title="owaParaStyle"><!--P {
MARGIN-TOP: 0px; MARGIN-BOTTOM: 0px
}
--></style>
</head>
<body lang="EN-US" vlink="purple" link="blue" ocsi="x">
<div style="FONT-FAMILY: Tahoma; DIRECTION: ltr; COLOR: #000000; FONT-SIZE: 13px">
<div>Paul:</div>
<div><font size="2" face="tahoma"></font> </div>
<div><font size="2" face="tahoma">Thanks for the reply. I'm afraid I'm not quite following you: why does using the API provided by Storport to get the MSI vector associated with the current request imply that you're submitting on a different core than completing?
The core you're executing on is the one assigned the vector that the API gives you, you put it on the submission queue that has been assigned to that vector, and it's completed on the completion queue for that vector as well.</font></div>
<div><font size="2" face="tahoma"></font> </div>
<div><font size="2" face="tahoma">What am I missing here?</font></div>
<div><font size="2" face="tahoma"></font> </div>
<div><font size="2" face="tahoma">Greg</font></div>
<div dir="ltr"><font color="#000000" size="2" face="Tahoma"></font> </div>
<div style="DIRECTION: ltr" id="divRpF346993">
<hr tabindex="-1">
<font color="#000000" size="2" face="Tahoma"><b>From:</b> Luse, Paul E [paul.e.luse@intel.com]<br>
<b>Sent:</b> Wednesday, February 15, 2012 10:57 AM<br>
<b>To:</b> Greg de Valois; nvmewin@lists.openfabrics.org<br>
<b>Subject:</b> RE: Question on Processor to MSI vector mapping<br>
</font><br>
</div>
<div></div>
<div>
<div class="WordSection1">
<p class="MsoNormal"><font color="#1f497d" size="2" face="Calibri"><span style="COLOR: #1f497d; FONT-SIZE: 10pt">Hi Greg-</span></font></p>
<p class="MsoNormal"><font color="#1f497d" size="2" face="Calibri"><span style="COLOR: #1f497d; FONT-SIZE: 10pt"></span></font> </p>
<p class="MsoNormal"><font color="#1f497d" size="2" face="Calibri"><span style="COLOR: #1f497d; FONT-SIZE: 10pt">Thanks for the question, it’s a good one!<span>
</span>The reasoning was mainly because it was fairly straightforward and had one benefit over the method you mention – we could assure NUMA optimization and vector matching for both SQ and CQ.<span>
</span>There are many, many ways that one could approach this problem and we discussed<span>
</span>a few as part of the <span class="SpellE">dev</span> of this driver and then individually discussed experiences with various methods with other
<span class="SpellE">dev</span> teams at our respective companies.</span></font></p>
<p class="MsoNormal"><font color="#1f497d" size="2" face="Calibri"><span style="COLOR: #1f497d; FONT-SIZE: 10pt"></span></font> </p>
<p class="MsoNormal"><font color="#1f497d" size="2" face="Calibri"><span style="COLOR: #1f497d; FONT-SIZE: 10pt">If one uses the method you mention below, we’d create our SQ/CQ per core NUMA optimized and then be submitting on a different core than we complete
on however we’d still be optimizing the completion side.<span> </span>The more I thought about this the more I realize
<span class="SpellE">its</span> actually not buying us much of anything over using the Msft API due to the effects of CPU cache and the fact that the SQ access on submission is a write and not a read.<span>
</span>I also heard from various other folks that they didn’t find the API below to be accurate all of the time, I can’t say from experience that I’ve seen this.</span></font></p>
<p class="MsoNormal"><font color="#1f497d" size="2" face="Calibri"><span style="COLOR: #1f497d; FONT-SIZE: 10pt"></span></font> </p>
<p class="MsoNormal"><font color="#1f497d" size="2" face="Calibri"><span style="COLOR: #1f497d; FONT-SIZE: 10pt">That said, I will likely be proposing an alternate method in the near future so I’ll go ahead and propose it now since you brought up the subject:</span></font></p>
<p class="MsoNormal"><font color="#1f497d" size="2" face="Calibri"><span style="COLOR: #1f497d; FONT-SIZE: 10pt"></span></font> </p>
<p class="MsoNormal"><font color="#1f497d" size="2" face="Calibri"><span style="COLOR: #1f497d; FONT-SIZE: 10pt">Proposal:<span>
</span>no long decompose the MSI address to populate the mapping table.<span> </span>
Instead, start off with a 1:1 mapping and ‘learn and update’ the mapping table on the completion side.<span>
</span>Would still avoid the storport API because I don’t think it adds value over the learned method and requires us to use the DPC steering option which I’ve witnessed to have unpredictable side effects.<span>
</span>I do plan on following up with Msft (and have already had several internal discussions at Intel with other storport
<span class="SpellE">devs</span>) on exactly how some of these optimizations within storport so we can better gauge whether they’re a good fit for us or now.</span></font></p>
<p class="MsoNormal"><font color="#1f497d" size="2" face="Calibri"><span style="COLOR: #1f497d; FONT-SIZE: 10pt"></span></font> </p>
<p class="MsoNormal"><font color="#1f497d" size="2" face="Calibri"><span style="COLOR: #1f497d; FONT-SIZE: 10pt">Pro:<span>
</span>This will *<b><span style="FONT-WEIGHT: bold">always</span></b>* work whereas the method we have no does not work for APIC logical mode.<span>
</span>I prefer a simple “one size fits all” solution every time over a 2 path solution even if one path is slightly optimized.<span>
</span>It makes the driver more maintainable and gives us less variables in debug (right now we don’t even store whether we found the APIC in
<span class="SpellE">phy</span> or logical mode so during debug you don’t really know).</span></font></p>
<p class="MsoNormal"><font color="#1f497d" size="2" face="Calibri"><span style="COLOR: #1f497d; FONT-SIZE: 10pt"></span></font> </p>
<p class="MsoNormal"><font color="#1f497d" size="2" face="Calibri"><span style="COLOR: #1f497d; FONT-SIZE: 10pt">Con:<span>
</span>SQ <span class="SpellE">mem</span> will be on a different core than the submitting thread but I don’t believe this is a measurable issue.<span>
</span>Certainly can perform some experiments to check though</span></font></p>
<p class="MsoNormal"><font color="#1f497d" size="2" face="Calibri"><span style="COLOR: #1f497d; FONT-SIZE: 10pt"></span></font> </p>
<p class="MsoNormal"><font color="#1f497d" size="2" face="Calibri"><span style="COLOR: #1f497d; FONT-SIZE: 10pt">Other thoughts?</span></font></p>
<p class="MsoNormal"><font color="#1f497d" size="2" face="Calibri"><span style="COLOR: #1f497d; FONT-SIZE: 10pt"></span></font> </p>
<p class="MsoNormal"><font color="#1f497d" size="2" face="Calibri"><span style="COLOR: #1f497d; FONT-SIZE: 10pt">Thx</span></font></p>
<p class="MsoNormal"><font color="#1f497d" size="2" face="Calibri"><span style="COLOR: #1f497d; FONT-SIZE: 10pt">Paul</span></font></p>
<p class="MsoNormal"><font color="#1f497d" size="2" face="Calibri"><span style="COLOR: #1f497d; FONT-SIZE: 10pt"></span></font> </p>
<div>
<div style="BORDER-BOTTOM: medium none; BORDER-LEFT: medium none; PADDING-BOTTOM: 0in; PADDING-LEFT: 0in; PADDING-RIGHT: 0in; BORDER-TOP: #b5c4df 1pt solid; BORDER-RIGHT: medium none; PADDING-TOP: 3pt">
<p class="MsoNormal"><b><font size="2" face="Tahoma"><span style="FONT-FAMILY: 'Tahoma','sans-serif'; FONT-SIZE: 10pt; FONT-WEIGHT: bold">From:</span></font></b><font size="2" face="Tahoma"><span style="FONT-FAMILY: 'Tahoma','sans-serif'; FONT-SIZE: 10pt">
nvmewin-bounces@lists.openfabrics.org [mailto:nvmewin-bounces@lists.openfabrics.org]
<b><span style="FONT-WEIGHT: bold">On Behalf Of </span></b>Greg de Valois<br>
<b><span style="FONT-WEIGHT: bold">Sent:</span></b> Tuesday, February 14, 2012 5:02 PM<br>
<b><span style="FONT-WEIGHT: bold">To:</span></b> nvmewin@lists.openfabrics.org<br>
<b><span style="FONT-WEIGHT: bold">Subject:</span></b> [nvmewin] Question on Processor to MSI vector mapping</span></font></p>
</div>
</div>
<p class="MsoNormal"><font size="2" face="Calibri"><span style="FONT-SIZE: 11pt"></span></font> </p>
<div>
<div>
<p class="MsoNormal"><font color="#1f497d" size="2" face="Tahoma"><span style="FONT-FAMILY: 'Tahoma','sans-serif'; COLOR: #1f497d; FONT-SIZE: 10pt">All:</span></font><font color="black" size="2" face="Tahoma"><span style="FONT-FAMILY: 'Tahoma','sans-serif'; COLOR: black; FONT-SIZE: 10pt"></span></font></p>
</div>
<div>
<p class="MsoNormal"><font color="black" size="2" face="Tahoma"><span style="FONT-FAMILY: 'Tahoma','sans-serif'; COLOR: black; FONT-SIZE: 10pt"></span></font> </p>
</div>
<div>
<div>
<p class="MsoNormal"><font color="#1f497d" size="2" face="Tahoma"><span style="FONT-FAMILY: 'Tahoma','sans-serif'; COLOR: #1f497d; FONT-SIZE: 10pt">I'm wondering if anyone can explain to me the reasoning behind the processor to MSI vector translation that is
being done by the driver, instead of using the vector returned from StorPortGetStartIoPerfParams for each IO? Are there cases where this doesn't work properly?</span></font></p>
</div>
<div>
<p class="MsoNormal"><font color="#1f497d" size="2" face="Tahoma"><span style="FONT-FAMILY: 'Tahoma','sans-serif'; COLOR: #1f497d; FONT-SIZE: 10pt"></span></font> </p>
</div>
<div>
<p class="MsoNormal"><font color="#1f497d" size="2" face="Tahoma"><span style="FONT-FAMILY: 'Tahoma','sans-serif'; COLOR: #1f497d; FONT-SIZE: 10pt">Thanks,
</span></font></p>
</div>
<div>
<p class="MsoNormal"><font color="#1f497d" size="2" face="Tahoma"><span style="FONT-FAMILY: 'Tahoma','sans-serif'; COLOR: #1f497d; FONT-SIZE: 10pt"></span></font> </p>
</div>
<div>
<p class="MsoNormal"><font color="#1f497d" size="2" face="Tahoma"><span style="FONT-FAMILY: 'Tahoma','sans-serif'; COLOR: #1f497d; FONT-SIZE: 10pt">Greg de Valois</span></font></p>
</div>
<div>
<p class="MsoNormal"><font color="#1f497d" size="2" face="Tahoma"><span style="FONT-FAMILY: 'Tahoma','sans-serif'; COLOR: #1f497d; FONT-SIZE: 10pt">SanDisk</span></font><font color="black" size="2" face="Tahoma"><span style="FONT-FAMILY: 'Tahoma','sans-serif'; COLOR: black; FONT-SIZE: 10pt"></span></font></p>
</div>
</div>
</div>
<p class="MsoNormal"><font size="3" face="Times New Roman"><span style="FONT-FAMILY: 'Times New Roman','serif'; FONT-SIZE: 12pt"></span></font> </p>
<div style="TEXT-ALIGN: center" class="MsoNormal" align="center"><font size="3" face="Times New Roman"><span style="FONT-FAMILY: 'Times New Roman','serif'; FONT-SIZE: 12pt">
<hr align="center" size="3" width="100%">
</span></font></div>
<p style="MARGIN-BOTTOM: 12pt" class="MsoNormal"><font color="gray" size="2" face="Arial"><span style="FONT-FAMILY: 'Arial','sans-serif'; COLOR: gray; FONT-SIZE: 10pt">PLEASE NOTE: The information contained in this electronic mail message is intended only for
the use of the designated recipient(s) named above. If the reader of this message is not the intended recipient, you are hereby notified that you have received this message in error and that any review, dissemination, distribution, or copying of this message
is strictly prohibited. If you have received this communication in error, please notify the sender by telephone or e-mail (as shown above) immediately and destroy any and all copies of this message in your possession (whether hard copies or electronically
stored copies).</span></font><font size="3" face="Times New Roman"><span style="FONT-FAMILY: 'Times New Roman','serif'; FONT-SIZE: 12pt"></span></font></p>
</div>
</div>
</div>
</body>
</html>