<html xmlns:v="urn:schemas-microsoft-com:vml" xmlns:o="urn:schemas-microsoft-com:office:office" xmlns:w="urn:schemas-microsoft-com:office:word" xmlns:x="urn:schemas-microsoft-com:office:excel" xmlns:m="http://schemas.microsoft.com/office/2004/12/omml" xmlns="http://www.w3.org/TR/REC-html40">
<head>
<meta http-equiv="Content-Type" content="text/html; charset=utf-8">
<meta name="Generator" content="Microsoft Word 15 (filtered medium)">
<!--[if !mso]><style>v\:* {behavior:url(#default#VML);}
o\:* {behavior:url(#default#VML);}
w\:* {behavior:url(#default#VML);}
.shape {behavior:url(#default#VML);}
</style><![endif]-->
<title>Samsung Enterprise Portal mySingle</title>
<style><!--
/* Font Definitions */
@font-face
{font-family:"Cambria Math";
panose-1:2 4 5 3 5 4 6 3 2 4;}
@font-face
{font-family:Calibri;
panose-1:2 15 5 2 2 2 4 3 2 4;}
@font-face
{font-family:Verdana;
panose-1:2 11 6 4 3 5 4 4 2 4;}
/* Style Definitions */
p.MsoNormal, li.MsoNormal, div.MsoNormal
{mso-margin-top-alt:auto;
margin-right:0in;
mso-margin-bottom-alt:auto;
margin-left:0in;
font-size:12.0pt;
font-family:"Times New Roman",serif;
mso-believe-normal-left:yes;}
a:link, span.MsoHyperlink
{mso-style-priority:99;
color:#0563C1;
text-decoration:underline;}
a:visited, span.MsoHyperlinkFollowed
{mso-style-priority:99;
color:#954F72;
text-decoration:underline;}
p
{mso-style-priority:99;
margin-top:3.75pt;
margin-right:0in;
margin-bottom:3.75pt;
margin-left:0in;
font-size:9.0pt;
font-family:"Arial",sans-serif;}
p.MsoNoSpacing, li.MsoNoSpacing, div.MsoNoSpacing
{mso-style-priority:1;
mso-margin-top-alt:auto;
margin-right:0in;
mso-margin-bottom-alt:auto;
margin-left:0in;
font-size:12.0pt;
font-family:"Times New Roman",serif;}
p.MsoListParagraph, li.MsoListParagraph, div.MsoListParagraph
{mso-style-priority:34;
mso-margin-top-alt:auto;
margin-right:0in;
mso-margin-bottom-alt:auto;
margin-left:.5in;
font-size:12.0pt;
font-family:"Times New Roman",serif;}
p.p1, li.p1, div.p1
{mso-style-name:p1;
mso-style-priority:99;
margin-top:3.75pt;
margin-right:0in;
margin-bottom:3.75pt;
margin-left:0in;
font-size:9.0pt;
font-family:"Arial",sans-serif;}
span.apple-style-span
{mso-style-name:apple-style-span;}
span.EmailStyle22
{mso-style-type:personal;
font-family:"Calibri",sans-serif;
color:#1F497D;
font-weight:normal;
font-style:normal;}
span.EmailStyle23
{mso-style-type:personal;
font-family:"Calibri",sans-serif;
color:windowtext;
font-weight:normal;
font-style:normal;}
span.s1
{mso-style-name:s1;}
span.EmailStyle25
{mso-style-type:personal;
font-family:"Calibri",sans-serif;
color:#1F497D;}
span.EmailStyle26
{mso-style-type:personal;
font-family:"Calibri",sans-serif;
color:#1F497D;}
span.EmailStyle28
{mso-style-type:personal-reply;
font-family:"Calibri",sans-serif;
color:#1F497D;}
.MsoChpDefault
{mso-style-type:export-only;
font-size:10.0pt;}
@page WordSection1
{size:8.5in 11.0in;
margin:1.0in 1.0in 1.0in 1.0in;}
div.WordSection1
{page:WordSection1;}
--></style><![if mso 9]><style>p.MsoNormal
{margin-left:7.5pt;}
</style><![endif]><!--[if gte mso 9]><xml>
<o:shapedefaults v:ext="edit" spidmax="1026" />
</xml><![endif]--><!--[if gte mso 9]><xml>
<o:shapelayout v:ext="edit">
<o:idmap v:ext="edit" data="1" />
</o:shapelayout></xml><![endif]-->
</head>
<body lang="EN-US" link="#0563C1" vlink="#954F72" style="margin-left:7.5pt;margin-top:7.5pt;margin-right:7.5pt;margin-bottom:7.5pt">
<div class="WordSection1">
<p class="MsoNormal" style="margin:0in;margin-bottom:.0001pt"><span style="font-size:11.0pt;font-family:"Calibri",sans-serif;color:#1F497D">Hi Suman,
<o:p></o:p></span></p>
<p class="MsoNormal" style="margin:0in;margin-bottom:.0001pt"><span style="font-size:11.0pt;font-family:"Calibri",sans-serif;color:#1F497D">I have an update on our performance data. It appears on systems with more than 32 and more than 64 cores that there
was no performance delta in the following workloads:<o:p></o:p></span></p>
<p class="MsoNormal" style="margin:0in;margin-bottom:.0001pt"><span style="font-size:11.0pt;font-family:"Calibri",sans-serif;color:#1F497D">4 workers, 32 Queue Depth<o:p></o:p></span></p>
<p class="MsoNormal" style="margin:0in;margin-bottom:.0001pt"><span style="font-size:11.0pt;font-family:"Calibri",sans-serif;color:#1F497D">8 workers, 16 Queue Depth<o:p></o:p></span></p>
<p class="MsoNormal" style="margin:0in;margin-bottom:.0001pt"><span style="font-size:11.0pt;font-family:"Calibri",sans-serif;color:#1F497D"><o:p> </o:p></span></p>
<p class="MsoNormal" style="margin:0in;margin-bottom:.0001pt"><span style="font-size:11.0pt;font-family:"Calibri",sans-serif;color:#1F497D">We did observe what looks like a more consistent (and much smaller) 10% drop on workload with a single worker, single
queue depth, on systems with 32 cores.<o:p></o:p></span></p>
<p class="MsoNormal" style="margin:0in;margin-bottom:.0001pt"><span style="font-size:11.0pt;font-family:"Calibri",sans-serif;color:#1F497D"><o:p> </o:p></span></p>
<p class="MsoNormal" style="margin:0in;margin-bottom:.0001pt"><span style="font-size:11.0pt;font-family:"Calibri",sans-serif;color:#1F497D">It seems that our initial results might have been flawed and based on your comments and performance analysis, there may
be no major issue here.<o:p></o:p></span></p>
<p class="MsoNormal" style="margin:0in;margin-bottom:.0001pt"><span style="font-size:11.0pt;font-family:"Calibri",sans-serif;color:#1F497D"><o:p> </o:p></span></p>
<p class="MsoNormal" style="margin:0in;margin-bottom:.0001pt"><span style="font-size:11.0pt;font-family:"Calibri",sans-serif;color:#1F497D">Carolyn<o:p></o:p></span></p>
<p class="MsoNormal" style="margin:0in;margin-bottom:.0001pt"><span style="font-size:11.0pt;font-family:"Calibri",sans-serif;color:#1F497D"><o:p> </o:p></span></p>
<div>
<div style="border:none;border-top:solid #E1E1E1 1.0pt;padding:3.0pt 0in 0in 0in">
<p class="MsoNormal" style="margin:0in;margin-bottom:.0001pt"><b><span style="font-size:11.0pt;font-family:"Calibri",sans-serif">From:</span></b><span style="font-size:11.0pt;font-family:"Calibri",sans-serif"> Foster, Carolyn D
<br>
<b>Sent:</b> Wednesday, April 06, 2016 2:54 PM<br>
<b>To:</b> 'suman.p@samsung.com' <suman.p@samsung.com>; Thomas Freeman <thomas.freeman@hgst.com>; nvmewin@lists.openfabrics.org<br>
<b>Cc:</b> Seokhwan Kim <sukka.kim@samsung.com>; ANSHUL SHARMA <anshul@samsung.com>; MANOJ THAPLIYAL <m.thapliyal@samsung.com>; tru.nguyen@ssi.samsung.com<br>
<b>Subject:</b> RE: RE: [nvmewin] Patch with changes for Optimizing disk initialization performance<o:p></o:p></span></p>
</div>
</div>
<p class="MsoNormal"><o:p> </o:p></p>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:"Calibri",sans-serif;color:#1F497D">Hi Suman, thank you for the clarification. I will confirm the rest of the workload details and have that information for you tomorrow. In the mean time I will
also rerun our performance tests to confirm that the results are reproducible, and will run the tests without line 594 as you suggested.<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:"Calibri",sans-serif;color:#1F497D">Carolyn<o:p></o:p></span></p>
<div>
<div style="border:none;border-top:solid #E1E1E1 1.0pt;padding:3.0pt 0in 0in 0in">
<p class="MsoNormal" style="margin:0in;margin-bottom:.0001pt"><b><span style="font-size:11.0pt;font-family:"Calibri",sans-serif">From:</span></b><span style="font-size:11.0pt;font-family:"Calibri",sans-serif"> SUMAN PRAKASH B [<a href="mailto:suman.p@samsung.com">mailto:suman.p@samsung.com</a>]
<br>
<b>Sent:</b> Wednesday, April 06, 2016 9:01 AM<br>
<b>To:</b> Foster, Carolyn D <<a href="mailto:carolyn.d.foster@intel.com">carolyn.d.foster@intel.com</a>>; Thomas Freeman <<a href="mailto:thomas.freeman@hgst.com">thomas.freeman@hgst.com</a>>;
<a href="mailto:nvmewin@lists.openfabrics.org">nvmewin@lists.openfabrics.org</a><br>
<b>Cc:</b> Seokhwan Kim <<a href="mailto:sukka.kim@samsung.com">sukka.kim@samsung.com</a>>; ANSHUL SHARMA <<a href="mailto:anshul@samsung.com">anshul@samsung.com</a>>; MANOJ THAPLIYAL <<a href="mailto:m.thapliyal@samsung.com">m.thapliyal@samsung.com</a>>;
<a href="mailto:tru.nguyen@ssi.samsung.com">tru.nguyen@ssi.samsung.com</a><br>
<b>Subject:</b> Re: RE: [nvmewin] Patch with changes for Optimizing disk initialization performance<o:p></o:p></span></p>
</div>
</div>
<p class="MsoNormal"><o:p> </o:p></p>
<p><span style="font-size:11.0pt;font-family:"Calibri",sans-serif">Hi Carolyn,</span><o:p></o:p></p>
<p> <o:p></o:p></p>
<p><span style="font-size:11.0pt;font-family:"Calibri",sans-serif">Thanks for the comments and suggestions. Please find my comments below:</span><o:p></o:p></p>
<p> <o:p></o:p></p>
<p><strong><u><span style="font-size:11.0pt;font-family:"Calibri",sans-serif;color:black">1. Observed performance degradation, potentially related to line 594 in nvmeInit.c</span></u></strong><span style="font-size:11.0pt;font-family:"Calibri",sans-serif"><br>
a. We tested on Servers with 32 and 64 logical processors, Windows 8.1 and 2012 R2 OS, multiple vendor devices with both 1-to-1 core-queue mapping and many-to-1 core-queue mapping, with both R133 and latest drivers, and we did not observe any performance drop
for 100% Sequential Read(128K) with 32 and 64 worker threads, and queue depth 32. We have tested both the StorPortInitializePerfOpts() pass case and fail case(learning cores will be executed).</span><o:p></o:p></p>
<p><span style="font-size:11.0pt;font-family:"Calibri",sans-serif">b. Regarding MsgID-- in nvmeInit.c line number 594, NVMe device supports number of msg ids equal to 1 admin + N IO queues. But in OFA driver, since the msg id 0 is shared between admin queue
and io queue, always ((1 admin + N io queues) - 1) number of msgids is used. With MsgID--, we make sure all IO queues are created with unique msg id and msg id 0 is shared with admin queue and 1 io queue. If the device that you are testing has total number
of msg ids equal to 1 admin queue + N io queues, then MsgID-- should not be a problem. But if you strongly feel that MsgID-- could be an issue, can you please take the perf benchmark after removing MsgID--.</span><o:p></o:p></p>
<p><span style="font-size:11.0pt;font-family:"Calibri",sans-serif">c. Can you please let us know the number of queues and number of Messaged IDs supported by the target device that you are testing with.</span><o:p></o:p></p>
<p><span style="font-size:11.0pt;font-family:"Calibri",sans-serif">d. On servers, we usually test on Windows Server edition OSes. When we tested with Windows 8.1, we observed that the number of logical processors supported in Windows 8.1 is maximum 32, even
when the server have more than 32 logical processors.</span><o:p></o:p></p>
<p><span style="font-size:11.0pt;font-family:"Calibri",sans-serif">We will try to reproduce the performance degradation behavior, meanwhile
<strong><u><span style="font-family:"Calibri",sans-serif">can you please provide us more debug data</span></u></strong>.</span><o:p></o:p></p>
<p> <o:p></o:p></p>
<p><strong><u><span style="font-size:11.0pt;font-family:"Calibri",sans-serif">2. nvmeStat.c @ line 784
</span></u></strong><span style="font-size:11.0pt;font-family:"Calibri",sans-serif">: Agreed. We will change as suggested.</span><o:p></o:p></p>
<p> <o:p></o:p></p>
<p><strong><u><span style="font-size:11.0pt;font-family:"Calibri",sans-serif">3. nvmeStat.c @line 899</span></u></strong><span style="font-size:11.0pt;font-family:"Calibri",sans-serif"> : Agreed. We will change as suggested.</span><o:p></o:p></p>
<p> <o:p></o:p></p>
<p><span style="font-size:11.0pt;font-family:"Calibri",sans-serif">Thanks,<br>
Suman</span><o:p></o:p></p>
<p> <o:p></o:p></p>
<p>------- <b>Original Message</b> -------<o:p></o:p></p>
<p><b>Sender</b> : Foster, Carolyn D<<a href="mailto:carolyn.d.foster@intel.com">carolyn.d.foster@intel.com</a>><o:p></o:p></p>
<p><b>Date</b> : Apr 06, 2016 05:33 (GMT+05:30)<o:p></o:p></p>
<p><b>Title</b> : RE: [nvmewin] Patch with changes for Optimizing disk initialization performance<o:p></o:p></p>
<p> <o:p></o:p></p>
<p class="MsoNormalCxSpFirst" style="margin:0in;margin-bottom:.0001pt;mso-add-space:auto">
<span style="font-size:11.0pt;font-family:"Calibri",sans-serif;color:#1F497D">Hi Suman,<o:p></o:p></span></p>
<p class="MsoNormalCxSpMiddle" style="margin:0in;margin-bottom:.0001pt;mso-add-space:auto">
<span style="font-size:11.0pt;font-family:"Calibri",sans-serif;color:#1F497D"><o:p> </o:p></span></p>
<p class="MsoNormalCxSpLast" style="margin:0in;margin-bottom:.0001pt;mso-add-space:auto">
<span style="font-size:11.0pt;font-family:"Calibri",sans-serif;color:#1F497D">I have a few comments and suggestions:<o:p></o:p></span></p>
<p class="MsoListParagraphCxSpFirst" style="mso-margin-top-alt:7.5pt;margin-right:7.5pt;margin-bottom:7.5pt;margin-left:.5in;mso-add-space:auto;text-indent:-.25in">
<span style="font-size:11.0pt;font-family:"Calibri",sans-serif;color:#1F497D">1.</span><span style="font-size:7.0pt;color:#1F497D">
</span><span style="font-size:11.0pt;font-family:"Calibri",sans-serif;color:#1F497D">Observed performance degradation, potentially related to line 594 in nvmeInit.c – We noticed on some systems we saw a degradation in performance and we suspect it’s related
to this change. If we don’t share MSIX vector 0 between the admin queue and an IO queue we are creating one fewer queue to submit IO to. Did you execute any performance testing before and after these changes? I have included some details about the system
configurations we tested and the observed results below.<o:p></o:p></span></p>
<p class="MsoListParagraphCxSpMiddle" style="mso-margin-top-alt:0in;margin-right:0in;margin-bottom:0in;margin-left:.5in;margin-bottom:.0001pt;mso-add-space:auto;text-indent:-.25in">
<span style="font-size:11.0pt;font-family:"Calibri",sans-serif;color:#1F497D">2.</span><span style="font-size:7.0pt;color:#1F497D">
</span><span style="font-size:11.0pt;font-family:"Calibri",sans-serif;color:#1F497D">nvmeStat.c @ line 784 : if there is zero namespace it is not necessary to go to NVMeWaitOnNamespaceReady , instead we can directly start NVMeStartComplete.<o:p></o:p></span></p>
<p class="MsoListParagraphCxSpLast" style="mso-margin-top-alt:0in;margin-right:0in;margin-bottom:0in;margin-left:.5in;margin-bottom:.0001pt;mso-add-space:auto;text-indent:-.25in">
<span style="font-size:11.0pt;font-family:"Calibri",sans-serif;color:#1F497D">3.</span><span style="font-size:7.0pt;color:#1F497D">
</span><span style="font-size:11.0pt;font-family:"Calibri",sans-serif;color:#1F497D">nvmeStat.c @line 899 : In crash/Hibernate mode it is not necessary to go to the NVMeWaitOnNamespaceReady.<o:p></o:p></span></p>
<p class="MsoNormalCxSpFirst" style="margin:0in;margin-bottom:.0001pt;mso-add-space:auto">
<span style="font-size:11.0pt;font-family:"Calibri",sans-serif;color:#1F497D"><o:p> </o:p></span></p>
<p class="MsoNormalCxSpMiddle" style="margin:0in;margin-bottom:.0001pt;mso-add-space:auto">
<span style="font-size:11.0pt;font-family:"Calibri",sans-serif;color:#1F497D"><o:p> </o:p></span></p>
<p class="MsoNormalCxSpMiddle" style="margin:0in;margin-bottom:.0001pt;mso-add-space:auto">
<span style="font-size:11.0pt;font-family:"Calibri",sans-serif;color:#1F497D">Performance configuration and data:<o:p></o:p></span></p>
<p class="MsoNormalCxSpMiddle" style="margin:0in;margin-bottom:.0001pt;mso-add-space:auto">
<span style="font-size:11.0pt;font-family:"Calibri",sans-serif;color:#1F497D"><o:p> </o:p></span></p>
<p class="MsoNormalCxSpMiddle" style="margin:0in;margin-bottom:.0001pt;mso-add-space:auto">
<span style="font-size:11.0pt;font-family:"Calibri",sans-serif;color:#1F497D">OS: Windows 8.1 x64<o:p></o:p></span></p>
<p class="MsoNormalCxSpMiddle" style="margin:0in;margin-bottom:.0001pt;mso-add-space:auto">
<span style="font-size:11.0pt;font-family:"Calibri",sans-serif;color:#1F497D">Workload: 100% sequential Read<o:p></o:p></span></p>
<p class="MsoNormalCxSpMiddle" style="margin:0in;margin-bottom:.0001pt;mso-add-space:auto">
<span style="font-size:11.0pt;font-family:"Calibri",sans-serif;color:#1F497D">Compared the OFA trunk to the Samsung patch<o:p></o:p></span></p>
<p class="MsoNormalCxSpMiddle" style="mso-margin-top-alt:7.5pt;margin-right:15.0pt;margin-bottom:7.5pt;margin-left:0in;mso-add-space:auto">
<span style="font-size:11.0pt;font-family:"Calibri",sans-serif;color:#1F497D"><o:p> </o:p></span></p>
<p class="MsoNormalCxSpLast" style="mso-margin-top-alt:7.5pt;margin-right:15.0pt;margin-bottom:7.5pt;margin-left:0in;mso-add-space:auto">
<span style="font-size:11.0pt;font-family:"Calibri",sans-serif;color:#1F497D">Summary of observed results:<o:p></o:p></span></p>
<p class="MsoListParagraphCxSpFirst" style="mso-margin-top-alt:7.5pt;margin-right:45.0pt;margin-bottom:7.5pt;margin-left:.25in;mso-add-space:auto;text-indent:-.25in">
<span style="font-size:11.0pt;font-family:Symbol;color:#1F497D">·</span><span style="font-size:7.0pt;color:#1F497D">
</span><span style="font-size:11.0pt;font-family:"Calibri",sans-serif;color:#1F497D">System with fewer than 32 logical CPU cores:
<o:p></o:p></span></p>
<p class="MsoListParagraphCxSpMiddle" style="mso-margin-top-alt:7.5pt;margin-right:52.5pt;margin-bottom:7.5pt;margin-left:.75in;mso-add-space:auto;text-indent:-.25in">
<span style="font-size:11.0pt;font-family:"Courier New";color:#1F497D">o</span><span style="font-size:7.0pt;color:#1F497D">
</span><span style="font-size:11.0pt;font-family:"Calibri",sans-serif;color:#1F497D">No delta in performance observed<o:p></o:p></span></p>
<p class="MsoListParagraphCxSpMiddle" style="mso-margin-top-alt:7.5pt;margin-right:52.5pt;margin-bottom:7.5pt;margin-left:.25in;mso-add-space:auto;text-indent:-.25in">
<span style="font-size:11.0pt;font-family:Symbol;color:#1F497D">·</span><span style="font-size:7.0pt;color:#1F497D">
</span><span style="font-size:11.0pt;font-family:"Calibri",sans-serif;color:#1F497D">System with between 32 and 64 cores:
<o:p></o:p></span></p>
<p class="MsoListParagraphCxSpMiddle" style="mso-margin-top-alt:7.5pt;margin-right:22.5pt;margin-bottom:7.5pt;margin-left:.75in;mso-add-space:auto;text-indent:-.25in">
<span style="font-size:11.0pt;font-family:"Courier New";color:#1F497D">o</span><span style="font-size:7.0pt;color:#1F497D">
</span><span style="font-size:11.0pt;font-family:"Calibri",sans-serif;color:#1F497D">20%-50% drop in performance observed<o:p></o:p></span></p>
<p class="MsoListParagraphCxSpMiddle" style="mso-margin-top-alt:7.5pt;margin-right:22.5pt;margin-bottom:7.5pt;margin-left:.25in;mso-add-space:auto;text-indent:-.25in">
<span style="font-size:11.0pt;font-family:Symbol;color:#1F497D">·</span><span style="font-size:7.0pt;color:#1F497D">
</span><span style="font-size:11.0pt;font-family:"Calibri",sans-serif;color:#1F497D">System with more than 64 cores:<o:p></o:p></span></p>
<p class="MsoListParagraphCxSpLast" style="mso-margin-top-alt:7.5pt;margin-right:22.5pt;margin-bottom:7.5pt;margin-left:.75in;mso-add-space:auto;text-indent:-.25in">
<span style="font-size:11.0pt;font-family:"Courier New";color:#1F497D">o</span><span style="font-size:7.0pt;color:#1F497D">
</span><span style="font-size:11.0pt;font-family:"Calibri",sans-serif;color:#1F497D">30%-40% drop in performance observed<o:p></o:p></span></p>
<p class="MsoNormalCxSpFirst" style="margin:0in;margin-bottom:.0001pt;mso-add-space:auto">
<span style="font-size:11.0pt;font-family:"Calibri",sans-serif;color:#1F497D"><o:p> </o:p></span></p>
<p class="MsoNormalCxSpMiddle" style="margin:0in;margin-bottom:.0001pt;mso-add-space:auto">
<span style="font-size:11.0pt;font-family:"Calibri",sans-serif;color:#1F497D">Carolyn<o:p></o:p></span></p>
<p class="MsoNormalCxSpMiddle" style="margin:0in;margin-bottom:.0001pt;mso-add-space:auto">
<span style="font-size:11.0pt;font-family:"Calibri",sans-serif;color:#1F497D"><o:p> </o:p></span></p>
<div>
<div style="border:none;border-top:solid #E1E1E1 1.0pt;padding:3.0pt 0in 0in 0in">
<p class="MsoNormal" style="margin:0in;margin-bottom:.0001pt"><b><span style="font-size:11.0pt;font-family:"Calibri",sans-serif">From:</span></b><span style="font-size:11.0pt;font-family:"Calibri",sans-serif"> nvmewin [<a href="mailto:nvmewin-bounces@lists.openfabrics.org">mailto:nvmewin-bounces@lists.openfabrics.org</a>]
<b>On Behalf Of </b>SUMAN PRAKASH B<br>
<b>Sent:</b> Monday, April 04, 2016 6:35 AM<br>
<b>To:</b> Thomas Freeman <<a href="mailto:thomas.freeman@hgst.com">thomas.freeman@hgst.com</a>>;
<a href="mailto:nvmewin@lists.openfabrics.org">nvmewin@lists.openfabrics.org</a><br>
<b>Cc:</b> Seokhwan Kim <<a href="mailto:sukka.kim@samsung.com">sukka.kim@samsung.com</a>>;
<a href="mailto:anshul@samsung.com">anshul@samsung.com</a>; MANOJ THAPLIYAL <<a href="mailto:m.thapliyal@samsung.com">m.thapliyal@samsung.com</a>>;
<a href="mailto:tru.nguyen@ssi.samsung.com">tru.nguyen@ssi.samsung.com</a><br>
<b>Subject:</b> Re: [nvmewin] Patch with changes for Optimizing disk initialization performance<o:p></o:p></span></p>
</div>
</div>
<p class="MsoNormal"><o:p> </o:p></p>
<p><span class="apple-style-span"><span style="font-size:11.0pt;font-family:"Calibri",sans-serif;color:black">Hi all,</span></span><o:p></o:p></p>
<p> <o:p></o:p></p>
<p><span class="apple-style-span"><span style="font-size:11.0pt;font-family:"Calibri",sans-serif;color:black">I am sending the updated patch incorporating feedback from Tom. The changes are listed below. The password is samsungnvme</span></span><o:p></o:p></p>
<p> <o:p></o:p></p>
<p><span class="apple-style-span"><span style="font-size:11.0pt;font-family:"Calibri",sans-serif;color:black">1. Moved the StorPortFreePool()from IoCompletionRoutine() to NvmeInitCallback() - NvmeWaitOnNamespaceReady.</span></span><o:p></o:p></p>
<p><span class="apple-style-span"><span style="font-size:11.0pt;font-family:"Calibri",sans-serif;color:black">2. In NVMeRunningWaitOnNamespaceReady(), the READ command will be sent only when the NS is ATTACHED and the pLunExt->slotStatus is ONLINE. pLunExt->slotStatus
willnot be ONLINE if lun extension is zero'ed out.</span></span><o:p></o:p></p>
<p><span class="apple-style-span"><span style="font-size:11.0pt;font-family:"Calibri",sans-serif;color:black">3. In NvmeInitCallBack(), in case NVMeWaitOnNamespaceReady, the READ will be retried only if SC = 0x82, else move to the next NS. If the NS LBA format
is unsupported, miniport sends the READ command, for which device will return SC=0xb, and miniport will move to next NS.</span></span><o:p></o:p></p>
<p> <o:p></o:p></p>
<p><strong><span style="font-size:11.0pt;font-family:"Calibri",sans-serif;color:black">To Mandatory reviewers</span></strong><span class="apple-style-span"><span style="font-size:11.0pt;font-family:"Calibri",sans-serif;color:black">: Can we get feedback or
approval for this patch before 7th April.</span></span><o:p></o:p></p>
<p><span style="font-size:11.0pt;font-family:"Calibri",sans-serif;color:black"> </span><o:p></o:p></p>
<p><span style="font-size:11.0pt;font-family:"Calibri",sans-serif;color:black">Thanks,</span><o:p></o:p></p>
<p><span style="font-size:11.0pt;font-family:"Calibri",sans-serif;color:black">Suman</span><span style="color:black"><o:p></o:p></span></p>
<p> <o:p></o:p></p>
<p>------- <b>Original Message</b> -------<o:p></o:p></p>
<p><b>Sender</b> : SUMAN PRAKASH B<<a href="mailto:suman.p@samsung.com">suman.p@samsung.com</a>> Senior Chief Engineer/SSIR-SSD Solutions/Samsung Electronics<o:p></o:p></p>
<p><b>Date</b> : Mar 29, 2016 20:27 (GMT+05:30)<o:p></o:p></p>
<p><b>Title</b> : Re: RE: [nvmewin] Patch with changes for Optimizing disk initialization performance<o:p></o:p></p>
<p> <o:p></o:p></p>
<p><span style="font-size:11.0pt;font-family:"Calibri",sans-serif">Hi Tom,</span><o:p></o:p></p>
<p> <o:p></o:p></p>
<p><span style="font-size:11.0pt;font-family:"Calibri",sans-serif">Thanks for the review comments. Please find my replies below:</span><o:p></o:p></p>
<p> <o:p></o:p></p>
<p><span style="font-size:11.0pt;font-family:"Calibri",sans-serif">1. nvmeStd.c::IoCompletionRoutine, when checking for NVMeWaitOnNamespaceReady, would it be better to make this check and free the buffer in NvmeInitCallback (when processing NVMeWaitOnNamespaceReady).
The check in IoCompletionRoutine is executed during the processing of every IO command, but it will only ever be TRUE during initialization.<br>
<strong><u><span style="font-family:"Calibri",sans-serif;color:blue">[Suman]</span></u></strong><strong><span style="font-family:"Calibri",sans-serif;color:blue">
</span></strong><span style="color:blue">Agreed. We will move the StorPortFreePool to NvmeInitCallback.</span></span><o:p></o:p></p>
<p> <o:p></o:p></p>
<p><span style="font-size:11.0pt;font-family:"Calibri",sans-serif">2.a. With your change, in the method NVMeRunningWaitOnNamespaceReady the driver picks the next lun in the list and issues a READ to that namespace.
<br>
<strong><u><span style="font-family:"Calibri",sans-serif;color:blue">[Suman]</span></u></strong><span style="color:blue"> Following changes are made:<br>
a. In NVMeRunningWaitOnNamespaceReady(), the READ command will be sent only when the NS is ATTACHED and the pLunExt->slotStatus is ONLINE. pLunExt->slotStatus will not be ONLINE if lun extension is zero'ed out.<br>
b. In NvmeInitCallBack(), in case NVMeWaitOnNamespaceReady, the READ will be retried only if SC = 0x82, else move to the next NS. If the NS LBA format is unsupported, miniport sends the READ command, for which device will return SC=0xb, and miniport will move
to next NS.</span></span><o:p></o:p></p>
<p><span style="font-size:11.0pt;font-family:"Calibri",sans-serif">Let me know if you have any questions.</span><o:p></o:p></p>
<p><span style="font-size:11.0pt;font-family:"Calibri",sans-serif"><br>
We will share the modified code once others share their feedback. Can we get feedback from other companies by 5th April?</span><o:p></o:p></p>
<p><span style="font-size:11.0pt;font-family:"Calibri",sans-serif"><br>
Thanks,<br>
Suman</span><o:p></o:p></p>
<p> <o:p></o:p></p>
<p>------- <b>Original Message</b> -------<o:p></o:p></p>
<p><b>Sender</b> : Thomas Freeman<<a href="mailto:thomas.freeman@hgst.com">thomas.freeman@hgst.com</a>><o:p></o:p></p>
<p><b>Date</b> : Mar 29, 2016 00:48 (GMT+05:30)<o:p></o:p></p>
<p><b>Title</b> : RE: [nvmewin] Patch with changes for Optimizing disk initialization performance<o:p></o:p></p>
<p> <o:p></o:p></p>
<p class="MsoNormal" style="mso-margin-top-alt:0in;margin-right:7.5pt;margin-bottom:0in;margin-left:0in;margin-bottom:.0001pt">
<span style="font-size:11.0pt;font-family:"Calibri",sans-serif;color:#1F497D">Hi Suman,<o:p></o:p></span></p>
<p class="MsoNormal" style="mso-margin-top-alt:0in;margin-right:15.0pt;margin-bottom:0in;margin-left:0in;margin-bottom:.0001pt">
<span style="font-size:11.0pt;font-family:"Calibri",sans-serif;color:#1F497D">It looks good.
<o:p></o:p></span></p>
<p class="MsoNormal" style="mso-margin-top-alt:0in;margin-right:15.0pt;margin-bottom:0in;margin-left:0in;margin-bottom:.0001pt">
<span style="font-size:11.0pt;font-family:"Calibri",sans-serif;color:#1F497D">I have a few comments here:<o:p></o:p></span></p>
<p class="MsoListParagraph" style="mso-margin-top-alt:5.0pt;margin-right:22.5pt;margin-bottom:5.0pt;margin-left:10.5pt;text-indent:-.25in">
<span style="font-size:11.0pt;font-family:"Calibri",sans-serif;color:#1F497D">1.</span><span style="font-size:7.0pt;color:#1F497D">
</span><span style="font-size:11.0pt;font-family:"Calibri",sans-serif;color:#1F497D">nvmeStd.c::IoCompletionRoutine, when checking for NVMeWaitOnNamespaceReady, would it be better to make this check and free the buffer in NvmeInitCallback (when processing NVMeWaitOnNamespaceReady).
The check in IoCompletionRoutine is executed during the processing of every IO command, but it will only ever be TRUE during initialization.<o:p></o:p></span></p>
<p class="MsoListParagraph" style="mso-margin-top-alt:5.0pt;margin-right:37.5pt;margin-bottom:5.0pt;margin-left:10.5pt;text-indent:-.25in">
<span style="font-size:11.0pt;font-family:"Calibri",sans-serif;color:#1F497D">2.</span><span style="font-size:7.0pt;color:#1F497D">
</span><span style="font-size:11.0pt;font-family:"Calibri",sans-serif;color:#1F497D">I ran into a few problems, here are the details:<o:p></o:p></span></p>
<p class="MsoListParagraph" style="mso-margin-top-alt:5.0pt;margin-right:37.5pt;margin-bottom:5.0pt;margin-left:10.5pt">
<span style="font-size:11.0pt;font-family:"Calibri",sans-serif;color:#1F497D">*My device configuration: I'm testing a device that supports NS management and it has multiple namespaces. Some of those namespaces are not attached. The format of some of those namespaces
is not supported by the driver (e.g. LBA contains metadata) <o:p></o:p></span></p>
<p class="MsoListParagraph" style="mso-margin-top-alt:5.0pt;margin-right:15.0pt;margin-bottom:5.0pt;margin-left:46.5pt;text-indent:-.25in">
<span style="font-size:11.0pt;font-family:"Calibri",sans-serif;color:#1F497D">a.</span><span style="font-size:7.0pt;color:#1F497D">
</span><span style="font-size:11.0pt;font-family:"Calibri",sans-serif;color:#1F497D">With your change, in the method NVMeRunningWaitOnNamespaceReady the driver picks the next lun in the list and issues a READ to that namespace.
<o:p></o:p></span></p>
<p class="MsoListParagraph" style="mso-margin-top-alt:5.0pt;margin-right:15.0pt;margin-bottom:5.0pt;margin-left:82.5pt;text-indent:-9.0pt">
<span style="font-size:7.0pt;color:#1F497D">
</span><span style="font-size:11.0pt;font-family:"Calibri",sans-serif;color:#1F497D">i.</span><span style="font-size:7.0pt;color:#1F497D">
</span><span style="font-size:11.0pt;font-family:"Calibri",sans-serif;color:#1F497D">If that lun is a detached namespace, the READ fails with a status code of 0xb. The driver attempts to retry until the READ is successful, but the command will never succeed.
<o:p></o:p></span></p>
<p class="MsoListParagraph" style="mso-margin-top-alt:5.0pt;margin-right:15.0pt;margin-bottom:5.0pt;margin-left:82.5pt;text-indent:-9.0pt">
<span style="font-size:7.0pt;color:#1F497D">
</span><span style="font-size:11.0pt;font-family:"Calibri",sans-serif;color:#1F497D">ii.</span><span style="font-size:7.0pt;color:#1F497D">
</span><span style="font-size:11.0pt;font-family:"Calibri",sans-serif;color:#1F497D">During initialization, if the driver detects a namespace that is in an unsupported format, it zero's out that LUN entry, but leaves that zero’ed entry in the LUN extension
list. When NVMeRunningWaitOnNamespaceReady is processing the list, it does not recognize this as a zero'ed out entry. Rather is attempts a READ from this namespace (the NSID is 0 since the init code zero'ed out that Lun list entry). The READ and all of its
retries fail with a status code of 0xb.<o:p></o:p></span></p>
<p class="MsoNormal" style="margin-right:7.5pt"><span style="font-size:11.0pt;font-family:"Calibri",sans-serif;color:#1F497D">Proposed fix:<o:p></o:p></span></p>
<p class="MsoListParagraph" style="mso-margin-top-alt:5.0pt;margin-right:30.0pt;margin-bottom:5.0pt;margin-left:.25in;text-indent:-.25in">
<span style="font-size:11.0pt;font-family:"Calibri",sans-serif;color:#1F497D">1.</span><span style="font-size:7.0pt;color:#1F497D">
</span><span style="font-size:11.0pt;font-family:"Calibri",sans-serif;color:#1F497D">Before issuing a READ, ensure that namespace is attached and a valid format. If not, increment the counters and move to the next Lun.
<o:p></o:p></span></p>
<p class="MsoListParagraph" style="mso-margin-top-alt:5.0pt;margin-right:30.0pt;margin-bottom:5.0pt;margin-left:.25in;text-indent:-.25in">
<span style="font-size:11.0pt;font-family:"Calibri",sans-serif;color:#1F497D">2.</span><span style="font-size:7.0pt;color:#1F497D">
</span><span style="font-size:11.0pt;font-family:"Calibri",sans-serif;color:#1F497D">Also, in NVMeInitCallback, when handling the case NVMeWaitOnNamespaceReady, instead of looking for an SC of 0x00, only issue a retry if the command fails with SC = 0x82 (NS
not ready). <o:p></o:p></span></p>
<p class="MsoNormal" style="margin-right:37.5pt"><span style="font-size:11.0pt;font-family:"Calibri",sans-serif;color:#1F497D">Let me know if you have any questions.<o:p></o:p></span></p>
<div>
<p class="MsoNormal" style="margin:0in;margin-bottom:.0001pt"><b><span style="font-size:10.0pt;font-family:"Arial",sans-serif;color:#1F497D">Tom Freeman<o:p></o:p></span></b></p>
<p class="MsoNormal" style="margin:0in;margin-bottom:.0001pt"><b><span style="font-size:10.0pt;font-family:"Arial",sans-serif;color:#1F497D">Software Engineer, Device Manager and Driver Development<o:p></o:p></span></b></p>
<p class="MsoNormal" style="margin:0in;margin-bottom:.0001pt"><span style="font-size:10.0pt;font-family:"Arial",sans-serif;color:#1F497D">Western Digital Corporation<o:p></o:p></span></p>
<p class="MsoNormal" style="margin:0in;margin-bottom:.0001pt"><span style="font-size:10.0pt;font-family:"Arial",sans-serif;color:#1F497D">e.
<a href="mailto:Thomas.freeman@hgst.com">Thomas.freeman@hgst.com</a><o:p></o:p></span></p>
<p class="MsoNormal" style="margin:0in;margin-bottom:.0001pt"><span style="font-size:10.0pt;font-family:"Arial",sans-serif;color:#1F497D">o. +1-507-322-2311<o:p></o:p></span></p>
<p class="MsoNormal" style="margin:0in;margin-bottom:.0001pt"><span style="font-size:11.0pt;font-family:"Calibri",sans-serif;color:#1F497D"><o:p> </o:p></span></p>
<p class="MsoNormal" style="margin:0in;margin-bottom:.0001pt"><span style="font-size:11.0pt;font-family:"Calibri",sans-serif;color:#1F497D"><img border="0" width="150" height="47" id="Picture_x0020_5" src="cid:image001.jpg@01D190D5.DF18E700" alt="http://www.samsung.net/service/ml/AttachController/image001.jpg?cmd=downdirectly&filepath=/LOCAL/ML/CACHE/s/20160406/image001.jpg@01D18E55.4C7BA130309suman.p&contentType=IMAGE/JPEG;charset=UTF8&msgno=309&partno=2&foldername=INBOX&msguid=48143"><o:p></o:p></span></p>
</div>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:"Calibri",sans-serif;color:#1F497D"><o:p> </o:p></span></p>
<div>
<div style="border:none;border-top:solid #E1E1E1 1.0pt;padding:3.0pt 0in 0in 0in">
<p class="MsoNormal" style="margin:0in;margin-bottom:.0001pt"><b><span style="font-size:11.0pt;font-family:"Calibri",sans-serif">From:</span></b><span style="font-size:11.0pt;font-family:"Calibri",sans-serif"> nvmewin [<a href="mailto:nvmewin-bounces@lists.openfabrics.org">mailto:nvmewin-bounces@lists.openfabrics.org</a>]
<b>On Behalf Of </b>SUMAN PRAKASH B<br>
<b>Sent:</b> Wednesday, March 23, 2016 7:27 AM<br>
<b>To:</b> <a href="mailto:nvmewin@lists.openfabrics.org">nvmewin@lists.openfabrics.org</a><br>
<b>Cc:</b> Seokhwan Kim <<a href="mailto:sukka.kim@samsung.com">sukka.kim@samsung.com</a>>; MANOJ THAPLIYAL <<a href="mailto:m.thapliyal@samsung.com">m.thapliyal@samsung.com</a>>;
<a href="mailto:tru.nguyen@ssi.samsung.com">tru.nguyen@ssi.samsung.com</a><br>
<b>Subject:</b> [nvmewin] Patch with changes for Optimizing disk initialization performance<o:p></o:p></span></p>
</div>
</div>
<p class="MsoNormal"><o:p> </o:p></p>
<p><span class="apple-style-span"><span style="font-size:11.5pt;font-family:"Calibri",sans-serif;color:black">Hi all,</span></span><o:p></o:p></p>
<p> <o:p></o:p></p>
<p><span class="apple-style-span"><span style="font-size:11.5pt;font-family:"Calibri",sans-serif;color:black">This patch includes changes for
</span></span><span class="apple-style-span"><span style="font-size:11.0pt;font-family:"Calibri",sans-serif;color:black">optimizing the disk initialization performance and relevant changes.</span></span><o:p></o:p></p>
<p><span class="apple-style-span"><span style="font-size:11.5pt;font-family:"Calibri",sans-serif;color:black">I have made a detailed overview of the changes in the attached doc file(the contents are also copied here below) and the attached zip file contains
the source code.</span></span><o:p></o:p></p>
<p> <o:p></o:p></p>
<p><span class="apple-style-span"><span style="font-size:11.5pt;font-family:"Calibri",sans-serif;color:black">Password is samsungnvme</span></span><o:p></o:p></p>
<p> <o:p></o:p></p>
<p><span class="apple-style-span"><span style="font-size:11.5pt;font-family:"Calibri",sans-serif;color:black">Please let me know if you have any questions.<o:p></o:p></span></span></p>
<p> <o:p></o:p></p>
<p><span style="font-size:11.0pt;font-family:"Calibri",sans-serif;color:black">Thanks,</span><span style="color:black"><o:p></o:p></span></p>
<p><span style="font-size:11.0pt;font-family:"Calibri",sans-serif;color:black">Suman</span><span style="color:black"><o:p></o:p></span></p>
<p> <o:p></o:p></p>
<p><span class="apple-style-span"><span style="font-size:11.5pt;font-family:"Calibri",sans-serif;color:black">******************</span></span><o:p></o:p></p>
<p><span class="apple-style-span"><span style="font-size:12.0pt;font-family:"Verdana",sans-serif;color:black"> <o:p></o:p></span></span></p>
<p><b><u><span style="font-size:11.0pt;font-family:"Calibri",sans-serif">Disk Initialization Performance Optimization:</span></u></b><span style="font-size:11.0pt;font-family:"Calibri",sans-serif"><o:p></o:p></span></p>
<p><span style="font-size:11.0pt;font-family:"Calibri",sans-serif"><br>
We can use the StorPortInitializePerfOpts(), PERF_CONFIGURATION_DATA.MessageTargets which provides the array of MSI numbers corresponding to each logical processor. This is an alternative of using the Learning cores logic implemented in the OFA driver.<o:p></o:p></span></p>
<p><span style="font-size:11.0pt;font-family:"Calibri",sans-serif"> <o:p></o:p></span></p>
<p><span style="font-size:11.0pt;font-family:"Calibri",sans-serif">Also this will directly reduce the time taken for the disk to be enumerated after a device hot insert.<o:p></o:p></span></p>
<p><span style="font-size:11.0pt;font-family:"Calibri",sans-serif">The current OFA driver does the following in its initialization path, let’s say on a server which has 32 logical processors and device which supports 32 queues –<o:p></o:p></span></p>
<p style="margin-left:.5in;mso-add-space:auto;text-indent:-.25in"><span style="font-size:11.0pt;font-family:"Calibri",sans-serif">1.</span><span style="font-size:7.0pt;font-family:"Times New Roman",serif">
</span><span style="font-size:11.0pt;font-family:"Calibri",sans-serif">Identify controller<o:p></o:p></span></p>
<p style="margin-left:.5in;mso-add-space:auto;text-indent:-.25in"><span style="font-size:11.0pt;font-family:"Calibri",sans-serif">2.</span><span style="font-size:7.0pt;font-family:"Times New Roman",serif">
</span><span style="font-size:11.0pt;font-family:"Calibri",sans-serif">Identify namespace - for N number of namespaces<o:p></o:p></span></p>
<p style="margin-left:.5in;mso-add-space:auto;text-indent:-.25in"><span style="font-size:11.0pt;font-family:"Calibri",sans-serif">3.</span><span style="font-size:7.0pt;font-family:"Times New Roman",serif">
</span><span style="font-size:11.0pt;font-family:"Calibri",sans-serif">Set features - Interrupt coalescing, number of queues, lba range type.<o:p></o:p></span></p>
<p style="margin-left:.5in;mso-add-space:auto;text-indent:-.25in"><span style="font-size:11.0pt;font-family:"Calibri",sans-serif">4.</span><span style="font-size:7.0pt;font-family:"Times New Roman",serif">
</span><span style="font-size:11.0pt;font-family:"Calibri",sans-serif">Create IO completion queue - 32 commands<o:p></o:p></span></p>
<p style="margin-left:.5in;mso-add-space:auto;text-indent:-.25in"><span style="font-size:11.0pt;font-family:"Calibri",sans-serif">5.</span><span style="font-size:7.0pt;font-family:"Times New Roman",serif">
</span><span style="font-size:11.0pt;font-family:"Calibri",sans-serif">Create IO submission queue - 32 commands<o:p></o:p></span></p>
<p style="margin-left:.5in;mso-add-space:auto;text-indent:-.25in"><span style="font-size:11.0pt;font-family:"Calibri",sans-serif">6.</span><span style="font-size:7.0pt;font-family:"Times New Roman",serif">
</span><span style="font-size:11.0pt;font-family:"Calibri",sans-serif">LearnMapping - 32 Read commands<o:p></o:p></span></p>
<p style="margin-left:.5in;mso-add-space:auto;text-indent:-.25in"><span style="font-size:11.0pt;font-family:"Calibri",sans-serif">7.</span><span style="font-size:7.0pt;font-family:"Times New Roman",serif">
</span><span style="font-size:11.0pt;font-family:"Calibri",sans-serif">ReSetupQueues - 32 Delete Sub queues + 32 Delete completion queues<o:p></o:p></span></p>
<p style="margin-left:.5in;mso-add-space:auto;text-indent:-.25in"><span style="font-size:11.0pt;font-family:"Calibri",sans-serif">8.</span><span style="font-size:7.0pt;font-family:"Times New Roman",serif">
</span><span style="font-size:11.0pt;font-family:"Calibri",sans-serif">Create IO completion queue - 32 commands<o:p></o:p></span></p>
<p style="margin-left:.5in;mso-add-space:auto;text-indent:-.25in"><span style="font-size:11.0pt;font-family:"Calibri",sans-serif">9.</span><span style="font-size:7.0pt;font-family:"Times New Roman",serif">
</span><span style="font-size:11.0pt;font-family:"Calibri",sans-serif">Create IO submission queue - 32 commands<o:p></o:p></span></p>
<p style="margin-left:.5in;mso-add-space:auto;text-indent:-.25in"><span style="font-size:11.0pt;font-family:"Calibri",sans-serif">10.</span><span style="font-size:7.0pt;font-family:"Times New Roman",serif">
</span><span style="font-size:11.0pt;font-family:"Calibri",sans-serif">Complete initialization state machine<o:p></o:p></span></p>
<p> <span style="font-size:11.0pt;font-family:"Calibri",sans-serif"><o:p></o:p></span></p>
<p><span style="font-size:11.0pt;font-family:"Calibri",sans-serif">As can be observed, during disk initialization, around 224 commands are processed for setting up the IO queues and associate the MSI-x number to each queues. If we use StorPortInitializePerfOpts(),
we required only 64 commands instead of 224 commands. On a server which as 120 logical processors, 840 commands are required for setting up the IO queues and associate the MSI-x number to each queues. If learning cores is avoided, only 240 commands are required
instead of 840 commands.<o:p></o:p></span></p>
<p> <span style="font-size:11.0pt;font-family:"Calibri",sans-serif"><o:p></o:p></span></p>
<p><span style="font-size:11.0pt;font-family:"Calibri",sans-serif">Also we can fall back to learning cores if the API StorPortInitiailzePerfOpts() fails.<o:p></o:p></span></p>
<p> <span style="font-size:11.0pt;font-family:"Calibri",sans-serif"><o:p></o:p></span></p>
<p><span style="font-size:11.0pt;font-family:"Calibri",sans-serif">We see improved device up time after this change. Also, if the number of queues supported by the device is less than the number of logical processors, the driver does not execute the learning
cores, hence there won’t be any improvement even if we use StorPortInitializePerfOpts().<o:p></o:p></span></p>
<p style="margin-bottom:7.5pt"> <span style="font-size:11.0pt;font-family:"Calibri",sans-serif"><o:p></o:p></span></p>
<table class="MsoNormalTable" border="0" cellspacing="0" cellpadding="0" width="495" style="width:371.0pt;border-collapse:collapse">
<tbody>
<tr style="height:16.6pt">
<td width="494" colspan="4" valign="bottom" style="width:370.5pt;border:solid black 1.0pt;border-right:solid windowtext 1.0pt;background:#FFCC99;padding:0in 0in 0in 0in;height:16.6pt;border-image: none">
<p align="center" style="text-align:center"><span style="font-size:11.0pt;font-family:"Calibri",sans-serif;color:#3F3F76">Server with 32 logical processors:</span><span style="font-size:11.0pt;font-family:"Calibri",sans-serif"><o:p></o:p></span></p>
</td>
</tr>
<tr style="height:15.0pt">
<td width="83" rowspan="3" style="width:62.25pt;border-top:none;border-left:solid black 1.0pt;border-bottom:solid black 1.0pt;border-right:solid windowtext 1.0pt;background:#A5A5A5;padding:0in 0in 0in 0in;height:15.0pt">
<p align="center" style="text-align:center"><b><span style="font-size:11.0pt;font-family:"Calibri",sans-serif;color:white">OFA version</span></b><span style="font-size:11.0pt;font-family:"Calibri",sans-serif"><o:p></o:p></span></p>
</td>
<td width="409" colspan="3" style="width:306.75pt;border-top:none;border-left:none;border-bottom:solid black 1.0pt;border-right:solid windowtext 1.0pt;background:#A5A5A5;padding:0in 0in 0in 0in;height:15.0pt">
<p align="center" style="text-align:center"><b><span style="font-size:11.0pt;font-family:"Calibri",sans-serif;color:white">disk up time in seconds</span></b><span style="font-size:11.0pt;font-family:"Calibri",sans-serif"><o:p></o:p></span></p>
</td>
</tr>
<tr style="height:15.0pt">
<td width="274" colspan="2" style="width:205.5pt;border-top:none;border-left:none;border-bottom:solid black 1.0pt;border-right:solid windowtext 1.0pt;background:#A5A5A5;padding:0in 0in 0in 0in;height:15.0pt">
<p align="center" style="text-align:center"><b><span style="font-size:11.0pt;font-family:"Calibri",sans-serif;color:white">Disk capacity = 400 GB</span></b><span style="font-size:11.0pt;font-family:"Calibri",sans-serif"><o:p></o:p></span></p>
</td>
<td width="134" rowspan="2" style="width:100.5pt;border-top:none;border-left:none;border-bottom:solid black 1.0pt;border-right:solid windowtext 1.0pt;background:#A5A5A5;padding:0in 0in 0in 0in;height:15.0pt">
<p align="center" style="text-align:center"><b><span style="font-size:11.0pt;font-family:"Calibri",sans-serif;color:white">Disk capacity = 1.6 TB</span></b><span style="font-size:11.0pt;font-family:"Calibri",sans-serif"><o:p></o:p></span></p>
</td>
</tr>
<tr style="height:15.0pt">
<td width="149" style="width:111.75pt;border-top:none;border-left:none;border-bottom:solid black 1.0pt;border-right:solid windowtext 1.0pt;background:#A5A5A5;padding:0in 0in 0in 0in;height:15.0pt">
<p><b><span style="font-size:11.0pt;font-family:"Calibri",sans-serif;color:white">Disk from vendor 1</span></b><span style="font-size:11.0pt;font-family:"Calibri",sans-serif"><o:p></o:p></span></p>
</td>
<td width="124" style="width:93.0pt;border-top:none;border-left:none;border-bottom:solid black 1.0pt;border-right:solid windowtext 1.0pt;background:#A5A5A5;padding:0in 0in 0in 0in;height:15.0pt">
<p><b><span style="font-size:11.0pt;font-family:"Calibri",sans-serif;color:white">Disk from vendor 2</span></b><span style="font-size:11.0pt;font-family:"Calibri",sans-serif"><o:p></o:p></span></p>
</td>
</tr>
<tr style="height:15.0pt">
<td width="83" valign="bottom" style="width:62.25pt;border-top:none;border-left:solid black 1.0pt;border-bottom:solid black 1.0pt;border-right:solid windowtext 1.0pt;padding:0in 0in 0in 0in;height:15.0pt">
<p><span style="font-size:11.0pt;font-family:"Calibri",sans-serif;color:black">Rev 133</span><span style="font-size:11.0pt;font-family:"Calibri",sans-serif"><o:p></o:p></span></p>
</td>
<td width="149" valign="bottom" style="width:111.75pt;border-top:none;border-left:none;border-bottom:solid black 1.0pt;border-right:solid windowtext 1.0pt;padding:0in 0in 0in 0in;height:15.0pt">
<p align="right" style="text-align:right"><span style="font-size:11.0pt;font-family:"Calibri",sans-serif;color:black">14</span><span style="font-size:11.0pt;font-family:"Calibri",sans-serif"><o:p></o:p></span></p>
</td>
<td width="124" valign="bottom" style="width:93.0pt;border-top:none;border-left:none;border-bottom:solid black 1.0pt;border-right:solid windowtext 1.0pt;padding:0in 0in 0in 0in;height:15.0pt">
<p align="right" style="text-align:right"><span style="font-size:11.0pt;font-family:"Calibri",sans-serif;color:black">6.5</span><span style="font-size:11.0pt;font-family:"Calibri",sans-serif"><o:p></o:p></span></p>
</td>
<td width="134" valign="bottom" style="width:100.5pt;border-top:none;border-left:none;border-bottom:solid black 1.0pt;border-right:solid windowtext 1.0pt;padding:0in 0in 0in 0in;height:15.0pt">
<p align="right" style="text-align:right"><span style="font-size:11.0pt;font-family:"Calibri",sans-serif;color:black">14.5</span><span style="font-size:11.0pt;font-family:"Calibri",sans-serif"><o:p></o:p></span></p>
</td>
</tr>
<tr style="height:8.05pt">
<td width="83" valign="bottom" style="width:62.25pt;border-top:none;border-left:solid black 1.0pt;border-bottom:solid black 1.0pt;border-right:solid windowtext 1.0pt;padding:0in 0in 0in 0in;height:8.05pt">
<p><span style="font-size:11.0pt;font-family:"Calibri",sans-serif;color:black">Latest</span><span style="font-size:11.0pt;font-family:"Calibri",sans-serif"><o:p></o:p></span></p>
</td>
<td width="149" valign="bottom" style="width:111.75pt;border-top:none;border-left:none;border-bottom:solid black 1.0pt;border-right:solid windowtext 1.0pt;padding:0in 0in 0in 0in;height:8.05pt">
<p align="right" style="text-align:right"><span style="font-size:11.0pt;font-family:"Calibri",sans-serif;color:black">5</span><span style="font-size:11.0pt;font-family:"Calibri",sans-serif"><o:p></o:p></span></p>
</td>
<td width="124" valign="bottom" style="width:93.0pt;border-top:none;border-left:none;border-bottom:solid black 1.0pt;border-right:solid windowtext 1.0pt;padding:0in 0in 0in 0in;height:8.05pt">
<p align="right" style="text-align:right"><span style="font-size:11.0pt;font-family:"Calibri",sans-serif;color:black">5</span><span style="font-size:11.0pt;font-family:"Calibri",sans-serif"><o:p></o:p></span></p>
</td>
<td width="134" valign="bottom" style="width:100.5pt;border-top:none;border-left:none;border-bottom:solid black 1.0pt;border-right:solid windowtext 1.0pt;padding:0in 0in 0in 0in;height:8.05pt">
<p align="right" style="text-align:right"><span style="font-size:11.0pt;font-family:"Calibri",sans-serif;color:black">13.5</span><span style="font-size:11.0pt;font-family:"Calibri",sans-serif"><o:p></o:p></span></p>
</td>
</tr>
</tbody>
</table>
<p><b><span style="font-size:11.0pt;font-family:"Calibri",sans-serif">PS</span></b><span style="font-size:11.0pt;font-family:"Calibri",sans-serif">: data may change for different vendor SSDs<o:p></o:p></span></p>
<p> <span style="font-size:11.0pt;font-family:"Calibri",sans-serif"><o:p></o:p></span></p>
<p><b><u><span style="font-size:11.0pt;font-family:"Calibri",sans-serif">Code changes:</span></u></b><span style="font-size:11.0pt;font-family:"Calibri",sans-serif"><o:p></o:p></span></p>
<p style="margin-left:.5in;mso-add-space:auto;text-indent:-.25in">1.<span style="font-size:7.0pt;font-family:"Times New Roman",serif">
</span><span style="font-size:11.0pt;font-family:"Calibri",sans-serif">Changes w.r.t StorPortInitializePerfOpts().<o:p></o:p></span></p>
<p style="margin-left:1.0in;mso-add-space:auto;text-indent:-.25in">a.<span style="font-size:7.0pt;font-family:"Times New Roman",serif">
</span><span style="font-size:11.0pt;font-family:"Calibri",sans-serif">In NVMeInitialize(), moved the StorPortInitializePerfOpts() after NVMeEnumMsiMessages() to set the LastRedirectionMessageNumber in StorPortInitializePerfOpts().<o:p></o:p></span></p>
<p style="margin-left:1.0in;mso-add-space:auto;text-indent:-.25in">b.<span style="font-size:7.0pt;font-family:"Times New Roman",serif">
</span><span style="font-size:11.0pt;font-family:"Calibri",sans-serif">Set the flags STOR_PERF_INTERRUPT_MESSAGE_RANGES and STOR_PERF_ADV_CONFIG_LOCALITY, and values FirstRedirectionMessageNumber, LastRedirectionMessageNumber and MessageTargets in StorPortInitializePerfOpts()
to get the MSIx-Core mapping in MessageTargets. If this API returns success, the learning cores can be skipped.<o:p></o:p></span></p>
<p style="margin-left:1.0in;mso-add-space:auto;text-indent:-.25in">c.<span style="font-size:7.0pt;font-family:"Times New Roman",serif">
</span><span style="font-size:11.0pt;font-family:"Calibri",sans-serif">If the StorPortInitializePerfOpts() fails, in NVMeMsiMapCores(), the mapping of msix to cores in assigned sequentially, and learning cores is executed. During learning cores, in IoCompletionRoutine(),
the msix to core is re-mapped. If the StorPortInitializePerfOpts() succeeds, in NVMeMsiMapCores(), the mapping of msix to cores is taken from MessageTargets and learnig cores is skipped.<o:p></o:p></span></p>
<p style="margin-left:1.0in;mso-add-space:auto"> <span style="font-size:11.0pt;font-family:"Calibri",sans-serif"><o:p></o:p></span></p>
<p style="margin-left:.5in;mso-add-space:auto;text-indent:-.25in">2.<span style="font-size:7.0pt;font-family:"Times New Roman",serif">
</span><span style="font-size:11.0pt;font-family:"Calibri",sans-serif">When the learning cores is skipped, the controller initialization completes faster. But we have observed that in some devices, the Namespace is not ready to process I/O at this point. And
when kernel send I/O, the device returns SC = NAMESPACE_NOT_READY and miniport returns SCSI_SENSEQ_BECOMING_READY, for which storport retries after some time. If the device takes too long to initialize the namespace, the storport gives up and shows as Uninitialized
in the disk mgmt.<br>
Hence the controller initialization has to be completed after Namespace is ready. For this, a new state is introduce in the NVMeRunning(), which waits till the NS is ready. <o:p></o:p></span></p>
<p style="margin-left:1.0in;mso-add-space:auto;text-indent:-.25in"><span style="font-size:11.0pt;font-family:"Calibri",sans-serif">a.</span><span style="font-size:7.0pt;font-family:"Times New Roman",serif">
</span><span style="font-size:11.0pt;font-family:"Calibri",sans-serif">Introduced a new state NVMeWaitOnNamespaceReady in NVMeRunning().<o:p></o:p></span></p>
<p style="margin-left:1.0in;mso-add-space:auto;text-indent:-.25in">b.<span style="font-size:7.0pt;font-family:"Times New Roman",serif">
</span><span style="font-size:11.0pt;font-family:"Calibri",sans-serif">In IoCompletionRoutine(), determine which CQ to look in based on WaitOnNamespaceReady state.<o:p></o:p></span></p>
<p style="margin-left:1.0in;mso-add-space:auto;text-indent:-.25in">c.<span style="font-size:7.0pt;font-family:"Times New Roman",serif">
</span><span style="font-size:11.0pt;font-family:"Calibri",sans-serif">In NVMeInitCallback(), implemented call back for NVMeWaitOnNamespaceReady.<o:p></o:p></span></p>
<p style="margin-left:1.0in;mso-add-space:auto;text-indent:-.25in">d.<span style="font-size:7.0pt;font-family:"Times New Roman",serif">
</span><span style="font-size:11.0pt;font-family:"Calibri",sans-serif">In IoCompletionRoutine(), free the read buffer for namespaceready.<o:p></o:p></span></p>
<p style="margin-left:.5in"><b><u><span style="font-size:11.0pt;font-family:"Calibri",sans-serif">Note</span></u></b><span style="font-size:11.0pt;font-family:"Calibri",sans-serif">:
<o:p></o:p></span></p>
<p style="margin-left:.75in;mso-add-space:auto;text-indent:-.25in">a.<span style="font-size:7.0pt;font-family:"Times New Roman",serif">
</span><span style="font-size:11.0pt;font-family:"Calibri",sans-serif">We have observed that higher capacity Namespaces take too long to initialize, hence the passiveTimeout value in NVMePassiveInitialize() is not sufficient. We need to increase the timeout
value based on vendor requirements. <o:p></o:p></span></p>
<p style="margin-left:.75in;mso-add-space:auto;text-indent:-.25in">b.<span style="font-size:7.0pt;font-family:"Times New Roman",serif">
</span><span style="font-size:11.0pt;font-family:"Calibri",sans-serif">b. Checking for Namespace ready is skipped during dump/hibernation mode and resume from hibernation.<o:p></o:p></span></p>
<p> <span style="font-size:11.0pt;font-family:"Calibri",sans-serif"><o:p></o:p></span></p>
<p style="margin-left:.5in;mso-add-space:auto;text-indent:-.25in">3.<span style="font-size:7.0pt;font-family:"Times New Roman",serif">
</span><span style="font-size:11.0pt;font-family:"Calibri",sans-serif">Usually, the number of MSIx supported by device and MSIx granted(StorPortGetMSIInfo) will be number of IO Queue + 1 Admin Queue. But, we share the Admin queue and first I/O queue in core
0, and hence MSIx 0 is shared between admin queue and first I/O queue. Incase of active cores more than Queues supported, one of the MSGID should not be considered. Made changes in In NVMeEnumMsiMessages() accordingly.<br>
For example, cores = 32, Admin + IO queues = 1 + 8, then MsgID(in NVMeEnumMsiMessages()) should be 8.<o:p></o:p></span></p>
<p style="margin-left:.5in;mso-add-space:auto"> <span style="font-size:11.0pt;font-family:"Calibri",sans-serif"><o:p></o:p></span></p>
<p style="margin-left:.5in;mso-add-space:auto;text-indent:-.25in">4.<span style="font-size:7.0pt;font-family:"Times New Roman",serif">
</span><span style="font-size:11.0pt;font-family:"Calibri",sans-serif">In IoCompletionRoutine(), for learning cores, only if MSIGranted is less than active cores, the QueueNo will be remapped in sequential manner. Otherwise QueueNo remains same as before.<o:p></o:p></span></p>
<p style="margin-bottom:7.5pt"> <span style="font-size:11.0pt;font-family:"Calibri",sans-serif"><o:p></o:p></span></p>
<p class="MsoNormal" style="margin:0in;margin-bottom:.0001pt"><span style="border:solid windowtext 1.0pt;padding:0in"><img border="0" width="100" height="100" id="Picture_x0020_1" src="cid:image002.jpg@01D190D5.DF18E700" alt="Image removed by sender."></span><o:p></o:p></p>
<p class="p1"><strong><span style="font-family:"Arial",sans-serif">Western Digital Corporation (and its subsidiaries) E-mail Confidentiality Notice & Disclaimer:</span></strong><o:p></o:p></p>
<p class="p1"><span class="s1">This e-mail and any files transmitted with it may contain confidential or legally privileged information of WDC and/or its affiliates, and are intended solely for the use of the individual or entity to which they are addressed.
If you are not the intended recipient, any disclosure, copying, distribution or any action taken or omitted to be taken in reliance on it, is prohibited. If you have received this e-mail in error, please notify the sender immediately and delete the e-mail
in its entirety from your system.</span><o:p></o:p></p>
<p> <o:p></o:p></p>
<p> <o:p></o:p></p>
<p> <o:p></o:p></p>
<p> <o:p></o:p></p>
<p> <o:p></o:p></p>
<p> <o:p></o:p></p>
<table class="MsoNormalTable" border="0" cellpadding="0" id="confidentialsignimg">
<tbody>
<tr>
<td style="padding:.75pt .75pt .75pt .75pt">
<p><img border="0" width="520" height="144" id="_x0000_i1027" src="cid:image003.gif@01D190D5.DF18E700"><o:p></o:p></p>
</td>
</tr>
</tbody>
</table>
<p class="MsoNormal" style="margin:0in;margin-bottom:.0001pt"><img border="0" id="_x0000_i1028" src="http://ext.samsung.net/mailcheck/SeenTimeChecker?do=5ffd028a7cd232fe9761cd2cae8791eea8b881028604b683b8a3fd6b4a260c0ed33a9d35f6e1735f20a30c65ae77ad69c7b41e955949e5c8a728c55b39cc59eacf878f9a26ce15a0"><o:p></o:p></p>
</div>
</body>
</html>