<html xmlns:v="urn:schemas-microsoft-com:vml" xmlns:o="urn:schemas-microsoft-com:office:office" xmlns:w="urn:schemas-microsoft-com:office:word" xmlns:m="http://schemas.microsoft.com/office/2004/12/omml" xmlns="http://www.w3.org/TR/REC-html40">
<head>
<meta http-equiv="Content-Type" content="text/html; charset=utf-8">
<meta name="Generator" content="Microsoft Word 15 (filtered medium)">
<!--[if !mso]><style>v\:* {behavior:url(#default#VML);}
o\:* {behavior:url(#default#VML);}
w\:* {behavior:url(#default#VML);}
.shape {behavior:url(#default#VML);}
</style><![endif]--><style><!--
/* Font Definitions */
@font-face
{font-family:Wingdings;
panose-1:5 0 0 0 0 0 0 0 0 0;}
@font-face
{font-family:SimSun;
panose-1:2 1 6 0 3 1 1 1 1 1;}
@font-face
{font-family:"Cambria Math";
panose-1:2 4 5 3 5 4 6 3 2 4;}
@font-face
{font-family:Calibri;
panose-1:2 15 5 2 2 2 4 3 2 4;}
@font-face
{font-family:Tahoma;
panose-1:2 11 6 4 3 5 4 4 2 4;}
@font-face
{font-family:"\@SimSun";
panose-1:2 1 6 0 3 1 1 1 1 1;}
@font-face
{font-family:Consolas;
panose-1:2 11 6 9 2 2 4 3 2 4;}
/* Style Definitions */
p.MsoNormal, li.MsoNormal, div.MsoNormal
{margin:0in;
margin-bottom:.0001pt;
font-size:12.0pt;
font-family:"Calibri",sans-serif;}
a:link, span.MsoHyperlink
{mso-style-priority:99;
color:#0563C1;
text-decoration:underline;}
a:visited, span.MsoHyperlinkFollowed
{mso-style-priority:99;
color:#954F72;
text-decoration:underline;}
p
{mso-style-priority:99;
margin-top:3.75pt;
margin-right:0in;
margin-bottom:3.75pt;
margin-left:0in;
font-size:10.0pt;
font-family:"Arial",sans-serif;}
pre
{mso-style-priority:99;
mso-style-link:"HTML Preformatted Char";
margin:0in;
margin-bottom:.0001pt;
font-size:10.0pt;
font-family:"Courier New";}
p.MsoAcetate, li.MsoAcetate, div.MsoAcetate
{mso-style-priority:99;
mso-style-link:"Balloon Text Char";
margin:0in;
margin-bottom:.0001pt;
font-size:8.0pt;
font-family:"Tahoma",sans-serif;}
p.MsoListParagraph, li.MsoListParagraph, div.MsoListParagraph
{mso-style-priority:34;
margin-top:0in;
margin-right:0in;
margin-bottom:0in;
margin-left:24.0pt;
margin-bottom:.0001pt;
mso-para-margin-top:0in;
mso-para-margin-right:0in;
mso-para-margin-bottom:0in;
mso-para-margin-left:2.0gd;
mso-para-margin-bottom:.0001pt;
font-size:12.0pt;
font-family:"Calibri",sans-serif;}
span.HTMLPreformattedChar
{mso-style-name:"HTML Preformatted Char";
mso-style-priority:99;
mso-style-link:"HTML Preformatted";
font-family:Consolas;}
span.BalloonTextChar
{mso-style-name:"Balloon Text Char";
mso-style-priority:99;
mso-style-link:"Balloon Text";
font-family:"Tahoma",sans-serif;}
p.msonormal0, li.msonormal0, div.msonormal0
{mso-style-name:msonormal;
mso-style-priority:99;
mso-margin-top-alt:auto;
margin-right:0in;
mso-margin-bottom-alt:auto;
margin-left:0in;
font-size:11.0pt;
font-family:"Calibri",sans-serif;}
p.search-word, li.search-word, div.search-word
{mso-style-name:search-word;
mso-style-priority:99;
margin-top:3.75pt;
margin-right:0in;
margin-bottom:3.75pt;
margin-left:0in;
background:#FFEE94;
font-size:10.0pt;
font-family:"Arial",sans-serif;}
span.EmailStyle25
{mso-style-type:personal;
font-family:"Calibri",sans-serif;
color:windowtext;}
span.EmailStyle26
{mso-style-type:personal;
font-family:"Calibri",sans-serif;
color:windowtext;}
span.EmailStyle27
{mso-style-type:personal;
font-family:"Calibri",sans-serif;
color:#002060;
font-weight:normal;
font-style:normal;}
span.EmailStyle28
{mso-style-type:personal;
font-family:"Calibri",sans-serif;
color:windowtext;}
span.EmailStyle29
{mso-style-type:personal;
font-family:"Calibri",sans-serif;
color:#1F497D;}
span.EmailStyle30
{mso-style-type:personal;
font-family:"Calibri",sans-serif;
color:#1F497D;}
span.EmailStyle31
{mso-style-type:personal;
font-family:"Calibri",sans-serif;
color:#1F497D;}
span.EmailStyle32
{mso-style-type:personal;
font-family:"Calibri",sans-serif;
color:#1F497D;
font-weight:normal;
font-style:normal;}
span.EmailStyle33
{mso-style-type:personal;
font-family:"Calibri",sans-serif;
color:#002060;
font-weight:normal;
font-style:normal;}
span.EmailStyle34
{mso-style-type:personal;
font-family:"Calibri",sans-serif;
color:#002060;
font-weight:normal;
font-style:normal;}
span.EmailStyle35
{mso-style-type:personal;
font-family:"Calibri",sans-serif;
color:#1F497D;
font-weight:normal;
font-style:normal;}
span.EmailStyle36
{mso-style-type:personal;
font-family:"Calibri",sans-serif;
color:#002060;
font-weight:normal;
font-style:normal;}
span.EmailStyle37
{mso-style-type:personal;
font-family:"Calibri",sans-serif;
color:#44546A;}
span.EmailStyle38
{mso-style-type:personal;
font-family:"Calibri",sans-serif;
color:#1F497D;}
span.EmailStyle39
{mso-style-type:personal;
font-family:"Calibri",sans-serif;
color:#44546A;}
span.EmailStyle41
{mso-style-type:personal-reply;
font-family:"Calibri",sans-serif;
color:#1F497D;
font-weight:normal;
font-style:normal;}
.MsoChpDefault
{mso-style-type:export-only;
font-size:10.0pt;}
@page WordSection1
{size:8.5in 11.0in;
margin:1.0in 1.25in 1.0in 1.25in;}
div.WordSection1
{page:WordSection1;}
--></style><!--[if gte mso 9]><xml>
<o:shapedefaults v:ext="edit" spidmax="1026" />
</xml><![endif]--><!--[if gte mso 9]><xml>
<o:shapelayout v:ext="edit">
<o:idmap v:ext="edit" data="1" />
</o:shapelayout></xml><![endif]-->
</head>
<body lang="EN-US" link="#0563C1" vlink="#954F72" style="text-justify-trim:punctuation">
<div class="WordSection1">
<p class="MsoNormal"><span style="font-size:11.0pt;color:#1F497D">Hi Judy and others,<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt;color:#1F497D">In Ray's absense, I'm trying to pick up this thread again. I'd like to verify we agree on what is required to complete this patch.<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt;color:#1F497D"><o:p> </o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt;color:#1F497D">1. As Judy stated before, the following is missing from Justina's original patch to nvmeIo.c on 5/25 - (cores == queues), all command completion handlers issuing a new command should require
ProcessIO to get the StartIO lock. <o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt;color:#1F497D"><o:p> </o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt;color:#1F497D">This affects calls to ProcessIo from these methods - SntiTranslatePersReserveOutResponse, SntiTranslateTemperatureResponse, SntiTranslateModeSenseResponse, SntiTranslateStartStopUnitResponse,
SntiTranslateWriteBufferResponse, NVMeHandleSmartThresholds, NVMeCompletionNsAttachment, NVMeCompletionNsAttachment, FormatNVMGetIdentify.<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt;color:#1F497D"><o:p> </o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt;color:#1F497D">2. Based on Ray's response on 6/27 "I’m ok with acquiring the StartIo lock as long as it’s in a new DPC spawned by the IOCompletionRoutine", instead of calling ProcessIo directly, IoCompletionRoutine
should issue a dpc that will take the startIo flag and invoke processIo.<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt;color:#1F497D"><o:p> </o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt;color:#1F497D">Do you agree? Is there anything else that should be included?<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt;color:#1F497D"><o:p> </o:p></span></p>
<div>
<p class="MsoNormal"><b><span style="color:#1F497D">Tom Freeman</span></b><b><span style="color:#1F497D"><o:p></o:p></span></b></p>
<p class="MsoNormal"><span style="font-size:10.0pt;color:#1F497D">Software Engineer, Device Manager and Driver Development<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:9.0pt;color:#1F497D"><br>
</span><b><span style="color:#1F497D">Western Digital</span></b><span style="color:#1F497D">®
</span><span style="font-size:10.0pt;color:#1F497D"><o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:10.0pt;color:#1F497D">Email: Thomas.Freeman@wdc.com<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:10.0pt;color:#1F497D">Office: +1-507-322-2311<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt;color:#1F497D"><o:p> </o:p></span></p>
</div>
<p class="MsoNormal"><span style="font-size:11.0pt;color:#1F497D"><o:p> </o:p></span></p>
<div>
<div style="border:none;border-top:solid #E1E1E1 1.0pt;padding:3.0pt 0in 0in 0in">
<p class="MsoNormal"><b><span style="font-size:11.0pt">From:</span></b><span style="font-size:11.0pt"> Judy Brock [mailto:judy.brock@samsung.com]
<br>
<b>Sent:</b> Friday, June 30, 2017 12:37 PM<br>
<b>To:</b> Robles, Raymond C <raymond.c.robles@intel.com>; Tom Freeman <thomas.freeman@wdc.com>; sm.kumar@samsung.com; Justina Lai <justina_lai@phison.com><br>
<b>Cc:</b> Larry Li <larry_li@phison.com>; nvmewin@lists.openfabrics.org<br>
<b>Subject:</b> RE: Re: [nvmewin] Compatibility issue with 1.5 version nvme driver<o:p></o:p></span></p>
</div>
</div>
<p class="MsoNormal"><o:p> </o:p></p>
<p class="MsoNormal"><span style="font-size:11.0pt;color:#44546A">Yes, we agree. That is why we included the following statement below:<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt;color:#44546A"><o:p> </o:p></span></p>
<p class="MsoNormal" style="text-indent:.5in"><span style="font-size:11.0pt">“The various translation completion command handlers should all be consistent and SHOULD take the StartIO lock in ProcessIO if trying to issue a new command”<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt;color:#44546A"><o:p> </o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt;color:#44546A">Right now, they are not all consistent.<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt;color:#44546A"><o:p> </o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt;color:#44546A">Thanks,<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt;color:#44546A">Judy<o:p></o:p></span></p>
<p class="MsoNormal"><a name="_MailEndCompose"></a><span style="font-size:11.0pt;color:#44546A"><o:p> </o:p></span></p>
<div>
<div style="border:none;border-top:solid #B5C4DF 1.0pt;padding:3.0pt 0in 0in 0in">
<p class="MsoNormal"><b><span style="font-size:10.0pt;font-family:"Tahoma",sans-serif">From:</span></b><span style="font-size:10.0pt;font-family:"Tahoma",sans-serif"> nvmewin [<a href="mailto:nvmewin-bounces@lists.openfabrics.org">mailto:nvmewin-bounces@lists.openfabrics.org</a>]
<b>On Behalf Of </b>Robles, Raymond C<br>
<b>Sent:</b> Friday, June 30, 2017 10:31 AM<br>
<b>To:</b> Judy Brock; Tom Freeman; <a href="mailto:sm.kumar@samsung.com">sm.kumar@samsung.com</a>; Justina Lai<br>
<b>Cc:</b> Larry Li; <a href="mailto:nvmewin@lists.openfabrics.org">nvmewin@lists.openfabrics.org</a><br>
<b>Subject:</b> Re: [nvmewin] Compatibility issue with 1.5 version nvme driver<o:p></o:p></span></p>
</div>
</div>
<p class="MsoNormal"><o:p> </o:p></p>
<p class="MsoNormal"><span style="font-size:11.0pt;color:#1F497D">Hi Judy, <o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt;color:#1F497D"><o:p> </o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt;color:#1F497D">Thank you for your feedback. I think we are all in agreement on Phison’s patch.
<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt;color:#1F497D"><o:p> </o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt;color:#1F497D">My only concern is the original patch did not address any changes in nvmeSnti.c where the completion path needed to invoke the StartIo lock appropriately. Did you observe the same?<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt;color:#1F497D"><o:p> </o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt;color:#1F497D">Thanks…<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt;color:#1F497D">Ray<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt;color:#1F497D"><o:p> </o:p></span></p>
<div>
<div style="border:none;border-top:solid #E1E1E1 1.0pt;padding:3.0pt 0in 0in 0in">
<p class="MsoNormal"><b><span style="font-size:11.0pt">From:</span></b><span style="font-size:11.0pt"> Judy Brock [<a href="mailto:judy.brock@samsung.com">mailto:judy.brock@samsung.com</a>]
<br>
<b>Sent:</b> Friday, June 30, 2017 6:16 AM<br>
<b>To:</b> Robles, Raymond C <<a href="mailto:raymond.c.robles@intel.com">raymond.c.robles@intel.com</a>>; Tom Freeman <<a href="mailto:thomas.freeman@wdc.com">thomas.freeman@wdc.com</a>>;
<a href="mailto:sm.kumar@samsung.com">sm.kumar@samsung.com</a>; Justina Lai <<a href="mailto:justina_lai@phison.com">justina_lai@phison.com</a>><br>
<b>Cc:</b> Larry Li <<a href="mailto:larry_li@phison.com">larry_li@phison.com</a>>;
<a href="mailto:nvmewin@lists.openfabrics.org">nvmewin@lists.openfabrics.org</a><br>
<b>Subject:</b> RE: Re: [nvmewin] Compatibility issue with 1.5 version nvme driver<o:p></o:p></span></p>
</div>
</div>
<p class="MsoNormal"><o:p> </o:p></p>
<p class="MsoNormal"><span style="font-size:11.0pt;color:#1F497D">Hi Ray et al,<o:p></o:p></span></p>
<p class="MsoNormal" style="margin-left:31.5pt"><span style="font-size:11.0pt;color:#1F497D">
<o:p></o:p></span></p>
<p class="MsoNormal" style="margin-left:.5in;text-indent:-.5in"><span style="font-size:11.0pt;color:#1F497D">We just want to clarify our position/feedback after all the recent back and forth.</span><o:p></o:p></p>
<p class="MsoNormal" style="margin-left:.5in;text-indent:-.5in"><span style="font-size:11.0pt;color:#1F497D"> </span><o:p></o:p></p>
<p class="MsoNormal" style="margin-left:.5in;text-indent:-.5in"><span style="font-size:11.0pt;color:#1F497D">In the
</span><span style="font-size:11.0pt">MultipleCoresToSingleQueueFlag ==<span style="color:#1F497D">
</span><span style="color:red">TRUE</span><span style="color:#1F497D"> completion path case:</span></span><o:p></o:p></p>
<p class="MsoListParagraph" style="margin-left:1.0in;mso-para-margin-left:0gd;text-indent:-.5in">
<span style="font-size:11.0pt;font-family:Symbol;color:#1F497D">·</span><span style="font-size:7.0pt;font-family:"Times New Roman",serif;color:#1F497D">
</span><span style="font-size:11.0pt;color:#1F497D">we understand and agree with acquiring the StartIO lock in the IOCompletion Routine. We do not support acquiring a DPC lock instead.</span><o:p></o:p></p>
<p class="MsoListParagraph" style="margin-left:1.0in;mso-para-margin-left:0gd;text-indent:-.5in">
<span style="font-size:11.0pt;font-family:Symbol;color:#1F497D">·</span><span style="font-size:7.0pt;font-family:"Times New Roman",serif;color:#1F497D">
</span><span style="font-size:11.0pt;color:#1F497D">we agree the code SHOULD NOT try to take the StartIO lock in ProcessIO if trying to issue a new secondary command (this caused the deadlock Justina reported)</span><o:p></o:p></p>
<p class="MsoListParagraph" style="margin-left:1.0in;mso-para-margin-left:0gd;text-indent:-.5in">
<span style="font-size:11.0pt;font-family:Symbol;color:#1F497D">·</span><span style="font-size:7.0pt;font-family:"Times New Roman",serif;color:#1F497D">
</span><span style="font-size:11.0pt;color:#1F497D">we are ok with Justina’s original fix with respect to this issue</span><o:p></o:p></p>
<p class="MsoNormal" style="margin-left:.5in;text-indent:-.5in"> <o:p></o:p></p>
<p class="MsoNormal" style="margin-left:.5in;text-indent:-.5in"><span style="font-size:11.0pt;color:#1F497D">In the
</span><span style="font-size:11.0pt">MultipleCoresToSingleQueueFlag ==<span style="color:#1F497D">
</span><span style="color:red">FALSE</span><span style="color:#1F497D"> completion path case (i.e. cores == queues):</span></span><o:p></o:p></p>
<p class="MsoListParagraph" style="margin-left:1.0in;mso-para-margin-left:0gd;text-indent:-.5in">
<span style="font-size:11.0pt;font-family:Symbol;color:#1F497D">·</span><span style="font-size:7.0pt;font-family:"Times New Roman",serif;color:#1F497D">
</span><span style="font-size:11.0pt;color:#1F497D">The IOCompletion routine should take a DPC lock (as it does today)</span><o:p></o:p></p>
<p class="MsoListParagraph" style="margin-left:1.0in;mso-para-margin-left:0gd;text-indent:-.5in">
<span style="font-family:Symbol;color:#215968">·</span><span style="font-size:7.0pt;font-family:"Times New Roman",serif;color:#215968">
</span><span style="color:#215968">The various translation completion command handlers should all be consistent and SHOULD take the StartIO lock in ProcessIO if trying to issue a new command</span><o:p></o:p></p>
<p class="MsoNormal" style="text-indent:-.5in"><span style="font-size:11.0pt;color:#44546A"><o:p> </o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt;color:#44546A">Thanks,<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt;color:#44546A">Judy<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt;color:#44546A"><o:p> </o:p></span></p>
<div>
<div style="border:none;border-top:solid #B5C4DF 1.0pt;padding:3.0pt 0in 0in 0in">
<p class="MsoNormal"><b><span style="font-size:10.0pt;font-family:"Tahoma",sans-serif">From:</span></b><span style="font-size:10.0pt;font-family:"Tahoma",sans-serif"> nvmewin [<a href="mailto:nvmewin-bounces@lists.openfabrics.org">mailto:nvmewin-bounces@lists.openfabrics.org</a>]
<b>On Behalf Of </b>Robles, Raymond C<br>
<b>Sent:</b> Tuesday, June 27, 2017 4:29 PM<br>
<b>To:</b> Tom Freeman; <a href="mailto:sm.kumar@samsung.com">sm.kumar@samsung.com</a>; Justina Lai<br>
<b>Cc:</b> Larry Li; <a href="mailto:nvmewin@lists.openfabrics.org">nvmewin@lists.openfabrics.org</a><br>
<b>Subject:</b> Re: [nvmewin] Compatibility issue with 1.5 version nvme driver<o:p></o:p></span></p>
</div>
</div>
<p class="MsoNormal"><o:p> </o:p></p>
<p class="MsoNormal"><span style="font-size:11.0pt;color:#002060">Hi Tom,<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt;color:#002060"><o:p> </o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt;color:#002060">Thanks for bearing with me on the details of this. After looking more closely at the code, I agree with your assessment in the case where a DPC is issued for a secondary, internal command (we
do need to acquire the StartIo lock). <o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt;color:#002060"><o:p> </o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt;color:#002060">I’ll wait for Phison to catch up on this thread and provide the update on their unit testing.
<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt;color:#002060"><o:p> </o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt;color:#002060">Thanks…<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt;color:#002060">Ray<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt;color:#002060"><o:p> </o:p></span></p>
<div>
<div style="border:none;border-top:solid #E1E1E1 1.0pt;padding:3.0pt 0in 0in 0in">
<p class="MsoNormal"><b><span style="font-size:11.0pt">From:</span></b><span style="font-size:11.0pt"> Tom Freeman [<a href="mailto:thomas.freeman@wdc.com">mailto:thomas.freeman@wdc.com</a>]
<br>
<b>Sent:</b> Tuesday, June 27, 2017 11:33 AM<br>
<b>To:</b> Robles, Raymond C <<a href="mailto:raymond.c.robles@intel.com">raymond.c.robles@intel.com</a>>;
<a href="mailto:sm.kumar@samsung.com">sm.kumar@samsung.com</a>; Justina Lai <<a href="mailto:justina_lai@phison.com">justina_lai@phison.com</a>><br>
<b>Cc:</b> Larry Li <<a href="mailto:larry_li@phison.com">larry_li@phison.com</a>>;
<a href="mailto:nvmewin@lists.openfabrics.org">nvmewin@lists.openfabrics.org</a><br>
<b>Subject:</b> RE: Re: [nvmewin] Compatibility issue with 1.5 version nvme driver<o:p></o:p></span></p>
</div>
</div>
<p class="MsoNormal"><o:p> </o:p></p>
<p class="MsoNormal"><span style="font-size:11.0pt;color:#1F497D">Thanks Ray,<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt;color:#1F497D">I'm still confused - and I think it relates to a basic question I have about dpc routing. If I understand it correctly, IoCompletionRoutine is queued to run on the core matching its submission/completion
queue. But, that instance of IoCompletionRoutine could simultaneously run on other cores. That's why we get the DpcLock, to synchronize multiple threads running the same instance of the dpc, IoCompletionRoutine.<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt;color:#1F497D"><o:p> </o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt;color:#1F497D">If that's correct, I think the following example shows why we need to get the StartIo lock in the completion path when it is issuing another NVMe command.<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt;color:#1F497D"><o:p> </o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt;color:#1F497D">The code path StartIo->...->ProcessIo issues i/o to submission queue 3 (on core 2). StartIo lock is automatically taken.<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt;color:#1F497D">Simultaneously, IoCompletionRoutine->....->ProcessIo() for comp queue 3 could be executing on a core other than core 2. If so, unless IoCompletionRoutine->....->ProcessIo() gets the StartIo lock,
it will not be synchronized with StartIo->...->Process() on core 2.<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt;color:#1F497D"><o:p> </o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt;color:#1F497D">Let me know your thoughts,<o:p></o:p></span></p>
<div>
<p class="MsoNormal"><b><span style="color:#1F497D">Tom Freeman<o:p></o:p></span></b></p>
<p class="MsoNormal"><span style="font-size:10.0pt;color:#1F497D">Software Engineer, Device Manager and Driver Development<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:9.0pt;color:#1F497D"><br>
</span><b><span style="color:#1F497D">Western Digital</span></b><span style="color:#1F497D">®
</span><span style="font-size:10.0pt;color:#1F497D"><o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:10.0pt;color:#1F497D">Email: <a href="mailto:Thomas.Freeman@wdc.com">
Thomas.Freeman@wdc.com</a><o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:10.0pt;color:#1F497D">Office: +1-507-322-2311<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt;color:#1F497D"><o:p> </o:p></span></p>
</div>
<p class="MsoNormal"><span style="font-size:11.0pt;color:#1F497D"><o:p> </o:p></span></p>
<div>
<div style="border:none;border-top:solid #E1E1E1 1.0pt;padding:3.0pt 0in 0in 0in">
<p class="MsoNormal"><b><span style="font-size:11.0pt">From:</span></b><span style="font-size:11.0pt"> Robles, Raymond C [<a href="mailto:raymond.c.robles@intel.com">mailto:raymond.c.robles@intel.com</a>]
<br>
<b>Sent:</b> Monday, June 26, 2017 7:46 PM<br>
<b>To:</b> Tom Freeman <<a href="mailto:thomas.freeman@wdc.com">thomas.freeman@wdc.com</a>>;
<a href="mailto:sm.kumar@samsung.com">sm.kumar@samsung.com</a>; Justina Lai <<a href="mailto:justina_lai@phison.com">justina_lai@phison.com</a>><br>
<b>Cc:</b> Larry Li <<a href="mailto:larry_li@phison.com">larry_li@phison.com</a>>;
<a href="mailto:nvmewin@lists.openfabrics.org">nvmewin@lists.openfabrics.org</a><br>
<b>Subject:</b> RE: Re: [nvmewin] Compatibility issue with 1.5 version nvme driver<o:p></o:p></span></p>
</div>
</div>
<p class="MsoNormal"><o:p> </o:p></p>
<p class="MsoNormal"><span style="font-size:11.0pt;color:#002060">Hi Tom,<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt;color:#002060"><o:p> </o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt;color:#002060">Your statement is correct. I’m probably not being clear… sorry for the confusion.
<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt;color:#002060"><o:p> </o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt;color:#002060">My assertion is that when cpu == cores, the driver should not be grabbing the StartIo lock as a practice. The StartIo lock is meant for Storport to aquire for call to the HwStartIo callback.
If we are in the completion DPC, then that means we have already acquired the DPC lock, and we should not be acquiring any other lock as part of that DPC calling back into ProcessIo for a second command. That is the deadlock to which I’m referring (and the
issue that Meenakshikumar from Samsung references).<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt;color:#002060"><o:p> </o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt;color:#002060">In my opinion, this issue should be resolved as part of this patch.<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt;color:#002060"><o:p> </o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt;color:#002060">Thoughts?<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt;color:#002060"><o:p> </o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt;color:#002060">Thanks…<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt;color:#002060">Ray<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt;color:#002060"><o:p> </o:p></span></p>
<div>
<div style="border:none;border-top:solid #E1E1E1 1.0pt;padding:3.0pt 0in 0in 0in">
<p class="MsoNormal"><b><span style="font-size:11.0pt">From:</span></b><span style="font-size:11.0pt"> Tom Freeman [<a href="mailto:thomas.freeman@wdc.com">mailto:thomas.freeman@wdc.com</a>]
<br>
<b>Sent:</b> Monday, June 26, 2017 1:31 PM<br>
<b>To:</b> Robles, Raymond C <<a href="mailto:raymond.c.robles@intel.com">raymond.c.robles@intel.com</a>>;
<a href="mailto:sm.kumar@samsung.com">sm.kumar@samsung.com</a>; Justina Lai <<a href="mailto:justina_lai@phison.com">justina_lai@phison.com</a>><br>
<b>Cc:</b> Larry Li <<a href="mailto:larry_li@phison.com">larry_li@phison.com</a>>;
<a href="mailto:nvmewin@lists.openfabrics.org">nvmewin@lists.openfabrics.org</a><br>
<b>Subject:</b> RE: Re: [nvmewin] Compatibility issue with 1.5 version nvme driver<o:p></o:p></span></p>
</div>
</div>
<p class="MsoNormal"><o:p> </o:p></p>
<p class="MsoNormal"><span style="font-size:11.0pt;color:#1F497D">Hi Ray,<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt;color:#1F497D">I missing something in the deadlock discussion. Since IoCompletionRoutine and StartIo run at the same IRQL, wouldn’t one run to completion before the other is scheduled thus preventing the deadlock?
<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt;color:#1F497D"><o:p> </o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt;color:#1F497D">Thanks,<o:p></o:p></span></p>
<div>
<p class="MsoNormal"><b><span style="color:#1F497D">Tom Freeman<o:p></o:p></span></b></p>
<p class="MsoNormal"><span style="font-size:10.0pt;color:#1F497D">Software Engineer, Device Manager and Driver Development<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:9.0pt;color:#1F497D"><br>
</span><b><span style="color:#1F497D">Western Digital</span></b><span style="color:#1F497D">®
</span><span style="font-size:10.0pt;color:#1F497D"><o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:10.0pt;color:#1F497D">Email: </span>
<a href="mailto:Thomas.Freeman@wdc.com"><span style="font-size:10.0pt">Thomas.Freeman@wdc.com</span></a><span style="font-size:10.0pt;color:#1F497D"><o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:10.0pt;color:#1F497D">Office: +1-507-322-2311<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt;color:#1F497D"><o:p> </o:p></span></p>
</div>
<p class="MsoNormal"><span style="font-size:11.0pt;color:#1F497D"><o:p> </o:p></span></p>
<div>
<div style="border:none;border-top:solid #E1E1E1 1.0pt;padding:3.0pt 0in 0in 0in">
<p class="MsoNormal"><b><span style="font-size:11.0pt">From:</span></b><span style="font-size:11.0pt"> nvmewin [</span><a href="mailto:nvmewin-bounces@lists.openfabrics.org"><span style="font-size:11.0pt">mailto:nvmewin-bounces@lists.openfabrics.org</span></a><span style="font-size:11.0pt">]
<b>On Behalf Of </b>Robles, Raymond C<br>
<b>Sent:</b> Friday, June 23, 2017 2:17 PM<br>
<b>To:</b> </span><a href="mailto:sm.kumar@samsung.com"><span style="font-size:11.0pt">sm.kumar@samsung.com</span></a><span style="font-size:11.0pt">; Justina Lai <</span><a href="mailto:justina_lai@phison.com"><span style="font-size:11.0pt">justina_lai@phison.com</span></a><span style="font-size:11.0pt">><br>
<b>Cc:</b> Larry Li <</span><a href="mailto:larry_li@phison.com"><span style="font-size:11.0pt">larry_li@phison.com</span></a><span style="font-size:11.0pt">>;
</span><a href="mailto:nvmewin@lists.openfabrics.org"><span style="font-size:11.0pt">nvmewin@lists.openfabrics.org</span></a><span style="font-size:11.0pt"><br>
<b>Subject:</b> Re: [nvmewin] Compatibility issue with 1.5 version nvme driver<o:p></o:p></span></p>
</div>
</div>
<p class="MsoNormal"><o:p> </o:p></p>
<p class="MsoNormal"><span style="font-size:11.0pt;color:#1F497D">Hi Meenakshikumar,<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt;color:#1F497D"><o:p> </o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt;color:#1F497D">Thanks for reviewing this patch and my comments. I agree with your statement below (highlighted).<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt;color:#1F497D"><o:p> </o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt;color:#1F497D">I was attempting to drive the possible solution where cores == queues… when admin commands can come from both Storport and the completion path. As you state, there are still several areas where
a the lock is acquired for commands in the completion path. In order to address, I believe the correct solution is to actually not grab the StartIO lock in the completion path, but instead a DPC lock. Both run at the same IRQL, and will be handled appropriately
by the OS with no deadlock. <o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt;color:#1F497D"><o:p> </o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt;color:#1F497D">Essentially:<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt;color:#1F497D"><o:p> </o:p></span></p>
<p class="MsoListParagraph" style="margin-left:.5in;mso-para-margin-left:0gd;text-indent:-.25in">
<span style="font-size:11.0pt;font-family:Symbol;color:#1F497D">·</span><span style="font-size:7.0pt;font-family:"Times New Roman",serif;color:#1F497D">
</span><span style="font-size:11.0pt;color:#1F497D">The OFA driver is acquiring StartIO Lock from processIo & IOCompletionDpcRoutine. So there is possibility for deadlock when we have an OS command requires more than one NVMe command.<o:p></o:p></span></p>
<p class="MsoListParagraph" style="margin-left:.5in;mso-para-margin-left:0gd;text-indent:-.25in">
<span style="font-size:11.0pt;font-family:Symbol;color:#1F497D">·</span><span style="font-size:7.0pt;font-family:"Times New Roman",serif;color:#1F497D">
</span><span style="font-size:11.0pt;color:#1F497D">I’m proposing we acquire the StartIO Lock from processIo and a DPC Lock from IOCompletionDpcRoutine. So there is no possibility of same lock acquired twice in single call stack.
<o:p></o:p></span></p>
<p class="MsoListParagraph" style="margin-left:.5in;mso-para-margin-left:0gd;text-indent:-.25in">
<span style="font-size:11.0pt;font-family:Symbol;color:#1F497D">·</span><span style="font-size:7.0pt;font-family:"Times New Roman",serif;color:#1F497D">
</span><span style="font-size:11.0pt;color:#1F497D">I would also suggest removing any lingering areas where we issue a new DPC from IOCompletionDpcRoutine to call processIo… so that DPC waiting time for lock is reduced.
<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt;color:#1F497D"><o:p> </o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt;color:#1F497D">Thanks…<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt;color:#1F497D">Ray<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt;color:#1F497D"><o:p> </o:p></span></p>
<div>
<div style="border:none;border-top:solid #E1E1E1 1.0pt;padding:3.0pt 0in 0in 0in">
<p class="MsoNormal"><b><span style="font-size:11.0pt">From:</span></b><span style="font-size:11.0pt"> MEENAKSHIKUMAR SOMASUNDARAM [</span><a href="mailto:sm.kumar@samsung.com"><span style="font-size:11.0pt">mailto:sm.kumar@samsung.com</span></a><span style="font-size:11.0pt">]
<br>
<b>Sent:</b> Friday, June 23, 2017 12:31 AM<br>
<b>To:</b> Robles, Raymond C <</span><a href="mailto:raymond.c.robles@intel.com"><span style="font-size:11.0pt">raymond.c.robles@intel.com</span></a><span style="font-size:11.0pt">>; Justina Lai <</span><a href="mailto:justina_lai@phison.com"><span style="font-size:11.0pt">justina_lai@phison.com</span></a><span style="font-size:11.0pt">><br>
<b>Cc:</b> Larry Li <</span><a href="mailto:larry_li@phison.com"><span style="font-size:11.0pt">larry_li@phison.com</span></a><span style="font-size:11.0pt">>;
</span><a href="mailto:nvmewin@lists.openfabrics.org"><span style="font-size:11.0pt">nvmewin@lists.openfabrics.org</span></a><span style="font-size:11.0pt"><br>
<b>Subject:</b> RE: Re: [nvmewin] Compatibility issue with 1.5 version nvme driver<o:p></o:p></span></p>
</div>
</div>
<p class="MsoNormal"><o:p> </o:p></p>
<p><span style="font-size:11.0pt;font-family:"Calibri",sans-serif">Hi Ray,</span><o:p></o:p></p>
<p> <o:p></o:p></p>
<p><span style="font-size:11.0pt;font-family:"Calibri",sans-serif">This is Meenakshikumar from Samsung. I have reviewed the fix and it looks fine to handle <strong><span style="font-family:"Calibri",sans-serif;color:black">pAE->MultipleCoresToSingleQueueFlag
= TRUE </span></strong><span style="color:black">case while a new admin command is issued in completion path.</span></span><o:p></o:p></p>
<p> <o:p></o:p></p>
<p><span style="font-size:11.0pt;font-family:"Calibri",sans-serif;color:black">For your comment, I am sharing my thoughts below :</span><o:p></o:p></p>
<p style="mso-margin-top-alt:0in;margin-right:0in;margin-bottom:0in;margin-left:.5in;margin-bottom:.0001pt;text-indent:-.25in">
<span style="font-size:11.0pt;font-family:Symbol;color:#1F497D">·</span><span style="font-size:7.0pt;font-family:"Times New Roman",serif;color:#1F497D">
</span><span style="font-size:11.0pt;font-family:"Calibri",sans-serif;color:#1F497D">When multiple cpu cores are mapped with single queue, the patch is removing startIoLock from processIO and startIoLock used only in IOCompletionDpcRoutine:</span><span style="font-size:12.0pt;font-family:"Calibri",sans-serif;color:black"><o:p></o:p></span></p>
<p style="mso-margin-top-alt:0in;margin-right:0in;margin-bottom:0in;margin-left:.5in;margin-bottom:.0001pt;text-indent:-.25in">
<span style="font-size:11.0pt;font-family:Symbol;color:#1F497D">·</span><span style="font-size:7.0pt;font-family:"Times New Roman",serif;color:#1F497D">
</span><span style="font-size:11.0pt;font-family:"Calibri",sans-serif;color:#1F497D">This will block NvmeStartIo when driver is processing a completion entry.
</span><span style="font-size:12.0pt;font-family:"Calibri",sans-serif;color:black"><o:p></o:p></span></p>
<p style="mso-margin-top-alt:0in;margin-right:0in;margin-bottom:0in;margin-left:.5in;margin-bottom:.0001pt;text-indent:-.25in">
<span style="font-size:12.0pt;font-family:"Calibri",sans-serif;color:black"> <o:p></o:p></span></p>
<p style="mso-margin-top-alt:0in;margin-right:0in;margin-bottom:0in;margin-left:.5in;margin-bottom:.0001pt;text-indent:-.25in">
<span style="font-size:11.0pt;font-family:"Calibri",sans-serif;color:black;background:yellow;mso-highlight:yellow">==> In OFA driver, Storport will acquire the StartIoLock before calling NVMeStartIo. We acquire StartIo lock only in the completion path in case
if cores != queues, to synchronize multiple cores trying to access single queue.</span><span style="font-size:12.0pt;font-family:"Calibri",sans-serif;color:black;background:yellow;mso-highlight:yellow"><o:p></o:p></span></p>
<p style="mso-margin-top-alt:0in;margin-right:0in;margin-bottom:0in;margin-left:.5in;margin-bottom:.0001pt;text-indent:-.25in">
<span style="font-size:11.0pt;font-family:"Calibri",sans-serif;color:black;background:yellow;mso-highlight:yellow">So in OFA driver, only if cores == queues, then both StartIO and DPCs will run concurrently.</span><span style="font-size:12.0pt;font-family:"Calibri",sans-serif"> <o:p></o:p></span></p>
<p> <o:p></o:p></p>
<p><span style="font-size:11.0pt;font-family:"Calibri",sans-serif;color:black">Apart from this, in
<strong><span style="font-family:"Calibri",sans-serif">pAE->MultipleCoresToSingleQueueFlag = FALSE case, </span></strong>the intention to synchronize issuing admin command from StartIO & various completion paths is not handled in similar fashion in all cases.
</span><o:p></o:p></p>
<p><span style="font-size:11.0pt;font-family:"Calibri",sans-serif;color:black">For example, in SntiTranslateModeSenseResponse(), FormatNVMGetIdentify(), etc., lock is acquired. But in SntiTranslateTemperatureResponse(), SntiTranslateStartStopUnitResponse(),
SntiTranslateWriteBufferResponse() etc., StartIO lock is not acquired. </span><o:p></o:p></p>
<p><span style="font-size:11.0pt;font-family:"Calibri",sans-serif;color:black">There could be race condition b/w an IOCTL and Completion path trying to issue admin command. This should be addressed in the OFA driver, might be in a different patch.</span><o:p></o:p></p>
<p> <o:p></o:p></p>
<p><span style="font-size:11.0pt;font-family:"Calibri",sans-serif">Thanks,</span><o:p></o:p></p>
<p><span style="font-size:11.0pt;font-family:"Calibri",sans-serif">Meenakshikumar</span><o:p></o:p></p>
<p> <o:p></o:p></p>
<p>--------- <b>Original Message</b> ---------<o:p></o:p></p>
<p><b>Sender</b> : Robles, Raymond C <<a href="mailto:raymond.c.robles@intel.com">raymond.c.robles@intel.com</a>><o:p></o:p></p>
<p><b>Date</b> : 2017-06-23 04:50 (GMT+5:30)<o:p></o:p></p>
<p><b>Title</b> : Re: [nvmewin] Compatibility issue with 1.5 version nvme driver<o:p></o:p></p>
<p> <o:p></o:p></p>
<p class="MsoNormal"><span style="font-size:11.0pt;color:#1F497D">Hi Justina,</span><o:p></o:p></p>
<p class="MsoNormal"><span style="font-size:11.0pt;color:#1F497D"> </span><o:p></o:p></p>
<p class="MsoNormal"><span style="font-size:11.0pt;color:#1F497D">Intel has reviewed your and tested your patch. Please see the following comments:</span><o:p></o:p></p>
<p class="MsoNormal"><span style="font-size:11.0pt;color:#1F497D"> </span><o:p></o:p></p>
<p class="MsoListParagraph" style="margin-left:.5in;mso-para-margin-left:0gd;text-indent:-.25in">
<span style="font-size:11.0pt;font-family:Symbol;color:#1F497D">·</span><span style="font-size:7.0pt;font-family:"Times New Roman",serif;color:#1F497D">
</span><span style="font-size:11.0pt;color:#1F497D">Need to remove code blocks in #if 0
</span><o:p></o:p></p>
<p class="MsoListParagraph" style="margin-left:.5in;mso-para-margin-left:0gd;text-indent:-.25in">
<span style="font-size:11.0pt;font-family:Symbol;color:#1F497D">·</span><span style="font-size:7.0pt;font-family:"Times New Roman",serif;color:#1F497D">
</span><span style="font-size:11.0pt;color:#1F497D">When multiple cpu cores are mapped with single queue, the patch is removing startIoLock from processIO and startIoLock used only in IOCompletionDpcRoutine:</span><o:p></o:p></p>
<p class="MsoListParagraph" style="margin-left:.5in;mso-para-margin-left:0gd;text-indent:-.25in">
<span style="font-size:11.0pt;font-family:Symbol;color:#1F497D">·</span><span style="font-size:7.0pt;font-family:"Times New Roman",serif;color:#1F497D">
</span><span style="font-size:11.0pt;color:#1F497D">This will block NvmeStartIo when driver is processing a completion entry.
</span><o:p></o:p></p>
<p class="MsoListParagraph" style="margin-left:.5in;mso-para-margin-left:0gd;text-indent:-.25in">
<span style="font-size:11.0pt;font-family:Symbol;color:#1F497D">·</span><span style="font-size:7.0pt;font-family:"Times New Roman",serif;color:#1F497D">
</span><span style="font-size:11.0pt;color:#1F497D">A couple of options:</span><o:p></o:p></p>
<p class="MsoListParagraph" style="margin-left:1.0in;mso-para-margin-left:0gd;text-indent:-.25in">
<span style="font-size:11.0pt;font-family:"Courier New";color:#1F497D">o</span><span style="font-size:7.0pt;font-family:"Times New Roman",serif;color:#1F497D">
</span><span style="font-size:11.0pt;color:#1F497D">Indicate in nvmeSnti.c when handling operations that require more than one command to the drive, and have that locking needs passed to ProcessIo appropriately. It does not seem plausible to isolate the changes
to nvmeIO.c only.</span><o:p></o:p></p>
<p class="MsoListParagraph" style="margin-left:1.0in;mso-para-margin-left:0gd;text-indent:-.25in">
<span style="font-size:11.0pt;font-family:"Courier New";color:#1F497D">o</span><span style="font-size:7.0pt;font-family:"Times New Roman",serif;color:#1F497D">
</span><span style="font-size:11.0pt;color:#1F497D">You could also instead remove startIoLock from IoCompletionDpcRoutine while keeping startIoLock at processIo. This way we can avoid double lock and also we can make NvmeStartIo and IoCompletionDpcRoutine parallel.</span><o:p></o:p></p>
<p class="MsoNormal"><span style="font-size:11.0pt;color:#1F497D"> </span><o:p></o:p></p>
<p class="MsoNormal"><span style="font-size:11.0pt;color:#1F497D">The key takeaway is that StartIo and IoCompletionDpcRoutine must be able to run concurrently with no deadlocks. As you’ve coded the patch, a deadlock is possible. Please revise your patch to
account for this scenario.</span><o:p></o:p></p>
<p class="MsoNormal"><span style="font-size:11.0pt;color:#1F497D"> </span><o:p></o:p></p>
<p class="MsoNormal"><span style="font-size:11.0pt;color:#1F497D">Thanks…</span><o:p></o:p></p>
<p class="MsoNormal"><span style="font-size:11.0pt;color:#1F497D">Ray</span><o:p></o:p></p>
<p class="MsoNormal"><span style="font-size:11.0pt;color:#1F497D"> </span><o:p></o:p></p>
<div>
<div style="border:none;border-top:solid #E1E1E1 1.0pt;padding:3.0pt 0in 0in 0in">
<p class="MsoNormal"><b><span style="font-size:11.0pt">From:</span></b><span style="font-size:11.0pt"> nvmewin [</span><a href="mailto:nvmewin-bounces@lists.openfabrics.org"><span style="font-size:11.0pt">mailto:nvmewin-bounces@lists.openfabrics.org</span></a><span style="font-size:11.0pt">]
<b>On Behalf Of </b>Robles, Raymond C<br>
<b>Sent:</b> Thursday, June 22, 2017 3:55 PM<br>
<b>To:</b> Justina Lai <</span><a href="mailto:justina_lai@phison.com"><span style="font-size:11.0pt">justina_lai@phison.com</span></a><span style="font-size:11.0pt">><br>
<b>Cc:</b> </span><a href="mailto:nvmewin@lists.openfabrics.org"><span style="font-size:11.0pt">nvmewin@lists.openfabrics.org</span></a><span style="font-size:11.0pt"><br>
<b>Subject:</b> Re: [nvmewin] Compatibility issue with 1.5 version nvme driver</span><o:p></o:p></p>
</div>
</div>
<p class="MsoNormal"> <o:p></o:p></p>
<p class="MsoNormal"><span style="font-size:11.0pt;color:#1F497D">Hi Justina, </span>
<o:p></o:p></p>
<p class="MsoNormal"><span style="font-size:11.0pt;color:#1F497D"> </span><o:p></o:p></p>
<p class="MsoNormal"><span style="font-size:11.0pt;color:#1F497D">Please go the Open Fabrics Alliance website and subscribe to our reflector on this page:
</span><a href="http://lists.openfabrics.org/mailman/listinfo/nvmewin"><span style="font-size:11.0pt">http://lists.openfabrics.org/mailman/listinfo/nvmewin</span></a><o:p></o:p></p>
<p class="MsoNormal"><span style="font-size:11.0pt;color:#1F497D"> </span><o:p></o:p></p>
<p class="MsoNormal"><span style="font-size:11.0pt;color:#1F497D">The main NVMe WG page can be found here:
</span><a href="https://www.openfabrics.org/index.php/working-groups.html"><span style="font-size:11.0pt">https://www.openfabrics.org/index.php/working-groups.html</span></a><o:p></o:p></p>
<p class="MsoNormal"><span style="font-size:11.0pt;color:#1F497D"> </span><o:p></o:p></p>
<p class="MsoNormal"><span style="font-size:11.0pt;color:#1F497D">Thanks…</span><o:p></o:p></p>
<p class="MsoNormal"><span style="font-size:11.0pt;color:#1F497D">Ray</span><o:p></o:p></p>
<p class="MsoNormal"><span style="font-size:11.0pt;color:#1F497D"> </span><o:p></o:p></p>
<div>
<div style="border:none;border-top:solid #E1E1E1 1.0pt;padding:3.0pt 0in 0in 0in">
<p class="MsoNormal"><b><span style="font-size:11.0pt">From:</span></b><span style="font-size:11.0pt"> Justina Lai [</span><a href="mailto:justina_lai@phison.com"><span style="font-size:11.0pt">mailto:justina_lai@phison.com</span></a><span style="font-size:11.0pt">]
<br>
<b>Sent:</b> Monday, June 19, 2017 11:45 PM<br>
<b>To:</b> Robles, Raymond C <</span><a href="mailto:raymond.c.robles@intel.com"><span style="font-size:11.0pt">raymond.c.robles@intel.com</span></a><span style="font-size:11.0pt">><br>
<b>Cc:</b> </span><a href="mailto:nvmewin@lists.openfabrics.org"><span style="font-size:11.0pt">nvmewin@lists.openfabrics.org</span></a><span style="font-size:11.0pt"><br>
<b>Subject:</b> RE: Compatibility issue with 1.5 version nvme driver</span><o:p></o:p></p>
</div>
</div>
<p class="MsoNormal"> <o:p></o:p></p>
<p class="MsoNormal"><span style="mso-fareast-language:ZH-TW">Dear Raymond,</span><o:p></o:p></p>
<p class="MsoNormal"><span style="mso-fareast-language:ZH-TW"> </span><o:p></o:p></p>
<p class="MsoNormal"><span style="mso-fareast-language:ZH-TW">Yes, I have received your mail regarding unit test.</span><o:p></o:p></p>
<p class="MsoNormal"><span style="mso-fareast-language:ZH-TW">We are now carrying on the test and hope to finish it within this week.</span><o:p></o:p></p>
<p class="MsoNormal"><span style="mso-fareast-language:ZH-TW"> </span><o:p></o:p></p>
<p class="MsoNormal"><span style="mso-fareast-language:ZH-TW">May I know how to officially subscribe to the reflector email list?</span><o:p></o:p></p>
<p class="MsoNormal"><span style="mso-fareast-language:ZH-TW">I saw my mail is already in the Non-digested Members of nvmewin. Should I set as digested mode?</span><o:p></o:p></p>
<p class="MsoNormal"><span style="mso-fareast-language:ZH-TW"> </span><o:p></o:p></p>
<p class="MsoNormal"><span style="mso-fareast-language:ZH-TW">Thank you!</span><o:p></o:p></p>
<p class="MsoNormal"><span style="mso-fareast-language:ZH-TW"> </span><o:p></o:p></p>
<div>
<p class="MsoNormal"><span style="mso-fareast-language:ZH-TW">Best Regards,</span><o:p></o:p></p>
<p class="MsoNormal"><span style="mso-fareast-language:ZH-TW">Justina Lai</span><o:p></o:p></p>
<p class="MsoNormal"><span style="mso-fareast-language:ZH-TW">#5707</span><o:p></o:p></p>
</div>
<p class="MsoNormal"><span style="mso-fareast-language:ZH-TW"> </span><o:p></o:p></p>
<div>
<div style="border:none;border-top:solid #E1E1E1 1.0pt;padding:3.0pt 0in 0in 0in">
<p class="MsoNormal"><b><span style="font-size:11.0pt;mso-fareast-language:ZH-TW">From:</span></b><span style="font-size:11.0pt;mso-fareast-language:ZH-TW"> Robles, Raymond C [</span><a href="mailto:raymond.c.robles@intel.com"><span style="font-size:11.0pt;mso-fareast-language:ZH-TW">mailto:raymond.c.robles@intel.com</span></a><span style="font-size:11.0pt;mso-fareast-language:ZH-TW">]
<br>
<b>Sent:</b> Tuesday, June 20, 2017 5:37 AM<br>
<b>To:</b> Justina Lai <</span><a href="mailto:justina_lai@phison.com"><span style="font-size:11.0pt;mso-fareast-language:ZH-TW">justina_lai@phison.com</span></a><span style="font-size:11.0pt;mso-fareast-language:ZH-TW">><br>
<b>Cc:</b> </span><a href="mailto:nvmewin@lists.openfabrics.org"><span style="font-size:11.0pt;mso-fareast-language:ZH-TW">nvmewin@lists.openfabrics.org</span></a><span style="font-size:11.0pt;mso-fareast-language:ZH-TW"><br>
<b>Subject:</b> RE: Compatibility issue with 1.5 version nvme driver</span><o:p></o:p></p>
</div>
</div>
<p class="MsoNormal"><span style="mso-fareast-language:ZH-TW"> </span><o:p></o:p></p>
<p class="MsoNormal"><span style="font-size:11.0pt;color:#002060;mso-fareast-language:ZH-TW">Hi Justina,</span><o:p></o:p></p>
<p class="MsoNormal"><span style="font-size:11.0pt;color:#002060;mso-fareast-language:ZH-TW"> </span><o:p></o:p></p>
<p class="MsoNormal"><span style="font-size:11.0pt;color:#002060;mso-fareast-language:ZH-TW">Yes, we did receive the email and have been emailing you. I notice that your emails keep getting bounced off the reflector list and I have to approve them. Could you
please officially subscribe to the reflector email list so that your emails are not bounced (which require me to approve and forward).
</span><o:p></o:p></p>
<p class="MsoNormal"><span style="font-size:11.0pt;color:#002060;mso-fareast-language:ZH-TW"> </span><o:p></o:p></p>
<p class="MsoNormal"><span style="font-size:11.0pt;color:#002060;mso-fareast-language:ZH-TW">There was an additional request for you to run the normal unit tests required for all OFA patches. I’ll forward that email again after you officially subscribe the
email list. Did you receive that email?</span><o:p></o:p></p>
<p class="MsoNormal"><span style="font-size:11.0pt;color:#002060;mso-fareast-language:ZH-TW"> </span><o:p></o:p></p>
<p class="MsoNormal"><span style="font-size:11.0pt;color:#002060;mso-fareast-language:ZH-TW">Thanks…</span><o:p></o:p></p>
<p class="MsoNormal"><span style="font-size:11.0pt;color:#002060;mso-fareast-language:ZH-TW">Ray</span><o:p></o:p></p>
<p class="MsoNormal"><span style="font-size:11.0pt;color:#002060;mso-fareast-language:ZH-TW"> </span><o:p></o:p></p>
<div>
<div style="border:none;border-top:solid #E1E1E1 1.0pt;padding:3.0pt 0in 0in 0in">
<p class="MsoNormal"><b><span style="font-size:11.0pt;mso-fareast-language:ZH-TW">From:</span></b><span style="font-size:11.0pt;mso-fareast-language:ZH-TW"> nvmewin [</span><a href="mailto:nvmewin-bounces@lists.openfabrics.org"><span style="font-size:11.0pt;mso-fareast-language:ZH-TW">mailto:nvmewin-bounces@lists.openfabrics.org</span></a><span style="font-size:11.0pt;mso-fareast-language:ZH-TW">]
<b>On Behalf Of </b>Justina Lai<br>
<b>Sent:</b> Wednesday, May 24, 2017 10:18 PM<br>
<b>To:</b> </span><a href="mailto:nvmewin@lists.openfabrics.org"><span style="font-size:11.0pt;mso-fareast-language:ZH-TW">nvmewin@lists.openfabrics.org</span></a><span style="font-size:11.0pt;mso-fareast-language:ZH-TW"><br>
<b>Subject:</b> Re: [nvmewin] Compatibility issue with 1.5 version nvme driver</span><o:p></o:p></p>
</div>
</div>
<p class="MsoNormal"><span style="mso-fareast-language:ZH-TW"> </span><o:p></o:p></p>
<p class="MsoNormal"><span style="mso-fareast-language:ZH-TW">Hi,</span><o:p></o:p></p>
<p class="MsoNormal"><span style="mso-fareast-language:ZH-TW"> </span><o:p></o:p></p>
<p class="MsoNormal"><span style="mso-fareast-language:ZH-TW">Have you received my mail on 5/22?</span><o:p></o:p></p>
<p class="MsoNormal"><span style="mso-fareast-language:ZH-TW">Thanks!</span><o:p></o:p></p>
<p class="MsoNormal"><span style="mso-fareast-language:ZH-TW"> </span><o:p></o:p></p>
<div>
<p class="MsoNormal"><span style="mso-fareast-language:ZH-TW">Best Regards,</span><o:p></o:p></p>
<p class="MsoNormal"><span style="mso-fareast-language:ZH-TW">Justina Lai</span><o:p></o:p></p>
<p class="MsoNormal"><span style="mso-fareast-language:ZH-TW">#5707</span><o:p></o:p></p>
</div>
<p class="MsoNormal"><span style="mso-fareast-language:ZH-TW"> </span><o:p></o:p></p>
<div>
<div style="border:none;border-top:solid #E1E1E1 1.0pt;padding:3.0pt 0in 0in 0in">
<p class="MsoNormal"><b><span style="font-size:11.0pt;mso-fareast-language:ZH-TW">From:</span></b><span style="font-size:11.0pt;mso-fareast-language:ZH-TW"> Justina Lai
<br>
<b>Sent:</b> Monday, May 22, 2017 1:14 PM<br>
<b>To:</b> 'nvmewin@lists.openfabrics.org' <</span><a href="mailto:nvmewin@lists.openfabrics.org"><span style="font-size:11.0pt;mso-fareast-language:ZH-TW">nvmewin@lists.openfabrics.org</span></a><span style="font-size:11.0pt;mso-fareast-language:ZH-TW">><br>
<b>Cc:</b> Larry Li <</span><a href="mailto:larry_li@phison.com"><span style="font-size:11.0pt;mso-fareast-language:ZH-TW">larry_li@phison.com</span></a><span style="font-size:11.0pt;mso-fareast-language:ZH-TW">><br>
<b>Subject:</b> Compatibility issue with 1.5 version nvme driver</span><o:p></o:p></p>
</div>
</div>
<p class="MsoNormal"><span style="mso-fareast-language:ZH-TW"> </span><o:p></o:p></p>
<p class="MsoNormal"><span style="mso-fareast-language:ZH-TW">Dear Sir/Madam,</span><o:p></o:p></p>
<p class="MsoNormal"><span style="mso-fareast-language:ZH-TW"> </span><o:p></o:p></p>
<p class="MsoNormal"><span style="mso-fareast-language:ZH-TW">We are facing compatibility issue on our PS5007 NVMe device with recent 1.5 version nvme driver.</span><o:p></o:p></p>
<p class="MsoNormal"><span style="mso-fareast-language:ZH-TW">Please check below issue description.</span><o:p></o:p></p>
<p class="MsoNormal"><span style="mso-fareast-language:ZH-TW"> </span><o:p></o:p></p>
<p class="MsoNormal"><span style="mso-fareast-language:ZH-TW">Phison PS5007 supports 7 IO queues, and if we use PS5007 with OFA driver on the platform core number>7, ex: 8-core or 12-core PC, we will see PC hang up and cannot enter OS like below:</span><o:p></o:p></p>
<p class="MsoNormal"><img border="0" width="538" height="302" id="Picture_x0020_1" src="cid:image001.jpg@01D30F72.69353760" alt="cid:image001.jpg@01D2EE8D.0DB3EAE0"><o:p></o:p></p>
<p class="MsoNormal"><span style="mso-fareast-language:ZH-TW"> </span><o:p></o:p></p>
<p class="MsoNormal"><span style="mso-fareast-language:ZH-TW">After debugging on our side, we found the fail is caused by below flow:</span><o:p></o:p></p>
<p class="MsoNormal"><span style="mso-fareast-language:ZH-TW">---------------------------------------------------------------------------------------------------------------------------------------------------------------------</span><o:p></o:p></p>
<p class="MsoNormal"><span style="mso-fareast-language:ZH-TW">……….</span><o:p></o:p></p>
<p class="MsoNormal" style="margin-left:.25in;text-indent:-.25in"><span style="mso-fareast-language:ZH-TW">1.</span><span style="font-size:7.0pt;font-family:"Times New Roman",serif;mso-fareast-language:ZH-TW">
</span><span style="mso-fareast-language:ZH-TW">SntiTranslateModeSense()</span><o:p></o:p></p>
<p class="MsoListParagraph" style="margin-left:.25in;mso-para-margin-left:0gd"><span style="mso-fareast-language:ZH-TW">……….</span><o:p></o:p></p>
<p class="MsoNormal"><span style="mso-fareast-language:ZH-TW">2.IoCompletionRoutine()</span><o:p></o:p></p>
<p class="MsoNormal"><span style="mso-fareast-language:ZH-TW"> if (pDpc != NULL) {</span><o:p></o:p></p>
<p class="MsoNormal"><span style="mso-fareast-language:ZH-TW"> ASSERT(pAE->ntldrDump == FALSE);</span><o:p></o:p></p>
<p class="MsoNormal"><span style="mso-fareast-language:ZH-TW"> if (<span style="color:red">pAE->MultipleCoresToSingleQueueFlag</span>) {</span><o:p></o:p></p>
<p class="MsoNormal"><span style="mso-fareast-language:ZH-TW"> <span style="color:red">
StorPortAcquireSpinLock(pAE, StartIoLock, NULL, &StartLockHandle); --------------------</span></span><span style="font-family:Wingdings;color:red;mso-fareast-language:ZH-TW">à</span><span style="color:red;mso-fareast-language:ZH-TW"> execute lock</span><o:p></o:p></p>
<p class="MsoNormal"><span style="mso-fareast-language:ZH-TW"> } else {</span><o:p></o:p></p>
<p class="MsoNormal"><span style="mso-fareast-language:ZH-TW"> StorPortAcquireSpinLock(pAE, DpcLock, pDpc, &DpcLockhandle);</span><o:p></o:p></p>
<p class="MsoNormal"><span style="mso-fareast-language:ZH-TW"> }</span><o:p></o:p></p>
<p class="MsoNormal"><span style="mso-fareast-language:ZH-TW"> }</span><o:p></o:p></p>
<p class="MsoNormal"><span style="mso-fareast-language:ZH-TW">………</span><o:p></o:p></p>
<p class="MsoNormal"><span style="mso-fareast-language:ZH-TW">callStorportNotification = pSrbExtension->pNvmeCompletionRoutine(pAE, (PVOID)pSrbExtension)</span><o:p></o:p></p>
<p class="MsoNormal"><span style="mso-fareast-language:ZH-TW"> && (pSrbExtension->pSrb != NULL);</span><o:p></o:p></p>
<p class="MsoNormal"><span style="mso-fareast-language:ZH-TW">…….</span><o:p></o:p></p>
<p class="MsoNormal"><span style="mso-fareast-language:ZH-TW">3.SntiTranslateModeSenseResponse()</span><o:p></o:p></p>
<p class="MsoListParagraph" style="margin-left:.25in;mso-para-margin-left:0gd"><span style="mso-fareast-language:ZH-TW">…….</span><o:p></o:p></p>
<p class="MsoNormal"><span style="mso-fareast-language:ZH-TW">case MODE_SENSE_RETURN_ALL:</span><o:p></o:p></p>
<p class="MsoNormal"><span style="mso-fareast-language:ZH-TW">…….</span><o:p></o:p></p>
<p class="MsoNormal"><span style="mso-fareast-language:ZH-TW">if (supportsVwc == TRUE) {</span><o:p></o:p></p>
<p class="MsoNormal"><span style="mso-fareast-language:ZH-TW"> </span><o:p></o:p></p>
<p class="MsoNormal"><span style="mso-fareast-language:ZH-TW"> pSrbExt->pNvmeCompletionRoutine = SntiCompletionCallbackRoutine;</span><o:p></o:p></p>
<p class="MsoNormal"><span style="mso-fareast-language:ZH-TW"> </span><o:p></o:p></p>
<p class="MsoNormal"><span style="mso-fareast-language:ZH-TW"> /* Finally, make sure we issue the GET FEATURES command */</span><o:p></o:p></p>
<p class="MsoNormal"><span style="mso-fareast-language:ZH-TW"> SntiBuildGetFeaturesCmd(pSrbExt, VOLATILE_WRITE_CACHE);</span><o:p></o:p></p>
<p class="MsoNormal"><span style="mso-fareast-language:ZH-TW"> </span><o:p></o:p></p>
<p class="MsoNormal"><span style="mso-fareast-language:ZH-TW"> ioStarted =
<span style="color:red">ProcessIo</span>(pSrbExt->pNvmeDevExt,</span><o:p></o:p></p>
<p class="MsoNormal"><span style="mso-fareast-language:ZH-TW"> pSrbExt,</span><o:p></o:p></p>
<p class="MsoNormal"><span style="mso-fareast-language:ZH-TW"> NVME_QUEUE_TYPE_ADMIN,</span><o:p></o:p></p>
<p class="MsoNormal"><span style="mso-fareast-language:ZH-TW">
<span style="color:red">TRUE</span>);</span><o:p></o:p></p>
<p class="MsoNormal"><span style="mso-fareast-language:ZH-TW">
</span><o:p></o:p></p>
<p class="MsoNormal"><span style="mso-fareast-language:ZH-TW"> </span><o:p></o:p></p>
<p class="MsoNormal"><span style="mso-fareast-language:ZH-TW">4.ProcessIo() </span>
<o:p></o:p></p>
<p class="MsoNormal"><span style="mso-fareast-language:ZH-TW"> ……..</span><o:p></o:p></p>
<p class="MsoNormal"><span style="mso-fareast-language:ZH-TW"> if (<span style="color:red">AcquireLock == TRUE</span>) {</span><o:p></o:p></p>
<p class="MsoNormal"><span style="mso-fareast-language:ZH-TW"> <span style="color:red">StorPortAcquireSpinLock</span>(pAdapterExtension,</span><o:p></o:p></p>
<p class="MsoNormal"><span style="mso-fareast-language:ZH-TW"> StartIoLock,</span><o:p></o:p></p>
<p class="MsoNormal"><span style="mso-fareast-language:ZH-TW"> NULL,</span><o:p></o:p></p>
<p class="MsoNormal"><span style="mso-fareast-language:ZH-TW"> &hStartIoLock);
<span style="color:red">----------------------</span></span><span style="font-family:Wingdings;color:red;mso-fareast-language:ZH-TW">à</span><span style="color:red;mso-fareast-language:ZH-TW"> double execute lock and cause (Assertion failure - code c0000420)</span><o:p></o:p></p>
<p class="MsoNormal"><span style="color:red;mso-fareast-language:ZH-TW"> </span><o:p></o:p></p>
<p class="MsoNormal"><span style="mso-fareast-language:ZH-TW"> </span><o:p></o:p></p>
<p class="MsoNormal"><span style="mso-fareast-language:ZH-TW"> } </span>
<o:p></o:p></p>
<p class="MsoNormal"><span style="mso-fareast-language:ZH-TW">---------------------------------------------------------------------------------------------------------------------------------------------------------------------</span><o:p></o:p></p>
<p class="MsoNormal"><span style="mso-fareast-language:ZH-TW"> </span><o:p></o:p></p>
<p class="MsoNormal"><span style="mso-fareast-language:ZH-TW">Since many users are now using our PS5007 device with 8-core or 12-core PC, they are facing this issue right now. Could you please help to modify driver to solve this problem asap?</span><o:p></o:p></p>
<p class="MsoNormal"><span style="mso-fareast-language:ZH-TW"> </span><o:p></o:p></p>
<p class="MsoNormal"><span style="mso-fareast-language:ZH-TW">Any unclear point, please kindly let us know.</span><o:p></o:p></p>
<p class="MsoNormal"><span style="mso-fareast-language:ZH-TW">Thank you very much for the help!</span><o:p></o:p></p>
<p class="MsoNormal"><span style="mso-fareast-language:ZH-TW"> </span><o:p></o:p></p>
<p class="MsoNormal"><span style="mso-fareast-language:ZH-TW">Best Regards,</span><o:p></o:p></p>
<p class="MsoNormal"><span style="mso-fareast-language:ZH-TW">Justina Lai</span><o:p></o:p></p>
<p class="MsoNormal"><span style="mso-fareast-language:ZH-TW">#5707</span><o:p></o:p></p>
<p class="MsoNormal"><span style="mso-fareast-language:ZH-TW"> </span><o:p></o:p></p>
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto"><span style="font-family:"Times New Roman",serif;mso-fareast-language:ZH-TW"><br>
</span><span style="font-family:"Arial",sans-serif;color:gray;mso-fareast-language:ZH-TW"><br>
This message and any attachments are confidential and may be legally privileged. Any unauthorized review, use or distribution by anyone other than the intended recipient is strictly prohibited. If you are not the intended recipient, please immediately notify
the sender, completely delete the message and any attachments, and destroy all copies. Your cooperation will be highly appreciated.</span><o:p></o:p></p>
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto"><span style="font-family:"Times New Roman",serif;mso-fareast-language:ZH-TW"><br>
</span><span style="font-family:"Arial",sans-serif;color:gray;mso-fareast-language:ZH-TW"><br>
This message and any attachments are confidential and may be legally privileged. Any unauthorized review, use or distribution by anyone other than the intended recipient is strictly prohibited. If you are not the intended recipient, please immediately notify
the sender, completely delete the message and any attachments, and destroy all copies. Your cooperation will be highly appreciated.</span><o:p></o:p></p>
<pre>_______________________________________________<o:p></o:p></pre>
<pre>nvmewin mailing list<o:p></o:p></pre>
<pre><a href="mailto:nvmewin@lists.openfabrics.org">nvmewin@lists.openfabrics.org</a><o:p></o:p></pre>
<pre><a href="http://lists.openfabrics.org/mailman/listinfo/nvmewin">http://lists.openfabrics.org/mailman/listinfo/nvmewin</a><o:p></o:p></pre>
<table class="MsoNormalTable" border="0" cellspacing="3" cellpadding="0" id="confidentialsignimg">
<tbody>
<tr>
<td style="padding:.75pt .75pt .75pt .75pt">
<p><img border="0" width="520" height="200" id="Picture_x0020_2" src="cid:image002.png@01D30F72.69353760" alt="cid:image002.png@01D2EE8D.0DB3EAE0"><o:p></o:p></p>
</td>
</tr>
</tbody>
</table>
<p class="MsoNormal"><span style="font-family:"Times New Roman",serif"><img border="0" width="32" height="32" id="_x0000_i1027" src="http://ext.samsung.net/mail/ext/v1/external/status/update?userid=sm.kumar&do=bWFpbElEPTIwMTcwNjIzMDczMTI4ZXBjbXM1cDY4MTZmNDk4MDA2ODllYzdlY2VlMGI0YjdlNzdmN2I3MSZyZWNpcGllbnRBZGRyZXNzPXJheW1vbmQuYy5yb2JsZXNAaW50ZWwuY29t" alt="http://ext.samsung.net/mail/ext/v1/external/status/update?userid=sm.kumar&do=bWFpbElEPTIwMTcwNjIzMDczMTI4ZXBjbXM1cDY4MTZmNDk4MDA2ODllYzdlY2VlMGI0YjdlNzdmN2I3MSZyZWNpcGllbnRBZGRyZXNzPXJheW1vbmQuYy5yb2JsZXNAaW50ZWwuY29t"><o:p></o:p></span></p>
<div>
<p class="MsoNormal"><span style="font-family:"Times New Roman",serif">Western Digital Corporation (and its subsidiaries) E-mail Confidentiality Notice & Disclaimer:<o:p></o:p></span></p>
<div>
<p class="MsoNormal"><span style="font-family:"Times New Roman",serif">This e-mail and any files transmitted with it may contain confidential or legally privileged information of WDC and/or its affiliates, and are intended solely for the use of the individual
or entity to which they are addressed. If you are not the intended recipient, any disclosure, copying, distribution or any action taken or omitted to be taken in reliance on it, is prohibited. If you have received this e-mail in error, please notify the sender
immediately and delete the e-mail in its entirety from your system.<o:p></o:p></span></p>
</div>
</div>
<div>
<p class="MsoNormal"><span style="font-family:"Times New Roman",serif">Western Digital Corporation (and its subsidiaries) E-mail Confidentiality Notice & Disclaimer:<o:p></o:p></span></p>
<div>
<p class="MsoNormal"><span style="font-family:"Times New Roman",serif">This e-mail and any files transmitted with it may contain confidential or legally privileged information of WDC and/or its affiliates, and are intended solely for the use of the individual
or entity to which they are addressed. If you are not the intended recipient, any disclosure, copying, distribution or any action taken or omitted to be taken in reliance on it, is prohibited. If you have received this e-mail in error, please notify the sender
immediately and delete the e-mail in its entirety from your system.<o:p></o:p></span></p>
</div>
</div>
</div>
</body>
</html>