<html xmlns:v="urn:schemas-microsoft-com:vml" xmlns:o="urn:schemas-microsoft-com:office:office" xmlns:w="urn:schemas-microsoft-com:office:word" xmlns:m="http://schemas.microsoft.com/office/2004/12/omml" xmlns="http://www.w3.org/TR/REC-html40">
<head>
<meta http-equiv="Content-Type" content="text/html; charset=gb2312">
<meta name="Generator" content="Microsoft Word 14 (filtered medium)">
<!--[if !mso]><style>v\:* {behavior:url(#default#VML);}
o\:* {behavior:url(#default#VML);}
w\:* {behavior:url(#default#VML);}
.shape {behavior:url(#default#VML);}
</style><![endif]--><style><!--
/* Font Definitions */
@font-face
{font-family:SimSun;
panose-1:2 1 6 0 3 1 1 1 1 1;}
@font-face
{font-family:SimSun;
panose-1:2 1 6 0 3 1 1 1 1 1;}
@font-face
{font-family:Cambria;
panose-1:2 4 5 3 5 4 6 3 2 4;}
@font-face
{font-family:Calibri;
panose-1:2 15 5 2 2 2 4 3 2 4;}
@font-face
{font-family:Tahoma;
panose-1:2 11 6 4 3 5 4 4 2 4;}
@font-face
{font-family:STXihei;}
@font-face
{font-family:"\@SimSun";
panose-1:2 1 6 0 3 1 1 1 1 1;}
/* Style Definitions */
p.MsoNormal, li.MsoNormal, div.MsoNormal
{margin:0in;
margin-bottom:.0001pt;
font-size:11.0pt;
font-family:"Calibri","sans-serif";
mso-fareast-language:ZH-CN;}
a:link, span.MsoHyperlink
{mso-style-priority:99;
color:blue;
text-decoration:underline;}
a:visited, span.MsoHyperlinkFollowed
{mso-style-priority:99;
color:purple;
text-decoration:underline;}
p.MsoAcetate, li.MsoAcetate, div.MsoAcetate
{mso-style-priority:99;
mso-style-link:"Balloon Text Char";
margin:0in;
margin-bottom:.0001pt;
font-size:8.0pt;
font-family:"Tahoma","sans-serif";
mso-fareast-language:ZH-CN;}
p.MsoListParagraph, li.MsoListParagraph, div.MsoListParagraph
{mso-style-priority:34;
margin-top:0in;
margin-right:0in;
margin-bottom:0in;
margin-left:.5in;
margin-bottom:.0001pt;
font-size:11.0pt;
font-family:"Calibri","sans-serif";
mso-fareast-language:ZH-CN;}
span.BalloonTextChar
{mso-style-name:"Balloon Text Char";
mso-style-priority:99;
mso-style-link:"Balloon Text";
font-family:"Tahoma","sans-serif";}
span.EmailStyle20
{mso-style-type:personal;
font-family:"Calibri","sans-serif";
color:windowtext;}
span.EmailStyle21
{mso-style-type:personal;
font-family:"Calibri","sans-serif";
color:#1F497D;}
span.EmailStyle22
{mso-style-type:personal;
font-family:"Calibri","sans-serif";
color:#1F497D;}
span.EmailStyle23
{mso-style-type:personal;
font-family:"Calibri","sans-serif";
color:#1F497D;}
span.EmailStyle24
{mso-style-type:personal;
font-family:"Calibri","sans-serif";
color:#1F497D;}
span.EmailStyle25
{mso-style-type:personal;
font-family:"Calibri","sans-serif";
color:#1F497D;}
span.EmailStyle26
{mso-style-type:personal;
font-family:"Calibri","sans-serif";
color:#1F497D;}
span.EmailStyle27
{mso-style-type:personal;
font-family:"Calibri","sans-serif";
color:#1F497D;}
span.EmailStyle28
{mso-style-type:personal;
font-family:"Calibri","sans-serif";
color:#1F497D;}
span.EmailStyle29
{mso-style-type:personal;
font-family:"Calibri","sans-serif";
color:#1F497D;}
span.EmailStyle30
{mso-style-type:personal-reply;
font-family:"Calibri","sans-serif";
color:#1F497D;}
.MsoChpDefault
{mso-style-type:export-only;
font-size:10.0pt;}
@page WordSection1
{size:8.5in 11.0in;
margin:1.0in 1.0in 1.0in 1.0in;}
div.WordSection1
{page:WordSection1;}
/* List Definitions */
@list l0
{mso-list-id:556473629;
mso-list-type:hybrid;
mso-list-template-ids:-563845728 67698703 67698713 67698715 67698703 67698713 67698715 67698703 67698713 67698715;}
@list l0:level1
{mso-level-tab-stop:none;
mso-level-number-position:left;
text-indent:-.25in;}
@list l0:level2
{mso-level-tab-stop:1.0in;
mso-level-number-position:left;
text-indent:-.25in;}
@list l0:level3
{mso-level-tab-stop:1.5in;
mso-level-number-position:left;
text-indent:-.25in;}
@list l0:level4
{mso-level-tab-stop:2.0in;
mso-level-number-position:left;
text-indent:-.25in;}
@list l0:level5
{mso-level-tab-stop:2.5in;
mso-level-number-position:left;
text-indent:-.25in;}
@list l0:level6
{mso-level-tab-stop:3.0in;
mso-level-number-position:left;
text-indent:-.25in;}
@list l0:level7
{mso-level-tab-stop:3.5in;
mso-level-number-position:left;
text-indent:-.25in;}
@list l0:level8
{mso-level-tab-stop:4.0in;
mso-level-number-position:left;
text-indent:-.25in;}
@list l0:level9
{mso-level-tab-stop:4.5in;
mso-level-number-position:left;
text-indent:-.25in;}
@list l1
{mso-list-id:1857302802;
mso-list-type:hybrid;
mso-list-template-ids:-1845313730 67698689 67698691 67698693 67698689 67698691 67698693 67698689 67698691 67698693;}
@list l1:level1
{mso-level-number-format:bullet;
mso-level-text:\F0B7;
mso-level-tab-stop:none;
mso-level-number-position:left;
text-indent:-.25in;
font-family:Symbol;}
@list l1:level2
{mso-level-tab-stop:1.0in;
mso-level-number-position:left;
text-indent:-.25in;}
@list l1:level3
{mso-level-tab-stop:1.5in;
mso-level-number-position:left;
text-indent:-.25in;}
@list l1:level4
{mso-level-tab-stop:2.0in;
mso-level-number-position:left;
text-indent:-.25in;}
@list l1:level5
{mso-level-tab-stop:2.5in;
mso-level-number-position:left;
text-indent:-.25in;}
@list l1:level6
{mso-level-tab-stop:3.0in;
mso-level-number-position:left;
text-indent:-.25in;}
@list l1:level7
{mso-level-tab-stop:3.5in;
mso-level-number-position:left;
text-indent:-.25in;}
@list l1:level8
{mso-level-tab-stop:4.0in;
mso-level-number-position:left;
text-indent:-.25in;}
@list l1:level9
{mso-level-tab-stop:4.5in;
mso-level-number-position:left;
text-indent:-.25in;}
ol
{margin-bottom:0in;}
ul
{margin-bottom:0in;}
--></style><!--[if gte mso 9]><xml>
<o:shapedefaults v:ext="edit" spidmax="1027" />
</xml><![endif]--><!--[if gte mso 9]><xml>
<o:shapelayout v:ext="edit">
<o:idmap v:ext="edit" data="1" />
</o:shapelayout></xml><![endif]-->
</head>
<body lang="EN-US" link="blue" vlink="purple">
<div class="WordSection1">
<p class="MsoNormal">Hi Yong, I also have some feedback and questions about the code changes.<o:p></o:p></p>
<p class="MsoNormal"><o:p> </o:p></p>
<p class="MsoListParagraph" style="text-indent:-.25in">1.<span style="font-size:7.0pt;font-family:"Times New Roman","serif""> </span>I am surprised that there are no ntldrDump checks around the DPC initialization and calls. I wouldn’t have expected the
DPCs to work properly on the Windows 7 systems. <o:p></o:p></p>
<p class="MsoListParagraph" style="text-indent:-.25in">2.<span style="font-size:7.0pt;font-family:"Times New Roman","serif"">
</span>In function NVMeAllocateMem, the ntldrDump check is wrapped such that if no buffer is allocated from the DumpBuffer, the code path could end up calling StorPortAllocateContiguousMemory in ntldrDump mode. Will this cause a BSOD? Or will it just fail?<o:p></o:p></p>
<p class="MsoListParagraph" style="text-indent:-.25in">3.<span style="font-size:7.0pt;font-family:"Times New Roman","serif"">
</span>In RecoveryDpcRoutine new code has been added above the reset adapter call not related to ntldrDump. If the controller isn’t responding, this additional delay time could cause a DPC watchdog timeout bugcheck if the maximum time allowed for a DPC is
exceeded. I have some concerns about this new code, what was your reasoning for adding it? <o:p></o:p></p>
<p class="MsoNormal"><o:p> </o:p></p>
<p class="MsoNormal">Thanks,<o:p></o:p></p>
<p class="MsoNormal">Carolyn<o:p></o:p></p>
<p class="MsoNormal"><span style="color:#1F497D"><o:p> </o:p></span></p>
<div>
<div style="border:none;border-top:solid #B5C4DF 1.0pt;padding:3.0pt 0in 0in 0in">
<p class="MsoNormal"><b><span style="font-size:10.0pt;font-family:"Tahoma","sans-serif"">From:</span></b><span style="font-size:10.0pt;font-family:"Tahoma","sans-serif""> nvmewin-bounces@lists.openfabrics.org [mailto:nvmewin-bounces@lists.openfabrics.org]
<b>On Behalf Of </b>Judy Brock-SSI<br>
<b>Sent:</b> Monday, November 25, 2013 3:18 PM<br>
<b>To:</b> Yong Chen; Alex Chang; nvmewin@lists.openfabrics.org<br>
<b>Subject:</b> Re: [nvmewin] code review: crash dump & hibernation support<o:p></o:p></span></p>
</div>
</div>
<p class="MsoNormal"><o:p> </o:p></p>
<p class="MsoNormal"><span style="color:#1F497D">Hi Yong,<o:p></o:p></span></p>
<p class="MsoNormal"><span style="color:#1F497D"><o:p> </o:p></span></p>
<p class="MsoNormal"><span style="color:#1F497D">Ensuring CSTS.RDY == 0 before setting CC.EN to ‘1’ is required by the spec – it is a necessary part of enabling the adapter. Therefore it is not overloading the function and should be kept together. It is
a sanity check that needs to take place <u>immediately </u>before writing CC.EN. They should not be separated by other blocks of code. That is not flexibility, it is a design flaw in my opinion.<o:p></o:p></span></p>
<p class="MsoNormal"><span style="color:#1F497D"><o:p> </o:p></span></p>
<p class="MsoNormal"><span style="color:#1F497D">I don’t see how it could possibly result in any destabilization of major features to make sure the RDY bit is not on before setting EN bit in the routine which is dedicated to enabling the controller. If you
are worried about removing other checks for CSTS.RDY == 0, then by all means, leave them in. It doesn’t a hurt a thing to have those xtra check points in the two non-runtime paths you mentioned.
<o:p></o:p></span></p>
<p class="MsoNormal"><span style="color:#1F497D"><o:p> </o:p></span></p>
<p class="MsoNormal"><span style="color:#1F497D">Conversely, it does potentially hurt to not have an explicit check in the NVMeEnableAdapter itself.
<o:p></o:p></span></p>
<p class="MsoNormal"><span style="color:#1F497D"><o:p> </o:p></span></p>
<p class="MsoNormal"><span style="color:#1F497D">As I mentioned previously, there is no check in the PassiveInitialization path for CSTS.RDY == 0 before calling NVMeEnableAdapter in the current code; so we are still violating the spec if we don’t enhance
your current changes one way or the other.<o:p></o:p></span></p>
<p class="MsoNormal"><span style="color:#1F497D"><o:p> </o:p></span></p>
<p class="MsoNormal"><span style="color:#1F497D">I say, put the fix in – it’s fairly trivial.<o:p></o:p></span></p>
<p class="MsoNormal"><span style="color:#1F497D"><o:p> </o:p></span></p>
<p class="MsoNormal"><span style="color:#1F497D">Thanks,<o:p></o:p></span></p>
<p class="MsoNormal"><span style="color:#1F497D">Judy<o:p></o:p></span></p>
<p class="MsoNormal"><span style="color:#1F497D"><o:p> </o:p></span></p>
<p class="MsoNormal"><span style="color:#1F497D"><o:p> </o:p></span></p>
<div>
<div style="border:none;border-top:solid #B5C4DF 1.0pt;padding:3.0pt 0in 0in 0in">
<p class="MsoNormal"><b><span style="font-size:10.0pt;font-family:"Tahoma","sans-serif"">From:</span></b><span style="font-size:10.0pt;font-family:"Tahoma","sans-serif""> Yong Chen [<a href="mailto:Yong.sc.Chen@huawei.com">mailto:Yong.sc.Chen@huawei.com</a>]
<br>
<b>Sent:</b> Monday, November 25, 2013 10:53 AM<br>
<b>To:</b> Judy Brock-SSI; Alex Chang; <a href="mailto:nvmewin@lists.openfabrics.org">
nvmewin@lists.openfabrics.org</a><br>
<b>Subject:</b> RE: [nvmewin] code review: crash dump & hibernation support<o:p></o:p></span></p>
</div>
</div>
<p class="MsoNormal"><o:p> </o:p></p>
<p class="MsoNormal"><span style="color:#1F497D">Hi, Judy,<o:p></o:p></span></p>
<p class="MsoNormal"><span style="color:#1F497D"><o:p> </o:p></span></p>
<p class="MsoNormal"><span style="color:#1F497D">Thanks for your input. I agreed with what you trying to achieve. I also think that block of cycling CC.EN 1->0 can be further refactored into one standalone helper function, to be called by RecoveryDpcRoutine()
and NVMeInitialize(). <o:p></o:p></span></p>
<p class="MsoNormal"><span style="color:#1F497D"><o:p> </o:p></span></p>
<p class="MsoNormal"><span style="color:#1F497D">Embedding into NVMeEnableAdapter() would make that function overloaded more than its name & meant to do, and losing the flexibility. Plus they are separated by other blocks of code and would materially change
the code, currently for no obvious reason. I would try the next guy to the right thing, it is always hard to fix something, if ever happened in the future.<o:p></o:p></span></p>
<p class="MsoNormal"><span style="color:#1F497D"><o:p> </o:p></span></p>
<p class="MsoNormal"><span style="color:#1F497D">Unless the test is completely reset for this check-in, I would delay this further improvement of refactoring to next time, to keep it separate from and avoid destabilizing this major feature work.<o:p></o:p></span></p>
<p class="MsoNormal"><span style="color:#1F497D"><o:p> </o:p></span></p>
<p class="MsoNormal"><span style="color:#1F497D">What do other folks think?<o:p></o:p></span></p>
<p class="MsoNormal"><span style="color:#1F497D"><o:p> </o:p></span></p>
<p class="MsoNormal"><span style="color:#1F497D">Thanks,<o:p></o:p></span></p>
<p class="MsoNormal"><span style="color:#1F497D"><o:p> </o:p></span></p>
<p class="MsoNormal"><span style="color:#1F497D">Yong<o:p></o:p></span></p>
<p class="MsoNormal"><span style="color:#1F497D"><o:p> </o:p></span></p>
<div>
<div style="border:none;border-top:solid #B5C4DF 1.0pt;padding:3.0pt 0in 0in 0in">
<p class="MsoNormal"><b><span style="font-size:10.0pt;font-family:"Tahoma","sans-serif"">From:</span></b><span style="font-size:10.0pt;font-family:"Tahoma","sans-serif""> Judy Brock-SSI [<a href="mailto:judy.brock@ssi.samsung.com">mailto:judy.brock@ssi.samsung.com</a>]
<br>
<b>Sent:</b> Sunday, November 24, 2013 8:20 AM<br>
<b>To:</b> Yong Chen; Alex Chang; <a href="mailto:nvmewin@lists.openfabrics.org">
nvmewin@lists.openfabrics.org</a><br>
<b>Subject:</b> RE: [nvmewin] code review: crash dump & hibernation support<o:p></o:p></span></p>
</div>
</div>
<p class="MsoNormal"><o:p> </o:p></p>
<p class="MsoNormal" style="text-autospace:none"><span style="font-family:"Courier New"">Hi Yong,<o:p></o:p></span></p>
<p class="MsoNormal" style="text-autospace:none"><span style="font-family:"Courier New""><o:p> </o:p></span></p>
<p class="MsoNormal" style="text-autospace:none"><span style="font-family:"Courier New"">I suggest you change the function NVMeEnableAdapter to check for CSTS.RDY == 0 before setting CC.EN to ‘1’.<o:p></o:p></span></p>
<p class="MsoNormal" style="text-autospace:none"><span style="font-family:"Courier New""><o:p> </o:p></span></p>
<p class="MsoNormal" style="text-autospace:none"><span style="font-family:"Courier New"">You added a check for this some lines above the call in NVMeInitialize. But I think we should avoid decoupling the check for CSTS.RDY == 0 from the controller enable itself.
If it is not in the same function, it can be overlooked. For exampley, there is another call to NVMeEnableAdapter in PassiveIntialize that doesn’t check before calling.<o:p></o:p></span></p>
<p class="MsoNormal" style="text-autospace:none"><span style="font-family:"Courier New""><o:p> </o:p></span></p>
<p class="MsoNormal" style="text-autospace:none"><span style="font-family:"Courier New"">I would modify NVMeEnableAdapter as below (additions/changes in highlight), change the prototype, and have callers check for success or failure:<o:p></o:p></span></p>
<p class="MsoNormal" style="text-autospace:none"><span style="font-family:"Courier New";color:green;background:yellow;mso-highlight:yellow"><o:p> </o:p></span></p>
<p class="MsoNormal" style="text-autospace:none"><span style="font-family:"Courier New";color:green;background:yellow;mso-highlight:yellow"><o:p> </o:p></span></p>
<p class="MsoNormal" style="text-autospace:none"><span style="font-family:"Courier New";color:green;background:yellow;mso-highlight:yellow"><o:p> </o:p></span></p>
<p class="MsoNormal" style="text-autospace:none"><span style="font-family:"Courier New";color:green;background:yellow;mso-highlight:yellow">* @return BOOLEAN<o:p></o:p></span></p>
<p class="MsoNormal" style="text-autospace:none"><span style="font-family:"Courier New";color:green;background:yellow;mso-highlight:yellow">* TRUE - If Adapter is enabled correctly<o:p></o:p></span></p>
<p class="MsoNormal" style="text-autospace:none"><span style="font-family:"Courier New";color:green;background:yellow;mso-highlight:yellow">* FALSE - If anything goes wrong</span><span style="font-family:"Courier New";color:green"><o:p></o:p></span></p>
<p class="MsoNormal" style="text-autospace:none"><span style="font-family:"Courier New";color:green">******************************************************************************/<o:p></o:p></span></p>
<p class="MsoNormal" style="text-autospace:none"><span style="font-family:"Courier New";background:yellow;mso-highlight:yellow">BOOLEAN</span><span style="font-family:"Courier New""> NVMeEnableAdapter(<o:p></o:p></span></p>
<p class="MsoNormal" style="text-autospace:none"><span style="font-family:"Courier New""> PNVME_DEVICE_EXTENSION pAE<o:p></o:p></span></p>
<p class="MsoNormal" style="text-autospace:none"><span style="font-family:"Courier New"">)<o:p></o:p></span></p>
<p class="MsoNormal" style="text-autospace:none"><span style="font-family:"Courier New"">{<o:p></o:p></span></p>
<p class="MsoNormal" style="text-autospace:none"><span style="font-family:"Courier New""> PQUEUE_INFO pQI = &pAE->QueueInfo;<o:p></o:p></span></p>
<p class="MsoNormal" style="text-autospace:none"><span style="font-family:"Courier New""> NVMe_CONTROLLER_CONFIGURATION CC = {0};<o:p></o:p></span></p>
<p class="MsoNormal" style="text-autospace:none"><span style="font-family:"Courier New"">
<span style="background:yellow;mso-highlight:yellow">NVMe_CONTROLLER_STATUS CSTS = {0};</span><o:p></o:p></span></p>
<p class="MsoNormal" style="text-autospace:none"><span style="font-family:"Courier New"">
<span style="background:yellow;mso-highlight:yellow">ULONG PollMax = pAE->uSecCrtlTimeout / MAX_STATE_STALL_us;<o:p></o:p></span></span></p>
<p class="MsoNormal" style="text-autospace:none"><span style="font-family:"Courier New";background:yellow;mso-highlight:yellow"> ULONG PollCount;</span><span style="font-family:"Courier New""><o:p></o:p></span></p>
<p class="MsoNormal" style="text-autospace:none"><span style="font-family:"Courier New""><o:p> </o:p></span></p>
<p class="MsoNormal" style="text-autospace:none"><span style="font-family:"Courier New"">
<span style="color:green">/*<o:p></o:p></span></span></p>
<p class="MsoNormal" style="text-autospace:none"><span style="font-family:"Courier New";color:green"> * Program Admin queue registers before enabling the adapter:<o:p></o:p></span></p>
<p class="MsoNormal" style="text-autospace:none"><span style="font-family:"Courier New";color:green"> * Admin Queue Attributes<o:p></o:p></span></p>
<p class="MsoNormal" style="text-autospace:none"><span style="font-family:"Courier New";color:green"> */<o:p></o:p></span></p>
<p class="MsoNormal" style="text-autospace:none"><span style="font-family:"Courier New""> StorPortWriteRegisterUlong(<o:p></o:p></span></p>
<p class="MsoNormal" style="text-autospace:none"><span style="font-family:"Courier New""> pAE,<o:p></o:p></span></p>
<p class="MsoNormal" style="text-autospace:none"><span style="font-family:"Courier New""> (PULONG)(&pAE->pCtrlRegister->AQA),<o:p></o:p></span></p>
<p class="MsoNormal" style="text-autospace:none"><span style="font-family:"Courier New""> (pQI->pSubQueueInfo->SubQEntries - 1) +<o:p></o:p></span></p>
<p class="MsoNormal" style="text-autospace:none"><span style="font-family:"Courier New""> ((pQI->pCplQueueInfo->CplQEntries - 1) << NVME_AQA_CQS_LSB));<o:p></o:p></span></p>
<p class="MsoNormal" style="margin-left:.5in"><b><span style="font-size:12.0pt">.<o:p></o:p></span></b></p>
<p class="MsoNormal" style="margin-left:.5in"><b><span style="font-size:12.0pt">.<o:p></o:p></span></b></p>
<p class="MsoNormal" style="margin-left:.5in"><b><span style="font-size:12.0pt">.<o:p></o:p></span></b></p>
<p class="MsoNormal" style="margin-left:.5in"><i><span style="font-family:"Courier New"">(further down):<o:p></o:p></span></i></p>
<p class="MsoNormal" style="margin-left:.5in;text-autospace:none"><span style="font-family:"Courier New""><o:p> </o:p></span></p>
<p class="MsoNormal" style="margin-left:.5in;text-autospace:none"><span style="font-family:"Courier New"">StorPortDebugPrint(INFO,
<span style="color:#A31515">"NVMeEnableAdapter: Setting EN...\n"</span>);<o:p></o:p></span></p>
<p class="MsoNormal" style="margin-left:.5in;text-autospace:none"><span style="font-family:"Courier New"">
<span style="color:green">/*<o:p></o:p></span></span></p>
<p class="MsoNormal" style="margin-left:.5in;text-autospace:none"><span style="font-family:"Courier New";color:green"> * Set up Controller Configuration Register<o:p></o:p></span></p>
<p class="MsoNormal" style="margin-left:.5in;text-autospace:none"><span style="font-family:"Courier New";color:green"> */<o:p></o:p></span></p>
<p class="MsoNormal" style="margin-left:.5in;text-autospace:none"><span style="font-family:"Courier New"">
<o:p></o:p></span></p>
<p class="MsoNormal" style="margin-left:.5in;text-autospace:none"><span style="font-family:"Courier New"">
<span style="color:green;background:yellow;mso-highlight:yellow">/* After reset, we must wait for CSTS.RDY == 0 before setting CC.EN to 1 */<o:p></o:p></span></span></p>
<p class="MsoNormal" style="margin-left:.5in;text-autospace:none"><span style="font-family:"Courier New";background:yellow;mso-highlight:yellow">
<o:p></o:p></span></p>
<p class="MsoNormal" style="margin-left:.5in;text-autospace:none"><span style="font-family:"Courier New";background:yellow;mso-highlight:yellow"> <span style="color:blue">for</span> (PollCount = 0; PollCount < PollMax; PollCount++) {<o:p></o:p></span></p>
<p class="MsoNormal" style="margin-left:.5in;text-autospace:none"><span style="font-family:"Courier New";background:yellow;mso-highlight:yellow"> CSTS.AsUlong = StorPortReadRegisterUlong(pAE,<o:p></o:p></span></p>
<p class="MsoNormal" style="margin-left:.5in;text-autospace:none"><span style="font-family:"Courier New";background:yellow;mso-highlight:yellow"> (PULONG)(&pAE->pCtrlRegister->CSTS.AsUlong));<o:p></o:p></span></p>
<p class="MsoNormal" style="margin-left:.5in;text-autospace:none"><span style="font-family:"Courier New";background:yellow;mso-highlight:yellow"><o:p> </o:p></span></p>
<p class="MsoNormal" style="margin-left:.5in;text-autospace:none"><span style="font-family:"Courier New";background:yellow;mso-highlight:yellow">
<span style="color:blue">if</span> (CSTS.RDY == 0) {<o:p></o:p></span></p>
<p class="MsoNormal" style="margin-left:.5in;text-autospace:none"><span style="font-family:"Courier New";background:yellow;mso-highlight:yellow">
<span style="color:green">/* Move on if RDY bit is cleared */<o:p></o:p></span></span></p>
<p class="MsoNormal" style="margin-left:.5in;text-autospace:none"><span style="font-family:"Courier New";background:yellow;mso-highlight:yellow">
<span style="color:blue">break</span>;<o:p></o:p></span></p>
<p class="MsoNormal" style="margin-left:.5in;text-autospace:none"><span style="font-family:"Courier New";background:yellow;mso-highlight:yellow"> }<o:p></o:p></span></p>
<p class="MsoNormal" style="margin-left:.5in;text-autospace:none"><span style="font-family:"Courier New";background:yellow;mso-highlight:yellow"><o:p> </o:p></span></p>
<p class="MsoNormal" style="margin-left:.5in;text-autospace:none"><span style="font-family:"Courier New";background:yellow;mso-highlight:yellow"> NVMeStallExecution(pAE, MAX_STATE_STALL_us);<o:p></o:p></span></p>
<p class="MsoNormal" style="margin-left:.5in;text-autospace:none"><span style="font-family:"Courier New";background:yellow;mso-highlight:yellow"> }<o:p></o:p></span></p>
<p class="MsoNormal" style="margin-left:.5in;text-autospace:none"><span style="font-family:"Courier New";background:yellow;mso-highlight:yellow"><o:p> </o:p></span></p>
<p class="MsoNormal" style="margin-left:.5in;text-autospace:none"><span style="font-family:"Courier New";background:yellow;mso-highlight:yellow">
<span style="color:blue">if</span> (CSTS.RDY != 0) {<o:p></o:p></span></p>
<p class="MsoNormal" style="margin-left:.5in;text-autospace:none"><span style="font-family:"Courier New";background:yellow;mso-highlight:yellow">
<span style="color:green">/* If RDY bit won't clear we can't enable the adapter */<o:p></o:p></span></span></p>
<p class="MsoNormal" style="margin-left:.5in;text-autospace:none"><span style="font-family:"Courier New";background:yellow;mso-highlight:yellow">
<span style="color:blue">return</span> FALSE;<o:p></o:p></span></p>
<p class="MsoNormal" style="margin-left:.5in;text-autospace:none"><span style="font-family:"Courier New";background:yellow;mso-highlight:yellow"> }</span><span style="font-family:"Courier New""><o:p></o:p></span></p>
<p class="MsoNormal" style="margin-left:.5in;text-autospace:none"><span style="font-family:"Courier New"">
<o:p></o:p></span></p>
<p class="MsoNormal" style="margin-left:.5in;text-autospace:none"><span style="font-family:"Courier New""> CC.EN = 1;<o:p></o:p></span></p>
<p class="MsoNormal" style="margin-left:.5in;text-autospace:none"><span style="font-family:"Courier New""> CC.CSS = NVME_CC_NVM_CMD;<o:p></o:p></span></p>
<p class="MsoNormal" style="margin-left:.5in;text-autospace:none"><span style="font-family:"Courier New""> CC.MPS = (PAGE_SIZE >> NVME_MEM_PAGE_SIZE_SHIFT);<o:p></o:p></span></p>
<p class="MsoNormal" style="margin-left:.5in;text-autospace:none"><span style="font-family:"Courier New""> CC.AMS = NVME_CC_ROUND_ROBIN;<o:p></o:p></span></p>
<p class="MsoNormal" style="margin-left:.5in;text-autospace:none"><span style="font-family:"Courier New""> CC.SHN = NVME_CC_SHUTDOWN_NONE;<o:p></o:p></span></p>
<p class="MsoNormal" style="margin-left:.5in;text-autospace:none"><span style="font-family:"Courier New""> CC.IOSQES = NVME_CC_IOSQES;<o:p></o:p></span></p>
<p class="MsoNormal" style="margin-left:.5in;text-autospace:none"><span style="font-family:"Courier New""> CC.IOCQES = NVME_CC_IOCQES;<o:p></o:p></span></p>
<p class="MsoNormal" style="margin-left:.5in;text-autospace:none"><span style="font-family:"Courier New""><o:p> </o:p></span></p>
<p class="MsoNormal" style="margin-left:.5in;text-autospace:none"><span style="font-family:"Courier New""> StorPortWriteRegisterUlong(pAE,<o:p></o:p></span></p>
<p class="MsoNormal" style="margin-left:.5in;text-autospace:none"><span style="font-family:"Courier New""> (PULONG)(&pAE->pCtrlRegister->CC),<o:p></o:p></span></p>
<p class="MsoNormal" style="margin-left:.5in;text-autospace:none"><span style="font-family:"Courier New""> CC.AsUlong);<o:p></o:p></span></p>
<p class="MsoNormal" style="margin-left:.5in;text-autospace:none"><span style="font-family:"Courier New"">
<span style="color:blue;background:yellow;mso-highlight:yellow">return</span><span style="background:yellow;mso-highlight:yellow"> TRUE;</span><o:p></o:p></span></p>
<p class="MsoNormal" style="margin-left:.5in"><span style="font-family:"Courier New"">}
<span style="color:green">/* NVMeEnableAdapter */</span></span><b><span style="font-size:16.0pt"><o:p></o:p></span></b></p>
<p class="MsoNormal"><span style="color:#1F497D"><o:p> </o:p></span></p>
<p class="MsoNormal"><span style="color:#1F497D"><o:p> </o:p></span></p>
<p class="MsoNormal">Thanks,<o:p></o:p></p>
<p class="MsoNormal">Judy<o:p></o:p></p>
<p class="MsoNormal"><span style="color:#1F497D"><o:p> </o:p></span></p>
<div>
<div style="border:none;border-top:solid #B5C4DF 1.0pt;padding:3.0pt 0in 0in 0in">
<p class="MsoNormal"><b><span style="font-size:10.0pt;font-family:"Tahoma","sans-serif"">From:</span></b><span style="font-size:10.0pt;font-family:"Tahoma","sans-serif"">
<a href="mailto:nvmewin-bounces@lists.openfabrics.org">nvmewin-bounces@lists.openfabrics.org</a> [<a href="mailto:nvmewin-bounces@lists.openfabrics.org">mailto:nvmewin-bounces@lists.openfabrics.org</a>]
<b>On Behalf Of </b>Yong Chen<br>
<b>Sent:</b> Friday, November 22, 2013 1:48 PM<br>
<b>To:</b> Alex Chang; <a href="mailto:nvmewin@lists.openfabrics.org">nvmewin@lists.openfabrics.org</a><br>
<b>Subject:</b> Re: [nvmewin] code review: crash dump & hibernation support<o:p></o:p></span></p>
</div>
</div>
<p class="MsoNormal"><o:p> </o:p></p>
<p class="MsoNormal"><span style="color:#1F497D">Hi, everyone,<o:p></o:p></span></p>
<p class="MsoNormal"><span style="color:#1F497D"><o:p> </o:p></span></p>
<p class="MsoNormal"><span style="color:#1F497D">I hope many are busy testing the changes on your devices. If you have any feedback to share, I’d very appreciate it. Holiday is upon us and we’d like to wrap up this much delayed soon.<o:p></o:p></span></p>
<p class="MsoNormal"><span style="color:#1F497D"><o:p> </o:p></span></p>
<p class="MsoNormal"><span style="color:#1F497D">Thanks,<o:p></o:p></span></p>
<p class="MsoNormal"><span style="color:#1F497D"><o:p> </o:p></span></p>
<p class="MsoNormal"><span style="color:#1F497D">Yong<o:p></o:p></span></p>
<p class="MsoNormal"><span style="color:#1F497D"><o:p> </o:p></span></p>
<div>
<div style="border:none;border-top:solid #B5C4DF 1.0pt;padding:3.0pt 0in 0in 0in">
<p class="MsoNormal"><b><span style="font-size:10.0pt;font-family:"Tahoma","sans-serif"">From:</span></b><span style="font-size:10.0pt;font-family:"Tahoma","sans-serif""> Yong Chen
<br>
<b>Sent:</b> Wednesday, November 20, 2013 12:52 AM<br>
<b>To:</b> 'Alex Chang'; Uma Parepalli<br>
<b>Cc:</b> 'nvmewin@lists.openfabrics.org'<br>
<b>Subject:</b> RE: [nvmewin] code review: crash dump & hibernation support<o:p></o:p></span></p>
</div>
</div>
<p class="MsoNormal"><o:p> </o:p></p>
<p class="MsoNormal"><span style="color:#1F497D">Object #1: crash dump when blue screen or manual triggered, for all SKUs (server or client).<o:p></o:p></span></p>
<p class="MsoNormal"><span style="color:#1F497D">Object #2: hibernate and then resume on all client SKUs.<o:p></o:p></span></p>
<p class="MsoNormal"><span style="color:#1F497D">+ Minor cleaning up and fixes along the way.<o:p></o:p></span></p>
<p class="MsoNormal"><span style="color:#1F497D"><o:p> </o:p></span></p>
<p class="MsoNormal"><span style="color:#1F497D">High-level Summary:<o:p></o:p></span></p>
<p class="MsoNormal"><span style="color:#1F497D"><o:p> </o:p></span></p>
<p class="MsoNormal"><span style="color:#1F497D">The major change is to enable ntldrDump mode so that during crash dump or hibernation, the system memory can be dumped to pre-allocated block locations (MEMORY.DMP or HIBERFIL.SYS file). The same nvme.sys driver
will be reloaded as another image into strict dumbed-down environment where usual APIs are not available anymore.<o:p></o:p></span></p>
<p class="MsoNormal"><span style="color:#1F497D"><o:p> </o:p></span></p>
<p class="MsoNormal"><span style="color:#1F497D">The next challenge is to re-initialize the controller properly after having resumed from hibernation image and to continue serve as system boot disk.
<o:p></o:p></span></p>
<p class="MsoNormal"><span style="color:#1F497D"><o:p> </o:p></span></p>
<p class="MsoNormal"><span style="color:#1F497D">I need to give credits to earlier contributors (Intel, LSI, IDT and others) for having laid solid building blocks needed for the dump mode. This change solved the buffer issue and introduced a different code
path for IOs in dump mode.<o:p></o:p></span></p>
<p class="MsoNormal"><span style="color:#1F497D"><o:p> </o:p></span></p>
<p class="MsoNormal"><span style="color:#1F497D">Detailed Briefs:<o:p></o:p></span></p>
<p class="MsoListParagraph" style="text-indent:-.25in;mso-list:l1 level1 lfo2"><![if !supportLists]><span style="font-family:Symbol;color:#1F497D"><span style="mso-list:Ignore">·<span style="font:7.0pt "Times New Roman"">
</span></span></span><![endif]><span style="color:#1F497D">nvmeInit.c<br>
changes to manage buffers and IO queues in dump modes.<o:p></o:p></span></p>
<p class="MsoListParagraph" style="text-indent:-.25in;mso-list:l1 level1 lfo2"><![if !supportLists]><span style="font-family:Symbol;color:#1F497D"><span style="mso-list:Ignore">·<span style="font:7.0pt "Times New Roman"">
</span></span></span><![endif]><span style="color:#1F497D">nvmeIo.c<br>
minor tweak in dump mode where only boot-CPU is available<o:p></o:p></span></p>
<p class="MsoListParagraph" style="text-indent:-.25in;mso-list:l1 level1 lfo2"><![if !supportLists]><span style="font-family:Symbol;color:#1F497D"><span style="mso-list:Ignore">·<span style="font:7.0pt "Times New Roman"">
</span></span></span><![endif]><span style="color:#1F497D">nvmeSnti.c<br>
fix existing bug where FLUSH cmd should include NSID (all NSs in this case).<o:p></o:p></span></p>
<p class="MsoListParagraph" style="text-indent:-.25in;mso-list:l1 level1 lfo2"><![if !supportLists]><span style="font-family:Symbol;color:#1F497D"><span style="mso-list:Ignore">·<span style="font:7.0pt "Times New Roman"">
</span></span></span><![endif]><span style="color:#1F497D">nvmeStat.c<br>
helper function change due to some timer related API not available in dump mode.<o:p></o:p></span></p>
<p class="MsoListParagraph" style="text-indent:-.25in;mso-list:l1 level1 lfo2"><![if !supportLists]><span style="font-family:Symbol;color:#1F497D"><span style="mso-list:Ignore">·<span style="font:7.0pt "Times New Roman"">
</span></span></span><![endif]><span style="color:#1F497D">nvmeStd.c<br>
A: refactored code into NVMeWaitForCtrlRDY() for waiting after setting CC.EN = 1.<br>
B: introduced same waiting logic after clearing CC.EN = 0.<br>
C: during power up, Reset will issue DPC to call RecoveryDpcRoutine() to re-initialize the driver, similarly the above A+B steps are introduced.
<br>
<br>
Using trunk version code, the hardware I have always timed out on initialization. I have had this fix since this spring. I think it is the same issue listed in Kwok’s laundry list. But I would need Judy to verify whether the issue she found is fixed or not
by this change.<br>
(</span><span style="font-size:10.0pt;font-family:"Arial","sans-serif"">Not handling CSTS.RDY status (from 1->0 and 0->1) properly on NVMe reset.</span><span style="color:#1F497D">)
<o:p></o:p></span></p>
<p class="MsoNormal"><span style="color:#1F497D"><o:p> </o:p></span></p>
<div>
<div style="border:none;border-top:solid #B5C4DF 1.0pt;padding:3.0pt 0in 0in 0in">
<p class="MsoNormal"><b><span style="font-size:10.0pt;font-family:"Tahoma","sans-serif"">From:</span></b><span style="font-size:10.0pt;font-family:"Tahoma","sans-serif""> Alex Chang [<a href="mailto:Alex.Chang@pmcs.com">mailto:Alex.Chang@pmcs.com</a>]
<br>
<b>Sent:</b> Tuesday, November 19, 2013 5:04 PM<br>
<b>To:</b> Uma Parepalli; Yong Chen<br>
<b>Subject:</b> RE: [nvmewin] code review: crash dump & hibernation support<o:p></o:p></span></p>
</div>
</div>
<p class="MsoNormal"><o:p> </o:p></p>
<p class="MsoNormal"><span style="color:#1F497D">Hi Yong,<o:p></o:p></span></p>
<p class="MsoNormal"><span style="color:#1F497D"><o:p> </o:p></span></p>
<p class="MsoNormal"><span style="color:#1F497D">Could you please summarize the changes you made? Normally, we list the changes under each file as high-level briefs.<o:p></o:p></span></p>
<p class="MsoNormal"><span style="color:#1F497D"><o:p> </o:p></span></p>
<p class="MsoNormal"><span style="color:#1F497D">Thanks,<br>
Alex<o:p></o:p></span></p>
<p class="MsoNormal"><span style="color:#1F497D"><o:p> </o:p></span></p>
<div>
<div style="border:none;border-top:solid #B5C4DF 1.0pt;padding:3.0pt 0in 0in 0in">
<p class="MsoNormal"><b><span style="font-size:10.0pt;font-family:"Tahoma","sans-serif"">From:</span></b><span style="font-size:10.0pt;font-family:"Tahoma","sans-serif""> Uma Parepalli [<a href="mailto:uma.parepalli@skhms.com">mailto:uma.parepalli@skhms.com</a>]
<br>
<b>Sent:</b> Tuesday, November 19, 2013 4:05 PM<br>
<b>To:</b> Alex Chang<br>
<b>Subject:</b> RE: [nvmewin] code review: crash dump & hibernation support<o:p></o:p></span></p>
</div>
</div>
<p class="MsoNormal"><o:p> </o:p></p>
<p class="MsoNormal"><span style="color:#1F497D">Is there a change log file or something that explains what changes are made without looking at the code?<o:p></o:p></span></p>
<p class="MsoNormal"><span style="color:#1F497D">Uma<o:p></o:p></span></p>
<p class="MsoNormal"><span style="color:#1F497D"><o:p> </o:p></span></p>
<div>
<div style="border:none;border-top:solid #B5C4DF 1.0pt;padding:3.0pt 0in 0in 0in">
<p class="MsoNormal"><b><span style="font-size:10.0pt;font-family:"Tahoma","sans-serif"">From:</span></b><span style="font-size:10.0pt;font-family:"Tahoma","sans-serif"">
<a href="mailto:nvmewin-bounces@lists.openfabrics.org">nvmewin-bounces@lists.openfabrics.org</a> [<a href="mailto:nvmewin-bounces@lists.openfabrics.org">mailto:nvmewin-bounces@lists.openfabrics.org</a>]
<b>On Behalf Of </b>Alex Chang<br>
<b>Sent:</b> Tuesday, November 19, 2013 4:05 PM<br>
<b>To:</b> <a href="mailto:nvmewin@lists.openfabrics.org">nvmewin@lists.openfabrics.org</a><br>
<b>Subject:</b> Re: [nvmewin] code review: crash dump & hibernation support<o:p></o:p></span></p>
</div>
</div>
<p class="MsoNormal"><o:p> </o:p></p>
<p class="MsoNormal"><span style="color:#1F497D">Hi all,<o:p></o:p></span></p>
<p class="MsoNormal"><span style="color:#1F497D"><o:p> </o:p></span></p>
<p class="MsoNormal"><span style="color:#1F497D">Please find the attached code changes made by Yong Chen from Huawei. Please review the changes, test them accordingly and provide your feedbacks.<o:p></o:p></span></p>
<p class="MsoNormal"><span style="color:#1F497D"><o:p> </o:p></span></p>
<p class="MsoNormal"><span style="color:#1F497D">Thanks,<br>
Alex<o:p></o:p></span></p>
<p class="MsoNormal"><span style="color:#1F497D"><o:p> </o:p></span></p>
<div>
<div style="border:none;border-top:solid #B5C4DF 1.0pt;padding:3.0pt 0in 0in 0in">
<p class="MsoNormal"><b><span style="font-size:10.0pt;font-family:"Tahoma","sans-serif"">From:</span></b><span style="font-size:10.0pt;font-family:"Tahoma","sans-serif""> Yong Chen [<a href="mailto:Yong.sc.Chen@huawei.com">mailto:Yong.sc.Chen@huawei.com</a>]
<br>
<b>Sent:</b> Tuesday, November 19, 2013 3:59 PM<br>
<b>To:</b> <a href="mailto:nvmewin@lists.openfabrics.org">nvmewin@lists.openfabrics.org</a>; Alex Chang<br>
<b>Subject:</b> RE: code review: crash dump & hibernation support<o:p></o:p></span></p>
</div>
</div>
<p class="MsoNormal"><o:p> </o:p></p>
<p class="MsoNormal"><span style="font-size:8.0pt;color:#1F497D"><Resending to due to attachment issue><o:p></o:p></span></p>
<p class="MsoNormal"><span style="color:#1F497D">Hi, all,<o:p></o:p></span></p>
<p class="MsoNormal"><span style="color:#1F497D"><o:p> </o:p></span></p>
<p class="MsoNormal"><span style="color:#1F497D">Please download the source code from the link in the attached email (you need to have Silverlight installed).<o:p></o:p></span></p>
<p class="MsoNormal"><span style="color:#1F497D"><o:p> </o:p></span></p>
<p class="MsoNormal"><span style="color:#1F497D">Or to save the trouble for everyone,… Alex, Could you reply back with the code change you downloaded. Testlog is attached and see below for the list of tests.<o:p></o:p></span></p>
<p class="MsoNormal"><span style="color:#1F497D"><o:p> </o:p></span></p>
<p class="MsoNormal"><span style="color:#1F497D">Thanks,<o:p></o:p></span></p>
<p class="MsoNormal"><span style="color:#1F497D"><o:p> </o:p></span></p>
<p class="MsoNormal"><span style="color:#1F497D">Yong<o:p></o:p></span></p>
<p class="MsoNormal"><span style="color:#1F497D"><o:p> </o:p></span></p>
<p class="MsoNormal" style="margin-bottom:12.0pt"><b><span style="font-size:10.0pt;font-family:"Tahoma","sans-serif"">From:</span></b><span style="font-size:10.0pt;font-family:"Tahoma","sans-serif""> Yong Chen
<br>
<b>Sent:</b> Monday, November 18, 2013 4:13 PM<br>
<b>To:</b> 'nvmewin@lists.openfabrics.org'; 'Alex Chang'</span><o:p></o:p></p>
<p class="MsoNormal">Hi, Alex and all,<o:p></o:p></p>
<p class="MsoNormal"><o:p> </o:p></p>
<p class="MsoNormal">Here is the code change to support crash dump and hibernation. Please review. Hopefully we can wrap up by this week before the meeting.<o:p></o:p></p>
<p class="MsoNormal"><o:p> </o:p></p>
<p class="MsoNormal">Using the trunk version I had problem with the initialization as well. The trunk version would timeout on me. I think it is the same
<span style="font-size:10.0pt;font-family:"Arial","sans-serif"">CSTS.RDY</span> issue Judy raised up. I refactored a bit and fixed it at least for the hardware I have.
<o:p></o:p></p>
<p class="MsoNormal"><o:p> </o:p></p>
<p class="MsoNormal">Thanks,<o:p></o:p></p>
<p class="MsoNormal"><o:p> </o:p></p>
<p class="MsoNormal">Yong<o:p></o:p></p>
<p class="MsoNormal"><o:p> </o:p></p>
<p class="MsoNormal">Tests that I have gone thru:<o:p></o:p></p>
<p class="MsoListParagraph" style="text-indent:-.25in;mso-list:l0 level1 lfo4"><![if !supportLists]><span style="mso-list:Ignore">1.<span style="font:7.0pt "Times New Roman"">
</span></span><![endif]>manual crash dump.<br>
KD>.crash and then reboot and KD -z -v MEMORY.DMP, do “!process -1 f”<o:p></o:p></p>
<p class="MsoListParagraph" style="text-indent:-.25in;mso-list:l0 level1 lfo4"><![if !supportLists]><span style="mso-list:Ignore">2.<span style="font:7.0pt "Times New Roman"">
</span></span><![endif]>manual hibernation or pwrtest /sleep /c:10 /d:30 /p:30 /s:4<o:p></o:p></p>
<p class="MsoListParagraph" style="text-indent:-.25in;mso-list:l0 level1 lfo4"><![if !supportLists]><span style="mso-list:Ignore">3.<span style="font:7.0pt "Times New Roman"">
</span></span><![endif]>SCSICompliance (log attached).<o:p></o:p></p>
<p class="MsoListParagraph" style="text-indent:-.25in;mso-list:l0 level1 lfo4"><![if !supportLists]><span style="mso-list:Ignore">4.<span style="font:7.0pt "Times New Roman"">
</span></span><![endif]>stresses: iostress, sdstress (log attached)<o:p></o:p></p>
<p class="MsoListParagraph" style="text-indent:-.25in;mso-list:l0 level1 lfo4"><![if !supportLists]><span style="mso-list:Ignore">5.<span style="font:7.0pt "Times New Roman"">
</span></span><![endif]>hibernation has been tested on win8.0 as well, but not extensively.<o:p></o:p></p>
<p class="MsoListParagraph" style="text-indent:-.25in;mso-list:l0 level1 lfo4"><![if !supportLists]><span style="mso-list:Ignore">6.<span style="font:7.0pt "Times New Roman"">
</span></span><![endif]>Hibernation has also been tested with both bootable OptionROM or the newly released UEFI driver.<o:p></o:p></p>
<p class="MsoListParagraph" style="text-indent:-.25in;mso-list:l0 level1 lfo4"><![if !supportLists]><span style="mso-list:Ignore">7.<span style="font:7.0pt "Times New Roman"">
</span></span><![endif]>All tests were conducted on x64 platform, involving 3 different hardware, plus another Intel MB which can’t do hibernation (no S4).<o:p></o:p></p>
<p class="MsoNormal"><o:p> </o:p></p>
<div class="MsoNormal" align="center" style="text-align:center">
<hr size="2" width="100%" align="center">
</div>
<p class="MsoNormal"><span style="font-size:10.0pt;font-family:STXihei;color:black">Yong Chen<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:10.0pt;font-family:STXihei;color:black">Storage Architect<o:p></o:p></span></p>
<p class="MsoNormal"><span lang="ZH-CN" style="font-size:10.0pt;font-family:SimSun;color:black">华为技术有限公司</span><span style="font-size:10.0pt;font-family:STXihei;color:black"> Huawei Technologies Co., Ltd<o:p></o:p></span></p>
<p class="MsoNormal"><!--[if gte vml 1]><v:shapetype id="_x0000_t75" coordsize="21600,21600" o:spt="75" o:preferrelative="t" path="m@4@5l@4@11@9@11@9@5xe" filled="f" stroked="f">
<v:stroke joinstyle="miter" />
<v:formulas>
<v:f eqn="if lineDrawn pixelLineWidth 0" />
<v:f eqn="sum @0 1 0" />
<v:f eqn="sum 0 0 @1" />
<v:f eqn="prod @2 1 2" />
<v:f eqn="prod @3 21600 pixelWidth" />
<v:f eqn="prod @3 21600 pixelHeight" />
<v:f eqn="sum @0 0 1" />
<v:f eqn="prod @6 1 2" />
<v:f eqn="prod @7 21600 pixelWidth" />
<v:f eqn="sum @8 21600 0" />
<v:f eqn="prod @7 21600 pixelHeight" />
<v:f eqn="sum @10 21600 0" />
</v:formulas>
<v:path o:extrusionok="f" gradientshapeok="t" o:connecttype="rect" />
<o:lock v:ext="edit" aspectratio="t" />
</v:shapetype><v:shape id="ridImg" o:spid="_x0000_s1026" type="#_x0000_t75" alt="Company_logo" style='position:absolute;margin-left:0;margin-top:0;width:76.5pt;height:24pt;z-index:251657728;visibility:visible;mso-wrap-style:square;mso-wrap-distance-left:0;mso-wrap-distance-top:0;mso-wrap-distance-right:0;mso-wrap-distance-bottom:0;mso-position-horizontal:left;mso-position-horizontal-relative:text;mso-position-vertical:absolute;mso-position-vertical-relative:line' o:allowoverlap="f">
<v:imagedata src="cid:image001.jpg@01CEE9FA.210589F0" o:title="company_logo" />
<w:wrap type="square"/>
</v:shape><![endif]--><![if !vml]><img width="102" height="32" src="cid:image001.jpg@01CEE9FA.210589F0" align="left" alt="Company_logo" v:shapes="ridImg"><![endif]><span style="font-size:10.0pt;font-family:STXihei;color:black">Office:
408-330-5482<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:10.0pt;font-family:"Cambria","serif";color:black">Mobile: 425-922-0658</span><span style="font-size:10.0pt;font-family:STXihei;color:black"><o:p></o:p></span></p>
<p class="MsoNormal"><span style="color:black">Email: <a href="mailto:yong.sc.chen@huawei.com">
yong.sc.chen@huawei.com</a><br>
2330 Central Expressway<br>
Santa Clara, CA 95050<o:p></o:p></span></p>
<p class="MsoNormal"><span style="color:black"><a href="http://www.huawei.com">http://www.huawei.com</a><o:p></o:p></span></p>
<p class="MsoNormal"><span lang="ZH-CN" style="font-size:7.5pt;font-family:SimSun;color:gray">本邮件及其附件含有华为公司的保密信息，仅限于发送给上面地址中列出的个人或群组。禁</span><span style="font-size:7.5pt;font-family:STXihei;color:gray"><br>
</span><span lang="ZH-CN" style="font-size:7.5pt;font-family:SimSun;color:gray">止任何其他人以任何形式使用（包括但不限于全部或部分地泄露、复制、或散发）本邮件中</span><span style="font-size:7.5pt;font-family:STXihei;color:gray"><br>
</span><span lang="ZH-CN" style="font-size:7.5pt;font-family:SimSun;color:gray">的信息。如果您错收了本邮件，请您立即电话或邮件通知发件人并删除本邮件！</span><span style="font-size:7.5pt;font-family:STXihei;color:gray"><br>
</span><span style="font-size:7.5pt;font-family:"Arial","sans-serif";color:gray">This e-mail and its attachments contain confidential information from HUAWEI, which
<br>
is intended only for the person or entity whose address is listed above. Any use of the
<br>
information contained herein in any way (including, but not limited to, total or partial
<br>
disclosure, reproduction, or dissemination) by persons other than the intended <br>
recipient(s) is prohibited. If you receive this e-mail in error, please notify the sender by
<br>
phone or email immediately and delete it!</span><span style="color:black"><o:p></o:p></span></p>
<p class="MsoNormal"><o:p> </o:p></p>
</div>
</body>
</html>