<html xmlns:v="urn:schemas-microsoft-com:vml" xmlns:o="urn:schemas-microsoft-com:office:office" xmlns:w="urn:schemas-microsoft-com:office:word" xmlns:x="urn:schemas-microsoft-com:office:excel" xmlns:m="http://schemas.microsoft.com/office/2004/12/omml" xmlns="http://www.w3.org/TR/REC-html40">
<head>
<meta http-equiv="Content-Type" content="text/html; charset=utf-8">
<meta name="Generator" content="Microsoft Word 15 (filtered medium)">
<!--[if !mso]><style>v\:* {behavior:url(#default#VML);}
o\:* {behavior:url(#default#VML);}
w\:* {behavior:url(#default#VML);}
.shape {behavior:url(#default#VML);}
</style><![endif]--><style><!--
/* Font Definitions */
@font-face
{font-family:"Cambria Math";
panose-1:2 4 5 3 5 4 6 3 2 4;}
@font-face
{font-family:Calibri;
panose-1:2 15 5 2 2 2 4 3 2 4;}
@font-face
{font-family:Tahoma;
panose-1:2 11 6 4 3 5 4 4 2 4;}
@font-face
{font-family:"Intel Clear";
panose-1:2 11 6 4 2 2 3 2 2 4;}
/* Style Definitions */
p.MsoNormal, li.MsoNormal, div.MsoNormal
{margin:0in;
margin-bottom:.0001pt;
font-size:12.0pt;
font-family:"Times New Roman",serif;}
a:link, span.MsoHyperlink
{mso-style-priority:99;
color:blue;
text-decoration:underline;}
a:visited, span.MsoHyperlinkFollowed
{mso-style-priority:99;
color:purple;
text-decoration:underline;}
p
{mso-style-priority:99;
mso-margin-top-alt:auto;
margin-right:0in;
mso-margin-bottom-alt:auto;
margin-left:0in;
font-size:12.0pt;
font-family:"Times New Roman",serif;}
span.EmailStyle18
{mso-style-type:personal-reply;
font-family:"Calibri",sans-serif;
color:#002060;
font-weight:normal;
font-style:normal;}
.MsoChpDefault
{mso-style-type:export-only;
font-family:"Calibri",sans-serif;}
@page WordSection1
{size:8.5in 11.0in;
margin:1.0in 1.0in 1.0in 1.0in;}
div.WordSection1
{page:WordSection1;}
--></style><!--[if gte mso 9]><xml>
<o:shapedefaults v:ext="edit" spidmax="1026" />
</xml><![endif]--><!--[if gte mso 9]><xml>
<o:shapelayout v:ext="edit">
<o:idmap v:ext="edit" data="1" />
</o:shapelayout></xml><![endif]-->
</head>
<body lang="EN-US" link="blue" vlink="purple">
<div class="WordSection1">
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:"Calibri",sans-serif;color:#002060">Crash dump mode is single threaded and no interrupts, so synchronization between submission and completion side is not necessary. We poll for completions in the
crash dump mode.<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:"Calibri",sans-serif;color:#002060"><o:p> </o:p></span></p>
<p class="MsoNormal"><b><span style="font-size:11.0pt;font-family:"Calibri",sans-serif">From:</span></b><span style="font-size:11.0pt;font-family:"Calibri",sans-serif"> Iuliu Rus [mailto:iuliur@google.com]
<br>
<b>Sent:</b> Tuesday, June 14, 2016 12:13 PM<br>
<b>To:</b> Thomas Freeman <thomas.freeman@hgst.com><br>
<b>Cc:</b> Robles, Raymond C <raymond.c.robles@intel.com>; nvmewin <nvmewin@lists.openfabrics.org><br>
<b>Subject:</b> Re: [nvmewin] review comments for patch from Google<o:p></o:p></span></p>
<p class="MsoNormal"><o:p> </o:p></p>
<div>
<p class="MsoNormal">Actually looking at it some more, it looks like that's needed in the dump mode.<o:p></o:p></p>
</div>
<div>
<p class="MsoNormal"><o:p> </o:p></p>
<div>
<p class="MsoNormal">On Tue, Jun 14, 2016 at 12:01 PM, Thomas Freeman <<a href="mailto:thomas.freeman@hgst.com" target="_blank">thomas.freeman@hgst.com</a>> wrote:<o:p></o:p></p>
<blockquote style="border:none;border-left:solid #CCCCCC 1.0pt;padding:0in 0in 0in 6.0pt;margin-left:4.8pt;margin-right:0in">
<div>
<div>
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto"><span style="font-size:11.0pt;font-family:"Calibri",sans-serif;color:#1F497D">I agree.</span><o:p></o:p></p>
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto"><span style="font-size:11.0pt;font-family:"Calibri",sans-serif;color:#1F497D">Thanks,</span><o:p></o:p></p>
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto"><b><span style="font-size:10.0pt;font-family:"Arial",sans-serif;color:#1F497D">Tom Freeman</span></b><o:p></o:p></p>
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto"><b><span style="font-size:10.0pt;font-family:"Arial",sans-serif;color:#1F497D">Software Engineer, Device Manager and Driver Development</span></b><o:p></o:p></p>
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto"><span style="font-size:10.0pt;font-family:"Arial",sans-serif;color:#1F497D">Western Digital Corporation</span><o:p></o:p></p>
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto"><span style="font-size:10.0pt;font-family:"Arial",sans-serif;color:#1F497D">e.
<a href="mailto:Thomas.freeman@hgst.com" target="_blank">Thomas.freeman@hgst.com</a></span><o:p></o:p></p>
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto"><span style="font-size:10.0pt;font-family:"Arial",sans-serif;color:#1F497D">o.
<a href="tel:%2B1-507-322-2311" target="_blank">+1-507-322-2311</a></span><o:p></o:p></p>
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto"><span style="font-size:11.0pt;font-family:"Calibri",sans-serif;color:#1F497D"> </span><o:p></o:p></p>
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto"><span style="font-size:11.0pt;font-family:"Calibri",sans-serif;color:#1F497D"><img border="0" width="150" height="47" id="_x0000_i1025" src="cid:image001.jpg@01D1C63F.A6983FD0"></span><o:p></o:p></p>
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto"><span style="font-size:11.0pt;font-family:"Calibri",sans-serif;color:#1F497D"> </span><o:p></o:p></p>
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto"><b><span style="font-size:11.0pt;font-family:"Calibri",sans-serif">From:</span></b><span style="font-size:11.0pt;font-family:"Calibri",sans-serif"> Iuliu Rus [mailto:<a href="mailto:iuliur@google.com" target="_blank">iuliur@google.com</a>]
<br>
<b>Sent:</b> Tuesday, June 14, 2016 1:25 PM</span><o:p></o:p></p>
<div>
<div>
<p class="MsoNormal"><br>
<b>To:</b> Thomas Freeman <<a href="mailto:thomas.freeman@hgst.com" target="_blank">thomas.freeman@hgst.com</a>><br>
<b>Cc:</b> Robles, Raymond C <<a href="mailto:raymond.c.robles@intel.com" target="_blank">raymond.c.robles@intel.com</a>>; nvmewin <<a href="mailto:nvmewin@lists.openfabrics.org" target="_blank">nvmewin@lists.openfabrics.org</a>><br>
<b>Subject:</b> Re: [nvmewin] review comments for patch from Google<o:p></o:p></p>
</div>
</div>
<div>
<div>
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto"> <o:p></o:p></p>
<div>
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto">I don't think it's needed. The IOCompletionRoutine always taking the dpc lock should be enough. Unless i'm missing something... If you agree I'll add these to the patch.<o:p></o:p></p>
</div>
<div>
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto"> <o:p></o:p></p>
<div>
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto">On Tue, Jun 14, 2016 at 11:08 AM, Thomas Freeman <<a href="mailto:thomas.freeman@hgst.com" target="_blank">thomas.freeman@hgst.com</a>> wrote:<o:p></o:p></p>
<blockquote style="border:none;border-left:solid #CCCCCC 1.0pt;padding:0in 0in 0in 6.0pt;margin-left:4.8pt;margin-top:5.0pt;margin-right:0in;margin-bottom:5.0pt">
<div>
<div>
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto"><span style="font-size:11.0pt;font-family:"Calibri",sans-serif;color:#1F497D">I think that is a good option.</span><o:p></o:p></p>
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto"><span style="font-size:11.0pt;font-family:"Calibri",sans-serif;color:#1F497D"> </span><o:p></o:p></p>
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto"><span style="font-size:11.0pt;font-family:"Calibri",sans-serif;color:#1F497D">I wonder at this point, is there any need for the StartIo lock?</span><o:p></o:p></p>
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto"><span style="font-size:11.0pt;font-family:"Calibri",sans-serif;color:#1F497D">Along with RecoveryDpcRoutine, I see the Startio lock is acquired in ProcessIo (when Acquire lock flag
is set) and IoCompletionRoutine (when MultipleCoresToSingleQueueFlag is set). </span>
<o:p></o:p></p>
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto"><span style="font-size:11.0pt;font-family:"Calibri",sans-serif;color:#1F497D">In this case, multiple IoCompletionRoutine DPC’s will be forced to operate serially when IoCompletionRoutine
is acquiring the StartIo lock.</span><o:p></o:p></p>
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto"><span style="font-size:11.0pt;font-family:"Calibri",sans-serif;color:#1F497D"> </span><o:p></o:p></p>
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto"><b><span style="font-size:10.0pt;font-family:"Arial",sans-serif;color:#1F497D">Tom Freeman</span></b><o:p></o:p></p>
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto"><b><span style="font-size:10.0pt;font-family:"Arial",sans-serif;color:#1F497D">Software Engineer, Device Manager and Driver Development</span></b><o:p></o:p></p>
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto"><span style="font-size:10.0pt;font-family:"Arial",sans-serif;color:#1F497D">Western Digital Corporation</span><o:p></o:p></p>
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto"><span style="font-size:10.0pt;font-family:"Arial",sans-serif;color:#1F497D">e.
<a href="mailto:Thomas.freeman@hgst.com" target="_blank">Thomas.freeman@hgst.com</a></span><o:p></o:p></p>
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto"><span style="font-size:10.0pt;font-family:"Arial",sans-serif;color:#1F497D">o.
<a href="tel:%2B1-507-322-2311" target="_blank">+1-507-322-2311</a></span><o:p></o:p></p>
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto"><span style="font-size:11.0pt;font-family:"Calibri",sans-serif;color:#1F497D"> </span><o:p></o:p></p>
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto"><span style="font-size:11.0pt;font-family:"Calibri",sans-serif;color:#1F497D"><img border="0" width="150" height="47" id="_x0000_i1026" src="cid:image002.jpg@01D1C63F.A6983FD0"></span><o:p></o:p></p>
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto"><span style="font-size:11.0pt;font-family:"Calibri",sans-serif;color:#1F497D"> </span><o:p></o:p></p>
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto"><b><span style="font-size:11.0pt;font-family:"Calibri",sans-serif">From:</span></b><span style="font-size:11.0pt;font-family:"Calibri",sans-serif"> Iuliu Rus [mailto:<a href="mailto:iuliur@google.com" target="_blank">iuliur@google.com</a>]
<br>
<b>Sent:</b> Tuesday, June 14, 2016 12:12 PM<br>
<b>To:</b> Thomas Freeman <<a href="mailto:thomas.freeman@hgst.com" target="_blank">thomas.freeman@hgst.com</a>><br>
<b>Cc:</b> Robles, Raymond C <<a href="mailto:raymond.c.robles@intel.com" target="_blank">raymond.c.robles@intel.com</a>>; nvmewin <<a href="mailto:nvmewin@lists.openfabrics.org" target="_blank">nvmewin@lists.openfabrics.org</a>></span><o:p></o:p></p>
<div>
<div>
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto"><br>
<b>Subject:</b> Re: [nvmewin] review comments for patch from Google<o:p></o:p></p>
</div>
</div>
<div>
<div>
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto"> <o:p></o:p></p>
<div>
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto">I think we can fix the recovery dpc routine by instead taking every dpc lock in addition or instead the startio lock. Do you see any problems with that?<o:p></o:p></p>
</div>
<div>
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto"> <o:p></o:p></p>
<div>
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto">On Tue, Jun 14, 2016 at 7:41 AM, Thomas Freeman <<a href="mailto:thomas.freeman@hgst.com" target="_blank">thomas.freeman@hgst.com</a>> wrote:<o:p></o:p></p>
<blockquote style="border:none;border-left:solid #CCCCCC 1.0pt;padding:0in 0in 0in 6.0pt;margin-left:4.8pt;margin-top:5.0pt;margin-right:0in;margin-bottom:5.0pt">
<div>
<div>
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto"><span style="font-size:11.0pt;font-family:"Calibri",sans-serif;color:#1F497D">Hi Iuliu,
</span><o:p></o:p></p>
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto"><span style="font-size:11.0pt;font-family:"Calibri",sans-serif;color:#1F497D">While looking at this change, it caused me to review other code paths that utilize the StartIo lock.</span><o:p></o:p></p>
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto"><span style="font-size:11.0pt;font-family:"Calibri",sans-serif;color:#1F497D"> </span><o:p></o:p></p>
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto"><span style="font-size:11.0pt;font-family:"Calibri",sans-serif;color:#1F497D">That brought up 2 concerns</span><o:p></o:p></p>
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto"><span style="font-size:11.0pt;font-family:"Calibri",sans-serif;color:#1F497D">1. The routine recoveryDpcRoutine gets the Startio lock to ensure no new sub/cpl queue entries are
processed on any CPU. With the changes for concurrent channels, that synchronization no longer occurs.</span><o:p></o:p></p>
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto"><span style="font-size:11.0pt;font-family:"Calibri",sans-serif;color:#1F497D"> </span><o:p></o:p></p>
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto"><span style="font-size:11.0pt;font-family:"Calibri",sans-serif;color:#1F497D">2. The second concern relates to an issue that existed before your change - There is a path for the
driver to deadlock on attempting to get the StartIo lock. </span><o:p></o:p></p>
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto"><span style="font-size:11.0pt;font-family:"Calibri",sans-serif;color:#1F497D"> </span><o:p></o:p></p>
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto;margin-left:.5in">
<span style="font-size:11.0pt;font-family:"Calibri",sans-serif;color:#1F497D">An example of the deadlock occurs when a command is issued to delete an attached namespace AND MultipleCoresToSingleQueueFlag is set to TRUE.</span><o:p></o:p></p>
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto;margin-left:.5in">
<span style="font-size:11.0pt;font-family:"Calibri",sans-serif;color:#1F497D">In this example, when the namespace delete is issued, here are the steps leading up to the deadlock:</span><o:p></o:p></p>
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto;margin-left:.5in">
<span style="font-size:11.0pt;font-family:"Calibri",sans-serif;color:#1F497D">-Driver issues command Namespace Attachment (SEL = 1, controller detach)</span><o:p></o:p></p>
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto;margin-left:.5in">
<span style="font-size:11.0pt;font-family:"Calibri",sans-serif;color:#1F497D">-Command completion is processed by IoCompletionRoutine. In that method the driver acquires the StartIo lock (if MultipleCoresToSingleQueueFlag is set)</span><o:p></o:p></p>
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto;margin-left:.5in">
<span style="font-size:11.0pt;font-family:"Calibri",sans-serif;color:#1F497D">-IoCompletionRoutine calls the completion method - pNvmeCompletionRoutine. That eventually ends up in NVMeCompletionNSAttachment.</span><o:p></o:p></p>
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto;margin-left:.5in">
<span style="font-size:11.0pt;font-family:"Calibri",sans-serif;color:#1F497D">-NVMeCompletionNSAttachment proceeds to the next step, building the Namespace management command (SEL = 1, delete).</span><o:p></o:p></p>
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto;margin-left:.5in">
<span style="font-size:11.0pt;font-family:"Calibri",sans-serif;color:#1F497D">-NVMeCompletionNSAttachment calls ProcessIo (with AcquireLock = TRUE).
</span><o:p></o:p></p>
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto;margin-left:.5in">
<span style="font-size:11.0pt;font-family:"Calibri",sans-serif;color:#1F497D">-The first step in ProcessIo is to acquire the StartIo lock, which is already held by IoCompletionRoutine.
</span><o:p></o:p></p>
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto;margin-left:.5in">
<span style="font-size:11.0pt;font-family:"Calibri",sans-serif;color:#1F497D"> </span><o:p></o:p></p>
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto;margin-left:.5in">
<span style="font-size:11.0pt;font-family:"Calibri",sans-serif;color:#1F497D">I'm not sure what value is provided by having ProcessIo getting the StartIo lock. Removing that step in ProcessIo would certainly solve this deadlock. I don't see it causing other
problems. Maybe some others get express an opinion on this.</span><o:p></o:p></p>
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto"><span style="font-size:11.0pt;font-family:"Calibri",sans-serif;color:#1F497D"> </span><o:p></o:p></p>
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto"><span style="font-size:11.0pt;font-family:"Calibri",sans-serif;color:#1F497D">Regards,</span><o:p></o:p></p>
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto"><b><span style="font-size:10.0pt;font-family:"Arial",sans-serif;color:#1F497D">Tom Freeman</span></b><o:p></o:p></p>
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto"><b><span style="font-size:10.0pt;font-family:"Arial",sans-serif;color:#1F497D">Software Engineer, Device Manager and Driver Development</span></b><o:p></o:p></p>
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto"><span style="font-size:10.0pt;font-family:"Arial",sans-serif;color:#1F497D">Western Digital Corporation</span><o:p></o:p></p>
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto"><span style="font-size:10.0pt;font-family:"Arial",sans-serif;color:#1F497D">e.
<a href="mailto:Thomas.freeman@hgst.com" target="_blank">Thomas.freeman@hgst.com</a></span><o:p></o:p></p>
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto"><span style="font-size:10.0pt;font-family:"Arial",sans-serif;color:#1F497D">o.
<a href="tel:%2B1-507-322-2311" target="_blank">+1-507-322-2311</a></span><o:p></o:p></p>
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto"><span style="font-size:11.0pt;font-family:"Calibri",sans-serif;color:#1F497D"> </span><o:p></o:p></p>
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto"><span style="font-size:11.0pt;font-family:"Calibri",sans-serif;color:#1F497D"><img border="0" width="150" height="47" id="_x0000_i1027" src="cid:image003.jpg@01D1C63F.A6983FD0"></span><o:p></o:p></p>
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto"><span style="font-size:11.0pt;font-family:"Calibri",sans-serif;color:#1F497D"> </span><o:p></o:p></p>
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto"><b><span style="font-size:11.0pt;font-family:"Calibri",sans-serif">From:</span></b><span style="font-size:11.0pt;font-family:"Calibri",sans-serif"> nvmewin [mailto:<a href="mailto:nvmewin-bounces@lists.openfabrics.org" target="_blank">nvmewin-bounces@lists.openfabrics.org</a>]
<b>On Behalf Of </b>Iuliu Rus<br>
<b>Sent:</b> Friday, June 10, 2016 2:31 PM<br>
<b>To:</b> Robles, Raymond C <<a href="mailto:raymond.c.robles@intel.com" target="_blank">raymond.c.robles@intel.com</a>><br>
<b>Cc:</b> nvmewin <<a href="mailto:nvmewin@lists.openfabrics.org" target="_blank">nvmewin@lists.openfabrics.org</a>></span><o:p></o:p></p>
<div>
<div>
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto"><br>
<b>Subject:</b> Re: [nvmewin] review comments for patch from Google<o:p></o:p></p>
</div>
</div>
<div>
<div>
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto"> <o:p></o:p></p>
<div>
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto">Our tests are still running and i'll be in meetings for the rest of the day, so i'm sending you my patch now so you can evaluate and review.<o:p></o:p></p>
</div>
</div>
</div>
<div>
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto"> <o:p></o:p></p>
<div>
<div>
<div>
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto">On Fri, Jun 10, 2016 at 10:24 AM, Iuliu Rus <<a href="mailto:iuliur@google.com" target="_blank">iuliur@google.com</a>> wrote:<o:p></o:p></p>
</div>
</div>
<blockquote style="border:none;border-left:solid #CCCCCC 1.0pt;padding:0in 0in 0in 6.0pt;margin-left:4.8pt;margin-top:5.0pt;margin-right:0in;margin-bottom:5.0pt">
<div>
<div>
<div>
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto">I'm just running through the tests and will send it to you as soon as they pass.<o:p></o:p></p>
</div>
</div>
</div>
<div>
<div>
<div>
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto"> <o:p></o:p></p>
<div>
<div>
<div>
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto">On Fri, Jun 10, 2016 at 10:22 AM, Robles, Raymond C <<a href="mailto:raymond.c.robles@intel.com" target="_blank">raymond.c.robles@intel.com</a>> wrote:<o:p></o:p></p>
</div>
</div>
<blockquote style="border:none;border-left:solid #CCCCCC 1.0pt;padding:0in 0in 0in 6.0pt;margin-left:4.8pt;margin-top:5.0pt;margin-right:0in;margin-bottom:5.0pt">
<div>
<div>
<div>
<div>
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto"><span style="font-size:11.0pt;font-family:"Calibri",sans-serif;color:#002060">Hi Iuliu,</span><o:p></o:p></p>
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto"><span style="font-size:11.0pt;font-family:"Calibri",sans-serif;color:#002060"> </span><o:p></o:p></p>
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto"><span style="font-size:11.0pt;font-family:"Calibri",sans-serif;color:#002060">If you send out the updated patch with the protection in place for both list entries and doorbells,
I can run it on an Intel NVMe SSD (P3700) and see if there is any performance impact (up or down).</span><o:p></o:p></p>
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto"><span style="font-size:11.0pt;font-family:"Calibri",sans-serif;color:#002060"> </span><o:p></o:p></p>
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto"><span style="font-size:11.0pt;font-family:"Calibri",sans-serif;color:#002060">Thank you…</span><o:p></o:p></p>
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto"><span style="font-size:11.0pt;font-family:"Calibri",sans-serif;color:#002060">Ray</span><o:p></o:p></p>
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto"><span style="font-size:11.0pt;font-family:"Calibri",sans-serif;color:#002060"> </span><o:p></o:p></p>
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto"><b><span style="font-size:11.0pt;font-family:"Calibri",sans-serif">From:</span></b><span style="font-size:11.0pt;font-family:"Calibri",sans-serif"> Iuliu Rus [mailto:<a href="mailto:iuliur@google.com" target="_blank">iuliur@google.com</a>]
<br>
<b>Sent:</b> Friday, June 10, 2016 10:17 AM<br>
<b>To:</b> <a href="mailto:suman.p@samsung.com" target="_blank">suman.p@samsung.com</a><br>
<b>Cc:</b> Robles, Raymond C <<a href="mailto:raymond.c.robles@intel.com" target="_blank">raymond.c.robles@intel.com</a>>; Judy Brock <<a href="mailto:judy.brock@samsung.com" target="_blank">judy.brock@samsung.com</a>>; nvmewin <<a href="mailto:nvmewin@lists.openfabrics.org" target="_blank">nvmewin@lists.openfabrics.org</a>><br>
<b>Subject:</b> Re: RE: review comments for patch from Google</span><o:p></o:p></p>
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto"> <o:p></o:p></p>
<div>
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto">we are running this in a virtual machine, and we do see a perf difference with concurrent channels. We did not observe any performance drop with synchronized access to the list
entry.<o:p></o:p></p>
</div>
</div>
</div>
<div>
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto"> <o:p></o:p></p>
<div>
<div>
<div>
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto">On Fri, Jun 10, 2016 at 7:15 AM, SUMAN PRAKASH B <<a href="mailto:suman.p@samsung.com" target="_blank">suman.p@samsung.com</a>> wrote:<o:p></o:p></p>
</div>
</div>
<blockquote style="border:none;border-left:solid #CCCCCC 1.0pt;padding:0in 0in 0in 6.0pt;margin-left:4.8pt;margin-top:5.0pt;margin-right:0in;margin-bottom:5.0pt">
<div>
<div>
<div>
<p><span style="font-size:11.0pt;font-family:"Calibri",sans-serif">Hi Iuliu,</span><o:p></o:p></p>
<p> <o:p></o:p></p>
<p><span style="font-size:11.0pt;font-family:"Calibri",sans-serif">As you have the implementation in place, can you let us know if there is any performance improvement when STOR_PERF_CONCURRENT_CHANNELS is set in case of #core and #queues not matching.</span><o:p></o:p></p>
<p><span style="font-size:11.0pt;font-family:"Calibri",sans-serif">Also, if there is any drop in performance if the list entry and doorbell is protected after setting STOR_PERF_CONCURRENT_CHANNELS.</span><o:p></o:p></p>
<p> <o:p></o:p></p>
<p><span style="font-size:11.0pt;font-family:"Calibri",sans-serif">Perhaps a bit off topic:</span><o:p></o:p></p>
<p><span style="font-size:11.0pt;font-family:"Calibri",sans-serif">Even in case when #cores and #queues are same and when STOR_PERF_CONCURRENT_CHANNELS is set, we did not observe any performance gain. Maybe we are hitting the max performance from device even
without setting STOR_PERF_CONCURRENT_CHANNELS. Can you please let us know your observation?</span><o:p></o:p></p>
<p> <o:p></o:p></p>
<p><span style="font-size:11.0pt;font-family:"Calibri",sans-serif">Thanks,</span><o:p></o:p></p>
<p><span style="font-size:11.0pt;font-family:"Calibri",sans-serif">Suman</span><o:p></o:p></p>
<p> <o:p></o:p></p>
<p>------- <b>Original Message</b> -------<o:p></o:p></p>
<p><b>Sender</b> : Robles, Raymond C<<a href="mailto:raymond.c.robles@intel.com" target="_blank">raymond.c.robles@intel.com</a>><o:p></o:p></p>
<p><b>Date</b> : Jun 10, 2016 08:30 (GMT+05:30)<o:p></o:p></p>
<p><b>Title</b> : RE: review comments for patch from Google<o:p></o:p></p>
</div>
</div>
<div>
<div>
<p> <o:p></o:p></p>
<div>
<div>
<div>
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto"><span style="font-size:11.0pt;font-family:"Calibri",sans-serif;color:#002060">Hi Judy,
</span><o:p></o:p></p>
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto"><span style="font-size:11.0pt;font-family:"Calibri",sans-serif;color:#002060"> </span><o:p></o:p></p>
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto"><span style="font-size:11.0pt;font-family:"Calibri",sans-serif;color:#002060">Thanks for the additional info. I agree with the assessment. My statement was directed at the general
direction of the solution… and the fact that I think option 1 is not as robust as option 2.</span><o:p></o:p></p>
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto"><span style="font-size:11.0pt;font-family:"Calibri",sans-serif;color:#002060"> </span><o:p></o:p></p>
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto"><span style="font-size:11.0pt;font-family:"Calibri",sans-serif;color:#002060">Thank you…</span><o:p></o:p></p>
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto"><span style="font-size:11.0pt;font-family:"Calibri",sans-serif;color:#002060">Ray</span><o:p></o:p></p>
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto"><a name="m_-702671920486349052_m_-605737012396966"><span style="font-size:11.0pt;font-family:"Calibri",sans-serif;color:#002060"> </span></a><o:p></o:p></p>
<div>
<div style="border:none;border-top:solid #E1E1E1 1.0pt;padding:3.0pt 0in 0in 0in">
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto"><b><span style="font-size:11.0pt;font-family:"Calibri",sans-serif">From:</span></b><span style="font-size:11.0pt;font-family:"Calibri",sans-serif"> Judy Brock [mailto:<a href="mailto:judy.brock@samsung.com" target="_blank">judy.brock@samsung.com</a>]
<br>
<b>Sent:</b> Thursday, June 09, 2016 5:14 PM<br>
<b>To:</b> Robles, Raymond C <<a href="mailto:raymond.c.robles@intel.com" target="_blank">raymond.c.robles@intel.com</a>>; Iuliu Rus <<a href="mailto:iuliur@google.com" target="_blank">iuliur@google.com</a>>;
<a href="mailto:suman.p@samsung.com" target="_blank">suman.p@samsung.com</a><br>
<b>Cc:</b> <a href="mailto:nvmewin@lists.openfabrics.org" target="_blank">nvmewin@lists.openfabrics.org</a><br>
<b>Subject:</b> RE: review comments for patch from Google</span><o:p></o:p></p>
</div>
</div>
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto"> <o:p></o:p></p>
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto"><span style="font-size:11.0pt;font-family:"Calibri",sans-serif;color:#1F497D">Hi Ray et al,</span><o:p></o:p></p>
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto;margin-left:.5in;text-indent:.5in">
<span style="font-size:11.0pt;font-family:"Calibri",sans-serif">[Suman wrote] >> Protect the listentry and doorbell before access/update</span><o:p></o:p></p>
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto;margin-left:.5in;text-indent:.5in">
<span style="font-size:11.0pt;font-family:"Calibri",sans-serif"> >>Do not set STOR_PERF_CONCURRENT_CHANNELS when number of cores and queues are not same.</span><o:p></o:p></p>
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto;margin-left:.5in;text-indent:.5in">
<span style="font-size:11.0pt;font-family:"Calibri",sans-serif;color:#002060"> </span><o:p></o:p></p>
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto;margin-left:.5in;text-indent:.5in">
<span style="font-size:11.0pt;font-family:"Calibri",sans-serif;color:#002060">Ray wrote] >>In the case Suman points out, concurrent channels should still be enabled even if the number of cores and queues do not match. This is especially true for large core
count systems (>128 logical cpu cores) but devices may only support 32 queues or less</span><o:p></o:p></p>
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto"><span style="font-size:11.0pt;font-family:"Calibri",sans-serif;color:#1F497D"> </span><o:p></o:p></p>
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto"><span style="font-size:11.0pt;font-family:"Calibri",sans-serif;color:#1F497D">I think Suman was saying, we have two options:
</span><o:p></o:p></p>
<p><span style="font-size:11.0pt;font-family:"Calibri",sans-serif;color:#1F497D">1.</span><span style="font-size:7.0pt;color:#1F497D">
</span><span style="font-size:11.0pt;font-family:"Calibri",sans-serif;color:#1F497D">either don’t enable it at all if # cores != #queues or
</span><o:p></o:p></p>
<p><span style="font-size:11.0pt;font-family:"Calibri",sans-serif;color:#1F497D">2.</span><span style="font-size:7.0pt;color:#1F497D">
</span><span style="font-size:11.0pt;font-family:"Calibri",sans-serif;color:#1F497D">protect both the list entries and doorbell writes before any accesses/updates if #cores != #queues.
</span><o:p></o:p></p>
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto"><span style="font-size:11.0pt;font-family:"Calibri",sans-serif;color:#1F497D"> </span><o:p></o:p></p>
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto"><span style="font-size:11.0pt;font-family:"Calibri",sans-serif;color:#1F497D">So we will need to check to see if we need to use locks or not in the performance path based on whether
#cores and # queues DO match or not if we want to enable concurrent channels no matter what. It’s probably better than option 1) above if this is a configuration that can come up frequently.</span><o:p></o:p></p>
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto"><span style="font-size:11.0pt;font-family:"Calibri",sans-serif;color:#1F497D"> </span><o:p></o:p></p>
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto"><span style="font-size:11.0pt;font-family:"Calibri",sans-serif;color:#1F497D">Thanks,</span><o:p></o:p></p>
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto"><span style="font-size:11.0pt;font-family:"Calibri",sans-serif;color:#1F497D">Judy</span><o:p></o:p></p>
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto"><span style="font-size:11.0pt;font-family:"Calibri",sans-serif;color:#1F497D"> </span><o:p></o:p></p>
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto"><span style="font-size:11.0pt;font-family:"Calibri",sans-serif;color:#1F497D"> </span><o:p></o:p></p>
<div>
<div style="border:none;border-top:solid #B5C4DF 1.0pt;padding:3.0pt 0in 0in 0in">
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto"><b><span style="font-size:10.0pt;font-family:"Tahoma",sans-serif">From:</span></b><span style="font-size:10.0pt;font-family:"Tahoma",sans-serif"> nvmewin [<a href="mailto:nvmewin-bounces@lists.openfabrics.org" target="_blank">mailto:nvmewin-bounces@lists.openfabrics.org</a>]
<b>On Behalf Of </b>Robles, Raymond C<br>
<b>Sent:</b> Thursday, June 09, 2016 2:32 PM<br>
<b>To:</b> Iuliu Rus; <a href="mailto:suman.p@samsung.com" target="_blank">suman.p@samsung.com</a><br>
<b>Cc:</b> <a href="mailto:nvmewin@lists.openfabrics.org" target="_blank">nvmewin@lists.openfabrics.org</a><br>
<b>Subject:</b> Re: [nvmewin] review comments for patch from Google</span><o:p></o:p></p>
</div>
</div>
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto"> <o:p></o:p></p>
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto"><span style="font-size:11.0pt;font-family:"Calibri",sans-serif;color:#002060">Hi Iuliu/Suman,</span><o:p></o:p></p>
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto"><span style="font-size:11.0pt;font-family:"Calibri",sans-serif;color:#002060"> </span><o:p></o:p></p>
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto"><span style="font-size:11.0pt;font-family:"Calibri",sans-serif;color:#002060">I’m of the opinion that any solution provided should be as robust as possible. In the case Suman points
out, concurrent channels should still be enabled even if the number of cores and queues do not match. This is especially true for large core count systems (>128 logical cpu cores) but devices may only support 32 queues or less.
</span><o:p></o:p></p>
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto"><span style="font-size:11.0pt;font-family:"Calibri",sans-serif;color:#002060"> </span><o:p></o:p></p>
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto"><span style="font-size:11.0pt;font-family:"Calibri",sans-serif;color:#002060">Thank you…</span><o:p></o:p></p>
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto"><span style="font-size:11.0pt;font-family:"Calibri",sans-serif;color:#002060">Ray</span><o:p></o:p></p>
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto"><span style="font-size:11.0pt;font-family:"Calibri",sans-serif;color:#002060"> </span><o:p></o:p></p>
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto"><b><span style="font-size:11.0pt;font-family:"Calibri",sans-serif">From:</span></b><span style="font-size:11.0pt;font-family:"Calibri",sans-serif"> Iuliu Rus
<a href="mailto:[mailto:iuliur@google.com]" target="_blank">[mailto:iuliur@google.com]</a>
<br>
<b>Sent:</b> Thursday, June 09, 2016 2:09 PM<br>
<b>To:</b> <a href="mailto:suman.p@samsung.com" target="_blank">suman.p@samsung.com</a><br>
<b>Cc:</b> Robles, Raymond C <<a href="mailto:raymond.c.robles@intel.com" target="_blank">raymond.c.robles@intel.com</a>>;
<a href="mailto:nvmewin@lists.openfabrics.org" target="_blank">nvmewin@lists.openfabrics.org</a><br>
<b>Subject:</b> Re: review comments for patch from Google</span><o:p></o:p></p>
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto"> <o:p></o:p></p>
<div>
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto">Which one do you prefer? Our google driver has the synchronization in place already for a different reason. Or, I can follow your second suggestion.<o:p></o:p></p>
</div>
</div>
</div>
<div>
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto"> <o:p></o:p></p>
<div>
<div>
<div>
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto">On Thu, Jun 9, 2016 at 7:39 AM, SUMAN PRAKASH B <<a href="mailto:suman.p@samsung.com" target="_blank">suman.p@samsung.com</a>> wrote:<o:p></o:p></p>
</div>
</div>
<blockquote style="border:none;border-left:solid #CCCCCC 1.0pt;padding:0in 0in 0in 6.0pt;margin-left:4.8pt;margin-top:5.0pt;margin-right:0in;margin-bottom:5.0pt">
<div>
<div>
<div>
<p><span style="font-size:11.0pt;font-family:"Calibri",sans-serif">Hi Iuliu,</span><o:p></o:p></p>
<p> <o:p></o:p></p>
<p><span style="font-size:11.0pt;font-family:"Calibri",sans-serif">Please find my review comments below:</span><o:p></o:p></p>
<p> <o:p></o:p></p>
<p><span style="font-size:11.0pt;font-family:"Calibri",sans-serif">1. If the number of Cores and Queues supported by device are not same, then setting STOR_PERF_CONCURRENT_CHANNELS will corrupt the submission queue. For example, if device has 4 queues and connected
to target PC which has 8 logical processors, and if STOR_PERF_CONCURRENT_CHANNELS is set, all the 8 logical processors will get startIo calls concurrently. As per OFA driver NUMA design, the IO Queue 1 will be shared by core 0 and core 4. There is possibility
that core 0 and core 4 will get startio() calls concurrently and will access/update the listenty and queue 1 doorbell register at same time, which will lead to listentry and doorbell register corruption.
<br>
This problem can be address by following ways:<br>
a. Protect the listentry and doorbell before access/update.<br>
b. Do not set STOR_PERF_CONCURRENT_CHANNELS when number of cores and queues are not same.</span><o:p></o:p></p>
<p> <o:p></o:p></p>
<p><span style="font-size:11.0pt;font-family:"Calibri",sans-serif">2. We can use pRMT->NumActiveCores which is updated in NVMeEnumNumaCores() instead of using KeQueryActiveProcessorCountEx/KeQueryActiveProcessorCount.</span><o:p></o:p></p>
<p> <o:p></o:p></p>
<p><span style="font-size:11.0pt;font-family:"Calibri",sans-serif">Please let us know your opinion.</span><o:p></o:p></p>
<p> <o:p></o:p></p>
<p><span style="font-size:11.0pt;font-family:"Calibri",sans-serif">Thanks,<br>
Suman</span><o:p></o:p></p>
<p> <o:p></o:p></p>
<p>------- <b>Original Message</b> -------<o:p></o:p></p>
<p><b>Sender</b> : Robles, Raymond C<<a href="mailto:raymond.c.robles@intel.com" target="_blank">raymond.c.robles@intel.com</a>><o:p></o:p></p>
<p><b>Date</b> : Jun 09, 2016 03:56 (GMT+05:30)<o:p></o:p></p>
<p><b>Title</b> : Re: [nvmewin] Happy New Year... and status update<o:p></o:p></p>
<p> <o:p></o:p></p>
<div>
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto"><span style="font-size:11.0pt;font-family:"Calibri",sans-serif;color:#002060">Thank you Iuliu!</span><o:p></o:p></p>
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto"><span style="font-size:11.0pt;font-family:"Calibri",sans-serif;color:#002060"> </span><o:p></o:p></p>
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto"><span style="font-size:11.0pt;font-family:"Calibri",sans-serif;color:#002060;background:yellow">Reviewing companies, please try to target the review completion for 2 weeks from
today (6/22/16).</span><span style="font-size:11.0pt;font-family:"Calibri",sans-serif;color:#002060">
</span><o:p></o:p></p>
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto"><span style="font-size:11.0pt;font-family:"Calibri",sans-serif;color:#002060"> </span><o:p></o:p></p>
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto"><span style="font-size:11.0pt;font-family:"Calibri",sans-serif;color:#002060">Thank you!</span><o:p></o:p></p>
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto"><span style="font-size:11.0pt;font-family:"Calibri",sans-serif;color:#002060">Ray</span><o:p></o:p></p>
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto"><span style="font-size:11.0pt;font-family:"Calibri",sans-serif;color:#002060"> </span><o:p></o:p></p>
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto"><b><span style="font-size:11.0pt;font-family:"Calibri",sans-serif">From:</span></b><span style="font-size:11.0pt;font-family:"Calibri",sans-serif"> Iuliu Rus [mailto:<a href="mailto:iuliur@google.com" target="_blank">iuliur@google.com</a>]
<br>
<b>Sent:</b> Wednesday, June 08, 2016 12:26 PM<br>
<b>To:</b> Robles, Raymond C <<a href="mailto:raymond.c.robles@intel.com" target="_blank">raymond.c.robles@intel.com</a>><br>
<b>Cc:</b> <a href="mailto:nvmewin@lists.openfabrics.org" target="_blank">nvmewin@lists.openfabrics.org</a><br>
<b>Subject:</b> Re: Happy New Year... and status update</span><o:p></o:p></p>
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto"> <o:p></o:p></p>
<div>
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto">I attached our patch. <o:p></o:p></p>
</div>
<div>
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto"> <o:p></o:p></p>
<div>
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto">On Tue, Jun 7, 2016 at 2:37 PM, Iuliu Rus <<a href="mailto:iuliur@google.com" target="_blank">iuliur@google.com</a>> wrote:<o:p></o:p></p>
<blockquote style="border:none;border-left:solid #CCCCCC 1.0pt;padding:0in 0in 0in 6.0pt;margin-left:4.8pt;margin-top:5.0pt;margin-right:0in;margin-bottom:5.0pt">
<div>
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto">Let me see if i can get it done by tomorrow eod...<o:p></o:p></p>
</div>
<div>
<div>
<div>
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto"> <o:p></o:p></p>
<div>
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto">On Tue, Jun 7, 2016 at 2:31 PM, Robles, Raymond C <<a href="mailto:raymond.c.robles@intel.com" target="_blank">raymond.c.robles@intel.com</a>> wrote:<o:p></o:p></p>
<blockquote style="border:none;border-left:solid #CCCCCC 1.0pt;padding:0in 0in 0in 6.0pt;margin-left:4.8pt;margin-top:5.0pt;margin-right:0in;margin-bottom:5.0pt">
<div>
<div>
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto"><span style="font-size:11.0pt;font-family:"Calibri",sans-serif;color:#002060">Hi Iuliu,</span><o:p></o:p></p>
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto"><span style="font-size:11.0pt;font-family:"Calibri",sans-serif;color:#002060"> </span><o:p></o:p></p>
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto"><span style="font-size:11.0pt;font-family:"Calibri",sans-serif;color:#002060">That’s great… thank you. Yes, the process is still the same (using SVN patches).</span><o:p></o:p></p>
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto"><span style="font-size:11.0pt;font-family:"Calibri",sans-serif;color:#002060"> </span><o:p></o:p></p>
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto"><span style="font-size:11.0pt;font-family:"Calibri",sans-serif;color:#002060">Please let me know if you have any questions/comments. My only question for you is an ETA for the patch
to be submitted.</span><o:p></o:p></p>
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto"><span style="font-size:11.0pt;font-family:"Calibri",sans-serif;color:#002060"> </span><o:p></o:p></p>
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto"><span style="font-size:11.0pt;font-family:"Calibri",sans-serif;color:#002060">Thank you…</span><o:p></o:p></p>
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto"><span style="font-size:11.0pt;font-family:"Calibri",sans-serif;color:#002060">Ray</span><o:p></o:p></p>
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto"><span style="font-size:11.0pt;font-family:"Calibri",sans-serif;color:#002060"> </span><o:p></o:p></p>
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto"><b><span style="font-size:11.0pt;font-family:"Calibri",sans-serif">From:</span></b><span style="font-size:11.0pt;font-family:"Calibri",sans-serif"> Iuliu Rus [mailto:<a href="mailto:iuliur@google.com" target="_blank">iuliur@google.com</a>]
<br>
<b>Sent:</b> Tuesday, June 07, 2016 2:28 PM<br>
<b>To:</b> Robles, Raymond C <<a href="mailto:raymond.c.robles@intel.com" target="_blank">raymond.c.robles@intel.com</a>><br>
<b>Cc:</b> <a href="mailto:nvmewin@lists.openfabrics.org" target="_blank">nvmewin@lists.openfabrics.org</a><br>
<b>Subject:</b> Re: Happy New Year... and status update</span><o:p></o:p></p>
<div>
<div>
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto"> <o:p></o:p></p>
<div>
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto">Yes we can send that to you. Is the process still the same?<o:p></o:p></p>
</div>
<div>
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto"> <o:p></o:p></p>
<div>
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto">On Tue, Jun 7, 2016 at 1:25 PM, Robles, Raymond C <<a href="mailto:raymond.c.robles@intel.com" target="_blank">raymond.c.robles@intel.com</a>> wrote:<o:p></o:p></p>
<blockquote style="border:none;border-left:solid #CCCCCC 1.0pt;padding:0in 0in 0in 6.0pt;margin-left:4.8pt;margin-top:5.0pt;margin-right:0in;margin-bottom:5.0pt">
<div>
<div>
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto"><span style="color:#002060">Hello Iuliu,</span><o:p></o:p></p>
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto"><span style="color:#002060"> </span><o:p></o:p></p>
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto"><span style="color:#002060">As the OFA community is closing on its mid-year release, we have only one more patch that was targeted… concurrent channels. Will Google still be submitting
the concurrent channels patch?</span><o:p></o:p></p>
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto"><span style="color:#002060"> </span><o:p></o:p></p>
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto"><span style="color:#002060">Thank you…</span><o:p></o:p></p>
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto"><span style="color:#002060">Ray</span><o:p></o:p></p>
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto"><span style="color:#002060"> </span><o:p></o:p></p>
<div>
<div style="border:none;border-top:solid #E1E1E1 1.0pt;padding:3.0pt 0in 0in 0in">
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto"><b>From:</b>
<a href="mailto:nvmewin-bounces@lists.openfabrics.org" target="_blank">nvmewin-bounces@lists.openfabrics.org</a> [mailto:<a href="mailto:nvmewin-bounces@lists.openfabrics.org" target="_blank">nvmewin-bounces@lists.openfabrics.org</a>]
<b>On Behalf Of </b>Robles, Raymond C<br>
<b>Sent:</b> Thursday, January 14, 2016 4:55 PM<br>
<b>To:</b> '<a href="mailto:nvmewin@lists.openfabrics.org" target="_blank">nvmewin@lists.openfabrics.org</a>' <<a href="mailto:nvmewin@lists.openfabrics.org" target="_blank">nvmewin@lists.openfabrics.org</a>><br>
<b>Subject:</b> [nvmewin] Happy New Year... and status update<o:p></o:p></p>
</div>
</div>
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto"> <o:p></o:p></p>
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto"><span style="font-size:10.0pt;font-family:"Intel Clear",sans-serif;color:#44546A">Hello and Happy New Year,</span><o:p></o:p></p>
<div>
<div>
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto"><span style="font-size:10.0pt;font-family:"Intel Clear",sans-serif;color:#44546A"> </span><o:p></o:p></p>
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto"><span style="font-size:10.0pt;font-family:"Intel Clear",sans-serif;color:#44546A">I hope everyone had a great holiday season and is off to a great start to the new year!</span><o:p></o:p></p>
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto"><span style="font-size:10.0pt;font-family:"Intel Clear",sans-serif;color:#44546A"> </span><o:p></o:p></p>
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto"><span style="font-size:10.0pt;font-family:"Intel Clear",sans-serif;color:#44546A">As communicated last month, the patch from HGST for the SCSI multi-initiator changes has been approved
and pushed. The holidays slowed down much of the progress on the OFA driver and there were several patches that did not get pushed prior to the end of the year. The list of patches remained to be pushed are as follows…</span><o:p></o:p></p>
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto"><span style="font-size:10.0pt;font-family:"Intel Clear",sans-serif;color:#44546A"> </span><o:p></o:p></p>
<p><span style="font-size:10.0pt;font-family:Symbol;color:#44546A">·</span><span style="font-size:7.0pt;color:#44546A">
</span><span style="font-size:10.0pt;font-family:"Intel Clear",sans-serif;color:#44546A">Namespace Management (Intel)</span><o:p></o:p></p>
<p><span style="font-size:10.0pt;font-family:Symbol;color:#44546A">·</span><span style="font-size:7.0pt;color:#44546A">
</span><span style="font-size:10.0pt;font-family:"Intel Clear",sans-serif;color:#44546A">Perf Opts (Samsung)
</span><o:p></o:p></p>
<p><span style="font-size:10.0pt;font-family:Symbol;color:#44546A">·</span><span style="font-size:7.0pt;color:#44546A">
</span><span style="font-size:10.0pt;font-family:"Intel Clear",sans-serif;color:#44546A">Win 8.1 Timers (Samsung)</span><o:p></o:p></p>
<p><span style="font-size:10.0pt;font-family:Symbol;color:#44546A">·</span><span style="font-size:7.0pt;color:#44546A">
</span><span style="font-size:10.0pt;font-family:"Intel Clear",sans-serif;color:#44546A">EOL Read Only (Samsung)</span><o:p></o:p></p>
<p><span style="font-size:10.0pt;font-family:Symbol;color:#44546A">·</span><span style="font-size:7.0pt;color:#44546A">
</span><span style="font-size:10.0pt;font-family:"Intel Clear",sans-serif;color:#44546A">Concurrent channels (Google)</span><o:p></o:p></p>
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto"><span style="font-size:10.0pt;font-family:"Intel Clear",sans-serif;color:#44546A"> </span><o:p></o:p></p>
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto"><span style="font-size:10.0pt;font-family:"Intel Clear",sans-serif;color:#44546A">The namespace management patch will be sent out for review tomorrow (look for the patch email from
Carolyn)… stay tuned!!!</span><o:p></o:p></p>
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto"><span style="font-size:10.0pt;font-family:"Intel Clear",sans-serif;color:#44546A"> </span><o:p></o:p></p>
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto"><span style="font-size:10.0pt;font-family:"Intel Clear",sans-serif;color:#44546A">However, once this patch is resolved, we as a community will have to make a decision on an official
release strategy. The patch process and cadence was significantly slower in 2015 which leaves us with a few options.
</span><o:p></o:p></p>
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto"><span style="font-size:10.0pt;font-family:"Intel Clear",sans-serif;color:#44546A"> </span><o:p></o:p></p>
<p><span style="font-size:10.0pt;font-family:"Intel Clear",sans-serif;color:#44546A">1.</span><span style="font-size:7.0pt;color:#44546A">
</span><span style="font-size:10.0pt;font-family:"Intel Clear",sans-serif;color:#44546A">Release what is in the trunk today (or after the namespace management patch)… and call that the 2015 release (albeit later than expected)</span><o:p></o:p></p>
<p><span style="font-size:10.0pt;font-family:"Intel Clear",sans-serif;color:#44546A">2.</span><span style="font-size:7.0pt;color:#44546A">
</span><span style="font-size:10.0pt;font-family:"Intel Clear",sans-serif;color:#44546A">In lieu of an official 2015 release, we push the remaining patches listed above… and then release in ~Q2 of 2015. Basically skip a 2015 release and go right into the mid-2016
release.</span><o:p></o:p></p>
<p><span style="font-size:10.0pt;font-family:"Intel Clear",sans-serif;color:#44546A">3.</span><span style="font-size:7.0pt;color:#44546A">
</span><span style="font-size:10.0pt;font-family:"Intel Clear",sans-serif;color:#44546A">Remove the concept of “official releases” from the OFA Windows NVMe driver and just allow people, companies, and users to pull from the OFA trunk as necessary.</span><o:p></o:p></p>
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto"><span style="font-size:10.0pt;font-family:"Intel Clear",sans-serif;color:#44546A"> </span><o:p></o:p></p>
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto"><span style="font-size:10.0pt;font-family:"Intel Clear",sans-serif;color:#44546A">For #3 above, my thoughts are that because the OFA driver is not a production driver… but a reference
and enabling driver, it should/could be managed as just that… a reference baseline driver that any potential user can go and grab the source, or contribute a patch. Nothing more… nothing less.</span><o:p></o:p></p>
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto"><span style="font-size:10.0pt;font-family:"Intel Clear",sans-serif;color:#44546A"> </span><o:p></o:p></p>
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto"><span style="font-size:10.0pt;font-family:"Intel Clear",sans-serif;color:#44546A;background:yellow">For the release decision, I’ll be happy to call a meeting… but we can also handle
it via email as well… just let me know. Feedback from all is welcome… but I would request mandatory feedback form the 4 reviewing companies: Samsung, HGST, PMC-Sierra, and Intel. How would you like to proceed?</span><o:p></o:p></p>
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto"><span style="font-size:10.0pt;font-family:"Intel Clear",sans-serif;color:#44546A"> </span><o:p></o:p></p>
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto"><span style="font-size:10.0pt;font-family:"Intel Clear",sans-serif;color:#44546A">Thanks,</span><o:p></o:p></p>
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto"><span style="font-size:10.0pt;font-family:"Intel Clear",sans-serif;color:#44546A">Ray</span><o:p></o:p></p>
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto"><span style="font-size:10.0pt;font-family:"Intel Clear",sans-serif;color:#44546A"> </span><o:p></o:p></p>
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto"><b><span style="font-size:10.0pt;font-family:"Intel Clear",sans-serif;color:#1F497D"><img border="0" width="54" height="39" id="_x0000_i1028" src="cid:image004.png@01D1C63F.A6983FD0" alt="cid:image001.png@01CB3870.4BB88E70"></span></b><o:p></o:p></p>
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto"><b><i><span style="font-family:"Intel Clear",sans-serif;color:#1F497D">Raymond C. Robles</span></i></b><o:p></o:p></p>
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto"><span style="font-size:10.0pt;font-family:"Intel Clear",sans-serif;color:#1F497D">Non-Volatile Memory Solutions Group</span><o:p></o:p></p>
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto"><span style="font-size:10.0pt;font-family:"Intel Clear",sans-serif;color:#1F497D">Intel Corporation</span><o:p></o:p></p>
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto"><span style="font-size:10.0pt;font-family:"Intel Clear",sans-serif;color:#1F497D">Office:
<a href="tel:480-554-2600" target="_blank">480-554-2600</a></span><o:p></o:p></p>
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto"><span style="font-size:10.0pt;font-family:"Intel Clear",sans-serif;color:#1F497D">Mobile:
<a href="tel:480-399-0645" target="_blank">480-399-0645</a></span><o:p></o:p></p>
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto"> <o:p></o:p></p>
</div>
</div>
</div>
</div>
</blockquote>
</div>
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto"> <o:p></o:p></p>
</div>
</div>
</div>
</div>
</div>
</blockquote>
</div>
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto"> <o:p></o:p></p>
</div>
</div>
</div>
</blockquote>
</div>
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto"> <o:p></o:p></p>
</div>
</div>
<p> <o:p></o:p></p>
<p> <o:p></o:p></p>
<table class="MsoNormalTable" border="0" cellpadding="0">
<tbody>
<tr>
<td style="padding:.75pt .75pt .75pt .75pt">
<p><img border="0" width="520" height="144" id="_x0000_i1029" src="cid:image005.gif@01D1C63F.A6983FD0"><o:p></o:p></p>
</td>
</tr>
</tbody>
</table>
</div>
</div>
</div>
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto"><span style="border:solid windowtext 1.0pt;padding:0in"><img border="0" width="100" height="100" id="_x0000_i1030" src="cid:image006.jpg@01D1C63F.A6983FD0" alt="Image removed by sender."></span><o:p></o:p></p>
</blockquote>
</div>
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto"> <o:p></o:p></p>
</div>
</div>
<p> <o:p></o:p></p>
<p> <o:p></o:p></p>
</div>
</div>
<table class="MsoNormalTable" border="0" cellpadding="0">
<tbody>
<tr>
<td style="padding:.75pt .75pt .75pt .75pt">
<p><img border="0" width="520" height="144" id="_x0000_i1031" src="cid:image005.gif@01D1C63F.A6983FD0"><o:p></o:p></p>
</td>
</tr>
</tbody>
</table>
</div>
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto"><span style="border:solid windowtext 1.0pt;padding:0in"><img border="0" width="100" height="100" id="_x0000_i1032" src="cid:image006.jpg@01D1C63F.A6983FD0" alt="Image removed by sender."></span><o:p></o:p></p>
</blockquote>
</div>
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto"> <o:p></o:p></p>
</div>
</div>
</div>
</blockquote>
</div>
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto"> <o:p></o:p></p>
</div>
</div>
</div>
</blockquote>
</div>
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto"> <o:p></o:p></p>
</div>
</div>
<div>
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto">Western Digital Corporation (and its subsidiaries) E-mail Confidentiality Notice & Disclaimer:<o:p></o:p></p>
<div>
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto">This e-mail and any files transmitted with it may contain confidential or legally privileged information of WDC and/or its affiliates, and are intended solely for the use of the
individual or entity to which they are addressed. If you are not the intended recipient, any disclosure, copying, distribution or any action taken or omitted to be taken in reliance on it, is prohibited. If you have received this e-mail in error, please notify
the sender immediately and delete the e-mail in its entirety from your system.<o:p></o:p></p>
</div>
</div>
</div>
</blockquote>
</div>
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto"> <o:p></o:p></p>
</div>
</div>
</div>
</div>
<div>
<div>
<div>
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto">Western Digital Corporation (and its subsidiaries) E-mail Confidentiality Notice & Disclaimer:<o:p></o:p></p>
<div>
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto">This e-mail and any files transmitted with it may contain confidential or legally privileged information of WDC and/or its affiliates, and are intended solely for the use of the
individual or entity to which they are addressed. If you are not the intended recipient, any disclosure, copying, distribution or any action taken or omitted to be taken in reliance on it, is prohibited. If you have received this e-mail in error, please notify
the sender immediately and delete the e-mail in its entirety from your system.<o:p></o:p></p>
</div>
</div>
</div>
</div>
</div>
</blockquote>
</div>
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto"> <o:p></o:p></p>
</div>
</div>
</div>
</div>
<div>
<div>
<div>
<p class="MsoNormal">Western Digital Corporation (and its subsidiaries) E-mail Confidentiality Notice & Disclaimer:<o:p></o:p></p>
<div>
<p class="MsoNormal">This e-mail and any files transmitted with it may contain confidential or legally privileged information of WDC and/or its affiliates, and are intended solely for the use of the individual or entity to which they are addressed. If you are
not the intended recipient, any disclosure, copying, distribution or any action taken or omitted to be taken in reliance on it, is prohibited. If you have received this e-mail in error, please notify the sender immediately and delete the e-mail in its entirety
from your system.<o:p></o:p></p>
</div>
</div>
</div>
</div>
</div>
</blockquote>
</div>
<p class="MsoNormal"><o:p> </o:p></p>
</div>
</div>
</body>
</html>