<html xmlns:v="urn:schemas-microsoft-com:vml" xmlns:o="urn:schemas-microsoft-com:office:office" xmlns:w="urn:schemas-microsoft-com:office:word" xmlns:m="http://schemas.microsoft.com/office/2004/12/omml" xmlns="http://www.w3.org/TR/REC-html40">
<head>
<meta http-equiv="Content-Type" content="text/html; charset=utf-8">
<meta name="Generator" content="Microsoft Word 12 (filtered medium)">
<!--[if !mso]><style>v\:* {behavior:url(#default#VML);}
o\:* {behavior:url(#default#VML);}
w\:* {behavior:url(#default#VML);}
.shape {behavior:url(#default#VML);}
</style><![endif]--><style><!--
/* Font Definitions */
@font-face
{font-family:Wingdings;
panose-1:5 0 0 0 0 0 0 0 0 0;}
@font-face
{font-family:Gulim;
panose-1:2 11 6 0 0 1 1 1 1 1;}
@font-face
{font-family:Gulim;
panose-1:2 11 6 0 0 1 1 1 1 1;}
@font-face
{font-family:Calibri;
panose-1:2 15 5 2 2 2 4 3 2 4;}
@font-face
{font-family:Tahoma;
panose-1:2 11 6 4 3 5 4 4 2 4;}
@font-face
{font-family:"\@Gulim";
panose-1:2 11 6 0 0 1 1 1 1 1;}
/* Style Definitions */
p.MsoNormal, li.MsoNormal, div.MsoNormal
{margin:0in;
margin-bottom:.0001pt;
font-size:12.0pt;
font-family:"Times New Roman","serif";}
a:link, span.MsoHyperlink
{mso-style-priority:99;
color:blue;
text-decoration:underline;}
a:visited, span.MsoHyperlinkFollowed
{mso-style-priority:99;
color:purple;
text-decoration:underline;}
p.MsoAcetate, li.MsoAcetate, div.MsoAcetate
{mso-style-priority:99;
mso-style-link:"Balloon Text Char";
margin:0in;
margin-bottom:.0001pt;
font-size:8.0pt;
font-family:"Tahoma","sans-serif";}
p.MsoListParagraph, li.MsoListParagraph, div.MsoListParagraph
{mso-style-priority:34;
margin-top:0in;
margin-right:0in;
margin-bottom:0in;
margin-left:.5in;
margin-bottom:.0001pt;
font-size:12.0pt;
font-family:"Times New Roman","serif";}
span.BalloonTextChar
{mso-style-name:"Balloon Text Char";
mso-style-priority:99;
mso-style-link:"Balloon Text";
font-family:"Tahoma","sans-serif";}
p.emailquote, li.emailquote, div.emailquote
{mso-style-name:emailquote;
mso-margin-top-alt:auto;
margin-right:0in;
mso-margin-bottom-alt:auto;
margin-left:1.0pt;
font-size:12.0pt;
font-family:"Times New Roman","serif";}
span.EmailStyle21
{mso-style-type:personal;
font-family:"Times New Roman","serif";
color:#1F497D;
font-weight:normal;
font-style:normal;}
span.EmailStyle22
{mso-style-type:personal;
font-family:"Calibri","sans-serif";
color:#1F497D;}
span.EmailStyle23
{mso-style-type:personal;
font-family:"Calibri","sans-serif";
color:#1F497D;}
span.EmailStyle24
{mso-style-type:personal;
font-family:"Calibri","sans-serif";
color:#1F497D;}
span.EmailStyle25
{mso-style-type:personal-reply;
font-family:"Calibri","sans-serif";
color:#1F497D;}
.MsoChpDefault
{mso-style-type:export-only;
font-size:10.0pt;}
@page WordSection1
{size:8.5in 11.0in;
margin:1.0in 1.0in 1.0in 1.0in;}
div.WordSection1
{page:WordSection1;}
--></style><!--[if gte mso 9]><xml>
<o:shapedefaults v:ext="edit" spidmax="1026" />
</xml><![endif]--><!--[if gte mso 9]><xml>
<o:shapelayout v:ext="edit">
<o:idmap v:ext="edit" data="1" />
</o:shapelayout></xml><![endif]-->
</head>
<body lang="EN-US" link="blue" vlink="purple">
<div class="WordSection1">
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:"Calibri","sans-serif";color:#1F497D">Hi Barret,<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:"Calibri","sans-serif";color:#1F497D"><o:p> </o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:"Calibri","sans-serif";color:#1F497D">To sum up the info you provide below, I think it confirms the following point I was trying to make:<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:"Calibri","sans-serif";color:#1F497D"><o:p> </o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:"Calibri","sans-serif";color:#1F497D">the miniport cannot assume that, after signally a Bus Change Notification to initiate the removal process, once the call to signal the Bus Change Notification
returns, the upper layers will already have done all the work involved in the device removal process.
<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:"Calibri","sans-serif";color:#1F497D"><o:p> </o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:"Calibri","sans-serif";color:#1F497D">In fact, that there is a lot of work involved in device removal (for ex, we both cited the new set of inquires to come in to the miniport which allow the miniport
to signal that a previously reported LUN is no longer present).<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:"Calibri","sans-serif";color:#1F497D"><o:p> </o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:"Calibri","sans-serif";color:#1F497D">Thanks,<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:"Calibri","sans-serif";color:#1F497D">Judy<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:"Calibri","sans-serif";color:#1F497D"><o:p> </o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:"Calibri","sans-serif";color:#1F497D"><o:p> </o:p></span></p>
<div>
<div style="border:none;border-top:solid #B5C4DF 1.0pt;padding:3.0pt 0in 0in 0in">
<p class="MsoNormal"><b><span style="font-size:10.0pt;font-family:"Tahoma","sans-serif"">From:</span></b><span style="font-size:10.0pt;font-family:"Tahoma","sans-serif""> Mayes, Barrett N [mailto:barrett.n.mayes@intel.com]
<br>
<b>Sent:</b> Wednesday, May 01, 2013 10:00 AM<br>
<b>To:</b> Judy Brock-SSI; Robles, Raymond C; nvmewin@lists.openfabrics.org<br>
<b>Subject:</b> RE: [nvmewin] Handling pending commands when processing Format [changing from NVMe WG dist. list to OFA NVMe Windows Driver dist. list]<o:p></o:p></span></p>
</div>
</div>
<p class="MsoNormal"><o:p> </o:p></p>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:"Calibri","sans-serif";color:#1F497D">I’ll take a quick stab at the first part. Will need Ray or someone with more experience on Format implementation to address the 2<sup>nd</sup> part as I’m
not sure on the synchronization of completing the inquiries following the Bus Change notification and starting of the format command.<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:"Calibri","sans-serif";color:#1F497D"><o:p> </o:p></span></p>
<p class="MsoListParagraph" style="text-indent:-.25in"><i><span style="font-size:10.0pt;color:#1F497D">a)</span></i><i><span style="font-size:7.0pt;color:#1F497D">
</span></i><i><span style="font-size:10.0pt;color:#1F497D">I don’t think it’s true that Storport can handle commands completing for a device it no longer has a record of – ie, after the target was removed due to re-enumeration.I think it will have torn down
its own structures for any old LUN(s) we had previously exposed and that would include any record it had of commands outstanding for those old LUNs. I think it isn’t going to hold on to ghost requests on behalf of devices that it no longer has a record of
because it has no place to store such requests anyway at that point/ no object to associate them with.<o:p></o:p></span></i></p>
<p class="MsoNormal"><i><span style="font-size:11.0pt;font-family:"Calibri","sans-serif";color:#1F497D"><o:p> </o:p></span></i></p>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:"Calibri","sans-serif";color:#1F497D">In general, the port driver will stall its queues, complete any IO already in flight or queued to the device then delete the device object. The process is
fairly involved and is documented here: <a href="http://msdn.microsoft.com/en-us/library/windows/hardware/ff561046(v=vs.85).aspx">
http://msdn.microsoft.com/en-us/library/windows/hardware/ff561046(v=vs.85).aspx</a> This describes how a WDM driver handles the remove process. The other piece to this is the storport<->miniport interaction. To initiate the removal process, the miniport
can signal a Bus Change Notification to cause storport to send a new set of inquiries. If a previously reported LUN is not reported in a subsequent inquiry, storport will call IoInvalidateDeviceRelations where it will in turn not report back a previously
reported Device Object. See the “BusRelations Request” section here: <a href="http://msdn.microsoft.com/en-us/library/windows/hardware/ff551670(v=vs.85).aspx">
http://msdn.microsoft.com/en-us/library/windows/hardware/ff551670(v=vs.85).aspx</a> and docs for IRP_MN_REMOVE_DEVICE here:
<a href="http://msdn.microsoft.com/en-us/library/windows/hardware/ff551738(v=vs.85).aspx">
http://msdn.microsoft.com/en-us/library/windows/hardware/ff551738(v=vs.85).aspx</a>.
<o:p></o:p></span></p>
<p class="MsoListParagraph"><i><span style="font-size:10.0pt;color:#1F497D"><o:p> </o:p></span></i></p>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:"Calibri","sans-serif";color:#1F497D"><o:p> </o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:"Calibri","sans-serif";color:#1F497D"><o:p> </o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:"Calibri","sans-serif";color:#1F497D"><o:p> </o:p></span></p>
<div>
<div style="border:none;border-top:solid #E1E1E1 1.0pt;padding:3.0pt 0in 0in 0in">
<p class="MsoNormal"><b><span style="font-size:11.0pt;font-family:"Calibri","sans-serif"">From:</span></b><span style="font-size:11.0pt;font-family:"Calibri","sans-serif"">
<a href="mailto:nvmewin-bounces@lists.openfabrics.org">nvmewin-bounces@lists.openfabrics.org</a> [<a href="mailto:nvmewin-bounces@lists.openfabrics.org">mailto:nvmewin-bounces@lists.openfabrics.org</a>]
<b>On Behalf Of </b>Judy Brock-SSI<br>
<b>Sent:</b> Wednesday, May 01, 2013 3:33 AM<br>
<b>To:</b> Robles, Raymond C; <a href="mailto:nvmewin@lists.openfabrics.org">nvmewin@lists.openfabrics.org</a><br>
<b>Subject:</b> Re: [nvmewin] Handling pending commands when processing Format [changing from NVMe WG dist. list to OFA NVMe Windows Driver dist. list]<o:p></o:p></span></p>
</div>
</div>
<p class="MsoNormal"><o:p> </o:p></p>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:"Calibri","sans-serif";color:#1F497D">Hi Ray,<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:"Calibri","sans-serif";color:#1F497D"><o:p> </o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:"Calibri","sans-serif";color:#1F497D">I was wondering if you or others had any thoughts about the below.<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:"Calibri","sans-serif";color:#1F497D"><o:p> </o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:"Calibri","sans-serif";color:#1F497D">Thanks,<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:"Calibri","sans-serif";color:#1F497D">Judy<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:"Calibri","sans-serif";color:#1F497D"><o:p> </o:p></span></p>
<div>
<div style="border:none;border-top:solid #B5C4DF 1.0pt;padding:3.0pt 0in 0in 0in">
<p class="MsoNormal"><b><span style="font-size:10.0pt;font-family:"Tahoma","sans-serif"">From:</span></b><span style="font-size:10.0pt;font-family:"Tahoma","sans-serif""> Judy Brock-SSI
<br>
<b>Sent:</b> Monday, April 22, 2013 11:27 PM<br>
<b>To:</b> 'Robles, Raymond C'; WONMOON CHEON; ???; <a href="mailto:nvmewin@lists.openfabrics.org">
nvmewin@lists.openfabrics.org</a><br>
<b>Subject:</b> RE: RE: Handling pending commands when processing Format [changing from NVMe WG dist. list to OFA NVMe Windows Driver dist. list]<o:p></o:p></span></p>
</div>
</div>
<p class="MsoNormal"><o:p> </o:p></p>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:"Calibri","sans-serif";color:#1F497D">Hi Ray,<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:"Calibri","sans-serif";color:#1F497D"><o:p> </o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:"Calibri","sans-serif";color:#1F497D">Thanks for all the details.<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:"Calibri","sans-serif";color:#1F497D"><o:p> </o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:"Calibri","sans-serif";color:#1F497D">The stuff about hot adding the namespaces back in, reissuing Identify Controller/Identify Device, etc was very easy to follow in the driver; it’s clear how
& why that’s done. <o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:"Calibri","sans-serif";color:#1F497D"><o:p> </o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:"Calibri","sans-serif";color:#1F497D">However I’m still having some difficulty with the finishing-up-IOs-outstanding before starting the format issue.<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:"Calibri","sans-serif";color:#1F497D"><o:p> </o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:"Calibri","sans-serif";color:#1F497D">I may be just covering ground the group covered a long time ago but I’m wondering if all the assumptions below are valid. I’m also interested in knowing how
the group validated them. Here are my concerns:<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:10.0pt;color:#1F497D"><o:p> </o:p></span></p>
<p class="MsoNormal"><span style="font-size:10.0pt;color:#1F497D"> >> if a format comes down for a namespace, any I/O outstanding to the controller (there won’t be anything that needs to be sent… I/O will either be at the device or on the CQ)<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:10.0pt;color:#1F497D"><o:p> </o:p></span></p>
<p class="MsoNormal"><span style="font-size:10.0pt;color:#1F497D">Why do we think there won’t be anything that needs to be sent? The app that sends a pass through IOCTL to do the format is presumably completely independent from say any file IO that might be
going on on behalf of other apps, or even raw IO from apps like Iometer. Seems like lots of IO could still be coming in when the format IOCTL is received.
<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:10.0pt;color:#1F497D"><o:p> </o:p></span></p>
<p class="MsoNormal"><span style="font-size:10.0pt;color:#1F497D"> > > any I/O outstanding to the controller …will simply complete via normal operation or be aborted at the controller… but Storport won’t care because the SCSI target was already removed upon
initially receiving the format command. Any I/O in the CQ will be completed and handled by Storport correctly.<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:10.0pt;color:#1F497D"><o:p> </o:p></span></p>
<p class="MsoNormal"><span style="font-size:10.0pt;color:#1F497D">I am wondering about two of the assumptions about timing in the paragraph above.
<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:10.0pt;color:#1F497D"><o:p> </o:p></span></p>
<p class="MsoListParagraph" style="text-indent:-.25in"><span style="font-size:10.0pt;color:#1F497D">a)</span><span style="font-size:7.0pt;color:#1F497D">
</span><span style="font-size:10.0pt;color:#1F497D">I don’t think it’s true that Storport can handle commands completing for a device it no longer has a record of – ie, after the target was removed due to re-enumeration.I think it will have torn down its own
structures for any old LUN(s) we had previously exposed and that would include any record it had of commands outstanding for those old LUNs. I think it isn’t going to hold on to ghost requests on behalf of devices that it no longer has a record of because
it has no place to store such requests anyway at that point/ no object to associate them with.<o:p></o:p></span></p>
<p class="MsoNormal" style="margin-left:.25in"><span style="font-size:10.0pt;color:#1F497D"><o:p> </o:p></span></p>
<p class="MsoListParagraph" style="text-indent:-.25in"><span style="font-size:10.0pt;color:#1F497D">b)</span><span style="font-size:7.0pt;color:#1F497D">
</span><span style="font-size:10.0pt;color:#1F497D">We are assuming that when we call StorPortNotification() with BusChangeDetected that Storport will come back in to the driver to rescan the bus – either via a bunch of Inquiry cmds or via a Report Luns cmd
– and will finish all the work associated with the bus scan, updating it’s record of device topology (ie remove any SCSI target/LUNs that were assocated with the NVMe dev we are about to format) – all before it returns from our call to ScsiPortNotification
and before the driver code continues on to call ProcessIO to start the real NVMe Format NVM operation.
<o:p></o:p></span></p>
<p class="MsoListParagraph"><span style="font-size:10.0pt;color:#1F497D"><o:p> </o:p></span></p>
<p class="MsoNormal" style="margin-left:.5in"><span style="font-size:10.0pt;color:#1F497D">I don’t know that that is a safe correct assumption to make. The bus scan could be deferred till the driver returns. Or even if it is launched right away, could the bus
scan take place on a different processor while the proc that is running through the driver just continues on its’ way?
<o:p></o:p></span></p>
<p class="MsoNormal" style="margin-left:.5in"><span style="font-size:10.0pt;color:#1F497D"><o:p> </o:p></span></p>
<p class="MsoNormal" style="margin-left:.25in"><span style="font-size:10.0pt;color:#1F497D">In my experience, it is the drivers and controllers’ joint responsibility to make sure that all outstanding IOs are completed back to the caller one way or the other
<u>before </u>starting the format op. That means aborting whatever can be aborted and also making absolutely sure that no live request left over from a Namespace that has been removed ever gets completed back to the host after the Namespace has been removed.
<o:p></o:p></span></p>
<p class="MsoNormal" style="margin-left:.25in"><span style="font-size:10.0pt;color:#1F497D"><o:p> </o:p></span></p>
<p class="MsoNormal" style="margin-left:.25in"><span style="font-size:10.0pt;color:#1F497D">Thanks,<o:p></o:p></span></p>
<p class="MsoNormal" style="margin-left:.25in"><span style="font-size:10.0pt;color:#1F497D"><o:p> </o:p></span></p>
<p class="MsoNormal" style="margin-left:.25in"><span style="font-size:10.0pt;color:#1F497D">Judy<o:p></o:p></span></p>
<p class="MsoNormal" style="margin-left:.25in"><span style="font-size:10.0pt;color:#1F497D"><o:p> </o:p></span></p>
<p class="MsoNormal" style="margin-left:.5in"><span style="font-size:10.0pt;color:#1F497D"><o:p> </o:p></span></p>
<p class="MsoListParagraph"><span style="font-size:10.0pt;color:#1F497D"><o:p> </o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:"Calibri","sans-serif";color:#1F497D"><o:p> </o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:"Calibri","sans-serif";color:#1F497D"><o:p> </o:p></span></p>
<div>
<div style="border:none;border-top:solid #B5C4DF 1.0pt;padding:3.0pt 0in 0in 0in">
<p class="MsoNormal"><b><span style="font-size:10.0pt;font-family:"Tahoma","sans-serif"">From:</span></b><span style="font-size:10.0pt;font-family:"Tahoma","sans-serif""> Robles, Raymond C [<a href="mailto:raymond.c.robles@intel.com">mailto:raymond.c.robles@intel.com</a>]
<br>
<b>Sent:</b> Thursday, April 18, 2013 6:08 PM<br>
<b>To:</b> Judy Brock-SSI; WONMOON CHEON; ???; <a href="mailto:nvmewin@lists.openfabrics.org">
nvmewin@lists.openfabrics.org</a><br>
<b>Subject:</b> RE: RE: Handling pending commands when processing Format [changing from NVMe WG dist. list to OFA NVMe Windows Driver dist. list]<o:p></o:p></span></p>
</div>
</div>
<p class="MsoNormal"><o:p> </o:p></p>
<p class="MsoNormal"><span style="font-size:10.0pt;color:#1F497D">Hi Judy,<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:10.0pt;color:#1F497D"><o:p> </o:p></span></p>
<p class="MsoNormal"><span style="font-size:10.0pt;color:#1F497D">Ahhh… I see now. I didn’t answer that question below. The format command is essentially built into the driver by a state machine.<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:10.0pt;color:#1F497D"><o:p> </o:p></span></p>
<p class="MsoNormal"><span style="font-size:10.0pt;color:#1F497D">When we receive the format command we immediately issue the *<b>hot remove</b>* command… but that is done inline. So, once we call Storport to kick off the enumeration, we simply return back
to handling the format command in NVMeStartIoProcessIoctl(). The appropriate states are set along the way to indicate progress. Once the namespace is removed from the “OS view”, then the format is processed like any other command (via ProcessIo). The callback
is setup to call NVMeIoctlFormatNVMCAllback() and the variable “FormatNvmInfo->AddNamespaceNeeded” is set to TRUE so that on the completion side we remember to have the OS re-enumerate after we are done. Once the NVM format completes, the callback is invoked
in the completion DPC. Then on the completion side we issue Identify Controller and Identify Namespace so that our cached driver data for the namespace(s) formatted are up to date. In the last state, after getting the Identify Namespace struct, we’ll call
*<b>hot add</b>* which is described below. <o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:10.0pt;color:#1F497D"><o:p> </o:p></span></p>
<p class="MsoNormal"><span style="font-size:10.0pt;color:#1F497D">Note that at no point do we “wait” for any I/O to finish. Format is a dangerous command… especially via pass through IOCTL. We talked about this quite a bit in the beginning of developing this
driver. But essentially, if a format comes down for a namespace, any I/O outstanding to the controller (there won’t be anything that needs to be sent… I/O will either be at the device or on the CQ) will simply complete via normal operation or be aborted at
the controller… but Storport won’t care because the SCSI target was already removed upon initially receiving the format command. Any I/O in the CQ will be completed and handled by Storport correctly.<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:10.0pt;color:#1F497D"><o:p> </o:p></span></p>
<p class="MsoNormal"><span style="font-size:10.0pt;color:#1F497D">Thanks,<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:10.0pt;color:#1F497D">Ray<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:10.0pt;color:#1F497D"><o:p> </o:p></span></p>
<div>
<div style="border:none;border-top:solid #B5C4DF 1.0pt;padding:3.0pt 0in 0in 0in">
<p class="MsoNormal"><b><span style="font-size:10.0pt;font-family:"Tahoma","sans-serif"">From:</span></b><span style="font-size:10.0pt;font-family:"Tahoma","sans-serif""> Judy Brock-SSI [<a href="mailto:judy.brock@ssi.samsung.com">mailto:judy.brock@ssi.samsung.com</a>]
<br>
<b>Sent:</b> Thursday, April 18, 2013 4:47 PM<br>
<b>To:</b> Robles, Raymond C; WONMOON CHEON; ???; <a href="mailto:nvmewin@lists.openfabrics.org">
nvmewin@lists.openfabrics.org</a><br>
<b>Subject:</b> RE: RE: Handling pending commands when processing Format [changing from NVMe WG dist. list to OFA NVMe Windows Driver dist. list]<o:p></o:p></span></p>
</div>
</div>
<p class="MsoNormal"><o:p> </o:p></p>
<div style="margin-top:5.0pt;margin-bottom:5.0pt">
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:"Calibri","sans-serif";color:#1F497D">Hi Ray,</span><span style="font-size:11.0pt;font-family:"Calibri","sans-serif""><o:p></o:p></span></p>
</div>
<div style="margin-top:5.0pt;margin-bottom:5.0pt">
<p class="MsoNormal"><span style="font-size:10.0pt;color:#1F497D"> [Ray wrote] Let me know if this answers your question.</span><span style="font-size:11.0pt;font-family:"Calibri","sans-serif""><o:p></o:p></span></p>
</div>
<div style="margin-top:5.0pt;margin-bottom:5.0pt">
<p class="MsoNormal"><span style="font-size:10.0pt;color:#1F497D">I don’t think it does. What I wrote below I think was pretty much the same as what you wrote - or at least that was my intention
</span><span style="font-size:10.0pt;font-family:Wingdings;color:#1F497D">J</span><span style="font-size:10.0pt;color:#1F497D">.
</span><span style="font-size:11.0pt;font-family:"Calibri","sans-serif""><o:p></o:p></span></p>
</div>
<div style="margin-top:5.0pt;margin-bottom:5.0pt">
<p class="MsoNormal"><span style="font-size:10.0pt;color:#1F497D">However, the piece I couldn’t explain (cause I haven’t looked into it) is how the driver holds off the beginning of the actual format NVM operation till whatever old IOs that were already in
progress for the namespace(s) before the format op request was received are completed back to the caller, aborted, or whatever - so there are no old live requests hanging around, still in the driver , before the format op begins.</span><span style="font-size:11.0pt;font-family:"Calibri","sans-serif""><o:p></o:p></span></p>
</div>
<div style="margin-top:5.0pt;margin-bottom:5.0pt">
<p class="MsoNormal"><span style="font-size:10.0pt;color:#1F497D">In other words, does the driver hold off starting the format cmd till the outstanding IOs are completed? Or do we perhaps just drop them on the floor and let the OS figure out that those requests
are permanently lost/gone due to the LUNs having disappeared (my guess is, the latter is what we do)? Or do we try to abort them all? And so on.</span><span style="font-size:11.0pt;font-family:"Calibri","sans-serif""><o:p></o:p></span></p>
</div>
<div style="margin-top:5.0pt;margin-bottom:5.0pt">
<p class="MsoNormal"><span style="font-size:10.0pt;color:#1F497D">So again, we do understand how to get the OS to avoid sending new I/O requests to stale namespaces but how exactly are the old I/O reqs (those existing at the time the format request comes in
) handled?</span><span style="font-size:11.0pt;font-family:"Calibri","sans-serif""><o:p></o:p></span></p>
</div>
<div style="margin-top:5.0pt;margin-bottom:5.0pt">
<p class="MsoNormal"><span style="font-size:10.0pt;color:#1F497D">At least that is my current question
</span><span style="font-size:10.0pt;font-family:Wingdings;color:#1F497D">J</span><span style="font-size:11.0pt;font-family:"Calibri","sans-serif""><o:p></o:p></span></p>
</div>
<div style="margin-top:5.0pt;margin-bottom:5.0pt">
<p class="MsoNormal"><span style="font-size:10.0pt;color:#1F497D">Thanks,</span><span style="font-size:11.0pt;font-family:"Calibri","sans-serif""><o:p></o:p></span></p>
</div>
<div style="margin-top:5.0pt;margin-bottom:5.0pt">
<p class="MsoNormal"><span style="font-size:10.0pt;color:#1F497D">Judy</span><span style="font-size:11.0pt;font-family:"Calibri","sans-serif""><o:p></o:p></span></p>
</div>
<div style="margin-top:5.0pt;margin-bottom:5.0pt">
<p class="MsoNormal"><span style="color:#1F497D"> </span><span style="font-size:11.0pt;font-family:"Calibri","sans-serif""><o:p></o:p></span></p>
</div>
<div style="margin-top:5.0pt;margin-bottom:5.0pt">
<p class="MsoNormal"><span style="color:#1F497D"> </span><span style="font-size:11.0pt;font-family:"Calibri","sans-serif""><o:p></o:p></span></p>
</div>
<div>
<p class="MsoNormal"><b><span style="font-size:10.0pt;font-family:"Tahoma","sans-serif"">From:</span></b><span style="font-size:10.0pt;font-family:"Tahoma","sans-serif""> Robles, Raymond C [</span><a href="mailto:raymond.c.robles@intel.com"><span style="font-size:10.0pt;font-family:"Tahoma","sans-serif"">mailto:raymond.c.robles@intel.com</span></a><span style="font-size:10.0pt;font-family:"Tahoma","sans-serif"">]
<br>
<b>Sent:</b> Thursday, April 18, 2013 2:46 PM<br>
<b>To:</b> Judy Brock-SSI; WONMOON CHEON; ???; </span><a href="mailto:nvmewin@lists.openfabrics.org"><span style="font-size:10.0pt;font-family:"Tahoma","sans-serif"">nvmewin@lists.openfabrics.org</span></a><span style="font-size:10.0pt;font-family:"Tahoma","sans-serif""><br>
<b>Subject:</b> RE: RE: Handling pending commands when processing Format [changing from NVMe WG dist. list to OFA NVMe Windows Driver dist. list]</span><span style="font-size:11.0pt;font-family:"Calibri","sans-serif""><o:p></o:p></span></p>
</div>
<div style="margin-top:5.0pt;margin-bottom:5.0pt">
<p class="MsoNormal"> <span style="font-size:11.0pt;font-family:"Calibri","sans-serif""><o:p></o:p></span></p>
</div>
<div style="margin-top:5.0pt;margin-bottom:5.0pt">
<p class="MsoNormal"><span style="font-size:10.0pt;color:#1F497D">Hello Judy/Wonmoon,</span><span style="font-size:11.0pt;font-family:"Calibri","sans-serif""><o:p></o:p></span></p>
</div>
<div style="margin-top:5.0pt;margin-bottom:5.0pt">
<p class="MsoNormal"><span style="font-size:10.0pt;color:#1F497D">Sorry for the late response. The “hot remove” state is just a state that we enter when the driver receives a Format command. Basically, this state will remove the namespace(s) from the topology
by calling StorPortNotification() with BusChangeDetected. This will remove the “SCSI target/disk” associated with each namespace form the OS (because Storport will re-enumerate the controller and the driver will not expose the namespaces about to be formatted)
so that the format can occur on the relevant namespaces.</span><span style="font-size:11.0pt;font-family:"Calibri","sans-serif""><o:p></o:p></span></p>
</div>
<div style="margin-top:5.0pt;margin-bottom:5.0pt">
<p class="MsoNormal"><span style="font-size:10.0pt;color:#1F497D">By signaling Windows that the namespaces have been “removed”, all I/O will be stopped by the OS. Then the format can complete. Once the format is complete, we perform the opposite action to
“hot add” the namespace back into the topology by calling StorPortNotification() with BusChangeDetected… only this time, we will surface the namespace(s) again when Storport re-enumerates.</span><span style="font-size:11.0pt;font-family:"Calibri","sans-serif""><o:p></o:p></span></p>
</div>
<div style="margin-top:5.0pt;margin-bottom:5.0pt">
<p class="MsoNormal"><span style="font-size:10.0pt;color:#1F497D">Not queues are deleted in this state, no memory is de-allocated, and nothing else changes about the namespace. This is simply the first step (in a 3 step sequence) when formatting a namespace
as we cannot format a namespace while the OS is aware of its presence and could be potentially sending I/O to a stale namespace config (i.e. changing LBA/sector size).</span><span style="font-size:11.0pt;font-family:"Calibri","sans-serif""><o:p></o:p></span></p>
</div>
<div style="margin-top:5.0pt;margin-bottom:5.0pt">
<p class="MsoNormal"><span style="font-size:10.0pt;color:#1F497D">Let me know if this answers your question.</span><span style="font-size:11.0pt;font-family:"Calibri","sans-serif""><o:p></o:p></span></p>
</div>
<div style="margin-top:5.0pt;margin-bottom:5.0pt">
<p class="MsoNormal"><span style="font-size:10.0pt;color:#1F497D">Thanks,</span><span style="font-size:11.0pt;font-family:"Calibri","sans-serif""><o:p></o:p></span></p>
</div>
<div style="margin-top:5.0pt;margin-bottom:5.0pt">
<p class="MsoNormal"><span style="font-size:10.0pt;color:#1F497D">Ray</span><span style="font-size:11.0pt;font-family:"Calibri","sans-serif""><o:p></o:p></span></p>
</div>
<div style="margin-top:5.0pt;margin-bottom:5.0pt">
<p class="MsoNormal"><span style="font-size:10.0pt;color:#1F497D"> </span><span style="font-size:11.0pt;font-family:"Calibri","sans-serif""><o:p></o:p></span></p>
</div>
<div>
<p class="MsoNormal"><b><span style="font-size:10.0pt;font-family:"Tahoma","sans-serif"">From:</span></b><span style="font-size:10.0pt;font-family:"Tahoma","sans-serif""> Judy Brock-SSI [</span><a href="mailto:judy.brock@ssi.samsung.com"><span style="font-size:10.0pt;font-family:"Tahoma","sans-serif"">mailto:judy.brock@ssi.samsung.com</span></a><span style="font-size:10.0pt;font-family:"Tahoma","sans-serif"">]
<br>
<b>Sent:</b> Wednesday, April 17, 2013 6:21 AM<br>
<b>To:</b> WONMOON CHEON; </span><span style="font-size:10.0pt;font-family:"Gulim","sans-serif"">강미경</span><span style="font-size:10.0pt;font-family:"Tahoma","sans-serif"">;
</span><a href="mailto:technical@nvmexpress.org"><span style="font-size:10.0pt;font-family:"Tahoma","sans-serif"">technical@nvmexpress.org</span></a><span style="font-size:10.0pt;font-family:"Tahoma","sans-serif""><br>
<b>Subject:</b> RE: RE: Handling pending commands when processing Format</span><span style="font-size:11.0pt;font-family:"Calibri","sans-serif""><o:p></o:p></span></p>
</div>
<div style="margin-top:5.0pt;margin-bottom:5.0pt">
<p class="MsoNormal"> <span style="font-size:11.0pt;font-family:"Calibri","sans-serif""><o:p></o:p></span></p>
</div>
<div style="margin-top:3.0pt;margin-bottom:3.0pt">
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:"Calibri","sans-serif""> >>Would you elaborate more about the "hot-remove" state? In this state, do you mean that all the IO SQ/CQs are deleted? Or, waiting for completions of all the outstanding
IOs?<o:p></o:p></span></p>
</div>
<div style="margin-top:5.0pt;margin-bottom:5.0pt">
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:"Calibri","sans-serif";color:#1F497D">The IO SQ/CQs are definitely NOT deleted. I would need to look more closely through the driver code to see how IOs previously sent to the namespaces which are
marked as OFFLINE are handled/finished/quiesced. </span><span style="font-size:11.0pt;font-family:"Calibri","sans-serif""><o:p></o:p></span></p>
</div>
<div style="margin-top:5.0pt;margin-bottom:5.0pt">
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:"Calibri","sans-serif";color:#1F497D">There are other folks on this thread who no doubt have more history/intimate knowledge of this driver than I do who may answer that question more quickly than
I can…also, perhaps this discussion should probably be moved to the OFA driver forum since it has turned into a driver-specific thread at this point.
</span><span style="font-size:11.0pt;font-family:"Calibri","sans-serif""><o:p></o:p></span></p>
</div>
<div style="margin-top:5.0pt;margin-bottom:5.0pt">
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:"Calibri","sans-serif";color:#1F497D">What do folks think?</span><span style="font-size:11.0pt;font-family:"Calibri","sans-serif""><o:p></o:p></span></p>
</div>
<div style="margin-top:5.0pt;margin-bottom:5.0pt">
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:"Calibri","sans-serif";color:#1F497D">Judy</span><span style="font-size:11.0pt;font-family:"Calibri","sans-serif""><o:p></o:p></span></p>
</div>
<div>
<p class="MsoNormal"><b><span style="font-size:10.0pt;font-family:"Tahoma","sans-serif"">From:</span></b><span style="font-size:10.0pt;font-family:"Tahoma","sans-serif"">
</span><span style="font-size:10.0pt;font-family:"Gulim","sans-serif"">천원문</span><span style="font-size:10.0pt;font-family:"Tahoma","sans-serif""> [</span><a href="mailto:wm.cheon@samsung.com"><span style="font-size:10.0pt;font-family:"Tahoma","sans-serif"">mailto:wm.cheon@samsung.com</span></a><span style="font-size:10.0pt;font-family:"Tahoma","sans-serif"">]
<br>
<b>Sent:</b> Wednesday, April 17, 2013 1:20 AM<br>
<b>To:</b> Judy Brock-SSI; </span><span style="font-size:10.0pt;font-family:"Gulim","sans-serif"">강미경</span><span style="font-size:10.0pt;font-family:"Tahoma","sans-serif"">;
</span><a href="mailto:technical@nvmexpress.org"><span style="font-size:10.0pt;font-family:"Tahoma","sans-serif"">technical@nvmexpress.org</span></a><span style="font-size:10.0pt;font-family:"Tahoma","sans-serif""><br>
<b>Subject:</b> Re: RE: Handling pending commands when processing Format</span><span style="font-size:11.0pt;font-family:"Calibri","sans-serif""><o:p></o:p></span></p>
</div>
<div style="margin-top:5.0pt;margin-bottom:5.0pt">
<p class="MsoNormal"> <span style="font-size:11.0pt;font-family:"Calibri","sans-serif""><o:p></o:p></span></p>
</div>
<div style="margin-top:3.0pt;margin-bottom:3.0pt">
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:"Calibri","sans-serif"">Hi Judy,<o:p></o:p></span></p>
</div>
<div style="margin-top:3.0pt;margin-bottom:3.0pt">
<p class="MsoNormal"><span style="font-size:9.0pt;font-family:"Arial","sans-serif""> </span><span style="font-size:11.0pt;font-family:"Calibri","sans-serif""><o:p></o:p></span></p>
</div>
<div style="margin-top:3.0pt;margin-bottom:3.0pt">
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:"Calibri","sans-serif"">Would you elaborate more about the "hot-remove" state? In this state, do you mean that all the IO SQ/CQs are deleted? Or, waiting for completions of all the outstanding
IOs?<o:p></o:p></span></p>
</div>
<div style="margin-top:3.0pt;margin-bottom:3.0pt">
<p class="MsoNormal"><span style="font-size:9.0pt;font-family:"Arial","sans-serif""> </span><span style="font-size:11.0pt;font-family:"Calibri","sans-serif""><o:p></o:p></span></p>
</div>
<div style="margin-top:3.0pt;margin-bottom:3.0pt">
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:"Calibri","sans-serif"">Thanks,<o:p></o:p></span></p>
</div>
<div style="margin-top:3.0pt;margin-bottom:3.0pt">
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:"Calibri","sans-serif"">Wonmoon<o:p></o:p></span></p>
</div>
<div style="margin-top:3.0pt;margin-bottom:3.0pt">
<p class="MsoNormal"><span style="font-size:9.0pt;font-family:"Arial","sans-serif""> </span><span style="font-size:11.0pt;font-family:"Calibri","sans-serif""><o:p></o:p></span></p>
</div>
<div style="margin-top:3.0pt;margin-bottom:3.0pt">
<p class="MsoNormal"><span style="font-size:9.0pt;font-family:"Arial","sans-serif"">-------
<b>Original Message</b> -------</span><span style="font-size:11.0pt;font-family:"Calibri","sans-serif""><o:p></o:p></span></p>
</div>
<div style="margin-top:3.0pt;margin-bottom:3.0pt">
<p class="MsoNormal"><b><span style="font-size:9.0pt;font-family:"Arial","sans-serif"">Sender</span></b><span style="font-size:9.0pt;font-family:"Arial","sans-serif""> : Judy Brock-SSI<</span><a href="mailto:judy.brock@ssi.samsung.com"><span style="font-size:9.0pt;font-family:"Arial","sans-serif"">judy.brock@ssi.samsung.com</span></a><span style="font-size:9.0pt;font-family:"Arial","sans-serif"">></span><span style="font-size:11.0pt;font-family:"Calibri","sans-serif""><o:p></o:p></span></p>
</div>
<div style="margin-top:3.0pt;margin-bottom:3.0pt">
<p class="MsoNormal"><b><span style="font-size:9.0pt;font-family:"Arial","sans-serif"">Date</span></b><span style="font-size:9.0pt;font-family:"Arial","sans-serif""> : 2013-04-17 16:39 (GMT+09:00)</span><span style="font-size:11.0pt;font-family:"Calibri","sans-serif""><o:p></o:p></span></p>
</div>
<div style="margin-top:3.0pt;margin-bottom:3.0pt">
<p class="MsoNormal"><b><span style="font-size:9.0pt;font-family:"Arial","sans-serif"">Title</span></b><span style="font-size:9.0pt;font-family:"Arial","sans-serif""> : RE: Handling pending commands when processing Format</span><span style="font-size:11.0pt;font-family:"Calibri","sans-serif""><o:p></o:p></span></p>
</div>
<div style="margin-top:3.0pt;margin-bottom:3.0pt">
<p class="MsoNormal"><span style="font-size:9.0pt;font-family:"Arial","sans-serif""> </span><span style="font-size:11.0pt;font-family:"Calibri","sans-serif""><o:p></o:p></span></p>
</div>
<div style="margin-top:5.0pt;margin-bottom:5.0pt">
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:"Calibri","sans-serif";color:#1F497D">Hi,</span><span style="font-size:11.0pt;font-family:"Calibri","sans-serif""><o:p></o:p></span></p>
</div>
<div style="margin-top:5.0pt;margin-bottom:5.0pt">
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:"Calibri","sans-serif";color:#1F497D">I should clarify that it is not the Windows operating system – but rather the Windows OFA NVMe driver - that, from what I can see, does a “hot-remove” of
all namespace(s) associated with a device before allowing a format operation to begin; “hot remove” is just the name for an internal state in the driver format nvm state machine.
</span><span style="font-size:11.0pt;font-family:"Calibri","sans-serif""><o:p></o:p></span></p>
</div>
<div style="margin-top:5.0pt;margin-bottom:5.0pt">
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:"Calibri","sans-serif";color:#1F497D">Before beginning the actual format op, the driver internally marks all namespaces associated with the format operation “offline”. It then notifies the OS that
there has been a “bus change” event (via an OS-specific API).</span><span style="font-size:11.0pt;font-family:"Calibri","sans-serif""><o:p></o:p></span></p>
</div>
<div style="margin-top:5.0pt;margin-bottom:5.0pt">
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:"Calibri","sans-serif";color:#1F497D">This in turn will cause the OS to rescan (re-enumerate) the “bus” (the pseudo SCSI bus, that is – we expose NVM namespaces as SCSI luns).</span><span style="font-size:11.0pt;font-family:"Calibri","sans-serif""><o:p></o:p></span></p>
</div>
<div style="margin-top:5.0pt;margin-bottom:5.0pt">
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:"Calibri","sans-serif";color:#1F497D">Since all the pertinent namespaces have been marked offline internally, the bus rescan won’t detect any valid SCSI luns (because the driver will not report
any).</span><span style="font-size:11.0pt;font-family:"Calibri","sans-serif""><o:p></o:p></span></p>
</div>
<div style="margin-top:5.0pt;margin-bottom:5.0pt">
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:"Calibri","sans-serif";color:#1F497D">Hence from the OS point of view, any SCSI lun(s) previously mapped to the namespace(s) to be formatted will have disappeared/will be unaddressable while the
format operation is in progress.</span><span style="font-size:11.0pt;font-family:"Calibri","sans-serif""><o:p></o:p></span></p>
</div>
<div style="margin-top:5.0pt;margin-bottom:5.0pt">
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:"Calibri","sans-serif";color:#1F497D">Judy</span><span style="font-size:11.0pt;font-family:"Calibri","sans-serif""><o:p></o:p></span></p>
</div>
<div style="margin-top:5.0pt;margin-bottom:5.0pt">
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:"Calibri","sans-serif";color:#1F497D"> </span><span style="font-size:11.0pt;font-family:"Calibri","sans-serif""><o:p></o:p></span></p>
</div>
<div>
<p class="MsoNormal"><b><span style="font-size:10.0pt;font-family:"Tahoma","sans-serif"">From:</span></b><span style="font-size:10.0pt;font-family:"Tahoma","sans-serif""> Judy Brock-SSI
<br>
<b>Sent:</b> Tuesday, April 16, 2013 8:22 PM<br>
<b>To:</b> 'mkkang.kang@samsung.com'; </span><a href="mailto:technical@nvmexpress.org"><span style="font-size:10.0pt;font-family:"Tahoma","sans-serif"">technical@nvmexpress.org</span></a><span style="font-size:10.0pt;font-family:"Tahoma","sans-serif""><br>
<b>Subject:</b> RE: Handling pending commands when processing Format</span><span style="font-size:11.0pt;font-family:"Calibri","sans-serif""><o:p></o:p></span></p>
</div>
<div style="margin-top:5.0pt;margin-bottom:5.0pt">
<p class="MsoNormal"> <span style="font-size:11.0pt;font-family:"Calibri","sans-serif""><o:p></o:p></span></p>
</div>
<div style="margin-top:5.0pt;margin-bottom:5.0pt">
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:"Calibri","sans-serif";color:#1F497D">Mikyeong,</span><span style="font-size:11.0pt;font-family:"Calibri","sans-serif""><o:p></o:p></span></p>
</div>
<div style="margin-top:5.0pt;margin-bottom:5.0pt">
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:"Calibri","sans-serif";color:#1F497D">I haven’t looked at the Linux driver but I know that Windows hot-removes all namespace(s) associated with a device before allowing a format operation to begin.
And a namespace can’t be removed while there is IO outstanding to it so that answers your question regarding IOs being completed before format begins. It also answers the question about requests being sent to a namespace while format is in progress – can’t
happen.</span><span style="font-size:11.0pt;font-family:"Calibri","sans-serif""><o:p></o:p></span></p>
</div>
<div style="margin-top:5.0pt;margin-bottom:5.0pt">
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:"Calibri","sans-serif";color:#1F497D">Thanks,</span><span style="font-size:11.0pt;font-family:"Calibri","sans-serif""><o:p></o:p></span></p>
</div>
<div style="margin-top:5.0pt;margin-bottom:5.0pt">
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:"Calibri","sans-serif";color:#1F497D">Judy</span><span style="font-size:11.0pt;font-family:"Calibri","sans-serif""><o:p></o:p></span></p>
</div>
<div style="margin-top:5.0pt;margin-bottom:5.0pt">
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:"Calibri","sans-serif";color:#1F497D"> </span><span style="font-size:11.0pt;font-family:"Calibri","sans-serif""><o:p></o:p></span></p>
</div>
<div style="margin-top:5.0pt;margin-bottom:5.0pt">
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:"Calibri","sans-serif";color:#1F497D"> </span><span style="font-size:11.0pt;font-family:"Calibri","sans-serif""><o:p></o:p></span></p>
</div>
<div style="margin-top:5.0pt;margin-bottom:5.0pt">
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:"Calibri","sans-serif";color:#1F497D"> </span><span style="font-size:11.0pt;font-family:"Calibri","sans-serif""><o:p></o:p></span></p>
</div>
<div>
<p class="MsoNormal"><b><span style="font-size:10.0pt;font-family:"Tahoma","sans-serif"">From:</span></b><span style="font-size:10.0pt;font-family:"Tahoma","sans-serif"">
</span><span style="font-size:10.0pt;font-family:"Gulim","sans-serif"">강미경</span><span style="font-size:10.0pt;font-family:"Tahoma","sans-serif""> [</span><a href="mailto:mkkang.kang@samsung.com"><span style="font-size:10.0pt;font-family:"Tahoma","sans-serif"">mailto:mkkang.kang@samsung.com</span></a><span style="font-size:10.0pt;font-family:"Tahoma","sans-serif"">]
<br>
<b>Sent:</b> Tuesday, April 16, 2013 7:07 PM<br>
<b>To:</b> </span><a href="mailto:technical@nvmexpress.org"><span style="font-size:10.0pt;font-family:"Tahoma","sans-serif"">technical@nvmexpress.org</span></a><span style="font-size:10.0pt;font-family:"Tahoma","sans-serif""><br>
<b>Subject:</b> Handling pending commands when processing Format</span><span style="font-size:11.0pt;font-family:"Calibri","sans-serif""><o:p></o:p></span></p>
</div>
<div style="margin-top:5.0pt;margin-bottom:5.0pt">
<p class="MsoNormal"> <span style="font-size:11.0pt;font-family:"Calibri","sans-serif""><o:p></o:p></span></p>
</div>
<div style="margin-top:3.0pt;margin-bottom:3.0pt">
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:"Calibri","sans-serif"">Dear All,<o:p></o:p></span></p>
</div>
<div style="margin-top:3.0pt;margin-bottom:3.0pt">
<p class="MsoNormal"><span style="font-size:9.0pt;font-family:"Arial","sans-serif""> </span><span style="font-size:11.0pt;font-family:"Calibri","sans-serif""><o:p></o:p></span></p>
</div>
<div style="margin-top:3.0pt;margin-bottom:3.0pt">
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:"Calibri","sans-serif"">Format NVM command may change the Namespace repository, and it will be executed out of order like any other commands. Therefore, Format NVM command may affect other commands
that are pending execution in the device, if any. <o:p></o:p></span></p>
</div>
<div style="margin-top:3.0pt;margin-bottom:3.0pt">
<p class="MsoNormal"><span style="font-size:9.0pt;font-family:"Arial","sans-serif""> </span><span style="font-size:11.0pt;font-family:"Calibri","sans-serif""><o:p></o:p></span></p>
</div>
<div style="margin-top:3.0pt;margin-bottom:3.0pt">
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:"Calibri","sans-serif"">1) How does an OFA/linux driver handle 'Format NVM command'? Does a host make sure that all commands for a particular NSID are completed before sending 'Format NVM command'?<o:p></o:p></span></p>
</div>
<div style="margin-top:3.0pt;margin-bottom:3.0pt">
<p class="MsoNormal"><span style="font-size:9.0pt;font-family:"Arial","sans-serif""> </span><span style="font-size:11.0pt;font-family:"Calibri","sans-serif""><o:p></o:p></span></p>
</div>
<div style="margin-top:3.0pt;margin-bottom:3.0pt">
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:"Calibri","sans-serif"">2) If a host driver does not behave like 1) above, how can a device handle other pending commands which were previously submitted in a SQ? It seems like we need an additional
status code. e.g. Abort due to Namespace Format<o:p></o:p></span></p>
</div>
<div style="margin-top:3.0pt;margin-bottom:3.0pt">
<p class="MsoNormal"><span style="font-size:9.0pt;font-family:"Arial","sans-serif""> </span><span style="font-size:11.0pt;font-family:"Calibri","sans-serif""><o:p></o:p></span></p>
</div>
<div style="margin-top:3.0pt;margin-bottom:3.0pt">
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:"Calibri","sans-serif"">3) Let's suppose that 'Format NVM command' is in progress. If the host driver sends subsequent commands to the namespace being formatted, should the device reply directly
with a 'Namespace not Ready'?<o:p></o:p></span></p>
</div>
<div style="margin-top:3.0pt;margin-bottom:3.0pt">
<p class="MsoNormal"><span style="font-size:9.0pt;font-family:"Arial","sans-serif""> </span><span style="font-size:11.0pt;font-family:"Calibri","sans-serif""><o:p></o:p></span></p>
</div>
<div style="margin-top:3.0pt;margin-bottom:3.0pt">
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:"Calibri","sans-serif"">[1.0e spec] If the device does not reply directly and the format operation takes long time, then, I/O command will timeout and the host may send the reset. But if commands
are responded with 'Namespace Not Ready', host may not issue the reset. Therefore, direct reply seems to be needed.
<o:p></o:p></span></p>
</div>
<div style="margin-top:3.0pt;margin-bottom:3.0pt">
<p class="MsoNormal"><span style="font-size:9.0pt;font-family:"Arial","sans-serif""> </span><span style="font-size:11.0pt;font-family:"Calibri","sans-serif""><o:p></o:p></span></p>
</div>
<div style="margin-top:3.0pt;margin-bottom:3.0pt">
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:"Calibri","sans-serif"">[1.1 spec. ECN 001] There is Format progress indicator. The host driver can check format progress any time, therefore, there is no concern about reset during format command.<o:p></o:p></span></p>
</div>
<div style="margin-top:3.0pt;margin-bottom:3.0pt">
<p class="MsoNormal"><span style="font-size:9.0pt;font-family:"Arial","sans-serif""> </span><span style="font-size:11.0pt;font-family:"Calibri","sans-serif""><o:p></o:p></span></p>
</div>
<div style="margin-top:3.0pt;margin-bottom:3.0pt">
<p class="MsoNormal"><span style="font-size:9.0pt;font-family:"Arial","sans-serif""> </span><span style="font-size:11.0pt;font-family:"Calibri","sans-serif""><o:p></o:p></span></p>
</div>
<div style="margin-top:3.0pt;margin-bottom:3.0pt">
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:"Calibri","sans-serif"">Best Regards,<o:p></o:p></span></p>
</div>
<div style="margin-top:3.0pt;margin-bottom:3.0pt">
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:"Calibri","sans-serif"">Mikyeong Kang<o:p></o:p></span></p>
</div>
<div style="margin-top:3.0pt;margin-bottom:3.0pt">
<p class="MsoNormal"><span style="font-size:9.0pt;font-family:"Arial","sans-serif""> </span><span style="font-size:11.0pt;font-family:"Calibri","sans-serif""><o:p></o:p></span></p>
</div>
<div>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:"Calibri","sans-serif""><o:p> </o:p></span></p>
</div>
<div style="margin-top:3.0pt;margin-bottom:3.0pt">
<p class="MsoNormal"><b><span style="font-size:10.0pt;font-family:"Calibri","sans-serif"">Kang MiKyeong</span></b><span style="font-size:11.0pt;font-family:"Calibri","sans-serif""><o:p></o:p></span></p>
</div>
<div style="margin-top:3.0pt;margin-bottom:3.0pt">
<p class="MsoNormal"><span style="font-size:9.0pt;font-family:"Arial","sans-serif""> </span><span style="font-size:11.0pt;font-family:"Calibri","sans-serif""><o:p></o:p></span></p>
</div>
<div style="margin-top:5.0pt;margin-bottom:5.0pt">
<p class="MsoNormal"><span style="font-size:10.0pt;font-family:"Calibri","sans-serif"">Flash Memory Planning/Enabling Group, Memory Div.</span><span style="font-size:11.0pt;font-family:"Calibri","sans-serif""><o:p></o:p></span></p>
</div>
<div style="margin-top:5.0pt;margin-bottom:5.0pt">
<p class="MsoNormal"><span style="font-size:10.0pt;font-family:"Calibri","sans-serif"">SAMSUNG ELECTRONICS, Co., Ltd..</span><span style="font-size:11.0pt;font-family:"Calibri","sans-serif""><o:p></o:p></span></p>
</div>
<div style="margin-top:5.0pt;margin-bottom:5.0pt">
<p class="MsoNormal"><span style="font-size:10.0pt;font-family:"Calibri","sans-serif"">Phone: 82-31-208-3857</span><span style="font-size:11.0pt;font-family:"Calibri","sans-serif""><o:p></o:p></span></p>
</div>
<div style="margin-top:5.0pt;margin-bottom:5.0pt">
<p class="MsoNormal"><span style="font-size:10.0pt;font-family:"Calibri","sans-serif"">Mobile: 82-10-9369-0177</span><span style="font-size:11.0pt;font-family:"Calibri","sans-serif""><o:p></o:p></span></p>
</div>
<div style="margin-top:5.0pt;margin-bottom:5.0pt">
<p class="MsoNormal"><span style="font-size:10.0pt;font-family:"Calibri","sans-serif"">E-mail:
</span><a href="mailto:mkkang.kang@samsung.com"><span style="font-size:10.0pt;font-family:"Calibri","sans-serif"">mkkang.kang@samsung.com</span></a><span style="font-size:11.0pt;font-family:"Calibri","sans-serif""><o:p></o:p></span></p>
</div>
<div>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:"Calibri","sans-serif""><o:p> </o:p></span></p>
</div>
<div style="margin-top:3.0pt;margin-bottom:3.0pt">
<p class="MsoNormal"><span style="font-size:9.0pt;font-family:"Arial","sans-serif""> </span><span style="font-size:11.0pt;font-family:"Calibri","sans-serif""><o:p></o:p></span></p>
</div>
<div style="margin-top:3.0pt;margin-bottom:3.0pt">
<p class="MsoNormal"><span style="font-size:9.0pt;font-family:"Arial","sans-serif""> </span><span style="font-size:11.0pt;font-family:"Calibri","sans-serif""><o:p></o:p></span></p>
</div>
<div style="margin-top:3.0pt;margin-bottom:3.0pt">
<p class="MsoNormal"><span style="font-size:9.0pt;font-family:"Arial","sans-serif""> </span><span style="font-size:11.0pt;font-family:"Calibri","sans-serif""><o:p></o:p></span></p>
</div>
<table class="MsoNormalTable" border="0" cellspacing="3" cellpadding="0" width="659" style="width:395.1pt;margin-left:2.85pt">
<tbody>
<tr>
<td style="padding:.75pt .75pt .75pt .75pt">
<p class="MsoNormal" style="mso-margin-top-alt:3.0pt;margin-right:0in;margin-bottom:3.0pt;margin-left:0in">
<span style="font-size:9.0pt;font-family:"Arial","sans-serif""><img border="0" width="520" height="144" id="_x0000_i1025" src="cid:image001.jpg@01CE4688.32390FA0" alt="cid:image001.jpg@01CE3C59.BF34AF60"></span><o:p></o:p></p>
</td>
</tr>
</tbody>
</table>
<div>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:"Calibri","sans-serif""><o:p> </o:p></span></p>
</div>
<div>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:"Calibri","sans-serif""><o:p> </o:p></span></p>
</div>
<div>
<p class="MsoNormal"> <span style="font-size:11.0pt;font-family:"Calibri","sans-serif""><o:p></o:p></span></p>
</div>
</div>
</body>
</html>