<html xmlns:v="urn:schemas-microsoft-com:vml" xmlns:o="urn:schemas-microsoft-com:office:office" xmlns:w="urn:schemas-microsoft-com:office:word" xmlns:m="http://schemas.microsoft.com/office/2004/12/omml" xmlns="http://www.w3.org/TR/REC-html40">
<head>
<meta http-equiv="Content-Type" content="text/html; charset=us-ascii">
<meta name="Generator" content="Microsoft Word 12 (filtered medium)">
<style><!--
/* Font Definitions */
@font-face
{font-family:Wingdings;
panose-1:5 0 0 0 0 0 0 0 0 0;}
@font-face
{font-family:"Cambria Math";
panose-1:2 4 5 3 5 4 6 3 2 4;}
@font-face
{font-family:Calibri;
panose-1:2 15 5 2 2 2 4 3 2 4;}
@font-face
{font-family:Tahoma;
panose-1:2 11 6 4 3 5 4 4 2 4;}
/* Style Definitions */
p.MsoNormal, li.MsoNormal, div.MsoNormal
{margin:0in;
margin-bottom:.0001pt;
font-size:11.0pt;
font-family:"Calibri","sans-serif";}
a:link, span.MsoHyperlink
{mso-style-priority:99;
color:blue;
text-decoration:underline;}
a:visited, span.MsoHyperlinkFollowed
{mso-style-priority:99;
color:purple;
text-decoration:underline;}
p.MsoAcetate, li.MsoAcetate, div.MsoAcetate
{mso-style-priority:99;
mso-style-link:"Balloon Text Char";
margin:0in;
margin-bottom:.0001pt;
font-size:8.0pt;
font-family:"Tahoma","sans-serif";}
p.MsoListParagraph, li.MsoListParagraph, div.MsoListParagraph
{mso-style-priority:34;
margin-top:0in;
margin-right:0in;
margin-bottom:0in;
margin-left:.5in;
margin-bottom:.0001pt;
font-size:11.0pt;
font-family:"Calibri","sans-serif";}
span.BalloonTextChar
{mso-style-name:"Balloon Text Char";
mso-style-priority:99;
mso-style-link:"Balloon Text";
font-family:"Tahoma","sans-serif";}
span.EmailStyle20
{mso-style-type:personal;
font-family:"Calibri","sans-serif";
color:windowtext;}
span.EmailStyle21
{mso-style-type:personal;
font-family:"Calibri","sans-serif";
color:#1F497D;}
span.EmailStyle22
{mso-style-type:personal;
font-family:"Calibri","sans-serif";
color:#1F497D;}
span.EmailStyle23
{mso-style-type:personal;
font-family:"Calibri","sans-serif";
color:#1F497D;}
span.EmailStyle24
{mso-style-type:personal-reply;
font-family:"Calibri","sans-serif";
color:#1F497D;}
.MsoChpDefault
{mso-style-type:export-only;
font-size:10.0pt;}
@page WordSection1
{size:8.5in 11.0in;
margin:1.0in 1.0in 1.0in 1.0in;}
div.WordSection1
{page:WordSection1;}
/* List Definitions */
@list l0
{mso-list-id:1472215286;
mso-list-type:hybrid;
mso-list-template-ids:-1858799420 67698705 67698713 67698715 67698703 67698713 67698715 67698703 67698713 67698715;}
@list l0:level1
{mso-level-text:"%1\)";
mso-level-tab-stop:none;
mso-level-number-position:left;
text-indent:-.25in;}
@list l0:level2
{mso-level-number-format:alpha-lower;
mso-level-tab-stop:none;
mso-level-number-position:left;
text-indent:-.25in;}
ol
{margin-bottom:0in;}
ul
{margin-bottom:0in;}
--></style><!--[if gte mso 9]><xml>
<o:shapedefaults v:ext="edit" spidmax="1026" />
</xml><![endif]--><!--[if gte mso 9]><xml>
<o:shapelayout v:ext="edit">
<o:idmap v:ext="edit" data="1" />
</o:shapelayout></xml><![endif]-->
</head>
<body lang="EN-US" link="blue" vlink="purple">
<div class="WordSection1">
<p class="MsoListParagraph" style="text-indent:-.25in;mso-list:l0 level1 lfo1"><![if !supportLists]><span style="color:#1F497D"><span style="mso-list:Ignore">1)<span style="font:7.0pt "Times New Roman"">
</span></span></span><![endif]><span style="color:#1F497D">Regarding one of the previous emails – there is no restriction preventing the SCSI block size from changing during format. Yes, it can! Unlikely, but not forbidden; therefore the READ CAPACITY command
– which gives you everything to discover this information. There were/are so-called “soft sectored” drives where the LBA block can be resized during format. Whether they are in wide use anymore … not sure. But no restrictions in SCSI.<o:p></o:p></span></p>
<p class="MsoListParagraph" style="text-indent:-.25in;mso-list:l0 level1 lfo1"><![if !supportLists]><span style="color:#1F497D"><span style="mso-list:Ignore">2)<span style="font:7.0pt "Times New Roman"">
</span></span></span><![endif]><span style="color:#1F497D">Namespaces absolutely can disappear … and reappear … and be renamed. Can even have different namespaces referencing exactly the same storage area as other namespaces … or overlapping. Namespaces are
Wild West – anything goes. And, they must be “instantiated”, i.e. Create and Delete Namespace. Yes, it is not specified in the NVMe spec how that is done (outside the scope), but it mentions that this must happen. Point of reference, there are companies that
are developing drives that do exactly this, in addition to allowing a variable programmable number of namespaces to be instantiated within. You may be sensing I’m not a fan. Yes, I have a problem with the namespace concept or lack of ground rules regarding.
But this is what we got. Now, how do we get it to work consistently, while at the same time mapping to the SCSI LU concept? Yes, we can lay down rules – but where does one go to reference those? Who’s the keeper of the rules? And do they change by manufacturer?
OS? Platform?<o:p></o:p></span></p>
<p class="MsoListParagraph" style="text-indent:-.25in;mso-list:l0 level1 lfo1"><![if !supportLists]><span style="color:#1F497D"><span style="mso-list:Ignore">3)<span style="font:7.0pt "Times New Roman"">
</span></span></span><![endif]><span style="color:#1F497D">SCSI LU’s and therefore LUN’s (the number/address) ARE NEVER instantiated. They all exist in the device and are constant from the moment of power-on. Internally LU’s usually have a physical relationship
to hardware, i.e. if a device has N LU’s, then there’s usually N data paths and N buffers and N storage areas for N independent units. That is, in SCSI parlance, there are N Device Servers – one per each LUN (see SCSI Architecture Model or SPC4, etc.).<o:p></o:p></span></p>
<p class="MsoListParagraph" style="text-indent:-.25in;mso-list:l0 level1 lfo1"><![if !supportLists]><span style="color:#1F497D"><span style="mso-list:Ignore">4)<span style="font:7.0pt "Times New Roman"">
</span></span></span><![endif]><span style="color:#1F497D">Someone mentioned this earlier – yes, you can continue sending commands to a SCSI device WHILE it is formatting. It is still online! It will report BUSY in status or NOT READY in the SENSE KEY. SCSI
can also produce a UNIT ATTENTION condition after formatting – meaning, “host check with me, something radical has changed”. NVMe specification does not speak to these conditions. Yes, we can make something up … but again, who’s the keeper of that info?<o:p></o:p></span></p>
<p class="MsoListParagraph" style="margin-left:1.0in;text-indent:-.25in;mso-list:l0 level2 lfo1">
<![if !supportLists]><span style="color:#1F497D"><span style="mso-list:Ignore">a.<span style="font:7.0pt "Times New Roman"">
</span></span></span><![endif]><span style="color:#1F497D">Also, using SAM, there is a TASK MANAGEMENT function that can return information indicating 0-100% completion of Format (or any long duration command).<o:p></o:p></span></p>
<p class="MsoNormal"><span style="color:#1F497D"><o:p> </o:p></span></p>
<p class="MsoNormal"><span style="color:#1F497D">Bottom line, IMHO – NVMe specification has serious omissions. The specification should be the SINGLE point of reference for these issues. It needs to address these issues and have defined protocols on how to
handle them. But, that ship has sailed.<o:p></o:p></span></p>
<p class="MsoNormal"><span style="color:#1F497D"><o:p> </o:p></span></p>
<p class="MsoNormal"><span style="color:#1F497D">Regards,<o:p></o:p></span></p>
<p class="MsoNormal"><span style="color:#1F497D">Neal<o:p></o:p></span></p>
<p class="MsoNormal"><span style="color:#1F497D"><o:p> </o:p></span></p>
<p class="MsoNormal"><span style="color:#1F497D"><o:p> </o:p></span></p>
<div>
<div style="border:none;border-top:solid #B5C4DF 1.0pt;padding:3.0pt 0in 0in 0in">
<p class="MsoNormal"><b><span style="font-size:10.0pt;font-family:"Tahoma","sans-serif"">From:</span></b><span style="font-size:10.0pt;font-family:"Tahoma","sans-serif""> Judy Brock-SSI [mailto:judy.brock@ssi.samsung.com]
<br>
<b>Sent:</b> Tuesday, January 07, 2014 9:42 PM<br>
<b>To:</b> Neal Galbo (ngalbo); Mayes, Barrett N; nvmewin@lists.openfabrics.org<br>
<b>Subject:</b> RE: Handling IO when Format NVM op is in progress<o:p></o:p></span></p>
</div>
</div>
<p class="MsoNormal"><o:p> </o:p></p>
<p class="MsoNormal"><span style="color:#1F497D"> >>In general, LU’s and LUN’s never disappear in SCSI; they are not transient. They are not dynamic. They are static. They either exist or they don’t. They don’t hide, once enumerated/attached/located. Unlike
namespaces.<o:p></o:p></span></p>
<p class="MsoNormal"><span style="color:#1F497D"><o:p> </o:p></span></p>
<p class="MsoNormal"><span style="color:#1F497D">Like SCSI LUNs, NVMe namespaces are also not transient and (I now submit, having reversed positions
</span><span style="font-family:Wingdings;color:#1F497D">J</span><span style="color:#1F497D">) that they should not hide either.<o:p></o:p></span></p>
<p class="MsoNormal"><span style="color:#1F497D"><o:p> </o:p></span></p>
<p class="MsoNormal"><span style="color:#1F497D">Format NVM command does not result in a different number of Namespaces then existed before the operation after it is finished. While namespaces can be formatted with different LBA Format than previously, they
are still static in terms of their existence/non-existence.<o:p></o:p></span></p>
<p class="MsoNormal"><span style="color:#1F497D"><o:p> </o:p></span></p>
<p class="MsoNormal"><span style="color:#1F497D">Additionally , the current code is not achieving what it intended to do since it does not actually hide any LUNS before launching the Format NVM op. That is it does not wait till the existing LUNs are “gone”
(ie until the miniport fails to report them in a subsequent inquiry) before it starts the operation. <o:p></o:p></span></p>
<p class="MsoNormal"><span style="color:#1F497D"><o:p> </o:p></span></p>
<p class="MsoNormal"><span style="color:#1F497D">Thanks,<o:p></o:p></span></p>
<p class="MsoNormal"><span style="color:#1F497D">Judy<o:p></o:p></span></p>
<p class="MsoNormal"><span style="color:#1F497D"><o:p> </o:p></span></p>
<div>
<div style="border:none;border-top:solid #B5C4DF 1.0pt;padding:3.0pt 0in 0in 0in">
<p class="MsoNormal"><b><span style="font-size:10.0pt;font-family:"Tahoma","sans-serif"">From:</span></b><span style="font-size:10.0pt;font-family:"Tahoma","sans-serif""> Neal Galbo (ngalbo) [mailto:ngalbo@micron.com]
<br>
<b>Sent:</b> Tuesday, January 07, 2014 6:27 AM<br>
<b>To:</b> Mayes, Barrett N; Judy Brock-SSI; nvmewin@lists.openfabrics.org<br>
<b>Subject:</b> RE: Handling IO when Format NVM op is in progress<o:p></o:p></span></p>
</div>
</div>
<p class="MsoNormal"><o:p> </o:p></p>
<p class="MsoNormal"><span style="color:#1F497D">The conditions you describe would never happen with a SCSI device. In general, LU’s and LUN’s never disappear in SCSI; they are not transient. They are not dynamic. They are static. They either exist or they
don’t. They don’t hide, once enumerated/attached/located. Unlike namespaces.<o:p></o:p></span></p>
<p class="MsoNormal"><span style="color:#1F497D"><o:p> </o:p></span></p>
<p class="MsoNormal"><span style="color:#1F497D">The media, backing storage or provisioning can change, but the LU would always be available for communication (commands). Other LU’s in the same device would not be affected either – they are independent entities
relative to each other.<o:p></o:p></span></p>
<p class="MsoNormal"><span style="color:#1F497D"><o:p> </o:p></span></p>
<p class="MsoNormal"><span style="color:#1F497D">-Neal<o:p></o:p></span></p>
<p class="MsoNormal"><span style="color:#1F497D"><o:p> </o:p></span></p>
<div>
<div style="border:none;border-top:solid #B5C4DF 1.0pt;padding:3.0pt 0in 0in 0in">
<p class="MsoNormal"><b><span style="font-size:10.0pt;font-family:"Tahoma","sans-serif"">From:</span></b><span style="font-size:10.0pt;font-family:"Tahoma","sans-serif"">
<a href="mailto:nvmewin-bounces@lists.openfabrics.org">nvmewin-bounces@lists.openfabrics.org</a> [<a href="mailto:nvmewin-bounces@lists.openfabrics.org">mailto:nvmewin-bounces@lists.openfabrics.org</a>]
<b>On Behalf Of </b>Mayes, Barrett N<br>
<b>Sent:</b> Tuesday, January 07, 2014 12:15 AM<br>
<b>To:</b> Judy Brock-SSI; <a href="mailto:nvmewin@lists.openfabrics.org">nvmewin@lists.openfabrics.org</a><br>
<b>Subject:</b> Re: [nvmewin] Handling IO when Format NVM op is in progress<o:p></o:p></span></p>
</div>
</div>
<p class="MsoNormal"><o:p> </o:p></p>
<p class="MsoNormal"><span style="color:#1F497D">What problem do you want to solve by keeping the block device around during a format and allowing IO through so it can be failed with check condition?
<o:p></o:p></span></p>
<p class="MsoNormal"><span style="color:#1F497D"><o:p> </o:p></span></p>
<p class="MsoNormal"><span style="color:#1F497D">Namespace not ready can’t generically be translated to SCSI check condition/not ready/Format In Progress. The driver would need to know that a format command is outstanding so it could translate that correctly
(for 1.0-based device support) since the namespace could be not ready for reasons other than a format in progress. If the driver already has to know a format is in progress, it could just fail commands without sending it to the device (so no need for the
new failure code). But if that’s the case, why _<i>not</i>_ hide the LUN until the format is complete. By hiding the LUN and bringing it back when the format is complete, you don’t have to worry about handling IO and you also take care of the re-enumeration
that has to happen when the format is complete (in the case of changing the LBA Format).<o:p></o:p></span></p>
<p class="MsoNormal"><span style="color:#1F497D"><o:p> </o:p></span></p>
<p class="MsoNormal"><span style="color:#1F497D">-Barrett<o:p></o:p></span></p>
<p class="MsoNormal"><span style="color:#1F497D"><o:p> </o:p></span></p>
<div>
<div style="border:none;border-top:solid #E1E1E1 1.0pt;padding:3.0pt 0in 0in 0in">
<p class="MsoNormal"><b>From:</b> <a href="mailto:nvmewin-bounces@lists.openfabrics.org">
nvmewin-bounces@lists.openfabrics.org</a> [<a href="mailto:nvmewin-bounces@lists.openfabrics.org">mailto:nvmewin-bounces@lists.openfabrics.org</a>]
<b>On Behalf Of </b>Judy Brock-SSI<br>
<b>Sent:</b> Monday, January 06, 2014 8:09 PM<br>
<b>To:</b> <a href="mailto:nvmewin@lists.openfabrics.org">nvmewin@lists.openfabrics.org</a><br>
<b>Subject:</b> [nvmewin] Handling IO when Format NVM op is in progress<o:p></o:p></p>
</div>
</div>
<p class="MsoNormal"><o:p> </o:p></p>
<p class="MsoNormal">All,<o:p></o:p></p>
<p class="MsoNormal"><o:p> </o:p></p>
<p class="MsoNormal">Many months ago, I initiated a thread (see attached) in which I argued that there were some holes in the current implementation of Format NVM ioctl passthru and in which I advocated for, among other things, the addition of logic to make
sure pseudo-SCSI bus re-enumeration had fully taken place in the driver (such that Storport was notified that no “luns” were present) before the actual Format NVM op was launched.<o:p></o:p></p>
<p class="MsoNormal"><o:p> </o:p></p>
<p class="MsoNormal">I intuitively understand – and up to this point have unquestioningly agreed with – the basic assumption that the reason the namespaces must be removed/ “luns” disappeared prior to formatting is because “<span style="font-size:10.0pt;color:#1F497D">we
cannot format a namespace while the OS is aware of its presence and could be potentially sending I/O to a stale namespace config (i.e. changing LBA/sector size).</span>” (excerpt from attached thread).<o:p></o:p></p>
<p class="MsoNormal"><o:p> </o:p></p>
<p class="MsoNormal">The question has recently arisen in internal discussion however as to whether or not we really have to do this. It was pointed out that SCSI devices (real ones) are capable of receiving IO commands while SCSI format commands are in process.
They will return the following error:<o:p></o:p></p>
<p class="MsoNormal"> <o:p></o:p></p>
<p class="MsoNormal"> SCSI status = Check condition, sense code = NOT READY (0x2), additional sense code = LUN NOT READY (0x4), additional sense code qualifier = FORMAT IN PROGRESS (0x4)<o:p></o:p></p>
<p class="MsoNormal"> <o:p></o:p></p>
<p class="MsoNormal">Why then, instead of removing namespaces/luns, can our Windows driver not return the same error status a real SCSI drive would return in such a situation? One would assume that upper layers of the storage stack have plenty years of experience
in knowing what to do when it sees that error. <o:p></o:p></p>
<p class="MsoNormal"><o:p> </o:p></p>
<p class="MsoNormal">As a point of comparison, there is no standard I am aware of which specifies that Storport miniports which support real SCSI devices, if they happen to provide a proprietary pass-thru to allow a SCSI format command to go through to a device,
must cause all LUNS to appear offline prior to formatting,<o:p></o:p></p>
<p class="MsoNormal"><o:p> </o:p></p>
<p class="MsoNormal">One could even argue (and they have!) that these IO commands could even be allowed to go through to the NVMe device itself (as in the real SCSI case); NVMe 1.1 Technical Proposal 005 has defined a new format-in-progress status code that
NVMe firmware will be able to return at some point in the future, current firmware could easily return NAMESPACE_NOT_READY and driver could translate to the above SCSI sense data, etc.<o:p></o:p></p>
<p class="MsoNormal"><o:p> </o:p></p>
<p class="MsoNormal">So … here I stand, devil’s advocate hat in hand, hoping to find out:<o:p></o:p></p>
<p class="MsoNormal"><o:p> </o:p></p>
<p class="MsoListParagraph" style="text-indent:-.25in">a)<span style="font-size:7.0pt;font-family:"Times New Roman","serif"">
</span> what the “back story” is on how this decision was ultimately made (the attached thread said a lot of discussion took place on the subject)<o:p></o:p></p>
<p class="MsoListParagraph" style="text-indent:-.25in">b)<span style="font-size:7.0pt;font-family:"Times New Roman","serif"">
</span> whether or not the diametrically-opposed alternative I am discussing above was thoroughly considered and if so, why it was rejected<o:p></o:p></p>
<p class="MsoListParagraph" style="text-indent:-.25in">c)<span style="font-size:7.0pt;font-family:"Times New Roman","serif"">
</span> whether the topic bears reconsidering at this point.<o:p></o:p></p>
<p class="MsoNormal"><o:p> </o:p></p>
<p class="MsoNormal">Thanks in advance for your collective consideration,<o:p></o:p></p>
<p class="MsoNormal"><o:p> </o:p></p>
<p class="MsoNormal">Judy<o:p></o:p></p>
<p class="MsoNormal"><o:p> </o:p></p>
</div>
</body>
</html>