<html xmlns:v="urn:schemas-microsoft-com:vml" xmlns:o="urn:schemas-microsoft-com:office:office" xmlns:w="urn:schemas-microsoft-com:office:word" xmlns:m="http://schemas.microsoft.com/office/2004/12/omml" xmlns="http://www.w3.org/TR/REC-html40">
<head>
<meta http-equiv="Content-Type" content="text/html; charset=us-ascii">
<meta name="Generator" content="Microsoft Word 12 (filtered medium)">
<style><!--
/* Font Definitions */
@font-face
{font-family:"Cambria Math";
panose-1:2 4 5 3 5 4 6 3 2 4;}
@font-face
{font-family:Calibri;
panose-1:2 15 5 2 2 2 4 3 2 4;}
@font-face
{font-family:Tahoma;
panose-1:2 11 6 4 3 5 4 4 2 4;}
@font-face
{font-family:Consolas;
panose-1:2 11 6 9 2 2 4 3 2 4;}
/* Style Definitions */
p.MsoNormal, li.MsoNormal, div.MsoNormal
{margin:0in;
margin-bottom:.0001pt;
font-size:11.0pt;
font-family:"Calibri","sans-serif";}
a:link, span.MsoHyperlink
{mso-style-priority:99;
color:blue;
text-decoration:underline;}
a:visited, span.MsoHyperlinkFollowed
{mso-style-priority:99;
color:purple;
text-decoration:underline;}
p.MsoPlainText, li.MsoPlainText, div.MsoPlainText
{mso-style-priority:99;
mso-style-link:"Plain Text Char";
margin:0in;
margin-bottom:.0001pt;
font-size:10.5pt;
font-family:Consolas;}
p.MsoAcetate, li.MsoAcetate, div.MsoAcetate
{mso-style-priority:99;
mso-style-link:"Balloon Text Char";
margin:0in;
margin-bottom:.0001pt;
font-size:8.0pt;
font-family:"Tahoma","sans-serif";}
span.PlainTextChar
{mso-style-name:"Plain Text Char";
mso-style-priority:99;
mso-style-link:"Plain Text";
font-family:Consolas;}
span.BalloonTextChar
{mso-style-name:"Balloon Text Char";
mso-style-priority:99;
mso-style-link:"Balloon Text";
font-family:"Tahoma","sans-serif";}
.MsoChpDefault
{mso-style-type:export-only;}
@page WordSection1
{size:8.5in 11.0in;
margin:1.0in 1.0in 1.0in 1.0in;}
div.WordSection1
{page:WordSection1;}
/* List Definitions */
@list l0
{mso-list-id:329914338;
mso-list-type:hybrid;
mso-list-template-ids:1924160558 67698703 67698713 67698715 67698703 67698713 67698715 67698703 67698713 67698715;}
@list l0:level1
{mso-level-tab-stop:none;
mso-level-number-position:left;
text-indent:-.25in;}
@list l1
{mso-list-id:612133458;
mso-list-type:hybrid;
mso-list-template-ids:942724774 67698705 67698713 67698715 67698703 67698713 67698715 67698703 67698713 67698715;}
@list l1:level1
{mso-level-text:"%1\)";
mso-level-tab-stop:none;
mso-level-number-position:left;
text-indent:-.25in;}
@list l2
{mso-list-id:1027485136;
mso-list-type:hybrid;
mso-list-template-ids:942724774 67698705 67698713 67698715 67698703 67698713 67698715 67698703 67698713 67698715;}
@list l2:level1
{mso-level-text:"%1\)";
mso-level-tab-stop:none;
mso-level-number-position:left;
text-indent:-.25in;}
ol
{margin-bottom:0in;}
ul
{margin-bottom:0in;}
--></style><!--[if gte mso 9]><xml>
<o:shapedefaults v:ext="edit" spidmax="1026" />
</xml><![endif]--><!--[if gte mso 9]><xml>
<o:shapelayout v:ext="edit">
<o:idmap v:ext="edit" data="1" />
</o:shapelayout></xml><![endif]-->
</head>
<body lang="EN-US" link="blue" vlink="purple">
<div class="WordSection1">
<p class="MsoPlainText">Hi,<o:p></o:p></p>
<p class="MsoPlainText">Responses inline below in <span style="color:red">red</span> - thanks.<o:p></o:p></p>
<p class="MsoPlainText"><o:p> </o:p></p>
<p class="MsoPlainText">-----Original Message-----<br>
From: nvmewin-bounces@lists.openfabrics.org [mailto:nvmewin-bounces@lists.openfabrics.org] On Behalf Of Kwok Kong<br>
Sent: Tuesday, January 07, 2014 9:57 AM<br>
To: nvmewin@lists.openfabrics.org<br>
Subject: Re: [nvmewin] Handling IO when Format NVM op is in progress<o:p></o:p></p>
<p class="MsoPlainText"><o:p> </o:p></p>
<p class="MsoPlainText">I would agree with Barrett. What problem do you want to solve here ?<o:p></o:p></p>
<p class="MsoPlainText"><span style="color:black"><o:p> </o:p></span></p>
<p class="MsoPlainText" style="margin-left:.5in;text-indent:-.25in;mso-list:l2 level1 lfo1">
<![if !supportLists]><span style="color:red"><span style="mso-list:Ignore">1)<span style="font:7.0pt "Times New Roman"">
</span></span></span><![endif]><span style="color:red">Bug-fixing current logic which does
<u>not</u> in fact hide any LUNs before initiating the format NVM op. As per the thread I attached at the beginning of this revived thread, it has been confirmed that this logic is missing. So right now we are not providing any protection against the scenario
we claim to want to guard against.<o:p></o:p></span></p>
<p class="MsoPlainText" style="margin-left:.5in;text-indent:-.25in;mso-list:l2 level1 lfo1">
<![if !supportLists]><span style="color:red"><span style="mso-list:Ignore">2)<span style="font:7.0pt "Times New Roman"">
</span></span></span><![endif]><span style="color:red">Simplification of the Windows driver. It is simpler logic to merely return an errors during format op like a real SCSI dev would than it is to wait for all the inquiries to come back down, in the meantime
returning errors on new IOS that come in after the Format NVM ioctl is received but before the entire bus rescan has completed (we have no control over how long it will take Storport to do that btw) and thus the existing LUNS are not yet offlined, etc. – the
whole fencing state machine is a lot more complicated if part of it depends on actions to be taken by Storport on the front end.<o:p></o:p></span></p>
<p class="MsoPlainText" style="margin-left:.5in;text-indent:-.25in;mso-list:l2 level1 lfo1">
<![if !supportLists]><span style="color:red"><span style="mso-list:Ignore">3)<span style="font:7.0pt "Times New Roman"">
</span></span></span><![endif]><span style="color:red">Compatibility with SCSI behavior – this is a SCSI miniport and thus inputs/outputs should behave pretty much the same as other SCSI miniports.<o:p></o:p></span></p>
<p class="MsoPlainText" style="margin-left:.25in"><span style="color:black"><o:p> </o:p></span></p>
<p class="MsoPlainText"><o:p> </o:p></p>
<p class="MsoPlainText">Why do you want to send IO to a device while you are doing a NVMe format ?<o:p></o:p></p>
<p class="MsoPlainText"><span style="color:black"><o:p> </o:p></span></p>
<p class="MsoPlainText" style="margin-left:.5in;text-indent:-.25in;mso-list:l1 level1 lfo2">
<![if !supportLists]><span style="color:red"><span style="mso-list:Ignore">1)<span style="font:7.0pt "Times New Roman"">
</span></span></span><![endif]><span style="color:red">I don’t necessarily want to. What I am saying is that the driver should not be in the business of orchestrating special behavior that is inconsistent with other SCSI-to-NVMe translations, emulations, etc.
I am curious as to what the open-source Linux driver does. Also I am not necessarily saying IOs need go thru all the way to the device. Since our driver knows when a Format NVM op is in prog, it could easily cook the sense data to be returned. Either way
we’d be closer to “normal” SCSI dev behavior.<o:p></o:p></span></p>
<p class="MsoPlainText"><span style="color:black"><o:p> </o:p></span></p>
<p class="MsoPlainText"><o:p> </o:p></p>
<p class="MsoPlainText">Since a NVMe format may change the sector size and all data are gone, a NVMe format should behave as if a drive had been removed and a new drive was added.<o:p></o:p></p>
<p class="MsoPlainText"><o:p> </o:p></p>
<p class="MsoPlainText"><span style="color:red">It can accomplish that by signaling a Bus Change at the end of the Format Op.</span><span style="color:black">
<o:p></o:p></span></p>
<p class="MsoPlainText">I think the simplest procedure to do a NVMe format is to:<o:p></o:p></p>
<p class="MsoPlainText">- offline the LUN (or remove the LUN from the system)<o:p></o:p></p>
<p class="MsoPlainText">- NVMe format <o:p></o:p></p>
<p class="MsoPlainText">- online the LUN<o:p></o:p></p>
<p class="MsoPlainText"><o:p> </o:p></p>
<p class="MsoPlainText">Do you see any problem with this simple approach ?<o:p></o:p></p>
<p class="MsoPlainText"><span style="color:black"><o:p> </o:p></span></p>
<p class="MsoPlainText" style="margin-left:.5in;text-indent:-.25in;mso-list:l0 level1 lfo3">
<![if !supportLists]><span style="color:red"><span style="mso-list:Ignore">1.<span style="font:7.0pt "Times New Roman"">
</span></span></span><![endif]><span style="color:red">The offlining is overkill – returning NOT READY while format is in prog and Bus Change Detected at end of format op will accomplish what is desired.<o:p></o:p></span></p>
<p class="MsoPlainText" style="margin-left:.5in;text-indent:-.25in;mso-list:l0 level1 lfo3">
<![if !supportLists]><span style="color:red"><span style="mso-list:Ignore">2.<span style="font:7.0pt "Times New Roman"">
</span></span></span><![endif]><span style="color:red">The offlining is <u>not taking</u> place before the NVM format is launched in the current driver – see the originally-attached thread. So the current driver is broken any way we look at it – it’s just a
matter of the easiest way to fix it imo.<o:p></o:p></span></p>
<p class="MsoPlainText"><span style="color:black"><o:p> </o:p></span></p>
<p class="MsoPlainText"><span style="color:black"><o:p> </o:p></span></p>
<p class="MsoPlainText">Have you a use case that this procedure does not work ?<o:p></o:p></p>
<p class="MsoPlainText"><o:p> </o:p></p>
<p class="MsoPlainText">Thanks<o:p></o:p></p>
<p class="MsoPlainText"><o:p> </o:p></p>
<p class="MsoPlainText">-Kwok<o:p></o:p></p>
<p class="MsoPlainText"><o:p> </o:p></p>
<p class="MsoPlainText"><o:p> </o:p></p>
<p class="MsoPlainText">----------------------------------------------------------------------<o:p></o:p></p>
<p class="MsoPlainText"><o:p> </o:p></p>
<p class="MsoPlainText">Message: 1<o:p></o:p></p>
<p class="MsoPlainText">Date: Tue, 7 Jan 2014 05:15:24 +0000<o:p></o:p></p>
<p class="MsoPlainText">From: "Mayes, Barrett N" <<a href="mailto:barrett.n.mayes@intel.com"><span style="color:windowtext;text-decoration:none">barrett.n.mayes@intel.com</span></a>><o:p></o:p></p>
<p class="MsoPlainText">To: Judy Brock-SSI <<a href="mailto:judy.brock@ssi.samsung.com"><span style="color:windowtext;text-decoration:none">judy.brock@ssi.samsung.com</span></a>>,<o:p></o:p></p>
<p class="MsoPlainText"> "<a href="mailto:nvmewin@lists.openfabrics.org"><span style="color:windowtext;text-decoration:none">nvmewin@lists.openfabrics.org</span></a>" <<a href="mailto:nvmewin@lists.openfabrics.org"><span style="color:windowtext;text-decoration:none">nvmewin@lists.openfabrics.org</span></a>><o:p></o:p></p>
<p class="MsoPlainText">Subject: Re: [nvmewin] Handling IO when Format NVM op is in progress<o:p></o:p></p>
<p class="MsoPlainText">Message-ID:<o:p></o:p></p>
<p class="MsoPlainText"> <<a href="mailto:B8DB29EA17EB2742AC3405C31A9B9EE9609824DF@ORSMSX106.amr.corp.intel.com"><span style="color:windowtext;text-decoration:none">B8DB29EA17EB2742AC3405C31A9B9EE9609824DF@ORSMSX106.amr.corp.intel.com</span></a>><o:p></o:p></p>
<p class="MsoPlainText"> <o:p></o:p></p>
<p class="MsoPlainText">Content-Type: text/plain; charset="us-ascii"<o:p></o:p></p>
<p class="MsoPlainText"><o:p> </o:p></p>
<p class="MsoPlainText">What problem do you want to solve by keeping the block device around during a format and allowing IO through so it can be failed with check condition?<o:p></o:p></p>
<p class="MsoPlainText"><o:p> </o:p></p>
<p class="MsoPlainText">Namespace not ready can't generically be translated to SCSI check condition/not ready/Format In Progress. The driver would need to know that a format command is outstanding so it could translate that correctly (for 1.0-based device
support) since the namespace could be not ready for reasons other than a format in progress. If the driver already has to know a format is in progress, it could just fail commands without sending it to the device (so no need for the new failure code). But
if that's the case, why _not_ hide the LUN until the format is complete. By hiding the LUN and bringing it back when the format is complete, you don't have to worry about handling IO and you also take care of the re-enumeration that has to happen when the
format is complete (in the case of changing the LBA Format).<o:p></o:p></p>
<p class="MsoPlainText"><o:p> </o:p></p>
<p class="MsoPlainText">-Barrett<o:p></o:p></p>
<p class="MsoPlainText"><o:p> </o:p></p>
<p class="MsoPlainText">From: <a href="mailto:nvmewin-bounces@lists.openfabrics.org">
<span style="color:windowtext;text-decoration:none">nvmewin-bounces@lists.openfabrics.org</span></a> [<a href="mailto:nvmewin-bounces@lists.openfabrics.org"><span style="color:windowtext;text-decoration:none">mailto:nvmewin-bounces@lists.openfabrics.org</span></a>]
On Behalf Of Judy Brock-SSI<o:p></o:p></p>
<p class="MsoPlainText">Sent: Monday, January 06, 2014 8:09 PM<o:p></o:p></p>
<p class="MsoPlainText">To: <a href="mailto:nvmewin@lists.openfabrics.org"><span style="color:windowtext;text-decoration:none">nvmewin@lists.openfabrics.org</span></a><o:p></o:p></p>
<p class="MsoPlainText">Subject: [nvmewin] Handling IO when Format NVM op is in progress<o:p></o:p></p>
<p class="MsoPlainText"><o:p> </o:p></p>
<p class="MsoPlainText">All,<o:p></o:p></p>
<p class="MsoPlainText"><o:p> </o:p></p>
<p class="MsoPlainText">Many months ago, I initiated a thread (see attached) in which I argued that there were some holes in the current implementation of Format NVM ioctl passthru and in which I advocated for, among other things, the addition of logic to
make sure pseudo-SCSI bus re-enumeration had fully taken place in the driver (such that Storport was notified that no "luns" were present) before the actual Format NVM op was launched.<o:p></o:p></p>
<p class="MsoPlainText"><o:p> </o:p></p>
<p class="MsoPlainText">I intuitively understand - and up to this point have unquestioningly agreed with - the basic assumption that the reason the namespaces must be removed/ "luns" disappeared prior to formatting is because "we cannot format a namespace
while the OS is aware of its presence and could be potentially sending I/O to a stale namespace config (i.e. changing LBA/sector size)." (excerpt from attached thread).<o:p></o:p></p>
<p class="MsoPlainText"><o:p> </o:p></p>
<p class="MsoPlainText">The question has recently arisen in internal discussion however as to whether or not we really have to do this. It was pointed out that SCSI devices (real ones) are capable of receiving IO commands while SCSI format commands are in process.
They will return the following error:<o:p></o:p></p>
<p class="MsoPlainText"><o:p> </o:p></p>
<p class="MsoPlainText"> SCSI status = Check condition, sense code = NOT READY (0x2), additional sense code = LUN NOT READY (0x4), additional sense code qualifier = FORMAT IN PROGRESS (0x4)<o:p></o:p></p>
<p class="MsoPlainText"><o:p> </o:p></p>
<p class="MsoPlainText">Why then, instead of removing namespaces/luns, can our Windows driver not return the same error status a real SCSI drive would return in such a situation? One would assume that upper layers of the storage stack have plenty years of experience
in knowing what to do when it sees that error.<o:p></o:p></p>
<p class="MsoPlainText"><o:p> </o:p></p>
<p class="MsoPlainText">As a point of comparison, there is no standard I am aware of which specifies that Storport miniports which support real SCSI devices, if they happen to provide a proprietary pass-thru to allow a SCSI format command to go through to a
device, must cause all LUNS to appear offline prior to formatting,<o:p></o:p></p>
<p class="MsoPlainText"><o:p> </o:p></p>
<p class="MsoPlainText">One could even argue (and they have!) that these IO commands could even be allowed to go through to the NVMe device itself (as in the real SCSI case); NVMe 1.1 Technical Proposal 005 has defined a new format-in-progress status code that
NVMe firmware will be able to return at some point in the future, current firmware could easily return NAMESPACE_NOT_READY and driver could translate to the above SCSI sense data, etc.<o:p></o:p></p>
<p class="MsoPlainText"><o:p> </o:p></p>
<p class="MsoPlainText">So ... here I stand, devil's advocate hat in hand, hoping to find out:<o:p></o:p></p>
<p class="MsoPlainText"><o:p> </o:p></p>
<p class="MsoPlainText"><o:p> </o:p></p>
<p class="MsoPlainText">a) what the "back story" is on how this decision was ultimately made (the attached thread said a lot of discussion took place on the subject)<o:p></o:p></p>
<p class="MsoPlainText"><o:p> </o:p></p>
<p class="MsoPlainText">b) whether or not the diametrically-opposed alternative I am discussing above was thoroughly considered and if so, why it was rejected<o:p></o:p></p>
<p class="MsoPlainText"><o:p> </o:p></p>
<p class="MsoPlainText">c) whether the topic bears reconsidering at this point.<o:p></o:p></p>
<p class="MsoPlainText"><o:p> </o:p></p>
<p class="MsoPlainText">Thanks in advance for your collective consideration,<o:p></o:p></p>
<p class="MsoPlainText"><o:p> </o:p></p>
<p class="MsoPlainText">Judy<o:p></o:p></p>
<p class="MsoPlainText"><o:p> </o:p></p>
<p class="MsoPlainText">_______________________________________________<o:p></o:p></p>
<p class="MsoPlainText">nvmewin mailing list<o:p></o:p></p>
<p class="MsoPlainText"><a href="mailto:nvmewin@lists.openfabrics.org"><span style="color:windowtext;text-decoration:none">nvmewin@lists.openfabrics.org</span></a><o:p></o:p></p>
<p class="MsoPlainText"><a href="http://lists.openfabrics.org/cgi-bin/mailman/listinfo/nvmewin"><span style="color:windowtext;text-decoration:none">http://lists.openfabrics.org/cgi-bin/mailman/listinfo/nvmewin</span></a><o:p></o:p></p>
</div>
</body>
</html>