<html>
<head>
<meta http-equiv="Content-Type" content="text/html; charset=utf-8">
<meta name="Generator" content="Microsoft Exchange Server">
<!-- converted from rtf -->
<style><!-- .EmailQuote { margin-left: 1pt; padding-left: 4pt; border-left: #800000 2px solid; } --></style>
</head>
<body>
<font face="Calibri" size="2"><span style="font-size:11pt;">
<div style="margin-top:5pt;margin-bottom:5pt;"><font color="#1F497D">Hi Ray,</font></div>
<div style="margin-top:5pt;margin-bottom:5pt;"><font face="Times New Roman" size="2" color="#1F497D"><span style="font-size:10pt;"> [Ray wrote] Let me know if this answers your question.</span></font></div>
<div style="margin-top:5pt;margin-bottom:5pt;"><font face="Times New Roman" size="2" color="#1F497D"><span style="font-size:10pt;">I don’t think it does. What I wrote below I think was pretty much the same as what you wrote - or at least that was my intention
<font face="Wingdings">J</font>. </span></font></div>
<div style="margin-top:5pt;margin-bottom:5pt;"><font face="Times New Roman" size="2" color="#1F497D"><span style="font-size:10pt;">However, the piece I couldn’t explain (cause I haven’t looked into it) is how the driver holds off the beginning of the actual
format NVM operation till whatever old IOs that were already in progress for the namespace(s) before the format op request was received are completed back to the caller, aborted, or whatever - so there are no old live requests hanging around, still in the
driver , before the format op begins.</span></font></div>
<div style="margin-top:5pt;margin-bottom:5pt;"><font face="Times New Roman" size="2" color="#1F497D"><span style="font-size:10pt;">In other words, does the driver hold off starting the format cmd till the outstanding IOs are completed? Or do we perhaps just
drop them on the floor and let the OS figure out that those requests are permanently lost/gone due to the LUNs having disappeared (my guess is, the latter is what we do)? Or do we try to abort them all? And so on.</span></font></div>
<div style="margin-top:5pt;margin-bottom:5pt;"><font face="Times New Roman" size="2" color="#1F497D"><span style="font-size:10pt;">So again, we do understand how to get the OS to avoid sending new I/O requests to stale namespaces but how exactly are the old
I/O reqs (those existing at the time the format request comes in ) handled?</span></font></div>
<div style="margin-top:5pt;margin-bottom:5pt;"><font face="Times New Roman" size="2" color="#1F497D"><span style="font-size:10pt;">At least that is my current question <font face="Wingdings">J</font></span></font></div>
<div style="margin-top:5pt;margin-bottom:5pt;"><font face="Times New Roman" size="2" color="#1F497D"><span style="font-size:10pt;">Thanks,</span></font></div>
<div style="margin-top:5pt;margin-bottom:5pt;"><font face="Times New Roman" size="2" color="#1F497D"><span style="font-size:10pt;">Judy</span></font></div>
<div style="margin-top:5pt;margin-bottom:5pt;"><font face="Times New Roman" size="3" color="#1F497D"><span style="font-size:12pt;"> </span></font></div>
<div style="margin-top:5pt;margin-bottom:5pt;"><font face="Times New Roman" size="3" color="#1F497D"><span style="font-size:12pt;"> </span></font></div>
<div><font face="Tahoma" size="2"><span style="font-size:10pt;"><b>From:</b> Robles, Raymond C [<a href="mailto:raymond.c.robles@intel.com">mailto:raymond.c.robles@intel.com</a>]
<br>
<b>Sent:</b> Thursday, April 18, 2013 2:46 PM<br>
<b>To:</b> Judy Brock-SSI; WONMOON CHEON; ???; nvmewin@lists.openfabrics.org<br>
<b>Subject:</b> RE: RE: Handling pending commands when processing Format [changing from NVMe WG dist. list to OFA NVMe Windows Driver dist. list]</span></font></div>
<div style="margin-top:5pt;margin-bottom:5pt;"><font face="Times New Roman" size="3"><span style="font-size:12pt;"> </span></font></div>
<div style="margin-top:5pt;margin-bottom:5pt;"><font face="Times New Roman" size="2" color="#1F497D"><span style="font-size:10pt;">Hello Judy/Wonmoon,</span></font></div>
<div style="margin-top:5pt;margin-bottom:5pt;"><font face="Times New Roman" size="2" color="#1F497D"><span style="font-size:10pt;">Sorry for the late response. The “hot remove” state is just a state that we enter when the driver receives a Format command.
Basically, this state will remove the namespace(s) from the topology by calling StorPortNotification() with BusChangeDetected. This will remove the “SCSI target/disk” associated with each namespace form the OS (because Storport will re-enumerate the controller
and the driver will not expose the namespaces about to be formatted) so that the format can occur on the relevant namespaces.</span></font></div>
<div style="margin-top:5pt;margin-bottom:5pt;"><font face="Times New Roman" size="2" color="#1F497D"><span style="font-size:10pt;">By signaling Windows that the namespaces have been “removed”, all I/O will be stopped by the OS. Then the format can complete.
Once the format is complete, we perform the opposite action to “hot add” the namespace back into the topology by calling StorPortNotification() with BusChangeDetected… only this time, we will surface the namespace(s) again when Storport re-enumerates.</span></font></div>
<div style="margin-top:5pt;margin-bottom:5pt;"><font face="Times New Roman" size="2" color="#1F497D"><span style="font-size:10pt;">Not queues are deleted in this state, no memory is de-allocated, and nothing else changes about the namespace. This is simply
the first step (in a 3 step sequence) when formatting a namespace as we cannot format a namespace while the OS is aware of its presence and could be potentially sending I/O to a stale namespace config (i.e. changing LBA/sector size).</span></font></div>
<div style="margin-top:5pt;margin-bottom:5pt;"><font face="Times New Roman" size="2" color="#1F497D"><span style="font-size:10pt;">Let me know if this answers your question.</span></font></div>
<div style="margin-top:5pt;margin-bottom:5pt;"><font face="Times New Roman" size="2" color="#1F497D"><span style="font-size:10pt;">Thanks,</span></font></div>
<div style="margin-top:5pt;margin-bottom:5pt;"><font face="Times New Roman" size="2" color="#1F497D"><span style="font-size:10pt;">Ray</span></font></div>
<div style="margin-top:5pt;margin-bottom:5pt;"><font face="Times New Roman" size="2" color="#1F497D"><span style="font-size:10pt;"> </span></font></div>
<div><font face="Tahoma" size="2"><span style="font-size:10pt;"><b>From:</b> Judy Brock-SSI [<a href="mailto:judy.brock@ssi.samsung.com"><font color="blue"><u>mailto:judy.brock@ssi.samsung.com</u></font></a>]
<br>
<b>Sent:</b> Wednesday, April 17, 2013 6:21 AM<br>
<b>To:</b> WONMOON CHEON; <font face="Gulim">강미경</font>; <a href="mailto:technical@nvmexpress.org"><font color="blue"><u>technical@nvmexpress.org</u></font></a><br>
<b>Subject:</b> RE: RE: Handling pending commands when processing Format</span></font></div>
<div style="margin-top:5pt;margin-bottom:5pt;"><font face="Times New Roman" size="3"><span style="font-size:12pt;"> </span></font></div>
<div style="margin-top:3pt;margin-bottom:3pt;"> >>Would you elaborate more about the "hot-remove" state? In this state, do you mean that all the IO SQ/CQs are deleted? Or, waiting for completions of all the outstanding IOs?</div>
<div style="margin-top:5pt;margin-bottom:5pt;"><font color="#1F497D">The IO SQ/CQs are definitely NOT deleted. I would need to look more closely through the driver code to see how IOs previously sent to the namespaces which are marked as OFFLINE are handled/finished/quiesced.
</font></div>
<div style="margin-top:5pt;margin-bottom:5pt;"><font color="#1F497D">There are other folks on this thread who no doubt have more history/intimate knowledge of this driver than I do who may answer that question more quickly than I can…also, perhaps this discussion
should probably be moved to the OFA driver forum since it has turned into a driver-specific thread at this point. </font></div>
<div style="margin-top:5pt;margin-bottom:5pt;"><font color="#1F497D">What do folks think?</font></div>
<div style="margin-top:5pt;margin-bottom:5pt;"><font color="#1F497D">Judy</font></div>
<div><font face="Tahoma" size="2"><span style="font-size:10pt;"><b>From:</b> <font face="Gulim">천원문</font> [<a href="mailto:wm.cheon@samsung.com"><font color="blue"><u>mailto:wm.cheon@samsung.com</u></font></a>]
<br>
<b>Sent:</b> Wednesday, April 17, 2013 1:20 AM<br>
<b>To:</b> Judy Brock-SSI; <font face="Gulim">강미경</font>; <a href="mailto:technical@nvmexpress.org"><font color="blue"><u>technical@nvmexpress.org</u></font></a><br>
<b>Subject:</b> Re: RE: Handling pending commands when processing Format</span></font></div>
<div style="margin-top:5pt;margin-bottom:5pt;"><font face="Times New Roman" size="3"><span style="font-size:12pt;"> </span></font></div>
<div style="margin-top:3pt;margin-bottom:3pt;">Hi Judy,</div>
<div style="margin-top:3pt;margin-bottom:3pt;"><font face="Arial" size="2"><span style="font-size:9pt;"> </span></font></div>
<div style="margin-top:3pt;margin-bottom:3pt;">Would you elaborate more about the "hot-remove" state? In this state, do you mean that all the IO SQ/CQs are deleted? Or, waiting for completions of all the outstanding IOs?</div>
<div style="margin-top:3pt;margin-bottom:3pt;"><font face="Arial" size="2"><span style="font-size:9pt;"> </span></font></div>
<div style="margin-top:3pt;margin-bottom:3pt;">Thanks,</div>
<div style="margin-top:3pt;margin-bottom:3pt;">Wonmoon</div>
<div style="margin-top:3pt;margin-bottom:3pt;"><font face="Arial" size="2"><span style="font-size:9pt;"> </span></font></div>
<div style="margin-top:3pt;margin-bottom:3pt;"><font face="Arial" size="2"><span style="font-size:9pt;">------- <b>Original Message</b> -------</span></font></div>
<div style="margin-top:3pt;margin-bottom:3pt;"><font face="Arial" size="2"><span style="font-size:9pt;"><b>Sender</b> : Judy Brock-SSI<<a href="mailto:judy.brock@ssi.samsung.com"><font color="blue"><u>judy.brock@ssi.samsung.com</u></font></a>></span></font></div>
<div style="margin-top:3pt;margin-bottom:3pt;"><font face="Arial" size="2"><span style="font-size:9pt;"><b>Date</b> : 2013-04-17 16:39 (GMT+09:00)</span></font></div>
<div style="margin-top:3pt;margin-bottom:3pt;"><font face="Arial" size="2"><span style="font-size:9pt;"><b>Title</b> : RE: Handling pending commands when processing Format</span></font></div>
<div style="margin-top:3pt;margin-bottom:3pt;"><font face="Arial" size="2"><span style="font-size:9pt;"> </span></font></div>
<div style="margin-top:5pt;margin-bottom:5pt;"><font color="#1F497D">Hi,</font></div>
<div style="margin-top:5pt;margin-bottom:5pt;"><font color="#1F497D">I should clarify that it is not the Windows operating system – but rather the Windows OFA NVMe driver - that, from what I can see, does a “hot-remove” of all namespace(s) associated with
a device before allowing a format operation to begin; “hot remove” is just the name for an internal state in the driver format nvm state machine. </font></div>
<div style="margin-top:5pt;margin-bottom:5pt;"><font color="#1F497D">Before beginning the actual format op, the driver internally marks all namespaces associated with the format operation “offline”. It then notifies the OS that there has been a “bus change”
event (via an OS-specific API).</font></div>
<div style="margin-top:5pt;margin-bottom:5pt;"><font color="#1F497D">This in turn will cause the OS to rescan (re-enumerate) the “bus” (the pseudo SCSI bus, that is – we expose NVM namespaces as SCSI luns).</font></div>
<div style="margin-top:5pt;margin-bottom:5pt;"><font color="#1F497D">Since all the pertinent namespaces have been marked offline internally, the bus rescan won’t detect any valid SCSI luns (because the driver will not report any).</font></div>
<div style="margin-top:5pt;margin-bottom:5pt;"><font color="#1F497D">Hence from the OS point of view, any SCSI lun(s) previously mapped to the namespace(s) to be formatted will have disappeared/will be unaddressable while the format operation is in progress.</font></div>
<div style="margin-top:5pt;margin-bottom:5pt;"><font color="#1F497D">Judy</font></div>
<div style="margin-top:5pt;margin-bottom:5pt;"><font color="#1F497D"> </font></div>
<div><font face="Tahoma" size="2"><span style="font-size:10pt;"><b>From:</b> Judy Brock-SSI
<br>
<b>Sent:</b> Tuesday, April 16, 2013 8:22 PM<br>
<b>To:</b> 'mkkang.kang@samsung.com'; <a href="mailto:technical@nvmexpress.org"><font color="blue"><u>technical@nvmexpress.org</u></font></a><br>
<b>Subject:</b> RE: Handling pending commands when processing Format</span></font></div>
<div style="margin-top:5pt;margin-bottom:5pt;"><font face="Times New Roman" size="3"><span style="font-size:12pt;"> </span></font></div>
<div style="margin-top:5pt;margin-bottom:5pt;"><font color="#1F497D">Mikyeong,</font></div>
<div style="margin-top:5pt;margin-bottom:5pt;"><font color="#1F497D">I haven’t looked at the Linux driver but I know that Windows hot-removes all namespace(s) associated with a device before allowing a format operation to begin. And a namespace can’t be removed
while there is IO outstanding to it so that answers your question regarding IOs being completed before format begins. It also answers the question about requests being sent to a namespace while format is in progress – can’t happen.</font></div>
<div style="margin-top:5pt;margin-bottom:5pt;"><font color="#1F497D">Thanks,</font></div>
<div style="margin-top:5pt;margin-bottom:5pt;"><font color="#1F497D">Judy</font></div>
<div style="margin-top:5pt;margin-bottom:5pt;"><font color="#1F497D"> </font></div>
<div style="margin-top:5pt;margin-bottom:5pt;"><font color="#1F497D"> </font></div>
<div style="margin-top:5pt;margin-bottom:5pt;"><font color="#1F497D"> </font></div>
<div><font face="Tahoma" size="2"><span style="font-size:10pt;"><b>From:</b> <font face="Gulim">강미경</font> [<a href="mailto:mkkang.kang@samsung.com"><font color="blue"><u>mailto:mkkang.kang@samsung.com</u></font></a>]
<br>
<b>Sent:</b> Tuesday, April 16, 2013 7:07 PM<br>
<b>To:</b> <a href="mailto:technical@nvmexpress.org"><font color="blue"><u>technical@nvmexpress.org</u></font></a><br>
<b>Subject:</b> Handling pending commands when processing Format</span></font></div>
<div style="margin-top:5pt;margin-bottom:5pt;"><font face="Times New Roman" size="3"><span style="font-size:12pt;"> </span></font></div>
<div style="margin-top:3pt;margin-bottom:3pt;">Dear All,</div>
<div style="margin-top:3pt;margin-bottom:3pt;"><font face="Arial" size="2"><span style="font-size:9pt;"> </span></font></div>
<div style="margin-top:3pt;margin-bottom:3pt;">Format NVM command may change the Namespace repository, and it will be executed out of order like any other commands. Therefore, Format NVM command may affect other commands that are pending execution in the device,
if any. </div>
<div style="margin-top:3pt;margin-bottom:3pt;"><font face="Arial" size="2"><span style="font-size:9pt;"> </span></font></div>
<div style="margin-top:3pt;margin-bottom:3pt;">1) How does an OFA/linux driver handle 'Format NVM command'? Does a host make sure that all commands for a particular NSID are completed before sending 'Format NVM command'?</div>
<div style="margin-top:3pt;margin-bottom:3pt;"><font face="Arial" size="2"><span style="font-size:9pt;"> </span></font></div>
<div style="margin-top:3pt;margin-bottom:3pt;">2) If a host driver does not behave like 1) above, how can a device handle other pending commands which were previously submitted in a SQ? It seems like we need an additional status code. e.g. Abort due to Namespace
Format</div>
<div style="margin-top:3pt;margin-bottom:3pt;"><font face="Arial" size="2"><span style="font-size:9pt;"> </span></font></div>
<div style="margin-top:3pt;margin-bottom:3pt;">3) Let's suppose that 'Format NVM command' is in progress. If the host driver sends subsequent commands to the namespace being formatted, should the device reply directly with a 'Namespace not Ready'?</div>
<div style="margin-top:3pt;margin-bottom:3pt;"><font face="Arial" size="2"><span style="font-size:9pt;"> </span></font></div>
<div style="margin-top:3pt;margin-bottom:3pt;">[1.0e spec] If the device does not reply directly and the format operation takes long time, then, I/O command will timeout and the host may send the reset. But if commands are responded with 'Namespace Not Ready',
host may not issue the reset. Therefore, direct reply seems to be needed. </div>
<div style="margin-top:3pt;margin-bottom:3pt;"><font face="Arial" size="2"><span style="font-size:9pt;"> </span></font></div>
<div style="margin-top:3pt;margin-bottom:3pt;">[1.1 spec. ECN 001] There is Format progress indicator. The host driver can check format progress any time, therefore, there is no concern about reset during format command.</div>
<div style="margin-top:3pt;margin-bottom:3pt;"><font face="Arial" size="2"><span style="font-size:9pt;"> </span></font></div>
<div style="margin-top:3pt;margin-bottom:3pt;"><font face="Arial" size="2"><span style="font-size:9pt;"> </span></font></div>
<div style="margin-top:3pt;margin-bottom:3pt;">Best Regards,</div>
<div style="margin-top:3pt;margin-bottom:3pt;">Mikyeong Kang</div>
<div style="margin-top:3pt;margin-bottom:3pt;"><font face="Arial" size="2"><span style="font-size:9pt;"> </span></font></div>
<div><font size="2"><span style="font-size:10pt;"><img width="83" height="41" src="rtfimage://"></span></font></div>
<div style="margin-top:3pt;margin-bottom:3pt;"><font size="2"><span style="font-size:10pt;"><b>Kang MiKyeong</b></span></font></div>
<div style="margin-top:3pt;margin-bottom:3pt;"><font face="Arial" size="2"><span style="font-size:9pt;"> </span></font></div>
<div style="margin-top:5pt;margin-bottom:5pt;"><font size="2"><span style="font-size:10pt;">Flash Memory Planning/Enabling Group, Memory Div.</span></font></div>
<div style="margin-top:5pt;margin-bottom:5pt;"><font size="2"><span style="font-size:10pt;">SAMSUNG ELECTRONICS, Co., Ltd..</span></font></div>
<div style="margin-top:5pt;margin-bottom:5pt;"><font size="2"><span style="font-size:10pt;">Phone: 82-31-208-3857</span></font></div>
<div style="margin-top:5pt;margin-bottom:5pt;"><font size="2"><span style="font-size:10pt;">Mobile: 82-10-9369-0177</span></font></div>
<div style="margin-top:5pt;margin-bottom:5pt;"><font size="2"><span style="font-size:10pt;">E-mail: <a href="mailto:mkkang.kang@samsung.com"><font color="blue"><u>mkkang.kang@samsung.com</u></font></a></span></font></div>
<div><font size="2"><span style="font-size:10pt;"><img width="83" height="41" src="rtfimage://"></span></font></div>
<div style="margin-top:3pt;margin-bottom:3pt;"><font face="Arial" size="2"><span style="font-size:9pt;"> </span></font></div>
<div style="margin-top:3pt;margin-bottom:3pt;"><font face="Arial" size="2"><span style="font-size:9pt;"> </span></font></div>
<div style="margin-top:3pt;margin-bottom:3pt;"><font face="Arial" size="2"><span style="font-size:9pt;"> </span></font></div>
<table width="658" style="width:395.1pt;margin-left:2.85pt;">
<col width="658" style="width:395.1pt;">
<tr>
<td style="margin-top:3pt;margin-bottom:3pt;"><font face="Arial" size="2"><span style="font-size:9pt;"><img src="cid:8F5983B92F8B7845A5CAF6DF730AE942@ssi.samsung.com"> </span></font></td>
</tr>
</table>
<div><font face="Times New Roman" size="3"><span style="font-size:12pt;"><img src="http://ext.samsung.net/mailcheck/SeenTimeChecker?do=dd90ab357fccaa0879ef49d135999623709a3a8db170c13642590acae5bdcb6a1b41a3ea5b94b2ed546e5f4bdee86ec72cb0372888719872db9fdddda33e82cbe4a391424e62fcf6cf878f9a26ce15a0"></span></font></div>
<div><font face="Times New Roman" size="3"><span style="font-size:12pt;"><img src="http://ext.samsung.net/mailcheck/SeenTimeChecker?do=dd90ab357fccaa0810067d8e81f72583c0a10306535965788c7e8eca81cb564b226b8d944a9d91d2f73afb615c53a27ef4bcdeced46ed5ee08cece8541bc14eacf878f9a26ce15a0"></span></font></div>
<div><font face="Times New Roman" size="3"><span style="font-size:12pt;"> </span></font></div>
</span></font>
</body>
</html>