From suman.p at samsung.com  Mon May  2 04:05:50 2016
From: suman.p at samsung.com (SUMAN PRAKASH B)
Date: Mon, 02 May 2016 11:05:50 +0000 (GMT)
Subject: [nvmewin] Patch with changes for disk Read only support
Message-ID: <AE.61.04928.E8437275@epcpsbgx1.samsung.com>

An HTML attachment was scrubbed...
URL: <http://lists.openfabrics.org/pipermail/nvmewin/attachments/20160502/08b87125/attachment.html>
-------------- next part --------------
A non-text attachment was scrubbed...
Name: 201605021636142_Z5JE7EUA.jpg
Type: image/jpeg
Size: 2934 bytes
Desc: not available
URL: <http://lists.openfabrics.org/pipermail/nvmewin/attachments/20160502/08b87125/attachment.jpg>
-------------- next part --------------
A non-text attachment was scrubbed...
Name: 201605021636147_LK7CT9SZ.jpg
Type: image/jpeg
Size: 2934 bytes
Desc: not available
URL: <http://lists.openfabrics.org/pipermail/nvmewin/attachments/20160502/08b87125/attachment-0001.jpg>
-------------- next part --------------
A non-text attachment was scrubbed...
Name: 201605021636152_BSL8PYMC.jpg
Type: image/jpeg
Size: 2938 bytes
Desc: not available
URL: <http://lists.openfabrics.org/pipermail/nvmewin/attachments/20160502/08b87125/attachment-0002.jpg>
-------------- next part --------------
A non-text attachment was scrubbed...
Name: 201605021636162_4XEV4D4T.jpg
Type: image/jpeg
Size: 823 bytes
Desc: not available
URL: <http://lists.openfabrics.org/pipermail/nvmewin/attachments/20160502/08b87125/attachment-0003.jpg>
-------------- next part --------------
A non-text attachment was scrubbed...
Name: 201605021636167_UIXHLLKJ.gif
Type: image/gif
Size: 13168 bytes
Desc: not available
URL: <http://lists.openfabrics.org/pipermail/nvmewin/attachments/20160502/08b87125/attachment.gif>

From raymond.c.robles at intel.com  Tue May  3 10:53:10 2016
From: raymond.c.robles at intel.com (Robles, Raymond C)
Date: Tue, 3 May 2016 17:53:10 +0000
Subject: [nvmewin] Patch with changes for disk Read only support
In-Reply-To: <BLUPR04MB673757B3D56ED8CEA8E64C7E9660@BLUPR04MB673.namprd04.prod.outlook.com>
References: <BE.71.05093.B4C33275@epcpsbgx4.samsung.com>
 <49158E750348AA499168FD41D88983607C515EF3@fmsmsx117.amr.corp.intel.com>
 <BLUPR04MB673757B3D56ED8CEA8E64C7E9660@BLUPR04MB673.namprd04.prod.outlook.com>
Message-ID: <49158E750348AA499168FD41D88983607C519FF1@fmsmsx117.amr.corp.intel.com>

Thanks Tom… and Suman. I’ve pushed the patch.
Suman, could you please double check the trunk to ensure your patch was applied properly? I got an error while applying the patch, and I want to make sure your patch got applied properly.
Thanks… Ray
From: Thomas Freeman [mailto:thomas.freeman at hgst.com]
Sent: Friday, April 29, 2016 10:52 AM
To: Robles, Raymond C <raymond.c.robles at intel.com>; suman.p at samsung.com; nvmewin at lists.openfabrics.org
Cc: PRAKASH BABU VEMULA <prakash.v at samsung.com>; ANSHUL SHARMA <anshul at samsung.com>; MANOJ THAPLIYAL <m.thapliyal at samsung.com>
Subject: RE: [nvmewin] Patch with changes for disk Read only support

I accept the change.
Tom Freeman
Software Engineer, Device Manager and Driver Development
Western Digital Corporation
e.  Thomas.freeman at hgst.com<mailto:Thomas.freeman at hgst.com>
o.  +1-507-322-2311

[cid:image001.jpg at 01D1A529.F9DB5C40]

From: nvmewin [mailto:nvmewin-bounces at lists.openfabrics.org] On Behalf Of Robles, Raymond C
Sent: Friday, April 29, 2016 12:47 PM
To: suman.p at samsung.com<mailto:suman.p at samsung.com>; nvmewin at lists.openfabrics.org<mailto:nvmewin at lists.openfabrics.org>
Cc: PRAKASH BABU VEMULA <prakash.v at samsung.com<mailto:prakash.v at samsung.com>>; ANSHUL SHARMA <anshul at samsung.com<mailto:anshul at samsung.com>>; MANOJ THAPLIYAL <m.thapliyal at samsung.com<mailto:m.thapliyal at samsung.com>>
Subject: Re: [nvmewin] Patch with changes for disk Read only support

Hi Suman,
I was waiting to hear back from Tom (HGST) on your latest changes. Once I get confirmation, we can close out this patch.
Thanks… Ray
From: SUMAN PRAKASH B [mailto:suman.p at samsung.com]
Sent: Friday, April 29, 2016 3:50 AM
To: Robles, Raymond C <raymond.c.robles at intel.com<mailto:raymond.c.robles at intel.com>>; nvmewin at lists.openfabrics.org<mailto:nvmewin at lists.openfabrics.org>
Cc: PRAKASH BABU VEMULA <prakash.v at samsung.com<mailto:prakash.v at samsung.com>>; MANOJ THAPLIYAL <m.thapliyal at samsung.com<mailto:m.thapliyal at samsung.com>>; ANSHUL SHARMA <anshul at samsung.com<mailto:anshul at samsung.com>>
Subject: Re: Re: RE: [nvmewin] Patch with changes for disk Read only support


Hi Ray,



Since Intel and HGST have reviewed the code, and I have incorporated the review comments provided by HGST, can we close out this patch?

We will be ready with our next patch by next week.



Thanks,

Suman



------- Original Message -------

Sender : SUMAN PRAKASH B<suman.p at samsung.com<mailto:suman.p at samsung.com>> Senior Chief Engineer/SSIR-SSD Solutions/Samsung Electronics

Date : Apr 27, 2016 20:09 (GMT+05:30)

Title : Re: RE: [nvmewin] Patch with changes for disk Read only support



Hi All,



I am sending the updated patch incorporating feedback from Tom. The changes are listed below. The password is samsungnvme



1. In SntiCompletionCallbackRoutine(), if the NVMe command Get Log Page fails, (SCT != Generic_command_status || SC != Successful completion), the buffer pSrbExt->pDatBuffer is freed.



2. In SntiTranslateModeSenseResponse(), replaced

        Lun = pLunExt->namespaceId - 1;
        pDevExt->pLunExtensionTable[Lun]->IsNamespaceReadOnly = TRUE;

with

        pLunExt->IsNamespaceReadOnly = TRUE;



3. In SntiReturnAllModePages(), following code is removed, as the same copy occurs in SntiTranslateModeSenseResponse()

         if (GET_DATA_BUFFER(pSrb) != NULL) {
              StorPortCopyMemory((PVOID)GET_DATA_BUFFER(pSrb),
              (PVOID)(pSrbExt->modeSenseBuf), GET_DATA_LENGTH(pSrb));
         }



Please let me know if you have any questions.



Thanks,
Suman



------- Original Message -------

Sender : Foster, Carolyn D<carolyn.d.foster at intel.com<mailto:carolyn.d.foster at intel.com>>

Date : Apr 22, 2016 20:49 (GMT+05:30)

Title : RE: [nvmewin] Patch with changes for disk Read only support


Hi Suman, I don’t have any additional feedback for this patch.
Carolyn
From: nvmewin [mailto:nvmewin-bounces at lists.openfabrics.org] On Behalf Of Thomas Freeman
Sent: Thursday, April 21, 2016 9:18 AM
To: suman.p at samsung.com<mailto:suman.p at samsung.com>; nvmewin at lists.openfabrics.org<mailto:nvmewin at lists.openfabrics.org>
Subject: Re: [nvmewin] Patch with changes for disk Read only support

Hi Suman,
Thank you for the quick response. I agree with your comments.
Thank you,
Tom Freeman
Software Engineer, Device Manager and Driver Development
Western Digital Corporation
e.  Thomas.freeman at hgst.com<mailto:Thomas.freeman at hgst.com>
o.  +1-507-322-2311

[cid:image001.jpg at 01D1A529.F9DB5C40]

From: SUMAN PRAKASH B [mailto:suman.p at samsung.com]
Sent: Thursday, April 21, 2016 9:55 AM
To: Thomas Freeman <thomas.freeman at hgst.com<mailto:thomas.freeman at hgst.com>>; nvmewin at lists.openfabrics.org<mailto:nvmewin at lists.openfabrics.org>
Cc: anshul at samsung.com<mailto:anshul at samsung.com>; prakash.v at samsung.com<mailto:prakash.v at samsung.com>; MANOJ THAPLIYAL <m.thapliyal at samsung.com<mailto:m.thapliyal at samsung.com>>
Subject: Re: RE: [nvmewin] Patch with changes for disk Read only support


Hi Tom,



Thanks for the comments. Please find my replies below:



1. For mode sense with 0x3f, you set the WP in the response header. Shouldn WP also be set if any of those pages(0x8, 0xa, 0x1a, 0x1c) are individually requested?

[Suman] Below is our observations:

a. During driver install/device enable from dev manager/hot insert, the  mode pages 0x8 and 0x3f are invoked.

b. During online/offline of disk, only 0x3f is invoked.

So as per our understanding, during disk initialization, not all the mode pages will be called. But driver gets the mode page 0x3f every time during disk initialization. Also for Detection during run time, when driver returns SCSI_SENSE_DATA_PROTECT for sense data, driver gets the mode page 0x3f  consistently. So we feel, setting the WP bit for mode page 0x3f will suffice.



2. snti.c:8242

[Suman] Agreed. We will use pLunExt->IsNamespaceReadOnly = TRUE;



3. Along with the new member, IsNamespaceReadOnly, the nvme_lun_extension also has ReadOnly. It seems like the setting of WP should take into account the value of both members.

[Suman] The OFA driver supports only the first lba range type for a namespace, though spec supports 64 lba range types per NS. This has to be corrected first.

Also we have to decide if the disk should be exposed as Read Only if any of the LBA range type is read only or only if the LBA 0 is read only.

I feel this should be taken as a separate patch since this involves too many changes.



4. If the NVMe command Get Log Page fails, (SCT != Generic_command_status || SC != Successful completion), the buffer pSrbExt->pDatBuffer is not freed. This corresponds to the allocation at snti.c:6530.

[Suman] Agreed.



5. snti.c:6539: I think the following can be eliminated. The same copy occurs during SntiTranslateModeSenseResponse - snti.c:8272.

[Suman] Agreed.



Please let us know your opinion.



Regards,

Suman



------- Original Message -------

Sender : Thomas Freeman<thomas.freeman at hgst.com<mailto:thomas.freeman at hgst.com>>

Date : Apr 20, 2016 21:59 (GMT+05:30)

Title : RE: [nvmewin] Patch with changes for disk Read only support


Hi Suman,
After reviewing the code, I have a few questions/comments:


1.       For mode sense with 0x3f, you set the WP in the response header. Shouldn WP also be set if any of those pages(0x8, 0xa, 0x1a, 0x1c) are individually requested?


2. snti.c:8242
Lun = pLunExt->namespaceId - 1;
pDevExt->pLunExtensionTable[Lun]->IsNamespaceReadOnly = TRUE;

These 2 lines can be replaced with
pLunExt->IsNamespaceReadOnly = TRUE;

Also, the original code is not a reliable way to determine the LUN id.
Here is an example where there doesn't work. The device has attached namespaces 1,3 & 4 and Existing namespaces of 1, 2, 3 & 4. LUNs 0-3 will correspond to namespaces 1,3,4,2. For namespace 3, the calculation NSID-1=lun will incorrectly give you LUNid of 2.

3. Along with the new member, IsNamespaceReadOnly, the nvme_lun_extension also has ReadOnly. It seems like the setting of WP should take into account the value of both members.

4. If the NVMe command Get Log Page fails, (SCT != Generic_command_status || SC != Successful completion), the buffer pSrbExt->pDatBuffer is not freed. This corresponds to the allocation at snti.c:6530.

5. snti.c:6539: I think the following can be eliminated. The same copy occurs during SntiTranslateModeSenseResponse - snti.c:8272.
if (GET_DATA_BUFFER(pSrb) != NULL) {
            StorPortCopyMemory((PVOID)GET_DATA_BUFFER(pSrb),
                (PVOID)(pSrbExt->modeSenseBuf), GET_DATA_LENGTH(pSrb));
        }
Let me know if you have questions,
Tom Freeman
Software Engineer, Device Manager and Driver Development
Western Digital Corporation
e.  Thomas.freeman at hgst.com<mailto:Thomas.freeman at hgst.com>
o.  +1-507-322-2311

[cid:image002.jpg at 01D1A529.F9DB5C40]

From: nvmewin [mailto:nvmewin-bounces at lists.openfabrics.org] On Behalf Of SUMAN PRAKASH B
Sent: Tuesday, April 19, 2016 8:45 AM
To: nvmewin at lists.openfabrics.org<mailto:nvmewin at lists.openfabrics.org>
Cc: sukka.kim at samsung.com<mailto:sukka.kim at samsung.com>; prakash.v at samsung.com<mailto:prakash.v at samsung.com>; anshul at samsung.com<mailto:anshul at samsung.com>; MANOJ THAPLIYAL <m.thapliyal at samsung.com<mailto:m.thapliyal at samsung.com>>; tru.nguyen at ssi.samsung.com<mailto:tru.nguyen at ssi.samsung.com>
Subject: [nvmewin] Patch with changes for disk Read only support


Hi all,

This patch includes changes for supporting NVMe Disk read only mode.

I have made a detailed overview of the changes in the attached doc file(the contents are also copied here below) and the attached zip file contains the source code.

Password is samsungnvme

Please let me know if you have any questions.

Thanks,
Suman



******************



NVMe Disk End of Life support:

Whenever NVMe disk exhausts the P/E cycles, the disk become Read only(reaches End Of Life). In this case, the user should be able to read the data from the disk for backup or migration purpose. To achieve this, the driver should inform the kernel that disk has become read only. If driver does not inform the kernel, the disk will be unusable from Windows.

The device has to be detected as Read only in following 2 scenarios –

a. Detection during device hot plug

When a Read only device is hot inserted, the kernel should be able to enumerate the device as Read only and alert the user accordingly. When the SSD is hot inserted, as part of disk initialization process, a SCSI mode sense command with page code ‘Return all pages’ (0x3f) is requested by the kernel. The mode page has a mode parameter header, which has a WP bit in the 'Device specific Parameter' field which indicates if the device is Write Protected for some reason. We can make use of this field to report to the kernel that the device has become Read only.

When the miniport driver receives this request, the NVM Express command Get log page is built with log identifier 'SMART / Health Information' (0x2) and send to the device. The SMART data has a 'Critical Warning' field in which a bit 'MediaInReadOnlyMode' is set whenever the media becomes Read only. So if the device returns SMART data with this bit set, the miniport driver sets the Device specific parameter – WP bit in mode parameter header and completes the command.

When the WP bit is set in the mode parameter header, the kernel will understand that the device is Write protected and hence kernel will not send any more write requests.

b. Detection during run time

When the device is in use and the Write exhausts and device becomes Read only, the kernel has to immediately report to the user that device has become write protected. To achieve this, whenever the device receives a NVMe Write request after it has become Read only, the device sets SCT to Command Specific Status and SC to 'Attempted Write to Read Only Range' in response to the write command.  For this the following sense data is returned for the corresponding SCSI write command.

Sense data – SCSI_SENSE_DATA_PROTECT, ASC – SCSI_ADSENSE_WRITE_PROTECT and ASCQ – SCSI_ADSENSE_NO_SENSE.

With this sense data, the kernel will understand that the device is in Write protected state for which the Mode sense command with mode page 'Return all pages' will be send to the device. Again with the NVM Express Get log page – SMART command, the miniport driver will return the mode sense 'Data Specific parameter' accordingly.



Code changes:

1. In SntiReturnAllModePages(), build get log page for SMART/health information and send to device.

2. In SntiTranslateModeSenseResponse(), for log page MODE_SENSE_RETURN_ALL, set the Write protect bit in device specific parameter in the mode header based on the media in read only mode bit(03) in critical warning field returned in SMART/health log page.

3. The checking for volatile write cache is moved from SntiReturnAllModePages() to SntiTranslateModeSenseResponse() after successful completion of get log page command.



We have tested the following:

a. On a Read only NVMe SSD, install OFA driver with these changes. In the disk management tool, the status of disk is shown as Read Only. Please find attached “DiskMgmt.jpg” (sometimes requires a system restart after driver installation).

b. Hot insert a RO NVMe SSD and observe status as Read Only in disk management tool.

c. On NVMe SSD, which has less % of available spare(for example 10%), execute io meter tool with write commands. When available spare reaches 0%, the error count in io meter tools starts increasing(i.e. write commands fails with the sense data, as explained in above sections), and status becomes Read Only in disk management tool.

d. After disk becomes RO, when we try to copy files to the RO drive, Windows show message "The disk is write protected". Please find attached “FileCopy.jpg”

Note:

a. As per NVMe spec 1.2, section 5.10.1.2, "There is not namespace specific information defined in the SMART / Health log page in this revision, thus the global log page and namespaces specific log page contain identical information". So when testing with multi namespace, when 1 namespace becomes RO, all the namespace will become RO. Spec has to be defined to have separate SMART /Health data per namespace.

b. For testing, if there is no NVMe SSD which is in RO state, the following changes can to be made in the driver to test this feature:

    1. In SntiTranslateModeSenseResponse(), hardcode pNvmeLogPage->CriticalWarning.MediaInReadOnlyMode to 1, before checking for the value. This can be done for per namespace also.

    2. In SntiMapCompletionStatus(), for NVMe write command, hardcode statusCodeType to COMMAND_SPECIFIC_ERRORS and statusCode to 0x82. This can be done for per namespace also.



[Image removed by sender.]

Western Digital Corporation (and its subsidiaries) E-mail Confidentiality Notice & Disclaimer:

This e-mail and any files transmitted with it may contain confidential or legally privileged information of WDC and/or its affiliates, and are intended solely for the use of the individual or entity to which they are addressed. If you are not the intended recipient, any disclosure, copying, distribution or any action taken or omitted to be taken in reliance on it, is prohibited. If you have received this e-mail in error, please notify the sender immediately and delete the e-mail in its entirety from your system.













[cid:image004.gif at 01D1A529.F9DB5C40]

[Image removed by sender.]

Western Digital Corporation (and its subsidiaries) E-mail Confidentiality Notice & Disclaimer:

This e-mail and any files transmitted with it may contain confidential or legally privileged information of WDC and/or its affiliates, and are intended solely for the use of the individual or entity to which they are addressed. If you are not the intended recipient, any disclosure, copying, distribution or any action taken or omitted to be taken in reliance on it, is prohibited. If you have received this e-mail in error, please notify the sender immediately and delete the e-mail in its entirety from your system.
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.openfabrics.org/pipermail/nvmewin/attachments/20160503/faab407f/attachment.html>
-------------- next part --------------
A non-text attachment was scrubbed...
Name: image001.jpg
Type: image/jpeg
Size: 2934 bytes
Desc: image001.jpg
URL: <http://lists.openfabrics.org/pipermail/nvmewin/attachments/20160503/faab407f/attachment.jpg>
-------------- next part --------------
A non-text attachment was scrubbed...
Name: image002.jpg
Type: image/jpeg
Size: 2938 bytes
Desc: image002.jpg
URL: <http://lists.openfabrics.org/pipermail/nvmewin/attachments/20160503/faab407f/attachment-0001.jpg>
-------------- next part --------------
A non-text attachment was scrubbed...
Name: image003.jpg
Type: image/jpeg
Size: 823 bytes
Desc: image003.jpg
URL: <http://lists.openfabrics.org/pipermail/nvmewin/attachments/20160503/faab407f/attachment-0002.jpg>
-------------- next part --------------
A non-text attachment was scrubbed...
Name: image004.gif
Type: image/gif
Size: 13168 bytes
Desc: image004.gif
URL: <http://lists.openfabrics.org/pipermail/nvmewin/attachments/20160503/faab407f/attachment.gif>

From suman.p at samsung.com  Wed May  4 00:15:53 2016
From: suman.p at samsung.com (SUMAN PRAKASH B)
Date: Wed, 04 May 2016 07:15:53 +0000 (GMT)
Subject: [nvmewin] Patch with changes for disk Read only support
Message-ID: <94.D6.04928.9A1A9275@epcpsbgx1.samsung.com>

An HTML attachment was scrubbed...
URL: <http://lists.openfabrics.org/pipermail/nvmewin/attachments/20160504/20fcba7a/attachment.html>
-------------- next part --------------
A non-text attachment was scrubbed...
Name: 201605041246887_0EB565ZP.jpg
Type: image/jpeg
Size: 2934 bytes
Desc: not available
URL: <http://lists.openfabrics.org/pipermail/nvmewin/attachments/20160504/20fcba7a/attachment.jpg>
-------------- next part --------------
A non-text attachment was scrubbed...
Name: 201605041246932_JZ9R2KEK.jpg
Type: image/jpeg
Size: 2938 bytes
Desc: not available
URL: <http://lists.openfabrics.org/pipermail/nvmewin/attachments/20160504/20fcba7a/attachment-0001.jpg>
-------------- next part --------------
A non-text attachment was scrubbed...
Name: 201605041246936_OVF2CJ4M.jpg
Type: image/jpeg
Size: 823 bytes
Desc: not available
URL: <http://lists.openfabrics.org/pipermail/nvmewin/attachments/20160504/20fcba7a/attachment-0002.jpg>
-------------- next part --------------
A non-text attachment was scrubbed...
Name: 201605041246940_OZ8NJVZ7.gif
Type: image/gif
Size: 13168 bytes
Desc: not available
URL: <http://lists.openfabrics.org/pipermail/nvmewin/attachments/20160504/20fcba7a/attachment.gif>

From raymond.c.robles at intel.com  Wed May  4 09:38:01 2016
From: raymond.c.robles at intel.com (Robles, Raymond C)
Date: Wed, 4 May 2016 16:38:01 +0000
Subject: [nvmewin] Patch with changes for disk Read only support
In-Reply-To: <44.D6.04928.8A1A9275@epcpsbgx1.samsung.com>
References: <44.D6.04928.8A1A9275@epcpsbgx1.samsung.com>
Message-ID: <49158E750348AA499168FD41D88983607C51C4AB@fmsmsx117.amr.corp.intel.com>

Thanks Suman. I manually copied over your files to the trunk source and updated. Could you please check the trunk again to see if all of your changes are there (they look fine on my end)?
I’m not sure what happened initially, but it looked like the patch only had one file in it (nvmeSnti.c) and not all the changes successfully merged in. Not sure if the patch file somehow got corrupted.
Thanks… Ray

From: SUMAN PRAKASH B [mailto:suman.p at samsung.com]
Sent: Wednesday, May 04, 2016 12:16 AM
To: Robles, Raymond C <raymond.c.robles at intel.com>; 'Thomas Freeman' <thomas.freeman at hgst.com>; nvmewin at lists.openfabrics.org
Cc: PRAKASH BABU VEMULA <prakash.v at samsung.com>; ANSHUL SHARMA <anshul at samsung.com>; MANOJ THAPLIYAL <m.thapliyal at samsung.com>
Subject: Re: RE: [nvmewin] Patch with changes for disk Read only support


Hi Ray,



The patch is not applied to trunk correctly. I have changed nvmeSnti.c, nvmeSnti.h and nvmeStd.h files for patch 2. And only nvmeSnti.c is updated in trunk and in that, only some of the changes are updated.



Thanks,

Suman



------- Original Message -------

Sender : Robles, Raymond C<raymond.c.robles at intel.com<mailto:raymond.c.robles at intel.com>>

Date : May 03, 2016 23:23 (GMT+05:30)

Title : RE: [nvmewin] Patch with changes for disk Read only support


Thanks Tom… and Suman. I’ve pushed the patch.
Suman, could you please double check the trunk to ensure your patch was applied properly? I got an error while applying the patch, and I want to make sure your patch got applied properly.
Thanks… Ray
From: Thomas Freeman [mailto:thomas.freeman at hgst.com]
Sent: Friday, April 29, 2016 10:52 AM
To: Robles, Raymond C <raymond.c.robles at intel.com<mailto:raymond.c.robles at intel.com>>; suman.p at samsung.com<mailto:suman.p at samsung.com>; nvmewin at lists.openfabrics.org<mailto:nvmewin at lists.openfabrics.org>
Cc: PRAKASH BABU VEMULA <prakash.v at samsung.com<mailto:prakash.v at samsung.com>>; ANSHUL SHARMA <anshul at samsung.com<mailto:anshul at samsung.com>>; MANOJ THAPLIYAL <m.thapliyal at samsung.com<mailto:m.thapliyal at samsung.com>>
Subject: RE: [nvmewin] Patch with changes for disk Read only support

I accept the change.
Tom Freeman
Software Engineer, Device Manager and Driver Development
Western Digital Corporation
e.  Thomas.freeman at hgst.com<mailto:Thomas.freeman at hgst.com>
o.  +1-507-322-2311

[cid:image001.jpg at 01D1A5E8.A5013310]

From: nvmewin [mailto:nvmewin-bounces at lists.openfabrics.org] On Behalf Of Robles, Raymond C
Sent: Friday, April 29, 2016 12:47 PM
To: suman.p at samsung.com<mailto:suman.p at samsung.com>; nvmewin at lists.openfabrics.org<mailto:nvmewin at lists.openfabrics.org>
Cc: PRAKASH BABU VEMULA <prakash.v at samsung.com<mailto:prakash.v at samsung.com>>; ANSHUL SHARMA <anshul at samsung.com<mailto:anshul at samsung.com>>; MANOJ THAPLIYAL <m.thapliyal at samsung.com<mailto:m.thapliyal at samsung.com>>
Subject: Re: [nvmewin] Patch with changes for disk Read only support

Hi Suman,
I was waiting to hear back from Tom (HGST) on your latest changes. Once I get confirmation, we can close out this patch.
Thanks… Ray
From: SUMAN PRAKASH B [mailto:suman.p at samsung.com]
Sent: Friday, April 29, 2016 3:50 AM
To: Robles, Raymond C <raymond.c.robles at intel.com<mailto:raymond.c.robles at intel.com>>; nvmewin at lists.openfabrics.org<mailto:nvmewin at lists.openfabrics.org>
Cc: PRAKASH BABU VEMULA <prakash.v at samsung.com<mailto:prakash.v at samsung.com>>; MANOJ THAPLIYAL <m.thapliyal at samsung.com<mailto:m.thapliyal at samsung.com>>; ANSHUL SHARMA <anshul at samsung.com<mailto:anshul at samsung.com>>
Subject: Re: Re: RE: [nvmewin] Patch with changes for disk Read only support


Hi Ray,



Since Intel and HGST have reviewed the code, and I have incorporated the review comments provided by HGST, can we close out this patch?

We will be ready with our next patch by next week.



Thanks,

Suman



------- Original Message -------

Sender : SUMAN PRAKASH B<suman.p at samsung.com<mailto:suman.p at samsung.com>> Senior Chief Engineer/SSIR-SSD Solutions/Samsung Electronics

Date : Apr 27, 2016 20:09 (GMT+05:30)

Title : Re: RE: [nvmewin] Patch with changes for disk Read only support



Hi All,



I am sending the updated patch incorporating feedback from Tom. The changes are listed below. The password is samsungnvme



1. In SntiCompletionCallbackRoutine(), if the NVMe command Get Log Page fails, (SCT != Generic_command_status || SC != Successful completion), the buffer pSrbExt->pDatBuffer is freed.



2. In SntiTranslateModeSenseResponse(), replaced

        Lun = pLunExt->namespaceId - 1;
        pDevExt->pLunExtensionTable[Lun]->IsNamespaceReadOnly = TRUE;

with

        pLunExt->IsNamespaceReadOnly = TRUE;



3. In SntiReturnAllModePages(), following code is removed, as the same copy occurs in SntiTranslateModeSenseResponse()

         if (GET_DATA_BUFFER(pSrb) != NULL) {
              StorPortCopyMemory((PVOID)GET_DATA_BUFFER(pSrb),
              (PVOID)(pSrbExt->modeSenseBuf), GET_DATA_LENGTH(pSrb));
         }



Please let me know if you have any questions.



Thanks,
Suman



------- Original Message -------

Sender : Foster, Carolyn D<carolyn.d.foster at intel.com<mailto:carolyn.d.foster at intel.com>>

Date : Apr 22, 2016 20:49 (GMT+05:30)

Title : RE: [nvmewin] Patch with changes for disk Read only support


Hi Suman, I don’t have any additional feedback for this patch.
Carolyn
From: nvmewin [mailto:nvmewin-bounces at lists.openfabrics.org] On Behalf Of Thomas Freeman
Sent: Thursday, April 21, 2016 9:18 AM
To: suman.p at samsung.com<mailto:suman.p at samsung.com>; nvmewin at lists.openfabrics.org<mailto:nvmewin at lists.openfabrics.org>
Subject: Re: [nvmewin] Patch with changes for disk Read only support

Hi Suman,
Thank you for the quick response. I agree with your comments.
Thank you,
Tom Freeman
Software Engineer, Device Manager and Driver Development
Western Digital Corporation
e.  Thomas.freeman at hgst.com<mailto:Thomas.freeman at hgst.com>
o.  +1-507-322-2311

[cid:image001.jpg at 01D1A5E8.A5013310]

From: SUMAN PRAKASH B [mailto:suman.p at samsung.com]
Sent: Thursday, April 21, 2016 9:55 AM
To: Thomas Freeman <thomas.freeman at hgst.com<mailto:thomas.freeman at hgst.com>>; nvmewin at lists.openfabrics.org<mailto:nvmewin at lists.openfabrics.org>
Cc: anshul at samsung.com<mailto:anshul at samsung.com>; prakash.v at samsung.com<mailto:prakash.v at samsung.com>; MANOJ THAPLIYAL <m.thapliyal at samsung.com<mailto:m.thapliyal at samsung.com>>
Subject: Re: RE: [nvmewin] Patch with changes for disk Read only support


Hi Tom,



Thanks for the comments. Please find my replies below:



1. For mode sense with 0x3f, you set the WP in the response header. Shouldn WP also be set if any of those pages(0x8, 0xa, 0x1a, 0x1c) are individually requested?

[Suman] Below is our observations:

a. During driver install/device enable from dev manager/hot insert, the  mode pages 0x8 and 0x3f are invoked.

b. During online/offline of disk, only 0x3f is invoked.

So as per our understanding, during disk initialization, not all the mode pages will be called. But driver gets the mode page 0x3f every time during disk initialization. Also for Detection during run time, when driver returns SCSI_SENSE_DATA_PROTECT for sense data, driver gets the mode page 0x3f  consistently. So we feel, setting the WP bit for mode page 0x3f will suffice.



2. snti.c:8242

[Suman] Agreed. We will use pLunExt->IsNamespaceReadOnly = TRUE;



3. Along with the new member, IsNamespaceReadOnly, the nvme_lun_extension also has ReadOnly. It seems like the setting of WP should take into account the value of both members.

[Suman] The OFA driver supports only the first lba range type for a namespace, though spec supports 64 lba range types per NS. This has to be corrected first.

Also we have to decide if the disk should be exposed as Read Only if any of the LBA range type is read only or only if the LBA 0 is read only.

I feel this should be taken as a separate patch since this involves too many changes.



4. If the NVMe command Get Log Page fails, (SCT != Generic_command_status || SC != Successful completion), the buffer pSrbExt->pDatBuffer is not freed. This corresponds to the allocation at snti.c:6530.

[Suman] Agreed.



5. snti.c:6539: I think the following can be eliminated. The same copy occurs during SntiTranslateModeSenseResponse - snti.c:8272.

[Suman] Agreed.



Please let us know your opinion.



Regards,

Suman



------- Original Message -------

Sender : Thomas Freeman<thomas.freeman at hgst.com<mailto:thomas.freeman at hgst.com>>

Date : Apr 20, 2016 21:59 (GMT+05:30)

Title : RE: [nvmewin] Patch with changes for disk Read only support


Hi Suman,
After reviewing the code, I have a few questions/comments:


1.       For mode sense with 0x3f, you set the WP in the response header. Shouldn WP also be set if any of those pages(0x8, 0xa, 0x1a, 0x1c) are individually requested?


2. snti.c:8242
Lun = pLunExt->namespaceId - 1;
pDevExt->pLunExtensionTable[Lun]->IsNamespaceReadOnly = TRUE;

These 2 lines can be replaced with
pLunExt->IsNamespaceReadOnly = TRUE;

Also, the original code is not a reliable way to determine the LUN id.
Here is an example where there doesn't work. The device has attached namespaces 1,3 & 4 and Existing namespaces of 1, 2, 3 & 4. LUNs 0-3 will correspond to namespaces 1,3,4,2. For namespace 3, the calculation NSID-1=lun will incorrectly give you LUNid of 2.

3. Along with the new member, IsNamespaceReadOnly, the nvme_lun_extension also has ReadOnly. It seems like the setting of WP should take into account the value of both members.

4. If the NVMe command Get Log Page fails, (SCT != Generic_command_status || SC != Successful completion), the buffer pSrbExt->pDatBuffer is not freed. This corresponds to the allocation at snti.c:6530.

5. snti.c:6539: I think the following can be eliminated. The same copy occurs during SntiTranslateModeSenseResponse - snti.c:8272.
if (GET_DATA_BUFFER(pSrb) != NULL) {
            StorPortCopyMemory((PVOID)GET_DATA_BUFFER(pSrb),
                (PVOID)(pSrbExt->modeSenseBuf), GET_DATA_LENGTH(pSrb));
        }
Let me know if you have questions,
Tom Freeman
Software Engineer, Device Manager and Driver Development
Western Digital Corporation
e.  Thomas.freeman at hgst.com<mailto:Thomas.freeman at hgst.com>
o.  +1-507-322-2311

[cid:image002.jpg at 01D1A5E8.A5013310]

From: nvmewin [mailto:nvmewin-bounces at lists.openfabrics.org] On Behalf Of SUMAN PRAKASH B
Sent: Tuesday, April 19, 2016 8:45 AM
To: nvmewin at lists.openfabrics.org<mailto:nvmewin at lists.openfabrics.org>
Cc: sukka.kim at samsung.com<mailto:sukka.kim at samsung.com>; prakash.v at samsung.com<mailto:prakash.v at samsung.com>; anshul at samsung.com<mailto:anshul at samsung.com>; MANOJ THAPLIYAL <m.thapliyal at samsung.com<mailto:m.thapliyal at samsung.com>>; tru.nguyen at ssi.samsung.com<mailto:tru.nguyen at ssi.samsung.com>
Subject: [nvmewin] Patch with changes for disk Read only support


Hi all,

This patch includes changes for supporting NVMe Disk read only mode.

I have made a detailed overview of the changes in the attached doc file(the contents are also copied here below) and the attached zip file contains the source code.

Password is samsungnvme

Please let me know if you have any questions.

Thanks,
Suman



******************



NVMe Disk End of Life support:

Whenever NVMe disk exhausts the P/E cycles, the disk become Read only(reaches End Of Life). In this case, the user should be able to read the data from the disk for backup or migration purpose. To achieve this, the driver should inform the kernel that disk has become read only. If driver does not inform the kernel, the disk will be unusable from Windows.

The device has to be detected as Read only in following 2 scenarios –

a. Detection during device hot plug

When a Read only device is hot inserted, the kernel should be able to enumerate the device as Read only and alert the user accordingly. When the SSD is hot inserted, as part of disk initialization process, a SCSI mode sense command with page code ‘Return all pages’ (0x3f) is requested by the kernel. The mode page has a mode parameter header, which has a WP bit in the 'Device specific Parameter' field which indicates if the device is Write Protected for some reason. We can make use of this field to report to the kernel that the device has become Read only.

When the miniport driver receives this request, the NVM Express command Get log page is built with log identifier 'SMART / Health Information' (0x2) and send to the device. The SMART data has a 'Critical Warning' field in which a bit 'MediaInReadOnlyMode' is set whenever the media becomes Read only. So if the device returns SMART data with this bit set, the miniport driver sets the Device specific parameter – WP bit in mode parameter header and completes the command.

When the WP bit is set in the mode parameter header, the kernel will understand that the device is Write protected and hence kernel will not send any more write requests.

b. Detection during run time

When the device is in use and the Write exhausts and device becomes Read only, the kernel has to immediately report to the user that device has become write protected. To achieve this, whenever the device receives a NVMe Write request after it has become Read only, the device sets SCT to Command Specific Status and SC to 'Attempted Write to Read Only Range' in response to the write command.  For this the following sense data is returned for the corresponding SCSI write command.

Sense data – SCSI_SENSE_DATA_PROTECT, ASC – SCSI_ADSENSE_WRITE_PROTECT and ASCQ – SCSI_ADSENSE_NO_SENSE.

With this sense data, the kernel will understand that the device is in Write protected state for which the Mode sense command with mode page 'Return all pages' will be send to the device. Again with the NVM Express Get log page – SMART command, the miniport driver will return the mode sense 'Data Specific parameter' accordingly.



Code changes:

1. In SntiReturnAllModePages(), build get log page for SMART/health information and send to device.

2. In SntiTranslateModeSenseResponse(), for log page MODE_SENSE_RETURN_ALL, set the Write protect bit in device specific parameter in the mode header based on the media in read only mode bit(03) in critical warning field returned in SMART/health log page.

3. The checking for volatile write cache is moved from SntiReturnAllModePages() to SntiTranslateModeSenseResponse() after successful completion of get log page command.



We have tested the following:

a. On a Read only NVMe SSD, install OFA driver with these changes. In the disk management tool, the status of disk is shown as Read Only. Please find attached “DiskMgmt.jpg” (sometimes requires a system restart after driver installation).

b. Hot insert a RO NVMe SSD and observe status as Read Only in disk management tool.

c. On NVMe SSD, which has less % of available spare(for example 10%), execute io meter tool with write commands. When available spare reaches 0%, the error count in io meter tools starts increasing(i.e. write commands fails with the sense data, as explained in above sections), and status becomes Read Only in disk management tool.

d. After disk becomes RO, when we try to copy files to the RO drive, Windows show message "The disk is write protected". Please find attached “FileCopy.jpg”

Note:

a. As per NVMe spec 1.2, section 5.10.1.2, "There is not namespace specific information defined in the SMART / Health log page in this revision, thus the global log page and namespaces specific log page contain identical information". So when testing with multi namespace, when 1 namespace becomes RO, all the namespace will become RO. Spec has to be defined to have separate SMART /Health data per namespace.

b. For testing, if there is no NVMe SSD which is in RO state, the following changes can to be made in the driver to test this feature:

    1. In SntiTranslateModeSenseResponse(), hardcode pNvmeLogPage->CriticalWarning.MediaInReadOnlyMode to 1, before checking for the value. This can be done for per namespace also.

    2. In SntiMapCompletionStatus(), for NVMe write command, hardcode statusCodeType to COMMAND_SPECIFIC_ERRORS and statusCode to 0x82. This can be done for per namespace also.



[Image removed by sender.]

Western Digital Corporation (and its subsidiaries) E-mail Confidentiality Notice & Disclaimer:

This e-mail and any files transmitted with it may contain confidential or legally privileged information of WDC and/or its affiliates, and are intended solely for the use of the individual or entity to which they are addressed. If you are not the intended recipient, any disclosure, copying, distribution or any action taken or omitted to be taken in reliance on it, is prohibited. If you have received this e-mail in error, please notify the sender immediately and delete the e-mail in its entirety from your system.

















[cid:image004.gif at 01D1A5E8.A5013310]

[http://ext.samsung.net/mailcheck/SeenTimeChecker?do=3af3b79521f261fadf6a404fbc8c6b76262f1de94e6a59f8313cb48408a29da1bbef98290b9ea6844a835b19b8ec809fc7b41e955949e5c8a728c55b39cc59eacf878f9a26ce15a0]
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.openfabrics.org/pipermail/nvmewin/attachments/20160504/8fca140b/attachment.html>
-------------- next part --------------
A non-text attachment was scrubbed...
Name: image001.jpg
Type: image/jpeg
Size: 2934 bytes
Desc: image001.jpg
URL: <http://lists.openfabrics.org/pipermail/nvmewin/attachments/20160504/8fca140b/attachment.jpg>
-------------- next part --------------
A non-text attachment was scrubbed...
Name: image002.jpg
Type: image/jpeg
Size: 2938 bytes
Desc: image002.jpg
URL: <http://lists.openfabrics.org/pipermail/nvmewin/attachments/20160504/8fca140b/attachment-0001.jpg>
-------------- next part --------------
A non-text attachment was scrubbed...
Name: image003.jpg
Type: image/jpeg
Size: 823 bytes
Desc: image003.jpg
URL: <http://lists.openfabrics.org/pipermail/nvmewin/attachments/20160504/8fca140b/attachment-0002.jpg>
-------------- next part --------------
A non-text attachment was scrubbed...
Name: image004.gif
Type: image/gif
Size: 13168 bytes
Desc: image004.gif
URL: <http://lists.openfabrics.org/pipermail/nvmewin/attachments/20160504/8fca140b/attachment.gif>

From suman.p at samsung.com  Thu May  5 06:58:12 2016
From: suman.p at samsung.com (SUMAN PRAKASH B)
Date: Thu, 05 May 2016 13:58:12 +0000 (GMT)
Subject: [nvmewin] Patch with changes for disk Read only support
Message-ID: <05.EB.05093.4715B275@epcpsbgx4.samsung.com>

An HTML attachment was scrubbed...
URL: <http://lists.openfabrics.org/pipermail/nvmewin/attachments/20160505/ea9c126c/attachment.html>
-------------- next part --------------
A non-text attachment was scrubbed...
Name: 201605051929437_Z5JE7EUA.jpg
Type: image/jpeg
Size: 2934 bytes
Desc: not available
URL: <http://lists.openfabrics.org/pipermail/nvmewin/attachments/20160505/ea9c126c/attachment.jpg>
-------------- next part --------------
A non-text attachment was scrubbed...
Name: 201605051929443_LK7CT9SZ.jpg
Type: image/jpeg
Size: 2938 bytes
Desc: not available
URL: <http://lists.openfabrics.org/pipermail/nvmewin/attachments/20160505/ea9c126c/attachment-0001.jpg>
-------------- next part --------------
A non-text attachment was scrubbed...
Name: 201605051929448_BSL8PYMC.jpg
Type: image/jpeg
Size: 823 bytes
Desc: not available
URL: <http://lists.openfabrics.org/pipermail/nvmewin/attachments/20160505/ea9c126c/attachment-0002.jpg>
-------------- next part --------------
A non-text attachment was scrubbed...
Name: 201605051929452_4XEV4D4T.gif
Type: image/gif
Size: 13168 bytes
Desc: not available
URL: <http://lists.openfabrics.org/pipermail/nvmewin/attachments/20160505/ea9c126c/attachment.gif>

From suman.p at samsung.com  Tue May 10 09:33:12 2016
From: suman.p at samsung.com (SUMAN PRAKASH B)
Date: Tue, 10 May 2016 16:33:12 +0000 (GMT)
Subject: [nvmewin] Samsung's next patch
Message-ID: <1A.21.05252.84D02375@epcpsbgx3.samsung.com>

An HTML attachment was scrubbed...
URL: <http://lists.openfabrics.org/pipermail/nvmewin/attachments/20160510/57b24bcd/attachment.html>
-------------- next part --------------
A non-text attachment was scrubbed...
Name: 201605102205457_Z5JE7EUA.gif
Type: image/gif
Size: 13168 bytes
Desc: not available
URL: <http://lists.openfabrics.org/pipermail/nvmewin/attachments/20160510/57b24bcd/attachment.gif>

From suman.p at samsung.com  Mon May 16 06:21:08 2016
From: suman.p at samsung.com (SUMAN PRAKASH B)
Date: Mon, 16 May 2016 13:21:08 +0000 (GMT)
Subject: [nvmewin] Patch for handling surprise removal in IOCTL path
Message-ID: <41.82.05093.449C9375@epcpsbgx4.samsung.com>

An HTML attachment was scrubbed...
URL: <http://lists.openfabrics.org/pipermail/nvmewin/attachments/20160516/74e9a487/attachment.html>
-------------- next part --------------
A non-text attachment was scrubbed...
Name: Samsung_SurpriseRemovalDuringIOCTL_v1.7z
Type: application/octet-stream
Size: 149104 bytes
Desc: not available
URL: <http://lists.openfabrics.org/pipermail/nvmewin/attachments/20160516/74e9a487/attachment.obj>
-------------- next part --------------
A non-text attachment was scrubbed...
Name: Handling device Surprise removal in IOCTL path.docx
Type: application/octet-stream
Size: 16148 bytes
Desc: not available
URL: <http://lists.openfabrics.org/pipermail/nvmewin/attachments/20160516/74e9a487/attachment-0001.obj>

From thomas.freeman at hgst.com  Thu May 26 14:15:32 2016
From: thomas.freeman at hgst.com (Thomas Freeman)
Date: Thu, 26 May 2016 21:15:32 +0000
Subject: [nvmewin] Patch for handling surprise removal in IOCTL path
In-Reply-To: <41.82.05093.449C9375@epcpsbgx4.samsung.com>
References: <41.82.05093.449C9375@epcpsbgx4.samsung.com>
Message-ID: <BLUPR04MB67302DDD16F529360FB5029E9410@BLUPR04MB673.namprd04.prod.outlook.com>

Suman,
I've reviewed changes and have no comments/concerns.
Thank you,
Tom Freeman
Software Engineer, Device Manager and Driver Development
Western Digital Corporation
e.  Thomas.freeman at hgst.com
o.  +1-507-322-2311

[cid:image002.jpg at 01D1B769.CD9DE060]

From: nvmewin [mailto:nvmewin-bounces at lists.openfabrics.org] On Behalf Of SUMAN PRAKASH B
Sent: Monday, May 16, 2016 8:21 AM
To: nvmewin at lists.openfabrics.org; raymond.c.robles at intel.com
Subject: [nvmewin] Patch for handling surprise removal in IOCTL path


Hi All,



This patch includes changes for supporting device surprise removal in IOCTL path. Samsung already submitted a patch to handle surprise removal during normal I/O in 2014. This patch is an extention to the previous patch.

I have made a detailed overview of the changes in the attached doc file(the contents are also copied here below) and the attached zip file contains the source code.

Password is samsungnvme

Please let me know if you have any questions.

Thanks,
Suman

***************************************



Handling device Surprise removal in IOCTL path:

    Current OFA driver does not need any change to support surprise removal when no I/Os are outstanding. But when I/Os are outstanding and the device is surprise removed, the tool(lets say IOMeter) hangs and the device is not removed from the device manager for very long time. Since NVMe is suprise remove capable device, user expectation is that the tool should exit/stop gracefully and the device should be removed from the device manage immediately.

    The Windows storage stack currently does not handle surprise removal during outstanding I/Os. To handle this scenario, we have submitted a patch in 2014. But this patch was handling surprise removal during normal I/O commands and not when IOCTL commands from proprietary tools were outstanding.
    In the previous patch, we had implemented the StorPortNotification(RequestTimerCall,...) to handle surprise removal durign I/O. This API does not detect surprise removal when executing IOCTL commands. For example, when executing Format NVM command with Secure erase or Cryptographic erase, when device is surprise removed, the device is not removed from the device manager and system hangs.

    To fix this issue, in Windows 8 and later versions, there is support for StorPortRequestTimer() which does the same as StorPortNotification(RequestTimerCall,...). When StorPortRequestTimer() is used, device surprise removal during IOCTL commands is also detected. Hence for Win 7 kernel, StorPortNotification(RequestTimerCall,...) and for Win 8 and above kernels, StorPortRequestTimer() can be used.



Code changes:

1. In NVMeFindAdapter(), initialize the timer routine using StorPortInitializeTimer() API.



2. In NVMeRunning(), in NVMeStartComplete, start the surprise removal timer routine.



3. Stop the timer routine using StorPortRequestTimer() in following functions:
 a. In NVMeAdapterControl(), in ScsiStopAdapter, when shutting down the device
 b. In NVMeBuildIo(), for SRB_FUNCTION_SHUTDOWN
 c. In NVMeBuildIo(), for SRB_FUNCTION_PNP, if PnPAction is StorRemoveDevice or StorSurpriseRemoval and ntldrDump is FALSE



4. Free the timer routine using StorPortFreeTimer() in following functions:
 a. In NVMePassiveInitialize(), when the initialization state machine fails
 b. In NVMeAdapterControl(), in ScsiStopAdapter, when shutting down the device
 c. In NVMeBuildIo(), for SRB_FUNCTION_SHUTDOWN,
 d. In NVMeBuildIo(), for SRB_FUNCTION_PNP, if PnPAction is StorRemoveDevice or StorSurpriseRemoval and ntldrDump is FALSE
 e. In NVMeStartIo(), when NextDriverState is not NVMeStartComplete
 f. In RecoveryDpcRoutine()



5. New Surprise removal timer routine IsDeviceRemoved() for kernels above Windows 8 which uses StorPortFreeTimer() and StorPortRequestTimer().



6. In FormatNVMGetIdentify(), when the structure to retrieve is Identify namespace, the lunId is set to INVALID_LUN_EXTN, because of which BSOD is observed. Invoked the function NVMeIsNamespaceVisible() to correct this. We observed this when secure erase in IOCTL is in progress, and device is surprise removed.



7. In NVMeBuildIo(), to block read/write commands when format nvm is in progress, we have a check. But when formatNVM is in progress, we get requests other than SRB_FUNCTION_EXECUTE_SCSI, like SRB_FUNCTION_PNP or SRB_FUNCTION_IO_CONTROL, in which case opCode = GET_OPCODE(Srb) in line number 1125 results in BSOD. This is observed when secure erase is executed in IOCTL path, and system is hibernated and then device is surprise removed. Hence moved the blocking of Read/Write commands while Format NVM is in progress code inside SRB_FUNCTION_EXECUTE_SCSI.



8. Changes in inf file:
    a. Following line has been included to allow interrupts on processors beyond group 0. For the StorPortInitializePerfOpts() MessageTargets to work correctly when a system has multiple groups are, the following line is required, otherwise we observe BSOD. Also, when we tested in a system which has 1 group by default and 32 logical processors in Server 2012R2, some WHQL tests create multiple groups after which the driver does not work properly if the below line is not included in inf file.
         HKR, Interrupt Management\Affinity Policy, GroupPolicy, %REG_DWORD%, 1

    b. Following line has been included to specifies that the device's interrupts are of high priority.
         HKR, Interrupt Management\Affinity Policy, DevicePriority, %REG_DWORD%, 3



9. As requested by HGST, the MAX_NAMESPACES is changed to 128. Correspondingly, the DUMP_BUFFER_SIZE is also changed from (5*64*1024) to ((5*64*1024) + (sizeof(NVME_LUN_EXTENSION)*MAX_NAMESPACES)). If DUMP_BUFFER_SIZE is not increased, in dump mode, in NVMeInitialize(), memory allocation for pAE->pLunExtensionTable will fail.

Also, the correct method to implement the above would be to allocate memory for pAE->pLunExtensionTable in NVMeInitCallback() in the NVMeWaitOnIdentifyCtrl state, where we get the number of Namespaces supported by the device, and remove the memory allocated for pAE->pLunExtensionTable in dump mode(NVMeInitialize()) and in normal mode(NVMePassiveInitialize()).

For this patch, we have set DUMP_BUFFER_SIZE to ((5*64*1024) + (sizeof(NVME_LUN_EXTENSION) * MAX_NAMESPACES)). Please note that as per MSDN, we should not allocate more than 32 KB for RequestedDumpBufferSize.



10. Implemented ScsiAdapterPower controltype in NVMeAdapterControl(). ScsiAdapterPower executes in <= DISPATCH_IRQL. This is supported only from Windows 8. Since we need to free the buffer and stop the timer only at IRQL <= DISPATCH_IRQL, this is implemented. Also, this will avoid executing initialization state machine in NVMeReInitializeController in ScsiRestartAdapter(DIRQL) during resume from hibernation.



We have tested following:

a. WHQL tests

b. Connect device -> execute secure erase in IOCTL path -> during secure erase execution remove device -> expectation: disk should be removed and tool(executing the secure erase) should exit gracefully

c. Connect device -> hibernation -> remove device -> resume -> expectation: disk should be removed

d. Connect device -> run IOMeter -> execute secure erase in IOCTL path -> hibernation -> remove device -> resume -> expectation: disk should be removed and tools should exit gracefully

c. OS installation




[Image removed by sender.]
Western Digital Corporation (and its subsidiaries) E-mail Confidentiality Notice & Disclaimer:

This e-mail and any files transmitted with it may contain confidential or legally privileged information of WDC and/or its affiliates, and are intended solely for the use of the individual or entity to which they are addressed. If you are not the intended recipient, any disclosure, copying, distribution or any action taken or omitted to be taken in reliance on it, is prohibited. If you have received this e-mail in error, please notify the sender immediately and delete the e-mail in its entirety from your system.
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.openfabrics.org/pipermail/nvmewin/attachments/20160526/109359ab/attachment.html>
-------------- next part --------------
A non-text attachment was scrubbed...
Name: ~WRD000.jpg
Type: image/jpeg
Size: 823 bytes
Desc: ~WRD000.jpg
URL: <http://lists.openfabrics.org/pipermail/nvmewin/attachments/20160526/109359ab/attachment.jpg>
-------------- next part --------------
A non-text attachment was scrubbed...
Name: image002.jpg
Type: image/jpeg
Size: 2937 bytes
Desc: image002.jpg
URL: <http://lists.openfabrics.org/pipermail/nvmewin/attachments/20160526/109359ab/attachment-0001.jpg>

From suman.p at samsung.com  Mon May 30 07:59:43 2016
From: suman.p at samsung.com (SUMAN PRAKASH B)
Date: Mon, 30 May 2016 14:59:43 +0000 (GMT)
Subject: [nvmewin] Patch for handling surprise removal in IOCTL path
Message-ID: <85.BD.05252.F555C475@epcpsbgx3.samsung.com>

An HTML attachment was scrubbed...
URL: <http://lists.openfabrics.org/pipermail/nvmewin/attachments/20160530/9dfd2df7/attachment.html>

From raymond.c.robles at intel.com  Tue May 31 14:23:06 2016
From: raymond.c.robles at intel.com (Robles, Raymond C)
Date: Tue, 31 May 2016 21:23:06 +0000
Subject: [nvmewin] Patch for handling surprise removal in IOCTL path
In-Reply-To: <A5.BD.05252.F555C475@epcpsbgx3.samsung.com>
References: <A5.BD.05252.F555C475@epcpsbgx3.samsung.com>
Message-ID: <49158E750348AA499168FD41D88983607C5852E5@fmsmsx117.amr.corp.intel.com>

Hi Suman… Intel has no comments/concerns. Everything looks good. I’ll push the patch in the next day.
I believe Samsung had one more patch for concurrent channels, correct? Was there anything after that? I’d like to start planning for the mid-year release.
Thank you… Ray
From: SUMAN PRAKASH B [mailto:suman.p at samsung.com]
Sent: Monday, May 30, 2016 8:00 AM
To: Thomas Freeman <thomas.freeman at hgst.com>; nvmewin at lists.openfabrics.org; Robles, Raymond C <raymond.c.robles at intel.com>
Subject: Re: RE: [nvmewin] Patch for handling surprise removal in IOCTL path




Thanks Tom.



To other reviewers: Please let me know if there is any comments/concerns.



- Suman



------- Original Message -------

Sender : Thomas Freeman<thomas.freeman at hgst.com<mailto:thomas.freeman at hgst.com>>

Date : May 27, 2016 02:45 (GMT+05:30)

Title : RE: [nvmewin] Patch for handling surprise removal in IOCTL path


Suman,
I’ve reviewed changes and have no comments/concerns.
Thank you,
Tom Freeman
Software Engineer, Device Manager and Driver Development
Western Digital Corporation
e.  Thomas.freeman at hgst.com<mailto:Thomas.freeman at hgst.com>
o.  +1-507-322-2311


From: nvmewin [mailto:nvmewin-bounces at lists.openfabrics.org] On Behalf Of SUMAN PRAKASH B
Sent: Monday, May 16, 2016 8:21 AM
To: nvmewin at lists.openfabrics.org<mailto:nvmewin at lists.openfabrics.org>; raymond.c.robles at intel.com<mailto:raymond.c.robles at intel.com>
Subject: [nvmewin] Patch for handling surprise removal in IOCTL path


Hi All,



This patch includes changes for supporting device surprise removal in IOCTL path. Samsung already submitted a patch to handle surprise removal during normal I/O in 2014. This patch is an extention to the previous patch.

I have made a detailed overview of the changes in the attached doc file(the contents are also copied here below) and the attached zip file contains the source code.

Password is samsungnvme

Please let me know if you have any questions.

Thanks,
Suman

***************************************



Handling device Surprise removal in IOCTL path:

    Current OFA driver does not need any change to support surprise removal when no I/Os are outstanding. But when I/Os are outstanding and the device is surprise removed, the tool(lets say IOMeter) hangs and the device is not removed from the device manager for very long time. Since NVMe is suprise remove capable device, user expectation is that the tool should exit/stop gracefully and the device should be removed from the device manage immediately.

    The Windows storage stack currently does not handle surprise removal during outstanding I/Os. To handle this scenario, we have submitted a patch in 2014. But this patch was handling surprise removal during normal I/O commands and not when IOCTL commands from proprietary tools were outstanding.
    In the previous patch, we had implemented the StorPortNotification(RequestTimerCall,…) to handle surprise removal durign I/O. This API does not detect surprise removal when executing IOCTL commands. For example, when executing Format NVM command with Secure erase or Cryptographic erase, when device is surprise removed, the device is not removed from the device manager and system hangs.

    To fix this issue, in Windows 8 and later versions, there is support for StorPortRequestTimer() which does the same as StorPortNotification(RequestTimerCall,…). When StorPortRequestTimer() is used, device surprise removal during IOCTL commands is also detected. Hence for Win 7 kernel, StorPortNotification(RequestTimerCall,…) and for Win 8 and above kernels, StorPortRequestTimer() can be used.



Code changes:

1. In NVMeFindAdapter(), initialize the timer routine using StorPortInitializeTimer() API.



2. In NVMeRunning(), in NVMeStartComplete, start the surprise removal timer routine.



3. Stop the timer routine using StorPortRequestTimer() in following functions:
 a. In NVMeAdapterControl(), in ScsiStopAdapter, when shutting down the device
 b. In NVMeBuildIo(), for SRB_FUNCTION_SHUTDOWN
 c. In NVMeBuildIo(), for SRB_FUNCTION_PNP, if PnPAction is StorRemoveDevice or StorSurpriseRemoval and ntldrDump is FALSE



4. Free the timer routine using StorPortFreeTimer() in following functions:
 a. In NVMePassiveInitialize(), when the initialization state machine fails
 b. In NVMeAdapterControl(), in ScsiStopAdapter, when shutting down the device
 c. In NVMeBuildIo(), for SRB_FUNCTION_SHUTDOWN,
 d. In NVMeBuildIo(), for SRB_FUNCTION_PNP, if PnPAction is StorRemoveDevice or StorSurpriseRemoval and ntldrDump is FALSE
 e. In NVMeStartIo(), when NextDriverState is not NVMeStartComplete
 f. In RecoveryDpcRoutine()



5. New Surprise removal timer routine IsDeviceRemoved() for kernels above Windows 8 which uses StorPortFreeTimer() and StorPortRequestTimer().



6. In FormatNVMGetIdentify(), when the structure to retrieve is Identify namespace, the lunId is set to INVALID_LUN_EXTN, because of which BSOD is observed. Invoked the function NVMeIsNamespaceVisible() to correct this. We observed this when secure erase in IOCTL is in progress, and device is surprise removed.



7. In NVMeBuildIo(), to block read/write commands when format nvm is in progress, we have a check. But when formatNVM is in progress, we get requests other than SRB_FUNCTION_EXECUTE_SCSI, like SRB_FUNCTION_PNP or SRB_FUNCTION_IO_CONTROL, in which case opCode = GET_OPCODE(Srb) in line number 1125 results in BSOD. This is observed when secure erase is executed in IOCTL path, and system is hibernated and then device is surprise removed. Hence moved the blocking of Read/Write commands while Format NVM is in progress code inside SRB_FUNCTION_EXECUTE_SCSI.



8. Changes in inf file:
    a. Following line has been included to allow interrupts on processors beyond group 0. For the StorPortInitializePerfOpts() MessageTargets to work correctly when a system has multiple groups are, the following line is required, otherwise we observe BSOD. Also, when we tested in a system which has 1 group by default and 32 logical processors in Server 2012R2, some WHQL tests create multiple groups after which the driver does not work properly if the below line is not included in inf file.
         HKR, Interrupt Management\Affinity Policy, GroupPolicy, %REG_DWORD%, 1

    b. Following line has been included to specifies that the device's interrupts are of high priority.
         HKR, Interrupt Management\Affinity Policy, DevicePriority, %REG_DWORD%, 3



9. As requested by HGST, the MAX_NAMESPACES is changed to 128. Correspondingly, the DUMP_BUFFER_SIZE is also changed from (5*64*1024) to ((5*64*1024) + (sizeof(NVME_LUN_EXTENSION)*MAX_NAMESPACES)). If DUMP_BUFFER_SIZE is not increased, in dump mode, in NVMeInitialize(), memory allocation for pAE->pLunExtensionTable will fail.

Also, the correct method to implement the above would be to allocate memory for pAE->pLunExtensionTable in NVMeInitCallback() in the NVMeWaitOnIdentifyCtrl state, where we get the number of Namespaces supported by the device, and remove the memory allocated for pAE->pLunExtensionTable in dump mode(NVMeInitialize()) and in normal mode(NVMePassiveInitialize()).

For this patch, we have set DUMP_BUFFER_SIZE to ((5*64*1024) + (sizeof(NVME_LUN_EXTENSION) * MAX_NAMESPACES)). Please note that as per MSDN, we should not allocate more than 32 KB for RequestedDumpBufferSize.



10. Implemented ScsiAdapterPower controltype in NVMeAdapterControl(). ScsiAdapterPower executes in <= DISPATCH_IRQL. This is supported only from Windows 8. Since we need to free the buffer and stop the timer only at IRQL <= DISPATCH_IRQL, this is implemented. Also, this will avoid executing initialization state machine in NVMeReInitializeController in ScsiRestartAdapter(DIRQL) during resume from hibernation.



We have tested following:

a. WHQL tests

b. Connect device -> execute secure erase in IOCTL path -> during secure erase execution remove device -> expectation: disk should be removed and tool(executing the secure erase) should exit gracefully

c. Connect device -> hibernation -> remove device -> resume -> expectation: disk should be removed

d. Connect device -> run IOMeter -> execute secure erase in IOCTL path -> hibernation -> remove device -> resume -> expectation: disk should be removed and tools should exit gracefully

c. OS installation




[http://ext.samsung.net/mailcheck/SeenTimeChecker?do=e7bc4897eaf7ac7a189c3eecb549349dcf4c3e2988f12776313cb48408a29da1bbef98290b9ea6844a835b19b8ec809fc7b41e955949e5c8a728c55b39cc59eacf878f9a26ce15a0]
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.openfabrics.org/pipermail/nvmewin/attachments/20160531/29da19e4/attachment.html>