From olivier.mallinger at ip-maker.com Fri Mar 4 06:47:16 2016 From: olivier.mallinger at ip-maker.com (Olivier Mallinger - IP Maker) Date: Fri, 4 Mar 2016 15:47:16 +0100 Subject: [nvmewin] Tested LBA size ? Message-ID: <56D99FF4.6090504@ip-maker.com> Hello. Does anyone already experience to use a NVMe SSD with a namespace defined with a LBA size higher or equal to 8kB. I successfully tested a device with LBA sizes of 512 bytes, 1kB, 2kB or 4kB. But when I use a LBA size of 8kB, the NVMe initialization is good but I can not see any read command access to drive after. The controller is enable but the host does not perform any access expect for CSTS read. My host page size is 4kB. Any idea (bad configuration, ...) ? I don't understand what is expecting the driver. Thanks. Regards. Olivier. -------------- next part -------------- A non-text attachment was scrubbed... Name: olivier_mallinger.vcf Type: text/x-vcard Size: 316 bytes Desc: not available URL: From raymond.c.robles at intel.com Mon Mar 7 11:12:27 2016 From: raymond.c.robles at intel.com (Robles, Raymond C) Date: Mon, 7 Mar 2016 19:12:27 +0000 Subject: [nvmewin] Tested LBA size ? In-Reply-To: <56D99FF4.6090504@ip-maker.com> References: <56D99FF4.6090504@ip-maker.com> Message-ID: <49158E750348AA499168FD41D889836072665635@fmsmsx117.amr.corp.intel.com> Hi Olivier, Windows OS only supports 512b and 4K LBA sector sizes. Using any other sectors size is untested for the Windows OFA NVMe driver... and results will be undefined and may lead to issues. Thanks, Ray -----Original Message----- From: nvmewin [mailto:nvmewin-bounces at lists.openfabrics.org] On Behalf Of Olivier Mallinger - IP Maker Sent: Friday, March 04, 2016 7:47 AM To: nvmewin at lists.openfabrics.org Subject: [nvmewin] Tested LBA size ? Hello. Does anyone already experience to use a NVMe SSD with a namespace defined with a LBA size higher or equal to 8kB. I successfully tested a device with LBA sizes of 512 bytes, 1kB, 2kB or 4kB. But when I use a LBA size of 8kB, the NVMe initialization is good but I can not see any read command access to drive after. The controller is enable but the host does not perform any access expect for CSTS read. My host page size is 4kB. Any idea (bad configuration, ...) ? I don't understand what is expecting the driver. Thanks. Regards. Olivier. From suman.p at samsung.com Tue Mar 8 02:47:53 2016 From: suman.p at samsung.com (SUMAN PRAKASH B) Date: Tue, 08 Mar 2016 10:47:53 +0000 (GMT) Subject: [nvmewin] Patch with changes for Namespace Management Message-ID: <35.D2.05149.9DDAED65@epcpsbgx2.samsung.com> An HTML attachment was scrubbed... URL: -------------- next part -------------- A non-text attachment was scrubbed... Name: 201603081618776_Z5JE7EUA.gif Type: image/gif Size: 13168 bytes Desc: not available URL: From olivier.mallinger at gmail.com Fri Mar 4 06:27:36 2016 From: olivier.mallinger at gmail.com (Olivier Mallinger) Date: Fri, 4 Mar 2016 15:27:36 +0100 Subject: [nvmewin] Tested LBA size ? Message-ID: <56D99B58.3050204@gmail.com> Hello. Does anyone already experience to use a NVMe SSD with a namespace defined with a LBA size higher or equal to 8kB. I successfully tested a device with LBA sizes of 512 bytes, 1kB, 2kB or 4kB. But when I use a LBA size of 8kB, the NVMe initialization is good but I can not see any read command access to drive after. The controller is enable but the host does not perform any access expect for CSTS read. My host page size is 4kB. Any idea (bad configuration, ...) ? I don't understand what is expecting the driver. Thanks. Regards. Olivier. From support at ip-maker.com Fri Mar 4 06:28:27 2016 From: support at ip-maker.com (Support (IP-maker)) Date: Fri, 4 Mar 2016 15:28:27 +0100 Subject: [nvmewin] Tested LBA size ? Message-ID: <56D99B8B.305@ip-maker.com> Hello. Does anyone already experience to use a NVMe SSD with a namespace defined with a LBA size higher or equal to 8kB. I successfully tested a device with LBA sizes of 512 bytes, 1kB, 2kB or 4kB. But when I use a LBA size of 8kB, the NVMe initialization is good but I can not see any read command access to drive after. The controller is enable but the host does not perform any access expect for CSTS read. My host page size is 4kB. Any idea (bad configuration, ...) ? I don't understand what is expecting the driver. Thanks. Regards. Olivier. From carolyn.d.foster at intel.com Mon Mar 14 16:35:11 2016 From: carolyn.d.foster at intel.com (Foster, Carolyn D) Date: Mon, 14 Mar 2016 23:35:11 +0000 Subject: [nvmewin] Patch with changes for Namespace Management In-Reply-To: References: Message-ID: Hi All, I have one final patch update, I tried to make minimal changes to speed up the review. Suman I did not make your suggested changes because they were non-functional changes. However, I was able to go back and update the code to resolve the issues with 0 namespaces. I incorporated some additional comments that Tom had made, and they are included as follows: nvmeInit.c: Line 1814: Modified condition in function “NVMeInitCallback” from “<=” to “==”. Line 1529/1635: Added a condition to check “NumKnownNamespaces == 0” to avoid calling identify ns if no namespaces are present. nvmeStd.c: Line 3055/3059 – Added a condition to modify the DataBufferSize for all the combinations of CNS value. nvmeStd.h: Line 83 - Added definition for 4K IDENTIFY_LIST_SIZE nvmeStat.c: Line 484 – Added a condition to avoid filling NSID to 0 in case no namespaces are present. Tom made an additional comment on issues with report luns, and I am not able to recreate the failure. The password is intelnvme Carolyn From: SUMAN PRAKASH B [mailto:suman.p at samsung.com] Sent: Tuesday, March 08, 2016 3:48 AM To: Foster, Carolyn D ; nvmewin at lists.openfabrics.org Cc: Robles, Raymond C ; tru.nguyen at ssi.samsung.com; judy.brock at ssi.samsung.com; MANOJ THAPLIYAL Subject: Re: Re: [nvmewin] Patch with changes for Namespace Management Hi Carolyn, We have reviewed the code and the changes look good, apart from the following minor comment: 1) File: nvmeStd.c, Function: NVMeIoctlNamespaceMgmt(), Line no: 3978 memset is not required as NVMeAllocatePool() will allocate memory for buffer and memset that buffer to zero. Same is applicable for pRMT->pProcGroupTbl in NVMeEnumNumaCores() in nvmeInit.c, line no:370, which is not part of Intel's patch. Anyway, we feel, if there is no other review comments from others, no need for Intel to submit updated patch after correcting the above. Samsung can make these changes when we submit our next patch after Intel's NS mgmt patch. Thanks, Suman ------- Original Message ------- Sender : Foster, Carolyn D> Date : Feb 26, 2016 21:57 (GMT+05:30) Title : Re: [nvmewin] Patch with changes for Namespace Management Hi all, I am sending the updated patch incorporating feedback I received. The changes are listed below, along with my comments about the feedback I did not address. The password is intelnvme Patch updates: - Server 2012 was failing to shut down because of mismatched calls to StorportDeviceBusy and StorportDeviceReady after delete or detach commands. - NVMeGetNamespaceStatusAndSlot(pDevExt, currentNSID, &lunId); moved inside the below condition. if (pNsMgmtDW10->SEL == NAMESPACE_MANAGEMENT_DELETE) { } - In function: NVMeIoctlNamespaceMgmt - allocate memory for pNsCtrlsList and free the memory after command completion. - In the Namespace Detach/Delete path – removed unnecessary calls to StorportNotification - In functions NVMeCompletionNsAttachment and FormatNVMGetIdentify changed PRP preparation to size of Identify Namespace - In function NVMeSetFeaturesCompletion added check for SC==INVALID_NAMESPACE_OR_FORMAT - In function NVMeSetFeaturesCompletion For the case where NS Mgmt is not supported, nsStatus should be set to ATTACHED - In function NVMeSetFeaturesCompletion In the case where NS mgmt is not supported, nsStatus will be "INVALID" Suggested feedback I did not incorporate: - I was not able to make any improvements to the driver handling for the zero attached namespace case. If the feedback is that this must be implemented for this patch, I will need two more weeks. - pAE->DriverState.NumQueuesSet – I could not find any such field in the driver - Check should validate that the SCT is GENERIC_COMMAND_STATUS – I did not change this because I’d have to add a new else condition, and I could not commit to testing that kind of a change. - I didn’t make any of the suggested changes to the LBA RANGE buffer size, because again, I could not commit to testing them - It is safe to set SEL field to NAMESPACE_MANAGEMENT_DELETE in the pNsMgmtCmd structure – as far as I can see when stepping through the code we set this, line 4102 in nvmeStd.c Carolyn From: nvmewin-bounces at lists.openfabrics.org [mailto:nvmewin-bounces at lists.openfabrics.org] On Behalf Of Foster, Carolyn D Sent: Friday, January 15, 2016 4:57 PM To: nvmewin at lists.openfabrics.org Subject: [nvmewin] Patch with changes for Namespace Management Hi all, This patch includes changes to support Namespace Management updates from the NVMe specification 1.2. This patch implements some fixes for handling non-continuous namespaces, adds handling for attached and detached namespaces, and implements new IOCTLs to create, delete, attach and detach namespaces. I have made a detailed overview of the changes in the text file in the attached zip file, the contents are also copied here below. Password is intelnvme Please let me know if you have any questions. Carolyn Foster This patch includes changes to support Namespace management, including create, delete, attach and detach namespace operations. The new functionality in this patch was tested using proprietary tools. We tested on Server 2008 R2, Server 2012 R2 and Windows 8.1 ****************** Design Assumptions ****************** 1. The numbering of namespaces need not be consecutive. 2. The namespace ID can be any number between 1 and 2^32. 3. A namespace is considered “active” only when it is created and attached to this controller. 4. A detached namespace, or one that is just created but not yet attached is considered “inactive”. 5. A non-existent, or deleted namespace is considered “invalid”. 6. An active namespace will result in one (and only one) “Online” LUN. 7. Assuming single-host, and single-controller NVMe system. ********************* Architecture Overview ********************* There are four new IOCTLs for namespace management, Create, Delete, Attach and Detach. An attached namespace will result in a visible LUN to the Windows OS. The LUN extension table has been updated to have a Namespace status: typedef enum _NS_STATUS { INVALID = 0, //Namespace ID does not exist (not known to controller). INACTIVE, //Namespace is created, but not attached to controller. ATTACHED //Namespace is created and attached to controller. } NS_STATUS; In order to properly build the LUN extension table during initialization, we made changes to identify all namespaces, and to determine each namespace’s status. These changes include new states in the Init State Machine NVMeRnningWaitOnListAttachedNs and NVMeRunningWaitOnListExistingNs. The updated state machine steps are as follows: 1. Send an Identify Namespace command with CNS set to 02h. This should return a list of all active (created and attached) namespaces. 2. Go through the list and update LUN extension entries accordingly, one entry for each namespace. Set all LUN status to online. 3. Send an Identify Namespace command with CNS set to 10h. This should return a list of all existing namespaces in the system, active and inactive. 4. Go through the list. 5. If a corresponding LUN entry exists, skip this step, as this must have been an active namespace that was covered in previous steps. LUN extension entries are populated as follows: When a namespace is created: - namespaceId is set. - nsStatus is set to “INACTIVE” - slotStatus is set to “FREE” - identifyData is partially set When a namespace is attached: - drive is pulled for namespace identify - identifyData is set accordingly - nsStatus is set to “ATTACHED” - slotStatus is set to “ONLINE” - ReadOnly is set to FALSE When a namespace is detached: - nsStatus is set to “INACTIVE” - slotStatus is set to “FREE” - ReadOnly is set to TRUE When a namespace is deleted: - The entire LUN extension entry is set to 0. There are also new reasons for the LUN to be offline: typedef enum _LUN_OFFLINE_REASON { NOT_OFFLINE, FORMAT_IN_PROGRESS, DETACH_IN_PROGRESS, DELETE_IN_PROGRESS // Add more as needed } LUN_OFFLINE_REASON; When delete or detach requests are made, the driver will call StorportDeviceBusy to pause incoming requests, and the LUN is marked as offline with the appropriate reason. When a user tries to delete an attached namespace, the driver will first send a detach command, and then the delete command. ***************** Known Limitations ***************** 1. If no namepsaces are present, the driver will fail to load. 2. If an error happens on any one namespace during initialization the driver will fail to load. The handling for these two scenarios could be strengthened and improved, which we did not get to in this patch. [cid:image001.gif at 01D17E0E.040E6830] [http://ext.samsung.net/mailcheck/SeenTimeChecker?do=22f8650517ba68655e9ebd4ee858fbd4c5713e275360f6b7b8a3fd6b4a260c0ed33a9d35f6e1735f20a30c65ae77ad69c7b41e955949e5c8a728c55b39cc59eacf878f9a26ce15a0] -------------- next part -------------- An HTML attachment was scrubbed... URL: -------------- next part -------------- A non-text attachment was scrubbed... Name: image001.gif Type: image/gif Size: 13168 bytes Desc: image001.gif URL: -------------- next part -------------- A non-text attachment was scrubbed... Name: IntelNamespaceManagementV3.zip Type: application/x-zip-compressed Size: 226124 bytes Desc: IntelNamespaceManagementV3.zip URL: From raymond.c.robles at intel.com Tue Mar 15 12:46:10 2016 From: raymond.c.robles at intel.com (Robles, Raymond C) Date: Tue, 15 Mar 2016 19:46:10 +0000 Subject: [nvmewin] Patch with changes for Namespace Management In-Reply-To: References: Message-ID: <49158E750348AA499168FD41D889836072673F7E@fmsmsx117.amr.corp.intel.com> Thanks Carolyn. I agree with your decision to speed up the review of this patch (it’s been outstanding for quite a while now due to the several revisions). The minor, non-functional requests can be modified either at a later time or via each individual pull and update. There are several other patches in line, plus the WHQL “test run” to release a driver by mid-year. Suman and Tom, Once you have reviewed and approved this latest revision, let me know and I will push the patch. Thanks, Ray From: Foster, Carolyn D Sent: Monday, March 14, 2016 4:35 PM To: suman.p at samsung.com; nvmewin at lists.openfabrics.org Cc: Robles, Raymond C ; tru.nguyen at ssi.samsung.com; judy.brock at ssi.samsung.com; MANOJ THAPLIYAL Subject: RE: Re: [nvmewin] Patch with changes for Namespace Management Hi All, I have one final patch update, I tried to make minimal changes to speed up the review. Suman I did not make your suggested changes because they were non-functional changes. However, I was able to go back and update the code to resolve the issues with 0 namespaces. I incorporated some additional comments that Tom had made, and they are included as follows: nvmeInit.c: Line 1814: Modified condition in function “NVMeInitCallback” from “<=” to “==”. Line 1529/1635: Added a condition to check “NumKnownNamespaces == 0” to avoid calling identify ns if no namespaces are present. nvmeStd.c: Line 3055/3059 – Added a condition to modify the DataBufferSize for all the combinations of CNS value. nvmeStd.h: Line 83 - Added definition for 4K IDENTIFY_LIST_SIZE nvmeStat.c: Line 484 – Added a condition to avoid filling NSID to 0 in case no namespaces are present. Tom made an additional comment on issues with report luns, and I am not able to recreate the failure. The password is intelnvme Carolyn From: SUMAN PRAKASH B [mailto:suman.p at samsung.com] Sent: Tuesday, March 08, 2016 3:48 AM To: Foster, Carolyn D >; nvmewin at lists.openfabrics.org Cc: Robles, Raymond C >; tru.nguyen at ssi.samsung.com; judy.brock at ssi.samsung.com; MANOJ THAPLIYAL > Subject: Re: Re: [nvmewin] Patch with changes for Namespace Management Hi Carolyn, We have reviewed the code and the changes look good, apart from the following minor comment: 1) File: nvmeStd.c, Function: NVMeIoctlNamespaceMgmt(), Line no: 3978 memset is not required as NVMeAllocatePool() will allocate memory for buffer and memset that buffer to zero. Same is applicable for pRMT->pProcGroupTbl in NVMeEnumNumaCores() in nvmeInit.c, line no:370, which is not part of Intel's patch. Anyway, we feel, if there is no other review comments from others, no need for Intel to submit updated patch after correcting the above. Samsung can make these changes when we submit our next patch after Intel's NS mgmt patch. Thanks, Suman ------- Original Message ------- Sender : Foster, Carolyn D> Date : Feb 26, 2016 21:57 (GMT+05:30) Title : Re: [nvmewin] Patch with changes for Namespace Management Hi all, I am sending the updated patch incorporating feedback I received. The changes are listed below, along with my comments about the feedback I did not address. The password is intelnvme Patch updates: - Server 2012 was failing to shut down because of mismatched calls to StorportDeviceBusy and StorportDeviceReady after delete or detach commands. - NVMeGetNamespaceStatusAndSlot(pDevExt, currentNSID, &lunId); moved inside the below condition. if (pNsMgmtDW10->SEL == NAMESPACE_MANAGEMENT_DELETE) { } - In function: NVMeIoctlNamespaceMgmt - allocate memory for pNsCtrlsList and free the memory after command completion. - In the Namespace Detach/Delete path – removed unnecessary calls to StorportNotification - In functions NVMeCompletionNsAttachment and FormatNVMGetIdentify changed PRP preparation to size of Identify Namespace - In function NVMeSetFeaturesCompletion added check for SC==INVALID_NAMESPACE_OR_FORMAT - In function NVMeSetFeaturesCompletion For the case where NS Mgmt is not supported, nsStatus should be set to ATTACHED - In function NVMeSetFeaturesCompletion In the case where NS mgmt is not supported, nsStatus will be "INVALID" Suggested feedback I did not incorporate: - I was not able to make any improvements to the driver handling for the zero attached namespace case. If the feedback is that this must be implemented for this patch, I will need two more weeks. - pAE->DriverState.NumQueuesSet – I could not find any such field in the driver - Check should validate that the SCT is GENERIC_COMMAND_STATUS – I did not change this because I’d have to add a new else condition, and I could not commit to testing that kind of a change. - I didn’t make any of the suggested changes to the LBA RANGE buffer size, because again, I could not commit to testing them - It is safe to set SEL field to NAMESPACE_MANAGEMENT_DELETE in the pNsMgmtCmd structure – as far as I can see when stepping through the code we set this, line 4102 in nvmeStd.c Carolyn From: nvmewin-bounces at lists.openfabrics.org [mailto:nvmewin-bounces at lists.openfabrics.org] On Behalf Of Foster, Carolyn D Sent: Friday, January 15, 2016 4:57 PM To: nvmewin at lists.openfabrics.org Subject: [nvmewin] Patch with changes for Namespace Management Hi all, This patch includes changes to support Namespace Management updates from the NVMe specification 1.2. This patch implements some fixes for handling non-continuous namespaces, adds handling for attached and detached namespaces, and implements new IOCTLs to create, delete, attach and detach namespaces. I have made a detailed overview of the changes in the text file in the attached zip file, the contents are also copied here below. Password is intelnvme Please let me know if you have any questions. Carolyn Foster This patch includes changes to support Namespace management, including create, delete, attach and detach namespace operations. The new functionality in this patch was tested using proprietary tools. We tested on Server 2008 R2, Server 2012 R2 and Windows 8.1 ****************** Design Assumptions ****************** 1. The numbering of namespaces need not be consecutive. 2. The namespace ID can be any number between 1 and 2^32. 3. A namespace is considered “active” only when it is created and attached to this controller. 4. A detached namespace, or one that is just created but not yet attached is considered “inactive”. 5. A non-existent, or deleted namespace is considered “invalid”. 6. An active namespace will result in one (and only one) “Online” LUN. 7. Assuming single-host, and single-controller NVMe system. ********************* Architecture Overview ********************* There are four new IOCTLs for namespace management, Create, Delete, Attach and Detach. An attached namespace will result in a visible LUN to the Windows OS. The LUN extension table has been updated to have a Namespace status: typedef enum _NS_STATUS { INVALID = 0, //Namespace ID does not exist (not known to controller). INACTIVE, //Namespace is created, but not attached to controller. ATTACHED //Namespace is created and attached to controller. } NS_STATUS; In order to properly build the LUN extension table during initialization, we made changes to identify all namespaces, and to determine each namespace’s status. These changes include new states in the Init State Machine NVMeRnningWaitOnListAttachedNs and NVMeRunningWaitOnListExistingNs. The updated state machine steps are as follows: 1. Send an Identify Namespace command with CNS set to 02h. This should return a list of all active (created and attached) namespaces. 2. Go through the list and update LUN extension entries accordingly, one entry for each namespace. Set all LUN status to online. 3. Send an Identify Namespace command with CNS set to 10h. This should return a list of all existing namespaces in the system, active and inactive. 4. Go through the list. 5. If a corresponding LUN entry exists, skip this step, as this must have been an active namespace that was covered in previous steps. LUN extension entries are populated as follows: When a namespace is created: - namespaceId is set. - nsStatus is set to “INACTIVE” - slotStatus is set to “FREE” - identifyData is partially set When a namespace is attached: - drive is pulled for namespace identify - identifyData is set accordingly - nsStatus is set to “ATTACHED” - slotStatus is set to “ONLINE” - ReadOnly is set to FALSE When a namespace is detached: - nsStatus is set to “INACTIVE” - slotStatus is set to “FREE” - ReadOnly is set to TRUE When a namespace is deleted: - The entire LUN extension entry is set to 0. There are also new reasons for the LUN to be offline: typedef enum _LUN_OFFLINE_REASON { NOT_OFFLINE, FORMAT_IN_PROGRESS, DETACH_IN_PROGRESS, DELETE_IN_PROGRESS // Add more as needed } LUN_OFFLINE_REASON; When delete or detach requests are made, the driver will call StorportDeviceBusy to pause incoming requests, and the LUN is marked as offline with the appropriate reason. When a user tries to delete an attached namespace, the driver will first send a detach command, and then the delete command. ***************** Known Limitations ***************** 1. If no namepsaces are present, the driver will fail to load. 2. If an error happens on any one namespace during initialization the driver will fail to load. The handling for these two scenarios could be strengthened and improved, which we did not get to in this patch. [cid:image001.gif at 01D17EB8.A58F01B0] [http://ext.samsung.net/mailcheck/SeenTimeChecker?do=22f8650517ba68655e9ebd4ee858fbd4c5713e275360f6b7b8a3fd6b4a260c0ed33a9d35f6e1735f20a30c65ae77ad69c7b41e955949e5c8a728c55b39cc59eacf878f9a26ce15a0] -------------- next part -------------- An HTML attachment was scrubbed... URL: -------------- next part -------------- A non-text attachment was scrubbed... Name: image001.gif Type: image/gif Size: 13168 bytes Desc: image001.gif URL: From thomas.freeman at hgst.com Wed Mar 16 09:55:57 2016 From: thomas.freeman at hgst.com (Thomas Freeman) Date: Wed, 16 Mar 2016 16:55:57 +0000 Subject: [nvmewin] Patch with changes for Namespace Management In-Reply-To: References: Message-ID: Carolyn, Thanks for the updates. HGST gives approval to push this patch. But, I would like to document the 2 errors I am seeing related to Scsi Report Luns. Problem 1 (Extra entry at the end of the list of LUNS) 1. Configuration - namespace #1 attached, namespace #2 detached, namespace #3 attached 2. Reset the device 3. Here is the response data I see from a Scsi Report Luns (there is an extra entry with LUN value 0): 00 00 00 18 00 00 00 00 00 00 00 00 00 00 00 00 00 01 00 00 00 00 00 00 00 00 00 00 00 00 00 00 Problem 2 (First LUN is not LUN 0) 1. Start with the configuration from problem 1. 2. Attach namespace #2 (do not reset the device) 3. Detach namespace #1 (do not reset the device) 4. Here is the response data I see from Scsi Report Luns. The first entry should be 0. There is an extra entry at the end. And the first entry specifies Lun 1. From the SNTL 1.5, Logical unit numbers shall begin with 0 00 00 00 18 00 00 00 00 00 01 00 00 00 00 00 00 00 02 00 00 00 00 00 00 00 00 00 00 00 00 00 00 5. After resetting the device, the response data now shows the first lun as 0 00 00 00 18 00 00 00 00 00 00 00 00 00 00 00 00 00 01 00 00 00 00 00 00 00 00 00 00 00 00 00 00 Tom Freeman Software Engineer, Device Manager and Driver Development Western Digital Corporation e. Thomas.freeman at hgst.com o. +1-507-322-2311 [cid:image001.jpg at 01D17F7A.CB1E3FF0] From: nvmewin [mailto:nvmewin-bounces at lists.openfabrics.org] On Behalf Of Foster, Carolyn D Sent: Monday, March 14, 2016 6:35 PM To: suman.p at samsung.com; nvmewin at lists.openfabrics.org Cc: tru.nguyen at ssi.samsung.com; MANOJ THAPLIYAL Subject: Re: [nvmewin] Patch with changes for Namespace Management Hi All, I have one final patch update, I tried to make minimal changes to speed up the review. Suman I did not make your suggested changes because they were non-functional changes. However, I was able to go back and update the code to resolve the issues with 0 namespaces. I incorporated some additional comments that Tom had made, and they are included as follows: nvmeInit.c: Line 1814: Modified condition in function “NVMeInitCallback” from “<=” to “==”. Line 1529/1635: Added a condition to check “NumKnownNamespaces == 0” to avoid calling identify ns if no namespaces are present. nvmeStd.c: Line 3055/3059 – Added a condition to modify the DataBufferSize for all the combinations of CNS value. nvmeStd.h: Line 83 - Added definition for 4K IDENTIFY_LIST_SIZE nvmeStat.c: Line 484 – Added a condition to avoid filling NSID to 0 in case no namespaces are present. Tom made an additional comment on issues with report luns, and I am not able to recreate the failure. The password is intelnvme Carolyn From: SUMAN PRAKASH B [mailto:suman.p at samsung.com] Sent: Tuesday, March 08, 2016 3:48 AM To: Foster, Carolyn D >; nvmewin at lists.openfabrics.org Cc: Robles, Raymond C >; tru.nguyen at ssi.samsung.com; judy.brock at ssi.samsung.com; MANOJ THAPLIYAL > Subject: Re: Re: [nvmewin] Patch with changes for Namespace Management Hi Carolyn, We have reviewed the code and the changes look good, apart from the following minor comment: 1) File: nvmeStd.c, Function: NVMeIoctlNamespaceMgmt(), Line no: 3978 memset is not required as NVMeAllocatePool() will allocate memory for buffer and memset that buffer to zero. Same is applicable for pRMT->pProcGroupTbl in NVMeEnumNumaCores() in nvmeInit.c, line no:370, which is not part of Intel's patch. Anyway, we feel, if there is no other review comments from others, no need for Intel to submit updated patch after correcting the above. Samsung can make these changes when we submit our next patch after Intel's NS mgmt patch. Thanks, Suman ------- Original Message ------- Sender : Foster, Carolyn D> Date : Feb 26, 2016 21:57 (GMT+05:30) Title : Re: [nvmewin] Patch with changes for Namespace Management Hi all, I am sending the updated patch incorporating feedback I received. The changes are listed below, along with my comments about the feedback I did not address. The password is intelnvme Patch updates: - Server 2012 was failing to shut down because of mismatched calls to StorportDeviceBusy and StorportDeviceReady after delete or detach commands. - NVMeGetNamespaceStatusAndSlot(pDevExt, currentNSID, &lunId); moved inside the below condition. if (pNsMgmtDW10->SEL == NAMESPACE_MANAGEMENT_DELETE) { } - In function: NVMeIoctlNamespaceMgmt - allocate memory for pNsCtrlsList and free the memory after command completion. - In the Namespace Detach/Delete path – removed unnecessary calls to StorportNotification - In functions NVMeCompletionNsAttachment and FormatNVMGetIdentify changed PRP preparation to size of Identify Namespace - In function NVMeSetFeaturesCompletion added check for SC==INVALID_NAMESPACE_OR_FORMAT - In function NVMeSetFeaturesCompletion For the case where NS Mgmt is not supported, nsStatus should be set to ATTACHED - In function NVMeSetFeaturesCompletion In the case where NS mgmt is not supported, nsStatus will be "INVALID" Suggested feedback I did not incorporate: - I was not able to make any improvements to the driver handling for the zero attached namespace case. If the feedback is that this must be implemented for this patch, I will need two more weeks. - pAE->DriverState.NumQueuesSet – I could not find any such field in the driver - Check should validate that the SCT is GENERIC_COMMAND_STATUS – I did not change this because I’d have to add a new else condition, and I could not commit to testing that kind of a change. - I didn’t make any of the suggested changes to the LBA RANGE buffer size, because again, I could not commit to testing them - It is safe to set SEL field to NAMESPACE_MANAGEMENT_DELETE in the pNsMgmtCmd structure – as far as I can see when stepping through the code we set this, line 4102 in nvmeStd.c Carolyn From: nvmewin-bounces at lists.openfabrics.org [mailto:nvmewin-bounces at lists.openfabrics.org] On Behalf Of Foster, Carolyn D Sent: Friday, January 15, 2016 4:57 PM To: nvmewin at lists.openfabrics.org Subject: [nvmewin] Patch with changes for Namespace Management Hi all, This patch includes changes to support Namespace Management updates from the NVMe specification 1.2. This patch implements some fixes for handling non-continuous namespaces, adds handling for attached and detached namespaces, and implements new IOCTLs to create, delete, attach and detach namespaces. I have made a detailed overview of the changes in the text file in the attached zip file, the contents are also copied here below. Password is intelnvme Please let me know if you have any questions. Carolyn Foster This patch includes changes to support Namespace management, including create, delete, attach and detach namespace operations. The new functionality in this patch was tested using proprietary tools. We tested on Server 2008 R2, Server 2012 R2 and Windows 8.1 ****************** Design Assumptions ****************** 1. The numbering of namespaces need not be consecutive. 2. The namespace ID can be any number between 1 and 2^32. 3. A namespace is considered “active” only when it is created and attached to this controller. 4. A detached namespace, or one that is just created but not yet attached is considered “inactive”. 5. A non-existent, or deleted namespace is considered “invalid”. 6. An active namespace will result in one (and only one) “Online” LUN. 7. Assuming single-host, and single-controller NVMe system. ********************* Architecture Overview ********************* There are four new IOCTLs for namespace management, Create, Delete, Attach and Detach. An attached namespace will result in a visible LUN to the Windows OS. The LUN extension table has been updated to have a Namespace status: typedef enum _NS_STATUS { INVALID = 0, //Namespace ID does not exist (not known to controller). INACTIVE, //Namespace is created, but not attached to controller. ATTACHED //Namespace is created and attached to controller. } NS_STATUS; In order to properly build the LUN extension table during initialization, we made changes to identify all namespaces, and to determine each namespace’s status. These changes include new states in the Init State Machine NVMeRnningWaitOnListAttachedNs and NVMeRunningWaitOnListExistingNs. The updated state machine steps are as follows: 1. Send an Identify Namespace command with CNS set to 02h. This should return a list of all active (created and attached) namespaces. 2. Go through the list and update LUN extension entries accordingly, one entry for each namespace. Set all LUN status to online. 3. Send an Identify Namespace command with CNS set to 10h. This should return a list of all existing namespaces in the system, active and inactive. 4. Go through the list. 5. If a corresponding LUN entry exists, skip this step, as this must have been an active namespace that was covered in previous steps. LUN extension entries are populated as follows: When a namespace is created: - namespaceId is set. - nsStatus is set to “INACTIVE” - slotStatus is set to “FREE” - identifyData is partially set When a namespace is attached: - drive is pulled for namespace identify - identifyData is set accordingly - nsStatus is set to “ATTACHED” - slotStatus is set to “ONLINE” - ReadOnly is set to FALSE When a namespace is detached: - nsStatus is set to “INACTIVE” - slotStatus is set to “FREE” - ReadOnly is set to TRUE When a namespace is deleted: - The entire LUN extension entry is set to 0. There are also new reasons for the LUN to be offline: typedef enum _LUN_OFFLINE_REASON { NOT_OFFLINE, FORMAT_IN_PROGRESS, DETACH_IN_PROGRESS, DELETE_IN_PROGRESS // Add more as needed } LUN_OFFLINE_REASON; When delete or detach requests are made, the driver will call StorportDeviceBusy to pause incoming requests, and the LUN is marked as offline with the appropriate reason. When a user tries to delete an attached namespace, the driver will first send a detach command, and then the delete command. ***************** Known Limitations ***************** 1. If no namepsaces are present, the driver will fail to load. 2. If an error happens on any one namespace during initialization the driver will fail to load. The handling for these two scenarios could be strengthened and improved, which we did not get to in this patch. [cid:image004.gif at 01D17F76.223AEA90] [http://ext.samsung.net/mailcheck/SeenTimeChecker?do=22f8650517ba68655e9ebd4ee858fbd4c5713e275360f6b7b8a3fd6b4a260c0ed33a9d35f6e1735f20a30c65ae77ad69c7b41e955949e5c8a728c55b39cc59eacf878f9a26ce15a0] Western Digital Corporation (and its subsidiaries) E-mail Confidentiality Notice & Disclaimer: This e-mail and any files transmitted with it may contain confidential or legally privileged information of WDC and/or its affiliates, and are intended solely for the use of the individual or entity to which they are addressed. If you are not the intended recipient, any disclosure, copying, distribution or any action taken or omitted to be taken in reliance on it, is prohibited. If you have received this e-mail in error, please notify the sender immediately and delete the e-mail in its entirety from your system. -------------- next part -------------- An HTML attachment was scrubbed... URL: -------------- next part -------------- A non-text attachment was scrubbed... Name: image004.gif Type: image/gif Size: 13168 bytes Desc: image004.gif URL: -------------- next part -------------- A non-text attachment was scrubbed... Name: image001.jpg Type: image/jpeg Size: 2938 bytes Desc: image001.jpg URL: From raymond.c.robles at intel.com Wed Mar 16 11:57:22 2016 From: raymond.c.robles at intel.com (Robles, Raymond C) Date: Wed, 16 Mar 2016 18:57:22 +0000 Subject: [nvmewin] Patch with changes for Namespace Management In-Reply-To: References: Message-ID: <49158E750348AA499168FD41D889836072675A29@fmsmsx117.amr.corp.intel.com> Hi Tom… thanks for the timely response. We’ll definitely log the two errors below associated with Report LUNs. Perhaps you can provide a patch to resolve this two issues? Now, just need approval form Suman (Samsung). Thanks, Ray From: Thomas Freeman [mailto:thomas.freeman at hgst.com] Sent: Wednesday, March 16, 2016 9:56 AM To: Foster, Carolyn D ; nvmewin at lists.openfabrics.org Cc: Robles, Raymond C Subject: RE: Re: [nvmewin] Patch with changes for Namespace Management Carolyn, Thanks for the updates. HGST gives approval to push this patch. But, I would like to document the 2 errors I am seeing related to Scsi Report Luns. Problem 1 (Extra entry at the end of the list of LUNS) 1. Configuration - namespace #1 attached, namespace #2 detached, namespace #3 attached 2. Reset the device 3. Here is the response data I see from a Scsi Report Luns (there is an extra entry with LUN value 0): 00 00 00 18 00 00 00 00 00 00 00 00 00 00 00 00 00 01 00 00 00 00 00 00 00 00 00 00 00 00 00 00 Problem 2 (First LUN is not LUN 0) 1. Start with the configuration from problem 1. 2. Attach namespace #2 (do not reset the device) 3. Detach namespace #1 (do not reset the device) 4. Here is the response data I see from Scsi Report Luns. The first entry should be 0. There is an extra entry at the end. And the first entry specifies Lun 1. From the SNTL 1.5, Logical unit numbers shall begin with 0 00 00 00 18 00 00 00 00 00 01 00 00 00 00 00 00 00 02 00 00 00 00 00 00 00 00 00 00 00 00 00 00 5. After resetting the device, the response data now shows the first lun as 0 00 00 00 18 00 00 00 00 00 00 00 00 00 00 00 00 00 01 00 00 00 00 00 00 00 00 00 00 00 00 00 00 Tom Freeman Software Engineer, Device Manager and Driver Development Western Digital Corporation e. Thomas.freeman at hgst.com o. +1-507-322-2311 [cid:image001.jpg at 01D17F7A.FDC7EA50] From: nvmewin [mailto:nvmewin-bounces at lists.openfabrics.org] On Behalf Of Foster, Carolyn D Sent: Monday, March 14, 2016 6:35 PM To: suman.p at samsung.com; nvmewin at lists.openfabrics.org Cc: tru.nguyen at ssi.samsung.com; MANOJ THAPLIYAL > Subject: Re: [nvmewin] Patch with changes for Namespace Management Hi All, I have one final patch update, I tried to make minimal changes to speed up the review. Suman I did not make your suggested changes because they were non-functional changes. However, I was able to go back and update the code to resolve the issues with 0 namespaces. I incorporated some additional comments that Tom had made, and they are included as follows: nvmeInit.c: Line 1814: Modified condition in function “NVMeInitCallback” from “<=” to “==”. Line 1529/1635: Added a condition to check “NumKnownNamespaces == 0” to avoid calling identify ns if no namespaces are present. nvmeStd.c: Line 3055/3059 – Added a condition to modify the DataBufferSize for all the combinations of CNS value. nvmeStd.h: Line 83 - Added definition for 4K IDENTIFY_LIST_SIZE nvmeStat.c: Line 484 – Added a condition to avoid filling NSID to 0 in case no namespaces are present. Tom made an additional comment on issues with report luns, and I am not able to recreate the failure. The password is intelnvme Carolyn From: SUMAN PRAKASH B [mailto:suman.p at samsung.com] Sent: Tuesday, March 08, 2016 3:48 AM To: Foster, Carolyn D >; nvmewin at lists.openfabrics.org Cc: Robles, Raymond C >; tru.nguyen at ssi.samsung.com; judy.brock at ssi.samsung.com; MANOJ THAPLIYAL > Subject: Re: Re: [nvmewin] Patch with changes for Namespace Management Hi Carolyn, We have reviewed the code and the changes look good, apart from the following minor comment: 1) File: nvmeStd.c, Function: NVMeIoctlNamespaceMgmt(), Line no: 3978 memset is not required as NVMeAllocatePool() will allocate memory for buffer and memset that buffer to zero. Same is applicable for pRMT->pProcGroupTbl in NVMeEnumNumaCores() in nvmeInit.c, line no:370, which is not part of Intel's patch. Anyway, we feel, if there is no other review comments from others, no need for Intel to submit updated patch after correcting the above. Samsung can make these changes when we submit our next patch after Intel's NS mgmt patch. Thanks, Suman ------- Original Message ------- Sender : Foster, Carolyn D> Date : Feb 26, 2016 21:57 (GMT+05:30) Title : Re: [nvmewin] Patch with changes for Namespace Management Hi all, I am sending the updated patch incorporating feedback I received. The changes are listed below, along with my comments about the feedback I did not address. The password is intelnvme Patch updates: - Server 2012 was failing to shut down because of mismatched calls to StorportDeviceBusy and StorportDeviceReady after delete or detach commands. - NVMeGetNamespaceStatusAndSlot(pDevExt, currentNSID, &lunId); moved inside the below condition. if (pNsMgmtDW10->SEL == NAMESPACE_MANAGEMENT_DELETE) { } - In function: NVMeIoctlNamespaceMgmt - allocate memory for pNsCtrlsList and free the memory after command completion. - In the Namespace Detach/Delete path – removed unnecessary calls to StorportNotification - In functions NVMeCompletionNsAttachment and FormatNVMGetIdentify changed PRP preparation to size of Identify Namespace - In function NVMeSetFeaturesCompletion added check for SC==INVALID_NAMESPACE_OR_FORMAT - In function NVMeSetFeaturesCompletion For the case where NS Mgmt is not supported, nsStatus should be set to ATTACHED - In function NVMeSetFeaturesCompletion In the case where NS mgmt is not supported, nsStatus will be "INVALID" Suggested feedback I did not incorporate: - I was not able to make any improvements to the driver handling for the zero attached namespace case. If the feedback is that this must be implemented for this patch, I will need two more weeks. - pAE->DriverState.NumQueuesSet – I could not find any such field in the driver - Check should validate that the SCT is GENERIC_COMMAND_STATUS – I did not change this because I’d have to add a new else condition, and I could not commit to testing that kind of a change. - I didn’t make any of the suggested changes to the LBA RANGE buffer size, because again, I could not commit to testing them - It is safe to set SEL field to NAMESPACE_MANAGEMENT_DELETE in the pNsMgmtCmd structure – as far as I can see when stepping through the code we set this, line 4102 in nvmeStd.c Carolyn From: nvmewin-bounces at lists.openfabrics.org [mailto:nvmewin-bounces at lists.openfabrics.org] On Behalf Of Foster, Carolyn D Sent: Friday, January 15, 2016 4:57 PM To: nvmewin at lists.openfabrics.org Subject: [nvmewin] Patch with changes for Namespace Management Hi all, This patch includes changes to support Namespace Management updates from the NVMe specification 1.2. This patch implements some fixes for handling non-continuous namespaces, adds handling for attached and detached namespaces, and implements new IOCTLs to create, delete, attach and detach namespaces. I have made a detailed overview of the changes in the text file in the attached zip file, the contents are also copied here below. Password is intelnvme Please let me know if you have any questions. Carolyn Foster This patch includes changes to support Namespace management, including create, delete, attach and detach namespace operations. The new functionality in this patch was tested using proprietary tools. We tested on Server 2008 R2, Server 2012 R2 and Windows 8.1 ****************** Design Assumptions ****************** 1. The numbering of namespaces need not be consecutive. 2. The namespace ID can be any number between 1 and 2^32. 3. A namespace is considered “active” only when it is created and attached to this controller. 4. A detached namespace, or one that is just created but not yet attached is considered “inactive”. 5. A non-existent, or deleted namespace is considered “invalid”. 6. An active namespace will result in one (and only one) “Online” LUN. 7. Assuming single-host, and single-controller NVMe system. ********************* Architecture Overview ********************* There are four new IOCTLs for namespace management, Create, Delete, Attach and Detach. An attached namespace will result in a visible LUN to the Windows OS. The LUN extension table has been updated to have a Namespace status: typedef enum _NS_STATUS { INVALID = 0, //Namespace ID does not exist (not known to controller). INACTIVE, //Namespace is created, but not attached to controller. ATTACHED //Namespace is created and attached to controller. } NS_STATUS; In order to properly build the LUN extension table during initialization, we made changes to identify all namespaces, and to determine each namespace’s status. These changes include new states in the Init State Machine NVMeRnningWaitOnListAttachedNs and NVMeRunningWaitOnListExistingNs. The updated state machine steps are as follows: 1. Send an Identify Namespace command with CNS set to 02h. This should return a list of all active (created and attached) namespaces. 2. Go through the list and update LUN extension entries accordingly, one entry for each namespace. Set all LUN status to online. 3. Send an Identify Namespace command with CNS set to 10h. This should return a list of all existing namespaces in the system, active and inactive. 4. Go through the list. 5. If a corresponding LUN entry exists, skip this step, as this must have been an active namespace that was covered in previous steps. LUN extension entries are populated as follows: When a namespace is created: - namespaceId is set. - nsStatus is set to “INACTIVE” - slotStatus is set to “FREE” - identifyData is partially set When a namespace is attached: - drive is pulled for namespace identify - identifyData is set accordingly - nsStatus is set to “ATTACHED” - slotStatus is set to “ONLINE” - ReadOnly is set to FALSE When a namespace is detached: - nsStatus is set to “INACTIVE” - slotStatus is set to “FREE” - ReadOnly is set to TRUE When a namespace is deleted: - The entire LUN extension entry is set to 0. There are also new reasons for the LUN to be offline: typedef enum _LUN_OFFLINE_REASON { NOT_OFFLINE, FORMAT_IN_PROGRESS, DETACH_IN_PROGRESS, DELETE_IN_PROGRESS // Add more as needed } LUN_OFFLINE_REASON; When delete or detach requests are made, the driver will call StorportDeviceBusy to pause incoming requests, and the LUN is marked as offline with the appropriate reason. When a user tries to delete an attached namespace, the driver will first send a detach command, and then the delete command. ***************** Known Limitations ***************** 1. If no namepsaces are present, the driver will fail to load. 2. If an error happens on any one namespace during initialization the driver will fail to load. The handling for these two scenarios could be strengthened and improved, which we did not get to in this patch. [cid:image002.gif at 01D17F7A.FDC7EA50] [http://ext.samsung.net/mailcheck/SeenTimeChecker?do=22f8650517ba68655e9ebd4ee858fbd4c5713e275360f6b7b8a3fd6b4a260c0ed33a9d35f6e1735f20a30c65ae77ad69c7b41e955949e5c8a728c55b39cc59eacf878f9a26ce15a0] -------------- next part -------------- An HTML attachment was scrubbed... URL: -------------- next part -------------- A non-text attachment was scrubbed... Name: image001.jpg Type: image/jpeg Size: 2938 bytes Desc: image001.jpg URL: -------------- next part -------------- A non-text attachment was scrubbed... Name: image002.gif Type: image/gif Size: 13168 bytes Desc: image002.gif URL: From suman.p at samsung.com Wed Mar 16 22:40:57 2016 From: suman.p at samsung.com (SUMAN PRAKASH B) Date: Thu, 17 Mar 2016 05:40:57 +0000 (GMT) Subject: [nvmewin] Patch with changes for Namespace Management Message-ID: <3A.B5.05161.9634AE65@epcpsbgx4.samsung.com> An HTML attachment was scrubbed... URL: -------------- next part -------------- A non-text attachment was scrubbed... Name: 201603171111008_Z5JE7EUA.jpg Type: image/jpeg Size: 2938 bytes Desc: not available URL: -------------- next part -------------- A non-text attachment was scrubbed... Name: 201603171111023_LK7CT9SZ.gif Type: image/gif Size: 13168 bytes Desc: not available URL: From raymond.c.robles at intel.com Thu Mar 17 15:56:59 2016 From: raymond.c.robles at intel.com (Robles, Raymond C) Date: Thu, 17 Mar 2016 22:56:59 +0000 Subject: [nvmewin] Patch with changes for Namespace Management In-Reply-To: References: Message-ID: <49158E750348AA499168FD41D889836072677296@fmsmsx117.amr.corp.intel.com> Thanks Suman! From: SUMAN PRAKASH B [mailto:suman.p at samsung.com] Sent: Wednesday, March 16, 2016 10:41 PM To: Robles, Raymond C ; 'Thomas Freeman' ; Foster, Carolyn D ; nvmewin at lists.openfabrics.org Cc: MANOJ THAPLIYAL Subject: Re: Re: [nvmewin] Patch with changes for Namespace Management Hi Ray, Carolyn, SAMSUNG approves this patch. Thanks, Suman ------- Original Message ------- Sender : Robles, Raymond C> Date : Mar 17, 2016 00:27 (GMT+05:30) Title : Re: [nvmewin] Patch with changes for Namespace Management Hi Tom… thanks for the timely response. We’ll definitely log the two errors below associated with Report LUNs. Perhaps you can provide a patch to resolve this two issues? Now, just need approval form Suman (Samsung). Thanks, Ray From: Thomas Freeman [mailto:thomas.freeman at hgst.com] Sent: Wednesday, March 16, 2016 9:56 AM To: Foster, Carolyn D >; nvmewin at lists.openfabrics.org Cc: Robles, Raymond C > Subject: RE: Re: [nvmewin] Patch with changes for Namespace Management Carolyn, Thanks for the updates. HGST gives approval to push this patch. But, I would like to document the 2 errors I am seeing related to Scsi Report Luns. Problem 1 (Extra entry at the end of the list of LUNS) 1. Configuration - namespace #1 attached, namespace #2 detached, namespace #3 attached 2. Reset the device 3. Here is the response data I see from a Scsi Report Luns (there is an extra entry with LUN value 0): 00 00 00 18 00 00 00 00 00 00 00 00 00 00 00 00 00 01 00 00 00 00 00 00 00 00 00 00 00 00 00 00 Problem 2 (First LUN is not LUN 0) 1. Start with the configuration from problem 1. 2. Attach namespace #2 (do not reset the device) 3. Detach namespace #1 (do not reset the device) 4. Here is the response data I see from Scsi Report Luns. The first entry should be 0. There is an extra entry at the end. And the first entry specifies Lun 1. From the SNTL 1.5, Logical unit numbers shall begin with 0 00 00 00 18 00 00 00 00 00 01 00 00 00 00 00 00 00 02 00 00 00 00 00 00 00 00 00 00 00 00 00 00 5. After resetting the device, the response data now shows the first lun as 0 00 00 00 18 00 00 00 00 00 00 00 00 00 00 00 00 00 01 00 00 00 00 00 00 00 00 00 00 00 00 00 00 Tom Freeman Software Engineer, Device Manager and Driver Development Western Digital Corporation e. Thomas.freeman at hgst.com o. +1-507-322-2311 [cid:image001.jpg at 01D18065.A29067C0] From: nvmewin [mailto:nvmewin-bounces at lists.openfabrics.org] On Behalf Of Foster, Carolyn D Sent: Monday, March 14, 2016 6:35 PM To: suman.p at samsung.com; nvmewin at lists.openfabrics.org Cc: tru.nguyen at ssi.samsung.com; MANOJ THAPLIYAL > Subject: Re: [nvmewin] Patch with changes for Namespace Management Hi All, I have one final patch update, I tried to make minimal changes to speed up the review. Suman I did not make your suggested changes because they were non-functional changes. However, I was able to go back and update the code to resolve the issues with 0 namespaces. I incorporated some additional comments that Tom had made, and they are included as follows: nvmeInit.c: Line 1814: Modified condition in function “NVMeInitCallback” from “<=” to “==”. Line 1529/1635: Added a condition to check “NumKnownNamespaces == 0” to avoid calling identify ns if no namespaces are present. nvmeStd.c: Line 3055/3059 – Added a condition to modify the DataBufferSize for all the combinations of CNS value. nvmeStd.h: Line 83 - Added definition for 4K IDENTIFY_LIST_SIZE nvmeStat.c: Line 484 – Added a condition to avoid filling NSID to 0 in case no namespaces are present. Tom made an additional comment on issues with report luns, and I am not able to recreate the failure. The password is intelnvme Carolyn From: SUMAN PRAKASH B [mailto:suman.p at samsung.com] Sent: Tuesday, March 08, 2016 3:48 AM To: Foster, Carolyn D >; nvmewin at lists.openfabrics.org Cc: Robles, Raymond C >; tru.nguyen at ssi.samsung.com; judy.brock at ssi.samsung.com; MANOJ THAPLIYAL > Subject: Re: Re: [nvmewin] Patch with changes for Namespace Management Hi Carolyn, We have reviewed the code and the changes look good, apart from the following minor comment: 1) File: nvmeStd.c, Function: NVMeIoctlNamespaceMgmt(), Line no: 3978 memset is not required as NVMeAllocatePool() will allocate memory for buffer and memset that buffer to zero. Same is applicable for pRMT->pProcGroupTbl in NVMeEnumNumaCores() in nvmeInit.c, line no:370, which is not part of Intel's patch. Anyway, we feel, if there is no other review comments from others, no need for Intel to submit updated patch after correcting the above. Samsung can make these changes when we submit our next patch after Intel's NS mgmt patch. Thanks, Suman ------- Original Message ------- Sender : Foster, Carolyn D> Date : Feb 26, 2016 21:57 (GMT+05:30) Title : Re: [nvmewin] Patch with changes for Namespace Management Hi all, I am sending the updated patch incorporating feedback I received. The changes are listed below, along with my comments about the feedback I did not address. The password is intelnvme Patch updates: - Server 2012 was failing to shut down because of mismatched calls to StorportDeviceBusy and StorportDeviceReady after delete or detach commands. - NVMeGetNamespaceStatusAndSlot(pDevExt, currentNSID, &lunId); moved inside the below condition. if (pNsMgmtDW10->SEL == NAMESPACE_MANAGEMENT_DELETE) { } - In function: NVMeIoctlNamespaceMgmt - allocate memory for pNsCtrlsList and free the memory after command completion. - In the Namespace Detach/Delete path – removed unnecessary calls to StorportNotification - In functions NVMeCompletionNsAttachment and FormatNVMGetIdentify changed PRP preparation to size of Identify Namespace - In function NVMeSetFeaturesCompletion added check for SC==INVALID_NAMESPACE_OR_FORMAT - In function NVMeSetFeaturesCompletion For the case where NS Mgmt is not supported, nsStatus should be set to ATTACHED - In function NVMeSetFeaturesCompletion In the case where NS mgmt is not supported, nsStatus will be "INVALID" Suggested feedback I did not incorporate: - I was not able to make any improvements to the driver handling for the zero attached namespace case. If the feedback is that this must be implemented for this patch, I will need two more weeks. - pAE->DriverState.NumQueuesSet – I could not find any such field in the driver - Check should validate that the SCT is GENERIC_COMMAND_STATUS – I did not change this because I’d have to add a new else condition, and I could not commit to testing that kind of a change. - I didn’t make any of the suggested changes to the LBA RANGE buffer size, because again, I could not commit to testing them - It is safe to set SEL field to NAMESPACE_MANAGEMENT_DELETE in the pNsMgmtCmd structure – as far as I can see when stepping through the code we set this, line 4102 in nvmeStd.c Carolyn From: nvmewin-bounces at lists.openfabrics.org [mailto:nvmewin-bounces at lists.openfabrics.org] On Behalf Of Foster, Carolyn D Sent: Friday, January 15, 2016 4:57 PM To: nvmewin at lists.openfabrics.org Subject: [nvmewin] Patch with changes for Namespace Management Hi all, This patch includes changes to support Namespace Management updates from the NVMe specification 1.2. This patch implements some fixes for handling non-continuous namespaces, adds handling for attached and detached namespaces, and implements new IOCTLs to create, delete, attach and detach namespaces. I have made a detailed overview of the changes in the text file in the attached zip file, the contents are also copied here below. Password is intelnvme Please let me know if you have any questions. Carolyn Foster This patch includes changes to support Namespace management, including create, delete, attach and detach namespace operations. The new functionality in this patch was tested using proprietary tools. We tested on Server 2008 R2, Server 2012 R2 and Windows 8.1 ****************** Design Assumptions ****************** 1. The numbering of namespaces need not be consecutive. 2. The namespace ID can be any number between 1 and 2^32. 3. A namespace is considered “active” only when it is created and attached to this controller. 4. A detached namespace, or one that is just created but not yet attached is considered “inactive”. 5. A non-existent, or deleted namespace is considered “invalid”. 6. An active namespace will result in one (and only one) “Online” LUN. 7. Assuming single-host, and single-controller NVMe system. ********************* Architecture Overview ********************* There are four new IOCTLs for namespace management, Create, Delete, Attach and Detach. An attached namespace will result in a visible LUN to the Windows OS. The LUN extension table has been updated to have a Namespace status: typedef enum _NS_STATUS { INVALID = 0, //Namespace ID does not exist (not known to controller). INACTIVE, //Namespace is created, but not attached to controller. ATTACHED //Namespace is created and attached to controller. } NS_STATUS; In order to properly build the LUN extension table during initialization, we made changes to identify all namespaces, and to determine each namespace’s status. These changes include new states in the Init State Machine NVMeRnningWaitOnListAttachedNs and NVMeRunningWaitOnListExistingNs. The updated state machine steps are as follows: 1. Send an Identify Namespace command with CNS set to 02h. This should return a list of all active (created and attached) namespaces. 2. Go through the list and update LUN extension entries accordingly, one entry for each namespace. Set all LUN status to online. 3. Send an Identify Namespace command with CNS set to 10h. This should return a list of all existing namespaces in the system, active and inactive. 4. Go through the list. 5. If a corresponding LUN entry exists, skip this step, as this must have been an active namespace that was covered in previous steps. LUN extension entries are populated as follows: When a namespace is created: - namespaceId is set. - nsStatus is set to “INACTIVE” - slotStatus is set to “FREE” - identifyData is partially set When a namespace is attached: - drive is pulled for namespace identify - identifyData is set accordingly - nsStatus is set to “ATTACHED” - slotStatus is set to “ONLINE” - ReadOnly is set to FALSE When a namespace is detached: - nsStatus is set to “INACTIVE” - slotStatus is set to “FREE” - ReadOnly is set to TRUE When a namespace is deleted: - The entire LUN extension entry is set to 0. There are also new reasons for the LUN to be offline: typedef enum _LUN_OFFLINE_REASON { NOT_OFFLINE, FORMAT_IN_PROGRESS, DETACH_IN_PROGRESS, DELETE_IN_PROGRESS // Add more as needed } LUN_OFFLINE_REASON; When delete or detach requests are made, the driver will call StorportDeviceBusy to pause incoming requests, and the LUN is marked as offline with the appropriate reason. When a user tries to delete an attached namespace, the driver will first send a detach command, and then the delete command. ***************** Known Limitations ***************** 1. If no namepsaces are present, the driver will fail to load. 2. If an error happens on any one namespace during initialization the driver will fail to load. The handling for these two scenarios could be strengthened and improved, which we did not get to in this patch. [cid:image002.gif at 01D18065.A29067C0] [http://ext.samsung.net/mailcheck/SeenTimeChecker?do=b48cc3aad7a292c11c43ae296b4bb7295b7eb5752d94bc69313cb48408a29da1bbef98290b9ea6844a835b19b8ec809fc7b41e955949e5c8a728c55b39cc59eacf878f9a26ce15a0] -------------- next part -------------- An HTML attachment was scrubbed... URL: -------------- next part -------------- A non-text attachment was scrubbed... Name: image001.jpg Type: image/jpeg Size: 2938 bytes Desc: image001.jpg URL: -------------- next part -------------- A non-text attachment was scrubbed... Name: image002.gif Type: image/gif Size: 13168 bytes Desc: image002.gif URL: From raymond.c.robles at intel.com Thu Mar 17 17:47:42 2016 From: raymond.c.robles at intel.com (Robles, Raymond C) Date: Fri, 18 Mar 2016 00:47:42 +0000 Subject: [nvmewin] Patch with changes for Namespace Management In-Reply-To: References: Message-ID: <49158E750348AA499168FD41D889836072677524@fmsmsx117.amr.corp.intel.com> Hello, I have applied the namespace management patch from Intel. Please rebase your local copies of the nvmewin repo to get the latest updates. NOTE: There have been web server difficulties on the OFA website. I’ve been working with Ken Strandberg to get these resolved. Most everything seems to be working again… however, I was unable to create a branch/tag for this path. But the patch should be applied to the trunk. Let me know if there are any questions. Thanks, Ray From: Foster, Carolyn D Sent: Monday, March 14, 2016 4:35 PM To: suman.p at samsung.com; nvmewin at lists.openfabrics.org Cc: Robles, Raymond C ; tru.nguyen at ssi.samsung.com; judy.brock at ssi.samsung.com; MANOJ THAPLIYAL Subject: RE: Re: [nvmewin] Patch with changes for Namespace Management Hi All, I have one final patch update, I tried to make minimal changes to speed up the review. Suman I did not make your suggested changes because they were non-functional changes. However, I was able to go back and update the code to resolve the issues with 0 namespaces. I incorporated some additional comments that Tom had made, and they are included as follows: nvmeInit.c: Line 1814: Modified condition in function “NVMeInitCallback” from “<=” to “==”. Line 1529/1635: Added a condition to check “NumKnownNamespaces == 0” to avoid calling identify ns if no namespaces are present. nvmeStd.c: Line 3055/3059 – Added a condition to modify the DataBufferSize for all the combinations of CNS value. nvmeStd.h: Line 83 - Added definition for 4K IDENTIFY_LIST_SIZE nvmeStat.c: Line 484 – Added a condition to avoid filling NSID to 0 in case no namespaces are present. Tom made an additional comment on issues with report luns, and I am not able to recreate the failure. The password is intelnvme Carolyn From: SUMAN PRAKASH B [mailto:suman.p at samsung.com] Sent: Tuesday, March 08, 2016 3:48 AM To: Foster, Carolyn D >; nvmewin at lists.openfabrics.org Cc: Robles, Raymond C >; tru.nguyen at ssi.samsung.com; judy.brock at ssi.samsung.com; MANOJ THAPLIYAL > Subject: Re: Re: [nvmewin] Patch with changes for Namespace Management Hi Carolyn, We have reviewed the code and the changes look good, apart from the following minor comment: 1) File: nvmeStd.c, Function: NVMeIoctlNamespaceMgmt(), Line no: 3978 memset is not required as NVMeAllocatePool() will allocate memory for buffer and memset that buffer to zero. Same is applicable for pRMT->pProcGroupTbl in NVMeEnumNumaCores() in nvmeInit.c, line no:370, which is not part of Intel's patch. Anyway, we feel, if there is no other review comments from others, no need for Intel to submit updated patch after correcting the above. Samsung can make these changes when we submit our next patch after Intel's NS mgmt patch. Thanks, Suman ------- Original Message ------- Sender : Foster, Carolyn D> Date : Feb 26, 2016 21:57 (GMT+05:30) Title : Re: [nvmewin] Patch with changes for Namespace Management Hi all, I am sending the updated patch incorporating feedback I received. The changes are listed below, along with my comments about the feedback I did not address. The password is intelnvme Patch updates: - Server 2012 was failing to shut down because of mismatched calls to StorportDeviceBusy and StorportDeviceReady after delete or detach commands. - NVMeGetNamespaceStatusAndSlot(pDevExt, currentNSID, &lunId); moved inside the below condition. if (pNsMgmtDW10->SEL == NAMESPACE_MANAGEMENT_DELETE) { } - In function: NVMeIoctlNamespaceMgmt - allocate memory for pNsCtrlsList and free the memory after command completion. - In the Namespace Detach/Delete path – removed unnecessary calls to StorportNotification - In functions NVMeCompletionNsAttachment and FormatNVMGetIdentify changed PRP preparation to size of Identify Namespace - In function NVMeSetFeaturesCompletion added check for SC==INVALID_NAMESPACE_OR_FORMAT - In function NVMeSetFeaturesCompletion For the case where NS Mgmt is not supported, nsStatus should be set to ATTACHED - In function NVMeSetFeaturesCompletion In the case where NS mgmt is not supported, nsStatus will be "INVALID" Suggested feedback I did not incorporate: - I was not able to make any improvements to the driver handling for the zero attached namespace case. If the feedback is that this must be implemented for this patch, I will need two more weeks. - pAE->DriverState.NumQueuesSet – I could not find any such field in the driver - Check should validate that the SCT is GENERIC_COMMAND_STATUS – I did not change this because I’d have to add a new else condition, and I could not commit to testing that kind of a change. - I didn’t make any of the suggested changes to the LBA RANGE buffer size, because again, I could not commit to testing them - It is safe to set SEL field to NAMESPACE_MANAGEMENT_DELETE in the pNsMgmtCmd structure – as far as I can see when stepping through the code we set this, line 4102 in nvmeStd.c Carolyn From: nvmewin-bounces at lists.openfabrics.org [mailto:nvmewin-bounces at lists.openfabrics.org] On Behalf Of Foster, Carolyn D Sent: Friday, January 15, 2016 4:57 PM To: nvmewin at lists.openfabrics.org Subject: [nvmewin] Patch with changes for Namespace Management Hi all, This patch includes changes to support Namespace Management updates from the NVMe specification 1.2. This patch implements some fixes for handling non-continuous namespaces, adds handling for attached and detached namespaces, and implements new IOCTLs to create, delete, attach and detach namespaces. I have made a detailed overview of the changes in the text file in the attached zip file, the contents are also copied here below. Password is intelnvme Please let me know if you have any questions. Carolyn Foster This patch includes changes to support Namespace management, including create, delete, attach and detach namespace operations. The new functionality in this patch was tested using proprietary tools. We tested on Server 2008 R2, Server 2012 R2 and Windows 8.1 ****************** Design Assumptions ****************** 1. The numbering of namespaces need not be consecutive. 2. The namespace ID can be any number between 1 and 2^32. 3. A namespace is considered “active” only when it is created and attached to this controller. 4. A detached namespace, or one that is just created but not yet attached is considered “inactive”. 5. A non-existent, or deleted namespace is considered “invalid”. 6. An active namespace will result in one (and only one) “Online” LUN. 7. Assuming single-host, and single-controller NVMe system. ********************* Architecture Overview ********************* There are four new IOCTLs for namespace management, Create, Delete, Attach and Detach. An attached namespace will result in a visible LUN to the Windows OS. The LUN extension table has been updated to have a Namespace status: typedef enum _NS_STATUS { INVALID = 0, //Namespace ID does not exist (not known to controller). INACTIVE, //Namespace is created, but not attached to controller. ATTACHED //Namespace is created and attached to controller. } NS_STATUS; In order to properly build the LUN extension table during initialization, we made changes to identify all namespaces, and to determine each namespace’s status. These changes include new states in the Init State Machine NVMeRnningWaitOnListAttachedNs and NVMeRunningWaitOnListExistingNs. The updated state machine steps are as follows: 1. Send an Identify Namespace command with CNS set to 02h. This should return a list of all active (created and attached) namespaces. 2. Go through the list and update LUN extension entries accordingly, one entry for each namespace. Set all LUN status to online. 3. Send an Identify Namespace command with CNS set to 10h. This should return a list of all existing namespaces in the system, active and inactive. 4. Go through the list. 5. If a corresponding LUN entry exists, skip this step, as this must have been an active namespace that was covered in previous steps. LUN extension entries are populated as follows: When a namespace is created: - namespaceId is set. - nsStatus is set to “INACTIVE” - slotStatus is set to “FREE” - identifyData is partially set When a namespace is attached: - drive is pulled for namespace identify - identifyData is set accordingly - nsStatus is set to “ATTACHED” - slotStatus is set to “ONLINE” - ReadOnly is set to FALSE When a namespace is detached: - nsStatus is set to “INACTIVE” - slotStatus is set to “FREE” - ReadOnly is set to TRUE When a namespace is deleted: - The entire LUN extension entry is set to 0. There are also new reasons for the LUN to be offline: typedef enum _LUN_OFFLINE_REASON { NOT_OFFLINE, FORMAT_IN_PROGRESS, DETACH_IN_PROGRESS, DELETE_IN_PROGRESS // Add more as needed } LUN_OFFLINE_REASON; When delete or detach requests are made, the driver will call StorportDeviceBusy to pause incoming requests, and the LUN is marked as offline with the appropriate reason. When a user tries to delete an attached namespace, the driver will first send a detach command, and then the delete command. ***************** Known Limitations ***************** 1. If no namepsaces are present, the driver will fail to load. 2. If an error happens on any one namespace during initialization the driver will fail to load. The handling for these two scenarios could be strengthened and improved, which we did not get to in this patch. [cid:image001.gif at 01D18075.19D2A820] [http://ext.samsung.net/mailcheck/SeenTimeChecker?do=22f8650517ba68655e9ebd4ee858fbd4c5713e275360f6b7b8a3fd6b4a260c0ed33a9d35f6e1735f20a30c65ae77ad69c7b41e955949e5c8a728c55b39cc59eacf878f9a26ce15a0] -------------- next part -------------- An HTML attachment was scrubbed... URL: -------------- next part -------------- A non-text attachment was scrubbed... Name: image001.gif Type: image/gif Size: 13168 bytes Desc: image001.gif URL: From raymond.c.robles at intel.com Thu Mar 17 17:56:51 2016 From: raymond.c.robles at intel.com (Robles, Raymond C) Date: Fri, 18 Mar 2016 00:56:51 +0000 Subject: [nvmewin] Q1`2016 NVMe Win Update Message-ID: <49158E750348AA499168FD41D889836072677577@fmsmsx117.amr.corp.intel.com> Hello, With this latest patch from Intel for namespace management, the OFA driver is open for the next set of patches from Samsung (outlined below). As agreed upon in the previous email chain, option #2 was selected by majority and will be the plan for 2016. After Samsung pushes their three patches, the WHCK/WHQL test suite will be on the OFA driver (by a volunteer company) against a released NVMe SSD. Of course, an official WHQL certification cannot happen with this driver since it would need to be against a company's specific NVMe device, but the WHQL test suite passing will act as our release criteria for the mid-year 2016 release. Please let me know if there are any questions or concerns around the current release plan for 2016. Thanks, Ray From: nvmewin-bounces at lists.openfabrics.org [mailto:nvmewin-bounces at lists.openfabrics.org] On Behalf Of Robles, Raymond C Sent: Thursday, January 14, 2016 4:55 PM To: 'nvmewin at lists.openfabrics.org' Subject: [nvmewin] Happy New Year... and status update Hello and Happy New Year, I hope everyone had a great holiday season and is off to a great start to the new year! As communicated last month, the patch from HGST for the SCSI multi-initiator changes has been approved and pushed. The holidays slowed down much of the progress on the OFA driver and there were several patches that did not get pushed prior to the end of the year. The list of patches remained to be pushed are as follows... * Namespace Management (Intel) * Perf Opts (Samsung) * Win 8.1 Timers (Samsung) * EOL Read Only (Samsung) * Concurrent channels (Google) The namespace management patch will be sent out for review tomorrow (look for the patch email from Carolyn)... stay tuned!!! However, once this patch is resolved, we as a community will have to make a decision on an official release strategy. The patch process and cadence was significantly slower in 2015 which leaves us with a few options. 1. Release what is in the trunk today (or after the namespace management patch)... and call that the 2015 release (albeit later than expected) 2. In lieu of an official 2015 release, we push the remaining patches listed above... and then release in ~Q2 of 2015. Basically skip a 2015 release and go right into the mid-2016 release. 3. Remove the concept of "official releases" from the OFA Windows NVMe driver and just allow people, companies, and users to pull from the OFA trunk as necessary. For #3 above, my thoughts are that because the OFA driver is not a production driver... but a reference and enabling driver, it should/could be managed as just that... a reference baseline driver that any potential user can go and grab the source, or contribute a patch. Nothing more... nothing less. For the release decision, I'll be happy to call a meeting... but we can also handle it via email as well... just let me know. Feedback from all is welcome... but I would request mandatory feedback form the 4 reviewing companies: Samsung, HGST, PMC-Sierra, and Intel. How would you like to proceed? Thanks, Ray [cid:image001.png at 01CB3870.4BB88E70] Raymond C. Robles Non-Volatile Memory Solutions Group Intel Corporation Office: 480-554-2600 Mobile: 480-399-0645 -------------- next part -------------- An HTML attachment was scrubbed... URL: -------------- next part -------------- A non-text attachment was scrubbed... Name: image001.png Type: image/png Size: 1756 bytes Desc: image001.png URL: From suman.p at samsung.com Fri Mar 18 06:58:28 2016 From: suman.p at samsung.com (SUMAN PRAKASH B) Date: Fri, 18 Mar 2016 13:58:28 +0000 (GMT) Subject: [nvmewin] Q1`2016 NVMe Win Update Message-ID: <3C.44.05140.4890CE65@epcpsbgx1.samsung.com> An HTML attachment was scrubbed... URL: -------------- next part -------------- A non-text attachment was scrubbed... Name: 201603181928226_Z5JE7EUA.png Type: image/png Size: 1756 bytes Desc: not available URL: -------------- next part -------------- A non-text attachment was scrubbed... Name: 201603181928232_LK7CT9SZ.gif Type: image/gif Size: 13168 bytes Desc: not available URL: From mschoberg at micron.com Tue Mar 22 08:28:47 2016 From: mschoberg at micron.com (Michael Schoberg (mschoberg)) Date: Tue, 22 Mar 2016 15:28:47 +0000 Subject: [nvmewin] Sense data format - fixed vs descriptor Message-ID: <5d97977f695c46b4af626954ce73470b@bowex36a.micron.com> I noticed that the sense data returned is using the "Fixed" format, yet the Control Page sets D_SENSE to 1b. I'm confused by this since the specification leads me to believe it should be returning sense data in "Descriptor" format: SPC-4 A descriptor format sense data (D_SENSE) bit set to zero specifies that the device server shall return fixed format sense data when returning sense data in the same I_T_L_Q nexus transaction as the status. A D_SENSE bit set to one specifies that the device server shall return descriptor format sense data. NVM Express: SCSI Translation Reference D_SENSE - Shall be set to 1b indicating that sense data is returned in descriptor format. Thanks. -------------- next part -------------- An HTML attachment was scrubbed... URL: From suman.p at samsung.com Wed Mar 23 05:26:37 2016 From: suman.p at samsung.com (SUMAN PRAKASH B) Date: Wed, 23 Mar 2016 12:26:37 +0000 (GMT) Subject: [nvmewin] Patch with changes for Optimizing disk initialization performance Message-ID: <9C.56.05168.D7B82F65@epcpsbgx3.samsung.com> An HTML attachment was scrubbed... URL: -------------- next part -------------- A non-text attachment was scrubbed... Name: Disk Initialization Performance Optimization.docx Type: application/octet-stream Size: 20321 bytes Desc: not available URL: -------------- next part -------------- A non-text attachment was scrubbed... Name: Samsung_DiskInitPerfOpt_v1.7z Type: application/octet-stream Size: 148520 bytes Desc: not available URL: From thomas.freeman at hgst.com Mon Mar 28 12:18:04 2016 From: thomas.freeman at hgst.com (Thomas Freeman) Date: Mon, 28 Mar 2016 19:18:04 +0000 Subject: [nvmewin] Patch with changes for Optimizing disk initialization performance Message-ID: Hi Suman, It looks good. I have a few comments here: 1. nvmeStd.c::IoCompletionRoutine, when checking for NVMeWaitOnNamespaceReady, would it be better to make this check and free the buffer in NvmeInitCallback (when processing NVMeWaitOnNamespaceReady). The check in IoCompletionRoutine is executed during the processing of every IO command, but it will only ever be TRUE during initialization. 2. I ran into a few problems, here are the details: *My device configuration: I'm testing a device that supports NS management and it has multiple namespaces. Some of those namespaces are not attached. The format of some of those namespaces is not supported by the driver (e.g. LBA contains metadata) a. With your change, in the method NVMeRunningWaitOnNamespaceReady the driver picks the next lun in the list and issues a READ to that namespace. i. If that lun is a detached namespace, the READ fails with a status code of 0xb. The driver attempts to retry until the READ is successful, but the command will never succeed. ii. During initialization, if the driver detects a namespace that is in an unsupported format, it zero's out that LUN entry, but leaves that zero'ed entry in the LUN extension list. When NVMeRunningWaitOnNamespaceReady is processing the list, it does not recognize this as a zero'ed out entry. Rather is attempts a READ from this namespace (the NSID is 0 since the init code zero'ed out that Lun list entry). The READ and all of its retries fail with a status code of 0xb. Proposed fix: 1. Before issuing a READ, ensure that namespace is attached and a valid format. If not, increment the counters and move to the next Lun. 2. Also, in NVMeInitCallback, when handling the case NVMeWaitOnNamespaceReady, instead of looking for an SC of 0x00, only issue a retry if the command fails with SC = 0x82 (NS not ready). Let me know if you have any questions. Tom Freeman Software Engineer, Device Manager and Driver Development Western Digital Corporation e. Thomas.freeman at hgst.com o. +1-507-322-2311 [cid:image003.jpg at 01D188FC.A143DB50] From: nvmewin [mailto:nvmewin-bounces at lists.openfabrics.org] On Behalf Of SUMAN PRAKASH B Sent: Wednesday, March 23, 2016 7:27 AM To: nvmewin at lists.openfabrics.org Cc: Seokhwan Kim ; MANOJ THAPLIYAL ; tru.nguyen at ssi.samsung.com Subject: [nvmewin] Patch with changes for Optimizing disk initialization performance Hi all, This patch includes changes for optimizing the disk initialization performance and relevant changes. I have made a detailed overview of the changes in the attached doc file(the contents are also copied here below) and the attached zip file contains the source code. Password is samsungnvme Please let me know if you have any questions. Thanks, Suman ****************** Disk Initialization Performance Optimization: We can use the StorPortInitializePerfOpts(), PERF_CONFIGURATION_DATA.MessageTargets which provides the array of MSI numbers corresponding to each logical processor. This is an alternative of using the Learning cores logic implemented in the OFA driver. Also this will directly reduce the time taken for the disk to be enumerated after a device hot insert. The current OFA driver does the following in its initialization path, let's say on a server which has 32 logical processors and device which supports 32 queues - 1. Identify controller 2. Identify namespace - for N number of namespaces 3. Set features - Interrupt coalescing, number of queues, lba range type. 4. Create IO completion queue - 32 commands 5. Create IO submission queue - 32 commands 6. LearnMapping - 32 Read commands 7. ReSetupQueues - 32 Delete Sub queues + 32 Delete completion queues 8. Create IO completion queue - 32 commands 9. Create IO submission queue - 32 commands 10. Complete initialization state machine As can be observed, during disk initialization, around 224 commands are processed for setting up the IO queues and associate the MSI-x number to each queues. If we use StorPortInitializePerfOpts(), we required only 64 commands instead of 224 commands. On a server which as 120 logical processors, 840 commands are required for setting up the IO queues and associate the MSI-x number to each queues. If learning cores is avoided, only 240 commands are required instead of 840 commands. Also we can fall back to learning cores if the API StorPortInitiailzePerfOpts() fails. We see improved device up time after this change. Also, if the number of queues supported by the device is less than the number of logical processors, the driver does not execute the learning cores, hence there won't be any improvement even if we use StorPortInitializePerfOpts(). Server with 32 logical processors: OFA version disk up time in seconds Disk capacity = 400 GB Disk capacity = 1.6 TB Disk from vendor 1 Disk from vendor 2 Rev 133 14 6.5 14.5 Latest 5 5 13.5 PS: data may change for different vendor SSDs Code changes: 1. Changes w.r.t StorPortInitializePerfOpts(). a. In NVMeInitialize(), moved the StorPortInitializePerfOpts() after NVMeEnumMsiMessages() to set the LastRedirectionMessageNumber in StorPortInitializePerfOpts(). b. Set the flags STOR_PERF_INTERRUPT_MESSAGE_RANGES and STOR_PERF_ADV_CONFIG_LOCALITY, and values FirstRedirectionMessageNumber, LastRedirectionMessageNumber and MessageTargets in StorPortInitializePerfOpts() to get the MSIx-Core mapping in MessageTargets. If this API returns success, the learning cores can be skipped. c. If the StorPortInitializePerfOpts() fails, in NVMeMsiMapCores(), the mapping of msix to cores in assigned sequentially, and learning cores is executed. During learning cores, in IoCompletionRoutine(), the msix to core is re-mapped. If the StorPortInitializePerfOpts() succeeds, in NVMeMsiMapCores(), the mapping of msix to cores is taken from MessageTargets and learnig cores is skipped. 2. When the learning cores is skipped, the controller initialization completes faster. But we have observed that in some devices, the Namespace is not ready to process I/O at this point. And when kernel send I/O, the device returns SC = NAMESPACE_NOT_READY and miniport returns SCSI_SENSEQ_BECOMING_READY, for which storport retries after some time. If the device takes too long to initialize the namespace, the storport gives up and shows as Uninitialized in the disk mgmt. Hence the controller initialization has to be completed after Namespace is ready. For this, a new state is introduce in the NVMeRunning(), which waits till the NS is ready. a. Introduced a new state NVMeWaitOnNamespaceReady in NVMeRunning(). b. In IoCompletionRoutine(), determine which CQ to look in based on WaitOnNamespaceReady state. c. In NVMeInitCallback(), implemented call back for NVMeWaitOnNamespaceReady. d. In IoCompletionRoutine(), free the read buffer for namespaceready. Note: a. We have observed that higher capacity Namespaces take too long to initialize, hence the passiveTimeout value in NVMePassiveInitialize() is not sufficient. We need to increase the timeout value based on vendor requirements. b. b. Checking for Namespace ready is skipped during dump/hibernation mode and resume from hibernation. 3. Usually, the number of MSIx supported by device and MSIx granted(StorPortGetMSIInfo) will be number of IO Queue + 1 Admin Queue. But, we share the Admin queue and first I/O queue in core 0, and hence MSIx 0 is shared between admin queue and first I/O queue. Incase of active cores more than Queues supported, one of the MSGID should not be considered. Made changes in In NVMeEnumMsiMessages() accordingly. For example, cores = 32, Admin + IO queues = 1 + 8, then MsgID(in NVMeEnumMsiMessages()) should be 8. 4. In IoCompletionRoutine(), for learning cores, only if MSIGranted is less than active cores, the QueueNo will be remapped in sequential manner. Otherwise QueueNo remains same as before. [http://ext.samsung.net/mailcheck/SeenTimeChecker?do=ffcf397c39a8fbf282cc457579413d8ac3ad024398f8c8597d9badbdf7e30042d1afaaba7860cdcd9564217c646641ad61e16949eaa607501b20909a04efd4d2748cfe1d4e847419cf878f9a26ce15a0] Western Digital Corporation (and its subsidiaries) E-mail Confidentiality Notice & Disclaimer: This e-mail and any files transmitted with it may contain confidential or legally privileged information of WDC and/or its affiliates, and are intended solely for the use of the individual or entity to which they are addressed. If you are not the intended recipient, any disclosure, copying, distribution or any action taken or omitted to be taken in reliance on it, is prohibited. If you have received this e-mail in error, please notify the sender immediately and delete the e-mail in its entirety from your system. -------------- next part -------------- An HTML attachment was scrubbed... URL: -------------- next part -------------- A non-text attachment was scrubbed... Name: image003.jpg Type: image/jpeg Size: 2938 bytes Desc: image003.jpg URL: From suman.p at samsung.com Tue Mar 29 07:57:46 2016 From: suman.p at samsung.com (SUMAN PRAKASH B) Date: Tue, 29 Mar 2016 14:57:46 +0000 (GMT) Subject: [nvmewin] Patch with changes for Optimizing disk initialization performance Message-ID: An HTML attachment was scrubbed... URL: -------------- next part -------------- A non-text attachment was scrubbed... Name: 201603292028218_Z5JE7EUA.jpg Type: image/jpeg Size: 2938 bytes Desc: not available URL: -------------- next part -------------- A non-text attachment was scrubbed... Name: 201603292028224_LK7CT9SZ.gif Type: image/gif Size: 13168 bytes Desc: not available URL: