From judy.brock at ssi.samsung.com Wed May 1 03:32:53 2013 From: judy.brock at ssi.samsung.com (Judy Brock-SSI) Date: Wed, 1 May 2013 10:32:53 +0000 Subject: [nvmewin] Handling pending commands when processing Format [changing from NVMe WG dist. list to OFA NVMe Windows Driver dist. list] References: <49158E750348AA499168FD41D88983606254536E@ORSMSX152.amr.corp.intel.com> <36E8D38D6B771A4BBDB1C0D800158A512FBDBF5E@SSIEXCH-MB2.ssi.samsung.com> <49158E750348AA499168FD41D889836062545566@ORSMSX152.amr.corp.intel.com> Message-ID: <36E8D38D6B771A4BBDB1C0D800158A512FD60B88@SSIEXCH-MB2.ssi.samsung.com> Hi Ray, I was wondering if you or others had any thoughts about the below. Thanks, Judy From: Judy Brock-SSI Sent: Monday, April 22, 2013 11:27 PM To: 'Robles, Raymond C'; WONMOON CHEON; ???; nvmewin at lists.openfabrics.org Subject: RE: RE: Handling pending commands when processing Format [changing from NVMe WG dist. list to OFA NVMe Windows Driver dist. list] Hi Ray, Thanks for all the details. The stuff about hot adding the namespaces back in, reissuing Identify Controller/Identify Device, etc was very easy to follow in the driver; it’s clear how & why that’s done. However I’m still having some difficulty with the finishing-up-IOs-outstanding before starting the format issue. I may be just covering ground the group covered a long time ago but I’m wondering if all the assumptions below are valid. I’m also interested in knowing how the group validated them. Here are my concerns: >> if a format comes down for a namespace, any I/O outstanding to the controller (there won’t be anything that needs to be sent… I/O will either be at the device or on the CQ) Why do we think there won’t be anything that needs to be sent? The app that sends a pass through IOCTL to do the format is presumably completely independent from say any file IO that might be going on on behalf of other apps, or even raw IO from apps like Iometer. Seems like lots of IO could still be coming in when the format IOCTL is received. > > any I/O outstanding to the controller …will simply complete via normal operation or be aborted at the controller… but Storport won’t care because the SCSI target was already removed upon initially receiving the format command. Any I/O in the CQ will be completed and handled by Storport correctly. I am wondering about two of the assumptions about timing in the paragraph above. a) I don’t think it’s true that Storport can handle commands completing for a device it no longer has a record of – ie, after the target was removed due to re-enumeration.I think it will have torn down its own structures for any old LUN(s) we had previously exposed and that would include any record it had of commands outstanding for those old LUNs. I think it isn’t going to hold on to ghost requests on behalf of devices that it no longer has a record of because it has no place to store such requests anyway at that point/ no object to associate them with. b) We are assuming that when we call StorPortNotification() with BusChangeDetected that Storport will come back in to the driver to rescan the bus – either via a bunch of Inquiry cmds or via a Report Luns cmd – and will finish all the work associated with the bus scan, updating it’s record of device topology (ie remove any SCSI target/LUNs that were assocated with the NVMe dev we are about to format) – all before it returns from our call to ScsiPortNotification and before the driver code continues on to call ProcessIO to start the real NVMe Format NVM operation. I don’t know that that is a safe correct assumption to make. The bus scan could be deferred till the driver returns. Or even if it is launched right away, could the bus scan take place on a different processor while the proc that is running through the driver just continues on its’ way? In my experience, it is the drivers and controllers’ joint responsibility to make sure that all outstanding IOs are completed back to the caller one way or the other before starting the format op. That means aborting whatever can be aborted and also making absolutely sure that no live request left over from a Namespace that has been removed ever gets completed back to the host after the Namespace has been removed. Thanks, Judy From: Robles, Raymond C [mailto:raymond.c.robles at intel.com] Sent: Thursday, April 18, 2013 6:08 PM To: Judy Brock-SSI; WONMOON CHEON; ???; nvmewin at lists.openfabrics.org Subject: RE: RE: Handling pending commands when processing Format [changing from NVMe WG dist. list to OFA NVMe Windows Driver dist. list] Hi Judy, Ahhh… I see now. I didn’t answer that question below. The format command is essentially built into the driver by a state machine. When we receive the format command we immediately issue the *hot remove* command… but that is done inline. So, once we call Storport to kick off the enumeration, we simply return back to handling the format command in NVMeStartIoProcessIoctl(). The appropriate states are set along the way to indicate progress. Once the namespace is removed from the “OS view”, then the format is processed like any other command (via ProcessIo). The callback is setup to call NVMeIoctlFormatNVMCAllback() and the variable “FormatNvmInfo->AddNamespaceNeeded” is set to TRUE so that on the completion side we remember to have the OS re-enumerate after we are done. Once the NVM format completes, the callback is invoked in the completion DPC. Then on the completion side we issue Identify Controller and Identify Namespace so that our cached driver data for the namespace(s) formatted are up to date. In the last state, after getting the Identify Namespace struct, we’ll call *hot add* which is described below. Note that at no point do we “wait” for any I/O to finish. Format is a dangerous command… especially via pass through IOCTL. We talked about this quite a bit in the beginning of developing this driver. But essentially, if a format comes down for a namespace, any I/O outstanding to the controller (there won’t be anything that needs to be sent… I/O will either be at the device or on the CQ) will simply complete via normal operation or be aborted at the controller… but Storport won’t care because the SCSI target was already removed upon initially receiving the format command. Any I/O in the CQ will be completed and handled by Storport correctly. Thanks, Ray From: Judy Brock-SSI [mailto:judy.brock at ssi.samsung.com] Sent: Thursday, April 18, 2013 4:47 PM To: Robles, Raymond C; WONMOON CHEON; ???; nvmewin at lists.openfabrics.org Subject: RE: RE: Handling pending commands when processing Format [changing from NVMe WG dist. list to OFA NVMe Windows Driver dist. list] Hi Ray, [Ray wrote] Let me know if this answers your question. I don’t think it does. What I wrote below I think was pretty much the same as what you wrote - or at least that was my intention ☺. However, the piece I couldn’t explain (cause I haven’t looked into it) is how the driver holds off the beginning of the actual format NVM operation till whatever old IOs that were already in progress for the namespace(s) before the format op request was received are completed back to the caller, aborted, or whatever - so there are no old live requests hanging around, still in the driver , before the format op begins. In other words, does the driver hold off starting the format cmd till the outstanding IOs are completed? Or do we perhaps just drop them on the floor and let the OS figure out that those requests are permanently lost/gone due to the LUNs having disappeared (my guess is, the latter is what we do)? Or do we try to abort them all? And so on. So again, we do understand how to get the OS to avoid sending new I/O requests to stale namespaces but how exactly are the old I/O reqs (those existing at the time the format request comes in ) handled? At least that is my current question ☺ Thanks, Judy From: Robles, Raymond C [mailto:raymond.c.robles at intel.com] Sent: Thursday, April 18, 2013 2:46 PM To: Judy Brock-SSI; WONMOON CHEON; ???; nvmewin at lists.openfabrics.org Subject: RE: RE: Handling pending commands when processing Format [changing from NVMe WG dist. list to OFA NVMe Windows Driver dist. list] Hello Judy/Wonmoon, Sorry for the late response. The “hot remove” state is just a state that we enter when the driver receives a Format command. Basically, this state will remove the namespace(s) from the topology by calling StorPortNotification() with BusChangeDetected. This will remove the “SCSI target/disk” associated with each namespace form the OS (because Storport will re-enumerate the controller and the driver will not expose the namespaces about to be formatted) so that the format can occur on the relevant namespaces. By signaling Windows that the namespaces have been “removed”, all I/O will be stopped by the OS. Then the format can complete. Once the format is complete, we perform the opposite action to “hot add” the namespace back into the topology by calling StorPortNotification() with BusChangeDetected… only this time, we will surface the namespace(s) again when Storport re-enumerates. Not queues are deleted in this state, no memory is de-allocated, and nothing else changes about the namespace. This is simply the first step (in a 3 step sequence) when formatting a namespace as we cannot format a namespace while the OS is aware of its presence and could be potentially sending I/O to a stale namespace config (i.e. changing LBA/sector size). Let me know if this answers your question. Thanks, Ray From: Judy Brock-SSI [mailto:judy.brock at ssi.samsung.com] Sent: Wednesday, April 17, 2013 6:21 AM To: WONMOON CHEON; 강미경; technical at nvmexpress.org Subject: RE: RE: Handling pending commands when processing Format >>Would you elaborate more about the "hot-remove" state? In this state, do you mean that all the IO SQ/CQs are deleted? Or, waiting for completions of all the outstanding IOs? The IO SQ/CQs are definitely NOT deleted. I would need to look more closely through the driver code to see how IOs previously sent to the namespaces which are marked as OFFLINE are handled/finished/quiesced. There are other folks on this thread who no doubt have more history/intimate knowledge of this driver than I do who may answer that question more quickly than I can…also, perhaps this discussion should probably be moved to the OFA driver forum since it has turned into a driver-specific thread at this point. What do folks think? Judy From: 천원문 [mailto:wm.cheon at samsung.com] Sent: Wednesday, April 17, 2013 1:20 AM To: Judy Brock-SSI; 강미경; technical at nvmexpress.org Subject: Re: RE: Handling pending commands when processing Format Hi Judy, Would you elaborate more about the "hot-remove" state? In this state, do you mean that all the IO SQ/CQs are deleted? Or, waiting for completions of all the outstanding IOs? Thanks, Wonmoon ------- Original Message ------- Sender : Judy Brock-SSI> Date : 2013-04-17 16:39 (GMT+09:00) Title : RE: Handling pending commands when processing Format Hi, I should clarify that it is not the Windows operating system – but rather the Windows OFA NVMe driver - that, from what I can see, does a “hot-remove” of all namespace(s) associated with a device before allowing a format operation to begin; “hot remove” is just the name for an internal state in the driver format nvm state machine. Before beginning the actual format op, the driver internally marks all namespaces associated with the format operation “offline”. It then notifies the OS that there has been a “bus change” event (via an OS-specific API). This in turn will cause the OS to rescan (re-enumerate) the “bus” (the pseudo SCSI bus, that is – we expose NVM namespaces as SCSI luns). Since all the pertinent namespaces have been marked offline internally, the bus rescan won’t detect any valid SCSI luns (because the driver will not report any). Hence from the OS point of view, any SCSI lun(s) previously mapped to the namespace(s) to be formatted will have disappeared/will be unaddressable while the format operation is in progress. Judy From: Judy Brock-SSI Sent: Tuesday, April 16, 2013 8:22 PM To: 'mkkang.kang at samsung.com'; technical at nvmexpress.org Subject: RE: Handling pending commands when processing Format Mikyeong, I haven’t looked at the Linux driver but I know that Windows hot-removes all namespace(s) associated with a device before allowing a format operation to begin. And a namespace can’t be removed while there is IO outstanding to it so that answers your question regarding IOs being completed before format begins. It also answers the question about requests being sent to a namespace while format is in progress – can’t happen. Thanks, Judy From: 강미경 [mailto:mkkang.kang at samsung.com] Sent: Tuesday, April 16, 2013 7:07 PM To: technical at nvmexpress.org Subject: Handling pending commands when processing Format Dear All, Format NVM command may change the Namespace repository, and it will be executed out of order like any other commands. Therefore, Format NVM command may affect other commands that are pending execution in the device, if any. 1) How does an OFA/linux driver handle 'Format NVM command'? Does a host make sure that all commands for a particular NSID are completed before sending 'Format NVM command'? 2) If a host driver does not behave like 1) above, how can a device handle other pending commands which were previously submitted in a SQ? It seems like we need an additional status code. e.g. Abort due to Namespace Format 3) Let's suppose that 'Format NVM command' is in progress. If the host driver sends subsequent commands to the namespace being formatted, should the device reply directly with a 'Namespace not Ready'? [1.0e spec] If the device does not reply directly and the format operation takes long time, then, I/O command will timeout and the host may send the reset. But if commands are responded with 'Namespace Not Ready', host may not issue the reset. Therefore, direct reply seems to be needed. [1.1 spec. ECN 001] There is Format progress indicator. The host driver can check format progress any time, therefore, there is no concern about reset during format command. Best Regards, Mikyeong Kang Kang MiKyeong Flash Memory Planning/Enabling Group, Memory Div. SAMSUNG ELECTRONICS, Co., Ltd.. Phone: 82-31-208-3857 Mobile: 82-10-9369-0177 E-mail: mkkang.kang at samsung.com [cid:image001.jpg at 01CE3C59.BF34AF60] -------------- next part -------------- An HTML attachment was scrubbed... URL: -------------- next part -------------- A non-text attachment was scrubbed... Name: image001.jpg Type: image/jpeg Size: 34869 bytes Desc: image001.jpg URL: From Dharani.Kotte at sandisk.com Wed May 1 08:55:33 2013 From: Dharani.Kotte at sandisk.com (Dharani Kotte) Date: Wed, 1 May 2013 15:55:33 +0000 Subject: [nvmewin] ***UNCHECKED*** [WARNING - ENCRYPTED ATTACHMENT NOT VIRUS SCANNED] New Patch From Sandisk References: <23EC73C80FB59046A6B7B8EB7B38265923248B63@MILMBXIP02.sdcorp.global.sandisk.com> <548C5470AAD9DA4A85D259B663190D361FFE04D2@corpmail1.na.ads.idt.com> <23EC73C80FB59046A6B7B8EB7B38265923248BD1@MILMBXIP02.sdcorp.global.sandisk.com> <49158E750348AA499168FD41D88983606254CB45@ORSMSX152.amr.corp.intel.com> Message-ID: <23EC73C80FB59046A6B7B8EB7B38265923248D8D@MILMBXIP02.sdcorp.global.sandisk.com> Hi Ray, I have incorporated the changes according to your suggestions, I tried to keep up with the coding convention please feel free to let me know if anything has to be changed. I have incorporated one more fix in the nvmeInit.c which can lead to memory corruptions and used the SrbExt for the buffer allocation. PassWd: sndk1234 Thanks, Dharani. From: Dharani Kotte Sent: Tuesday, April 30, 2013 12:03 PM To: 'Robles, Raymond C'; nvmewin at lists.openfabrics.org Subject: RE: New Patch From Sandisk Thank you for the feedback, I agree that the global buffer is not good I have made the changes to allocate this buffer from the srbext I will provide the code with the below modifications soon. Thanks, Dharani. From: Robles, Raymond C [mailto:raymond.c.robles at intel.com] Sent: Tuesday, April 30, 2013 11:52 AM To: Dharani Kotte; nvmewin at lists.openfabrics.org Subject: RE: New Patch From Sandisk Hi Dharani, Thank you for putting this patch together. Please see my feedback below: - Line 52: Note that we have a file called nvmeSntiTypes.h. This file contains all #defines. As per our existing convention, all #defines should be placed in this file (versus at the top of a .c file). - Line 3339/3412/3502/3593/3756/5391/5433/5442/5453: Our coding convention states no line be longer than 80 characters. These lines all exceed 80 characters. - Line 5410: SntiTranslateAllModePagesResponse() contains new if-else blocks. Our coding convention dictates that open curly braces be placed on the same line as the if or else statement. You can refer to other instances of the code and if-else clauses as examples. - Line 4017: SntiCreateModeDataHeader() no longer needs the SRB pointer to be passed in as a parameter since you are using a global temp buffer to build up the mode sense data. That parameter can be removed. - General: The method of using a global buffer for a single mode sense command opens up a potential race condition for multiple mode sense commands submitted on top of each other. The idea behind putting the SCSI to NVMe translation phase in the HwStorBuildIo phase was to help performance. We initially knew that we would never be sharing any data structures or memory in calls to BuildIo. There is no spinlock or protection for calls to BuildIo and MSDN makes it clear that these BuildIo calls are not synchronized and that any synchronization of memory/data must be done within the function implementations or just use StartIo instead (since Storport will acquire the StartIoSpinLock). So there is hole here w/r/t the global buffer being populated for multiple mode sense commands at the same time. I believe you have the right idea, but the correct solution is to create a temp mode sense buffer inside the SRB extension. This way, each command has its own temp mode sense buffer for building the mode sense response, and then when ready to copy over to the SRB data buffer, it's pulling from a buffer that can only be accessed by the current command (instead of a global buffer that may have been overwritten by another mode sense request call from another BuildIo call). Note that the driver collaboration group made the decision to never acquire any type of lock in BuildIo. Let me know if you have questions on this solution. The additional size to the SRB extension is not an issue. This fix is critical for this patch to be accepted. Thanks, Ray From: nvmewin-bounces at lists.openfabrics.org [mailto:nvmewin-bounces at lists.openfabrics.org] On Behalf Of Dharani Kotte Sent: Tuesday, April 30, 2013 11:14 AM To: Chang, Alex; nvmewin at lists.openfabrics.org Subject: [nvmewin] ***UNCHECKED*** [WARNING - ENCRYPTED ATTACHMENT NOT VIRUS SCANNED] New Patch From Sandisk Attaching the code with according to Alex suggestion. Passwd: sndk1234 Thanks, Dharani. From: Chang, Alex [mailto:Alex.Chang at idt.com] Sent: Tuesday, April 30, 2013 9:23 AM To: Dharani Kotte; nvmewin at lists.openfabrics.org Subject: RE: New Patch From Sandisk Hi Dharani, I'd suggest that it's safer to initialize gModeSenseBuf as all zeros each time before it is used, just in case there are some unexpected data in the buffer. Other than that, I am fine with the patch. Thanks, Alex ________________________________ From: nvmewin-bounces at lists.openfabrics.org [mailto:nvmewin-bounces at lists.openfabrics.org] On Behalf Of Dharani Kotte Sent: Tuesday, April 30, 2013 8:37 AM To: nvmewin at lists.openfabrics.org Subject: [nvmewin] New Patch From Sandisk Hi all, It's been almost two weeks after I sent out the patch. please let us know if you're okay with it. Thanks, Dharani. From: Dharani Kotte Sent: Friday, April 12, 2013 9:58 AM To: nvmewin at lists.openfabrics.org Subject: New Patch From Sandisk Hi all, I am attaching a new patch that includes the following changes: 1. In nvmeSnti.c a. Allocate a global buffer of 256bytes size for modesense return data preparation b. For any mode sense this buffer is used for modesense data along with the requested pages in it. c. Copy the required data according to the alloc_length requested in the cdb to the Srb->DataBuffer and update the dataTransferLength accordingly d. In function SntiTranslateReturnAllModePagesResponse() the modesense data header offset is calculated wrong which ends up in BSOD in some cases modified code to fix this issue Please review the changes and provide feedbacks if you have any. If nobody disagrees with the changes, I will remind Ray to merge them in two weeks. Thanks, Dharani. ________________________________ PLEASE NOTE: The information contained in this electronic mail message is intended only for the use of the designated recipient(s) named above. If the reader of this message is not the intended recipient, you are hereby notified that you have received this message in error and that any review, dissemination, distribution, or copying of this message is strictly prohibited. If you have received this communication in error, please notify the sender by telephone or e-mail (as shown above) immediately and destroy any and all copies of this message in your possession (whether hard copies or electronically stored copies). -------------- next part -------------- An HTML attachment was scrubbed... URL: -------------- next part -------------- A non-text attachment was scrubbed... Name: source_sndk_05_01_2013.zip Type: application/x-zip-compressed Size: 168229 bytes Desc: source_sndk_05_01_2013.zip URL: From barrett.n.mayes at intel.com Wed May 1 10:00:10 2013 From: barrett.n.mayes at intel.com (Mayes, Barrett N) Date: Wed, 1 May 2013 17:00:10 +0000 Subject: [nvmewin] Handling pending commands when processing Format [changing from NVMe WG dist. list to OFA NVMe Windows Driver dist. list] In-Reply-To: <36E8D38D6B771A4BBDB1C0D800158A512FD60B88@SSIEXCH-MB2.ssi.samsung.com> References: <49158E750348AA499168FD41D88983606254536E@ORSMSX152.amr.corp.intel.com> <36E8D38D6B771A4BBDB1C0D800158A512FBDBF5E@SSIEXCH-MB2.ssi.samsung.com> <49158E750348AA499168FD41D889836062545566@ORSMSX152.amr.corp.intel.com> <36E8D38D6B771A4BBDB1C0D800158A512FD60B88@SSIEXCH-MB2.ssi.samsung.com> Message-ID: I’ll take a quick stab at the first part. Will need Ray or someone with more experience on Format implementation to address the 2nd part as I’m not sure on the synchronization of completing the inquiries following the Bus Change notification and starting of the format command. a) I don’t think it’s true that Storport can handle commands completing for a device it no longer has a record of – ie, after the target was removed due to re-enumeration.I think it will have torn down its own structures for any old LUN(s) we had previously exposed and that would include any record it had of commands outstanding for those old LUNs. I think it isn’t going to hold on to ghost requests on behalf of devices that it no longer has a record of because it has no place to store such requests anyway at that point/ no object to associate them with. In general, the port driver will stall its queues, complete any IO already in flight or queued to the device then delete the device object. The process is fairly involved and is documented here: http://msdn.microsoft.com/en-us/library/windows/hardware/ff561046(v=vs.85).aspx This describes how a WDM driver handles the remove process. The other piece to this is the storport<->miniport interaction. To initiate the removal process, the miniport can signal a Bus Change Notification to cause storport to send a new set of inquiries. If a previously reported LUN is not reported in a subsequent inquiry, storport will call IoInvalidateDeviceRelations where it will in turn not report back a previously reported Device Object. See the “BusRelations Request” section here: http://msdn.microsoft.com/en-us/library/windows/hardware/ff551670(v=vs.85).aspx and docs for IRP_MN_REMOVE_DEVICE here: http://msdn.microsoft.com/en-us/library/windows/hardware/ff551738(v=vs.85).aspx. From: nvmewin-bounces at lists.openfabrics.org [mailto:nvmewin-bounces at lists.openfabrics.org] On Behalf Of Judy Brock-SSI Sent: Wednesday, May 01, 2013 3:33 AM To: Robles, Raymond C; nvmewin at lists.openfabrics.org Subject: Re: [nvmewin] Handling pending commands when processing Format [changing from NVMe WG dist. list to OFA NVMe Windows Driver dist. list] Hi Ray, I was wondering if you or others had any thoughts about the below. Thanks, Judy From: Judy Brock-SSI Sent: Monday, April 22, 2013 11:27 PM To: 'Robles, Raymond C'; WONMOON CHEON; ???; nvmewin at lists.openfabrics.org Subject: RE: RE: Handling pending commands when processing Format [changing from NVMe WG dist. list to OFA NVMe Windows Driver dist. list] Hi Ray, Thanks for all the details. The stuff about hot adding the namespaces back in, reissuing Identify Controller/Identify Device, etc was very easy to follow in the driver; it’s clear how & why that’s done. However I’m still having some difficulty with the finishing-up-IOs-outstanding before starting the format issue. I may be just covering ground the group covered a long time ago but I’m wondering if all the assumptions below are valid. I’m also interested in knowing how the group validated them. Here are my concerns: >> if a format comes down for a namespace, any I/O outstanding to the controller (there won’t be anything that needs to be sent… I/O will either be at the device or on the CQ) Why do we think there won’t be anything that needs to be sent? The app that sends a pass through IOCTL to do the format is presumably completely independent from say any file IO that might be going on on behalf of other apps, or even raw IO from apps like Iometer. Seems like lots of IO could still be coming in when the format IOCTL is received. > > any I/O outstanding to the controller …will simply complete via normal operation or be aborted at the controller… but Storport won’t care because the SCSI target was already removed upon initially receiving the format command. Any I/O in the CQ will be completed and handled by Storport correctly. I am wondering about two of the assumptions about timing in the paragraph above. a) I don’t think it’s true that Storport can handle commands completing for a device it no longer has a record of – ie, after the target was removed due to re-enumeration.I think it will have torn down its own structures for any old LUN(s) we had previously exposed and that would include any record it had of commands outstanding for those old LUNs. I think it isn’t going to hold on to ghost requests on behalf of devices that it no longer has a record of because it has no place to store such requests anyway at that point/ no object to associate them with. b) We are assuming that when we call StorPortNotification() with BusChangeDetected that Storport will come back in to the driver to rescan the bus – either via a bunch of Inquiry cmds or via a Report Luns cmd – and will finish all the work associated with the bus scan, updating it’s record of device topology (ie remove any SCSI target/LUNs that were assocated with the NVMe dev we are about to format) – all before it returns from our call to ScsiPortNotification and before the driver code continues on to call ProcessIO to start the real NVMe Format NVM operation. I don’t know that that is a safe correct assumption to make. The bus scan could be deferred till the driver returns. Or even if it is launched right away, could the bus scan take place on a different processor while the proc that is running through the driver just continues on its’ way? In my experience, it is the drivers and controllers’ joint responsibility to make sure that all outstanding IOs are completed back to the caller one way or the other before starting the format op. That means aborting whatever can be aborted and also making absolutely sure that no live request left over from a Namespace that has been removed ever gets completed back to the host after the Namespace has been removed. Thanks, Judy From: Robles, Raymond C [mailto:raymond.c.robles at intel.com] Sent: Thursday, April 18, 2013 6:08 PM To: Judy Brock-SSI; WONMOON CHEON; ???; nvmewin at lists.openfabrics.org Subject: RE: RE: Handling pending commands when processing Format [changing from NVMe WG dist. list to OFA NVMe Windows Driver dist. list] Hi Judy, Ahhh… I see now. I didn’t answer that question below. The format command is essentially built into the driver by a state machine. When we receive the format command we immediately issue the *hot remove* command… but that is done inline. So, once we call Storport to kick off the enumeration, we simply return back to handling the format command in NVMeStartIoProcessIoctl(). The appropriate states are set along the way to indicate progress. Once the namespace is removed from the “OS view”, then the format is processed like any other command (via ProcessIo). The callback is setup to call NVMeIoctlFormatNVMCAllback() and the variable “FormatNvmInfo->AddNamespaceNeeded” is set to TRUE so that on the completion side we remember to have the OS re-enumerate after we are done. Once the NVM format completes, the callback is invoked in the completion DPC. Then on the completion side we issue Identify Controller and Identify Namespace so that our cached driver data for the namespace(s) formatted are up to date. In the last state, after getting the Identify Namespace struct, we’ll call *hot add* which is described below. Note that at no point do we “wait” for any I/O to finish. Format is a dangerous command… especially via pass through IOCTL. We talked about this quite a bit in the beginning of developing this driver. But essentially, if a format comes down for a namespace, any I/O outstanding to the controller (there won’t be anything that needs to be sent… I/O will either be at the device or on the CQ) will simply complete via normal operation or be aborted at the controller… but Storport won’t care because the SCSI target was already removed upon initially receiving the format command. Any I/O in the CQ will be completed and handled by Storport correctly. Thanks, Ray From: Judy Brock-SSI [mailto:judy.brock at ssi.samsung.com] Sent: Thursday, April 18, 2013 4:47 PM To: Robles, Raymond C; WONMOON CHEON; ???; nvmewin at lists.openfabrics.org Subject: RE: RE: Handling pending commands when processing Format [changing from NVMe WG dist. list to OFA NVMe Windows Driver dist. list] Hi Ray, [Ray wrote] Let me know if this answers your question. I don’t think it does. What I wrote below I think was pretty much the same as what you wrote - or at least that was my intention ☺. However, the piece I couldn’t explain (cause I haven’t looked into it) is how the driver holds off the beginning of the actual format NVM operation till whatever old IOs that were already in progress for the namespace(s) before the format op request was received are completed back to the caller, aborted, or whatever - so there are no old live requests hanging around, still in the driver , before the format op begins. In other words, does the driver hold off starting the format cmd till the outstanding IOs are completed? Or do we perhaps just drop them on the floor and let the OS figure out that those requests are permanently lost/gone due to the LUNs having disappeared (my guess is, the latter is what we do)? Or do we try to abort them all? And so on. So again, we do understand how to get the OS to avoid sending new I/O requests to stale namespaces but how exactly are the old I/O reqs (those existing at the time the format request comes in ) handled? At least that is my current question ☺ Thanks, Judy From: Robles, Raymond C [mailto:raymond.c.robles at intel.com] Sent: Thursday, April 18, 2013 2:46 PM To: Judy Brock-SSI; WONMOON CHEON; ???; nvmewin at lists.openfabrics.org Subject: RE: RE: Handling pending commands when processing Format [changing from NVMe WG dist. list to OFA NVMe Windows Driver dist. list] Hello Judy/Wonmoon, Sorry for the late response. The “hot remove” state is just a state that we enter when the driver receives a Format command. Basically, this state will remove the namespace(s) from the topology by calling StorPortNotification() with BusChangeDetected. This will remove the “SCSI target/disk” associated with each namespace form the OS (because Storport will re-enumerate the controller and the driver will not expose the namespaces about to be formatted) so that the format can occur on the relevant namespaces. By signaling Windows that the namespaces have been “removed”, all I/O will be stopped by the OS. Then the format can complete. Once the format is complete, we perform the opposite action to “hot add” the namespace back into the topology by calling StorPortNotification() with BusChangeDetected… only this time, we will surface the namespace(s) again when Storport re-enumerates. Not queues are deleted in this state, no memory is de-allocated, and nothing else changes about the namespace. This is simply the first step (in a 3 step sequence) when formatting a namespace as we cannot format a namespace while the OS is aware of its presence and could be potentially sending I/O to a stale namespace config (i.e. changing LBA/sector size). Let me know if this answers your question. Thanks, Ray From: Judy Brock-SSI [mailto:judy.brock at ssi.samsung.com] Sent: Wednesday, April 17, 2013 6:21 AM To: WONMOON CHEON; 강미경; technical at nvmexpress.org Subject: RE: RE: Handling pending commands when processing Format >>Would you elaborate more about the "hot-remove" state? In this state, do you mean that all the IO SQ/CQs are deleted? Or, waiting for completions of all the outstanding IOs? The IO SQ/CQs are definitely NOT deleted. I would need to look more closely through the driver code to see how IOs previously sent to the namespaces which are marked as OFFLINE are handled/finished/quiesced. There are other folks on this thread who no doubt have more history/intimate knowledge of this driver than I do who may answer that question more quickly than I can…also, perhaps this discussion should probably be moved to the OFA driver forum since it has turned into a driver-specific thread at this point. What do folks think? Judy From: 천원문 [mailto:wm.cheon at samsung.com] Sent: Wednesday, April 17, 2013 1:20 AM To: Judy Brock-SSI; 강미경; technical at nvmexpress.org Subject: Re: RE: Handling pending commands when processing Format Hi Judy, Would you elaborate more about the "hot-remove" state? In this state, do you mean that all the IO SQ/CQs are deleted? Or, waiting for completions of all the outstanding IOs? Thanks, Wonmoon ------- Original Message ------- Sender : Judy Brock-SSI> Date : 2013-04-17 16:39 (GMT+09:00) Title : RE: Handling pending commands when processing Format Hi, I should clarify that it is not the Windows operating system – but rather the Windows OFA NVMe driver - that, from what I can see, does a “hot-remove” of all namespace(s) associated with a device before allowing a format operation to begin; “hot remove” is just the name for an internal state in the driver format nvm state machine. Before beginning the actual format op, the driver internally marks all namespaces associated with the format operation “offline”. It then notifies the OS that there has been a “bus change” event (via an OS-specific API). This in turn will cause the OS to rescan (re-enumerate) the “bus” (the pseudo SCSI bus, that is – we expose NVM namespaces as SCSI luns). Since all the pertinent namespaces have been marked offline internally, the bus rescan won’t detect any valid SCSI luns (because the driver will not report any). Hence from the OS point of view, any SCSI lun(s) previously mapped to the namespace(s) to be formatted will have disappeared/will be unaddressable while the format operation is in progress. Judy From: Judy Brock-SSI Sent: Tuesday, April 16, 2013 8:22 PM To: 'mkkang.kang at samsung.com'; technical at nvmexpress.org Subject: RE: Handling pending commands when processing Format Mikyeong, I haven’t looked at the Linux driver but I know that Windows hot-removes all namespace(s) associated with a device before allowing a format operation to begin. And a namespace can’t be removed while there is IO outstanding to it so that answers your question regarding IOs being completed before format begins. It also answers the question about requests being sent to a namespace while format is in progress – can’t happen. Thanks, Judy From: 강미경 [mailto:mkkang.kang at samsung.com] Sent: Tuesday, April 16, 2013 7:07 PM To: technical at nvmexpress.org Subject: Handling pending commands when processing Format Dear All, Format NVM command may change the Namespace repository, and it will be executed out of order like any other commands. Therefore, Format NVM command may affect other commands that are pending execution in the device, if any. 1) How does an OFA/linux driver handle 'Format NVM command'? Does a host make sure that all commands for a particular NSID are completed before sending 'Format NVM command'? 2) If a host driver does not behave like 1) above, how can a device handle other pending commands which were previously submitted in a SQ? It seems like we need an additional status code. e.g. Abort due to Namespace Format 3) Let's suppose that 'Format NVM command' is in progress. If the host driver sends subsequent commands to the namespace being formatted, should the device reply directly with a 'Namespace not Ready'? [1.0e spec] If the device does not reply directly and the format operation takes long time, then, I/O command will timeout and the host may send the reset. But if commands are responded with 'Namespace Not Ready', host may not issue the reset. Therefore, direct reply seems to be needed. [1.1 spec. ECN 001] There is Format progress indicator. The host driver can check format progress any time, therefore, there is no concern about reset during format command. Best Regards, Mikyeong Kang Kang MiKyeong Flash Memory Planning/Enabling Group, Memory Div. SAMSUNG ELECTRONICS, Co., Ltd.. Phone: 82-31-208-3857 Mobile: 82-10-9369-0177 E-mail: mkkang.kang at samsung.com [cid:image001.jpg at 01CE3C59.BF34AF60] -------------- next part -------------- An HTML attachment was scrubbed... URL: -------------- next part -------------- A non-text attachment was scrubbed... Name: image001.jpg Type: image/jpeg Size: 34869 bytes Desc: image001.jpg URL: From Alex.Chang at idt.com Wed May 1 10:30:20 2013 From: Alex.Chang at idt.com (Chang, Alex) Date: Wed, 1 May 2013 17:30:20 +0000 Subject: [nvmewin] Handling pending commands when processing Format [changing from NVMe WG dist. list to OFA NVMe Windows Driver dist. list] In-Reply-To: <36E8D38D6B771A4BBDB1C0D800158A512FD60B88@SSIEXCH-MB2.ssi.samsung.com> References: <49158E750348AA499168FD41D88983606254536E@ORSMSX152.amr.corp.intel.com> <36E8D38D6B771A4BBDB1C0D800158A512FBDBF5E@SSIEXCH-MB2.ssi.samsung.com> <49158E750348AA499168FD41D889836062545566@ORSMSX152.amr.corp.intel.com> <36E8D38D6B771A4BBDB1C0D800158A512FD60B88@SSIEXCH-MB2.ssi.samsung.com> Message-ID: <548C5470AAD9DA4A85D259B663190D361FFE05C9@corpmail1.na.ads.idt.com> Hi Judy, In terms of handling pending IOs before Format NVM, I agree that, after receiving Format NVM request, we need to stop accepting IOs for the specific namespace from Storport in BuildIo once we mark down the associated namespace in the LUN extension as OFFLINE. Meanwhile, ensure the completions of all the pending IOs before sending down Format NVM command to device. Apparently, there is a race condition in current implementation between Format NVM command being issued to device in StartIo and coming IOs in BuildIo, where the LUN extension is being checked to see if it's ONLINE. Thanks, Alex ________________________________ From: nvmewin-bounces at lists.openfabrics.org [mailto:nvmewin-bounces at lists.openfabrics.org] On Behalf Of Judy Brock-SSI Sent: Wednesday, May 01, 2013 3:33 AM To: Robles, Raymond C; nvmewin at lists.openfabrics.org Subject: Re: [nvmewin] Handling pending commands when processing Format [changing from NVMe WG dist. list to OFA NVMe Windows Driver dist. list] Hi Ray, I was wondering if you or others had any thoughts about the below. Thanks, Judy From: Judy Brock-SSI Sent: Monday, April 22, 2013 11:27 PM To: 'Robles, Raymond C'; WONMOON CHEON; ???; nvmewin at lists.openfabrics.org Subject: RE: RE: Handling pending commands when processing Format [changing from NVMe WG dist. list to OFA NVMe Windows Driver dist. list] Hi Ray, Thanks for all the details. The stuff about hot adding the namespaces back in, reissuing Identify Controller/Identify Device, etc was very easy to follow in the driver; it’s clear how & why that’s done. However I’m still having some difficulty with the finishing-up-IOs-outstanding before starting the format issue. I may be just covering ground the group covered a long time ago but I’m wondering if all the assumptions below are valid. I’m also interested in knowing how the group validated them. Here are my concerns: >> if a format comes down for a namespace, any I/O outstanding to the controller (there won’t be anything that needs to be sent… I/O will either be at the device or on the CQ) Why do we think there won’t be anything that needs to be sent? The app that sends a pass through IOCTL to do the format is presumably completely independent from say any file IO that might be going on on behalf of other apps, or even raw IO from apps like Iometer. Seems like lots of IO could still be coming in when the format IOCTL is received. > > any I/O outstanding to the controller …will simply complete via normal operation or be aborted at the controller… but Storport won’t care because the SCSI target was already removed upon initially receiving the format command. Any I/O in the CQ will be completed and handled by Storport correctly. I am wondering about two of the assumptions about timing in the paragraph above. a) I don’t think it’s true that Storport can handle commands completing for a device it no longer has a record of - ie, after the target was removed due to re-enumeration.I think it will have torn down its own structures for any old LUN(s) we had previously exposed and that would include any record it had of commands outstanding for those old LUNs. I think it isn’t going to hold on to ghost requests on behalf of devices that it no longer has a record of because it has no place to store such requests anyway at that point/ no object to associate them with. b) We are assuming that when we call StorPortNotification() with BusChangeDetected that Storport will come back in to the driver to rescan the bus - either via a bunch of Inquiry cmds or via a Report Luns cmd - and will finish all the work associated with the bus scan, updating it’s record of device topology (ie remove any SCSI target/LUNs that were assocated with the NVMe dev we are about to format) - all before it returns from our call to ScsiPortNotification and before the driver code continues on to call ProcessIO to start the real NVMe Format NVM operation. I don’t know that that is a safe correct assumption to make. The bus scan could be deferred till the driver returns. Or even if it is launched right away, could the bus scan take place on a different processor while the proc that is running through the driver just continues on its’ way? In my experience, it is the drivers and controllers’ joint responsibility to make sure that all outstanding IOs are completed back to the caller one way or the other before starting the format op. That means aborting whatever can be aborted and also making absolutely sure that no live request left over from a Namespace that has been removed ever gets completed back to the host after the Namespace has been removed. Thanks, Judy From: Robles, Raymond C [mailto:raymond.c.robles at intel.com] Sent: Thursday, April 18, 2013 6:08 PM To: Judy Brock-SSI; WONMOON CHEON; ???; nvmewin at lists.openfabrics.org Subject: RE: RE: Handling pending commands when processing Format [changing from NVMe WG dist. list to OFA NVMe Windows Driver dist. list] Hi Judy, Ahhh… I see now. I didn’t answer that question below. The format command is essentially built into the driver by a state machine. When we receive the format command we immediately issue the *hot remove* command… but that is done inline. So, once we call Storport to kick off the enumeration, we simply return back to handling the format command in NVMeStartIoProcessIoctl(). The appropriate states are set along the way to indicate progress. Once the namespace is removed from the “OS view”, then the format is processed like any other command (via ProcessIo). The callback is setup to call NVMeIoctlFormatNVMCAllback() and the variable “FormatNvmInfo->AddNamespaceNeeded” is set to TRUE so that on the completion side we remember to have the OS re-enumerate after we are done. Once the NVM format completes, the callback is invoked in the completion DPC. Then on the completion side we issue Identify Controller and Identify Namespace so that our cached driver data for the namespace(s) formatted are up to date. In the last state, after getting the Identify Namespace struct, we’ll call *hot add* which is described below. Note that at no point do we “wait” for any I/O to finish. Format is a dangerous command… especially via pass through IOCTL. We talked about this quite a bit in the beginning of developing this driver. But essentially, if a format comes down for a namespace, any I/O outstanding to the controller (there won’t be anything that needs to be sent… I/O will either be at the device or on the CQ) will simply complete via normal operation or be aborted at the controller… but Storport won’t care because the SCSI target was already removed upon initially receiving the format command. Any I/O in the CQ will be completed and handled by Storport correctly. Thanks, Ray From: Judy Brock-SSI [mailto:judy.brock at ssi.samsung.com] Sent: Thursday, April 18, 2013 4:47 PM To: Robles, Raymond C; WONMOON CHEON; ???; nvmewin at lists.openfabrics.org Subject: RE: RE: Handling pending commands when processing Format [changing from NVMe WG dist. list to OFA NVMe Windows Driver dist. list] Hi Ray, [Ray wrote] Let me know if this answers your question. I don’t think it does. What I wrote below I think was pretty much the same as what you wrote - or at least that was my intention :). However, the piece I couldn’t explain (cause I haven’t looked into it) is how the driver holds off the beginning of the actual format NVM operation till whatever old IOs that were already in progress for the namespace(s) before the format op request was received are completed back to the caller, aborted, or whatever - so there are no old live requests hanging around, still in the driver , before the format op begins. In other words, does the driver hold off starting the format cmd till the outstanding IOs are completed? Or do we perhaps just drop them on the floor and let the OS figure out that those requests are permanently lost/gone due to the LUNs having disappeared (my guess is, the latter is what we do)? Or do we try to abort them all? And so on. So again, we do understand how to get the OS to avoid sending new I/O requests to stale namespaces but how exactly are the old I/O reqs (those existing at the time the format request comes in ) handled? At least that is my current question :) Thanks, Judy From: Robles, Raymond C [mailto:raymond.c.robles at intel.com] Sent: Thursday, April 18, 2013 2:46 PM To: Judy Brock-SSI; WONMOON CHEON; ???; nvmewin at lists.openfabrics.org Subject: RE: RE: Handling pending commands when processing Format [changing from NVMe WG dist. list to OFA NVMe Windows Driver dist. list] Hello Judy/Wonmoon, Sorry for the late response. The “hot remove” state is just a state that we enter when the driver receives a Format command. Basically, this state will remove the namespace(s) from the topology by calling StorPortNotification() with BusChangeDetected. This will remove the “SCSI target/disk” associated with each namespace form the OS (because Storport will re-enumerate the controller and the driver will not expose the namespaces about to be formatted) so that the format can occur on the relevant namespaces. By signaling Windows that the namespaces have been “removed”, all I/O will be stopped by the OS. Then the format can complete. Once the format is complete, we perform the opposite action to “hot add” the namespace back into the topology by calling StorPortNotification() with BusChangeDetected… only this time, we will surface the namespace(s) again when Storport re-enumerates. Not queues are deleted in this state, no memory is de-allocated, and nothing else changes about the namespace. This is simply the first step (in a 3 step sequence) when formatting a namespace as we cannot format a namespace while the OS is aware of its presence and could be potentially sending I/O to a stale namespace config (i.e. changing LBA/sector size). Let me know if this answers your question. Thanks, Ray From: Judy Brock-SSI [mailto:judy.brock at ssi.samsung.com] Sent: Wednesday, April 17, 2013 6:21 AM To: WONMOON CHEON; 강미경; technical at nvmexpress.org Subject: RE: RE: Handling pending commands when processing Format >>Would you elaborate more about the "hot-remove" state? In this state, do you mean that all the IO SQ/CQs are deleted? Or, waiting for completions of all the outstanding IOs? The IO SQ/CQs are definitely NOT deleted. I would need to look more closely through the driver code to see how IOs previously sent to the namespaces which are marked as OFFLINE are handled/finished/quiesced. There are other folks on this thread who no doubt have more history/intimate knowledge of this driver than I do who may answer that question more quickly than I can…also, perhaps this discussion should probably be moved to the OFA driver forum since it has turned into a driver-specific thread at this point. What do folks think? Judy From: 천원문 [mailto:wm.cheon at samsung.com] Sent: Wednesday, April 17, 2013 1:20 AM To: Judy Brock-SSI; 강미경; technical at nvmexpress.org Subject: Re: RE: Handling pending commands when processing Format Hi Judy, Would you elaborate more about the "hot-remove" state? In this state, do you mean that all the IO SQ/CQs are deleted? Or, waiting for completions of all the outstanding IOs? Thanks, Wonmoon ------- Original Message ------- Sender : Judy Brock-SSI> Date : 2013-04-17 16:39 (GMT+09:00) Title : RE: Handling pending commands when processing Format Hi, I should clarify that it is not the Windows operating system - but rather the Windows OFA NVMe driver - that, from what I can see, does a “hot-remove” of all namespace(s) associated with a device before allowing a format operation to begin; “hot remove” is just the name for an internal state in the driver format nvm state machine. Before beginning the actual format op, the driver internally marks all namespaces associated with the format operation “offline”. It then notifies the OS that there has been a “bus change” event (via an OS-specific API). This in turn will cause the OS to rescan (re-enumerate) the “bus” (the pseudo SCSI bus, that is - we expose NVM namespaces as SCSI luns). Since all the pertinent namespaces have been marked offline internally, the bus rescan won’t detect any valid SCSI luns (because the driver will not report any). Hence from the OS point of view, any SCSI lun(s) previously mapped to the namespace(s) to be formatted will have disappeared/will be unaddressable while the format operation is in progress. Judy From: Judy Brock-SSI Sent: Tuesday, April 16, 2013 8:22 PM To: 'mkkang.kang at samsung.com'; technical at nvmexpress.org Subject: RE: Handling pending commands when processing Format Mikyeong, I haven’t looked at the Linux driver but I know that Windows hot-removes all namespace(s) associated with a device before allowing a format operation to begin. And a namespace can’t be removed while there is IO outstanding to it so that answers your question regarding IOs being completed before format begins. It also answers the question about requests being sent to a namespace while format is in progress - can’t happen. Thanks, Judy From: 강미경 [mailto:mkkang.kang at samsung.com] Sent: Tuesday, April 16, 2013 7:07 PM To: technical at nvmexpress.org Subject: Handling pending commands when processing Format Dear All, Format NVM command may change the Namespace repository, and it will be executed out of order like any other commands. Therefore, Format NVM command may affect other commands that are pending execution in the device, if any. 1) How does an OFA/linux driver handle 'Format NVM command'? Does a host make sure that all commands for a particular NSID are completed before sending 'Format NVM command'? 2) If a host driver does not behave like 1) above, how can a device handle other pending commands which were previously submitted in a SQ? It seems like we need an additional status code. e.g. Abort due to Namespace Format 3) Let's suppose that 'Format NVM command' is in progress. If the host driver sends subsequent commands to the namespace being formatted, should the device reply directly with a 'Namespace not Ready'? [1.0e spec] If the device does not reply directly and the format operation takes long time, then, I/O command will timeout and the host may send the reset. But if commands are responded with 'Namespace Not Ready', host may not issue the reset. Therefore, direct reply seems to be needed. [1.1 spec. ECN 001] There is Format progress indicator. The host driver can check format progress any time, therefore, there is no concern about reset during format command. Best Regards, Mikyeong Kang Kang MiKyeong Flash Memory Planning/Enabling Group, Memory Div. SAMSUNG ELECTRONICS, Co., Ltd.. Phone: 82-31-208-3857 Mobile: 82-10-9369-0177 E-mail: mkkang.kang at samsung.com [cid:image001.jpg at 01CE3C59.BF34AF60] -------------- next part -------------- An HTML attachment was scrubbed... URL: -------------- next part -------------- A non-text attachment was scrubbed... Name: image001.jpg Type: image/jpeg Size: 34869 bytes Desc: image001.jpg URL: From judy.brock at ssi.samsung.com Wed May 1 16:15:23 2013 From: judy.brock at ssi.samsung.com (Judy Brock-SSI) Date: Wed, 1 May 2013 23:15:23 +0000 Subject: [nvmewin] Handling pending commands when processing Format [changing from NVMe WG dist. list to OFA NVMe Windows Driver dist. list] In-Reply-To: <548C5470AAD9DA4A85D259B663190D361FFE05C9@corpmail1.na.ads.idt.com> References: <49158E750348AA499168FD41D88983606254536E@ORSMSX152.amr.corp.intel.com> <36E8D38D6B771A4BBDB1C0D800158A512FBDBF5E@SSIEXCH-MB2.ssi.samsung.com> <49158E750348AA499168FD41D889836062545566@ORSMSX152.amr.corp.intel.com> <36E8D38D6B771A4BBDB1C0D800158A512FD60B88@SSIEXCH-MB2.ssi.samsung.com> <548C5470AAD9DA4A85D259B663190D361FFE05C9@corpmail1.na.ads.idt.com> Message-ID: <36E8D38D6B771A4BBDB1C0D800158A512FD60D9D@SSIEXCH-MB2.ssi.samsung.com> Hi Alex, Thanks for the follow-up investigation. So I believe we are in agreement that the current Format NVM state machine needs to be modified as it can cause data corruption/system crashes at the present time. The design needs to be modified to add logic for the following: a) after receiving Format NVM request, ensure the completions of all the pending IOs before sending down Format NVM command to device. This includes aborting whatever can be aborted and also making absolutely sure that no live request left over from a Namespace that has been removed ever gets completed back to the host after the Namespace has been removed. b) after receiving Format NVM request, we need to stop accepting IOs for the specific namespace from Storport in BuildIo once we mark down the associated namespace in the LUN extension as OFFLINE, including eliminating the race condition in current implementation between Format NVM command being issued to device in StartIo and coming IOs in BuildIo, where the LUN extension is being checked to see if it's ONLINE. Thanks, Judy From: Chang, Alex [mailto:Alex.Chang at idt.com] Sent: Wednesday, May 01, 2013 10:30 AM To: Judy Brock-SSI; Robles, Raymond C; nvmewin at lists.openfabrics.org Subject: RE: [nvmewin] Handling pending commands when processing Format [changing from NVMe WG dist. list to OFA NVMe Windows Driver dist. list] Hi Judy, In terms of handling pending IOs before Format NVM, I agree that, after receiving Format NVM request, we need to stop accepting IOs for the specific namespace from Storport in BuildIo once we mark down the associated namespace in the LUN extension as OFFLINE. Meanwhile, ensure the completions of all the pending IOs before sending down Format NVM command to device. Apparently, there is a race condition in current implementation between Format NVM command being issued to device in StartIo and coming IOs in BuildIo, where the LUN extension is being checked to see if it's ONLINE. Thanks, Alex ________________________________ From: nvmewin-bounces at lists.openfabrics.org [mailto:nvmewin-bounces at lists.openfabrics.org] On Behalf Of Judy Brock-SSI Sent: Wednesday, May 01, 2013 3:33 AM To: Robles, Raymond C; nvmewin at lists.openfabrics.org Subject: Re: [nvmewin] Handling pending commands when processing Format [changing from NVMe WG dist. list to OFA NVMe Windows Driver dist. list] Hi Ray, I was wondering if you or others had any thoughts about the below. Thanks, Judy From: Judy Brock-SSI Sent: Monday, April 22, 2013 11:27 PM To: 'Robles, Raymond C'; WONMOON CHEON; ???; nvmewin at lists.openfabrics.org Subject: RE: RE: Handling pending commands when processing Format [changing from NVMe WG dist. list to OFA NVMe Windows Driver dist. list] Hi Ray, Thanks for all the details. The stuff about hot adding the namespaces back in, reissuing Identify Controller/Identify Device, etc was very easy to follow in the driver; it’s clear how & why that’s done. However I’m still having some difficulty with the finishing-up-IOs-outstanding before starting the format issue. I may be just covering ground the group covered a long time ago but I’m wondering if all the assumptions below are valid. I’m also interested in knowing how the group validated them. Here are my concerns: >> if a format comes down for a namespace, any I/O outstanding to the controller (there won’t be anything that needs to be sent… I/O will either be at the device or on the CQ) Why do we think there won’t be anything that needs to be sent? The app that sends a pass through IOCTL to do the format is presumably completely independent from say any file IO that might be going on on behalf of other apps, or even raw IO from apps like Iometer. Seems like lots of IO could still be coming in when the format IOCTL is received. > > any I/O outstanding to the controller …will simply complete via normal operation or be aborted at the controller… but Storport won’t care because the SCSI target was already removed upon initially receiving the format command. Any I/O in the CQ will be completed and handled by Storport correctly. I am wondering about two of the assumptions about timing in the paragraph above. a) I don’t think it’s true that Storport can handle commands completing for a device it no longer has a record of – ie, after the target was removed due to re-enumeration.I think it will have torn down its own structures for any old LUN(s) we had previously exposed and that would include any record it had of commands outstanding for those old LUNs. I think it isn’t going to hold on to ghost requests on behalf of devices that it no longer has a record of because it has no place to store such requests anyway at that point/ no object to associate them with. b) We are assuming that when we call StorPortNotification() with BusChangeDetected that Storport will come back in to the driver to rescan the bus – either via a bunch of Inquiry cmds or via a Report Luns cmd – and will finish all the work associated with the bus scan, updating it’s record of device topology (ie remove any SCSI target/LUNs that were assocated with the NVMe dev we are about to format) – all before it returns from our call to ScsiPortNotification and before the driver code continues on to call ProcessIO to start the real NVMe Format NVM operation. I don’t know that that is a safe correct assumption to make. The bus scan could be deferred till the driver returns. Or even if it is launched right away, could the bus scan take place on a different processor while the proc that is running through the driver just continues on its’ way? In my experience, it is the drivers and controllers’ joint responsibility to make sure that all outstanding IOs are completed back to the caller one way or the other before starting the format op. That means aborting whatever can be aborted and also making absolutely sure that no live request left over from a Namespace that has been removed ever gets completed back to the host after the Namespace has been removed. Thanks, Judy From: Robles, Raymond C [mailto:raymond.c.robles at intel.com] Sent: Thursday, April 18, 2013 6:08 PM To: Judy Brock-SSI; WONMOON CHEON; ???; nvmewin at lists.openfabrics.org Subject: RE: RE: Handling pending commands when processing Format [changing from NVMe WG dist. list to OFA NVMe Windows Driver dist. list] Hi Judy, Ahhh… I see now. I didn’t answer that question below. The format command is essentially built into the driver by a state machine. When we receive the format command we immediately issue the *hot remove* command… but that is done inline. So, once we call Storport to kick off the enumeration, we simply return back to handling the format command in NVMeStartIoProcessIoctl(). The appropriate states are set along the way to indicate progress. Once the namespace is removed from the “OS view”, then the format is processed like any other command (via ProcessIo). The callback is setup to call NVMeIoctlFormatNVMCAllback() and the variable “FormatNvmInfo->AddNamespaceNeeded” is set to TRUE so that on the completion side we remember to have the OS re-enumerate after we are done. Once the NVM format completes, the callback is invoked in the completion DPC. Then on the completion side we issue Identify Controller and Identify Namespace so that our cached driver data for the namespace(s) formatted are up to date. In the last state, after getting the Identify Namespace struct, we’ll call *hot add* which is described below. Note that at no point do we “wait” for any I/O to finish. Format is a dangerous command… especially via pass through IOCTL. We talked about this quite a bit in the beginning of developing this driver. But essentially, if a format comes down for a namespace, any I/O outstanding to the controller (there won’t be anything that needs to be sent… I/O will either be at the device or on the CQ) will simply complete via normal operation or be aborted at the controller… but Storport won’t care because the SCSI target was already removed upon initially receiving the format command. Any I/O in the CQ will be completed and handled by Storport correctly. Thanks, Ray From: Judy Brock-SSI [mailto:judy.brock at ssi.samsung.com] Sent: Thursday, April 18, 2013 4:47 PM To: Robles, Raymond C; WONMOON CHEON; ???; nvmewin at lists.openfabrics.org Subject: RE: RE: Handling pending commands when processing Format [changing from NVMe WG dist. list to OFA NVMe Windows Driver dist. list] Hi Ray, [Ray wrote] Let me know if this answers your question. I don’t think it does. What I wrote below I think was pretty much the same as what you wrote - or at least that was my intention ☺. However, the piece I couldn’t explain (cause I haven’t looked into it) is how the driver holds off the beginning of the actual format NVM operation till whatever old IOs that were already in progress for the namespace(s) before the format op request was received are completed back to the caller, aborted, or whatever - so there are no old live requests hanging around, still in the driver , before the format op begins. In other words, does the driver hold off starting the format cmd till the outstanding IOs are completed? Or do we perhaps just drop them on the floor and let the OS figure out that those requests are permanently lost/gone due to the LUNs having disappeared (my guess is, the latter is what we do)? Or do we try to abort them all? And so on. So again, we do understand how to get the OS to avoid sending new I/O requests to stale namespaces but how exactly are the old I/O reqs (those existing at the time the format request comes in ) handled? At least that is my current question ☺ Thanks, Judy From: Robles, Raymond C [mailto:raymond.c.robles at intel.com] Sent: Thursday, April 18, 2013 2:46 PM To: Judy Brock-SSI; WONMOON CHEON; ???; nvmewin at lists.openfabrics.org Subject: RE: RE: Handling pending commands when processing Format [changing from NVMe WG dist. list to OFA NVMe Windows Driver dist. list] Hello Judy/Wonmoon, Sorry for the late response. The “hot remove” state is just a state that we enter when the driver receives a Format command. Basically, this state will remove the namespace(s) from the topology by calling StorPortNotification() with BusChangeDetected. This will remove the “SCSI target/disk” associated with each namespace form the OS (because Storport will re-enumerate the controller and the driver will not expose the namespaces about to be formatted) so that the format can occur on the relevant namespaces. By signaling Windows that the namespaces have been “removed”, all I/O will be stopped by the OS. Then the format can complete. Once the format is complete, we perform the opposite action to “hot add” the namespace back into the topology by calling StorPortNotification() with BusChangeDetected… only this time, we will surface the namespace(s) again when Storport re-enumerates. Not queues are deleted in this state, no memory is de-allocated, and nothing else changes about the namespace. This is simply the first step (in a 3 step sequence) when formatting a namespace as we cannot format a namespace while the OS is aware of its presence and could be potentially sending I/O to a stale namespace config (i.e. changing LBA/sector size). Let me know if this answers your question. Thanks, Ray From: Judy Brock-SSI [mailto:judy.brock at ssi.samsung.com] Sent: Wednesday, April 17, 2013 6:21 AM To: WONMOON CHEON; 강미경; technical at nvmexpress.org Subject: RE: RE: Handling pending commands when processing Format >>Would you elaborate more about the "hot-remove" state? In this state, do you mean that all the IO SQ/CQs are deleted? Or, waiting for completions of all the outstanding IOs? The IO SQ/CQs are definitely NOT deleted. I would need to look more closely through the driver code to see how IOs previously sent to the namespaces which are marked as OFFLINE are handled/finished/quiesced. There are other folks on this thread who no doubt have more history/intimate knowledge of this driver than I do who may answer that question more quickly than I can…also, perhaps this discussion should probably be moved to the OFA driver forum since it has turned into a driver-specific thread at this point. What do folks think? Judy From: 천원문 [mailto:wm.cheon at samsung.com] Sent: Wednesday, April 17, 2013 1:20 AM To: Judy Brock-SSI; 강미경; technical at nvmexpress.org Subject: Re: RE: Handling pending commands when processing Format Hi Judy, Would you elaborate more about the "hot-remove" state? In this state, do you mean that all the IO SQ/CQs are deleted? Or, waiting for completions of all the outstanding IOs? Thanks, Wonmoon ------- Original Message ------- Sender : Judy Brock-SSI> Date : 2013-04-17 16:39 (GMT+09:00) Title : RE: Handling pending commands when processing Format Hi, I should clarify that it is not the Windows operating system – but rather the Windows OFA NVMe driver - that, from what I can see, does a “hot-remove” of all namespace(s) associated with a device before allowing a format operation to begin; “hot remove” is just the name for an internal state in the driver format nvm state machine. Before beginning the actual format op, the driver internally marks all namespaces associated with the format operation “offline”. It then notifies the OS that there has been a “bus change” event (via an OS-specific API). This in turn will cause the OS to rescan (re-enumerate) the “bus” (the pseudo SCSI bus, that is – we expose NVM namespaces as SCSI luns). Since all the pertinent namespaces have been marked offline internally, the bus rescan won’t detect any valid SCSI luns (because the driver will not report any). Hence from the OS point of view, any SCSI lun(s) previously mapped to the namespace(s) to be formatted will have disappeared/will be unaddressable while the format operation is in progress. Judy From: Judy Brock-SSI Sent: Tuesday, April 16, 2013 8:22 PM To: 'mkkang.kang at samsung.com'; technical at nvmexpress.org Subject: RE: Handling pending commands when processing Format Mikyeong, I haven’t looked at the Linux driver but I know that Windows hot-removes all namespace(s) associated with a device before allowing a format operation to begin. And a namespace can’t be removed while there is IO outstanding to it so that answers your question regarding IOs being completed before format begins. It also answers the question about requests being sent to a namespace while format is in progress – can’t happen. Thanks, Judy From: 강미경 [mailto:mkkang.kang at samsung.com] Sent: Tuesday, April 16, 2013 7:07 PM To: technical at nvmexpress.org Subject: Handling pending commands when processing Format Dear All, Format NVM command may change the Namespace repository, and it will be executed out of order like any other commands. Therefore, Format NVM command may affect other commands that are pending execution in the device, if any. 1) How does an OFA/linux driver handle 'Format NVM command'? Does a host make sure that all commands for a particular NSID are completed before sending 'Format NVM command'? 2) If a host driver does not behave like 1) above, how can a device handle other pending commands which were previously submitted in a SQ? It seems like we need an additional status code. e.g. Abort due to Namespace Format 3) Let's suppose that 'Format NVM command' is in progress. If the host driver sends subsequent commands to the namespace being formatted, should the device reply directly with a 'Namespace not Ready'? [1.0e spec] If the device does not reply directly and the format operation takes long time, then, I/O command will timeout and the host may send the reset. But if commands are responded with 'Namespace Not Ready', host may not issue the reset. Therefore, direct reply seems to be needed. [1.1 spec. ECN 001] There is Format progress indicator. The host driver can check format progress any time, therefore, there is no concern about reset during format command. Best Regards, Mikyeong Kang Kang MiKyeong Flash Memory Planning/Enabling Group, Memory Div. SAMSUNG ELECTRONICS, Co., Ltd.. Phone: 82-31-208-3857 Mobile: 82-10-9369-0177 E-mail: mkkang.kang at samsung.com [cid:image001.jpg at 01CE3C59.BF34AF60] -------------- next part -------------- An HTML attachment was scrubbed... URL: -------------- next part -------------- A non-text attachment was scrubbed... Name: image001.jpg Type: image/jpeg Size: 34869 bytes Desc: image001.jpg URL: From judy.brock at ssi.samsung.com Wed May 1 16:28:37 2013 From: judy.brock at ssi.samsung.com (Judy Brock-SSI) Date: Wed, 1 May 2013 23:28:37 +0000 Subject: [nvmewin] Handling pending commands when processing Format [changing from NVMe WG dist. list to OFA NVMe Windows Driver dist. list] In-Reply-To: References: <49158E750348AA499168FD41D88983606254536E@ORSMSX152.amr.corp.intel.com> <36E8D38D6B771A4BBDB1C0D800158A512FBDBF5E@SSIEXCH-MB2.ssi.samsung.com> <49158E750348AA499168FD41D889836062545566@ORSMSX152.amr.corp.intel.com> <36E8D38D6B771A4BBDB1C0D800158A512FD60B88@SSIEXCH-MB2.ssi.samsung.com> Message-ID: <36E8D38D6B771A4BBDB1C0D800158A512FD60DDA@SSIEXCH-MB2.ssi.samsung.com> Hi Barret, To sum up the info you provide below, I think it confirms the following point I was trying to make: the miniport cannot assume that, after signally a Bus Change Notification to initiate the removal process, once the call to signal the Bus Change Notification returns, the upper layers will already have done all the work involved in the device removal process. In fact, that there is a lot of work involved in device removal (for ex, we both cited the new set of inquires to come in to the miniport which allow the miniport to signal that a previously reported LUN is no longer present). Thanks, Judy From: Mayes, Barrett N [mailto:barrett.n.mayes at intel.com] Sent: Wednesday, May 01, 2013 10:00 AM To: Judy Brock-SSI; Robles, Raymond C; nvmewin at lists.openfabrics.org Subject: RE: [nvmewin] Handling pending commands when processing Format [changing from NVMe WG dist. list to OFA NVMe Windows Driver dist. list] I’ll take a quick stab at the first part. Will need Ray or someone with more experience on Format implementation to address the 2nd part as I’m not sure on the synchronization of completing the inquiries following the Bus Change notification and starting of the format command. a) I don’t think it’s true that Storport can handle commands completing for a device it no longer has a record of – ie, after the target was removed due to re-enumeration.I think it will have torn down its own structures for any old LUN(s) we had previously exposed and that would include any record it had of commands outstanding for those old LUNs. I think it isn’t going to hold on to ghost requests on behalf of devices that it no longer has a record of because it has no place to store such requests anyway at that point/ no object to associate them with. In general, the port driver will stall its queues, complete any IO already in flight or queued to the device then delete the device object. The process is fairly involved and is documented here: http://msdn.microsoft.com/en-us/library/windows/hardware/ff561046(v=vs.85).aspx This describes how a WDM driver handles the remove process. The other piece to this is the storport<->miniport interaction. To initiate the removal process, the miniport can signal a Bus Change Notification to cause storport to send a new set of inquiries. If a previously reported LUN is not reported in a subsequent inquiry, storport will call IoInvalidateDeviceRelations where it will in turn not report back a previously reported Device Object. See the “BusRelations Request” section here: http://msdn.microsoft.com/en-us/library/windows/hardware/ff551670(v=vs.85).aspx and docs for IRP_MN_REMOVE_DEVICE here: http://msdn.microsoft.com/en-us/library/windows/hardware/ff551738(v=vs.85).aspx. From: nvmewin-bounces at lists.openfabrics.org [mailto:nvmewin-bounces at lists.openfabrics.org] On Behalf Of Judy Brock-SSI Sent: Wednesday, May 01, 2013 3:33 AM To: Robles, Raymond C; nvmewin at lists.openfabrics.org Subject: Re: [nvmewin] Handling pending commands when processing Format [changing from NVMe WG dist. list to OFA NVMe Windows Driver dist. list] Hi Ray, I was wondering if you or others had any thoughts about the below. Thanks, Judy From: Judy Brock-SSI Sent: Monday, April 22, 2013 11:27 PM To: 'Robles, Raymond C'; WONMOON CHEON; ???; nvmewin at lists.openfabrics.org Subject: RE: RE: Handling pending commands when processing Format [changing from NVMe WG dist. list to OFA NVMe Windows Driver dist. list] Hi Ray, Thanks for all the details. The stuff about hot adding the namespaces back in, reissuing Identify Controller/Identify Device, etc was very easy to follow in the driver; it’s clear how & why that’s done. However I’m still having some difficulty with the finishing-up-IOs-outstanding before starting the format issue. I may be just covering ground the group covered a long time ago but I’m wondering if all the assumptions below are valid. I’m also interested in knowing how the group validated them. Here are my concerns: >> if a format comes down for a namespace, any I/O outstanding to the controller (there won’t be anything that needs to be sent… I/O will either be at the device or on the CQ) Why do we think there won’t be anything that needs to be sent? The app that sends a pass through IOCTL to do the format is presumably completely independent from say any file IO that might be going on on behalf of other apps, or even raw IO from apps like Iometer. Seems like lots of IO could still be coming in when the format IOCTL is received. > > any I/O outstanding to the controller …will simply complete via normal operation or be aborted at the controller… but Storport won’t care because the SCSI target was already removed upon initially receiving the format command. Any I/O in the CQ will be completed and handled by Storport correctly. I am wondering about two of the assumptions about timing in the paragraph above. a) I don’t think it’s true that Storport can handle commands completing for a device it no longer has a record of – ie, after the target was removed due to re-enumeration.I think it will have torn down its own structures for any old LUN(s) we had previously exposed and that would include any record it had of commands outstanding for those old LUNs. I think it isn’t going to hold on to ghost requests on behalf of devices that it no longer has a record of because it has no place to store such requests anyway at that point/ no object to associate them with. b) We are assuming that when we call StorPortNotification() with BusChangeDetected that Storport will come back in to the driver to rescan the bus – either via a bunch of Inquiry cmds or via a Report Luns cmd – and will finish all the work associated with the bus scan, updating it’s record of device topology (ie remove any SCSI target/LUNs that were assocated with the NVMe dev we are about to format) – all before it returns from our call to ScsiPortNotification and before the driver code continues on to call ProcessIO to start the real NVMe Format NVM operation. I don’t know that that is a safe correct assumption to make. The bus scan could be deferred till the driver returns. Or even if it is launched right away, could the bus scan take place on a different processor while the proc that is running through the driver just continues on its’ way? In my experience, it is the drivers and controllers’ joint responsibility to make sure that all outstanding IOs are completed back to the caller one way or the other before starting the format op. That means aborting whatever can be aborted and also making absolutely sure that no live request left over from a Namespace that has been removed ever gets completed back to the host after the Namespace has been removed. Thanks, Judy From: Robles, Raymond C [mailto:raymond.c.robles at intel.com] Sent: Thursday, April 18, 2013 6:08 PM To: Judy Brock-SSI; WONMOON CHEON; ???; nvmewin at lists.openfabrics.org Subject: RE: RE: Handling pending commands when processing Format [changing from NVMe WG dist. list to OFA NVMe Windows Driver dist. list] Hi Judy, Ahhh… I see now. I didn’t answer that question below. The format command is essentially built into the driver by a state machine. When we receive the format command we immediately issue the *hot remove* command… but that is done inline. So, once we call Storport to kick off the enumeration, we simply return back to handling the format command in NVMeStartIoProcessIoctl(). The appropriate states are set along the way to indicate progress. Once the namespace is removed from the “OS view”, then the format is processed like any other command (via ProcessIo). The callback is setup to call NVMeIoctlFormatNVMCAllback() and the variable “FormatNvmInfo->AddNamespaceNeeded” is set to TRUE so that on the completion side we remember to have the OS re-enumerate after we are done. Once the NVM format completes, the callback is invoked in the completion DPC. Then on the completion side we issue Identify Controller and Identify Namespace so that our cached driver data for the namespace(s) formatted are up to date. In the last state, after getting the Identify Namespace struct, we’ll call *hot add* which is described below. Note that at no point do we “wait” for any I/O to finish. Format is a dangerous command… especially via pass through IOCTL. We talked about this quite a bit in the beginning of developing this driver. But essentially, if a format comes down for a namespace, any I/O outstanding to the controller (there won’t be anything that needs to be sent… I/O will either be at the device or on the CQ) will simply complete via normal operation or be aborted at the controller… but Storport won’t care because the SCSI target was already removed upon initially receiving the format command. Any I/O in the CQ will be completed and handled by Storport correctly. Thanks, Ray From: Judy Brock-SSI [mailto:judy.brock at ssi.samsung.com] Sent: Thursday, April 18, 2013 4:47 PM To: Robles, Raymond C; WONMOON CHEON; ???; nvmewin at lists.openfabrics.org Subject: RE: RE: Handling pending commands when processing Format [changing from NVMe WG dist. list to OFA NVMe Windows Driver dist. list] Hi Ray, [Ray wrote] Let me know if this answers your question. I don’t think it does. What I wrote below I think was pretty much the same as what you wrote - or at least that was my intention ☺. However, the piece I couldn’t explain (cause I haven’t looked into it) is how the driver holds off the beginning of the actual format NVM operation till whatever old IOs that were already in progress for the namespace(s) before the format op request was received are completed back to the caller, aborted, or whatever - so there are no old live requests hanging around, still in the driver , before the format op begins. In other words, does the driver hold off starting the format cmd till the outstanding IOs are completed? Or do we perhaps just drop them on the floor and let the OS figure out that those requests are permanently lost/gone due to the LUNs having disappeared (my guess is, the latter is what we do)? Or do we try to abort them all? And so on. So again, we do understand how to get the OS to avoid sending new I/O requests to stale namespaces but how exactly are the old I/O reqs (those existing at the time the format request comes in ) handled? At least that is my current question ☺ Thanks, Judy From: Robles, Raymond C [mailto:raymond.c.robles at intel.com] Sent: Thursday, April 18, 2013 2:46 PM To: Judy Brock-SSI; WONMOON CHEON; ???; nvmewin at lists.openfabrics.org Subject: RE: RE: Handling pending commands when processing Format [changing from NVMe WG dist. list to OFA NVMe Windows Driver dist. list] Hello Judy/Wonmoon, Sorry for the late response. The “hot remove” state is just a state that we enter when the driver receives a Format command. Basically, this state will remove the namespace(s) from the topology by calling StorPortNotification() with BusChangeDetected. This will remove the “SCSI target/disk” associated with each namespace form the OS (because Storport will re-enumerate the controller and the driver will not expose the namespaces about to be formatted) so that the format can occur on the relevant namespaces. By signaling Windows that the namespaces have been “removed”, all I/O will be stopped by the OS. Then the format can complete. Once the format is complete, we perform the opposite action to “hot add” the namespace back into the topology by calling StorPortNotification() with BusChangeDetected… only this time, we will surface the namespace(s) again when Storport re-enumerates. Not queues are deleted in this state, no memory is de-allocated, and nothing else changes about the namespace. This is simply the first step (in a 3 step sequence) when formatting a namespace as we cannot format a namespace while the OS is aware of its presence and could be potentially sending I/O to a stale namespace config (i.e. changing LBA/sector size). Let me know if this answers your question. Thanks, Ray From: Judy Brock-SSI [mailto:judy.brock at ssi.samsung.com] Sent: Wednesday, April 17, 2013 6:21 AM To: WONMOON CHEON; 강미경; technical at nvmexpress.org Subject: RE: RE: Handling pending commands when processing Format >>Would you elaborate more about the "hot-remove" state? In this state, do you mean that all the IO SQ/CQs are deleted? Or, waiting for completions of all the outstanding IOs? The IO SQ/CQs are definitely NOT deleted. I would need to look more closely through the driver code to see how IOs previously sent to the namespaces which are marked as OFFLINE are handled/finished/quiesced. There are other folks on this thread who no doubt have more history/intimate knowledge of this driver than I do who may answer that question more quickly than I can…also, perhaps this discussion should probably be moved to the OFA driver forum since it has turned into a driver-specific thread at this point. What do folks think? Judy From: 천원문 [mailto:wm.cheon at samsung.com] Sent: Wednesday, April 17, 2013 1:20 AM To: Judy Brock-SSI; 강미경; technical at nvmexpress.org Subject: Re: RE: Handling pending commands when processing Format Hi Judy, Would you elaborate more about the "hot-remove" state? In this state, do you mean that all the IO SQ/CQs are deleted? Or, waiting for completions of all the outstanding IOs? Thanks, Wonmoon ------- Original Message ------- Sender : Judy Brock-SSI> Date : 2013-04-17 16:39 (GMT+09:00) Title : RE: Handling pending commands when processing Format Hi, I should clarify that it is not the Windows operating system – but rather the Windows OFA NVMe driver - that, from what I can see, does a “hot-remove” of all namespace(s) associated with a device before allowing a format operation to begin; “hot remove” is just the name for an internal state in the driver format nvm state machine. Before beginning the actual format op, the driver internally marks all namespaces associated with the format operation “offline”. It then notifies the OS that there has been a “bus change” event (via an OS-specific API). This in turn will cause the OS to rescan (re-enumerate) the “bus” (the pseudo SCSI bus, that is – we expose NVM namespaces as SCSI luns). Since all the pertinent namespaces have been marked offline internally, the bus rescan won’t detect any valid SCSI luns (because the driver will not report any). Hence from the OS point of view, any SCSI lun(s) previously mapped to the namespace(s) to be formatted will have disappeared/will be unaddressable while the format operation is in progress. Judy From: Judy Brock-SSI Sent: Tuesday, April 16, 2013 8:22 PM To: 'mkkang.kang at samsung.com'; technical at nvmexpress.org Subject: RE: Handling pending commands when processing Format Mikyeong, I haven’t looked at the Linux driver but I know that Windows hot-removes all namespace(s) associated with a device before allowing a format operation to begin. And a namespace can’t be removed while there is IO outstanding to it so that answers your question regarding IOs being completed before format begins. It also answers the question about requests being sent to a namespace while format is in progress – can’t happen. Thanks, Judy From: 강미경 [mailto:mkkang.kang at samsung.com] Sent: Tuesday, April 16, 2013 7:07 PM To: technical at nvmexpress.org Subject: Handling pending commands when processing Format Dear All, Format NVM command may change the Namespace repository, and it will be executed out of order like any other commands. Therefore, Format NVM command may affect other commands that are pending execution in the device, if any. 1) How does an OFA/linux driver handle 'Format NVM command'? Does a host make sure that all commands for a particular NSID are completed before sending 'Format NVM command'? 2) If a host driver does not behave like 1) above, how can a device handle other pending commands which were previously submitted in a SQ? It seems like we need an additional status code. e.g. Abort due to Namespace Format 3) Let's suppose that 'Format NVM command' is in progress. If the host driver sends subsequent commands to the namespace being formatted, should the device reply directly with a 'Namespace not Ready'? [1.0e spec] If the device does not reply directly and the format operation takes long time, then, I/O command will timeout and the host may send the reset. But if commands are responded with 'Namespace Not Ready', host may not issue the reset. Therefore, direct reply seems to be needed. [1.1 spec. ECN 001] There is Format progress indicator. The host driver can check format progress any time, therefore, there is no concern about reset during format command. Best Regards, Mikyeong Kang Kang MiKyeong Flash Memory Planning/Enabling Group, Memory Div. SAMSUNG ELECTRONICS, Co., Ltd.. Phone: 82-31-208-3857 Mobile: 82-10-9369-0177 E-mail: mkkang.kang at samsung.com [cid:image001.jpg at 01CE3C59.BF34AF60] -------------- next part -------------- An HTML attachment was scrubbed... URL: -------------- next part -------------- A non-text attachment was scrubbed... Name: image001.jpg Type: image/jpeg Size: 34869 bytes Desc: image001.jpg URL: From barrett.n.mayes at intel.com Thu May 9 09:31:35 2013 From: barrett.n.mayes at intel.com (Mayes, Barrett N) Date: Thu, 9 May 2013 16:31:35 +0000 Subject: [nvmewin] Handling pending commands when processing Format [changing from NVMe WG dist. list to OFA NVMe Windows Driver dist. list] In-Reply-To: <36E8D38D6B771A4BBDB1C0D800158A512FD60DDA@SSIEXCH-MB2.ssi.samsung.com> References: <49158E750348AA499168FD41D88983606254536E@ORSMSX152.amr.corp.intel.com> <36E8D38D6B771A4BBDB1C0D800158A512FBDBF5E@SSIEXCH-MB2.ssi.samsung.com> <49158E750348AA499168FD41D889836062545566@ORSMSX152.amr.corp.intel.com> <36E8D38D6B771A4BBDB1C0D800158A512FD60B88@SSIEXCH-MB2.ssi.samsung.com> <36E8D38D6B771A4BBDB1C0D800158A512FD60DDA@SSIEXCH-MB2.ssi.samsung.com> Message-ID: the miniport cannot assume that, after signally a Bus Change Notification to initiate the removal process, once the call to signal the Bus Change Notification returns, the upper layers will already have done all the work involved in the device removal process. Correct. The LUN isn’t “gone” until the miniport fails to report it in a subsequent inquiry. From: Judy Brock-SSI [mailto:judy.brock at ssi.samsung.com] Sent: Wednesday, May 01, 2013 4:29 PM To: Mayes, Barrett N; Robles, Raymond C; nvmewin at lists.openfabrics.org Subject: RE: [nvmewin] Handling pending commands when processing Format [changing from NVMe WG dist. list to OFA NVMe Windows Driver dist. list] Hi Barret, To sum up the info you provide below, I think it confirms the following point I was trying to make: the miniport cannot assume that, after signally a Bus Change Notification to initiate the removal process, once the call to signal the Bus Change Notification returns, the upper layers will already have done all the work involved in the device removal process. In fact, that there is a lot of work involved in device removal (for ex, we both cited the new set of inquires to come in to the miniport which allow the miniport to signal that a previously reported LUN is no longer present). Thanks, Judy From: Mayes, Barrett N [mailto:barrett.n.mayes at intel.com] Sent: Wednesday, May 01, 2013 10:00 AM To: Judy Brock-SSI; Robles, Raymond C; nvmewin at lists.openfabrics.org Subject: RE: [nvmewin] Handling pending commands when processing Format [changing from NVMe WG dist. list to OFA NVMe Windows Driver dist. list] I’ll take a quick stab at the first part. Will need Ray or someone with more experience on Format implementation to address the 2nd part as I’m not sure on the synchronization of completing the inquiries following the Bus Change notification and starting of the format command. a) I don’t think it’s true that Storport can handle commands completing for a device it no longer has a record of – ie, after the target was removed due to re-enumeration.I think it will have torn down its own structures for any old LUN(s) we had previously exposed and that would include any record it had of commands outstanding for those old LUNs. I think it isn’t going to hold on to ghost requests on behalf of devices that it no longer has a record of because it has no place to store such requests anyway at that point/ no object to associate them with. In general, the port driver will stall its queues, complete any IO already in flight or queued to the device then delete the device object. The process is fairly involved and is documented here: http://msdn.microsoft.com/en-us/library/windows/hardware/ff561046(v=vs.85).aspx This describes how a WDM driver handles the remove process. The other piece to this is the storport<->miniport interaction. To initiate the removal process, the miniport can signal a Bus Change Notification to cause storport to send a new set of inquiries. If a previously reported LUN is not reported in a subsequent inquiry, storport will call IoInvalidateDeviceRelations where it will in turn not report back a previously reported Device Object. See the “BusRelations Request” section here: http://msdn.microsoft.com/en-us/library/windows/hardware/ff551670(v=vs.85).aspx and docs for IRP_MN_REMOVE_DEVICE here: http://msdn.microsoft.com/en-us/library/windows/hardware/ff551738(v=vs.85).aspx. From: nvmewin-bounces at lists.openfabrics.org [mailto:nvmewin-bounces at lists.openfabrics.org] On Behalf Of Judy Brock-SSI Sent: Wednesday, May 01, 2013 3:33 AM To: Robles, Raymond C; nvmewin at lists.openfabrics.org Subject: Re: [nvmewin] Handling pending commands when processing Format [changing from NVMe WG dist. list to OFA NVMe Windows Driver dist. list] Hi Ray, I was wondering if you or others had any thoughts about the below. Thanks, Judy From: Judy Brock-SSI Sent: Monday, April 22, 2013 11:27 PM To: 'Robles, Raymond C'; WONMOON CHEON; ???; nvmewin at lists.openfabrics.org Subject: RE: RE: Handling pending commands when processing Format [changing from NVMe WG dist. list to OFA NVMe Windows Driver dist. list] Hi Ray, Thanks for all the details. The stuff about hot adding the namespaces back in, reissuing Identify Controller/Identify Device, etc was very easy to follow in the driver; it’s clear how & why that’s done. However I’m still having some difficulty with the finishing-up-IOs-outstanding before starting the format issue. I may be just covering ground the group covered a long time ago but I’m wondering if all the assumptions below are valid. I’m also interested in knowing how the group validated them. Here are my concerns: >> if a format comes down for a namespace, any I/O outstanding to the controller (there won’t be anything that needs to be sent… I/O will either be at the device or on the CQ) Why do we think there won’t be anything that needs to be sent? The app that sends a pass through IOCTL to do the format is presumably completely independent from say any file IO that might be going on on behalf of other apps, or even raw IO from apps like Iometer. Seems like lots of IO could still be coming in when the format IOCTL is received. > > any I/O outstanding to the controller …will simply complete via normal operation or be aborted at the controller… but Storport won’t care because the SCSI target was already removed upon initially receiving the format command. Any I/O in the CQ will be completed and handled by Storport correctly. I am wondering about two of the assumptions about timing in the paragraph above. a) I don’t think it’s true that Storport can handle commands completing for a device it no longer has a record of – ie, after the target was removed due to re-enumeration.I think it will have torn down its own structures for any old LUN(s) we had previously exposed and that would include any record it had of commands outstanding for those old LUNs. I think it isn’t going to hold on to ghost requests on behalf of devices that it no longer has a record of because it has no place to store such requests anyway at that point/ no object to associate them with. b) We are assuming that when we call StorPortNotification() with BusChangeDetected that Storport will come back in to the driver to rescan the bus – either via a bunch of Inquiry cmds or via a Report Luns cmd – and will finish all the work associated with the bus scan, updating it’s record of device topology (ie remove any SCSI target/LUNs that were assocated with the NVMe dev we are about to format) – all before it returns from our call to ScsiPortNotification and before the driver code continues on to call ProcessIO to start the real NVMe Format NVM operation. I don’t know that that is a safe correct assumption to make. The bus scan could be deferred till the driver returns. Or even if it is launched right away, could the bus scan take place on a different processor while the proc that is running through the driver just continues on its’ way? In my experience, it is the drivers and controllers’ joint responsibility to make sure that all outstanding IOs are completed back to the caller one way or the other before starting the format op. That means aborting whatever can be aborted and also making absolutely sure that no live request left over from a Namespace that has been removed ever gets completed back to the host after the Namespace has been removed. Thanks, Judy From: Robles, Raymond C [mailto:raymond.c.robles at intel.com] Sent: Thursday, April 18, 2013 6:08 PM To: Judy Brock-SSI; WONMOON CHEON; ???; nvmewin at lists.openfabrics.org Subject: RE: RE: Handling pending commands when processing Format [changing from NVMe WG dist. list to OFA NVMe Windows Driver dist. list] Hi Judy, Ahhh… I see now. I didn’t answer that question below. The format command is essentially built into the driver by a state machine. When we receive the format command we immediately issue the *hot remove* command… but that is done inline. So, once we call Storport to kick off the enumeration, we simply return back to handling the format command in NVMeStartIoProcessIoctl(). The appropriate states are set along the way to indicate progress. Once the namespace is removed from the “OS view”, then the format is processed like any other command (via ProcessIo). The callback is setup to call NVMeIoctlFormatNVMCAllback() and the variable “FormatNvmInfo->AddNamespaceNeeded” is set to TRUE so that on the completion side we remember to have the OS re-enumerate after we are done. Once the NVM format completes, the callback is invoked in the completion DPC. Then on the completion side we issue Identify Controller and Identify Namespace so that our cached driver data for the namespace(s) formatted are up to date. In the last state, after getting the Identify Namespace struct, we’ll call *hot add* which is described below. Note that at no point do we “wait” for any I/O to finish. Format is a dangerous command… especially via pass through IOCTL. We talked about this quite a bit in the beginning of developing this driver. But essentially, if a format comes down for a namespace, any I/O outstanding to the controller (there won’t be anything that needs to be sent… I/O will either be at the device or on the CQ) will simply complete via normal operation or be aborted at the controller… but Storport won’t care because the SCSI target was already removed upon initially receiving the format command. Any I/O in the CQ will be completed and handled by Storport correctly. Thanks, Ray From: Judy Brock-SSI [mailto:judy.brock at ssi.samsung.com] Sent: Thursday, April 18, 2013 4:47 PM To: Robles, Raymond C; WONMOON CHEON; ???; nvmewin at lists.openfabrics.org Subject: RE: RE: Handling pending commands when processing Format [changing from NVMe WG dist. list to OFA NVMe Windows Driver dist. list] Hi Ray, [Ray wrote] Let me know if this answers your question. I don’t think it does. What I wrote below I think was pretty much the same as what you wrote - or at least that was my intention ☺. However, the piece I couldn’t explain (cause I haven’t looked into it) is how the driver holds off the beginning of the actual format NVM operation till whatever old IOs that were already in progress for the namespace(s) before the format op request was received are completed back to the caller, aborted, or whatever - so there are no old live requests hanging around, still in the driver , before the format op begins. In other words, does the driver hold off starting the format cmd till the outstanding IOs are completed? Or do we perhaps just drop them on the floor and let the OS figure out that those requests are permanently lost/gone due to the LUNs having disappeared (my guess is, the latter is what we do)? Or do we try to abort them all? And so on. So again, we do understand how to get the OS to avoid sending new I/O requests to stale namespaces but how exactly are the old I/O reqs (those existing at the time the format request comes in ) handled? At least that is my current question ☺ Thanks, Judy From: Robles, Raymond C [mailto:raymond.c.robles at intel.com] Sent: Thursday, April 18, 2013 2:46 PM To: Judy Brock-SSI; WONMOON CHEON; ???; nvmewin at lists.openfabrics.org Subject: RE: RE: Handling pending commands when processing Format [changing from NVMe WG dist. list to OFA NVMe Windows Driver dist. list] Hello Judy/Wonmoon, Sorry for the late response. The “hot remove” state is just a state that we enter when the driver receives a Format command. Basically, this state will remove the namespace(s) from the topology by calling StorPortNotification() with BusChangeDetected. This will remove the “SCSI target/disk” associated with each namespace form the OS (because Storport will re-enumerate the controller and the driver will not expose the namespaces about to be formatted) so that the format can occur on the relevant namespaces. By signaling Windows that the namespaces have been “removed”, all I/O will be stopped by the OS. Then the format can complete. Once the format is complete, we perform the opposite action to “hot add” the namespace back into the topology by calling StorPortNotification() with BusChangeDetected… only this time, we will surface the namespace(s) again when Storport re-enumerates. Not queues are deleted in this state, no memory is de-allocated, and nothing else changes about the namespace. This is simply the first step (in a 3 step sequence) when formatting a namespace as we cannot format a namespace while the OS is aware of its presence and could be potentially sending I/O to a stale namespace config (i.e. changing LBA/sector size). Let me know if this answers your question. Thanks, Ray From: Judy Brock-SSI [mailto:judy.brock at ssi.samsung.com] Sent: Wednesday, April 17, 2013 6:21 AM To: WONMOON CHEON; 강미경; technical at nvmexpress.org Subject: RE: RE: Handling pending commands when processing Format >>Would you elaborate more about the "hot-remove" state? In this state, do you mean that all the IO SQ/CQs are deleted? Or, waiting for completions of all the outstanding IOs? The IO SQ/CQs are definitely NOT deleted. I would need to look more closely through the driver code to see how IOs previously sent to the namespaces which are marked as OFFLINE are handled/finished/quiesced. There are other folks on this thread who no doubt have more history/intimate knowledge of this driver than I do who may answer that question more quickly than I can…also, perhaps this discussion should probably be moved to the OFA driver forum since it has turned into a driver-specific thread at this point. What do folks think? Judy From: 천원문 [mailto:wm.cheon at samsung.com] Sent: Wednesday, April 17, 2013 1:20 AM To: Judy Brock-SSI; 강미경; technical at nvmexpress.org Subject: Re: RE: Handling pending commands when processing Format Hi Judy, Would you elaborate more about the "hot-remove" state? In this state, do you mean that all the IO SQ/CQs are deleted? Or, waiting for completions of all the outstanding IOs? Thanks, Wonmoon ------- Original Message ------- Sender : Judy Brock-SSI> Date : 2013-04-17 16:39 (GMT+09:00) Title : RE: Handling pending commands when processing Format Hi, I should clarify that it is not the Windows operating system – but rather the Windows OFA NVMe driver - that, from what I can see, does a “hot-remove” of all namespace(s) associated with a device before allowing a format operation to begin; “hot remove” is just the name for an internal state in the driver format nvm state machine. Before beginning the actual format op, the driver internally marks all namespaces associated with the format operation “offline”. It then notifies the OS that there has been a “bus change” event (via an OS-specific API). This in turn will cause the OS to rescan (re-enumerate) the “bus” (the pseudo SCSI bus, that is – we expose NVM namespaces as SCSI luns). Since all the pertinent namespaces have been marked offline internally, the bus rescan won’t detect any valid SCSI luns (because the driver will not report any). Hence from the OS point of view, any SCSI lun(s) previously mapped to the namespace(s) to be formatted will have disappeared/will be unaddressable while the format operation is in progress. Judy From: Judy Brock-SSI Sent: Tuesday, April 16, 2013 8:22 PM To: 'mkkang.kang at samsung.com'; technical at nvmexpress.org Subject: RE: Handling pending commands when processing Format Mikyeong, I haven’t looked at the Linux driver but I know that Windows hot-removes all namespace(s) associated with a device before allowing a format operation to begin. And a namespace can’t be removed while there is IO outstanding to it so that answers your question regarding IOs being completed before format begins. It also answers the question about requests being sent to a namespace while format is in progress – can’t happen. Thanks, Judy From: 강미경 [mailto:mkkang.kang at samsung.com] Sent: Tuesday, April 16, 2013 7:07 PM To: technical at nvmexpress.org Subject: Handling pending commands when processing Format Dear All, Format NVM command may change the Namespace repository, and it will be executed out of order like any other commands. Therefore, Format NVM command may affect other commands that are pending execution in the device, if any. 1) How does an OFA/linux driver handle 'Format NVM command'? Does a host make sure that all commands for a particular NSID are completed before sending 'Format NVM command'? 2) If a host driver does not behave like 1) above, how can a device handle other pending commands which were previously submitted in a SQ? It seems like we need an additional status code. e.g. Abort due to Namespace Format 3) Let's suppose that 'Format NVM command' is in progress. If the host driver sends subsequent commands to the namespace being formatted, should the device reply directly with a 'Namespace not Ready'? [1.0e spec] If the device does not reply directly and the format operation takes long time, then, I/O command will timeout and the host may send the reset. But if commands are responded with 'Namespace Not Ready', host may not issue the reset. Therefore, direct reply seems to be needed. [1.1 spec. ECN 001] There is Format progress indicator. The host driver can check format progress any time, therefore, there is no concern about reset during format command. Best Regards, Mikyeong Kang Kang MiKyeong Flash Memory Planning/Enabling Group, Memory Div. SAMSUNG ELECTRONICS, Co., Ltd.. Phone: 82-31-208-3857 Mobile: 82-10-9369-0177 E-mail: mkkang.kang at samsung.com [cid:image001.jpg at 01CE3C59.BF34AF60] -------------- next part -------------- An HTML attachment was scrubbed... URL: -------------- next part -------------- A non-text attachment was scrubbed... Name: image001.jpg Type: image/jpeg Size: 34869 bytes Desc: image001.jpg URL: From raymond.c.robles at intel.com Mon May 13 16:54:54 2013 From: raymond.c.robles at intel.com (Robles, Raymond C) Date: Mon, 13 May 2013 23:54:54 +0000 Subject: [nvmewin] New Patch From Sandisk In-Reply-To: <23EC73C80FB59046A6B7B8EB7B38265923248C18@MILMBXIP02.sdcorp.global.sandisk.com> References: <23EC73C80FB59046A6B7B8EB7B38265923248B63@MILMBXIP02.sdcorp.global.sandisk.com> <548C5470AAD9DA4A85D259B663190D361FFE04D2@corpmail1.na.ads.idt.com> <23EC73C80FB59046A6B7B8EB7B38265923248BD1@MILMBXIP02.sdcorp.global.sandisk.com> <49158E750348AA499168FD41D88983606254CB45@ORSMSX152.amr.corp.intel.com> <23EC73C80FB59046A6B7B8EB7B38265923248C18@MILMBXIP02.sdcorp.global.sandisk.com> Message-ID: <49158E750348AA499168FD41D88983606255F157@FMSMSX105.amr.corp.intel.com> Hello, I have not heard any feedback on the latest updates on Dharani's patch. IDT and LSI, can we close on this patch by this Friday (5/17)? I have one patch I would like to push after this one. Thanks, Ray From: Dharani Kotte [mailto:Dharani.Kotte at sandisk.com] Sent: Tuesday, April 30, 2013 12:03 PM To: Robles, Raymond C; nvmewin at lists.openfabrics.org Subject: RE: New Patch From Sandisk Thank you for the feedback, I agree that the global buffer is not good I have made the changes to allocate this buffer from the srbext I will provide the code with the below modifications soon. Thanks, Dharani. From: Robles, Raymond C [mailto:raymond.c.robles at intel.com] Sent: Tuesday, April 30, 2013 11:52 AM To: Dharani Kotte; nvmewin at lists.openfabrics.org Subject: RE: New Patch From Sandisk Hi Dharani, Thank you for putting this patch together. Please see my feedback below: - Line 52: Note that we have a file called nvmeSntiTypes.h. This file contains all #defines. As per our existing convention, all #defines should be placed in this file (versus at the top of a .c file). - Line 3339/3412/3502/3593/3756/5391/5433/5442/5453: Our coding convention states no line be longer than 80 characters. These lines all exceed 80 characters. - Line 5410: SntiTranslateAllModePagesResponse() contains new if-else blocks. Our coding convention dictates that open curly braces be placed on the same line as the if or else statement. You can refer to other instances of the code and if-else clauses as examples. - Line 4017: SntiCreateModeDataHeader() no longer needs the SRB pointer to be passed in as a parameter since you are using a global temp buffer to build up the mode sense data. That parameter can be removed. - General: The method of using a global buffer for a single mode sense command opens up a potential race condition for multiple mode sense commands submitted on top of each other. The idea behind putting the SCSI to NVMe translation phase in the HwStorBuildIo phase was to help performance. We initially knew that we would never be sharing any data structures or memory in calls to BuildIo. There is no spinlock or protection for calls to BuildIo and MSDN makes it clear that these BuildIo calls are not synchronized and that any synchronization of memory/data must be done within the function implementations or just use StartIo instead (since Storport will acquire the StartIoSpinLock). So there is hole here w/r/t the global buffer being populated for multiple mode sense commands at the same time. I believe you have the right idea, but the correct solution is to create a temp mode sense buffer inside the SRB extension. This way, each command has its own temp mode sense buffer for building the mode sense response, and then when ready to copy over to the SRB data buffer, it's pulling from a buffer that can only be accessed by the current command (instead of a global buffer that may have been overwritten by another mode sense request call from another BuildIo call). Note that the driver collaboration group made the decision to never acquire any type of lock in BuildIo. Let me know if you have questions on this solution. The additional size to the SRB extension is not an issue. This fix is critical for this patch to be accepted. Thanks, Ray From: nvmewin-bounces at lists.openfabrics.org [mailto:nvmewin-bounces at lists.openfabrics.org] On Behalf Of Dharani Kotte Sent: Tuesday, April 30, 2013 11:14 AM To: Chang, Alex; nvmewin at lists.openfabrics.org Subject: [nvmewin] ***UNCHECKED*** [WARNING - ENCRYPTED ATTACHMENT NOT VIRUS SCANNED] New Patch From Sandisk Attaching the code with according to Alex suggestion. Passwd: sndk1234 Thanks, Dharani. From: Chang, Alex [mailto:Alex.Chang at idt.com] Sent: Tuesday, April 30, 2013 9:23 AM To: Dharani Kotte; nvmewin at lists.openfabrics.org Subject: RE: New Patch From Sandisk Hi Dharani, I'd suggest that it's safer to initialize gModeSenseBuf as all zeros each time before it is used, just in case there are some unexpected data in the buffer. Other than that, I am fine with the patch. Thanks, Alex ________________________________ From: nvmewin-bounces at lists.openfabrics.org [mailto:nvmewin-bounces at lists.openfabrics.org] On Behalf Of Dharani Kotte Sent: Tuesday, April 30, 2013 8:37 AM To: nvmewin at lists.openfabrics.org Subject: [nvmewin] New Patch From Sandisk Hi all, It's been almost two weeks after I sent out the patch. please let us know if you're okay with it. Thanks, Dharani. From: Dharani Kotte Sent: Friday, April 12, 2013 9:58 AM To: nvmewin at lists.openfabrics.org Subject: New Patch From Sandisk Hi all, I am attaching a new patch that includes the following changes: 1. In nvmeSnti.c a. Allocate a global buffer of 256bytes size for modesense return data preparation b. For any mode sense this buffer is used for modesense data along with the requested pages in it. c. Copy the required data according to the alloc_length requested in the cdb to the Srb->DataBuffer and update the dataTransferLength accordingly d. In function SntiTranslateReturnAllModePagesResponse() the modesense data header offset is calculated wrong which ends up in BSOD in some cases modified code to fix this issue Please review the changes and provide feedbacks if you have any. If nobody disagrees with the changes, I will remind Ray to merge them in two weeks. Thanks, Dharani. ________________________________ PLEASE NOTE: The information contained in this electronic mail message is intended only for the use of the designated recipient(s) named above. If the reader of this message is not the intended recipient, you are hereby notified that you have received this message in error and that any review, dissemination, distribution, or copying of this message is strictly prohibited. If you have received this communication in error, please notify the sender by telephone or e-mail (as shown above) immediately and destroy any and all copies of this message in your possession (whether hard copies or electronically stored copies). -------------- next part -------------- An HTML attachment was scrubbed... URL: From Alex.Chang at idt.com Tue May 14 10:37:27 2013 From: Alex.Chang at idt.com (Chang, Alex) Date: Tue, 14 May 2013 17:37:27 +0000 Subject: [nvmewin] New Patch From Sandisk In-Reply-To: <49158E750348AA499168FD41D88983606255F157@FMSMSX105.amr.corp.intel.com> References: <23EC73C80FB59046A6B7B8EB7B38265923248B63@MILMBXIP02.sdcorp.global.sandisk.com> <548C5470AAD9DA4A85D259B663190D361FFE04D2@corpmail1.na.ads.idt.com> <23EC73C80FB59046A6B7B8EB7B38265923248BD1@MILMBXIP02.sdcorp.global.sandisk.com> <49158E750348AA499168FD41D88983606254CB45@ORSMSX152.amr.corp.intel.com> <23EC73C80FB59046A6B7B8EB7B38265923248C18@MILMBXIP02.sdcorp.global.sandisk.com> <49158E750348AA499168FD41D88983606255F157@FMSMSX105.amr.corp.intel.com> Message-ID: <548C5470AAD9DA4A85D259B663190D361FFE73DC@corpmail1.na.ads.idt.com> Hi Ray, Thanks for reminding us. I will re-test the patch and let you know ASAP. Alex ________________________________ From: nvmewin-bounces at lists.openfabrics.org [mailto:nvmewin-bounces at lists.openfabrics.org] On Behalf Of Robles, Raymond C Sent: Monday, May 13, 2013 4:55 PM To: nvmewin at lists.openfabrics.org Subject: Re: [nvmewin] New Patch From Sandisk Hello, I have not heard any feedback on the latest updates on Dharani's patch. IDT and LSI, can we close on this patch by this Friday (5/17)? I have one patch I would like to push after this one. Thanks, Ray From: Dharani Kotte [mailto:Dharani.Kotte at sandisk.com] Sent: Tuesday, April 30, 2013 12:03 PM To: Robles, Raymond C; nvmewin at lists.openfabrics.org Subject: RE: New Patch From Sandisk Thank you for the feedback, I agree that the global buffer is not good I have made the changes to allocate this buffer from the srbext I will provide the code with the below modifications soon. Thanks, Dharani. From: Robles, Raymond C [mailto:raymond.c.robles at intel.com] Sent: Tuesday, April 30, 2013 11:52 AM To: Dharani Kotte; nvmewin at lists.openfabrics.org Subject: RE: New Patch From Sandisk Hi Dharani, Thank you for putting this patch together. Please see my feedback below: - Line 52: Note that we have a file called nvmeSntiTypes.h. This file contains all #defines. As per our existing convention, all #defines should be placed in this file (versus at the top of a .c file). - Line 3339/3412/3502/3593/3756/5391/5433/5442/5453: Our coding convention states no line be longer than 80 characters. These lines all exceed 80 characters. - Line 5410: SntiTranslateAllModePagesResponse() contains new if-else blocks. Our coding convention dictates that open curly braces be placed on the same line as the if or else statement. You can refer to other instances of the code and if-else clauses as examples. - Line 4017: SntiCreateModeDataHeader() no longer needs the SRB pointer to be passed in as a parameter since you are using a global temp buffer to build up the mode sense data. That parameter can be removed. - General: The method of using a global buffer for a single mode sense command opens up a potential race condition for multiple mode sense commands submitted on top of each other. The idea behind putting the SCSI to NVMe translation phase in the HwStorBuildIo phase was to help performance. We initially knew that we would never be sharing any data structures or memory in calls to BuildIo. There is no spinlock or protection for calls to BuildIo and MSDN makes it clear that these BuildIo calls are not synchronized and that any synchronization of memory/data must be done within the function implementations or just use StartIo instead (since Storport will acquire the StartIoSpinLock). So there is hole here w/r/t the global buffer being populated for multiple mode sense commands at the same time. I believe you have the right idea, but the correct solution is to create a temp mode sense buffer inside the SRB extension. This way, each command has its own temp mode sense buffer for building the mode sense response, and then when ready to copy over to the SRB data buffer, it's pulling from a buffer that can only be accessed by the current command (instead of a global buffer that may have been overwritten by another mode sense request call from another BuildIo call). Note that the driver collaboration group made the decision to never acquire any type of lock in BuildIo. Let me know if you have questions on this solution. The additional size to the SRB extension is not an issue. This fix is critical for this patch to be accepted. Thanks, Ray From: nvmewin-bounces at lists.openfabrics.org [mailto:nvmewin-bounces at lists.openfabrics.org] On Behalf Of Dharani Kotte Sent: Tuesday, April 30, 2013 11:14 AM To: Chang, Alex; nvmewin at lists.openfabrics.org Subject: [nvmewin] ***UNCHECKED*** [WARNING - ENCRYPTED ATTACHMENT NOT VIRUS SCANNED] New Patch From Sandisk Attaching the code with according to Alex suggestion. Passwd: sndk1234 Thanks, Dharani. From: Chang, Alex [mailto:Alex.Chang at idt.com] Sent: Tuesday, April 30, 2013 9:23 AM To: Dharani Kotte; nvmewin at lists.openfabrics.org Subject: RE: New Patch From Sandisk Hi Dharani, I'd suggest that it's safer to initialize gModeSenseBuf as all zeros each time before it is used, just in case there are some unexpected data in the buffer. Other than that, I am fine with the patch. Thanks, Alex ________________________________ From: nvmewin-bounces at lists.openfabrics.org [mailto:nvmewin-bounces at lists.openfabrics.org] On Behalf Of Dharani Kotte Sent: Tuesday, April 30, 2013 8:37 AM To: nvmewin at lists.openfabrics.org Subject: [nvmewin] New Patch From Sandisk Hi all, It's been almost two weeks after I sent out the patch. please let us know if you're okay with it. Thanks, Dharani. From: Dharani Kotte Sent: Friday, April 12, 2013 9:58 AM To: nvmewin at lists.openfabrics.org Subject: New Patch From Sandisk Hi all, I am attaching a new patch that includes the following changes: 1. In nvmeSnti.c a. Allocate a global buffer of 256bytes size for modesense return data preparation b. For any mode sense this buffer is used for modesense data along with the requested pages in it. c. Copy the required data according to the alloc_length requested in the cdb to the Srb->DataBuffer and update the dataTransferLength accordingly d. In function SntiTranslateReturnAllModePagesResponse() the modesense data header offset is calculated wrong which ends up in BSOD in some cases modified code to fix this issue Please review the changes and provide feedbacks if you have any. If nobody disagrees with the changes, I will remind Ray to merge them in two weeks. Thanks, Dharani. ________________________________ PLEASE NOTE: The information contained in this electronic mail message is intended only for the use of the designated recipient(s) named above. If the reader of this message is not the intended recipient, you are hereby notified that you have received this message in error and that any review, dissemination, distribution, or copying of this message is strictly prohibited. If you have received this communication in error, please notify the sender by telephone or e-mail (as shown above) immediately and destroy any and all copies of this message in your possession (whether hard copies or electronically stored copies). -------------- next part -------------- An HTML attachment was scrubbed... URL: From Alex.Chang at idt.com Thu May 16 10:58:17 2013 From: Alex.Chang at idt.com (Chang, Alex) Date: Thu, 16 May 2013 17:58:17 +0000 Subject: [nvmewin] New Patch From Sandisk In-Reply-To: <23EC73C80FB59046A6B7B8EB7B38265923248D8D@MILMBXIP02.sdcorp.global.sandisk.com> References: <23EC73C80FB59046A6B7B8EB7B38265923248B63@MILMBXIP02.sdcorp.global.sandisk.com> <548C5470AAD9DA4A85D259B663190D361FFE04D2@corpmail1.na.ads.idt.com> <23EC73C80FB59046A6B7B8EB7B38265923248BD1@MILMBXIP02.sdcorp.global.sandisk.com> <49158E750348AA499168FD41D88983606254CB45@ORSMSX152.amr.corp.intel.com> <23EC73C80FB59046A6B7B8EB7B38265923248D8D@MILMBXIP02.sdcorp.global.sandisk.com> Message-ID: <548C5470AAD9DA4A85D259B663190D361FFE95AF@corpmail1.na.ads.idt.com> Hi Dharani, I tested your new patch and experienced a D1 bugcheck while running SCSICompliance test. Some preliminary debugging leads to Line#3633 of nvmestni.c, where it looks like: memset(GET_DATA_BUFFER(pSrb), 0, allocLength); The bugcheck happens when the value of "allocLength" is 0x1000. Apparently, the allocated length in CDB is somewhat a bogus value that is meant to test how driver handles it. Could you please fix it? Thanks, Alex ________________________________ From: nvmewin-bounces at lists.openfabrics.org [mailto:nvmewin-bounces at lists.openfabrics.org] On Behalf Of Dharani Kotte Sent: Wednesday, May 01, 2013 8:56 AM To: Robles, Raymond C; nvmewin at lists.openfabrics.org Subject: [nvmewin] ***UNCHECKED*** [WARNING - ENCRYPTED ATTACHMENT NOT VIRUS SCANNED] New Patch From Sandisk Hi Ray, I have incorporated the changes according to your suggestions, I tried to keep up with the coding convention please feel free to let me know if anything has to be changed. I have incorporated one more fix in the nvmeInit.c which can lead to memory corruptions and used the SrbExt for the buffer allocation. PassWd: sndk1234 Thanks, Dharani. From: Dharani Kotte Sent: Tuesday, April 30, 2013 12:03 PM To: 'Robles, Raymond C'; nvmewin at lists.openfabrics.org Subject: RE: New Patch From Sandisk Thank you for the feedback, I agree that the global buffer is not good I have made the changes to allocate this buffer from the srbext I will provide the code with the below modifications soon. Thanks, Dharani. From: Robles, Raymond C [mailto:raymond.c.robles at intel.com] Sent: Tuesday, April 30, 2013 11:52 AM To: Dharani Kotte; nvmewin at lists.openfabrics.org Subject: RE: New Patch From Sandisk Hi Dharani, Thank you for putting this patch together. Please see my feedback below: - Line 52: Note that we have a file called nvmeSntiTypes.h. This file contains all #defines. As per our existing convention, all #defines should be placed in this file (versus at the top of a .c file). - Line 3339/3412/3502/3593/3756/5391/5433/5442/5453: Our coding convention states no line be longer than 80 characters. These lines all exceed 80 characters. - Line 5410: SntiTranslateAllModePagesResponse() contains new if-else blocks. Our coding convention dictates that open curly braces be placed on the same line as the if or else statement. You can refer to other instances of the code and if-else clauses as examples. - Line 4017: SntiCreateModeDataHeader() no longer needs the SRB pointer to be passed in as a parameter since you are using a global temp buffer to build up the mode sense data. That parameter can be removed. - General: The method of using a global buffer for a single mode sense command opens up a potential race condition for multiple mode sense commands submitted on top of each other. The idea behind putting the SCSI to NVMe translation phase in the HwStorBuildIo phase was to help performance. We initially knew that we would never be sharing any data structures or memory in calls to BuildIo. There is no spinlock or protection for calls to BuildIo and MSDN makes it clear that these BuildIo calls are not synchronized and that any synchronization of memory/data must be done within the function implementations or just use StartIo instead (since Storport will acquire the StartIoSpinLock). So there is hole here w/r/t the global buffer being populated for multiple mode sense commands at the same time. I believe you have the right idea, but the correct solution is to create a temp mode sense buffer inside the SRB extension. This way, each command has its own temp mode sense buffer for building the mode sense response, and then when ready to copy over to the SRB data buffer, it's pulling from a buffer that can only be accessed by the current command (instead of a global buffer that may have been overwritten by another mode sense request call from another BuildIo call). Note that the driver collaboration group made the decision to never acquire any type of lock in BuildIo. Let me know if you have questions on this solution. The additional size to the SRB extension is not an issue. This fix is critical for this patch to be accepted. Thanks, Ray From: nvmewin-bounces at lists.openfabrics.org [mailto:nvmewin-bounces at lists.openfabrics.org] On Behalf Of Dharani Kotte Sent: Tuesday, April 30, 2013 11:14 AM To: Chang, Alex; nvmewin at lists.openfabrics.org Subject: [nvmewin] ***UNCHECKED*** [WARNING - ENCRYPTED ATTACHMENT NOT VIRUS SCANNED] New Patch From Sandisk Attaching the code with according to Alex suggestion. Passwd: sndk1234 Thanks, Dharani. From: Chang, Alex [mailto:Alex.Chang at idt.com] Sent: Tuesday, April 30, 2013 9:23 AM To: Dharani Kotte; nvmewin at lists.openfabrics.org Subject: RE: New Patch From Sandisk Hi Dharani, I'd suggest that it's safer to initialize gModeSenseBuf as all zeros each time before it is used, just in case there are some unexpected data in the buffer. Other than that, I am fine with the patch. Thanks, Alex ________________________________ From: nvmewin-bounces at lists.openfabrics.org [mailto:nvmewin-bounces at lists.openfabrics.org] On Behalf Of Dharani Kotte Sent: Tuesday, April 30, 2013 8:37 AM To: nvmewin at lists.openfabrics.org Subject: [nvmewin] New Patch From Sandisk Hi all, It's been almost two weeks after I sent out the patch. please let us know if you're okay with it. Thanks, Dharani. From: Dharani Kotte Sent: Friday, April 12, 2013 9:58 AM To: nvmewin at lists.openfabrics.org Subject: New Patch From Sandisk Hi all, I am attaching a new patch that includes the following changes: 1. In nvmeSnti.c a. Allocate a global buffer of 256bytes size for modesense return data preparation b. For any mode sense this buffer is used for modesense data along with the requested pages in it. c. Copy the required data according to the alloc_length requested in the cdb to the Srb->DataBuffer and update the dataTransferLength accordingly d. In function SntiTranslateReturnAllModePagesResponse() the modesense data header offset is calculated wrong which ends up in BSOD in some cases modified code to fix this issue Please review the changes and provide feedbacks if you have any. If nobody disagrees with the changes, I will remind Ray to merge them in two weeks. Thanks, Dharani. ________________________________ PLEASE NOTE: The information contained in this electronic mail message is intended only for the use of the designated recipient(s) named above. If the reader of this message is not the intended recipient, you are hereby notified that you have received this message in error and that any review, dissemination, distribution, or copying of this message is strictly prohibited. If you have received this communication in error, please notify the sender by telephone or e-mail (as shown above) immediately and destroy any and all copies of this message in your possession (whether hard copies or electronically stored copies). -------------- next part -------------- An HTML attachment was scrubbed... URL: From Rick.Knoblaugh at lsi.com Thu May 16 11:58:32 2013 From: Rick.Knoblaugh at lsi.com (Knoblaugh, Rick) Date: Thu, 16 May 2013 12:58:32 -0600 Subject: [nvmewin] New Patch From Sandisk In-Reply-To: <548C5470AAD9DA4A85D259B663190D361FFE95AF@corpmail1.na.ads.idt.com> References: <23EC73C80FB59046A6B7B8EB7B38265923248B63@MILMBXIP02.sdcorp.global.sandisk.com> <548C5470AAD9DA4A85D259B663190D361FFE04D2@corpmail1.na.ads.idt.com> <23EC73C80FB59046A6B7B8EB7B38265923248BD1@MILMBXIP02.sdcorp.global.sandisk.com> <49158E750348AA499168FD41D88983606254CB45@ORSMSX152.amr.corp.intel.com> <23EC73C80FB59046A6B7B8EB7B38265923248D8D@MILMBXIP02.sdcorp.global.sandisk.com> <548C5470AAD9DA4A85D259B663190D361FFE95AF@corpmail1.na.ads.idt.com> Message-ID: <4565AEA676113A449269C2F3A549520FDBA82CDD@cosmail03.lsi.com> Hi Dharani, When you change that one, may as well fix the comment in header of SntiCreateModeDataHeader function to reflect that srb extension is now the parameter that's passed in. Other stuff looks good. Thanks, -Rick From: nvmewin-bounces at lists.openfabrics.org [mailto:nvmewin-bounces at lists.openfabrics.org] On Behalf Of Chang, Alex Sent: Thursday, May 16, 2013 10:58 AM To: Dharani Kotte; Robles, Raymond C; nvmewin at lists.openfabrics.org Subject: Re: [nvmewin] New Patch From Sandisk Hi Dharani, I tested your new patch and experienced a D1 bugcheck while running SCSICompliance test. Some preliminary debugging leads to Line#3633 of nvmestni.c, where it looks like: memset(GET_DATA_BUFFER(pSrb), 0, allocLength); The bugcheck happens when the value of "allocLength" is 0x1000. Apparently, the allocated length in CDB is somewhat a bogus value that is meant to test how driver handles it. Could you please fix it? Thanks, Alex ________________________________ From: nvmewin-bounces at lists.openfabrics.org [mailto:nvmewin-bounces at lists.openfabrics.org] On Behalf Of Dharani Kotte Sent: Wednesday, May 01, 2013 8:56 AM To: Robles, Raymond C; nvmewin at lists.openfabrics.org Subject: [nvmewin] ***UNCHECKED*** [WARNING - ENCRYPTED ATTACHMENT NOT VIRUS SCANNED] New Patch From Sandisk Hi Ray, I have incorporated the changes according to your suggestions, I tried to keep up with the coding convention please feel free to let me know if anything has to be changed. I have incorporated one more fix in the nvmeInit.c which can lead to memory corruptions and used the SrbExt for the buffer allocation. PassWd: sndk1234 Thanks, Dharani. From: Dharani Kotte Sent: Tuesday, April 30, 2013 12:03 PM To: 'Robles, Raymond C'; nvmewin at lists.openfabrics.org Subject: RE: New Patch From Sandisk Thank you for the feedback, I agree that the global buffer is not good I have made the changes to allocate this buffer from the srbext I will provide the code with the below modifications soon. Thanks, Dharani. From: Robles, Raymond C [mailto:raymond.c.robles at intel.com] Sent: Tuesday, April 30, 2013 11:52 AM To: Dharani Kotte; nvmewin at lists.openfabrics.org Subject: RE: New Patch From Sandisk Hi Dharani, Thank you for putting this patch together. Please see my feedback below: - Line 52: Note that we have a file called nvmeSntiTypes.h. This file contains all #defines. As per our existing convention, all #defines should be placed in this file (versus at the top of a .c file). - Line 3339/3412/3502/3593/3756/5391/5433/5442/5453: Our coding convention states no line be longer than 80 characters. These lines all exceed 80 characters. - Line 5410: SntiTranslateAllModePagesResponse() contains new if-else blocks. Our coding convention dictates that open curly braces be placed on the same line as the if or else statement. You can refer to other instances of the code and if-else clauses as examples. - Line 4017: SntiCreateModeDataHeader() no longer needs the SRB pointer to be passed in as a parameter since you are using a global temp buffer to build up the mode sense data. That parameter can be removed. - General: The method of using a global buffer for a single mode sense command opens up a potential race condition for multiple mode sense commands submitted on top of each other. The idea behind putting the SCSI to NVMe translation phase in the HwStorBuildIo phase was to help performance. We initially knew that we would never be sharing any data structures or memory in calls to BuildIo. There is no spinlock or protection for calls to BuildIo and MSDN makes it clear that these BuildIo calls are not synchronized and that any synchronization of memory/data must be done within the function implementations or just use StartIo instead (since Storport will acquire the StartIoSpinLock). So there is hole here w/r/t the global buffer being populated for multiple mode sense commands at the same time. I believe you have the right idea, but the correct solution is to create a temp mode sense buffer inside the SRB extension. This way, each command has its own temp mode sense buffer for building the mode sense response, and then when ready to copy over to the SRB data buffer, it's pulling from a buffer that can only be accessed by the current command (instead of a global buffer that may have been overwritten by another mode sense request call from another BuildIo call). Note that the driver collaboration group made the decision to never acquire any type of lock in BuildIo. Let me know if you have questions on this solution. The additional size to the SRB extension is not an issue. This fix is critical for this patch to be accepted. Thanks, Ray From: nvmewin-bounces at lists.openfabrics.org [mailto:nvmewin-bounces at lists.openfabrics.org] On Behalf Of Dharani Kotte Sent: Tuesday, April 30, 2013 11:14 AM To: Chang, Alex; nvmewin at lists.openfabrics.org Subject: [nvmewin] ***UNCHECKED*** [WARNING - ENCRYPTED ATTACHMENT NOT VIRUS SCANNED] New Patch From Sandisk Attaching the code with according to Alex suggestion. Passwd: sndk1234 Thanks, Dharani. From: Chang, Alex [mailto:Alex.Chang at idt.com] Sent: Tuesday, April 30, 2013 9:23 AM To: Dharani Kotte; nvmewin at lists.openfabrics.org Subject: RE: New Patch From Sandisk Hi Dharani, I'd suggest that it's safer to initialize gModeSenseBuf as all zeros each time before it is used, just in case there are some unexpected data in the buffer. Other than that, I am fine with the patch. Thanks, Alex ________________________________ From: nvmewin-bounces at lists.openfabrics.org [mailto:nvmewin-bounces at lists.openfabrics.org] On Behalf Of Dharani Kotte Sent: Tuesday, April 30, 2013 8:37 AM To: nvmewin at lists.openfabrics.org Subject: [nvmewin] New Patch From Sandisk Hi all, It's been almost two weeks after I sent out the patch. please let us know if you're okay with it. Thanks, Dharani. From: Dharani Kotte Sent: Friday, April 12, 2013 9:58 AM To: nvmewin at lists.openfabrics.org Subject: New Patch From Sandisk Hi all, I am attaching a new patch that includes the following changes: 1. In nvmeSnti.c a. Allocate a global buffer of 256bytes size for modesense return data preparation b. For any mode sense this buffer is used for modesense data along with the requested pages in it. c. Copy the required data according to the alloc_length requested in the cdb to the Srb->DataBuffer and update the dataTransferLength accordingly d. In function SntiTranslateReturnAllModePagesResponse() the modesense data header offset is calculated wrong which ends up in BSOD in some cases modified code to fix this issue Please review the changes and provide feedbacks if you have any. If nobody disagrees with the changes, I will remind Ray to merge them in two weeks. Thanks, Dharani. ________________________________ PLEASE NOTE: The information contained in this electronic mail message is intended only for the use of the designated recipient(s) named above. If the reader of this message is not the intended recipient, you are hereby notified that you have received this message in error and that any review, dissemination, distribution, or copying of this message is strictly prohibited. If you have received this communication in error, please notify the sender by telephone or e-mail (as shown above) immediately and destroy any and all copies of this message in your possession (whether hard copies or electronically stored copies). -------------- next part -------------- An HTML attachment was scrubbed... URL: From Sumant.Patro at sandisk.com Thu May 16 12:38:20 2013 From: Sumant.Patro at sandisk.com (Sumant Patro) Date: Thu, 16 May 2013 19:38:20 +0000 Subject: [nvmewin] ***UNCHECKED*** [WARNING - ENCRYPTED ATTACHMENT NOT VIRUS SCANNED] RE: New Patch From Sandisk In-Reply-To: <4565AEA676113A449269C2F3A549520FDBA82CDD@cosmail03.lsi.com> References: <23EC73C80FB59046A6B7B8EB7B38265923248B63@MILMBXIP02.sdcorp.global.sandisk.com> <548C5470AAD9DA4A85D259B663190D361FFE04D2@corpmail1.na.ads.idt.com> <23EC73C80FB59046A6B7B8EB7B38265923248BD1@MILMBXIP02.sdcorp.global.sandisk.com> <49158E750348AA499168FD41D88983606254CB45@ORSMSX152.amr.corp.intel.com> <23EC73C80FB59046A6B7B8EB7B38265923248D8D@MILMBXIP02.sdcorp.global.sandisk.com> <548C5470AAD9DA4A85D259B663190D361FFE95AF@corpmail1.na.ads.idt.com> <4565AEA676113A449269C2F3A549520FDBA82CDD@cosmail03.lsi.com> Message-ID: Thanks Rick, Alex for your feedback. Please find attached updated files with following two changes : 1. Comment in header of SntiCreateModeDataHeader function changed to reflect passing of srb extension 2. memset(GET_DATA_BUFFER(pSrb), 0, allocLength); modified to use minimum of allocLength and DataTransferLength in SntiReturnAllModePages(..) Password : sndk1234 Regards, Sumant From: nvmewin-bounces at lists.openfabrics.org [mailto:nvmewin-bounces at lists.openfabrics.org] On Behalf Of Knoblaugh, Rick Sent: Thursday, May 16, 2013 11:59 AM To: Chang, Alex; Dharani Kotte; Robles, Raymond C; nvmewin at lists.openfabrics.org Subject: Re: [nvmewin] New Patch From Sandisk Hi Dharani, When you change that one, may as well fix the comment in header of SntiCreateModeDataHeader function to reflect that srb extension is now the parameter that's passed in. Other stuff looks good. Thanks, -Rick From: nvmewin-bounces at lists.openfabrics.org [mailto:nvmewin-bounces at lists.openfabrics.org] On Behalf Of Chang, Alex Sent: Thursday, May 16, 2013 10:58 AM To: Dharani Kotte; Robles, Raymond C; nvmewin at lists.openfabrics.org Subject: Re: [nvmewin] New Patch From Sandisk Hi Dharani, I tested your new patch and experienced a D1 bugcheck while running SCSICompliance test. Some preliminary debugging leads to Line#3633 of nvmestni.c, where it looks like: memset(GET_DATA_BUFFER(pSrb), 0, allocLength); The bugcheck happens when the value of "allocLength" is 0x1000. Apparently, the allocated length in CDB is somewhat a bogus value that is meant to test how driver handles it. Could you please fix it? Thanks, Alex ________________________________ From: nvmewin-bounces at lists.openfabrics.org [mailto:nvmewin-bounces at lists.openfabrics.org] On Behalf Of Dharani Kotte Sent: Wednesday, May 01, 2013 8:56 AM To: Robles, Raymond C; nvmewin at lists.openfabrics.org Subject: [nvmewin] ***UNCHECKED*** [WARNING - ENCRYPTED ATTACHMENT NOT VIRUS SCANNED] New Patch From Sandisk Hi Ray, I have incorporated the changes according to your suggestions, I tried to keep up with the coding convention please feel free to let me know if anything has to be changed. I have incorporated one more fix in the nvmeInit.c which can lead to memory corruptions and used the SrbExt for the buffer allocation. PassWd: sndk1234 Thanks, Dharani. From: Dharani Kotte Sent: Tuesday, April 30, 2013 12:03 PM To: 'Robles, Raymond C'; nvmewin at lists.openfabrics.org Subject: RE: New Patch From Sandisk Thank you for the feedback, I agree that the global buffer is not good I have made the changes to allocate this buffer from the srbext I will provide the code with the below modifications soon. Thanks, Dharani. From: Robles, Raymond C [mailto:raymond.c.robles at intel.com] Sent: Tuesday, April 30, 2013 11:52 AM To: Dharani Kotte; nvmewin at lists.openfabrics.org Subject: RE: New Patch From Sandisk Hi Dharani, Thank you for putting this patch together. Please see my feedback below: - Line 52: Note that we have a file called nvmeSntiTypes.h. This file contains all #defines. As per our existing convention, all #defines should be placed in this file (versus at the top of a .c file). - Line 3339/3412/3502/3593/3756/5391/5433/5442/5453: Our coding convention states no line be longer than 80 characters. These lines all exceed 80 characters. - Line 5410: SntiTranslateAllModePagesResponse() contains new if-else blocks. Our coding convention dictates that open curly braces be placed on the same line as the if or else statement. You can refer to other instances of the code and if-else clauses as examples. - Line 4017: SntiCreateModeDataHeader() no longer needs the SRB pointer to be passed in as a parameter since you are using a global temp buffer to build up the mode sense data. That parameter can be removed. - General: The method of using a global buffer for a single mode sense command opens up a potential race condition for multiple mode sense commands submitted on top of each other. The idea behind putting the SCSI to NVMe translation phase in the HwStorBuildIo phase was to help performance. We initially knew that we would never be sharing any data structures or memory in calls to BuildIo. There is no spinlock or protection for calls to BuildIo and MSDN makes it clear that these BuildIo calls are not synchronized and that any synchronization of memory/data must be done within the function implementations or just use StartIo instead (since Storport will acquire the StartIoSpinLock). So there is hole here w/r/t the global buffer being populated for multiple mode sense commands at the same time. I believe you have the right idea, but the correct solution is to create a temp mode sense buffer inside the SRB extension. This way, each command has its own temp mode sense buffer for building the mode sense response, and then when ready to copy over to the SRB data buffer, it's pulling from a buffer that can only be accessed by the current command (instead of a global buffer that may have been overwritten by another mode sense request call from another BuildIo call). Note that the driver collaboration group made the decision to never acquire any type of lock in BuildIo. Let me know if you have questions on this solution. The additional size to the SRB extension is not an issue. This fix is critical for this patch to be accepted. Thanks, Ray From: nvmewin-bounces at lists.openfabrics.org [mailto:nvmewin-bounces at lists.openfabrics.org] On Behalf Of Dharani Kotte Sent: Tuesday, April 30, 2013 11:14 AM To: Chang, Alex; nvmewin at lists.openfabrics.org Subject: [nvmewin] ***UNCHECKED*** [WARNING - ENCRYPTED ATTACHMENT NOT VIRUS SCANNED] New Patch From Sandisk Attaching the code with according to Alex suggestion. Passwd: sndk1234 Thanks, Dharani. From: Chang, Alex [mailto:Alex.Chang at idt.com] Sent: Tuesday, April 30, 2013 9:23 AM To: Dharani Kotte; nvmewin at lists.openfabrics.org Subject: RE: New Patch From Sandisk Hi Dharani, I'd suggest that it's safer to initialize gModeSenseBuf as all zeros each time before it is used, just in case there are some unexpected data in the buffer. Other than that, I am fine with the patch. Thanks, Alex ________________________________ From: nvmewin-bounces at lists.openfabrics.org [mailto:nvmewin-bounces at lists.openfabrics.org] On Behalf Of Dharani Kotte Sent: Tuesday, April 30, 2013 8:37 AM To: nvmewin at lists.openfabrics.org Subject: [nvmewin] New Patch From Sandisk Hi all, It's been almost two weeks after I sent out the patch. please let us know if you're okay with it. Thanks, Dharani. From: Dharani Kotte Sent: Friday, April 12, 2013 9:58 AM To: nvmewin at lists.openfabrics.org Subject: New Patch From Sandisk Hi all, I am attaching a new patch that includes the following changes: 1. In nvmeSnti.c a. Allocate a global buffer of 256bytes size for modesense return data preparation b. For any mode sense this buffer is used for modesense data along with the requested pages in it. c. Copy the required data according to the alloc_length requested in the cdb to the Srb->DataBuffer and update the dataTransferLength accordingly d. In function SntiTranslateReturnAllModePagesResponse() the modesense data header offset is calculated wrong which ends up in BSOD in some cases modified code to fix this issue Please review the changes and provide feedbacks if you have any. If nobody disagrees with the changes, I will remind Ray to merge them in two weeks. Thanks, Dharani. ________________________________ PLEASE NOTE: The information contained in this electronic mail message is intended only for the use of the designated recipient(s) named above. If the reader of this message is not the intended recipient, you are hereby notified that you have received this message in error and that any review, dissemination, distribution, or copying of this message is strictly prohibited. If you have received this communication in error, please notify the sender by telephone or e-mail (as shown above) immediately and destroy any and all copies of this message in your possession (whether hard copies or electronically stored copies). -------------- next part -------------- An HTML attachment was scrubbed... URL: -------------- next part -------------- A non-text attachment was scrubbed... Name: source_sndk_05_16_2013.zip Type: application/x-zip-compressed Size: 199239 bytes Desc: source_sndk_05_16_2013.zip URL: From Sumant.Patro at sandisk.com Thu May 16 12:43:29 2013 From: Sumant.Patro at sandisk.com (Sumant Patro) Date: Thu, 16 May 2013 19:43:29 +0000 Subject: [nvmewin] =?iso-2022-jp?b?KioqVU5DSEVDS0VEKioqIFtXQVJOSU5HIC0g?= =?iso-2022-jp?b?RU5DUllQVEVEIEFUVEFDSE1FTlQgTk9UIFZJUlVTIFNDQU5O?= =?iso-2022-jp?b?RURdICBSRTogARskQiVLGyhCW1dBUk5JTkcgLSBFTkNSWVBU?= =?iso-2022-jp?b?RUQgQVRUQUNITUVOVCBOT1QgVklSVVMgU0NBTk5FRF0gICAq?= =?iso-2022-jp?b?KipVTkNIRUNLRUQqKiogW1dBUk5JTkcgLSBFTkNSWVBURUQg?= =?iso-2022-jp?b?QVRUQUNITUVOVCBOT1QgVklSVVMJU0NBTk5FRF0gUkU6IE5l?= =?iso-2022-jp?b?dyBQYXRjaCBGcm9tIFNhbmRpc2s=?= Message-ID: Sorry, sent the wrong zip in the previous mail. Please ignore the previous attachment. Attached is the updated zip file. Regards, Sumant From: nvmewin-bounces at lists.openfabrics.org [mailto:nvmewin-bounces at lists.openfabrics.org] On Behalf Of Sumant Patro Sent: Thursday, May 16, 2013 12:38 PM To: Knoblaugh, Rick; Chang, Alex; Dharani Kotte; Robles, Raymond C; nvmewin at lists.openfabrics.org Subject: ニ[WARNING - ENCRYPTED ATTACHMENT NOT VIRUS SCANNED] [nvmewin] ***UNCHECKED*** [WARNING - ENCRYPTED ATTACHMENT NOT VIRUS SCANNED] RE: New Patch From Sandisk Thanks Rick, Alex for your feedback. Please find attached updated files with following two changes : 1. Comment in header of SntiCreateModeDataHeader function changed to reflect passing of srb extension 2. memset(GET_DATA_BUFFER(pSrb), 0, allocLength); modified to use minimum of allocLength and DataTransferLength in SntiReturnAllModePages(..) Password : sndk1234 Regards, Sumant From: nvmewin-bounces at lists.openfabrics.org [mailto:nvmewin-bounces at lists.openfabrics.org] On Behalf Of Knoblaugh, Rick Sent: Thursday, May 16, 2013 11:59 AM To: Chang, Alex; Dharani Kotte; Robles, Raymond C; nvmewin at lists.openfabrics.org Subject: Re: [nvmewin] New Patch From Sandisk Hi Dharani, When you change that one, may as well fix the comment in header of SntiCreateModeDataHeader function to reflect that srb extension is now the parameter that’s passed in. Other stuff looks good. Thanks, -Rick From: nvmewin-bounces at lists.openfabrics.org [mailto:nvmewin-bounces at lists.openfabrics.org] On Behalf Of Chang, Alex Sent: Thursday, May 16, 2013 10:58 AM To: Dharani Kotte; Robles, Raymond C; nvmewin at lists.openfabrics.org Subject: Re: [nvmewin] New Patch From Sandisk Hi Dharani, I tested your new patch and experienced a D1 bugcheck while running SCSICompliance test. Some preliminary debugging leads to Line#3633 of nvmestni.c, where it looks like: memset(GET_DATA_BUFFER(pSrb), 0, allocLength); The bugcheck happens when the value of "allocLength" is 0x1000. Apparently, the allocated length in CDB is somewhat a bogus value that is meant to test how driver handles it. Could you please fix it? Thanks, Alex ________________________________ From: nvmewin-bounces at lists.openfabrics.org [mailto:nvmewin-bounces at lists.openfabrics.org] On Behalf Of Dharani Kotte Sent: Wednesday, May 01, 2013 8:56 AM To: Robles, Raymond C; nvmewin at lists.openfabrics.org Subject: [nvmewin] ***UNCHECKED*** [WARNING - ENCRYPTED ATTACHMENT NOT VIRUS SCANNED] New Patch From Sandisk Hi Ray, I have incorporated the changes according to your suggestions, I tried to keep up with the coding convention please feel free to let me know if anything has to be changed. I have incorporated one more fix in the nvmeInit.c which can lead to memory corruptions and used the SrbExt for the buffer allocation. PassWd: sndk1234 Thanks, Dharani. From: Dharani Kotte Sent: Tuesday, April 30, 2013 12:03 PM To: 'Robles, Raymond C'; nvmewin at lists.openfabrics.org Subject: RE: New Patch From Sandisk Thank you for the feedback, I agree that the global buffer is not good I have made the changes to allocate this buffer from the srbext I will provide the code with the below modifications soon. Thanks, Dharani. From: Robles, Raymond C [mailto:raymond.c.robles at intel.com] Sent: Tuesday, April 30, 2013 11:52 AM To: Dharani Kotte; nvmewin at lists.openfabrics.org Subject: RE: New Patch From Sandisk Hi Dharani, Thank you for putting this patch together. Please see my feedback below: - Line 52: Note that we have a file called nvmeSntiTypes.h. This file contains all #defines. As per our existing convention, all #defines should be placed in this file (versus at the top of a .c file). - Line 3339/3412/3502/3593/3756/5391/5433/5442/5453: Our coding convention states no line be longer than 80 characters. These lines all exceed 80 characters. - Line 5410: SntiTranslateAllModePagesResponse() contains new if-else blocks. Our coding convention dictates that open curly braces be placed on the same line as the if or else statement. You can refer to other instances of the code and if-else clauses as examples. - Line 4017: SntiCreateModeDataHeader() no longer needs the SRB pointer to be passed in as a parameter since you are using a global temp buffer to build up the mode sense data. That parameter can be removed. - General: The method of using a global buffer for a single mode sense command opens up a potential race condition for multiple mode sense commands submitted on top of each other. The idea behind putting the SCSI to NVMe translation phase in the HwStorBuildIo phase was to help performance. We initially knew that we would never be sharing any data structures or memory in calls to BuildIo. There is no spinlock or protection for calls to BuildIo and MSDN makes it clear that these BuildIo calls are not synchronized and that any synchronization of memory/data must be done within the function implementations or just use StartIo instead (since Storport will acquire the StartIoSpinLock). So there is hole here w/r/t the global buffer being populated for multiple mode sense commands at the same time. I believe you have the right idea, but the correct solution is to create a temp mode sense buffer inside the SRB extension. This way, each command has its own temp mode sense buffer for building the mode sense response, and then when ready to copy over to the SRB data buffer, it’s pulling from a buffer that can only be accessed by the current command (instead of a global buffer that may have been overwritten by another mode sense request call from another BuildIo call). Note that the driver collaboration group made the decision to never acquire any type of lock in BuildIo. Let me know if you have questions on this solution. The additional size to the SRB extension is not an issue. This fix is critical for this patch to be accepted. Thanks, Ray From: nvmewin-bounces at lists.openfabrics.org [mailto:nvmewin-bounces at lists.openfabrics.org] On Behalf Of Dharani Kotte Sent: Tuesday, April 30, 2013 11:14 AM To: Chang, Alex; nvmewin at lists.openfabrics.org Subject: [nvmewin] ***UNCHECKED*** [WARNING - ENCRYPTED ATTACHMENT NOT VIRUS SCANNED] New Patch From Sandisk Attaching the code with according to Alex suggestion. Passwd: sndk1234 Thanks, Dharani. From: Chang, Alex [mailto:Alex.Chang at idt.com] Sent: Tuesday, April 30, 2013 9:23 AM To: Dharani Kotte; nvmewin at lists.openfabrics.org Subject: RE: New Patch From Sandisk Hi Dharani, I'd suggest that it's safer to initialize gModeSenseBuf as all zeros each time before it is used, just in case there are some unexpected data in the buffer. Other than that, I am fine with the patch. Thanks, Alex ________________________________ From: nvmewin-bounces at lists.openfabrics.org [mailto:nvmewin-bounces at lists.openfabrics.org] On Behalf Of Dharani Kotte Sent: Tuesday, April 30, 2013 8:37 AM To: nvmewin at lists.openfabrics.org Subject: [nvmewin] New Patch From Sandisk Hi all, It's been almost two weeks after I sent out the patch. please let us know if you're okay with it. Thanks, Dharani. From: Dharani Kotte Sent: Friday, April 12, 2013 9:58 AM To: nvmewin at lists.openfabrics.org Subject: New Patch From Sandisk Hi all, I am attaching a new patch that includes the following changes: 1. In nvmeSnti.c a. Allocate a global buffer of 256bytes size for modesense return data preparation b. For any mode sense this buffer is used for modesense data along with the requested pages in it. c. Copy the required data according to the alloc_length requested in the cdb to the Srb->DataBuffer and update the dataTransferLength accordingly d. In function SntiTranslateReturnAllModePagesResponse() the modesense data header offset is calculated wrong which ends up in BSOD in some cases modified code to fix this issue Please review the changes and provide feedbacks if you have any. If nobody disagrees with the changes, I will remind Ray to merge them in two weeks. Thanks, Dharani. ________________________________ PLEASE NOTE: The information contained in this electronic mail message is intended only for the use of the designated recipient(s) named above. If the reader of this message is not the intended recipient, you are hereby notified that you have received this message in error and that any review, dissemination, distribution, or copying of this message is strictly prohibited. If you have received this communication in error, please notify the sender by telephone or e-mail (as shown above) immediately and destroy any and all copies of this message in your possession (whether hard copies or electronically stored copies). -------------- next part -------------- An HTML attachment was scrubbed... URL: -------------- next part -------------- A non-text attachment was scrubbed... Name: source_sndk_05_16_2013.zip Type: application/x-zip-compressed Size: 199240 bytes Desc: source_sndk_05_16_2013.zip URL: From raymond.c.robles at intel.com Thu May 16 13:37:17 2013 From: raymond.c.robles at intel.com (Robles, Raymond C) Date: Thu, 16 May 2013 20:37:17 +0000 Subject: [nvmewin] New Patch From Sandisk Message-ID: <49158E750348AA499168FD41D88983606256082A@FMSMSX105.amr.corp.intel.com> Hi Sumant/Dharani, Thanks for the updated patch. A couple of comments/questions: - For all the mode sense page translations, the new temp buffer allocated in the SRB extension is used. But for the Modes Sense 0x3C (all mode pages), the SRB data buffer is still memset to 0 at the beginning of SntiReturnAllModePages(). I’m not sure we still need to do this considering how the temp buffer is used to build the mode sense data. - If you insist on keeping that memset in there, there is a #define in for all mode pages - MODE_SENSE_ALL_PAGES_LENGTH. It may be better to zero out the buffer based on the min between this define and allocLength. SRB data transfer length cannot be relied on for “data-in” commands (like mode sense). Thanks, Ray From: Sumant Patro [mailto:Sumant.Patro at sandisk.com] Sent: Thursday, May 16, 2013 12:43 PM To: Knoblaugh, Rick; Chang, Alex; Dharani Kotte; Robles, Raymond C; nvmewin at lists.openfabrics.org Subject: [WARNING - ENCRYPTED ATTACHMENT NOT VIRUS SCANNED] RE: ニ[WARNING - ENCRYPTED ATTACHMENT NOT VIRUS SCANNED] [nvmewin] ***UNCHECKED*** [WARNING - ENCRYPTED ATTACHMENT NOT VIRUS SCANNED] RE: New Patch From Sandisk Sorry, sent the wrong zip in the previous mail. Please ignore the previous attachment. Attached is the updated zip file. Regards, Sumant From: nvmewin-bounces at lists.openfabrics.org [mailto:nvmewin-bounces at lists.openfabrics.org] On Behalf Of Sumant Patro Sent: Thursday, May 16, 2013 12:38 PM To: Knoblaugh, Rick; Chang, Alex; Dharani Kotte; Robles, Raymond C; nvmewin at lists.openfabrics.org Subject: ニ[WARNING - ENCRYPTED ATTACHMENT NOT VIRUS SCANNED] [nvmewin] ***UNCHECKED*** [WARNING - ENCRYPTED ATTACHMENT NOT VIRUS SCANNED] RE: New Patch From Sandisk Thanks Rick, Alex for your feedback. Please find attached updated files with following two changes : 1. Comment in header of SntiCreateModeDataHeader function changed to reflect passing of srb extension 2. memset(GET_DATA_BUFFER(pSrb), 0, allocLength); modified to use minimum of allocLength and DataTransferLength in SntiReturnAllModePages(..) Password : sndk1234 Regards, Sumant From: nvmewin-bounces at lists.openfabrics.org [mailto:nvmewin-bounces at lists.openfabrics.org] On Behalf Of Knoblaugh, Rick Sent: Thursday, May 16, 2013 11:59 AM To: Chang, Alex; Dharani Kotte; Robles, Raymond C; nvmewin at lists.openfabrics.org Subject: Re: [nvmewin] New Patch From Sandisk Hi Dharani, When you change that one, may as well fix the comment in header of SntiCreateModeDataHeader function to reflect that srb extension is now the parameter that’s passed in. Other stuff looks good. Thanks, -Rick From: nvmewin-bounces at lists.openfabrics.org [mailto:nvmewin-bounces at lists.openfabrics.org] On Behalf Of Chang, Alex Sent: Thursday, May 16, 2013 10:58 AM To: Dharani Kotte; Robles, Raymond C; nvmewin at lists.openfabrics.org Subject: Re: [nvmewin] New Patch From Sandisk Hi Dharani, I tested your new patch and experienced a D1 bugcheck while running SCSICompliance test. Some preliminary debugging leads to Line#3633 of nvmestni.c, where it looks like: memset(GET_DATA_BUFFER(pSrb), 0, allocLength); The bugcheck happens when the value of "allocLength" is 0x1000. Apparently, the allocated length in CDB is somewhat a bogus value that is meant to test how driver handles it. Could you please fix it? Thanks, Alex ________________________________ From: nvmewin-bounces at lists.openfabrics.org [mailto:nvmewin-bounces at lists.openfabrics.org] On Behalf Of Dharani Kotte Sent: Wednesday, May 01, 2013 8:56 AM To: Robles, Raymond C; nvmewin at lists.openfabrics.org Subject: [nvmewin] ***UNCHECKED*** [WARNING - ENCRYPTED ATTACHMENT NOT VIRUS SCANNED] New Patch From Sandisk Hi Ray, I have incorporated the changes according to your suggestions, I tried to keep up with the coding convention please feel free to let me know if anything has to be changed. I have incorporated one more fix in the nvmeInit.c which can lead to memory corruptions and used the SrbExt for the buffer allocation. PassWd: sndk1234 Thanks, Dharani. From: Dharani Kotte Sent: Tuesday, April 30, 2013 12:03 PM To: 'Robles, Raymond C'; nvmewin at lists.openfabrics.org Subject: RE: New Patch From Sandisk Thank you for the feedback, I agree that the global buffer is not good I have made the changes to allocate this buffer from the srbext I will provide the code with the below modifications soon. Thanks, Dharani. From: Robles, Raymond C [mailto:raymond.c.robles at intel.com] Sent: Tuesday, April 30, 2013 11:52 AM To: Dharani Kotte; nvmewin at lists.openfabrics.org Subject: RE: New Patch From Sandisk Hi Dharani, Thank you for putting this patch together. Please see my feedback below: - Line 52: Note that we have a file called nvmeSntiTypes.h. This file contains all #defines. As per our existing convention, all #defines should be placed in this file (versus at the top of a .c file). - Line 3339/3412/3502/3593/3756/5391/5433/5442/5453: Our coding convention states no line be longer than 80 characters. These lines all exceed 80 characters. - Line 5410: SntiTranslateAllModePagesResponse() contains new if-else blocks. Our coding convention dictates that open curly braces be placed on the same line as the if or else statement. You can refer to other instances of the code and if-else clauses as examples. - Line 4017: SntiCreateModeDataHeader() no longer needs the SRB pointer to be passed in as a parameter since you are using a global temp buffer to build up the mode sense data. That parameter can be removed. - General: The method of using a global buffer for a single mode sense command opens up a potential race condition for multiple mode sense commands submitted on top of each other. The idea behind putting the SCSI to NVMe translation phase in the HwStorBuildIo phase was to help performance. We initially knew that we would never be sharing any data structures or memory in calls to BuildIo. There is no spinlock or protection for calls to BuildIo and MSDN makes it clear that these BuildIo calls are not synchronized and that any synchronization of memory/data must be done within the function implementations or just use StartIo instead (since Storport will acquire the StartIoSpinLock). So there is hole here w/r/t the global buffer being populated for multiple mode sense commands at the same time. I believe you have the right idea, but the correct solution is to create a temp mode sense buffer inside the SRB extension. This way, each command has its own temp mode sense buffer for building the mode sense response, and then when ready to copy over to the SRB data buffer, it’s pulling from a buffer that can only be accessed by the current command (instead of a global buffer that may have been overwritten by another mode sense request call from another BuildIo call). Note that the driver collaboration group made the decision to never acquire any type of lock in BuildIo. Let me know if you have questions on this solution. The additional size to the SRB extension is not an issue. This fix is critical for this patch to be accepted. Thanks, Ray From: nvmewin-bounces at lists.openfabrics.org [mailto:nvmewin-bounces at lists.openfabrics.org] On Behalf Of Dharani Kotte Sent: Tuesday, April 30, 2013 11:14 AM To: Chang, Alex; nvmewin at lists.openfabrics.org Subject: [nvmewin] ***UNCHECKED*** [WARNING - ENCRYPTED ATTACHMENT NOT VIRUS SCANNED] New Patch From Sandisk Attaching the code with according to Alex suggestion. Passwd: sndk1234 Thanks, Dharani. From: Chang, Alex [mailto:Alex.Chang at idt.com] Sent: Tuesday, April 30, 2013 9:23 AM To: Dharani Kotte; nvmewin at lists.openfabrics.org Subject: RE: New Patch From Sandisk Hi Dharani, I'd suggest that it's safer to initialize gModeSenseBuf as all zeros each time before it is used, just in case there are some unexpected data in the buffer. Other than that, I am fine with the patch. Thanks, Alex ________________________________ From: nvmewin-bounces at lists.openfabrics.org [mailto:nvmewin-bounces at lists.openfabrics.org] On Behalf Of Dharani Kotte Sent: Tuesday, April 30, 2013 8:37 AM To: nvmewin at lists.openfabrics.org Subject: [nvmewin] New Patch From Sandisk Hi all, It's been almost two weeks after I sent out the patch. please let us know if you're okay with it. Thanks, Dharani. From: Dharani Kotte Sent: Friday, April 12, 2013 9:58 AM To: nvmewin at lists.openfabrics.org Subject: New Patch From Sandisk Hi all, I am attaching a new patch that includes the following changes: 1. In nvmeSnti.c a. Allocate a global buffer of 256bytes size for modesense return data preparation b. For any mode sense this buffer is used for modesense data along with the requested pages in it. c. Copy the required data according to the alloc_length requested in the cdb to the Srb->DataBuffer and update the dataTransferLength accordingly d. In function SntiTranslateReturnAllModePagesResponse() the modesense data header offset is calculated wrong which ends up in BSOD in some cases modified code to fix this issue Please review the changes and provide feedbacks if you have any. If nobody disagrees with the changes, I will remind Ray to merge them in two weeks. Thanks, Dharani. ________________________________ PLEASE NOTE: The information contained in this electronic mail message is intended only for the use of the designated recipient(s) named above. If the reader of this message is not the intended recipient, you are hereby notified that you have received this message in error and that any review, dissemination, distribution, or copying of this message is strictly prohibited. If you have received this communication in error, please notify the sender by telephone or e-mail (as shown above) immediately and destroy any and all copies of this message in your possession (whether hard copies or electronically stored copies). CAUTION: Please confirm that the password protected .zip attachment which contains the file(s) of type source_sndk_05_16_2013.zip is legitimate prior to opening. To make sure this message is not infected with a virus, it is important to verify that you are expecting the message or else confirm its legitimacy with the sender. -------------- next part -------------- An HTML attachment was scrubbed... URL: From Alex.Chang at idt.com Thu May 16 13:48:29 2013 From: Alex.Chang at idt.com (Chang, Alex) Date: Thu, 16 May 2013 20:48:29 +0000 Subject: [nvmewin] New Patch From Sandisk In-Reply-To: <49158E750348AA499168FD41D88983606256082A@FMSMSX105.amr.corp.intel.com> References: <49158E750348AA499168FD41D88983606256082A@FMSMSX105.amr.corp.intel.com> Message-ID: <548C5470AAD9DA4A85D259B663190D361FFE95DF@corpmail1.na.ads.idt.com> Hi Sumant/Dharani, I did not receive your new patch sent out earlier today. Could you please send me a copy? Thanks, Alex ________________________________ From: Robles, Raymond C [mailto:raymond.c.robles at intel.com] Sent: Thursday, May 16, 2013 1:37 PM To: Sumant Patro; Knoblaugh, Rick; Chang, Alex; Dharani Kotte; nvmewin at lists.openfabrics.org Subject: RE: New Patch From Sandisk Hi Sumant/Dharani, Thanks for the updated patch. A couple of comments/questions: - For all the mode sense page translations, the new temp buffer allocated in the SRB extension is used. But for the Modes Sense 0x3C (all mode pages), the SRB data buffer is still memset to 0 at the beginning of SntiReturnAllModePages(). I’m not sure we still need to do this considering how the temp buffer is used to build the mode sense data. - If you insist on keeping that memset in there, there is a #define in for all mode pages - MODE_SENSE_ALL_PAGES_LENGTH. It may be better to zero out the buffer based on the min between this define and allocLength. SRB data transfer length cannot be relied on for “data-in” commands (like mode sense). Thanks, Ray From: Sumant Patro [mailto:Sumant.Patro at sandisk.com] Sent: Thursday, May 16, 2013 12:43 PM To: Knoblaugh, Rick; Chang, Alex; Dharani Kotte; Robles, Raymond C; nvmewin at lists.openfabrics.org Subject: [WARNING - ENCRYPTED ATTACHMENT NOT VIRUS SCANNED] RE: ニ[WARNING - ENCRYPTED ATTACHMENT NOT VIRUS SCANNED] [nvmewin] ***UNCHECKED*** [WARNING - ENCRYPTED ATTACHMENT NOT VIRUS SCANNED] RE: New Patch From Sandisk Sorry, sent the wrong zip in the previous mail. Please ignore the previous attachment. Attached is the updated zip file. Regards, Sumant From: nvmewin-bounces at lists.openfabrics.org [mailto:nvmewin-bounces at lists.openfabrics.org] On Behalf Of Sumant Patro Sent: Thursday, May 16, 2013 12:38 PM To: Knoblaugh, Rick; Chang, Alex; Dharani Kotte; Robles, Raymond C; nvmewin at lists.openfabrics.org Subject: ニ[WARNING - ENCRYPTED ATTACHMENT NOT VIRUS SCANNED] [nvmewin] ***UNCHECKED*** [WARNING - ENCRYPTED ATTACHMENT NOT VIRUS SCANNED] RE: New Patch From Sandisk Thanks Rick, Alex for your feedback. Please find attached updated files with following two changes : 1. Comment in header of SntiCreateModeDataHeader function changed to reflect passing of srb extension 2. memset(GET_DATA_BUFFER(pSrb), 0, allocLength); modified to use minimum of allocLength and DataTransferLength in SntiReturnAllModePages(..) Password : sndk1234 Regards, Sumant From: nvmewin-bounces at lists.openfabrics.org [mailto:nvmewin-bounces at lists.openfabrics.org] On Behalf Of Knoblaugh, Rick Sent: Thursday, May 16, 2013 11:59 AM To: Chang, Alex; Dharani Kotte; Robles, Raymond C; nvmewin at lists.openfabrics.org Subject: Re: [nvmewin] New Patch From Sandisk Hi Dharani, When you change that one, may as well fix the comment in header of SntiCreateModeDataHeader function to reflect that srb extension is now the parameter that’s passed in. Other stuff looks good. Thanks, -Rick From: nvmewin-bounces at lists.openfabrics.org [mailto:nvmewin-bounces at lists.openfabrics.org] On Behalf Of Chang, Alex Sent: Thursday, May 16, 2013 10:58 AM To: Dharani Kotte; Robles, Raymond C; nvmewin at lists.openfabrics.org Subject: Re: [nvmewin] New Patch From Sandisk Hi Dharani, I tested your new patch and experienced a D1 bugcheck while running SCSICompliance test. Some preliminary debugging leads to Line#3633 of nvmestni.c, where it looks like: memset(GET_DATA_BUFFER(pSrb), 0, allocLength); The bugcheck happens when the value of "allocLength" is 0x1000. Apparently, the allocated length in CDB is somewhat a bogus value that is meant to test how driver handles it. Could you please fix it? Thanks, Alex ________________________________ From: nvmewin-bounces at lists.openfabrics.org [mailto:nvmewin-bounces at lists.openfabrics.org] On Behalf Of Dharani Kotte Sent: Wednesday, May 01, 2013 8:56 AM To: Robles, Raymond C; nvmewin at lists.openfabrics.org Subject: [nvmewin] ***UNCHECKED*** [WARNING - ENCRYPTED ATTACHMENT NOT VIRUS SCANNED] New Patch From Sandisk Hi Ray, I have incorporated the changes according to your suggestions, I tried to keep up with the coding convention please feel free to let me know if anything has to be changed. I have incorporated one more fix in the nvmeInit.c which can lead to memory corruptions and used the SrbExt for the buffer allocation. PassWd: sndk1234 Thanks, Dharani. From: Dharani Kotte Sent: Tuesday, April 30, 2013 12:03 PM To: 'Robles, Raymond C'; nvmewin at lists.openfabrics.org Subject: RE: New Patch From Sandisk Thank you for the feedback, I agree that the global buffer is not good I have made the changes to allocate this buffer from the srbext I will provide the code with the below modifications soon. Thanks, Dharani. From: Robles, Raymond C [mailto:raymond.c.robles at intel.com] Sent: Tuesday, April 30, 2013 11:52 AM To: Dharani Kotte; nvmewin at lists.openfabrics.org Subject: RE: New Patch From Sandisk Hi Dharani, Thank you for putting this patch together. Please see my feedback below: - Line 52: Note that we have a file called nvmeSntiTypes.h. This file contains all #defines. As per our existing convention, all #defines should be placed in this file (versus at the top of a .c file). - Line 3339/3412/3502/3593/3756/5391/5433/5442/5453: Our coding convention states no line be longer than 80 characters. These lines all exceed 80 characters. - Line 5410: SntiTranslateAllModePagesResponse() contains new if-else blocks. Our coding convention dictates that open curly braces be placed on the same line as the if or else statement. You can refer to other instances of the code and if-else clauses as examples. - Line 4017: SntiCreateModeDataHeader() no longer needs the SRB pointer to be passed in as a parameter since you are using a global temp buffer to build up the mode sense data. That parameter can be removed. - General: The method of using a global buffer for a single mode sense command opens up a potential race condition for multiple mode sense commands submitted on top of each other. The idea behind putting the SCSI to NVMe translation phase in the HwStorBuildIo phase was to help performance. We initially knew that we would never be sharing any data structures or memory in calls to BuildIo. There is no spinlock or protection for calls to BuildIo and MSDN makes it clear that these BuildIo calls are not synchronized and that any synchronization of memory/data must be done within the function implementations or just use StartIo instead (since Storport will acquire the StartIoSpinLock). So there is hole here w/r/t the global buffer being populated for multiple mode sense commands at the same time. I believe you have the right idea, but the correct solution is to create a temp mode sense buffer inside the SRB extension. This way, each command has its own temp mode sense buffer for building the mode sense response, and then when ready to copy over to the SRB data buffer, it’s pulling from a buffer that can only be accessed by the current command (instead of a global buffer that may have been overwritten by another mode sense request call from another BuildIo call). Note that the driver collaboration group made the decision to never acquire any type of lock in BuildIo. Let me know if you have questions on this solution. The additional size to the SRB extension is not an issue. This fix is critical for this patch to be accepted. Thanks, Ray From: nvmewin-bounces at lists.openfabrics.org [mailto:nvmewin-bounces at lists.openfabrics.org] On Behalf Of Dharani Kotte Sent: Tuesday, April 30, 2013 11:14 AM To: Chang, Alex; nvmewin at lists.openfabrics.org Subject: [nvmewin] ***UNCHECKED*** [WARNING - ENCRYPTED ATTACHMENT NOT VIRUS SCANNED] New Patch From Sandisk Attaching the code with according to Alex suggestion. Passwd: sndk1234 Thanks, Dharani. From: Chang, Alex [mailto:Alex.Chang at idt.com] Sent: Tuesday, April 30, 2013 9:23 AM To: Dharani Kotte; nvmewin at lists.openfabrics.org Subject: RE: New Patch From Sandisk Hi Dharani, I'd suggest that it's safer to initialize gModeSenseBuf as all zeros each time before it is used, just in case there are some unexpected data in the buffer. Other than that, I am fine with the patch. Thanks, Alex ________________________________ From: nvmewin-bounces at lists.openfabrics.org [mailto:nvmewin-bounces at lists.openfabrics.org] On Behalf Of Dharani Kotte Sent: Tuesday, April 30, 2013 8:37 AM To: nvmewin at lists.openfabrics.org Subject: [nvmewin] New Patch From Sandisk Hi all, It's been almost two weeks after I sent out the patch. please let us know if you're okay with it. Thanks, Dharani. From: Dharani Kotte Sent: Friday, April 12, 2013 9:58 AM To: nvmewin at lists.openfabrics.org Subject: New Patch From Sandisk Hi all, I am attaching a new patch that includes the following changes: 1. In nvmeSnti.c a. Allocate a global buffer of 256bytes size for modesense return data preparation b. For any mode sense this buffer is used for modesense data along with the requested pages in it. c. Copy the required data according to the alloc_length requested in the cdb to the Srb->DataBuffer and update the dataTransferLength accordingly d. In function SntiTranslateReturnAllModePagesResponse() the modesense data header offset is calculated wrong which ends up in BSOD in some cases modified code to fix this issue Please review the changes and provide feedbacks if you have any. If nobody disagrees with the changes, I will remind Ray to merge them in two weeks. Thanks, Dharani. ________________________________ PLEASE NOTE: The information contained in this electronic mail message is intended only for the use of the designated recipient(s) named above. If the reader of this message is not the intended recipient, you are hereby notified that you have received this message in error and that any review, dissemination, distribution, or copying of this message is strictly prohibited. If you have received this communication in error, please notify the sender by telephone or e-mail (as shown above) immediately and destroy any and all copies of this message in your possession (whether hard copies or electronically stored copies). CAUTION: Please confirm that the password protected .zip attachment which contains the file(s) of type source_sndk_05_16_2013.zip is legitimate prior to opening. To make sure this message is not infected with a virus, it is important to verify that you are expecting the message or else confirm its legitimacy with the sender. -------------- next part -------------- An HTML attachment was scrubbed... URL: From Sumant.Patro at sandisk.com Thu May 16 14:12:09 2013 From: Sumant.Patro at sandisk.com (Sumant Patro) Date: Thu, 16 May 2013 21:12:09 +0000 Subject: [nvmewin] ***UNCHECKED*** [WARNING - ENCRYPTED ATTACHMENT NOT VIRUS SCANNED] RE: New Patch From Sandisk In-Reply-To: <548C5470AAD9DA4A85D259B663190D361FFE95DF@corpmail1.na.ads.idt.com> References: <49158E750348AA499168FD41D88983606256082A@FMSMSX105.amr.corp.intel.com> <548C5470AAD9DA4A85D259B663190D361FFE95DF@corpmail1.na.ads.idt.com> Message-ID: Alex, please find attached the files. Regards, Sumant From: Chang, Alex [mailto:Alex.Chang at idt.com] Sent: Thursday, May 16, 2013 1:48 PM To: Robles, Raymond C; Sumant Patro; Knoblaugh, Rick; Dharani Kotte; nvmewin at lists.openfabrics.org Subject: RE: New Patch From Sandisk Hi Sumant/Dharani, I did not receive your new patch sent out earlier today. Could you please send me a copy? Thanks, Alex ________________________________ From: Robles, Raymond C [mailto:raymond.c.robles at intel.com] Sent: Thursday, May 16, 2013 1:37 PM To: Sumant Patro; Knoblaugh, Rick; Chang, Alex; Dharani Kotte; nvmewin at lists.openfabrics.org Subject: RE: New Patch From Sandisk Hi Sumant/Dharani, Thanks for the updated patch. A couple of comments/questions: - For all the mode sense page translations, the new temp buffer allocated in the SRB extension is used. But for the Modes Sense 0x3C (all mode pages), the SRB data buffer is still memset to 0 at the beginning of SntiReturnAllModePages(). I’m not sure we still need to do this considering how the temp buffer is used to build the mode sense data. - If you insist on keeping that memset in there, there is a #define in for all mode pages - MODE_SENSE_ALL_PAGES_LENGTH. It may be better to zero out the buffer based on the min between this define and allocLength. SRB data transfer length cannot be relied on for “data-in” commands (like mode sense). Thanks, Ray From: Sumant Patro [mailto:Sumant.Patro at sandisk.com] Sent: Thursday, May 16, 2013 12:43 PM To: Knoblaugh, Rick; Chang, Alex; Dharani Kotte; Robles, Raymond C; nvmewin at lists.openfabrics.org Subject: [WARNING - ENCRYPTED ATTACHMENT NOT VIRUS SCANNED] RE: ニ[WARNING - ENCRYPTED ATTACHMENT NOT VIRUS SCANNED] [nvmewin] ***UNCHECKED*** [WARNING - ENCRYPTED ATTACHMENT NOT VIRUS SCANNED] RE: New Patch From Sandisk Sorry, sent the wrong zip in the previous mail. Please ignore the previous attachment. Attached is the updated zip file. Regards, Sumant From: nvmewin-bounces at lists.openfabrics.org [mailto:nvmewin-bounces at lists.openfabrics.org] On Behalf Of Sumant Patro Sent: Thursday, May 16, 2013 12:38 PM To: Knoblaugh, Rick; Chang, Alex; Dharani Kotte; Robles, Raymond C; nvmewin at lists.openfabrics.org Subject: ニ[WARNING - ENCRYPTED ATTACHMENT NOT VIRUS SCANNED] [nvmewin] ***UNCHECKED*** [WARNING - ENCRYPTED ATTACHMENT NOT VIRUS SCANNED] RE: New Patch From Sandisk Thanks Rick, Alex for your feedback. Please find attached updated files with following two changes : 1. Comment in header of SntiCreateModeDataHeader function changed to reflect passing of srb extension 2. memset(GET_DATA_BUFFER(pSrb), 0, allocLength); modified to use minimum of allocLength and DataTransferLength in SntiReturnAllModePages(..) Password : sndk1234 Regards, Sumant From: nvmewin-bounces at lists.openfabrics.org [mailto:nvmewin-bounces at lists.openfabrics.org] On Behalf Of Knoblaugh, Rick Sent: Thursday, May 16, 2013 11:59 AM To: Chang, Alex; Dharani Kotte; Robles, Raymond C; nvmewin at lists.openfabrics.org Subject: Re: [nvmewin] New Patch From Sandisk Hi Dharani, When you change that one, may as well fix the comment in header of SntiCreateModeDataHeader function to reflect that srb extension is now the parameter that’s passed in. Other stuff looks good. Thanks, -Rick From: nvmewin-bounces at lists.openfabrics.org [mailto:nvmewin-bounces at lists.openfabrics.org] On Behalf Of Chang, Alex Sent: Thursday, May 16, 2013 10:58 AM To: Dharani Kotte; Robles, Raymond C; nvmewin at lists.openfabrics.org Subject: Re: [nvmewin] New Patch From Sandisk Hi Dharani, I tested your new patch and experienced a D1 bugcheck while running SCSICompliance test. Some preliminary debugging leads to Line#3633 of nvmestni.c, where it looks like: memset(GET_DATA_BUFFER(pSrb), 0, allocLength); The bugcheck happens when the value of "allocLength" is 0x1000. Apparently, the allocated length in CDB is somewhat a bogus value that is meant to test how driver handles it. Could you please fix it? Thanks, Alex ________________________________ From: nvmewin-bounces at lists.openfabrics.org [mailto:nvmewin-bounces at lists.openfabrics.org] On Behalf Of Dharani Kotte Sent: Wednesday, May 01, 2013 8:56 AM To: Robles, Raymond C; nvmewin at lists.openfabrics.org Subject: [nvmewin] ***UNCHECKED*** [WARNING - ENCRYPTED ATTACHMENT NOT VIRUS SCANNED] New Patch From Sandisk Hi Ray, I have incorporated the changes according to your suggestions, I tried to keep up with the coding convention please feel free to let me know if anything has to be changed. I have incorporated one more fix in the nvmeInit.c which can lead to memory corruptions and used the SrbExt for the buffer allocation. PassWd: sndk1234 Thanks, Dharani. From: Dharani Kotte Sent: Tuesday, April 30, 2013 12:03 PM To: 'Robles, Raymond C'; nvmewin at lists.openfabrics.org Subject: RE: New Patch From Sandisk Thank you for the feedback, I agree that the global buffer is not good I have made the changes to allocate this buffer from the srbext I will provide the code with the below modifications soon. Thanks, Dharani. From: Robles, Raymond C [mailto:raymond.c.robles at intel.com] Sent: Tuesday, April 30, 2013 11:52 AM To: Dharani Kotte; nvmewin at lists.openfabrics.org Subject: RE: New Patch From Sandisk Hi Dharani, Thank you for putting this patch together. Please see my feedback below: - Line 52: Note that we have a file called nvmeSntiTypes.h. This file contains all #defines. As per our existing convention, all #defines should be placed in this file (versus at the top of a .c file). - Line 3339/3412/3502/3593/3756/5391/5433/5442/5453: Our coding convention states no line be longer than 80 characters. These lines all exceed 80 characters. - Line 5410: SntiTranslateAllModePagesResponse() contains new if-else blocks. Our coding convention dictates that open curly braces be placed on the same line as the if or else statement. You can refer to other instances of the code and if-else clauses as examples. - Line 4017: SntiCreateModeDataHeader() no longer needs the SRB pointer to be passed in as a parameter since you are using a global temp buffer to build up the mode sense data. That parameter can be removed. - General: The method of using a global buffer for a single mode sense command opens up a potential race condition for multiple mode sense commands submitted on top of each other. The idea behind putting the SCSI to NVMe translation phase in the HwStorBuildIo phase was to help performance. We initially knew that we would never be sharing any data structures or memory in calls to BuildIo. There is no spinlock or protection for calls to BuildIo and MSDN makes it clear that these BuildIo calls are not synchronized and that any synchronization of memory/data must be done within the function implementations or just use StartIo instead (since Storport will acquire the StartIoSpinLock). So there is hole here w/r/t the global buffer being populated for multiple mode sense commands at the same time. I believe you have the right idea, but the correct solution is to create a temp mode sense buffer inside the SRB extension. This way, each command has its own temp mode sense buffer for building the mode sense response, and then when ready to copy over to the SRB data buffer, it’s pulling from a buffer that can only be accessed by the current command (instead of a global buffer that may have been overwritten by another mode sense request call from another BuildIo call). Note that the driver collaboration group made the decision to never acquire any type of lock in BuildIo. Let me know if you have questions on this solution. The additional size to the SRB extension is not an issue. This fix is critical for this patch to be accepted. Thanks, Ray From: nvmewin-bounces at lists.openfabrics.org [mailto:nvmewin-bounces at lists.openfabrics.org] On Behalf Of Dharani Kotte Sent: Tuesday, April 30, 2013 11:14 AM To: Chang, Alex; nvmewin at lists.openfabrics.org Subject: [nvmewin] ***UNCHECKED*** [WARNING - ENCRYPTED ATTACHMENT NOT VIRUS SCANNED] New Patch From Sandisk Attaching the code with according to Alex suggestion. Passwd: sndk1234 Thanks, Dharani. From: Chang, Alex [mailto:Alex.Chang at idt.com] Sent: Tuesday, April 30, 2013 9:23 AM To: Dharani Kotte; nvmewin at lists.openfabrics.org Subject: RE: New Patch From Sandisk Hi Dharani, I'd suggest that it's safer to initialize gModeSenseBuf as all zeros each time before it is used, just in case there are some unexpected data in the buffer. Other than that, I am fine with the patch. Thanks, Alex ________________________________ From: nvmewin-bounces at lists.openfabrics.org [mailto:nvmewin-bounces at lists.openfabrics.org] On Behalf Of Dharani Kotte Sent: Tuesday, April 30, 2013 8:37 AM To: nvmewin at lists.openfabrics.org Subject: [nvmewin] New Patch From Sandisk Hi all, It's been almost two weeks after I sent out the patch. please let us know if you're okay with it. Thanks, Dharani. From: Dharani Kotte Sent: Friday, April 12, 2013 9:58 AM To: nvmewin at lists.openfabrics.org Subject: New Patch From Sandisk Hi all, I am attaching a new patch that includes the following changes: 1. In nvmeSnti.c a. Allocate a global buffer of 256bytes size for modesense return data preparation b. For any mode sense this buffer is used for modesense data along with the requested pages in it. c. Copy the required data according to the alloc_length requested in the cdb to the Srb->DataBuffer and update the dataTransferLength accordingly d. In function SntiTranslateReturnAllModePagesResponse() the modesense data header offset is calculated wrong which ends up in BSOD in some cases modified code to fix this issue Please review the changes and provide feedbacks if you have any. If nobody disagrees with the changes, I will remind Ray to merge them in two weeks. Thanks, Dharani. ________________________________ PLEASE NOTE: The information contained in this electronic mail message is intended only for the use of the designated recipient(s) named above. If the reader of this message is not the intended recipient, you are hereby notified that you have received this message in error and that any review, dissemination, distribution, or copying of this message is strictly prohibited. If you have received this communication in error, please notify the sender by telephone or e-mail (as shown above) immediately and destroy any and all copies of this message in your possession (whether hard copies or electronically stored copies). CAUTION: Please confirm that the password protected .zip attachment which contains the file(s) of type source_sndk_05_16_2013.zip is legitimate prior to opening. To make sure this message is not infected with a virus, it is important to verify that you are expecting the message or else confirm its legitimacy with the sender. -------------- next part -------------- An HTML attachment was scrubbed... URL: -------------- next part -------------- A non-text attachment was scrubbed... Name: source_sndk_05_16_2013.zip Type: application/x-zip-compressed Size: 199240 bytes Desc: source_sndk_05_16_2013.zip URL: From Alex.Chang at idt.com Thu May 16 15:43:56 2013 From: Alex.Chang at idt.com (Chang, Alex) Date: Thu, 16 May 2013 22:43:56 +0000 Subject: [nvmewin] New Patch From Sandisk In-Reply-To: <49158E750348AA499168FD41D88983606256082A@FMSMSX105.amr.corp.intel.com> References: <49158E750348AA499168FD41D88983606256082A@FMSMSX105.amr.corp.intel.com> Message-ID: <548C5470AAD9DA4A85D259B663190D361FFE961C@corpmail1.na.ads.idt.com> Hi Sumant, The only change you've made in the new patch is replacing: memset(GET_DATA_BUFFER(pSrb), 0, allocLength); with: memset(GET_DATA_BUFFER(pSrb), 0, min(pSrbExt->DataTransferLength, allocLength)); However, it generates compiling error because DataTransferLength is not a member of NVME_SRB_EXTENSION structure. Could you please fix it? Thanks a lot for you help, Alex ________________________________ From: Robles, Raymond C [mailto:raymond.c.robles at intel.com] Sent: Thursday, May 16, 2013 1:37 PM To: Sumant Patro; Knoblaugh, Rick; Chang, Alex; Dharani Kotte; nvmewin at lists.openfabrics.org Subject: RE: New Patch From Sandisk Hi Sumant/Dharani, Thanks for the updated patch. A couple of comments/questions: - For all the mode sense page translations, the new temp buffer allocated in the SRB extension is used. But for the Modes Sense 0x3C (all mode pages), the SRB data buffer is still memset to 0 at the beginning of SntiReturnAllModePages(). I’m not sure we still need to do this considering how the temp buffer is used to build the mode sense data. - If you insist on keeping that memset in there, there is a #define in for all mode pages - MODE_SENSE_ALL_PAGES_LENGTH. It may be better to zero out the buffer based on the min between this define and allocLength. SRB data transfer length cannot be relied on for “data-in” commands (like mode sense). Thanks, Ray From: Sumant Patro [mailto:Sumant.Patro at sandisk.com] Sent: Thursday, May 16, 2013 12:43 PM To: Knoblaugh, Rick; Chang, Alex; Dharani Kotte; Robles, Raymond C; nvmewin at lists.openfabrics.org Subject: [WARNING - ENCRYPTED ATTACHMENT NOT VIRUS SCANNED] RE: ニ[WARNING - ENCRYPTED ATTACHMENT NOT VIRUS SCANNED] [nvmewin] ***UNCHECKED*** [WARNING - ENCRYPTED ATTACHMENT NOT VIRUS SCANNED] RE: New Patch From Sandisk Sorry, sent the wrong zip in the previous mail. Please ignore the previous attachment. Attached is the updated zip file. Regards, Sumant From: nvmewin-bounces at lists.openfabrics.org [mailto:nvmewin-bounces at lists.openfabrics.org] On Behalf Of Sumant Patro Sent: Thursday, May 16, 2013 12:38 PM To: Knoblaugh, Rick; Chang, Alex; Dharani Kotte; Robles, Raymond C; nvmewin at lists.openfabrics.org Subject: ニ[WARNING - ENCRYPTED ATTACHMENT NOT VIRUS SCANNED] [nvmewin] ***UNCHECKED*** [WARNING - ENCRYPTED ATTACHMENT NOT VIRUS SCANNED] RE: New Patch From Sandisk Thanks Rick, Alex for your feedback. Please find attached updated files with following two changes : 1. Comment in header of SntiCreateModeDataHeader function changed to reflect passing of srb extension 2. memset(GET_DATA_BUFFER(pSrb), 0, allocLength); modified to use minimum of allocLength and DataTransferLength in SntiReturnAllModePages(..) Password : sndk1234 Regards, Sumant From: nvmewin-bounces at lists.openfabrics.org [mailto:nvmewin-bounces at lists.openfabrics.org] On Behalf Of Knoblaugh, Rick Sent: Thursday, May 16, 2013 11:59 AM To: Chang, Alex; Dharani Kotte; Robles, Raymond C; nvmewin at lists.openfabrics.org Subject: Re: [nvmewin] New Patch From Sandisk Hi Dharani, When you change that one, may as well fix the comment in header of SntiCreateModeDataHeader function to reflect that srb extension is now the parameter that’s passed in. Other stuff looks good. Thanks, -Rick From: nvmewin-bounces at lists.openfabrics.org [mailto:nvmewin-bounces at lists.openfabrics.org] On Behalf Of Chang, Alex Sent: Thursday, May 16, 2013 10:58 AM To: Dharani Kotte; Robles, Raymond C; nvmewin at lists.openfabrics.org Subject: Re: [nvmewin] New Patch From Sandisk Hi Dharani, I tested your new patch and experienced a D1 bugcheck while running SCSICompliance test. Some preliminary debugging leads to Line#3633 of nvmestni.c, where it looks like: memset(GET_DATA_BUFFER(pSrb), 0, allocLength); The bugcheck happens when the value of "allocLength" is 0x1000. Apparently, the allocated length in CDB is somewhat a bogus value that is meant to test how driver handles it. Could you please fix it? Thanks, Alex ________________________________ From: nvmewin-bounces at lists.openfabrics.org [mailto:nvmewin-bounces at lists.openfabrics.org] On Behalf Of Dharani Kotte Sent: Wednesday, May 01, 2013 8:56 AM To: Robles, Raymond C; nvmewin at lists.openfabrics.org Subject: [nvmewin] ***UNCHECKED*** [WARNING - ENCRYPTED ATTACHMENT NOT VIRUS SCANNED] New Patch From Sandisk Hi Ray, I have incorporated the changes according to your suggestions, I tried to keep up with the coding convention please feel free to let me know if anything has to be changed. I have incorporated one more fix in the nvmeInit.c which can lead to memory corruptions and used the SrbExt for the buffer allocation. PassWd: sndk1234 Thanks, Dharani. From: Dharani Kotte Sent: Tuesday, April 30, 2013 12:03 PM To: 'Robles, Raymond C'; nvmewin at lists.openfabrics.org Subject: RE: New Patch From Sandisk Thank you for the feedback, I agree that the global buffer is not good I have made the changes to allocate this buffer from the srbext I will provide the code with the below modifications soon. Thanks, Dharani. From: Robles, Raymond C [mailto:raymond.c.robles at intel.com] Sent: Tuesday, April 30, 2013 11:52 AM To: Dharani Kotte; nvmewin at lists.openfabrics.org Subject: RE: New Patch From Sandisk Hi Dharani, Thank you for putting this patch together. Please see my feedback below: - Line 52: Note that we have a file called nvmeSntiTypes.h. This file contains all #defines. As per our existing convention, all #defines should be placed in this file (versus at the top of a .c file). - Line 3339/3412/3502/3593/3756/5391/5433/5442/5453: Our coding convention states no line be longer than 80 characters. These lines all exceed 80 characters. - Line 5410: SntiTranslateAllModePagesResponse() contains new if-else blocks. Our coding convention dictates that open curly braces be placed on the same line as the if or else statement. You can refer to other instances of the code and if-else clauses as examples. - Line 4017: SntiCreateModeDataHeader() no longer needs the SRB pointer to be passed in as a parameter since you are using a global temp buffer to build up the mode sense data. That parameter can be removed. - General: The method of using a global buffer for a single mode sense command opens up a potential race condition for multiple mode sense commands submitted on top of each other. The idea behind putting the SCSI to NVMe translation phase in the HwStorBuildIo phase was to help performance. We initially knew that we would never be sharing any data structures or memory in calls to BuildIo. There is no spinlock or protection for calls to BuildIo and MSDN makes it clear that these BuildIo calls are not synchronized and that any synchronization of memory/data must be done within the function implementations or just use StartIo instead (since Storport will acquire the StartIoSpinLock). So there is hole here w/r/t the global buffer being populated for multiple mode sense commands at the same time. I believe you have the right idea, but the correct solution is to create a temp mode sense buffer inside the SRB extension. This way, each command has its own temp mode sense buffer for building the mode sense response, and then when ready to copy over to the SRB data buffer, it’s pulling from a buffer that can only be accessed by the current command (instead of a global buffer that may have been overwritten by another mode sense request call from another BuildIo call). Note that the driver collaboration group made the decision to never acquire any type of lock in BuildIo. Let me know if you have questions on this solution. The additional size to the SRB extension is not an issue. This fix is critical for this patch to be accepted. Thanks, Ray From: nvmewin-bounces at lists.openfabrics.org [mailto:nvmewin-bounces at lists.openfabrics.org] On Behalf Of Dharani Kotte Sent: Tuesday, April 30, 2013 11:14 AM To: Chang, Alex; nvmewin at lists.openfabrics.org Subject: [nvmewin] ***UNCHECKED*** [WARNING - ENCRYPTED ATTACHMENT NOT VIRUS SCANNED] New Patch From Sandisk Attaching the code with according to Alex suggestion. Passwd: sndk1234 Thanks, Dharani. From: Chang, Alex [mailto:Alex.Chang at idt.com] Sent: Tuesday, April 30, 2013 9:23 AM To: Dharani Kotte; nvmewin at lists.openfabrics.org Subject: RE: New Patch From Sandisk Hi Dharani, I'd suggest that it's safer to initialize gModeSenseBuf as all zeros each time before it is used, just in case there are some unexpected data in the buffer. Other than that, I am fine with the patch. Thanks, Alex ________________________________ From: nvmewin-bounces at lists.openfabrics.org [mailto:nvmewin-bounces at lists.openfabrics.org] On Behalf Of Dharani Kotte Sent: Tuesday, April 30, 2013 8:37 AM To: nvmewin at lists.openfabrics.org Subject: [nvmewin] New Patch From Sandisk Hi all, It's been almost two weeks after I sent out the patch. please let us know if you're okay with it. Thanks, Dharani. From: Dharani Kotte Sent: Friday, April 12, 2013 9:58 AM To: nvmewin at lists.openfabrics.org Subject: New Patch From Sandisk Hi all, I am attaching a new patch that includes the following changes: 1. In nvmeSnti.c a. Allocate a global buffer of 256bytes size for modesense return data preparation b. For any mode sense this buffer is used for modesense data along with the requested pages in it. c. Copy the required data according to the alloc_length requested in the cdb to the Srb->DataBuffer and update the dataTransferLength accordingly d. In function SntiTranslateReturnAllModePagesResponse() the modesense data header offset is calculated wrong which ends up in BSOD in some cases modified code to fix this issue Please review the changes and provide feedbacks if you have any. If nobody disagrees with the changes, I will remind Ray to merge them in two weeks. Thanks, Dharani. ________________________________ PLEASE NOTE: The information contained in this electronic mail message is intended only for the use of the designated recipient(s) named above. If the reader of this message is not the intended recipient, you are hereby notified that you have received this message in error and that any review, dissemination, distribution, or copying of this message is strictly prohibited. If you have received this communication in error, please notify the sender by telephone or e-mail (as shown above) immediately and destroy any and all copies of this message in your possession (whether hard copies or electronically stored copies). CAUTION: Please confirm that the password protected .zip attachment which contains the file(s) of type source_sndk_05_16_2013.zip is legitimate prior to opening. To make sure this message is not infected with a virus, it is important to verify that you are expecting the message or else confirm its legitimacy with the sender. -------------- next part -------------- An HTML attachment was scrubbed... URL: From Sumant.Patro at sandisk.com Thu May 16 15:46:47 2013 From: Sumant.Patro at sandisk.com (Sumant Patro) Date: Thu, 16 May 2013 22:46:47 +0000 Subject: [nvmewin] New Patch From Sandisk In-Reply-To: <548C5470AAD9DA4A85D259B663190D361FFE961C@corpmail1.na.ads.idt.com> References: <49158E750348AA499168FD41D88983606256082A@FMSMSX105.amr.corp.intel.com> <548C5470AAD9DA4A85D259B663190D361FFE961C@corpmail1.na.ads.idt.com> Message-ID: Thanks Alex. I compiled prior to sending. Let me see what happened and fix and resend. Regards, Sumant From: Chang, Alex [mailto:Alex.Chang at idt.com] Sent: Thursday, May 16, 2013 3:44 PM To: Robles, Raymond C; Sumant Patro; Knoblaugh, Rick; Dharani Kotte; nvmewin at lists.openfabrics.org Subject: RE: New Patch From Sandisk Hi Sumant, The only change you've made in the new patch is replacing: memset(GET_DATA_BUFFER(pSrb), 0, allocLength); with: memset(GET_DATA_BUFFER(pSrb), 0, min(pSrbExt->DataTransferLength, allocLength)); However, it generates compiling error because DataTransferLength is not a member of NVME_SRB_EXTENSION structure. Could you please fix it? Thanks a lot for you help, Alex ________________________________ From: Robles, Raymond C [mailto:raymond.c.robles at intel.com] Sent: Thursday, May 16, 2013 1:37 PM To: Sumant Patro; Knoblaugh, Rick; Chang, Alex; Dharani Kotte; nvmewin at lists.openfabrics.org Subject: RE: New Patch From Sandisk Hi Sumant/Dharani, Thanks for the updated patch. A couple of comments/questions: - For all the mode sense page translations, the new temp buffer allocated in the SRB extension is used. But for the Modes Sense 0x3C (all mode pages), the SRB data buffer is still memset to 0 at the beginning of SntiReturnAllModePages(). I’m not sure we still need to do this considering how the temp buffer is used to build the mode sense data. - If you insist on keeping that memset in there, there is a #define in for all mode pages - MODE_SENSE_ALL_PAGES_LENGTH. It may be better to zero out the buffer based on the min between this define and allocLength. SRB data transfer length cannot be relied on for “data-in” commands (like mode sense). Thanks, Ray From: Sumant Patro [mailto:Sumant.Patro at sandisk.com] Sent: Thursday, May 16, 2013 12:43 PM To: Knoblaugh, Rick; Chang, Alex; Dharani Kotte; Robles, Raymond C; nvmewin at lists.openfabrics.org Subject: [WARNING - ENCRYPTED ATTACHMENT NOT VIRUS SCANNED] RE: ニ[WARNING - ENCRYPTED ATTACHMENT NOT VIRUS SCANNED] [nvmewin] ***UNCHECKED*** [WARNING - ENCRYPTED ATTACHMENT NOT VIRUS SCANNED] RE: New Patch From Sandisk Sorry, sent the wrong zip in the previous mail. Please ignore the previous attachment. Attached is the updated zip file. Regards, Sumant From: nvmewin-bounces at lists.openfabrics.org [mailto:nvmewin-bounces at lists.openfabrics.org] On Behalf Of Sumant Patro Sent: Thursday, May 16, 2013 12:38 PM To: Knoblaugh, Rick; Chang, Alex; Dharani Kotte; Robles, Raymond C; nvmewin at lists.openfabrics.org Subject: ニ[WARNING - ENCRYPTED ATTACHMENT NOT VIRUS SCANNED] [nvmewin] ***UNCHECKED*** [WARNING - ENCRYPTED ATTACHMENT NOT VIRUS SCANNED] RE: New Patch From Sandisk Thanks Rick, Alex for your feedback. Please find attached updated files with following two changes : 1. Comment in header of SntiCreateModeDataHeader function changed to reflect passing of srb extension 2. memset(GET_DATA_BUFFER(pSrb), 0, allocLength); modified to use minimum of allocLength and DataTransferLength in SntiReturnAllModePages(..) Password : sndk1234 Regards, Sumant From: nvmewin-bounces at lists.openfabrics.org [mailto:nvmewin-bounces at lists.openfabrics.org] On Behalf Of Knoblaugh, Rick Sent: Thursday, May 16, 2013 11:59 AM To: Chang, Alex; Dharani Kotte; Robles, Raymond C; nvmewin at lists.openfabrics.org Subject: Re: [nvmewin] New Patch From Sandisk Hi Dharani, When you change that one, may as well fix the comment in header of SntiCreateModeDataHeader function to reflect that srb extension is now the parameter that’s passed in. Other stuff looks good. Thanks, -Rick From: nvmewin-bounces at lists.openfabrics.org [mailto:nvmewin-bounces at lists.openfabrics.org] On Behalf Of Chang, Alex Sent: Thursday, May 16, 2013 10:58 AM To: Dharani Kotte; Robles, Raymond C; nvmewin at lists.openfabrics.org Subject: Re: [nvmewin] New Patch From Sandisk Hi Dharani, I tested your new patch and experienced a D1 bugcheck while running SCSICompliance test. Some preliminary debugging leads to Line#3633 of nvmestni.c, where it looks like: memset(GET_DATA_BUFFER(pSrb), 0, allocLength); The bugcheck happens when the value of "allocLength" is 0x1000. Apparently, the allocated length in CDB is somewhat a bogus value that is meant to test how driver handles it. Could you please fix it? Thanks, Alex ________________________________ From: nvmewin-bounces at lists.openfabrics.org [mailto:nvmewin-bounces at lists.openfabrics.org] On Behalf Of Dharani Kotte Sent: Wednesday, May 01, 2013 8:56 AM To: Robles, Raymond C; nvmewin at lists.openfabrics.org Subject: [nvmewin] ***UNCHECKED*** [WARNING - ENCRYPTED ATTACHMENT NOT VIRUS SCANNED] New Patch From Sandisk Hi Ray, I have incorporated the changes according to your suggestions, I tried to keep up with the coding convention please feel free to let me know if anything has to be changed. I have incorporated one more fix in the nvmeInit.c which can lead to memory corruptions and used the SrbExt for the buffer allocation. PassWd: sndk1234 Thanks, Dharani. From: Dharani Kotte Sent: Tuesday, April 30, 2013 12:03 PM To: 'Robles, Raymond C'; nvmewin at lists.openfabrics.org Subject: RE: New Patch From Sandisk Thank you for the feedback, I agree that the global buffer is not good I have made the changes to allocate this buffer from the srbext I will provide the code with the below modifications soon. Thanks, Dharani. From: Robles, Raymond C [mailto:raymond.c.robles at intel.com] Sent: Tuesday, April 30, 2013 11:52 AM To: Dharani Kotte; nvmewin at lists.openfabrics.org Subject: RE: New Patch From Sandisk Hi Dharani, Thank you for putting this patch together. Please see my feedback below: - Line 52: Note that we have a file called nvmeSntiTypes.h. This file contains all #defines. As per our existing convention, all #defines should be placed in this file (versus at the top of a .c file). - Line 3339/3412/3502/3593/3756/5391/5433/5442/5453: Our coding convention states no line be longer than 80 characters. These lines all exceed 80 characters. - Line 5410: SntiTranslateAllModePagesResponse() contains new if-else blocks. Our coding convention dictates that open curly braces be placed on the same line as the if or else statement. You can refer to other instances of the code and if-else clauses as examples. - Line 4017: SntiCreateModeDataHeader() no longer needs the SRB pointer to be passed in as a parameter since you are using a global temp buffer to build up the mode sense data. That parameter can be removed. - General: The method of using a global buffer for a single mode sense command opens up a potential race condition for multiple mode sense commands submitted on top of each other. The idea behind putting the SCSI to NVMe translation phase in the HwStorBuildIo phase was to help performance. We initially knew that we would never be sharing any data structures or memory in calls to BuildIo. There is no spinlock or protection for calls to BuildIo and MSDN makes it clear that these BuildIo calls are not synchronized and that any synchronization of memory/data must be done within the function implementations or just use StartIo instead (since Storport will acquire the StartIoSpinLock). So there is hole here w/r/t the global buffer being populated for multiple mode sense commands at the same time. I believe you have the right idea, but the correct solution is to create a temp mode sense buffer inside the SRB extension. This way, each command has its own temp mode sense buffer for building the mode sense response, and then when ready to copy over to the SRB data buffer, it’s pulling from a buffer that can only be accessed by the current command (instead of a global buffer that may have been overwritten by another mode sense request call from another BuildIo call). Note that the driver collaboration group made the decision to never acquire any type of lock in BuildIo. Let me know if you have questions on this solution. The additional size to the SRB extension is not an issue. This fix is critical for this patch to be accepted. Thanks, Ray From: nvmewin-bounces at lists.openfabrics.org [mailto:nvmewin-bounces at lists.openfabrics.org] On Behalf Of Dharani Kotte Sent: Tuesday, April 30, 2013 11:14 AM To: Chang, Alex; nvmewin at lists.openfabrics.org Subject: [nvmewin] ***UNCHECKED*** [WARNING - ENCRYPTED ATTACHMENT NOT VIRUS SCANNED] New Patch From Sandisk Attaching the code with according to Alex suggestion. Passwd: sndk1234 Thanks, Dharani. From: Chang, Alex [mailto:Alex.Chang at idt.com] Sent: Tuesday, April 30, 2013 9:23 AM To: Dharani Kotte; nvmewin at lists.openfabrics.org Subject: RE: New Patch From Sandisk Hi Dharani, I'd suggest that it's safer to initialize gModeSenseBuf as all zeros each time before it is used, just in case there are some unexpected data in the buffer. Other than that, I am fine with the patch. Thanks, Alex ________________________________ From: nvmewin-bounces at lists.openfabrics.org [mailto:nvmewin-bounces at lists.openfabrics.org] On Behalf Of Dharani Kotte Sent: Tuesday, April 30, 2013 8:37 AM To: nvmewin at lists.openfabrics.org Subject: [nvmewin] New Patch From Sandisk Hi all, It's been almost two weeks after I sent out the patch. please let us know if you're okay with it. Thanks, Dharani. From: Dharani Kotte Sent: Friday, April 12, 2013 9:58 AM To: nvmewin at lists.openfabrics.org Subject: New Patch From Sandisk Hi all, I am attaching a new patch that includes the following changes: 1. In nvmeSnti.c a. Allocate a global buffer of 256bytes size for modesense return data preparation b. For any mode sense this buffer is used for modesense data along with the requested pages in it. c. Copy the required data according to the alloc_length requested in the cdb to the Srb->DataBuffer and update the dataTransferLength accordingly d. In function SntiTranslateReturnAllModePagesResponse() the modesense data header offset is calculated wrong which ends up in BSOD in some cases modified code to fix this issue Please review the changes and provide feedbacks if you have any. If nobody disagrees with the changes, I will remind Ray to merge them in two weeks. Thanks, Dharani. ________________________________ PLEASE NOTE: The information contained in this electronic mail message is intended only for the use of the designated recipient(s) named above. If the reader of this message is not the intended recipient, you are hereby notified that you have received this message in error and that any review, dissemination, distribution, or copying of this message is strictly prohibited. If you have received this communication in error, please notify the sender by telephone or e-mail (as shown above) immediately and destroy any and all copies of this message in your possession (whether hard copies or electronically stored copies). CAUTION: Please confirm that the password protected .zip attachment which contains the file(s) of type source_sndk_05_16_2013.zip is legitimate prior to opening. To make sure this message is not infected with a virus, it is important to verify that you are expecting the message or else confirm its legitimacy with the sender. -------------- next part -------------- An HTML attachment was scrubbed... URL: From raymond.c.robles at intel.com Thu May 16 15:49:28 2013 From: raymond.c.robles at intel.com (Robles, Raymond C) Date: Thu, 16 May 2013 22:49:28 +0000 Subject: [nvmewin] New Patch From Sandisk In-Reply-To: <548C5470AAD9DA4A85D259B663190D361FFE961C@corpmail1.na.ads.idt.com> References: <49158E750348AA499168FD41D88983606256082A@FMSMSX105.amr.corp.intel.com> <548C5470AAD9DA4A85D259B663190D361FFE961C@corpmail1.na.ads.idt.com> Message-ID: <49158E750348AA499168FD41D889836062560AA2@FMSMSX105.amr.corp.intel.com> Actually, I was going to type the same thing in my email below and then got derailed with the whole #define issue (thanks for capturing this Alex). There is no DataTransferLength field in the SRB Extension structure. That is defined in the SRB by Storport. As a general courtesy to other companies who are reviewing these patches, please insure that you are compiling and re-testing your patch prior to sending out for secondary or additional code reviews. This will help save valuable time in the code review process. If you are having issues compiling, or do not see the same errors, please contact me. Thanks, Ray From: Chang, Alex [mailto:Alex.Chang at idt.com] Sent: Thursday, May 16, 2013 3:44 PM To: Robles, Raymond C; Sumant Patro; Knoblaugh, Rick; Dharani Kotte; nvmewin at lists.openfabrics.org Subject: RE: New Patch From Sandisk Hi Sumant, The only change you've made in the new patch is replacing: memset(GET_DATA_BUFFER(pSrb), 0, allocLength); with: memset(GET_DATA_BUFFER(pSrb), 0, min(pSrbExt->DataTransferLength, allocLength)); However, it generates compiling error because DataTransferLength is not a member of NVME_SRB_EXTENSION structure. Could you please fix it? Thanks a lot for you help, Alex ________________________________ From: Robles, Raymond C [mailto:raymond.c.robles at intel.com] Sent: Thursday, May 16, 2013 1:37 PM To: Sumant Patro; Knoblaugh, Rick; Chang, Alex; Dharani Kotte; nvmewin at lists.openfabrics.org Subject: RE: New Patch From Sandisk Hi Sumant/Dharani, Thanks for the updated patch. A couple of comments/questions: - For all the mode sense page translations, the new temp buffer allocated in the SRB extension is used. But for the Modes Sense 0x3C (all mode pages), the SRB data buffer is still memset to 0 at the beginning of SntiReturnAllModePages(). I’m not sure we still need to do this considering how the temp buffer is used to build the mode sense data. - If you insist on keeping that memset in there, there is a #define in for all mode pages - MODE_SENSE_ALL_PAGES_LENGTH. It may be better to zero out the buffer based on the min between this define and allocLength. SRB data transfer length cannot be relied on for “data-in” commands (like mode sense). Thanks, Ray From: Sumant Patro [mailto:Sumant.Patro at sandisk.com] Sent: Thursday, May 16, 2013 12:43 PM To: Knoblaugh, Rick; Chang, Alex; Dharani Kotte; Robles, Raymond C; nvmewin at lists.openfabrics.org Subject: [WARNING - ENCRYPTED ATTACHMENT NOT VIRUS SCANNED] RE: ニ[WARNING - ENCRYPTED ATTACHMENT NOT VIRUS SCANNED] [nvmewin] ***UNCHECKED*** [WARNING - ENCRYPTED ATTACHMENT NOT VIRUS SCANNED] RE: New Patch From Sandisk Sorry, sent the wrong zip in the previous mail. Please ignore the previous attachment. Attached is the updated zip file. Regards, Sumant From: nvmewin-bounces at lists.openfabrics.org [mailto:nvmewin-bounces at lists.openfabrics.org] On Behalf Of Sumant Patro Sent: Thursday, May 16, 2013 12:38 PM To: Knoblaugh, Rick; Chang, Alex; Dharani Kotte; Robles, Raymond C; nvmewin at lists.openfabrics.org Subject: ニ[WARNING - ENCRYPTED ATTACHMENT NOT VIRUS SCANNED] [nvmewin] ***UNCHECKED*** [WARNING - ENCRYPTED ATTACHMENT NOT VIRUS SCANNED] RE: New Patch From Sandisk Thanks Rick, Alex for your feedback. Please find attached updated files with following two changes : 1. Comment in header of SntiCreateModeDataHeader function changed to reflect passing of srb extension 2. memset(GET_DATA_BUFFER(pSrb), 0, allocLength); modified to use minimum of allocLength and DataTransferLength in SntiReturnAllModePages(..) Password : sndk1234 Regards, Sumant From: nvmewin-bounces at lists.openfabrics.org [mailto:nvmewin-bounces at lists.openfabrics.org] On Behalf Of Knoblaugh, Rick Sent: Thursday, May 16, 2013 11:59 AM To: Chang, Alex; Dharani Kotte; Robles, Raymond C; nvmewin at lists.openfabrics.org Subject: Re: [nvmewin] New Patch From Sandisk Hi Dharani, When you change that one, may as well fix the comment in header of SntiCreateModeDataHeader function to reflect that srb extension is now the parameter that’s passed in. Other stuff looks good. Thanks, -Rick From: nvmewin-bounces at lists.openfabrics.org [mailto:nvmewin-bounces at lists.openfabrics.org] On Behalf Of Chang, Alex Sent: Thursday, May 16, 2013 10:58 AM To: Dharani Kotte; Robles, Raymond C; nvmewin at lists.openfabrics.org Subject: Re: [nvmewin] New Patch From Sandisk Hi Dharani, I tested your new patch and experienced a D1 bugcheck while running SCSICompliance test. Some preliminary debugging leads to Line#3633 of nvmestni.c, where it looks like: memset(GET_DATA_BUFFER(pSrb), 0, allocLength); The bugcheck happens when the value of "allocLength" is 0x1000. Apparently, the allocated length in CDB is somewhat a bogus value that is meant to test how driver handles it. Could you please fix it? Thanks, Alex ________________________________ From: nvmewin-bounces at lists.openfabrics.org [mailto:nvmewin-bounces at lists.openfabrics.org] On Behalf Of Dharani Kotte Sent: Wednesday, May 01, 2013 8:56 AM To: Robles, Raymond C; nvmewin at lists.openfabrics.org Subject: [nvmewin] ***UNCHECKED*** [WARNING - ENCRYPTED ATTACHMENT NOT VIRUS SCANNED] New Patch From Sandisk Hi Ray, I have incorporated the changes according to your suggestions, I tried to keep up with the coding convention please feel free to let me know if anything has to be changed. I have incorporated one more fix in the nvmeInit.c which can lead to memory corruptions and used the SrbExt for the buffer allocation. PassWd: sndk1234 Thanks, Dharani. From: Dharani Kotte Sent: Tuesday, April 30, 2013 12:03 PM To: 'Robles, Raymond C'; nvmewin at lists.openfabrics.org Subject: RE: New Patch From Sandisk Thank you for the feedback, I agree that the global buffer is not good I have made the changes to allocate this buffer from the srbext I will provide the code with the below modifications soon. Thanks, Dharani. From: Robles, Raymond C [mailto:raymond.c.robles at intel.com] Sent: Tuesday, April 30, 2013 11:52 AM To: Dharani Kotte; nvmewin at lists.openfabrics.org Subject: RE: New Patch From Sandisk Hi Dharani, Thank you for putting this patch together. Please see my feedback below: - Line 52: Note that we have a file called nvmeSntiTypes.h. This file contains all #defines. As per our existing convention, all #defines should be placed in this file (versus at the top of a .c file). - Line 3339/3412/3502/3593/3756/5391/5433/5442/5453: Our coding convention states no line be longer than 80 characters. These lines all exceed 80 characters. - Line 5410: SntiTranslateAllModePagesResponse() contains new if-else blocks. Our coding convention dictates that open curly braces be placed on the same line as the if or else statement. You can refer to other instances of the code and if-else clauses as examples. - Line 4017: SntiCreateModeDataHeader() no longer needs the SRB pointer to be passed in as a parameter since you are using a global temp buffer to build up the mode sense data. That parameter can be removed. - General: The method of using a global buffer for a single mode sense command opens up a potential race condition for multiple mode sense commands submitted on top of each other. The idea behind putting the SCSI to NVMe translation phase in the HwStorBuildIo phase was to help performance. We initially knew that we would never be sharing any data structures or memory in calls to BuildIo. There is no spinlock or protection for calls to BuildIo and MSDN makes it clear that these BuildIo calls are not synchronized and that any synchronization of memory/data must be done within the function implementations or just use StartIo instead (since Storport will acquire the StartIoSpinLock). So there is hole here w/r/t the global buffer being populated for multiple mode sense commands at the same time. I believe you have the right idea, but the correct solution is to create a temp mode sense buffer inside the SRB extension. This way, each command has its own temp mode sense buffer for building the mode sense response, and then when ready to copy over to the SRB data buffer, it’s pulling from a buffer that can only be accessed by the current command (instead of a global buffer that may have been overwritten by another mode sense request call from another BuildIo call). Note that the driver collaboration group made the decision to never acquire any type of lock in BuildIo. Let me know if you have questions on this solution. The additional size to the SRB extension is not an issue. This fix is critical for this patch to be accepted. Thanks, Ray From: nvmewin-bounces at lists.openfabrics.org [mailto:nvmewin-bounces at lists.openfabrics.org] On Behalf Of Dharani Kotte Sent: Tuesday, April 30, 2013 11:14 AM To: Chang, Alex; nvmewin at lists.openfabrics.org Subject: [nvmewin] ***UNCHECKED*** [WARNING - ENCRYPTED ATTACHMENT NOT VIRUS SCANNED] New Patch From Sandisk Attaching the code with according to Alex suggestion. Passwd: sndk1234 Thanks, Dharani. From: Chang, Alex [mailto:Alex.Chang at idt.com] Sent: Tuesday, April 30, 2013 9:23 AM To: Dharani Kotte; nvmewin at lists.openfabrics.org Subject: RE: New Patch From Sandisk Hi Dharani, I'd suggest that it's safer to initialize gModeSenseBuf as all zeros each time before it is used, just in case there are some unexpected data in the buffer. Other than that, I am fine with the patch. Thanks, Alex ________________________________ From: nvmewin-bounces at lists.openfabrics.org [mailto:nvmewin-bounces at lists.openfabrics.org] On Behalf Of Dharani Kotte Sent: Tuesday, April 30, 2013 8:37 AM To: nvmewin at lists.openfabrics.org Subject: [nvmewin] New Patch From Sandisk Hi all, It's been almost two weeks after I sent out the patch. please let us know if you're okay with it. Thanks, Dharani. From: Dharani Kotte Sent: Friday, April 12, 2013 9:58 AM To: nvmewin at lists.openfabrics.org Subject: New Patch From Sandisk Hi all, I am attaching a new patch that includes the following changes: 1. In nvmeSnti.c a. Allocate a global buffer of 256bytes size for modesense return data preparation b. For any mode sense this buffer is used for modesense data along with the requested pages in it. c. Copy the required data according to the alloc_length requested in the cdb to the Srb->DataBuffer and update the dataTransferLength accordingly d. In function SntiTranslateReturnAllModePagesResponse() the modesense data header offset is calculated wrong which ends up in BSOD in some cases modified code to fix this issue Please review the changes and provide feedbacks if you have any. If nobody disagrees with the changes, I will remind Ray to merge them in two weeks. Thanks, Dharani. ________________________________ PLEASE NOTE: The information contained in this electronic mail message is intended only for the use of the designated recipient(s) named above. If the reader of this message is not the intended recipient, you are hereby notified that you have received this message in error and that any review, dissemination, distribution, or copying of this message is strictly prohibited. If you have received this communication in error, please notify the sender by telephone or e-mail (as shown above) immediately and destroy any and all copies of this message in your possession (whether hard copies or electronically stored copies). CAUTION: Please confirm that the password protected .zip attachment which contains the file(s) of type source_sndk_05_16_2013.zip is legitimate prior to opening. To make sure this message is not infected with a virus, it is important to verify that you are expecting the message or else confirm its legitimacy with the sender. -------------- next part -------------- An HTML attachment was scrubbed... URL: From Alex.Chang at idt.com Tue May 28 11:05:28 2013 From: Alex.Chang at idt.com (Chang, Alex) Date: Tue, 28 May 2013 18:05:28 +0000 Subject: [nvmewin] OFA NVMe Windows driver Release 1.2 status In-Reply-To: <05CD7821AE397547A01AC160FBC231474BCA6EEF@corpmail1.na.ads.idt.com> References: <05CD7821AE397547A01AC160FBC231474BCA6EEF@corpmail1.na.ads.idt.com> Message-ID: <548C5470AAD9DA4A85D259B663190D361FFEBCFC@corpmail1.na.ads.idt.com> Hi all, In addition to adding changes to be compliant with NVMe 1.00e specification, I tested the followings on the supported Operating Systems : Drive format IOMeter SCSICompliance SDStress You may find the test results in the attachment. Thanks, Alex -----Original Message----- From: Kong, Kwok Sent: Thursday, May 23, 2013 1:17 PM To: Rick.Knoblaugh at lsi.com; Yong.sc.Chen at huawei.com; Chang, Alex Subject: OFA NVMe Windows driver Release 1.2 status All, I would like to get a status update on your task on the 1.2 release in June before I set up a working group meeting in June. Please let me know where you are and when you think you are ready to send out your patch for final review. Alex: - Support the following additional Windows 64-bit systems - Windows 8 - Windows Server 2008R2 - Windows Server 2012 - NVMe 1.00e enhancement Rick: - TRIM command support Yong: - Hibernation as a boot drive Thanks -Kwok -------------- next part -------------- A non-text attachment was scrubbed... Name: OFA_Test_Results.zip Type: application/x-zip-compressed Size: 226140 bytes Desc: OFA_Test_Results.zip URL: From raymond.c.robles at intel.com Thu May 30 13:23:45 2013 From: raymond.c.robles at intel.com (Robles, Raymond C) Date: Thu, 30 May 2013 20:23:45 +0000 Subject: [nvmewin] New Patch From Sandisk In-Reply-To: <49158E750348AA499168FD41D889836062560AA2@FMSMSX105.amr.corp.intel.com> References: <49158E750348AA499168FD41D88983606256082A@FMSMSX105.amr.corp.intel.com> <548C5470AAD9DA4A85D259B663190D361FFE961C@corpmail1.na.ads.idt.com> <49158E750348AA499168FD41D889836062560AA2@FMSMSX105.amr.corp.intel.com> Message-ID: <49158E750348AA499168FD41D889836062578A2D@FMSMSX105.amr.corp.intel.com> Dharani/Sumant, What is the status on this patch? I’ve not seen any follow up to my last email below. Hopefully we can close this out soon as there are other patches waiting to be submitted. Thanks for your quick attention. Thanks, Ray From: nvmewin-bounces at lists.openfabrics.org [mailto:nvmewin-bounces at lists.openfabrics.org] On Behalf Of Robles, Raymond C Sent: Thursday, May 16, 2013 3:49 PM To: Chang, Alex; Sumant Patro; Knoblaugh, Rick; Dharani Kotte; nvmewin at lists.openfabrics.org Subject: Re: [nvmewin] New Patch From Sandisk Actually, I was going to type the same thing in my email below and then got derailed with the whole #define issue (thanks for capturing this Alex). There is no DataTransferLength field in the SRB Extension structure. That is defined in the SRB by Storport. As a general courtesy to other companies who are reviewing these patches, please insure that you are compiling and re-testing your patch prior to sending out for secondary or additional code reviews. This will help save valuable time in the code review process. If you are having issues compiling, or do not see the same errors, please contact me. Thanks, Ray From: Chang, Alex [mailto:Alex.Chang at idt.com] Sent: Thursday, May 16, 2013 3:44 PM To: Robles, Raymond C; Sumant Patro; Knoblaugh, Rick; Dharani Kotte; nvmewin at lists.openfabrics.org Subject: RE: New Patch From Sandisk Hi Sumant, The only change you've made in the new patch is replacing: memset(GET_DATA_BUFFER(pSrb), 0, allocLength); with: memset(GET_DATA_BUFFER(pSrb), 0, min(pSrbExt->DataTransferLength, allocLength)); However, it generates compiling error because DataTransferLength is not a member of NVME_SRB_EXTENSION structure. Could you please fix it? Thanks a lot for you help, Alex ________________________________ From: Robles, Raymond C [mailto:raymond.c.robles at intel.com] Sent: Thursday, May 16, 2013 1:37 PM To: Sumant Patro; Knoblaugh, Rick; Chang, Alex; Dharani Kotte; nvmewin at lists.openfabrics.org Subject: RE: New Patch From Sandisk Hi Sumant/Dharani, Thanks for the updated patch. A couple of comments/questions: - For all the mode sense page translations, the new temp buffer allocated in the SRB extension is used. But for the Modes Sense 0x3C (all mode pages), the SRB data buffer is still memset to 0 at the beginning of SntiReturnAllModePages(). I’m not sure we still need to do this considering how the temp buffer is used to build the mode sense data. - If you insist on keeping that memset in there, there is a #define in for all mode pages - MODE_SENSE_ALL_PAGES_LENGTH. It may be better to zero out the buffer based on the min between this define and allocLength. SRB data transfer length cannot be relied on for “data-in” commands (like mode sense). Thanks, Ray From: Sumant Patro [mailto:Sumant.Patro at sandisk.com] Sent: Thursday, May 16, 2013 12:43 PM To: Knoblaugh, Rick; Chang, Alex; Dharani Kotte; Robles, Raymond C; nvmewin at lists.openfabrics.org Subject: [WARNING - ENCRYPTED ATTACHMENT NOT VIRUS SCANNED] RE: ニ[WARNING - ENCRYPTED ATTACHMENT NOT VIRUS SCANNED] [nvmewin] ***UNCHECKED*** [WARNING - ENCRYPTED ATTACHMENT NOT VIRUS SCANNED] RE: New Patch From Sandisk Sorry, sent the wrong zip in the previous mail. Please ignore the previous attachment. Attached is the updated zip file. Regards, Sumant From: nvmewin-bounces at lists.openfabrics.org [mailto:nvmewin-bounces at lists.openfabrics.org] On Behalf Of Sumant Patro Sent: Thursday, May 16, 2013 12:38 PM To: Knoblaugh, Rick; Chang, Alex; Dharani Kotte; Robles, Raymond C; nvmewin at lists.openfabrics.org Subject: ニ[WARNING - ENCRYPTED ATTACHMENT NOT VIRUS SCANNED] [nvmewin] ***UNCHECKED*** [WARNING - ENCRYPTED ATTACHMENT NOT VIRUS SCANNED] RE: New Patch From Sandisk Thanks Rick, Alex for your feedback. Please find attached updated files with following two changes : 1. Comment in header of SntiCreateModeDataHeader function changed to reflect passing of srb extension 2. memset(GET_DATA_BUFFER(pSrb), 0, allocLength); modified to use minimum of allocLength and DataTransferLength in SntiReturnAllModePages(..) Password : sndk1234 Regards, Sumant From: nvmewin-bounces at lists.openfabrics.org [mailto:nvmewin-bounces at lists.openfabrics.org] On Behalf Of Knoblaugh, Rick Sent: Thursday, May 16, 2013 11:59 AM To: Chang, Alex; Dharani Kotte; Robles, Raymond C; nvmewin at lists.openfabrics.org Subject: Re: [nvmewin] New Patch From Sandisk Hi Dharani, When you change that one, may as well fix the comment in header of SntiCreateModeDataHeader function to reflect that srb extension is now the parameter that’s passed in. Other stuff looks good. Thanks, -Rick From: nvmewin-bounces at lists.openfabrics.org [mailto:nvmewin-bounces at lists.openfabrics.org] On Behalf Of Chang, Alex Sent: Thursday, May 16, 2013 10:58 AM To: Dharani Kotte; Robles, Raymond C; nvmewin at lists.openfabrics.org Subject: Re: [nvmewin] New Patch From Sandisk Hi Dharani, I tested your new patch and experienced a D1 bugcheck while running SCSICompliance test. Some preliminary debugging leads to Line#3633 of nvmestni.c, where it looks like: memset(GET_DATA_BUFFER(pSrb), 0, allocLength); The bugcheck happens when the value of "allocLength" is 0x1000. Apparently, the allocated length in CDB is somewhat a bogus value that is meant to test how driver handles it. Could you please fix it? Thanks, Alex ________________________________ From: nvmewin-bounces at lists.openfabrics.org [mailto:nvmewin-bounces at lists.openfabrics.org] On Behalf Of Dharani Kotte Sent: Wednesday, May 01, 2013 8:56 AM To: Robles, Raymond C; nvmewin at lists.openfabrics.org Subject: [nvmewin] ***UNCHECKED*** [WARNING - ENCRYPTED ATTACHMENT NOT VIRUS SCANNED] New Patch From Sandisk Hi Ray, I have incorporated the changes according to your suggestions, I tried to keep up with the coding convention please feel free to let me know if anything has to be changed. I have incorporated one more fix in the nvmeInit.c which can lead to memory corruptions and used the SrbExt for the buffer allocation. PassWd: sndk1234 Thanks, Dharani. From: Dharani Kotte Sent: Tuesday, April 30, 2013 12:03 PM To: 'Robles, Raymond C'; nvmewin at lists.openfabrics.org Subject: RE: New Patch From Sandisk Thank you for the feedback, I agree that the global buffer is not good I have made the changes to allocate this buffer from the srbext I will provide the code with the below modifications soon. Thanks, Dharani. From: Robles, Raymond C [mailto:raymond.c.robles at intel.com] Sent: Tuesday, April 30, 2013 11:52 AM To: Dharani Kotte; nvmewin at lists.openfabrics.org Subject: RE: New Patch From Sandisk Hi Dharani, Thank you for putting this patch together. Please see my feedback below: - Line 52: Note that we have a file called nvmeSntiTypes.h. This file contains all #defines. As per our existing convention, all #defines should be placed in this file (versus at the top of a .c file). - Line 3339/3412/3502/3593/3756/5391/5433/5442/5453: Our coding convention states no line be longer than 80 characters. These lines all exceed 80 characters. - Line 5410: SntiTranslateAllModePagesResponse() contains new if-else blocks. Our coding convention dictates that open curly braces be placed on the same line as the if or else statement. You can refer to other instances of the code and if-else clauses as examples. - Line 4017: SntiCreateModeDataHeader() no longer needs the SRB pointer to be passed in as a parameter since you are using a global temp buffer to build up the mode sense data. That parameter can be removed. - General: The method of using a global buffer for a single mode sense command opens up a potential race condition for multiple mode sense commands submitted on top of each other. The idea behind putting the SCSI to NVMe translation phase in the HwStorBuildIo phase was to help performance. We initially knew that we would never be sharing any data structures or memory in calls to BuildIo. There is no spinlock or protection for calls to BuildIo and MSDN makes it clear that these BuildIo calls are not synchronized and that any synchronization of memory/data must be done within the function implementations or just use StartIo instead (since Storport will acquire the StartIoSpinLock). So there is hole here w/r/t the global buffer being populated for multiple mode sense commands at the same time. I believe you have the right idea, but the correct solution is to create a temp mode sense buffer inside the SRB extension. This way, each command has its own temp mode sense buffer for building the mode sense response, and then when ready to copy over to the SRB data buffer, it’s pulling from a buffer that can only be accessed by the current command (instead of a global buffer that may have been overwritten by another mode sense request call from another BuildIo call). Note that the driver collaboration group made the decision to never acquire any type of lock in BuildIo. Let me know if you have questions on this solution. The additional size to the SRB extension is not an issue. This fix is critical for this patch to be accepted. Thanks, Ray From: nvmewin-bounces at lists.openfabrics.org [mailto:nvmewin-bounces at lists.openfabrics.org] On Behalf Of Dharani Kotte Sent: Tuesday, April 30, 2013 11:14 AM To: Chang, Alex; nvmewin at lists.openfabrics.org Subject: [nvmewin] ***UNCHECKED*** [WARNING - ENCRYPTED ATTACHMENT NOT VIRUS SCANNED] New Patch From Sandisk Attaching the code with according to Alex suggestion. Passwd: sndk1234 Thanks, Dharani. From: Chang, Alex [mailto:Alex.Chang at idt.com] Sent: Tuesday, April 30, 2013 9:23 AM To: Dharani Kotte; nvmewin at lists.openfabrics.org Subject: RE: New Patch From Sandisk Hi Dharani, I'd suggest that it's safer to initialize gModeSenseBuf as all zeros each time before it is used, just in case there are some unexpected data in the buffer. Other than that, I am fine with the patch. Thanks, Alex ________________________________ From: nvmewin-bounces at lists.openfabrics.org [mailto:nvmewin-bounces at lists.openfabrics.org] On Behalf Of Dharani Kotte Sent: Tuesday, April 30, 2013 8:37 AM To: nvmewin at lists.openfabrics.org Subject: [nvmewin] New Patch From Sandisk Hi all, It's been almost two weeks after I sent out the patch. please let us know if you're okay with it. Thanks, Dharani. From: Dharani Kotte Sent: Friday, April 12, 2013 9:58 AM To: nvmewin at lists.openfabrics.org Subject: New Patch From Sandisk Hi all, I am attaching a new patch that includes the following changes: 1. In nvmeSnti.c a. Allocate a global buffer of 256bytes size for modesense return data preparation b. For any mode sense this buffer is used for modesense data along with the requested pages in it. c. Copy the required data according to the alloc_length requested in the cdb to the Srb->DataBuffer and update the dataTransferLength accordingly d. In function SntiTranslateReturnAllModePagesResponse() the modesense data header offset is calculated wrong which ends up in BSOD in some cases modified code to fix this issue Please review the changes and provide feedbacks if you have any. If nobody disagrees with the changes, I will remind Ray to merge them in two weeks. Thanks, Dharani. ________________________________ PLEASE NOTE: The information contained in this electronic mail message is intended only for the use of the designated recipient(s) named above. If the reader of this message is not the intended recipient, you are hereby notified that you have received this message in error and that any review, dissemination, distribution, or copying of this message is strictly prohibited. If you have received this communication in error, please notify the sender by telephone or e-mail (as shown above) immediately and destroy any and all copies of this message in your possession (whether hard copies or electronically stored copies). CAUTION: Please confirm that the password protected .zip attachment which contains the file(s) of type source_sndk_05_16_2013.zip is legitimate prior to opening. To make sure this message is not infected with a virus, it is important to verify that you are expecting the message or else confirm its legitimacy with the sender. -------------- next part -------------- An HTML attachment was scrubbed... URL: From Sumant.Patro at sandisk.com Thu May 30 14:11:22 2013 From: Sumant.Patro at sandisk.com (Sumant Patro) Date: Thu, 30 May 2013 21:11:22 +0000 Subject: [nvmewin] New Patch From Sandisk In-Reply-To: <49158E750348AA499168FD41D889836062578A2D@FMSMSX105.amr.corp.intel.com> References: <49158E750348AA499168FD41D88983606256082A@FMSMSX105.amr.corp.intel.com> <548C5470AAD9DA4A85D259B663190D361FFE961C@corpmail1.na.ads.idt.com> <49158E750348AA499168FD41D889836062560AA2@FMSMSX105.amr.corp.intel.com> <49158E750348AA499168FD41D889836062578A2D@FMSMSX105.amr.corp.intel.com> Message-ID: Hello Alex, I haven’t been able to repro the BSOD issue that you saw in your setup. Would it be possible for you to send me the crashdump please ? Regards, Sumant From: Robles, Raymond C [mailto:raymond.c.robles at intel.com] Sent: Thursday, May 30, 2013 1:24 PM To: Sumant Patro; Dharani Kotte; nvmewin at lists.openfabrics.org Subject: RE: New Patch From Sandisk Importance: High Dharani/Sumant, What is the status on this patch? I’ve not seen any follow up to my last email below. Hopefully we can close this out soon as there are other patches waiting to be submitted. Thanks for your quick attention. Thanks, Ray From: nvmewin-bounces at lists.openfabrics.org [mailto:nvmewin-bounces at lists.openfabrics.org] On Behalf Of Robles, Raymond C Sent: Thursday, May 16, 2013 3:49 PM To: Chang, Alex; Sumant Patro; Knoblaugh, Rick; Dharani Kotte; nvmewin at lists.openfabrics.org Subject: Re: [nvmewin] New Patch From Sandisk Actually, I was going to type the same thing in my email below and then got derailed with the whole #define issue (thanks for capturing this Alex). There is no DataTransferLength field in the SRB Extension structure. That is defined in the SRB by Storport. As a general courtesy to other companies who are reviewing these patches, please insure that you are compiling and re-testing your patch prior to sending out for secondary or additional code reviews. This will help save valuable time in the code review process. If you are having issues compiling, or do not see the same errors, please contact me. Thanks, Ray From: Chang, Alex [mailto:Alex.Chang at idt.com] Sent: Thursday, May 16, 2013 3:44 PM To: Robles, Raymond C; Sumant Patro; Knoblaugh, Rick; Dharani Kotte; nvmewin at lists.openfabrics.org Subject: RE: New Patch From Sandisk Hi Sumant, The only change you've made in the new patch is replacing: memset(GET_DATA_BUFFER(pSrb), 0, allocLength); with: memset(GET_DATA_BUFFER(pSrb), 0, min(pSrbExt->DataTransferLength, allocLength)); However, it generates compiling error because DataTransferLength is not a member of NVME_SRB_EXTENSION structure. Could you please fix it? Thanks a lot for you help, Alex ________________________________ From: Robles, Raymond C [mailto:raymond.c.robles at intel.com] Sent: Thursday, May 16, 2013 1:37 PM To: Sumant Patro; Knoblaugh, Rick; Chang, Alex; Dharani Kotte; nvmewin at lists.openfabrics.org Subject: RE: New Patch From Sandisk Hi Sumant/Dharani, Thanks for the updated patch. A couple of comments/questions: - For all the mode sense page translations, the new temp buffer allocated in the SRB extension is used. But for the Modes Sense 0x3C (all mode pages), the SRB data buffer is still memset to 0 at the beginning of SntiReturnAllModePages(). I’m not sure we still need to do this considering how the temp buffer is used to build the mode sense data. - If you insist on keeping that memset in there, there is a #define in for all mode pages - MODE_SENSE_ALL_PAGES_LENGTH. It may be better to zero out the buffer based on the min between this define and allocLength. SRB data transfer length cannot be relied on for “data-in” commands (like mode sense). Thanks, Ray From: Sumant Patro [mailto:Sumant.Patro at sandisk.com] Sent: Thursday, May 16, 2013 12:43 PM To: Knoblaugh, Rick; Chang, Alex; Dharani Kotte; Robles, Raymond C; nvmewin at lists.openfabrics.org Subject: [WARNING - ENCRYPTED ATTACHMENT NOT VIRUS SCANNED] RE: ニ[WARNING - ENCRYPTED ATTACHMENT NOT VIRUS SCANNED] [nvmewin] ***UNCHECKED*** [WARNING - ENCRYPTED ATTACHMENT NOT VIRUS SCANNED] RE: New Patch From Sandisk Sorry, sent the wrong zip in the previous mail. Please ignore the previous attachment. Attached is the updated zip file. Regards, Sumant From: nvmewin-bounces at lists.openfabrics.org [mailto:nvmewin-bounces at lists.openfabrics.org] On Behalf Of Sumant Patro Sent: Thursday, May 16, 2013 12:38 PM To: Knoblaugh, Rick; Chang, Alex; Dharani Kotte; Robles, Raymond C; nvmewin at lists.openfabrics.org Subject: ニ[WARNING - ENCRYPTED ATTACHMENT NOT VIRUS SCANNED] [nvmewin] ***UNCHECKED*** [WARNING - ENCRYPTED ATTACHMENT NOT VIRUS SCANNED] RE: New Patch From Sandisk Thanks Rick, Alex for your feedback. Please find attached updated files with following two changes : 1. Comment in header of SntiCreateModeDataHeader function changed to reflect passing of srb extension 2. memset(GET_DATA_BUFFER(pSrb), 0, allocLength); modified to use minimum of allocLength and DataTransferLength in SntiReturnAllModePages(..) Password : sndk1234 Regards, Sumant From: nvmewin-bounces at lists.openfabrics.org [mailto:nvmewin-bounces at lists.openfabrics.org] On Behalf Of Knoblaugh, Rick Sent: Thursday, May 16, 2013 11:59 AM To: Chang, Alex; Dharani Kotte; Robles, Raymond C; nvmewin at lists.openfabrics.org Subject: Re: [nvmewin] New Patch From Sandisk Hi Dharani, When you change that one, may as well fix the comment in header of SntiCreateModeDataHeader function to reflect that srb extension is now the parameter that’s passed in. Other stuff looks good. Thanks, -Rick From: nvmewin-bounces at lists.openfabrics.org [mailto:nvmewin-bounces at lists.openfabrics.org] On Behalf Of Chang, Alex Sent: Thursday, May 16, 2013 10:58 AM To: Dharani Kotte; Robles, Raymond C; nvmewin at lists.openfabrics.org Subject: Re: [nvmewin] New Patch From Sandisk Hi Dharani, I tested your new patch and experienced a D1 bugcheck while running SCSICompliance test. Some preliminary debugging leads to Line#3633 of nvmestni.c, where it looks like: memset(GET_DATA_BUFFER(pSrb), 0, allocLength); The bugcheck happens when the value of "allocLength" is 0x1000. Apparently, the allocated length in CDB is somewhat a bogus value that is meant to test how driver handles it. Could you please fix it? Thanks, Alex ________________________________ From: nvmewin-bounces at lists.openfabrics.org [mailto:nvmewin-bounces at lists.openfabrics.org] On Behalf Of Dharani Kotte Sent: Wednesday, May 01, 2013 8:56 AM To: Robles, Raymond C; nvmewin at lists.openfabrics.org Subject: [nvmewin] ***UNCHECKED*** [WARNING - ENCRYPTED ATTACHMENT NOT VIRUS SCANNED] New Patch From Sandisk Hi Ray, I have incorporated the changes according to your suggestions, I tried to keep up with the coding convention please feel free to let me know if anything has to be changed. I have incorporated one more fix in the nvmeInit.c which can lead to memory corruptions and used the SrbExt for the buffer allocation. PassWd: sndk1234 Thanks, Dharani. From: Dharani Kotte Sent: Tuesday, April 30, 2013 12:03 PM To: 'Robles, Raymond C'; nvmewin at lists.openfabrics.org Subject: RE: New Patch From Sandisk Thank you for the feedback, I agree that the global buffer is not good I have made the changes to allocate this buffer from the srbext I will provide the code with the below modifications soon. Thanks, Dharani. From: Robles, Raymond C [mailto:raymond.c.robles at intel.com] Sent: Tuesday, April 30, 2013 11:52 AM To: Dharani Kotte; nvmewin at lists.openfabrics.org Subject: RE: New Patch From Sandisk Hi Dharani, Thank you for putting this patch together. Please see my feedback below: - Line 52: Note that we have a file called nvmeSntiTypes.h. This file contains all #defines. As per our existing convention, all #defines should be placed in this file (versus at the top of a .c file). - Line 3339/3412/3502/3593/3756/5391/5433/5442/5453: Our coding convention states no line be longer than 80 characters. These lines all exceed 80 characters. - Line 5410: SntiTranslateAllModePagesResponse() contains new if-else blocks. Our coding convention dictates that open curly braces be placed on the same line as the if or else statement. You can refer to other instances of the code and if-else clauses as examples. - Line 4017: SntiCreateModeDataHeader() no longer needs the SRB pointer to be passed in as a parameter since you are using a global temp buffer to build up the mode sense data. That parameter can be removed. - General: The method of using a global buffer for a single mode sense command opens up a potential race condition for multiple mode sense commands submitted on top of each other. The idea behind putting the SCSI to NVMe translation phase in the HwStorBuildIo phase was to help performance. We initially knew that we would never be sharing any data structures or memory in calls to BuildIo. There is no spinlock or protection for calls to BuildIo and MSDN makes it clear that these BuildIo calls are not synchronized and that any synchronization of memory/data must be done within the function implementations or just use StartIo instead (since Storport will acquire the StartIoSpinLock). So there is hole here w/r/t the global buffer being populated for multiple mode sense commands at the same time. I believe you have the right idea, but the correct solution is to create a temp mode sense buffer inside the SRB extension. This way, each command has its own temp mode sense buffer for building the mode sense response, and then when ready to copy over to the SRB data buffer, it’s pulling from a buffer that can only be accessed by the current command (instead of a global buffer that may have been overwritten by another mode sense request call from another BuildIo call). Note that the driver collaboration group made the decision to never acquire any type of lock in BuildIo. Let me know if you have questions on this solution. The additional size to the SRB extension is not an issue. This fix is critical for this patch to be accepted. Thanks, Ray From: nvmewin-bounces at lists.openfabrics.org [mailto:nvmewin-bounces at lists.openfabrics.org] On Behalf Of Dharani Kotte Sent: Tuesday, April 30, 2013 11:14 AM To: Chang, Alex; nvmewin at lists.openfabrics.org Subject: [nvmewin] ***UNCHECKED*** [WARNING - ENCRYPTED ATTACHMENT NOT VIRUS SCANNED] New Patch From Sandisk Attaching the code with according to Alex suggestion. Passwd: sndk1234 Thanks, Dharani. From: Chang, Alex [mailto:Alex.Chang at idt.com] Sent: Tuesday, April 30, 2013 9:23 AM To: Dharani Kotte; nvmewin at lists.openfabrics.org Subject: RE: New Patch From Sandisk Hi Dharani, I'd suggest that it's safer to initialize gModeSenseBuf as all zeros each time before it is used, just in case there are some unexpected data in the buffer. Other than that, I am fine with the patch. Thanks, Alex ________________________________ From: nvmewin-bounces at lists.openfabrics.org [mailto:nvmewin-bounces at lists.openfabrics.org] On Behalf Of Dharani Kotte Sent: Tuesday, April 30, 2013 8:37 AM To: nvmewin at lists.openfabrics.org Subject: [nvmewin] New Patch From Sandisk Hi all, It's been almost two weeks after I sent out the patch. please let us know if you're okay with it. Thanks, Dharani. From: Dharani Kotte Sent: Friday, April 12, 2013 9:58 AM To: nvmewin at lists.openfabrics.org Subject: New Patch From Sandisk Hi all, I am attaching a new patch that includes the following changes: 1. In nvmeSnti.c a. Allocate a global buffer of 256bytes size for modesense return data preparation b. For any mode sense this buffer is used for modesense data along with the requested pages in it. c. Copy the required data according to the alloc_length requested in the cdb to the Srb->DataBuffer and update the dataTransferLength accordingly d. In function SntiTranslateReturnAllModePagesResponse() the modesense data header offset is calculated wrong which ends up in BSOD in some cases modified code to fix this issue Please review the changes and provide feedbacks if you have any. If nobody disagrees with the changes, I will remind Ray to merge them in two weeks. Thanks, Dharani. ________________________________ PLEASE NOTE: The information contained in this electronic mail message is intended only for the use of the designated recipient(s) named above. If the reader of this message is not the intended recipient, you are hereby notified that you have received this message in error and that any review, dissemination, distribution, or copying of this message is strictly prohibited. If you have received this communication in error, please notify the sender by telephone or e-mail (as shown above) immediately and destroy any and all copies of this message in your possession (whether hard copies or electronically stored copies). CAUTION: Please confirm that the password protected .zip attachment which contains the file(s) of type source_sndk_05_16_2013.zip is legitimate prior to opening. To make sure this message is not infected with a virus, it is important to verify that you are expecting the message or else confirm its legitimacy with the sender. -------------- next part -------------- An HTML attachment was scrubbed... URL: From Alex.Chang at idt.com Thu May 30 15:07:06 2013 From: Alex.Chang at idt.com (Chang, Alex) Date: Thu, 30 May 2013 22:07:06 +0000 Subject: [nvmewin] New Patch From Sandisk In-Reply-To: References: <49158E750348AA499168FD41D88983606256082A@FMSMSX105.amr.corp.intel.com> <548C5470AAD9DA4A85D259B663190D361FFE961C@corpmail1.na.ads.idt.com> <49158E750348AA499168FD41D889836062560AA2@FMSMSX105.amr.corp.intel.com> <49158E750348AA499168FD41D889836062578A2D@FMSMSX105.amr.corp.intel.com> Message-ID: <548C5470AAD9DA4A85D259B663190D361FFEC067@corpmail1.na.ads.idt.com> Hi Sumant, Actually I did some preliminary debugging on it and found out the crash was not introduced by your patch. When the system gets normal shutdown, NVMeNormalShutdown calls NVMeFreeBuffers to free up all queue related structures. And later, when NVMeDetectPendingCmds is called to check on the pending commands, it refers to the queue structures and causes the crash. I will include the fix in my next patch that includes changes to be compliant with 1.0e NVMe Specification. Thanks, Alex ________________________________ From: nvmewin-bounces at lists.openfabrics.org [mailto:nvmewin-bounces at lists.openfabrics.org] On Behalf Of Sumant Patro Sent: Thursday, May 30, 2013 2:11 PM To: Robles, Raymond C; Dharani Kotte; nvmewin at lists.openfabrics.org Subject: Re: [nvmewin] New Patch From Sandisk Hello Alex, I haven’t been able to repro the BSOD issue that you saw in your setup. Would it be possible for you to send me the crashdump please ? Regards, Sumant From: Robles, Raymond C [mailto:raymond.c.robles at intel.com] Sent: Thursday, May 30, 2013 1:24 PM To: Sumant Patro; Dharani Kotte; nvmewin at lists.openfabrics.org Subject: RE: New Patch From Sandisk Importance: High Dharani/Sumant, What is the status on this patch? I’ve not seen any follow up to my last email below. Hopefully we can close this out soon as there are other patches waiting to be submitted. Thanks for your quick attention. Thanks, Ray From: nvmewin-bounces at lists.openfabrics.org [mailto:nvmewin-bounces at lists.openfabrics.org] On Behalf Of Robles, Raymond C Sent: Thursday, May 16, 2013 3:49 PM To: Chang, Alex; Sumant Patro; Knoblaugh, Rick; Dharani Kotte; nvmewin at lists.openfabrics.org Subject: Re: [nvmewin] New Patch From Sandisk Actually, I was going to type the same thing in my email below and then got derailed with the whole #define issue (thanks for capturing this Alex). There is no DataTransferLength field in the SRB Extension structure. That is defined in the SRB by Storport. As a general courtesy to other companies who are reviewing these patches, please insure that you are compiling and re-testing your patch prior to sending out for secondary or additional code reviews. This will help save valuable time in the code review process. If you are having issues compiling, or do not see the same errors, please contact me. Thanks, Ray From: Chang, Alex [mailto:Alex.Chang at idt.com] Sent: Thursday, May 16, 2013 3:44 PM To: Robles, Raymond C; Sumant Patro; Knoblaugh, Rick; Dharani Kotte; nvmewin at lists.openfabrics.org Subject: RE: New Patch From Sandisk Hi Sumant, The only change you've made in the new patch is replacing: memset(GET_DATA_BUFFER(pSrb), 0, allocLength); with: memset(GET_DATA_BUFFER(pSrb), 0, min(pSrbExt->DataTransferLength, allocLength)); However, it generates compiling error because DataTransferLength is not a member of NVME_SRB_EXTENSION structure. Could you please fix it? Thanks a lot for you help, Alex ________________________________ From: Robles, Raymond C [mailto:raymond.c.robles at intel.com] Sent: Thursday, May 16, 2013 1:37 PM To: Sumant Patro; Knoblaugh, Rick; Chang, Alex; Dharani Kotte; nvmewin at lists.openfabrics.org Subject: RE: New Patch From Sandisk Hi Sumant/Dharani, Thanks for the updated patch. A couple of comments/questions: - For all the mode sense page translations, the new temp buffer allocated in the SRB extension is used. But for the Modes Sense 0x3C (all mode pages), the SRB data buffer is still memset to 0 at the beginning of SntiReturnAllModePages(). I’m not sure we still need to do this considering how the temp buffer is used to build the mode sense data. - If you insist on keeping that memset in there, there is a #define in for all mode pages - MODE_SENSE_ALL_PAGES_LENGTH. It may be better to zero out the buffer based on the min between this define and allocLength. SRB data transfer length cannot be relied on for “data-in” commands (like mode sense). Thanks, Ray From: Sumant Patro [mailto:Sumant.Patro at sandisk.com] Sent: Thursday, May 16, 2013 12:43 PM To: Knoblaugh, Rick; Chang, Alex; Dharani Kotte; Robles, Raymond C; nvmewin at lists.openfabrics.org Subject: [WARNING - ENCRYPTED ATTACHMENT NOT VIRUS SCANNED] RE: ニ[WARNING - ENCRYPTED ATTACHMENT NOT VIRUS SCANNED] [nvmewin] ***UNCHECKED*** [WARNING - ENCRYPTED ATTACHMENT NOT VIRUS SCANNED] RE: New Patch From Sandisk Sorry, sent the wrong zip in the previous mail. Please ignore the previous attachment. Attached is the updated zip file. Regards, Sumant From: nvmewin-bounces at lists.openfabrics.org [mailto:nvmewin-bounces at lists.openfabrics.org] On Behalf Of Sumant Patro Sent: Thursday, May 16, 2013 12:38 PM To: Knoblaugh, Rick; Chang, Alex; Dharani Kotte; Robles, Raymond C; nvmewin at lists.openfabrics.org Subject: ニ[WARNING - ENCRYPTED ATTACHMENT NOT VIRUS SCANNED] [nvmewin] ***UNCHECKED*** [WARNING - ENCRYPTED ATTACHMENT NOT VIRUS SCANNED] RE: New Patch From Sandisk Thanks Rick, Alex for your feedback. Please find attached updated files with following two changes : 1. Comment in header of SntiCreateModeDataHeader function changed to reflect passing of srb extension 2. memset(GET_DATA_BUFFER(pSrb), 0, allocLength); modified to use minimum of allocLength and DataTransferLength in SntiReturnAllModePages(..) Password : sndk1234 Regards, Sumant From: nvmewin-bounces at lists.openfabrics.org [mailto:nvmewin-bounces at lists.openfabrics.org] On Behalf Of Knoblaugh, Rick Sent: Thursday, May 16, 2013 11:59 AM To: Chang, Alex; Dharani Kotte; Robles, Raymond C; nvmewin at lists.openfabrics.org Subject: Re: [nvmewin] New Patch From Sandisk Hi Dharani, When you change that one, may as well fix the comment in header of SntiCreateModeDataHeader function to reflect that srb extension is now the parameter that’s passed in. Other stuff looks good. Thanks, -Rick From: nvmewin-bounces at lists.openfabrics.org [mailto:nvmewin-bounces at lists.openfabrics.org] On Behalf Of Chang, Alex Sent: Thursday, May 16, 2013 10:58 AM To: Dharani Kotte; Robles, Raymond C; nvmewin at lists.openfabrics.org Subject: Re: [nvmewin] New Patch From Sandisk Hi Dharani, I tested your new patch and experienced a D1 bugcheck while running SCSICompliance test. Some preliminary debugging leads to Line#3633 of nvmestni.c, where it looks like: memset(GET_DATA_BUFFER(pSrb), 0, allocLength); The bugcheck happens when the value of "allocLength" is 0x1000. Apparently, the allocated length in CDB is somewhat a bogus value that is meant to test how driver handles it. Could you please fix it? Thanks, Alex ________________________________ From: nvmewin-bounces at lists.openfabrics.org [mailto:nvmewin-bounces at lists.openfabrics.org] On Behalf Of Dharani Kotte Sent: Wednesday, May 01, 2013 8:56 AM To: Robles, Raymond C; nvmewin at lists.openfabrics.org Subject: [nvmewin] ***UNCHECKED*** [WARNING - ENCRYPTED ATTACHMENT NOT VIRUS SCANNED] New Patch From Sandisk Hi Ray, I have incorporated the changes according to your suggestions, I tried to keep up with the coding convention please feel free to let me know if anything has to be changed. I have incorporated one more fix in the nvmeInit.c which can lead to memory corruptions and used the SrbExt for the buffer allocation. PassWd: sndk1234 Thanks, Dharani. From: Dharani Kotte Sent: Tuesday, April 30, 2013 12:03 PM To: 'Robles, Raymond C'; nvmewin at lists.openfabrics.org Subject: RE: New Patch From Sandisk Thank you for the feedback, I agree that the global buffer is not good I have made the changes to allocate this buffer from the srbext I will provide the code with the below modifications soon. Thanks, Dharani. From: Robles, Raymond C [mailto:raymond.c.robles at intel.com] Sent: Tuesday, April 30, 2013 11:52 AM To: Dharani Kotte; nvmewin at lists.openfabrics.org Subject: RE: New Patch From Sandisk Hi Dharani, Thank you for putting this patch together. Please see my feedback below: - Line 52: Note that we have a file called nvmeSntiTypes.h. This file contains all #defines. As per our existing convention, all #defines should be placed in this file (versus at the top of a .c file). - Line 3339/3412/3502/3593/3756/5391/5433/5442/5453: Our coding convention states no line be longer than 80 characters. These lines all exceed 80 characters. - Line 5410: SntiTranslateAllModePagesResponse() contains new if-else blocks. Our coding convention dictates that open curly braces be placed on the same line as the if or else statement. You can refer to other instances of the code and if-else clauses as examples. - Line 4017: SntiCreateModeDataHeader() no longer needs the SRB pointer to be passed in as a parameter since you are using a global temp buffer to build up the mode sense data. That parameter can be removed. - General: The method of using a global buffer for a single mode sense command opens up a potential race condition for multiple mode sense commands submitted on top of each other. The idea behind putting the SCSI to NVMe translation phase in the HwStorBuildIo phase was to help performance. We initially knew that we would never be sharing any data structures or memory in calls to BuildIo. There is no spinlock or protection for calls to BuildIo and MSDN makes it clear that these BuildIo calls are not synchronized and that any synchronization of memory/data must be done within the function implementations or just use StartIo instead (since Storport will acquire the StartIoSpinLock). So there is hole here w/r/t the global buffer being populated for multiple mode sense commands at the same time. I believe you have the right idea, but the correct solution is to create a temp mode sense buffer inside the SRB extension. This way, each command has its own temp mode sense buffer for building the mode sense response, and then when ready to copy over to the SRB data buffer, it’s pulling from a buffer that can only be accessed by the current command (instead of a global buffer that may have been overwritten by another mode sense request call from another BuildIo call). Note that the driver collaboration group made the decision to never acquire any type of lock in BuildIo. Let me know if you have questions on this solution. The additional size to the SRB extension is not an issue. This fix is critical for this patch to be accepted. Thanks, Ray From: nvmewin-bounces at lists.openfabrics.org [mailto:nvmewin-bounces at lists.openfabrics.org] On Behalf Of Dharani Kotte Sent: Tuesday, April 30, 2013 11:14 AM To: Chang, Alex; nvmewin at lists.openfabrics.org Subject: [nvmewin] ***UNCHECKED*** [WARNING - ENCRYPTED ATTACHMENT NOT VIRUS SCANNED] New Patch From Sandisk Attaching the code with according to Alex suggestion. Passwd: sndk1234 Thanks, Dharani. From: Chang, Alex [mailto:Alex.Chang at idt.com] Sent: Tuesday, April 30, 2013 9:23 AM To: Dharani Kotte; nvmewin at lists.openfabrics.org Subject: RE: New Patch From Sandisk Hi Dharani, I'd suggest that it's safer to initialize gModeSenseBuf as all zeros each time before it is used, just in case there are some unexpected data in the buffer. Other than that, I am fine with the patch. Thanks, Alex ________________________________ From: nvmewin-bounces at lists.openfabrics.org [mailto:nvmewin-bounces at lists.openfabrics.org] On Behalf Of Dharani Kotte Sent: Tuesday, April 30, 2013 8:37 AM To: nvmewin at lists.openfabrics.org Subject: [nvmewin] New Patch From Sandisk Hi all, It's been almost two weeks after I sent out the patch. please let us know if you're okay with it. Thanks, Dharani. From: Dharani Kotte Sent: Friday, April 12, 2013 9:58 AM To: nvmewin at lists.openfabrics.org Subject: New Patch From Sandisk Hi all, I am attaching a new patch that includes the following changes: 1. In nvmeSnti.c a. Allocate a global buffer of 256bytes size for modesense return data preparation b. For any mode sense this buffer is used for modesense data along with the requested pages in it. c. Copy the required data according to the alloc_length requested in the cdb to the Srb->DataBuffer and update the dataTransferLength accordingly d. In function SntiTranslateReturnAllModePagesResponse() the modesense data header offset is calculated wrong which ends up in BSOD in some cases modified code to fix this issue Please review the changes and provide feedbacks if you have any. If nobody disagrees with the changes, I will remind Ray to merge them in two weeks. Thanks, Dharani. ________________________________ PLEASE NOTE: The information contained in this electronic mail message is intended only for the use of the designated recipient(s) named above. If the reader of this message is not the intended recipient, you are hereby notified that you have received this message in error and that any review, dissemination, distribution, or copying of this message is strictly prohibited. If you have received this communication in error, please notify the sender by telephone or e-mail (as shown above) immediately and destroy any and all copies of this message in your possession (whether hard copies or electronically stored copies). CAUTION: Please confirm that the password protected .zip attachment which contains the file(s) of type source_sndk_05_16_2013.zip is legitimate prior to opening. To make sure this message is not infected with a virus, it is important to verify that you are expecting the message or else confirm its legitimacy with the sender. -------------- next part -------------- An HTML attachment was scrubbed... URL: From raymond.c.robles at intel.com Thu May 30 15:10:22 2013 From: raymond.c.robles at intel.com (Robles, Raymond C) Date: Thu, 30 May 2013 22:10:22 +0000 Subject: [nvmewin] New Patch From Sandisk In-Reply-To: <548C5470AAD9DA4A85D259B663190D361FFEC067@corpmail1.na.ads.idt.com> References: <49158E750348AA499168FD41D88983606256082A@FMSMSX105.amr.corp.intel.com> <548C5470AAD9DA4A85D259B663190D361FFE961C@corpmail1.na.ads.idt.com> <49158E750348AA499168FD41D889836062560AA2@FMSMSX105.amr.corp.intel.com> <49158E750348AA499168FD41D889836062578A2D@FMSMSX105.amr.corp.intel.com> <548C5470AAD9DA4A85D259B663190D361FFEC067@corpmail1.na.ads.idt.com> Message-ID: <49158E750348AA499168FD41D889836062578B98@FMSMSX105.amr.corp.intel.com> Thanks Alex. Sumant, please continue with your updates from the last set of code review comments and resend the patch (after running all the required unit tests of course). Once we confirm the feedback changes are in place, I’ll push the patch. Thanks, Ray From: Chang, Alex [mailto:Alex.Chang at idt.com] Sent: Thursday, May 30, 2013 3:07 PM To: Sumant Patro; Robles, Raymond C; Dharani Kotte; nvmewin at lists.openfabrics.org Subject: RE: New Patch From Sandisk Hi Sumant, Actually I did some preliminary debugging on it and found out the crash was not introduced by your patch. When the system gets normal shutdown, NVMeNormalShutdown calls NVMeFreeBuffers to free up all queue related structures. And later, when NVMeDetectPendingCmds is called to check on the pending commands, it refers to the queue structures and causes the crash. I will include the fix in my next patch that includes changes to be compliant with 1.0e NVMe Specification. Thanks, Alex ________________________________ From: nvmewin-bounces at lists.openfabrics.org [mailto:nvmewin-bounces at lists.openfabrics.org] On Behalf Of Sumant Patro Sent: Thursday, May 30, 2013 2:11 PM To: Robles, Raymond C; Dharani Kotte; nvmewin at lists.openfabrics.org Subject: Re: [nvmewin] New Patch From Sandisk Hello Alex, I haven’t been able to repro the BSOD issue that you saw in your setup. Would it be possible for you to send me the crashdump please ? Regards, Sumant From: Robles, Raymond C [mailto:raymond.c.robles at intel.com] Sent: Thursday, May 30, 2013 1:24 PM To: Sumant Patro; Dharani Kotte; nvmewin at lists.openfabrics.org Subject: RE: New Patch From Sandisk Importance: High Dharani/Sumant, What is the status on this patch? I’ve not seen any follow up to my last email below. Hopefully we can close this out soon as there are other patches waiting to be submitted. Thanks for your quick attention. Thanks, Ray From: nvmewin-bounces at lists.openfabrics.org [mailto:nvmewin-bounces at lists.openfabrics.org] On Behalf Of Robles, Raymond C Sent: Thursday, May 16, 2013 3:49 PM To: Chang, Alex; Sumant Patro; Knoblaugh, Rick; Dharani Kotte; nvmewin at lists.openfabrics.org Subject: Re: [nvmewin] New Patch From Sandisk Actually, I was going to type the same thing in my email below and then got derailed with the whole #define issue (thanks for capturing this Alex). There is no DataTransferLength field in the SRB Extension structure. That is defined in the SRB by Storport. As a general courtesy to other companies who are reviewing these patches, please insure that you are compiling and re-testing your patch prior to sending out for secondary or additional code reviews. This will help save valuable time in the code review process. If you are having issues compiling, or do not see the same errors, please contact me. Thanks, Ray From: Chang, Alex [mailto:Alex.Chang at idt.com] Sent: Thursday, May 16, 2013 3:44 PM To: Robles, Raymond C; Sumant Patro; Knoblaugh, Rick; Dharani Kotte; nvmewin at lists.openfabrics.org Subject: RE: New Patch From Sandisk Hi Sumant, The only change you've made in the new patch is replacing: memset(GET_DATA_BUFFER(pSrb), 0, allocLength); with: memset(GET_DATA_BUFFER(pSrb), 0, min(pSrbExt->DataTransferLength, allocLength)); However, it generates compiling error because DataTransferLength is not a member of NVME_SRB_EXTENSION structure. Could you please fix it? Thanks a lot for you help, Alex ________________________________ From: Robles, Raymond C [mailto:raymond.c.robles at intel.com] Sent: Thursday, May 16, 2013 1:37 PM To: Sumant Patro; Knoblaugh, Rick; Chang, Alex; Dharani Kotte; nvmewin at lists.openfabrics.org Subject: RE: New Patch From Sandisk Hi Sumant/Dharani, Thanks for the updated patch. A couple of comments/questions: - For all the mode sense page translations, the new temp buffer allocated in the SRB extension is used. But for the Modes Sense 0x3C (all mode pages), the SRB data buffer is still memset to 0 at the beginning of SntiReturnAllModePages(). I’m not sure we still need to do this considering how the temp buffer is used to build the mode sense data. - If you insist on keeping that memset in there, there is a #define in for all mode pages - MODE_SENSE_ALL_PAGES_LENGTH. It may be better to zero out the buffer based on the min between this define and allocLength. SRB data transfer length cannot be relied on for “data-in” commands (like mode sense). Thanks, Ray From: Sumant Patro [mailto:Sumant.Patro at sandisk.com] Sent: Thursday, May 16, 2013 12:43 PM To: Knoblaugh, Rick; Chang, Alex; Dharani Kotte; Robles, Raymond C; nvmewin at lists.openfabrics.org Subject: [WARNING - ENCRYPTED ATTACHMENT NOT VIRUS SCANNED] RE: ニ[WARNING - ENCRYPTED ATTACHMENT NOT VIRUS SCANNED] [nvmewin] ***UNCHECKED*** [WARNING - ENCRYPTED ATTACHMENT NOT VIRUS SCANNED] RE: New Patch From Sandisk Sorry, sent the wrong zip in the previous mail. Please ignore the previous attachment. Attached is the updated zip file. Regards, Sumant From: nvmewin-bounces at lists.openfabrics.org [mailto:nvmewin-bounces at lists.openfabrics.org] On Behalf Of Sumant Patro Sent: Thursday, May 16, 2013 12:38 PM To: Knoblaugh, Rick; Chang, Alex; Dharani Kotte; Robles, Raymond C; nvmewin at lists.openfabrics.org Subject: ニ[WARNING - ENCRYPTED ATTACHMENT NOT VIRUS SCANNED] [nvmewin] ***UNCHECKED*** [WARNING - ENCRYPTED ATTACHMENT NOT VIRUS SCANNED] RE: New Patch From Sandisk Thanks Rick, Alex for your feedback. Please find attached updated files with following two changes : 1. Comment in header of SntiCreateModeDataHeader function changed to reflect passing of srb extension 2. memset(GET_DATA_BUFFER(pSrb), 0, allocLength); modified to use minimum of allocLength and DataTransferLength in SntiReturnAllModePages(..) Password : sndk1234 Regards, Sumant From: nvmewin-bounces at lists.openfabrics.org [mailto:nvmewin-bounces at lists.openfabrics.org] On Behalf Of Knoblaugh, Rick Sent: Thursday, May 16, 2013 11:59 AM To: Chang, Alex; Dharani Kotte; Robles, Raymond C; nvmewin at lists.openfabrics.org Subject: Re: [nvmewin] New Patch From Sandisk Hi Dharani, When you change that one, may as well fix the comment in header of SntiCreateModeDataHeader function to reflect that srb extension is now the parameter that’s passed in. Other stuff looks good. Thanks, -Rick From: nvmewin-bounces at lists.openfabrics.org [mailto:nvmewin-bounces at lists.openfabrics.org] On Behalf Of Chang, Alex Sent: Thursday, May 16, 2013 10:58 AM To: Dharani Kotte; Robles, Raymond C; nvmewin at lists.openfabrics.org Subject: Re: [nvmewin] New Patch From Sandisk Hi Dharani, I tested your new patch and experienced a D1 bugcheck while running SCSICompliance test. Some preliminary debugging leads to Line#3633 of nvmestni.c, where it looks like: memset(GET_DATA_BUFFER(pSrb), 0, allocLength); The bugcheck happens when the value of "allocLength" is 0x1000. Apparently, the allocated length in CDB is somewhat a bogus value that is meant to test how driver handles it. Could you please fix it? Thanks, Alex ________________________________ From: nvmewin-bounces at lists.openfabrics.org [mailto:nvmewin-bounces at lists.openfabrics.org] On Behalf Of Dharani Kotte Sent: Wednesday, May 01, 2013 8:56 AM To: Robles, Raymond C; nvmewin at lists.openfabrics.org Subject: [nvmewin] ***UNCHECKED*** [WARNING - ENCRYPTED ATTACHMENT NOT VIRUS SCANNED] New Patch From Sandisk Hi Ray, I have incorporated the changes according to your suggestions, I tried to keep up with the coding convention please feel free to let me know if anything has to be changed. I have incorporated one more fix in the nvmeInit.c which can lead to memory corruptions and used the SrbExt for the buffer allocation. PassWd: sndk1234 Thanks, Dharani. From: Dharani Kotte Sent: Tuesday, April 30, 2013 12:03 PM To: 'Robles, Raymond C'; nvmewin at lists.openfabrics.org Subject: RE: New Patch From Sandisk Thank you for the feedback, I agree that the global buffer is not good I have made the changes to allocate this buffer from the srbext I will provide the code with the below modifications soon. Thanks, Dharani. From: Robles, Raymond C [mailto:raymond.c.robles at intel.com] Sent: Tuesday, April 30, 2013 11:52 AM To: Dharani Kotte; nvmewin at lists.openfabrics.org Subject: RE: New Patch From Sandisk Hi Dharani, Thank you for putting this patch together. Please see my feedback below: - Line 52: Note that we have a file called nvmeSntiTypes.h. This file contains all #defines. As per our existing convention, all #defines should be placed in this file (versus at the top of a .c file). - Line 3339/3412/3502/3593/3756/5391/5433/5442/5453: Our coding convention states no line be longer than 80 characters. These lines all exceed 80 characters. - Line 5410: SntiTranslateAllModePagesResponse() contains new if-else blocks. Our coding convention dictates that open curly braces be placed on the same line as the if or else statement. You can refer to other instances of the code and if-else clauses as examples. - Line 4017: SntiCreateModeDataHeader() no longer needs the SRB pointer to be passed in as a parameter since you are using a global temp buffer to build up the mode sense data. That parameter can be removed. - General: The method of using a global buffer for a single mode sense command opens up a potential race condition for multiple mode sense commands submitted on top of each other. The idea behind putting the SCSI to NVMe translation phase in the HwStorBuildIo phase was to help performance. We initially knew that we would never be sharing any data structures or memory in calls to BuildIo. There is no spinlock or protection for calls to BuildIo and MSDN makes it clear that these BuildIo calls are not synchronized and that any synchronization of memory/data must be done within the function implementations or just use StartIo instead (since Storport will acquire the StartIoSpinLock). So there is hole here w/r/t the global buffer being populated for multiple mode sense commands at the same time. I believe you have the right idea, but the correct solution is to create a temp mode sense buffer inside the SRB extension. This way, each command has its own temp mode sense buffer for building the mode sense response, and then when ready to copy over to the SRB data buffer, it’s pulling from a buffer that can only be accessed by the current command (instead of a global buffer that may have been overwritten by another mode sense request call from another BuildIo call). Note that the driver collaboration group made the decision to never acquire any type of lock in BuildIo. Let me know if you have questions on this solution. The additional size to the SRB extension is not an issue. This fix is critical for this patch to be accepted. Thanks, Ray From: nvmewin-bounces at lists.openfabrics.org [mailto:nvmewin-bounces at lists.openfabrics.org] On Behalf Of Dharani Kotte Sent: Tuesday, April 30, 2013 11:14 AM To: Chang, Alex; nvmewin at lists.openfabrics.org Subject: [nvmewin] ***UNCHECKED*** [WARNING - ENCRYPTED ATTACHMENT NOT VIRUS SCANNED] New Patch From Sandisk Attaching the code with according to Alex suggestion. Passwd: sndk1234 Thanks, Dharani. From: Chang, Alex [mailto:Alex.Chang at idt.com] Sent: Tuesday, April 30, 2013 9:23 AM To: Dharani Kotte; nvmewin at lists.openfabrics.org Subject: RE: New Patch From Sandisk Hi Dharani, I'd suggest that it's safer to initialize gModeSenseBuf as all zeros each time before it is used, just in case there are some unexpected data in the buffer. Other than that, I am fine with the patch. Thanks, Alex ________________________________ From: nvmewin-bounces at lists.openfabrics.org [mailto:nvmewin-bounces at lists.openfabrics.org] On Behalf Of Dharani Kotte Sent: Tuesday, April 30, 2013 8:37 AM To: nvmewin at lists.openfabrics.org Subject: [nvmewin] New Patch From Sandisk Hi all, It's been almost two weeks after I sent out the patch. please let us know if you're okay with it. Thanks, Dharani. From: Dharani Kotte Sent: Friday, April 12, 2013 9:58 AM To: nvmewin at lists.openfabrics.org Subject: New Patch From Sandisk Hi all, I am attaching a new patch that includes the following changes: 1. In nvmeSnti.c a. Allocate a global buffer of 256bytes size for modesense return data preparation b. For any mode sense this buffer is used for modesense data along with the requested pages in it. c. Copy the required data according to the alloc_length requested in the cdb to the Srb->DataBuffer and update the dataTransferLength accordingly d. In function SntiTranslateReturnAllModePagesResponse() the modesense data header offset is calculated wrong which ends up in BSOD in some cases modified code to fix this issue Please review the changes and provide feedbacks if you have any. If nobody disagrees with the changes, I will remind Ray to merge them in two weeks. Thanks, Dharani. ________________________________ PLEASE NOTE: The information contained in this electronic mail message is intended only for the use of the designated recipient(s) named above. If the reader of this message is not the intended recipient, you are hereby notified that you have received this message in error and that any review, dissemination, distribution, or copying of this message is strictly prohibited. If you have received this communication in error, please notify the sender by telephone or e-mail (as shown above) immediately and destroy any and all copies of this message in your possession (whether hard copies or electronically stored copies). CAUTION: Please confirm that the password protected .zip attachment which contains the file(s) of type source_sndk_05_16_2013.zip is legitimate prior to opening. To make sure this message is not infected with a virus, it is important to verify that you are expecting the message or else confirm its legitimacy with the sender. -------------- next part -------------- An HTML attachment was scrubbed... URL: From Sumant.Patro at sandisk.com Thu May 30 15:20:24 2013 From: Sumant.Patro at sandisk.com (Sumant Patro) Date: Thu, 30 May 2013 22:20:24 +0000 Subject: [nvmewin] ***UNCHECKED*** [WARNING - ENCRYPTED ATTACHMENT NOT VIRUS SCANNED] RE: New Patch From Sandisk In-Reply-To: <49158E750348AA499168FD41D889836062578B98@FMSMSX105.amr.corp.intel.com> References: <49158E750348AA499168FD41D88983606256082A@FMSMSX105.amr.corp.intel.com> <548C5470AAD9DA4A85D259B663190D361FFE961C@corpmail1.na.ads.idt.com> <49158E750348AA499168FD41D889836062560AA2@FMSMSX105.amr.corp.intel.com> <49158E750348AA499168FD41D889836062578A2D@FMSMSX105.amr.corp.intel.com> <548C5470AAD9DA4A85D259B663190D361FFEC067@corpmail1.na.ads.idt.com> <49158E750348AA499168FD41D889836062578B98@FMSMSX105.amr.corp.intel.com> Message-ID: Hello Ray, Please find attached the patch that has fix for the last code review comment. This is the patch set that I had sent to Alex earlier. Password : sndk1234 Regards, Sumant From: Robles, Raymond C [mailto:raymond.c.robles at intel.com] Sent: Thursday, May 30, 2013 3:10 PM To: Chang, Alex; Sumant Patro; Dharani Kotte; nvmewin at lists.openfabrics.org Subject: RE: New Patch From Sandisk Thanks Alex. Sumant, please continue with your updates from the last set of code review comments and resend the patch (after running all the required unit tests of course). Once we confirm the feedback changes are in place, I’ll push the patch. Thanks, Ray From: Chang, Alex [mailto:Alex.Chang at idt.com] Sent: Thursday, May 30, 2013 3:07 PM To: Sumant Patro; Robles, Raymond C; Dharani Kotte; nvmewin at lists.openfabrics.org Subject: RE: New Patch From Sandisk Hi Sumant, Actually I did some preliminary debugging on it and found out the crash was not introduced by your patch. When the system gets normal shutdown, NVMeNormalShutdown calls NVMeFreeBuffers to free up all queue related structures. And later, when NVMeDetectPendingCmds is called to check on the pending commands, it refers to the queue structures and causes the crash. I will include the fix in my next patch that includes changes to be compliant with 1.0e NVMe Specification. Thanks, Alex ________________________________ From: nvmewin-bounces at lists.openfabrics.org [mailto:nvmewin-bounces at lists.openfabrics.org] On Behalf Of Sumant Patro Sent: Thursday, May 30, 2013 2:11 PM To: Robles, Raymond C; Dharani Kotte; nvmewin at lists.openfabrics.org Subject: Re: [nvmewin] New Patch From Sandisk Hello Alex, I haven’t been able to repro the BSOD issue that you saw in your setup. Would it be possible for you to send me the crashdump please ? Regards, Sumant From: Robles, Raymond C [mailto:raymond.c.robles at intel.com] Sent: Thursday, May 30, 2013 1:24 PM To: Sumant Patro; Dharani Kotte; nvmewin at lists.openfabrics.org Subject: RE: New Patch From Sandisk Importance: High Dharani/Sumant, What is the status on this patch? I’ve not seen any follow up to my last email below. Hopefully we can close this out soon as there are other patches waiting to be submitted. Thanks for your quick attention. Thanks, Ray From: nvmewin-bounces at lists.openfabrics.org [mailto:nvmewin-bounces at lists.openfabrics.org] On Behalf Of Robles, Raymond C Sent: Thursday, May 16, 2013 3:49 PM To: Chang, Alex; Sumant Patro; Knoblaugh, Rick; Dharani Kotte; nvmewin at lists.openfabrics.org Subject: Re: [nvmewin] New Patch From Sandisk Actually, I was going to type the same thing in my email below and then got derailed with the whole #define issue (thanks for capturing this Alex). There is no DataTransferLength field in the SRB Extension structure. That is defined in the SRB by Storport. As a general courtesy to other companies who are reviewing these patches, please insure that you are compiling and re-testing your patch prior to sending out for secondary or additional code reviews. This will help save valuable time in the code review process. If you are having issues compiling, or do not see the same errors, please contact me. Thanks, Ray From: Chang, Alex [mailto:Alex.Chang at idt.com] Sent: Thursday, May 16, 2013 3:44 PM To: Robles, Raymond C; Sumant Patro; Knoblaugh, Rick; Dharani Kotte; nvmewin at lists.openfabrics.org Subject: RE: New Patch From Sandisk Hi Sumant, The only change you've made in the new patch is replacing: memset(GET_DATA_BUFFER(pSrb), 0, allocLength); with: memset(GET_DATA_BUFFER(pSrb), 0, min(pSrbExt->DataTransferLength, allocLength)); However, it generates compiling error because DataTransferLength is not a member of NVME_SRB_EXTENSION structure. Could you please fix it? Thanks a lot for you help, Alex ________________________________ From: Robles, Raymond C [mailto:raymond.c.robles at intel.com] Sent: Thursday, May 16, 2013 1:37 PM To: Sumant Patro; Knoblaugh, Rick; Chang, Alex; Dharani Kotte; nvmewin at lists.openfabrics.org Subject: RE: New Patch From Sandisk Hi Sumant/Dharani, Thanks for the updated patch. A couple of comments/questions: - For all the mode sense page translations, the new temp buffer allocated in the SRB extension is used. But for the Modes Sense 0x3C (all mode pages), the SRB data buffer is still memset to 0 at the beginning of SntiReturnAllModePages(). I’m not sure we still need to do this considering how the temp buffer is used to build the mode sense data. - If you insist on keeping that memset in there, there is a #define in for all mode pages - MODE_SENSE_ALL_PAGES_LENGTH. It may be better to zero out the buffer based on the min between this define and allocLength. SRB data transfer length cannot be relied on for “data-in” commands (like mode sense). Thanks, Ray From: Sumant Patro [mailto:Sumant.Patro at sandisk.com] Sent: Thursday, May 16, 2013 12:43 PM To: Knoblaugh, Rick; Chang, Alex; Dharani Kotte; Robles, Raymond C; nvmewin at lists.openfabrics.org Subject: [WARNING - ENCRYPTED ATTACHMENT NOT VIRUS SCANNED] RE: ニ[WARNING - ENCRYPTED ATTACHMENT NOT VIRUS SCANNED] [nvmewin] ***UNCHECKED*** [WARNING - ENCRYPTED ATTACHMENT NOT VIRUS SCANNED] RE: New Patch From Sandisk Sorry, sent the wrong zip in the previous mail. Please ignore the previous attachment. Attached is the updated zip file. Regards, Sumant From: nvmewin-bounces at lists.openfabrics.org [mailto:nvmewin-bounces at lists.openfabrics.org] On Behalf Of Sumant Patro Sent: Thursday, May 16, 2013 12:38 PM To: Knoblaugh, Rick; Chang, Alex; Dharani Kotte; Robles, Raymond C; nvmewin at lists.openfabrics.org Subject: ニ[WARNING - ENCRYPTED ATTACHMENT NOT VIRUS SCANNED] [nvmewin] ***UNCHECKED*** [WARNING - ENCRYPTED ATTACHMENT NOT VIRUS SCANNED] RE: New Patch From Sandisk Thanks Rick, Alex for your feedback. Please find attached updated files with following two changes : 1. Comment in header of SntiCreateModeDataHeader function changed to reflect passing of srb extension 2. memset(GET_DATA_BUFFER(pSrb), 0, allocLength); modified to use minimum of allocLength and DataTransferLength in SntiReturnAllModePages(..) Password : sndk1234 Regards, Sumant From: nvmewin-bounces at lists.openfabrics.org [mailto:nvmewin-bounces at lists.openfabrics.org] On Behalf Of Knoblaugh, Rick Sent: Thursday, May 16, 2013 11:59 AM To: Chang, Alex; Dharani Kotte; Robles, Raymond C; nvmewin at lists.openfabrics.org Subject: Re: [nvmewin] New Patch From Sandisk Hi Dharani, When you change that one, may as well fix the comment in header of SntiCreateModeDataHeader function to reflect that srb extension is now the parameter that’s passed in. Other stuff looks good. Thanks, -Rick From: nvmewin-bounces at lists.openfabrics.org [mailto:nvmewin-bounces at lists.openfabrics.org] On Behalf Of Chang, Alex Sent: Thursday, May 16, 2013 10:58 AM To: Dharani Kotte; Robles, Raymond C; nvmewin at lists.openfabrics.org Subject: Re: [nvmewin] New Patch From Sandisk Hi Dharani, I tested your new patch and experienced a D1 bugcheck while running SCSICompliance test. Some preliminary debugging leads to Line#3633 of nvmestni.c, where it looks like: memset(GET_DATA_BUFFER(pSrb), 0, allocLength); The bugcheck happens when the value of "allocLength" is 0x1000. Apparently, the allocated length in CDB is somewhat a bogus value that is meant to test how driver handles it. Could you please fix it? Thanks, Alex ________________________________ From: nvmewin-bounces at lists.openfabrics.org [mailto:nvmewin-bounces at lists.openfabrics.org] On Behalf Of Dharani Kotte Sent: Wednesday, May 01, 2013 8:56 AM To: Robles, Raymond C; nvmewin at lists.openfabrics.org Subject: [nvmewin] ***UNCHECKED*** [WARNING - ENCRYPTED ATTACHMENT NOT VIRUS SCANNED] New Patch From Sandisk Hi Ray, I have incorporated the changes according to your suggestions, I tried to keep up with the coding convention please feel free to let me know if anything has to be changed. I have incorporated one more fix in the nvmeInit.c which can lead to memory corruptions and used the SrbExt for the buffer allocation. PassWd: sndk1234 Thanks, Dharani. From: Dharani Kotte Sent: Tuesday, April 30, 2013 12:03 PM To: 'Robles, Raymond C'; nvmewin at lists.openfabrics.org Subject: RE: New Patch From Sandisk Thank you for the feedback, I agree that the global buffer is not good I have made the changes to allocate this buffer from the srbext I will provide the code with the below modifications soon. Thanks, Dharani. From: Robles, Raymond C [mailto:raymond.c.robles at intel.com] Sent: Tuesday, April 30, 2013 11:52 AM To: Dharani Kotte; nvmewin at lists.openfabrics.org Subject: RE: New Patch From Sandisk Hi Dharani, Thank you for putting this patch together. Please see my feedback below: - Line 52: Note that we have a file called nvmeSntiTypes.h. This file contains all #defines. As per our existing convention, all #defines should be placed in this file (versus at the top of a .c file). - Line 3339/3412/3502/3593/3756/5391/5433/5442/5453: Our coding convention states no line be longer than 80 characters. These lines all exceed 80 characters. - Line 5410: SntiTranslateAllModePagesResponse() contains new if-else blocks. Our coding convention dictates that open curly braces be placed on the same line as the if or else statement. You can refer to other instances of the code and if-else clauses as examples. - Line 4017: SntiCreateModeDataHeader() no longer needs the SRB pointer to be passed in as a parameter since you are using a global temp buffer to build up the mode sense data. That parameter can be removed. - General: The method of using a global buffer for a single mode sense command opens up a potential race condition for multiple mode sense commands submitted on top of each other. The idea behind putting the SCSI to NVMe translation phase in the HwStorBuildIo phase was to help performance. We initially knew that we would never be sharing any data structures or memory in calls to BuildIo. There is no spinlock or protection for calls to BuildIo and MSDN makes it clear that these BuildIo calls are not synchronized and that any synchronization of memory/data must be done within the function implementations or just use StartIo instead (since Storport will acquire the StartIoSpinLock). So there is hole here w/r/t the global buffer being populated for multiple mode sense commands at the same time. I believe you have the right idea, but the correct solution is to create a temp mode sense buffer inside the SRB extension. This way, each command has its own temp mode sense buffer for building the mode sense response, and then when ready to copy over to the SRB data buffer, it’s pulling from a buffer that can only be accessed by the current command (instead of a global buffer that may have been overwritten by another mode sense request call from another BuildIo call). Note that the driver collaboration group made the decision to never acquire any type of lock in BuildIo. Let me know if you have questions on this solution. The additional size to the SRB extension is not an issue. This fix is critical for this patch to be accepted. Thanks, Ray From: nvmewin-bounces at lists.openfabrics.org [mailto:nvmewin-bounces at lists.openfabrics.org] On Behalf Of Dharani Kotte Sent: Tuesday, April 30, 2013 11:14 AM To: Chang, Alex; nvmewin at lists.openfabrics.org Subject: [nvmewin] ***UNCHECKED*** [WARNING - ENCRYPTED ATTACHMENT NOT VIRUS SCANNED] New Patch From Sandisk Attaching the code with according to Alex suggestion. Passwd: sndk1234 Thanks, Dharani. From: Chang, Alex [mailto:Alex.Chang at idt.com] Sent: Tuesday, April 30, 2013 9:23 AM To: Dharani Kotte; nvmewin at lists.openfabrics.org Subject: RE: New Patch From Sandisk Hi Dharani, I'd suggest that it's safer to initialize gModeSenseBuf as all zeros each time before it is used, just in case there are some unexpected data in the buffer. Other than that, I am fine with the patch. Thanks, Alex ________________________________ From: nvmewin-bounces at lists.openfabrics.org [mailto:nvmewin-bounces at lists.openfabrics.org] On Behalf Of Dharani Kotte Sent: Tuesday, April 30, 2013 8:37 AM To: nvmewin at lists.openfabrics.org Subject: [nvmewin] New Patch From Sandisk Hi all, It's been almost two weeks after I sent out the patch. please let us know if you're okay with it. Thanks, Dharani. From: Dharani Kotte Sent: Friday, April 12, 2013 9:58 AM To: nvmewin at lists.openfabrics.org Subject: New Patch From Sandisk Hi all, I am attaching a new patch that includes the following changes: 1. In nvmeSnti.c a. Allocate a global buffer of 256bytes size for modesense return data preparation b. For any mode sense this buffer is used for modesense data along with the requested pages in it. c. Copy the required data according to the alloc_length requested in the cdb to the Srb->DataBuffer and update the dataTransferLength accordingly d. In function SntiTranslateReturnAllModePagesResponse() the modesense data header offset is calculated wrong which ends up in BSOD in some cases modified code to fix this issue Please review the changes and provide feedbacks if you have any. If nobody disagrees with the changes, I will remind Ray to merge them in two weeks. Thanks, Dharani. ________________________________ PLEASE NOTE: The information contained in this electronic mail message is intended only for the use of the designated recipient(s) named above. If the reader of this message is not the intended recipient, you are hereby notified that you have received this message in error and that any review, dissemination, distribution, or copying of this message is strictly prohibited. If you have received this communication in error, please notify the sender by telephone or e-mail (as shown above) immediately and destroy any and all copies of this message in your possession (whether hard copies or electronically stored copies). CAUTION: Please confirm that the password protected .zip attachment which contains the file(s) of type source_sndk_05_16_2013.zip is legitimate prior to opening. To make sure this message is not infected with a virus, it is important to verify that you are expecting the message or else confirm its legitimacy with the sender. -------------- next part -------------- An HTML attachment was scrubbed... URL: -------------- next part -------------- A non-text attachment was scrubbed... Name: source_sndk_05_30_2013.zip Type: application/x-zip-compressed Size: 199239 bytes Desc: source_sndk_05_30_2013.zip URL: