From paul.e.luse at intel.com Thu Aug 9 07:58:11 2012 From: paul.e.luse at intel.com (Luse, Paul E) Date: Thu, 9 Aug 2012 14:58:11 +0000 Subject: [nvmewin] ***UNCHECKED*** Aug 9 - Patch (mainly IOCTL stuff, some general cleanup as well) Message-ID: <82C9F782B054C94B9FC04A331649C77A07B24E46@FMSMSX106.amr.corp.intel.com> All- Here's the next patch (pw is intel123), main focus is really IOCTL handling with some other small fixes as well. I'll be looking for feedback/approval by the end of next week (Aug 17); please let me know if you need more time. Tested as per our guidelines, no issues. Alex, you mentioned having some power transition changes almost ready - following acceptance of this patch can you being to prepare that one? Thx Paul PS: note that the SVN repo changed recently. Make sure you are setup according to the updated website as shown below: NVM Express for Windows subversion repository (SVN) can be accessed using an SVN client at the following: * svn://beany.openfabrics.org/nvmewin (note that the directory is case-sensitive) * http://beany.openfabrics.org/svn/nvmewin (note that the directory is case sensitive) - minor cleanup (some prints, extra chatham1 stuff, etc). There were a few "odd" issues in the trunk and I think we may have had some merge issues with the recent SVN change so once this is pushed be sure to update! - bug fix for reset handling: q ptrs were being reset to 0 in the wrong locations - bug fix: we weren't honoring the supports VWC bit in some SNTI routines - updates to format (NVM) handling based on new lunExt changes - slight change to how we handle initial touch of HW. Before we forced a reset by setting EN to 1 then reading it back then writing 1. We had a report of IHV HW where this was causing an issue because their interpretation of the spec, and thus their implementation, expected nothing to happen after setting EN to 1 until RDY is set, including setting EN back to 0. Rather than add an additional wait it seems reasonable to check if EN is already set and if so then set it to 0 to force an initial reset. If its 0 to begin with the n its either just been powered on (thus reset) or it made a transition from 1?0 thus was just reset. No change to where we enable the card for the first time, that's still in the init state machine - lots of structural rework in the IOCTL handlers. See code for details, let me know if there are questions, either as to what or why things were shuffled around. The initial implementation didn't take into account the need for IHVs to define their own private IOCTLs so decode need to be added/changed accordingly. Also, some IOCTLs need processing in startIO only (some private ones) and we didn't have a framework setup very well for that either. - removed a few no-op IOCTL placeholders. Note that NVMe does not define specific private IOCTLs other than the NVME PT IOCTL. We need (eventually, Intel will propose one soon if someone else doesn't) a standard struct to define a private IOCTL if we want one, or if IHVs want to define their own and keep both it and their private IOCTLs our of OFA that's fine. I left 2 privates in there (hot add, hot remove) because they don't have a corresponding structure, they're basically just op codes. - added a few IOCTLs for Msft public SMART commands (caused rework of the IOCTL generic callback) ____________________________________ Paul Luse Sr. Staff Engineer PCG Server Software Engineering Desk: 480.554.3688, Mobile: 480.334.4630 -------------- next part -------------- An HTML attachment was scrubbed... URL: -------------- next part -------------- A non-text attachment was scrubbed... Name: source.zip Type: application/x-zip-compressed Size: 911780 bytes Desc: source.zip URL: From Alex.Chang at idt.com Thu Aug 9 09:09:28 2012 From: Alex.Chang at idt.com (Chang, Alex) Date: Thu, 9 Aug 2012 16:09:28 +0000 Subject: [nvmewin] ***UNCHECKED*** Aug 9 - Patch (mainly IOCTL stuff, some general cleanup as well) In-Reply-To: <82C9F782B054C94B9FC04A331649C77A07B24E46@FMSMSX106.amr.corp.intel.com> References: <82C9F782B054C94B9FC04A331649C77A07B24E46@FMSMSX106.amr.corp.intel.com> Message-ID: <548C5470AAD9DA4A85D259B663190D360384CD@corpmail1.na.ads.idt.com> Paul, You're fast! I am just about to send out my patch either today or tomorrow after some more tests. Actually, I also changed the queue pointer resets while handling controller reset. I will add my patch after yours. Will also consider including the compliant changes for 1.0c specification that I promised a month ago. Thanks, Alex ________________________________ From: nvmewin-bounces at lists.openfabrics.org [mailto:nvmewin-bounces at lists.openfabrics.org] On Behalf Of Luse, Paul E Sent: Thursday, August 09, 2012 7:58 AM To: nvmewin at lists.openfabrics.org Subject: [nvmewin] ***UNCHECKED*** Aug 9 - Patch (mainly IOCTL stuff, some general cleanup as well) All- Here's the next patch (pw is intel123), main focus is really IOCTL handling with some other small fixes as well. I'll be looking for feedback/approval by the end of next week (Aug 17); please let me know if you need more time. Tested as per our guidelines, no issues. Alex, you mentioned having some power transition changes almost ready - following acceptance of this patch can you being to prepare that one? Thx Paul PS: note that the SVN repo changed recently. Make sure you are setup according to the updated website as shown below: NVM Express for Windows subversion repository (SVN) can be accessed using an SVN client at the following: * svn://beany.openfabrics.org/nvmewin (note that the directory is case-sensitive) * http://beany.openfabrics.org/svn/nvmewin (note that the directory is case sensitive) - minor cleanup (some prints, extra chatham1 stuff, etc). There were a few "odd" issues in the trunk and I think we may have had some merge issues with the recent SVN change so once this is pushed be sure to update! - bug fix for reset handling: q ptrs were being reset to 0 in the wrong locations - bug fix: we weren't honoring the supports VWC bit in some SNTI routines - updates to format (NVM) handling based on new lunExt changes - slight change to how we handle initial touch of HW. Before we forced a reset by setting EN to 1 then reading it back then writing 1. We had a report of IHV HW where this was causing an issue because their interpretation of the spec, and thus their implementation, expected nothing to happen after setting EN to 1 until RDY is set, including setting EN back to 0. Rather than add an additional wait it seems reasonable to check if EN is already set and if so then set it to 0 to force an initial reset. If its 0 to begin with the n its either just been powered on (thus reset) or it made a transition from 1?0 thus was just reset. No change to where we enable the card for the first time, that's still in the init state machine - lots of structural rework in the IOCTL handlers. See code for details, let me know if there are questions, either as to what or why things were shuffled around. The initial implementation didn't take into account the need for IHVs to define their own private IOCTLs so decode need to be added/changed accordingly. Also, some IOCTLs need processing in startIO only (some private ones) and we didn't have a framework setup very well for that either. - removed a few no-op IOCTL placeholders. Note that NVMe does not define specific private IOCTLs other than the NVME PT IOCTL. We need (eventually, Intel will propose one soon if someone else doesn't) a standard struct to define a private IOCTL if we want one, or if IHVs want to define their own and keep both it and their private IOCTLs our of OFA that's fine. I left 2 privates in there (hot add, hot remove) because they don't have a corresponding structure, they're basically just op codes. - added a few IOCTLs for Msft public SMART commands (caused rework of the IOCTL generic callback) ____________________________________ Paul Luse Sr. Staff Engineer PCG Server Software Engineering Desk: 480.554.3688, Mobile: 480.334.4630 -------------- next part -------------- An HTML attachment was scrubbed... URL: From paul.e.luse at intel.com Thu Aug 9 09:11:17 2012 From: paul.e.luse at intel.com (Luse, Paul E) Date: Thu, 9 Aug 2012 16:11:17 +0000 Subject: [nvmewin] ***UNCHECKED*** Aug 9 - Patch (mainly IOCTL stuff, some general cleanup as well) In-Reply-To: <548C5470AAD9DA4A85D259B663190D360384CD@corpmail1.na.ads.idt.com> References: <82C9F782B054C94B9FC04A331649C77A07B24E46@FMSMSX106.amr.corp.intel.com> <548C5470AAD9DA4A85D259B663190D360384CD@corpmail1.na.ads.idt.com> Message-ID: <82C9F782B054C94B9FC04A331649C77A07B25176@FMSMSX106.amr.corp.intel.com> Thanks Alex J From: Chang, Alex [mailto:Alex.Chang at idt.com] Sent: Thursday, August 09, 2012 9:09 AM To: Luse, Paul E; nvmewin at lists.openfabrics.org Subject: RE: [nvmewin] ***UNCHECKED*** Aug 9 - Patch (mainly IOCTL stuff, some general cleanup as well) Paul, You're fast! I am just about to send out my patch either today or tomorrow after some more tests. Actually, I also changed the queue pointer resets while handling controller reset. I will add my patch after yours. Will also consider including the compliant changes for 1.0c specification that I promised a month ago. Thanks, Alex ________________________________ From: nvmewin-bounces at lists.openfabrics.org [mailto:nvmewin-bounces at lists.openfabrics.org] On Behalf Of Luse, Paul E Sent: Thursday, August 09, 2012 7:58 AM To: nvmewin at lists.openfabrics.org Subject: [nvmewin] ***UNCHECKED*** Aug 9 - Patch (mainly IOCTL stuff, some general cleanup as well) All- Here's the next patch (pw is intel123), main focus is really IOCTL handling with some other small fixes as well. I'll be looking for feedback/approval by the end of next week (Aug 17); please let me know if you need more time. Tested as per our guidelines, no issues. Alex, you mentioned having some power transition changes almost ready - following acceptance of this patch can you being to prepare that one? Thx Paul PS: note that the SVN repo changed recently. Make sure you are setup according to the updated website as shown below: NVM Express for Windows subversion repository (SVN) can be accessed using an SVN client at the following: * svn://beany.openfabrics.org/nvmewin (note that the directory is case-sensitive) * http://beany.openfabrics.org/svn/nvmewin (note that the directory is case sensitive) - minor cleanup (some prints, extra chatham1 stuff, etc). There were a few "odd" issues in the trunk and I think we may have had some merge issues with the recent SVN change so once this is pushed be sure to update! - bug fix for reset handling: q ptrs were being reset to 0 in the wrong locations - bug fix: we weren't honoring the supports VWC bit in some SNTI routines - updates to format (NVM) handling based on new lunExt changes - slight change to how we handle initial touch of HW. Before we forced a reset by setting EN to 1 then reading it back then writing 1. We had a report of IHV HW where this was causing an issue because their interpretation of the spec, and thus their implementation, expected nothing to happen after setting EN to 1 until RDY is set, including setting EN back to 0. Rather than add an additional wait it seems reasonable to check if EN is already set and if so then set it to 0 to force an initial reset. If its 0 to begin with the n its either just been powered on (thus reset) or it made a transition from 1?0 thus was just reset. No change to where we enable the card for the first time, that's still in the init state machine - lots of structural rework in the IOCTL handlers. See code for details, let me know if there are questions, either as to what or why things were shuffled around. The initial implementation didn't take into account the need for IHVs to define their own private IOCTLs so decode need to be added/changed accordingly. Also, some IOCTLs need processing in startIO only (some private ones) and we didn't have a framework setup very well for that either. - removed a few no-op IOCTL placeholders. Note that NVMe does not define specific private IOCTLs other than the NVME PT IOCTL. We need (eventually, Intel will propose one soon if someone else doesn't) a standard struct to define a private IOCTL if we want one, or if IHVs want to define their own and keep both it and their private IOCTLs our of OFA that's fine. I left 2 privates in there (hot add, hot remove) because they don't have a corresponding structure, they're basically just op codes. - added a few IOCTLs for Msft public SMART commands (caused rework of the IOCTL generic callback) ____________________________________ Paul Luse Sr. Staff Engineer PCG Server Software Engineering Desk: 480.554.3688, Mobile: 480.334.4630 -------------- next part -------------- An HTML attachment was scrubbed... URL: From Alex.Chang at idt.com Fri Aug 10 14:59:00 2012 From: Alex.Chang at idt.com (Chang, Alex) Date: Fri, 10 Aug 2012 21:59:00 +0000 Subject: [nvmewin] ***UNCHECKED*** Aug 9 - Patch (mainly IOCTL stuff, some general cleanup as well) In-Reply-To: <82C9F782B054C94B9FC04A331649C77A07B24E46@FMSMSX106.amr.corp.intel.com> References: <82C9F782B054C94B9FC04A331649C77A07B24E46@FMSMSX106.amr.corp.intel.com> Message-ID: <548C5470AAD9DA4A85D259B663190D360385D0@corpmail1.na.ads.idt.com> Hi Paul, Some questions after browsing the difference between current source and your changes: 1. Why you removed "CC.SHN = 1;" in NVMeNormalShutdown routine? 2. In nvme.h, why you changed the opcode of NVM_DATASET_MANAGEMENT from 0x06 to 0x09? 3. Is there any specific reason(s), you added the following in NVMeInitialize, and then immediately clear EN without checking RDY = 1? /* Set EN to 1 (all others zero) */ CC.EN = 1; StorPortWriteRegisterUlong(pAE, (PULONG)(&pAE->pCtrlRegister->CC), CC.AsUlong); Thanks, Alex ________________________________ From: nvmewin-bounces at lists.openfabrics.org [mailto:nvmewin-bounces at lists.openfabrics.org] On Behalf Of Luse, Paul E Sent: Thursday, August 09, 2012 7:58 AM To: nvmewin at lists.openfabrics.org Subject: [nvmewin] ***UNCHECKED*** Aug 9 - Patch (mainly IOCTL stuff, some general cleanup as well) All- Here's the next patch (pw is intel123), main focus is really IOCTL handling with some other small fixes as well. I'll be looking for feedback/approval by the end of next week (Aug 17); please let me know if you need more time. Tested as per our guidelines, no issues. Alex, you mentioned having some power transition changes almost ready - following acceptance of this patch can you being to prepare that one? Thx Paul PS: note that the SVN repo changed recently. Make sure you are setup according to the updated website as shown below: NVM Express for Windows subversion repository (SVN) can be accessed using an SVN client at the following: * svn://beany.openfabrics.org/nvmewin (note that the directory is case-sensitive) * http://beany.openfabrics.org/svn/nvmewin (note that the directory is case sensitive) - minor cleanup (some prints, extra chatham1 stuff, etc). There were a few "odd" issues in the trunk and I think we may have had some merge issues with the recent SVN change so once this is pushed be sure to update! - bug fix for reset handling: q ptrs were being reset to 0 in the wrong locations - bug fix: we weren't honoring the supports VWC bit in some SNTI routines - updates to format (NVM) handling based on new lunExt changes - slight change to how we handle initial touch of HW. Before we forced a reset by setting EN to 1 then reading it back then writing 1. We had a report of IHV HW where this was causing an issue because their interpretation of the spec, and thus their implementation, expected nothing to happen after setting EN to 1 until RDY is set, including setting EN back to 0. Rather than add an additional wait it seems reasonable to check if EN is already set and if so then set it to 0 to force an initial reset. If its 0 to begin with the n its either just been powered on (thus reset) or it made a transition from 1?0 thus was just reset. No change to where we enable the card for the first time, that's still in the init state machine - lots of structural rework in the IOCTL handlers. See code for details, let me know if there are questions, either as to what or why things were shuffled around. The initial implementation didn't take into account the need for IHVs to define their own private IOCTLs so decode need to be added/changed accordingly. Also, some IOCTLs need processing in startIO only (some private ones) and we didn't have a framework setup very well for that either. - removed a few no-op IOCTL placeholders. Note that NVMe does not define specific private IOCTLs other than the NVME PT IOCTL. We need (eventually, Intel will propose one soon if someone else doesn't) a standard struct to define a private IOCTL if we want one, or if IHVs want to define their own and keep both it and their private IOCTLs our of OFA that's fine. I left 2 privates in there (hot add, hot remove) because they don't have a corresponding structure, they're basically just op codes. - added a few IOCTLs for Msft public SMART commands (caused rework of the IOCTL generic callback) ____________________________________ Paul Luse Sr. Staff Engineer PCG Server Software Engineering Desk: 480.554.3688, Mobile: 480.334.4630 -------------- next part -------------- An HTML attachment was scrubbed... URL: From paul.e.luse at intel.com Wed Aug 15 08:12:10 2012 From: paul.e.luse at intel.com (Luse, Paul E) Date: Wed, 15 Aug 2012 15:12:10 +0000 Subject: [nvmewin] patch review status Message-ID: <82C9F782B054C94B9FC04A331649C77A07B2AECD@FMSMSX106.amr.corp.intel.com> Just checking in... any questions to date and how are folks tracking for completing their review by this Fri? Thx Paul ____________________________________ Paul Luse Sr. Staff Engineer PCG Server Software Engineering Desk: 480.554.3688, Mobile: 480.334.4630 -------------- next part -------------- An HTML attachment was scrubbed... URL: From Alex.Chang at idt.com Wed Aug 15 08:50:45 2012 From: Alex.Chang at idt.com (Chang, Alex) Date: Wed, 15 Aug 2012 15:50:45 +0000 Subject: [nvmewin] patch review status In-Reply-To: <82C9F782B054C94B9FC04A331649C77A07B2AECD@FMSMSX106.amr.corp.intel.com> References: <82C9F782B054C94B9FC04A331649C77A07B2AECD@FMSMSX106.amr.corp.intel.com> Message-ID: <548C5470AAD9DA4A85D259B663190D36038760@corpmail1.na.ads.idt.com> Hi Paul, Some questions after browsing the difference between current source and your changes: 1. Why you removed "CC.SHN = 1;" in NVMeNormalShutdown routine? 2. In nvme.h, why you changed the opcode of NVM_DATASET_MANAGEMENT from 0x06 to 0x09? 3. Is there any specific reason(s), you added the following in NVMeInitialize, and then immediately clear EN without checking RDY = 1? /* Set EN to 1 (all others zero) */ CC.EN = 1; StorPortWriteRegisterUlong(pAE, (PULONG)(&pAE->pCtrlRegister->CC), CC.AsUlong); Thanks, Alex ________________________________ From: nvmewin-bounces at lists.openfabrics.org [mailto:nvmewin-bounces at lists.openfabrics.org] On Behalf Of Luse, Paul E Sent: Wednesday, August 15, 2012 8:12 AM To: nvmewin at lists.openfabrics.org Subject: [nvmewin] patch review status Just checking in... any questions to date and how are folks tracking for completing their review by this Fri? Thx Paul ____________________________________ Paul Luse Sr. Staff Engineer PCG Server Software Engineering Desk: 480.554.3688, Mobile: 480.334.4630 -------------- next part -------------- An HTML attachment was scrubbed... URL: From paul.e.luse at intel.com Wed Aug 15 09:03:05 2012 From: paul.e.luse at intel.com (Luse, Paul E) Date: Wed, 15 Aug 2012 16:03:05 +0000 Subject: [nvmewin] patch review status In-Reply-To: <548C5470AAD9DA4A85D259B663190D36038760@corpmail1.na.ads.idt.com> References: <82C9F782B054C94B9FC04A331649C77A07B2AECD@FMSMSX106.amr.corp.intel.com> <548C5470AAD9DA4A85D259B663190D36038760@corpmail1.na.ads.idt.com> Message-ID: <82C9F782B054C94B9FC04A331649C77A07B2AF6F@FMSMSX106.amr.corp.intel.com> Thanks Alex, see below. From: Chang, Alex [mailto:Alex.Chang at idt.com] Sent: Wednesday, August 15, 2012 8:51 AM To: Luse, Paul E; nvmewin at lists.openfabrics.org Subject: RE: patch review status Hi Paul, Some questions after browsing the difference between current source and your changes: 1. Why you removed "CC.SHN = 1;" in NVMeNormalShutdown routine? PL> Good eye, that was a mistake. I'll add it back before pushing 2. In nvme.h, why you changed the opcode of NVM_DATASET_MANAGEMENT from 0x06 to 0x09? PL> 6 was wrong, 9 is correct per spec 3. Is there any specific reason(s), you added the following in NVMeInitialize, and then immediately clear EN without checking RDY = 1? /* Set EN to 1 (all others zero) */ CC.EN = 1; StorPortWriteRegisterUlong(pAE, (PULONG)(&pAE->pCtrlRegister->CC), CC.AsUlong); PL> I wrote this up in the patch submission and I can't find that email right now. I didn't add code, I changed code. Before, we use to set EN to 1, read it back, and then set it to 0 to kick off a reset and then later we wait for RDY in the init state machine. We had an IHV complain that their implementation required waiting on RDY after setting EN 1 to even before clearing EN. So, our old implementation was a problem. So what I did now is to not force a reset unless EN is set to 1 when we come in. If its set to 0 then either (a) we're coming from power on or (b) the pre_OS reset the card when it was done. Either way if its already 0 then we're in a known state so we do nothing and then later we set to 1 and wait on RDY. If its 1 when we enter here we set to 0 and then later set to 1 and wait on RDY. Make sense? This assures we never enable without waiting for RDY ***** WAS ******* /* Set EN to 1 (all others zero) */ CC.EN = 1; StorPortWriteRegisterUlong(pAE, (PULONG)(&pAE->pCtrlRegister->CC), CC.AsUlong); /* Read it back */ CC.AsUlong = StorPortReadRegisterUlong(pAE, (PULONG)(&pAE->pCtrlRegister->CC)); /* Now reset */ CC.EN = 0; StorPortWriteRegisterUlong(pAE, (PULONG)(&pAE->pCtrlRegister->CC), CC.AsUlong); CC.AsUlong = StorPortReadRegisterUlong(pAE, (PULONG)(&pAE->pCtrlRegister->CC)); *** NOW IS ******* if (CC.EN == 1) { /* Now reset */ CC.EN = 0; StorPortWriteRegisterUlong(pAE, (PULONG)(&pAE->pCtrlRegister->CC), CC.AsUlong); } Thanks, Alex ________________________________ From: nvmewin-bounces at lists.openfabrics.org [mailto:nvmewin-bounces at lists.openfabrics.org] On Behalf Of Luse, Paul E Sent: Wednesday, August 15, 2012 8:12 AM To: nvmewin at lists.openfabrics.org Subject: [nvmewin] patch review status Just checking in... any questions to date and how are folks tracking for completing their review by this Fri? Thx Paul ____________________________________ Paul Luse Sr. Staff Engineer PCG Server Software Engineering Desk: 480.554.3688, Mobile: 480.334.4630 -------------- next part -------------- An HTML attachment was scrubbed... URL: From Alex.Chang at idt.com Wed Aug 15 09:51:48 2012 From: Alex.Chang at idt.com (Chang, Alex) Date: Wed, 15 Aug 2012 16:51:48 +0000 Subject: [nvmewin] patch review status In-Reply-To: <82C9F782B054C94B9FC04A331649C77A07B2AF6F@FMSMSX106.amr.corp.intel.com> References: <82C9F782B054C94B9FC04A331649C77A07B2AECD@FMSMSX106.amr.corp.intel.com> <548C5470AAD9DA4A85D259B663190D36038760@corpmail1.na.ads.idt.com> <82C9F782B054C94B9FC04A331649C77A07B2AF6F@FMSMSX106.amr.corp.intel.com> Message-ID: <548C5470AAD9DA4A85D259B663190D3603878D@corpmail1.na.ads.idt.com> Hi Paul, I agree with the new logic on #3. However, instead of checking CC.EN, should we check STS.RDY or both? Thanks, Alex ________________________________ From: Luse, Paul E [mailto:paul.e.luse at intel.com] Sent: Wednesday, August 15, 2012 9:03 AM To: Chang, Alex; nvmewin at lists.openfabrics.org Subject: RE: patch review status Thanks Alex, see below. From: Chang, Alex [mailto:Alex.Chang at idt.com] Sent: Wednesday, August 15, 2012 8:51 AM To: Luse, Paul E; nvmewin at lists.openfabrics.org Subject: RE: patch review status Hi Paul, Some questions after browsing the difference between current source and your changes: 1. Why you removed "CC.SHN = 1;" in NVMeNormalShutdown routine? PL> Good eye, that was a mistake. I'll add it back before pushing 2. In nvme.h, why you changed the opcode of NVM_DATASET_MANAGEMENT from 0x06 to 0x09? PL> 6 was wrong, 9 is correct per spec 3. Is there any specific reason(s), you added the following in NVMeInitialize, and then immediately clear EN without checking RDY = 1? /* Set EN to 1 (all others zero) */ CC.EN = 1; StorPortWriteRegisterUlong(pAE, (PULONG)(&pAE->pCtrlRegister->CC), CC.AsUlong); PL> I wrote this up in the patch submission and I can't find that email right now. I didn't add code, I changed code. Before, we use to set EN to 1, read it back, and then set it to 0 to kick off a reset and then later we wait for RDY in the init state machine. We had an IHV complain that their implementation required waiting on RDY after setting EN 1 to even before clearing EN. So, our old implementation was a problem. So what I did now is to not force a reset unless EN is set to 1 when we come in. If its set to 0 then either (a) we're coming from power on or (b) the pre_OS reset the card when it was done. Either way if its already 0 then we're in a known state so we do nothing and then later we set to 1 and wait on RDY. If its 1 when we enter here we set to 0 and then later set to 1 and wait on RDY. Make sense? This assures we never enable without waiting for RDY ***** WAS ******* /* Set EN to 1 (all others zero) */ CC.EN = 1; StorPortWriteRegisterUlong(pAE, (PULONG)(&pAE->pCtrlRegister->CC), CC.AsUlong); /* Read it back */ CC.AsUlong = StorPortReadRegisterUlong(pAE, (PULONG)(&pAE->pCtrlRegister->CC)); /* Now reset */ CC.EN = 0; StorPortWriteRegisterUlong(pAE, (PULONG)(&pAE->pCtrlRegister->CC), CC.AsUlong); CC.AsUlong = StorPortReadRegisterUlong(pAE, (PULONG)(&pAE->pCtrlRegister->CC)); *** NOW IS ******* if (CC.EN == 1) { /* Now reset */ CC.EN = 0; StorPortWriteRegisterUlong(pAE, (PULONG)(&pAE->pCtrlRegister->CC), CC.AsUlong); } Thanks, Alex ________________________________ From: nvmewin-bounces at lists.openfabrics.org [mailto:nvmewin-bounces at lists.openfabrics.org] On Behalf Of Luse, Paul E Sent: Wednesday, August 15, 2012 8:12 AM To: nvmewin at lists.openfabrics.org Subject: [nvmewin] patch review status Just checking in... any questions to date and how are folks tracking for completing their review by this Fri? Thx Paul ____________________________________ Paul Luse Sr. Staff Engineer PCG Server Software Engineering Desk: 480.554.3688, Mobile: 480.334.4630 -------------- next part -------------- An HTML attachment was scrubbed... URL: From paul.e.luse at intel.com Wed Aug 15 09:55:32 2012 From: paul.e.luse at intel.com (Luse, Paul E) Date: Wed, 15 Aug 2012 16:55:32 +0000 Subject: [nvmewin] patch review status In-Reply-To: <548C5470AAD9DA4A85D259B663190D3603878D@corpmail1.na.ads.idt.com> References: <82C9F782B054C94B9FC04A331649C77A07B2AECD@FMSMSX106.amr.corp.intel.com> <548C5470AAD9DA4A85D259B663190D36038760@corpmail1.na.ads.idt.com> <82C9F782B054C94B9FC04A331649C77A07B2AF6F@FMSMSX106.amr.corp.intel.com> <548C5470AAD9DA4A85D259B663190D3603878D@corpmail1.na.ads.idt.com> Message-ID: <82C9F782B054C94B9FC04A331649C77A07B2B051@FMSMSX106.amr.corp.intel.com> OK, I see what you're saying I think. We don't want to check for RDY without first looking at EN so does this make sense: if (CC.EN == 1) { -- now wait for RDY - /* Now reset */ CC.EN = 0; StorPortWriteRegisterUlong(pAE, (PULONG)(&pAE->pCtrlRegister->CC), CC.AsUlong); } From: Chang, Alex [mailto:Alex.Chang at idt.com] Sent: Wednesday, August 15, 2012 9:52 AM To: Luse, Paul E; nvmewin at lists.openfabrics.org Subject: RE: patch review status Hi Paul, I agree with the new logic on #3. However, instead of checking CC.EN, should we check STS.RDY or both? Thanks, Alex ________________________________ From: Luse, Paul E [mailto:paul.e.luse at intel.com] Sent: Wednesday, August 15, 2012 9:03 AM To: Chang, Alex; nvmewin at lists.openfabrics.org Subject: RE: patch review status Thanks Alex, see below. From: Chang, Alex [mailto:Alex.Chang at idt.com] Sent: Wednesday, August 15, 2012 8:51 AM To: Luse, Paul E; nvmewin at lists.openfabrics.org Subject: RE: patch review status Hi Paul, Some questions after browsing the difference between current source and your changes: 1. Why you removed "CC.SHN = 1;" in NVMeNormalShutdown routine? PL> Good eye, that was a mistake. I'll add it back before pushing 2. In nvme.h, why you changed the opcode of NVM_DATASET_MANAGEMENT from 0x06 to 0x09? PL> 6 was wrong, 9 is correct per spec 3. Is there any specific reason(s), you added the following in NVMeInitialize, and then immediately clear EN without checking RDY = 1? /* Set EN to 1 (all others zero) */ CC.EN = 1; StorPortWriteRegisterUlong(pAE, (PULONG)(&pAE->pCtrlRegister->CC), CC.AsUlong); PL> I wrote this up in the patch submission and I can't find that email right now. I didn't add code, I changed code. Before, we use to set EN to 1, read it back, and then set it to 0 to kick off a reset and then later we wait for RDY in the init state machine. We had an IHV complain that their implementation required waiting on RDY after setting EN 1 to even before clearing EN. So, our old implementation was a problem. So what I did now is to not force a reset unless EN is set to 1 when we come in. If its set to 0 then either (a) we're coming from power on or (b) the pre_OS reset the card when it was done. Either way if its already 0 then we're in a known state so we do nothing and then later we set to 1 and wait on RDY. If its 1 when we enter here we set to 0 and then later set to 1 and wait on RDY. Make sense? This assures we never enable without waiting for RDY ***** WAS ******* /* Set EN to 1 (all others zero) */ CC.EN = 1; StorPortWriteRegisterUlong(pAE, (PULONG)(&pAE->pCtrlRegister->CC), CC.AsUlong); /* Read it back */ CC.AsUlong = StorPortReadRegisterUlong(pAE, (PULONG)(&pAE->pCtrlRegister->CC)); /* Now reset */ CC.EN = 0; StorPortWriteRegisterUlong(pAE, (PULONG)(&pAE->pCtrlRegister->CC), CC.AsUlong); CC.AsUlong = StorPortReadRegisterUlong(pAE, (PULONG)(&pAE->pCtrlRegister->CC)); *** NOW IS ******* if (CC.EN == 1) { /* Now reset */ CC.EN = 0; StorPortWriteRegisterUlong(pAE, (PULONG)(&pAE->pCtrlRegister->CC), CC.AsUlong); } Thanks, Alex ________________________________ From: nvmewin-bounces at lists.openfabrics.org [mailto:nvmewin-bounces at lists.openfabrics.org] On Behalf Of Luse, Paul E Sent: Wednesday, August 15, 2012 8:12 AM To: nvmewin at lists.openfabrics.org Subject: [nvmewin] patch review status Just checking in... any questions to date and how are folks tracking for completing their review by this Fri? Thx Paul ____________________________________ Paul Luse Sr. Staff Engineer PCG Server Software Engineering Desk: 480.554.3688, Mobile: 480.334.4630 -------------- next part -------------- An HTML attachment was scrubbed... URL: From Alex.Chang at idt.com Wed Aug 15 10:20:48 2012 From: Alex.Chang at idt.com (Chang, Alex) Date: Wed, 15 Aug 2012 17:20:48 +0000 Subject: [nvmewin] patch review status In-Reply-To: <82C9F782B054C94B9FC04A331649C77A07B2B051@FMSMSX106.amr.corp.intel.com> References: <82C9F782B054C94B9FC04A331649C77A07B2AECD@FMSMSX106.amr.corp.intel.com> <548C5470AAD9DA4A85D259B663190D36038760@corpmail1.na.ads.idt.com> <82C9F782B054C94B9FC04A331649C77A07B2AF6F@FMSMSX106.amr.corp.intel.com> <548C5470AAD9DA4A85D259B663190D3603878D@corpmail1.na.ads.idt.com> <82C9F782B054C94B9FC04A331649C77A07B2B051@FMSMSX106.amr.corp.intel.com> Message-ID: <548C5470AAD9DA4A85D259B663190D360397B2@corpmail1.na.ads.idt.com> Absolutely. Now, the tricky part is when CC.EN=1 and STS.RDY never becomes 1. I'd say simply check STS.RDY only. If it's 1, then reset the device, otherwise, skip resetting it. What do you think? Thanks, Alex ________________________________ From: Luse, Paul E [mailto:paul.e.luse at intel.com] Sent: Wednesday, August 15, 2012 9:56 AM To: Chang, Alex; nvmewin at lists.openfabrics.org Subject: RE: patch review status OK, I see what you're saying I think. We don't want to check for RDY without first looking at EN so does this make sense: if (CC.EN == 1) { -- now wait for RDY - /* Now reset */ CC.EN = 0; StorPortWriteRegisterUlong(pAE, (PULONG)(&pAE->pCtrlRegister->CC), CC.AsUlong); } From: Chang, Alex [mailto:Alex.Chang at idt.com] Sent: Wednesday, August 15, 2012 9:52 AM To: Luse, Paul E; nvmewin at lists.openfabrics.org Subject: RE: patch review status Hi Paul, I agree with the new logic on #3. However, instead of checking CC.EN, should we check STS.RDY or both? Thanks, Alex ________________________________ From: Luse, Paul E [mailto:paul.e.luse at intel.com] Sent: Wednesday, August 15, 2012 9:03 AM To: Chang, Alex; nvmewin at lists.openfabrics.org Subject: RE: patch review status Thanks Alex, see below. From: Chang, Alex [mailto:Alex.Chang at idt.com] Sent: Wednesday, August 15, 2012 8:51 AM To: Luse, Paul E; nvmewin at lists.openfabrics.org Subject: RE: patch review status Hi Paul, Some questions after browsing the difference between current source and your changes: 1. Why you removed "CC.SHN = 1;" in NVMeNormalShutdown routine? PL> Good eye, that was a mistake. I'll add it back before pushing 2. In nvme.h, why you changed the opcode of NVM_DATASET_MANAGEMENT from 0x06 to 0x09? PL> 6 was wrong, 9 is correct per spec 3. Is there any specific reason(s), you added the following in NVMeInitialize, and then immediately clear EN without checking RDY = 1? /* Set EN to 1 (all others zero) */ CC.EN = 1; StorPortWriteRegisterUlong(pAE, (PULONG)(&pAE->pCtrlRegister->CC), CC.AsUlong); PL> I wrote this up in the patch submission and I can't find that email right now. I didn't add code, I changed code. Before, we use to set EN to 1, read it back, and then set it to 0 to kick off a reset and then later we wait for RDY in the init state machine. We had an IHV complain that their implementation required waiting on RDY after setting EN 1 to even before clearing EN. So, our old implementation was a problem. So what I did now is to not force a reset unless EN is set to 1 when we come in. If its set to 0 then either (a) we're coming from power on or (b) the pre_OS reset the card when it was done. Either way if its already 0 then we're in a known state so we do nothing and then later we set to 1 and wait on RDY. If its 1 when we enter here we set to 0 and then later set to 1 and wait on RDY. Make sense? This assures we never enable without waiting for RDY ***** WAS ******* /* Set EN to 1 (all others zero) */ CC.EN = 1; StorPortWriteRegisterUlong(pAE, (PULONG)(&pAE->pCtrlRegister->CC), CC.AsUlong); /* Read it back */ CC.AsUlong = StorPortReadRegisterUlong(pAE, (PULONG)(&pAE->pCtrlRegister->CC)); /* Now reset */ CC.EN = 0; StorPortWriteRegisterUlong(pAE, (PULONG)(&pAE->pCtrlRegister->CC), CC.AsUlong); CC.AsUlong = StorPortReadRegisterUlong(pAE, (PULONG)(&pAE->pCtrlRegister->CC)); *** NOW IS ******* if (CC.EN == 1) { /* Now reset */ CC.EN = 0; StorPortWriteRegisterUlong(pAE, (PULONG)(&pAE->pCtrlRegister->CC), CC.AsUlong); } Thanks, Alex ________________________________ From: nvmewin-bounces at lists.openfabrics.org [mailto:nvmewin-bounces at lists.openfabrics.org] On Behalf Of Luse, Paul E Sent: Wednesday, August 15, 2012 8:12 AM To: nvmewin at lists.openfabrics.org Subject: [nvmewin] patch review status Just checking in... any questions to date and how are folks tracking for completing their review by this Fri? Thx Paul ____________________________________ Paul Luse Sr. Staff Engineer PCG Server Software Engineering Desk: 480.554.3688, Mobile: 480.334.4630 -------------- next part -------------- An HTML attachment was scrubbed... URL: From bpowers at cadence.com Wed Aug 15 10:30:55 2012 From: bpowers at cadence.com (Brenda Powers) Date: Wed, 15 Aug 2012 10:30:55 -0700 Subject: [nvmewin] Setting a Windows Debugging Environment Message-ID: I have been trying to get a Windows debugging environment working and have had no luck. I have tried using Visual Studio 11 and going through a USB, 1394 and RS232 connection but have had no success. I can get the remote system provisioned but cannot get a connection through either a USB, 1394 or RS232. I have tried using Windbg and have the same problem of failing to get a connection established. Can anyone give me the details of their debugging setup which I can duplicate? Brenda Powers Cadence -------------- next part -------------- An HTML attachment was scrubbed... URL: From paul.e.luse at intel.com Wed Aug 15 10:43:21 2012 From: paul.e.luse at intel.com (Luse, Paul E) Date: Wed, 15 Aug 2012 17:43:21 +0000 Subject: [nvmewin] patch review status In-Reply-To: <548C5470AAD9DA4A85D259B663190D360397B2@corpmail1.na.ads.idt.com> References: <82C9F782B054C94B9FC04A331649C77A07B2AECD@FMSMSX106.amr.corp.intel.com> <548C5470AAD9DA4A85D259B663190D36038760@corpmail1.na.ads.idt.com> <82C9F782B054C94B9FC04A331649C77A07B2AF6F@FMSMSX106.amr.corp.intel.com> <548C5470AAD9DA4A85D259B663190D3603878D@corpmail1.na.ads.idt.com> <82C9F782B054C94B9FC04A331649C77A07B2B051@FMSMSX106.amr.corp.intel.com> <548C5470AAD9DA4A85D259B663190D360397B2@corpmail1.na.ads.idt.com> Message-ID: <82C9F782B054C94B9FC04A331649C77A07B2B0C9@FMSMSX106.amr.corp.intel.com> OK, I think I can get behind that. I was thinking STS might not be valid when EN is 0 but looking at the spec I think this is fine Ready (RDY): This field is set to '1' when the controller is ready to process commands after CC.EN is set to '1'. This field shall be cleared to '0' when CC.EN is cleared to '0'. Commands shall not be issued to the controller until this field is set to '1' after the CC.EN bit is set to '1'. Failure to follow this requirement produces undefined results. Host software shall wait a minimum of CAP.TO seconds for this field to be set to '1' after setting CC.EN to '1' from a previous value of '0'. So I'll just chance the check below for EN == 1 to RDY == 1 then Thanks! Paul From: Chang, Alex [mailto:Alex.Chang at idt.com] Sent: Wednesday, August 15, 2012 10:21 AM To: Luse, Paul E; nvmewin at lists.openfabrics.org Subject: RE: patch review status Absolutely. Now, the tricky part is when CC.EN=1 and STS.RDY never becomes 1. I'd say simply check STS.RDY only. If it's 1, then reset the device, otherwise, skip resetting it. What do you think? Thanks, Alex ________________________________ From: Luse, Paul E [mailto:paul.e.luse at intel.com] Sent: Wednesday, August 15, 2012 9:56 AM To: Chang, Alex; nvmewin at lists.openfabrics.org Subject: RE: patch review status OK, I see what you're saying I think. We don't want to check for RDY without first looking at EN so does this make sense: if (CC.EN == 1) { -- now wait for RDY - /* Now reset */ CC.EN = 0; StorPortWriteRegisterUlong(pAE, (PULONG)(&pAE->pCtrlRegister->CC), CC.AsUlong); } From: Chang, Alex [mailto:Alex.Chang at idt.com] Sent: Wednesday, August 15, 2012 9:52 AM To: Luse, Paul E; nvmewin at lists.openfabrics.org Subject: RE: patch review status Hi Paul, I agree with the new logic on #3. However, instead of checking CC.EN, should we check STS.RDY or both? Thanks, Alex ________________________________ From: Luse, Paul E [mailto:paul.e.luse at intel.com] Sent: Wednesday, August 15, 2012 9:03 AM To: Chang, Alex; nvmewin at lists.openfabrics.org Subject: RE: patch review status Thanks Alex, see below. From: Chang, Alex [mailto:Alex.Chang at idt.com] Sent: Wednesday, August 15, 2012 8:51 AM To: Luse, Paul E; nvmewin at lists.openfabrics.org Subject: RE: patch review status Hi Paul, Some questions after browsing the difference between current source and your changes: 1. Why you removed "CC.SHN = 1;" in NVMeNormalShutdown routine? PL> Good eye, that was a mistake. I'll add it back before pushing 2. In nvme.h, why you changed the opcode of NVM_DATASET_MANAGEMENT from 0x06 to 0x09? PL> 6 was wrong, 9 is correct per spec 3. Is there any specific reason(s), you added the following in NVMeInitialize, and then immediately clear EN without checking RDY = 1? /* Set EN to 1 (all others zero) */ CC.EN = 1; StorPortWriteRegisterUlong(pAE, (PULONG)(&pAE->pCtrlRegister->CC), CC.AsUlong); PL> I wrote this up in the patch submission and I can't find that email right now. I didn't add code, I changed code. Before, we use to set EN to 1, read it back, and then set it to 0 to kick off a reset and then later we wait for RDY in the init state machine. We had an IHV complain that their implementation required waiting on RDY after setting EN 1 to even before clearing EN. So, our old implementation was a problem. So what I did now is to not force a reset unless EN is set to 1 when we come in. If its set to 0 then either (a) we're coming from power on or (b) the pre_OS reset the card when it was done. Either way if its already 0 then we're in a known state so we do nothing and then later we set to 1 and wait on RDY. If its 1 when we enter here we set to 0 and then later set to 1 and wait on RDY. Make sense? This assures we never enable without waiting for RDY ***** WAS ******* /* Set EN to 1 (all others zero) */ CC.EN = 1; StorPortWriteRegisterUlong(pAE, (PULONG)(&pAE->pCtrlRegister->CC), CC.AsUlong); /* Read it back */ CC.AsUlong = StorPortReadRegisterUlong(pAE, (PULONG)(&pAE->pCtrlRegister->CC)); /* Now reset */ CC.EN = 0; StorPortWriteRegisterUlong(pAE, (PULONG)(&pAE->pCtrlRegister->CC), CC.AsUlong); CC.AsUlong = StorPortReadRegisterUlong(pAE, (PULONG)(&pAE->pCtrlRegister->CC)); *** NOW IS ******* if (CC.EN == 1) { /* Now reset */ CC.EN = 0; StorPortWriteRegisterUlong(pAE, (PULONG)(&pAE->pCtrlRegister->CC), CC.AsUlong); } Thanks, Alex ________________________________ From: nvmewin-bounces at lists.openfabrics.org [mailto:nvmewin-bounces at lists.openfabrics.org] On Behalf Of Luse, Paul E Sent: Wednesday, August 15, 2012 8:12 AM To: nvmewin at lists.openfabrics.org Subject: [nvmewin] patch review status Just checking in... any questions to date and how are folks tracking for completing their review by this Fri? Thx Paul ____________________________________ Paul Luse Sr. Staff Engineer PCG Server Software Engineering Desk: 480.554.3688, Mobile: 480.334.4630 -------------- next part -------------- An HTML attachment was scrubbed... URL: From Alex.Chang at idt.com Wed Aug 15 10:47:18 2012 From: Alex.Chang at idt.com (Chang, Alex) Date: Wed, 15 Aug 2012 17:47:18 +0000 Subject: [nvmewin] patch review status In-Reply-To: <82C9F782B054C94B9FC04A331649C77A07B2B0C9@FMSMSX106.amr.corp.intel.com> References: <82C9F782B054C94B9FC04A331649C77A07B2AECD@FMSMSX106.amr.corp.intel.com> <548C5470AAD9DA4A85D259B663190D36038760@corpmail1.na.ads.idt.com> <82C9F782B054C94B9FC04A331649C77A07B2AF6F@FMSMSX106.amr.corp.intel.com> <548C5470AAD9DA4A85D259B663190D3603878D@corpmail1.na.ads.idt.com> <82C9F782B054C94B9FC04A331649C77A07B2B051@FMSMSX106.amr.corp.intel.com> <548C5470AAD9DA4A85D259B663190D360397B2@corpmail1.na.ads.idt.com> <82C9F782B054C94B9FC04A331649C77A07B2B0C9@FMSMSX106.amr.corp.intel.com> Message-ID: <548C5470AAD9DA4A85D259B663190D360397E1@corpmail1.na.ads.idt.com> Thanks, Paul. Alex ________________________________ From: Luse, Paul E [mailto:paul.e.luse at intel.com] Sent: Wednesday, August 15, 2012 10:43 AM To: Chang, Alex; nvmewin at lists.openfabrics.org Subject: RE: patch review status OK, I think I can get behind that. I was thinking STS might not be valid when EN is 0 but looking at the spec I think this is fine Ready (RDY): This field is set to '1' when the controller is ready to process commands after CC.EN is set to '1'. This field shall be cleared to '0' when CC.EN is cleared to '0'. Commands shall not be issued to the controller until this field is set to '1' after the CC.EN bit is set to '1'. Failure to follow this requirement produces undefined results. Host software shall wait a minimum of CAP.TO seconds for this field to be set to '1' after setting CC.EN to '1' from a previous value of '0'. So I'll just chance the check below for EN == 1 to RDY == 1 then Thanks! Paul From: Chang, Alex [mailto:Alex.Chang at idt.com] Sent: Wednesday, August 15, 2012 10:21 AM To: Luse, Paul E; nvmewin at lists.openfabrics.org Subject: RE: patch review status Absolutely. Now, the tricky part is when CC.EN=1 and STS.RDY never becomes 1. I'd say simply check STS.RDY only. If it's 1, then reset the device, otherwise, skip resetting it. What do you think? Thanks, Alex ________________________________ From: Luse, Paul E [mailto:paul.e.luse at intel.com] Sent: Wednesday, August 15, 2012 9:56 AM To: Chang, Alex; nvmewin at lists.openfabrics.org Subject: RE: patch review status OK, I see what you're saying I think. We don't want to check for RDY without first looking at EN so does this make sense: if (CC.EN == 1) { -- now wait for RDY - /* Now reset */ CC.EN = 0; StorPortWriteRegisterUlong(pAE, (PULONG)(&pAE->pCtrlRegister->CC), CC.AsUlong); } From: Chang, Alex [mailto:Alex.Chang at idt.com] Sent: Wednesday, August 15, 2012 9:52 AM To: Luse, Paul E; nvmewin at lists.openfabrics.org Subject: RE: patch review status Hi Paul, I agree with the new logic on #3. However, instead of checking CC.EN, should we check STS.RDY or both? Thanks, Alex ________________________________ From: Luse, Paul E [mailto:paul.e.luse at intel.com] Sent: Wednesday, August 15, 2012 9:03 AM To: Chang, Alex; nvmewin at lists.openfabrics.org Subject: RE: patch review status Thanks Alex, see below. From: Chang, Alex [mailto:Alex.Chang at idt.com] Sent: Wednesday, August 15, 2012 8:51 AM To: Luse, Paul E; nvmewin at lists.openfabrics.org Subject: RE: patch review status Hi Paul, Some questions after browsing the difference between current source and your changes: 1. Why you removed "CC.SHN = 1;" in NVMeNormalShutdown routine? PL> Good eye, that was a mistake. I'll add it back before pushing 2. In nvme.h, why you changed the opcode of NVM_DATASET_MANAGEMENT from 0x06 to 0x09? PL> 6 was wrong, 9 is correct per spec 3. Is there any specific reason(s), you added the following in NVMeInitialize, and then immediately clear EN without checking RDY = 1? /* Set EN to 1 (all others zero) */ CC.EN = 1; StorPortWriteRegisterUlong(pAE, (PULONG)(&pAE->pCtrlRegister->CC), CC.AsUlong); PL> I wrote this up in the patch submission and I can't find that email right now. I didn't add code, I changed code. Before, we use to set EN to 1, read it back, and then set it to 0 to kick off a reset and then later we wait for RDY in the init state machine. We had an IHV complain that their implementation required waiting on RDY after setting EN 1 to even before clearing EN. So, our old implementation was a problem. So what I did now is to not force a reset unless EN is set to 1 when we come in. If its set to 0 then either (a) we're coming from power on or (b) the pre_OS reset the card when it was done. Either way if its already 0 then we're in a known state so we do nothing and then later we set to 1 and wait on RDY. If its 1 when we enter here we set to 0 and then later set to 1 and wait on RDY. Make sense? This assures we never enable without waiting for RDY ***** WAS ******* /* Set EN to 1 (all others zero) */ CC.EN = 1; StorPortWriteRegisterUlong(pAE, (PULONG)(&pAE->pCtrlRegister->CC), CC.AsUlong); /* Read it back */ CC.AsUlong = StorPortReadRegisterUlong(pAE, (PULONG)(&pAE->pCtrlRegister->CC)); /* Now reset */ CC.EN = 0; StorPortWriteRegisterUlong(pAE, (PULONG)(&pAE->pCtrlRegister->CC), CC.AsUlong); CC.AsUlong = StorPortReadRegisterUlong(pAE, (PULONG)(&pAE->pCtrlRegister->CC)); *** NOW IS ******* if (CC.EN == 1) { /* Now reset */ CC.EN = 0; StorPortWriteRegisterUlong(pAE, (PULONG)(&pAE->pCtrlRegister->CC), CC.AsUlong); } Thanks, Alex ________________________________ From: nvmewin-bounces at lists.openfabrics.org [mailto:nvmewin-bounces at lists.openfabrics.org] On Behalf Of Luse, Paul E Sent: Wednesday, August 15, 2012 8:12 AM To: nvmewin at lists.openfabrics.org Subject: [nvmewin] patch review status Just checking in... any questions to date and how are folks tracking for completing their review by this Fri? Thx Paul ____________________________________ Paul Luse Sr. Staff Engineer PCG Server Software Engineering Desk: 480.554.3688, Mobile: 480.334.4630 -------------- next part -------------- An HTML attachment was scrubbed... URL: From paul.e.luse at intel.com Wed Aug 15 10:49:57 2012 From: paul.e.luse at intel.com (Luse, Paul E) Date: Wed, 15 Aug 2012 17:49:57 +0000 Subject: [nvmewin] patch review status In-Reply-To: <548C5470AAD9DA4A85D259B663190D360397E1@corpmail1.na.ads.idt.com> References: <82C9F782B054C94B9FC04A331649C77A07B2AECD@FMSMSX106.amr.corp.intel.com> <548C5470AAD9DA4A85D259B663190D36038760@corpmail1.na.ads.idt.com> <82C9F782B054C94B9FC04A331649C77A07B2AF6F@FMSMSX106.amr.corp.intel.com> <548C5470AAD9DA4A85D259B663190D3603878D@corpmail1.na.ads.idt.com> <82C9F782B054C94B9FC04A331649C77A07B2B051@FMSMSX106.amr.corp.intel.com> <548C5470AAD9DA4A85D259B663190D360397B2@corpmail1.na.ads.idt.com> <82C9F782B054C94B9FC04A331649C77A07B2B0C9@FMSMSX106.amr.corp.intel.com> <548C5470AAD9DA4A85D259B663190D360397E1@corpmail1.na.ads.idt.com> Message-ID: <82C9F782B054C94B9FC04A331649C77A07B2B104@FMSMSX106.amr.corp.intel.com> So I heard from Rick and Alex I assume you're OK with the 2 small changes below. I'll go ahead and make the changes and do some regression and get this to Ray for pushing later this week. Next up... Alex's sets of changes Thanks Everyone Paul From: Chang, Alex [mailto:Alex.Chang at idt.com] Sent: Wednesday, August 15, 2012 10:47 AM To: Luse, Paul E; nvmewin at lists.openfabrics.org Subject: RE: patch review status Thanks, Paul. Alex ________________________________ From: Luse, Paul E [mailto:paul.e.luse at intel.com] Sent: Wednesday, August 15, 2012 10:43 AM To: Chang, Alex; nvmewin at lists.openfabrics.org Subject: RE: patch review status OK, I think I can get behind that. I was thinking STS might not be valid when EN is 0 but looking at the spec I think this is fine Ready (RDY): This field is set to '1' when the controller is ready to process commands after CC.EN is set to '1'. This field shall be cleared to '0' when CC.EN is cleared to '0'. Commands shall not be issued to the controller until this field is set to '1' after the CC.EN bit is set to '1'. Failure to follow this requirement produces undefined results. Host software shall wait a minimum of CAP.TO seconds for this field to be set to '1' after setting CC.EN to '1' from a previous value of '0'. So I'll just chance the check below for EN == 1 to RDY == 1 then Thanks! Paul From: Chang, Alex [mailto:Alex.Chang at idt.com] Sent: Wednesday, August 15, 2012 10:21 AM To: Luse, Paul E; nvmewin at lists.openfabrics.org Subject: RE: patch review status Absolutely. Now, the tricky part is when CC.EN=1 and STS.RDY never becomes 1. I'd say simply check STS.RDY only. If it's 1, then reset the device, otherwise, skip resetting it. What do you think? Thanks, Alex ________________________________ From: Luse, Paul E [mailto:paul.e.luse at intel.com] Sent: Wednesday, August 15, 2012 9:56 AM To: Chang, Alex; nvmewin at lists.openfabrics.org Subject: RE: patch review status OK, I see what you're saying I think. We don't want to check for RDY without first looking at EN so does this make sense: if (CC.EN == 1) { -- now wait for RDY - /* Now reset */ CC.EN = 0; StorPortWriteRegisterUlong(pAE, (PULONG)(&pAE->pCtrlRegister->CC), CC.AsUlong); } From: Chang, Alex [mailto:Alex.Chang at idt.com] Sent: Wednesday, August 15, 2012 9:52 AM To: Luse, Paul E; nvmewin at lists.openfabrics.org Subject: RE: patch review status Hi Paul, I agree with the new logic on #3. However, instead of checking CC.EN, should we check STS.RDY or both? Thanks, Alex ________________________________ From: Luse, Paul E [mailto:paul.e.luse at intel.com] Sent: Wednesday, August 15, 2012 9:03 AM To: Chang, Alex; nvmewin at lists.openfabrics.org Subject: RE: patch review status Thanks Alex, see below. From: Chang, Alex [mailto:Alex.Chang at idt.com] Sent: Wednesday, August 15, 2012 8:51 AM To: Luse, Paul E; nvmewin at lists.openfabrics.org Subject: RE: patch review status Hi Paul, Some questions after browsing the difference between current source and your changes: 1. Why you removed "CC.SHN = 1;" in NVMeNormalShutdown routine? PL> Good eye, that was a mistake. I'll add it back before pushing 2. In nvme.h, why you changed the opcode of NVM_DATASET_MANAGEMENT from 0x06 to 0x09? PL> 6 was wrong, 9 is correct per spec 3. Is there any specific reason(s), you added the following in NVMeInitialize, and then immediately clear EN without checking RDY = 1? /* Set EN to 1 (all others zero) */ CC.EN = 1; StorPortWriteRegisterUlong(pAE, (PULONG)(&pAE->pCtrlRegister->CC), CC.AsUlong); PL> I wrote this up in the patch submission and I can't find that email right now. I didn't add code, I changed code. Before, we use to set EN to 1, read it back, and then set it to 0 to kick off a reset and then later we wait for RDY in the init state machine. We had an IHV complain that their implementation required waiting on RDY after setting EN 1 to even before clearing EN. So, our old implementation was a problem. So what I did now is to not force a reset unless EN is set to 1 when we come in. If its set to 0 then either (a) we're coming from power on or (b) the pre_OS reset the card when it was done. Either way if its already 0 then we're in a known state so we do nothing and then later we set to 1 and wait on RDY. If its 1 when we enter here we set to 0 and then later set to 1 and wait on RDY. Make sense? This assures we never enable without waiting for RDY ***** WAS ******* /* Set EN to 1 (all others zero) */ CC.EN = 1; StorPortWriteRegisterUlong(pAE, (PULONG)(&pAE->pCtrlRegister->CC), CC.AsUlong); /* Read it back */ CC.AsUlong = StorPortReadRegisterUlong(pAE, (PULONG)(&pAE->pCtrlRegister->CC)); /* Now reset */ CC.EN = 0; StorPortWriteRegisterUlong(pAE, (PULONG)(&pAE->pCtrlRegister->CC), CC.AsUlong); CC.AsUlong = StorPortReadRegisterUlong(pAE, (PULONG)(&pAE->pCtrlRegister->CC)); *** NOW IS ******* if (CC.EN == 1) { /* Now reset */ CC.EN = 0; StorPortWriteRegisterUlong(pAE, (PULONG)(&pAE->pCtrlRegister->CC), CC.AsUlong); } Thanks, Alex ________________________________ From: nvmewin-bounces at lists.openfabrics.org [mailto:nvmewin-bounces at lists.openfabrics.org] On Behalf Of Luse, Paul E Sent: Wednesday, August 15, 2012 8:12 AM To: nvmewin at lists.openfabrics.org Subject: [nvmewin] patch review status Just checking in... any questions to date and how are folks tracking for completing their review by this Fri? Thx Paul ____________________________________ Paul Luse Sr. Staff Engineer PCG Server Software Engineering Desk: 480.554.3688, Mobile: 480.334.4630 -------------- next part -------------- An HTML attachment was scrubbed... URL: From raymond.c.robles at intel.com Wed Aug 15 11:09:12 2012 From: raymond.c.robles at intel.com (Robles, Raymond C) Date: Wed, 15 Aug 2012 18:09:12 +0000 Subject: [nvmewin] patch review status In-Reply-To: <82C9F782B054C94B9FC04A331649C77A07B2B104@FMSMSX106.amr.corp.intel.com> References: <82C9F782B054C94B9FC04A331649C77A07B2AECD@FMSMSX106.amr.corp.intel.com> <548C5470AAD9DA4A85D259B663190D36038760@corpmail1.na.ads.idt.com> <82C9F782B054C94B9FC04A331649C77A07B2AF6F@FMSMSX106.amr.corp.intel.com> <548C5470AAD9DA4A85D259B663190D3603878D@corpmail1.na.ads.idt.com> <82C9F782B054C94B9FC04A331649C77A07B2B051@FMSMSX106.amr.corp.intel.com> <548C5470AAD9DA4A85D259B663190D360397B2@corpmail1.na.ads.idt.com> <82C9F782B054C94B9FC04A331649C77A07B2B0C9@FMSMSX106.amr.corp.intel.com> <548C5470AAD9DA4A85D259B663190D360397E1@corpmail1.na.ads.idt.com> <82C9F782B054C94B9FC04A331649C77A07B2B104@FMSMSX106.amr.corp.intel.com> Message-ID: <49158E750348AA499168FD41D88983601805FC5C@FMSMSX105.amr.corp.intel.com> Sounds good. Alex, please wait for the push of Paul's changes before you send out your patch review request so that you can rebase his changes. Thanks, Ray From: nvmewin-bounces at lists.openfabrics.org [mailto:nvmewin-bounces at lists.openfabrics.org] On Behalf Of Luse, Paul E Sent: Wednesday, August 15, 2012 10:50 AM To: Chang, Alex; nvmewin at lists.openfabrics.org Subject: Re: [nvmewin] patch review status So I heard from Rick and Alex I assume you're OK with the 2 small changes below. I'll go ahead and make the changes and do some regression and get this to Ray for pushing later this week. Next up... Alex's sets of changes Thanks Everyone Paul From: Chang, Alex [mailto:Alex.Chang at idt.com] Sent: Wednesday, August 15, 2012 10:47 AM To: Luse, Paul E; nvmewin at lists.openfabrics.org Subject: RE: patch review status Thanks, Paul. Alex ________________________________ From: Luse, Paul E [mailto:paul.e.luse at intel.com] Sent: Wednesday, August 15, 2012 10:43 AM To: Chang, Alex; nvmewin at lists.openfabrics.org Subject: RE: patch review status OK, I think I can get behind that. I was thinking STS might not be valid when EN is 0 but looking at the spec I think this is fine Ready (RDY): This field is set to '1' when the controller is ready to process commands after CC.EN is set to '1'. This field shall be cleared to '0' when CC.EN is cleared to '0'. Commands shall not be issued to the controller until this field is set to '1' after the CC.EN bit is set to '1'. Failure to follow this requirement produces undefined results. Host software shall wait a minimum of CAP.TO seconds for this field to be set to '1' after setting CC.EN to '1' from a previous value of '0'. So I'll just chance the check below for EN == 1 to RDY == 1 then Thanks! Paul From: Chang, Alex [mailto:Alex.Chang at idt.com] Sent: Wednesday, August 15, 2012 10:21 AM To: Luse, Paul E; nvmewin at lists.openfabrics.org Subject: RE: patch review status Absolutely. Now, the tricky part is when CC.EN=1 and STS.RDY never becomes 1. I'd say simply check STS.RDY only. If it's 1, then reset the device, otherwise, skip resetting it. What do you think? Thanks, Alex ________________________________ From: Luse, Paul E [mailto:paul.e.luse at intel.com] Sent: Wednesday, August 15, 2012 9:56 AM To: Chang, Alex; nvmewin at lists.openfabrics.org Subject: RE: patch review status OK, I see what you're saying I think. We don't want to check for RDY without first looking at EN so does this make sense: if (CC.EN == 1) { -- now wait for RDY - /* Now reset */ CC.EN = 0; StorPortWriteRegisterUlong(pAE, (PULONG)(&pAE->pCtrlRegister->CC), CC.AsUlong); } From: Chang, Alex [mailto:Alex.Chang at idt.com] Sent: Wednesday, August 15, 2012 9:52 AM To: Luse, Paul E; nvmewin at lists.openfabrics.org Subject: RE: patch review status Hi Paul, I agree with the new logic on #3. However, instead of checking CC.EN, should we check STS.RDY or both? Thanks, Alex ________________________________ From: Luse, Paul E [mailto:paul.e.luse at intel.com] Sent: Wednesday, August 15, 2012 9:03 AM To: Chang, Alex; nvmewin at lists.openfabrics.org Subject: RE: patch review status Thanks Alex, see below. From: Chang, Alex [mailto:Alex.Chang at idt.com] Sent: Wednesday, August 15, 2012 8:51 AM To: Luse, Paul E; nvmewin at lists.openfabrics.org Subject: RE: patch review status Hi Paul, Some questions after browsing the difference between current source and your changes: 1. Why you removed "CC.SHN = 1;" in NVMeNormalShutdown routine? PL> Good eye, that was a mistake. I'll add it back before pushing 2. In nvme.h, why you changed the opcode of NVM_DATASET_MANAGEMENT from 0x06 to 0x09? PL> 6 was wrong, 9 is correct per spec 3. Is there any specific reason(s), you added the following in NVMeInitialize, and then immediately clear EN without checking RDY = 1? /* Set EN to 1 (all others zero) */ CC.EN = 1; StorPortWriteRegisterUlong(pAE, (PULONG)(&pAE->pCtrlRegister->CC), CC.AsUlong); PL> I wrote this up in the patch submission and I can't find that email right now. I didn't add code, I changed code. Before, we use to set EN to 1, read it back, and then set it to 0 to kick off a reset and then later we wait for RDY in the init state machine. We had an IHV complain that their implementation required waiting on RDY after setting EN 1 to even before clearing EN. So, our old implementation was a problem. So what I did now is to not force a reset unless EN is set to 1 when we come in. If its set to 0 then either (a) we're coming from power on or (b) the pre_OS reset the card when it was done. Either way if its already 0 then we're in a known state so we do nothing and then later we set to 1 and wait on RDY. If its 1 when we enter here we set to 0 and then later set to 1 and wait on RDY. Make sense? This assures we never enable without waiting for RDY ***** WAS ******* /* Set EN to 1 (all others zero) */ CC.EN = 1; StorPortWriteRegisterUlong(pAE, (PULONG)(&pAE->pCtrlRegister->CC), CC.AsUlong); /* Read it back */ CC.AsUlong = StorPortReadRegisterUlong(pAE, (PULONG)(&pAE->pCtrlRegister->CC)); /* Now reset */ CC.EN = 0; StorPortWriteRegisterUlong(pAE, (PULONG)(&pAE->pCtrlRegister->CC), CC.AsUlong); CC.AsUlong = StorPortReadRegisterUlong(pAE, (PULONG)(&pAE->pCtrlRegister->CC)); *** NOW IS ******* if (CC.EN == 1) { /* Now reset */ CC.EN = 0; StorPortWriteRegisterUlong(pAE, (PULONG)(&pAE->pCtrlRegister->CC), CC.AsUlong); } Thanks, Alex ________________________________ From: nvmewin-bounces at lists.openfabrics.org [mailto:nvmewin-bounces at lists.openfabrics.org] On Behalf Of Luse, Paul E Sent: Wednesday, August 15, 2012 8:12 AM To: nvmewin at lists.openfabrics.org Subject: [nvmewin] patch review status Just checking in... any questions to date and how are folks tracking for completing their review by this Fri? Thx Paul ____________________________________ Paul Luse Sr. Staff Engineer PCG Server Software Engineering Desk: 480.554.3688, Mobile: 480.334.4630 -------------- next part -------------- An HTML attachment was scrubbed... URL: From Alex.Chang at idt.com Wed Aug 15 11:23:45 2012 From: Alex.Chang at idt.com (Chang, Alex) Date: Wed, 15 Aug 2012 18:23:45 +0000 Subject: [nvmewin] patch review status In-Reply-To: <49158E750348AA499168FD41D88983601805FC5C@FMSMSX105.amr.corp.intel.com> References: <82C9F782B054C94B9FC04A331649C77A07B2AECD@FMSMSX106.amr.corp.intel.com> <548C5470AAD9DA4A85D259B663190D36038760@corpmail1.na.ads.idt.com> <82C9F782B054C94B9FC04A331649C77A07B2AF6F@FMSMSX106.amr.corp.intel.com> <548C5470AAD9DA4A85D259B663190D3603878D@corpmail1.na.ads.idt.com> <82C9F782B054C94B9FC04A331649C77A07B2B051@FMSMSX106.amr.corp.intel.com> <548C5470AAD9DA4A85D259B663190D360397B2@corpmail1.na.ads.idt.com> <82C9F782B054C94B9FC04A331649C77A07B2B0C9@FMSMSX106.amr.corp.intel.com> <548C5470AAD9DA4A85D259B663190D360397E1@corpmail1.na.ads.idt.com> <82C9F782B054C94B9FC04A331649C77A07B2B104@FMSMSX106.amr.corp.intel.com> <49158E750348AA499168FD41D88983601805FC5C@FMSMSX105.amr.corp.intel.com> Message-ID: <548C5470AAD9DA4A85D259B663190D36039810@corpmail1.na.ads.idt.com> Sure. I am still working on it. Thanks, Alex ________________________________ From: nvmewin-bounces at lists.openfabrics.org [mailto:nvmewin-bounces at lists.openfabrics.org] On Behalf Of Robles, Raymond C Sent: Wednesday, August 15, 2012 11:09 AM To: nvmewin at lists.openfabrics.org Subject: Re: [nvmewin] patch review status Sounds good. Alex, please wait for the push of Paul's changes before you send out your patch review request so that you can rebase his changes. Thanks, Ray From: nvmewin-bounces at lists.openfabrics.org [mailto:nvmewin-bounces at lists.openfabrics.org] On Behalf Of Luse, Paul E Sent: Wednesday, August 15, 2012 10:50 AM To: Chang, Alex; nvmewin at lists.openfabrics.org Subject: Re: [nvmewin] patch review status So I heard from Rick and Alex I assume you're OK with the 2 small changes below. I'll go ahead and make the changes and do some regression and get this to Ray for pushing later this week. Next up... Alex's sets of changes Thanks Everyone Paul From: Chang, Alex [mailto:Alex.Chang at idt.com] Sent: Wednesday, August 15, 2012 10:47 AM To: Luse, Paul E; nvmewin at lists.openfabrics.org Subject: RE: patch review status Thanks, Paul. Alex ________________________________ From: Luse, Paul E [mailto:paul.e.luse at intel.com] Sent: Wednesday, August 15, 2012 10:43 AM To: Chang, Alex; nvmewin at lists.openfabrics.org Subject: RE: patch review status OK, I think I can get behind that. I was thinking STS might not be valid when EN is 0 but looking at the spec I think this is fine Ready (RDY): This field is set to '1' when the controller is ready to process commands after CC.EN is set to '1'. This field shall be cleared to '0' when CC.EN is cleared to '0'. Commands shall not be issued to the controller until this field is set to '1' after the CC.EN bit is set to '1'. Failure to follow this requirement produces undefined results. Host software shall wait a minimum of CAP.TO seconds for this field to be set to '1' after setting CC.EN to '1' from a previous value of '0'. So I'll just chance the check below for EN == 1 to RDY == 1 then Thanks! Paul From: Chang, Alex [mailto:Alex.Chang at idt.com] Sent: Wednesday, August 15, 2012 10:21 AM To: Luse, Paul E; nvmewin at lists.openfabrics.org Subject: RE: patch review status Absolutely. Now, the tricky part is when CC.EN=1 and STS.RDY never becomes 1. I'd say simply check STS.RDY only. If it's 1, then reset the device, otherwise, skip resetting it. What do you think? Thanks, Alex ________________________________ From: Luse, Paul E [mailto:paul.e.luse at intel.com] Sent: Wednesday, August 15, 2012 9:56 AM To: Chang, Alex; nvmewin at lists.openfabrics.org Subject: RE: patch review status OK, I see what you're saying I think. We don't want to check for RDY without first looking at EN so does this make sense: if (CC.EN == 1) { -- now wait for RDY - /* Now reset */ CC.EN = 0; StorPortWriteRegisterUlong(pAE, (PULONG)(&pAE->pCtrlRegister->CC), CC.AsUlong); } From: Chang, Alex [mailto:Alex.Chang at idt.com] Sent: Wednesday, August 15, 2012 9:52 AM To: Luse, Paul E; nvmewin at lists.openfabrics.org Subject: RE: patch review status Hi Paul, I agree with the new logic on #3. However, instead of checking CC.EN, should we check STS.RDY or both? Thanks, Alex ________________________________ From: Luse, Paul E [mailto:paul.e.luse at intel.com] Sent: Wednesday, August 15, 2012 9:03 AM To: Chang, Alex; nvmewin at lists.openfabrics.org Subject: RE: patch review status Thanks Alex, see below. From: Chang, Alex [mailto:Alex.Chang at idt.com] Sent: Wednesday, August 15, 2012 8:51 AM To: Luse, Paul E; nvmewin at lists.openfabrics.org Subject: RE: patch review status Hi Paul, Some questions after browsing the difference between current source and your changes: 1. Why you removed "CC.SHN = 1;" in NVMeNormalShutdown routine? PL> Good eye, that was a mistake. I'll add it back before pushing 2. In nvme.h, why you changed the opcode of NVM_DATASET_MANAGEMENT from 0x06 to 0x09? PL> 6 was wrong, 9 is correct per spec 3. Is there any specific reason(s), you added the following in NVMeInitialize, and then immediately clear EN without checking RDY = 1? /* Set EN to 1 (all others zero) */ CC.EN = 1; StorPortWriteRegisterUlong(pAE, (PULONG)(&pAE->pCtrlRegister->CC), CC.AsUlong); PL> I wrote this up in the patch submission and I can't find that email right now. I didn't add code, I changed code. Before, we use to set EN to 1, read it back, and then set it to 0 to kick off a reset and then later we wait for RDY in the init state machine. We had an IHV complain that their implementation required waiting on RDY after setting EN 1 to even before clearing EN. So, our old implementation was a problem. So what I did now is to not force a reset unless EN is set to 1 when we come in. If its set to 0 then either (a) we're coming from power on or (b) the pre_OS reset the card when it was done. Either way if its already 0 then we're in a known state so we do nothing and then later we set to 1 and wait on RDY. If its 1 when we enter here we set to 0 and then later set to 1 and wait on RDY. Make sense? This assures we never enable without waiting for RDY ***** WAS ******* /* Set EN to 1 (all others zero) */ CC.EN = 1; StorPortWriteRegisterUlong(pAE, (PULONG)(&pAE->pCtrlRegister->CC), CC.AsUlong); /* Read it back */ CC.AsUlong = StorPortReadRegisterUlong(pAE, (PULONG)(&pAE->pCtrlRegister->CC)); /* Now reset */ CC.EN = 0; StorPortWriteRegisterUlong(pAE, (PULONG)(&pAE->pCtrlRegister->CC), CC.AsUlong); CC.AsUlong = StorPortReadRegisterUlong(pAE, (PULONG)(&pAE->pCtrlRegister->CC)); *** NOW IS ******* if (CC.EN == 1) { /* Now reset */ CC.EN = 0; StorPortWriteRegisterUlong(pAE, (PULONG)(&pAE->pCtrlRegister->CC), CC.AsUlong); } Thanks, Alex ________________________________ From: nvmewin-bounces at lists.openfabrics.org [mailto:nvmewin-bounces at lists.openfabrics.org] On Behalf Of Luse, Paul E Sent: Wednesday, August 15, 2012 8:12 AM To: nvmewin at lists.openfabrics.org Subject: [nvmewin] patch review status Just checking in... any questions to date and how are folks tracking for completing their review by this Fri? Thx Paul ____________________________________ Paul Luse Sr. Staff Engineer PCG Server Software Engineering Desk: 480.554.3688, Mobile: 480.334.4630 -------------- next part -------------- An HTML attachment was scrubbed... URL: From james.p.freyensee at intel.com Wed Aug 15 13:11:47 2012 From: james.p.freyensee at intel.com (Freyensee, James P) Date: Wed, 15 Aug 2012 20:11:47 +0000 Subject: [nvmewin] Setting a Windows Debugging Environment In-Reply-To: References: Message-ID: <2D98093777D3FD46A36253F35FE9D69346BFD379@ORSMSX101.amr.corp.intel.com> I just used a 1394 connection and used Microsoft's MSDN pages to set mine up: http://msdn.microsoft.com/en-us/library/windows/hardware/ff556866%28v=vs.85%29.aspx http://msdn.microsoft.com/en-us/library/windows/hardware/hh698272%28v=vs.85%29.aspx I use VS 2012 and the latest WDK for VS 2012 because if you are targeting Win8, you need that VS version (per Microsoft). VS 2012 allows you to set up and debug drivers within the IDE, but it falls well short of whatever "oohs-and-aahs" advertised. For example, all your breakpoints are reset when you reset target device, unlike windbg which allows you to insert a breakpoint upon boot before rebooting the target machine. I even filed a bug against VS 2012 on this behavior and the VS team closed it, saying they don't own this (which I can't understand). So I don't know what benefit Microsoft thinks they have with an IDE that can't seem to remember breakpoints when debugging a driver (among other things I won't list), thus I continue to use windbg and don't recommend learning how to debug a driver with VS 2012. One gotcha with installing VS 2012- you need to uninstall all your WDKs off your computer first before installing VS 2012 then the WDK. There's a bug in VS 2012 that will royally break driver compiling if you don't. Good luck :). From: nvmewin-bounces at lists.openfabrics.org [mailto:nvmewin-bounces at lists.openfabrics.org] On Behalf Of Brenda Powers Sent: Wednesday, August 15, 2012 10:31 AM To: nvmewin at lists.openfabrics.org Subject: [nvmewin] Setting a Windows Debugging Environment I have been trying to get a Windows debugging environment working and have had no luck. I have tried using Visual Studio 11 and going through a USB, 1394 and RS232 connection but have had no success. I can get the remote system provisioned but cannot get a connection through either a USB, 1394 or RS232. I have tried using Windbg and have the same problem of failing to get a connection established. Can anyone give me the details of their debugging setup which I can duplicate? Brenda Powers Cadence -------------- next part -------------- An HTML attachment was scrubbed... URL: From vishal.l.verma at intel.com Wed Aug 15 13:46:04 2012 From: vishal.l.verma at intel.com (Verma, Vishal L) Date: Wed, 15 Aug 2012 20:46:04 +0000 Subject: [nvmewin] patch review status In-Reply-To: <82C9F782B054C94B9FC04A331649C77A07B2B0C9@FMSMSX106.amr.corp.intel.com> Message-ID: Paul, Is this correct though? Making CC.EN 0 implies that RDY will become 0, but RDY being 0 does not guarantee that CC.EN is 0. If what I'm thinking is right, we could be in a state where CC.EN was 1 when the driver comes up, but the controller never became ready. In this case we will think everything is alright, and we will try to make CC.EN 1 again – Which is not something we'd want to do, right? Vishal From: , Paul E > Date: Wednesday, August 15, 2012 11:43 AM To: "Chang, Alex" >, "nvmewin at lists.openfabrics.org" > Subject: Re: [nvmewin] patch review status OK, I think I can get behind that. I was thinking STS might not be valid when EN is 0 but looking at the spec I think this is fine Ready (RDY): This field is set to ‘1’ when the controller is ready to process commands after CC.EN is set to ‘1’. This field shall be cleared to ‘0’ when CC.EN is cleared to ‘0’. Commands shall not be issued to the controller until this field is set to ‘1’ after the CC.EN bit is set to ‘1’. Failure to follow this requirement produces undefined results. Host software shall wait a minimum of CAP.TO seconds for this field to be set to ‘1’ after setting CC.EN to ‘1’ from a previous value of ‘0’. So I’ll just chance the check below for EN == 1 to RDY == 1 then Thanks! Paul From: Chang, Alex [mailto:Alex.Chang at idt.com] Sent: Wednesday, August 15, 2012 10:21 AM To: Luse, Paul E; nvmewin at lists.openfabrics.org Subject: RE: patch review status Absolutely. Now, the tricky part is when CC.EN=1 and STS.RDY never becomes 1. I'd say simply check STS.RDY only. If it's 1, then reset the device, otherwise, skip resetting it. What do you think? Thanks, Alex ________________________________ From: Luse, Paul E [mailto:paul.e.luse at intel.com] Sent: Wednesday, August 15, 2012 9:56 AM To: Chang, Alex; nvmewin at lists.openfabrics.org Subject: RE: patch review status OK, I see what you’re saying I think. We don’t want to check for RDY without first looking at EN so does this make sense: if (CC.EN == 1) { -- now wait for RDY - /* Now reset */ CC.EN = 0; StorPortWriteRegisterUlong(pAE, (PULONG)(&pAE->pCtrlRegister->CC), CC.AsUlong); } From: Chang, Alex [mailto:Alex.Chang at idt.com] Sent: Wednesday, August 15, 2012 9:52 AM To: Luse, Paul E; nvmewin at lists.openfabrics.org Subject: RE: patch review status Hi Paul, I agree with the new logic on #3. However, instead of checking CC.EN, should we check STS.RDY or both? Thanks, Alex ________________________________ From: Luse, Paul E [mailto:paul.e.luse at intel.com] Sent: Wednesday, August 15, 2012 9:03 AM To: Chang, Alex; nvmewin at lists.openfabrics.org Subject: RE: patch review status Thanks Alex, see below. From: Chang, Alex [mailto:Alex.Chang at idt.com] Sent: Wednesday, August 15, 2012 8:51 AM To: Luse, Paul E; nvmewin at lists.openfabrics.org Subject: RE: patch review status Hi Paul, Some questions after browsing the difference between current source and your changes: 1. Why you removed "CC.SHN = 1;" in NVMeNormalShutdown routine? PL> Good eye, that was a mistake. I’ll add it back before pushing 2. In nvme.h, why you changed the opcode of NVM_DATASET_MANAGEMENT from 0x06 to 0x09? PL> 6 was wrong, 9 is correct per spec 3. Is there any specific reason(s), you added the following in NVMeInitialize, and then immediately clear EN without checking RDY = 1? /* Set EN to 1 (all others zero) */ CC.EN = 1; StorPortWriteRegisterUlong(pAE, (PULONG)(&pAE->pCtrlRegister->CC), CC.AsUlong); PL> I wrote this up in the patch submission and I can’t find that email right now. I didn’t add code, I changed code. Before, we use to set EN to 1, read it back, and then set it to 0 to kick off a reset and then later we wait for RDY in the init state machine. We had an IHV complain that their implementation required waiting on RDY after setting EN 1 to even before clearing EN. So, our old implementation was a problem. So what I did now is to not force a reset unless EN is set to 1 when we come in. If its set to 0 then either (a) we’re coming from power on or (b) the pre_OS reset the card when it was done. Either way if its already 0 then we’re in a known state so we do nothing and then later we set to 1 and wait on RDY. If its 1 when we enter here we set to 0 and then later set to 1 and wait on RDY. Make sense? This assures we never enable without waiting for RDY ***** WAS ******* /* Set EN to 1 (all others zero) */ CC.EN = 1; StorPortWriteRegisterUlong(pAE, (PULONG)(&pAE->pCtrlRegister->CC), CC.AsUlong); /* Read it back */ CC.AsUlong = StorPortReadRegisterUlong(pAE, (PULONG)(&pAE->pCtrlRegister->CC)); /* Now reset */ CC.EN = 0; StorPortWriteRegisterUlong(pAE, (PULONG)(&pAE->pCtrlRegister->CC), CC.AsUlong); CC.AsUlong = StorPortReadRegisterUlong(pAE, (PULONG)(&pAE->pCtrlRegister->CC)); *** NOW IS ******* if (CC.EN == 1) { /* Now reset */ CC.EN = 0; StorPortWriteRegisterUlong(pAE, (PULONG)(&pAE->pCtrlRegister->CC), CC.AsUlong); } Thanks, Alex ________________________________ From:nvmewin-bounces at lists.openfabrics.org[mailto:nvmewin-bounces at lists.openfabrics.org]On Behalf Of Luse, Paul E Sent: Wednesday, August 15, 2012 8:12 AM To: nvmewin at lists.openfabrics.org Subject: [nvmewin] patch review status Just checking in… any questions to date and how are folks tracking for completing their review by this Fri? Thx Paul ____________________________________ Paul Luse Sr. Staff Engineer PCG Server Software Engineering Desk: 480.554.3688, Mobile: 480.334.4630 From paul.e.luse at intel.com Wed Aug 15 15:20:55 2012 From: paul.e.luse at intel.com (Luse, Paul E) Date: Wed, 15 Aug 2012 22:20:55 +0000 Subject: [nvmewin] patch review status In-Reply-To: References: <82C9F782B054C94B9FC04A331649C77A07B2B0C9@FMSMSX106.amr.corp.intel.com> Message-ID: <82C9F782B054C94B9FC04A331649C77A07B2B907@FMSMSX106.amr.corp.intel.com> I was thinking about that too... If EN was already 1 and we caught RDY before it was 0 then we would indeed set EN to 1 again (possibly before, during or after RDY is set). The spec, of course, does not dictate behavior in this case but one would assume it would be treated as a NOP and our init state machine would properly wait for RDY. I'm not a HW guy though, anyone else have a comment/concern on setting EN to 1 if tis already 1? From: Verma, Vishal L Sent: Wednesday, August 15, 2012 1:46 PM To: Luse, Paul E; Chang, Alex; nvmewin at lists.openfabrics.org Subject: Re: [nvmewin] patch review status Paul, Is this correct though? Making CC.EN 0 implies that RDY will become 0, but RDY being 0 does not guarantee that CC.EN is 0. If what I'm thinking is right, we could be in a state where CC.EN was 1 when the driver comes up, but the controller never became ready. In this case we will think everything is alright, and we will try to make CC.EN 1 again - Which is not something we'd want to do, right? Vishal From: , Paul E > Date: Wednesday, August 15, 2012 11:43 AM To: "Chang, Alex" >, "nvmewin at lists.openfabrics.org" > Subject: Re: [nvmewin] patch review status OK, I think I can get behind that. I was thinking STS might not be valid when EN is 0 but looking at the spec I think this is fine Ready (RDY): This field is set to '1' when the controller is ready to process commands after CC.EN is set to '1'. This field shall be cleared to '0' when CC.EN is cleared to '0'. Commands shall not be issued to the controller until this field is set to '1' after the CC.EN bit is set to '1'. Failure to follow this requirement produces undefined results. Host software shall wait a minimum of CAP.TO seconds for this field to be set to '1' after setting CC.EN to '1' from a previous value of '0'. So I'll just chance the check below for EN == 1 to RDY == 1 then Thanks! Paul From: Chang, Alex [mailto:Alex.Chang at idt.com] Sent: Wednesday, August 15, 2012 10:21 AM To: Luse, Paul E; nvmewin at lists.openfabrics.org Subject: RE: patch review status Absolutely. Now, the tricky part is when CC.EN=1 and STS.RDY never becomes 1. I'd say simply check STS.RDY only. If it's 1, then reset the device, otherwise, skip resetting it. What do you think? Thanks, Alex ________________________________ From: Luse, Paul E [mailto:paul.e.luse at intel.com] Sent: Wednesday, August 15, 2012 9:56 AM To: Chang, Alex; nvmewin at lists.openfabrics.org Subject: RE: patch review status OK, I see what you're saying I think. We don't want to check for RDY without first looking at EN so does this make sense: if (CC.EN == 1) { -- now wait for RDY - /* Now reset */ CC.EN = 0; StorPortWriteRegisterUlong(pAE, (PULONG)(&pAE->pCtrlRegister->CC), CC.AsUlong); } From: Chang, Alex [mailto:Alex.Chang at idt.com] Sent: Wednesday, August 15, 2012 9:52 AM To: Luse, Paul E; nvmewin at lists.openfabrics.org Subject: RE: patch review status Hi Paul, I agree with the new logic on #3. However, instead of checking CC.EN, should we check STS.RDY or both? Thanks, Alex ________________________________ From: Luse, Paul E [mailto:paul.e.luse at intel.com] Sent: Wednesday, August 15, 2012 9:03 AM To: Chang, Alex; nvmewin at lists.openfabrics.org Subject: RE: patch review status Thanks Alex, see below. From: Chang, Alex [mailto:Alex.Chang at idt.com] Sent: Wednesday, August 15, 2012 8:51 AM To: Luse, Paul E; nvmewin at lists.openfabrics.org Subject: RE: patch review status Hi Paul, Some questions after browsing the difference between current source and your changes: 1. Why you removed "CC.SHN = 1;" in NVMeNormalShutdown routine? PL> Good eye, that was a mistake. I'll add it back before pushing 2. In nvme.h, why you changed the opcode of NVM_DATASET_MANAGEMENT from 0x06 to 0x09? PL> 6 was wrong, 9 is correct per spec 3. Is there any specific reason(s), you added the following in NVMeInitialize, and then immediately clear EN without checking RDY = 1? /* Set EN to 1 (all others zero) */ CC.EN = 1; StorPortWriteRegisterUlong(pAE, (PULONG)(&pAE->pCtrlRegister->CC), CC.AsUlong); PL> I wrote this up in the patch submission and I can't find that email right now. I didn't add code, I changed code. Before, we use to set EN to 1, read it back, and then set it to 0 to kick off a reset and then later we wait for RDY in the init state machine. We had an IHV complain that their implementation required waiting on RDY after setting EN 1 to even before clearing EN. So, our old implementation was a problem. So what I did now is to not force a reset unless EN is set to 1 when we come in. If its set to 0 then either (a) we're coming from power on or (b) the pre_OS reset the card when it was done. Either way if its already 0 then we're in a known state so we do nothing and then later we set to 1 and wait on RDY. If its 1 when we enter here we set to 0 and then later set to 1 and wait on RDY. Make sense? This assures we never enable without waiting for RDY ***** WAS ******* /* Set EN to 1 (all others zero) */ CC.EN = 1; StorPortWriteRegisterUlong(pAE, (PULONG)(&pAE->pCtrlRegister->CC), CC.AsUlong); /* Read it back */ CC.AsUlong = StorPortReadRegisterUlong(pAE, (PULONG)(&pAE->pCtrlRegister->CC)); /* Now reset */ CC.EN = 0; StorPortWriteRegisterUlong(pAE, (PULONG)(&pAE->pCtrlRegister->CC), CC.AsUlong); CC.AsUlong = StorPortReadRegisterUlong(pAE, (PULONG)(&pAE->pCtrlRegister->CC)); *** NOW IS ******* if (CC.EN == 1) { /* Now reset */ CC.EN = 0; StorPortWriteRegisterUlong(pAE, (PULONG)(&pAE->pCtrlRegister->CC), CC.AsUlong); } Thanks, Alex ________________________________ From:nvmewin-bounces at lists.openfabrics.org[mailto:nvmewin-bounces at lists.openfabrics.org]On Behalf Of Luse, Paul E Sent: Wednesday, August 15, 2012 8:12 AM To: nvmewin at lists.openfabrics.org Subject: [nvmewin] patch review status Just checking in... any questions to date and how are folks tracking for completing their review by this Fri? Thx Paul ____________________________________ Paul Luse Sr. Staff Engineer PCG Server Software Engineering Desk: 480.554.3688, Mobile: 480.334.4630 -------------- next part -------------- An HTML attachment was scrubbed... URL: From paul.e.luse at intel.com Wed Aug 15 15:59:15 2012 From: paul.e.luse at intel.com (Luse, Paul E) Date: Wed, 15 Aug 2012 22:59:15 +0000 Subject: [nvmewin] patch review status References: <82C9F782B054C94B9FC04A331649C77A07B2B0C9@FMSMSX106.amr.corp.intel.com> Message-ID: <82C9F782B054C94B9FC04A331649C77A07B2B9EB@FMSMSX106.amr.corp.intel.com> So maybe a better proposal for this section of code... go back to what we had before (sorta) but include a wait on RDY after setting EN to 1 and only set EN to 1 if its zero to start. This is the most generic soln and since the behavior of the HW at this detailed level isn't specified we can call this the least common denominator and if IHVs want to optimize for their implementations they can. If this first RDY wait files, then we fail driver init. This code assures we always reset and we never touch EN after setting it to 1 without waiting for RDY IF EN == 0 { /* Set EN to 1 (all others zero) */ CC.EN = 1; StorPortWriteRegisterUlong(pAE, (PULONG)(&pAE->pCtrlRegister->CC), CC.AsUlong); WAIT ON RDY } /* Now reset */ CC.EN = 0; StorPortWriteRegisterUlong(pAE, (PULONG)(&pAE->pCtrlRegister->CC), CC.AsUlong); From: Luse, Paul E Sent: Wednesday, August 15, 2012 3:21 PM To: Verma, Vishal L; Chang, Alex; nvmewin at lists.openfabrics.org Subject: RE: [nvmewin] patch review status I was thinking about that too... If EN was already 1 and we caught RDY before it was 0 then we would indeed set EN to 1 again (possibly before, during or after RDY is set). The spec, of course, does not dictate behavior in this case but one would assume it would be treated as a NOP and our init state machine would properly wait for RDY. I'm not a HW guy though, anyone else have a comment/concern on setting EN to 1 if tis already 1? From: Verma, Vishal L Sent: Wednesday, August 15, 2012 1:46 PM To: Luse, Paul E; Chang, Alex; nvmewin at lists.openfabrics.org Subject: Re: [nvmewin] patch review status Paul, Is this correct though? Making CC.EN 0 implies that RDY will become 0, but RDY being 0 does not guarantee that CC.EN is 0. If what I'm thinking is right, we could be in a state where CC.EN was 1 when the driver comes up, but the controller never became ready. In this case we will think everything is alright, and we will try to make CC.EN 1 again - Which is not something we'd want to do, right? Vishal From: , Paul E > Date: Wednesday, August 15, 2012 11:43 AM To: "Chang, Alex" >, "nvmewin at lists.openfabrics.org" > Subject: Re: [nvmewin] patch review status OK, I think I can get behind that. I was thinking STS might not be valid when EN is 0 but looking at the spec I think this is fine Ready (RDY): This field is set to '1' when the controller is ready to process commands after CC.EN is set to '1'. This field shall be cleared to '0' when CC.EN is cleared to '0'. Commands shall not be issued to the controller until this field is set to '1' after the CC.EN bit is set to '1'. Failure to follow this requirement produces undefined results. Host software shall wait a minimum of CAP.TO seconds for this field to be set to '1' after setting CC.EN to '1' from a previous value of '0'. So I'll just chance the check below for EN == 1 to RDY == 1 then Thanks! Paul From: Chang, Alex [mailto:Alex.Chang at idt.com] Sent: Wednesday, August 15, 2012 10:21 AM To: Luse, Paul E; nvmewin at lists.openfabrics.org Subject: RE: patch review status Absolutely. Now, the tricky part is when CC.EN=1 and STS.RDY never becomes 1. I'd say simply check STS.RDY only. If it's 1, then reset the device, otherwise, skip resetting it. What do you think? Thanks, Alex ________________________________ From: Luse, Paul E [mailto:paul.e.luse at intel.com] Sent: Wednesday, August 15, 2012 9:56 AM To: Chang, Alex; nvmewin at lists.openfabrics.org Subject: RE: patch review status OK, I see what you're saying I think. We don't want to check for RDY without first looking at EN so does this make sense: if (CC.EN == 1) { -- now wait for RDY - /* Now reset */ CC.EN = 0; StorPortWriteRegisterUlong(pAE, (PULONG)(&pAE->pCtrlRegister->CC), CC.AsUlong); } From: Chang, Alex [mailto:Alex.Chang at idt.com] Sent: Wednesday, August 15, 2012 9:52 AM To: Luse, Paul E; nvmewin at lists.openfabrics.org Subject: RE: patch review status Hi Paul, I agree with the new logic on #3. However, instead of checking CC.EN, should we check STS.RDY or both? Thanks, Alex ________________________________ From: Luse, Paul E [mailto:paul.e.luse at intel.com] Sent: Wednesday, August 15, 2012 9:03 AM To: Chang, Alex; nvmewin at lists.openfabrics.org Subject: RE: patch review status Thanks Alex, see below. From: Chang, Alex [mailto:Alex.Chang at idt.com] Sent: Wednesday, August 15, 2012 8:51 AM To: Luse, Paul E; nvmewin at lists.openfabrics.org Subject: RE: patch review status Hi Paul, Some questions after browsing the difference between current source and your changes: 1. Why you removed "CC.SHN = 1;" in NVMeNormalShutdown routine? PL> Good eye, that was a mistake. I'll add it back before pushing 2. In nvme.h, why you changed the opcode of NVM_DATASET_MANAGEMENT from 0x06 to 0x09? PL> 6 was wrong, 9 is correct per spec 3. Is there any specific reason(s), you added the following in NVMeInitialize, and then immediately clear EN without checking RDY = 1? /* Set EN to 1 (all others zero) */ CC.EN = 1; StorPortWriteRegisterUlong(pAE, (PULONG)(&pAE->pCtrlRegister->CC), CC.AsUlong); PL> I wrote this up in the patch submission and I can't find that email right now. I didn't add code, I changed code. Before, we use to set EN to 1, read it back, and then set it to 0 to kick off a reset and then later we wait for RDY in the init state machine. We had an IHV complain that their implementation required waiting on RDY after setting EN 1 to even before clearing EN. So, our old implementation was a problem. So what I did now is to not force a reset unless EN is set to 1 when we come in. If its set to 0 then either (a) we're coming from power on or (b) the pre_OS reset the card when it was done. Either way if its already 0 then we're in a known state so we do nothing and then later we set to 1 and wait on RDY. If its 1 when we enter here we set to 0 and then later set to 1 and wait on RDY. Make sense? This assures we never enable without waiting for RDY ***** WAS ******* /* Set EN to 1 (all others zero) */ CC.EN = 1; StorPortWriteRegisterUlong(pAE, (PULONG)(&pAE->pCtrlRegister->CC), CC.AsUlong); /* Read it back */ CC.AsUlong = StorPortReadRegisterUlong(pAE, (PULONG)(&pAE->pCtrlRegister->CC)); /* Now reset */ CC.EN = 0; StorPortWriteRegisterUlong(pAE, (PULONG)(&pAE->pCtrlRegister->CC), CC.AsUlong); CC.AsUlong = StorPortReadRegisterUlong(pAE, (PULONG)(&pAE->pCtrlRegister->CC)); *** NOW IS ******* if (CC.EN == 1) { /* Now reset */ CC.EN = 0; StorPortWriteRegisterUlong(pAE, (PULONG)(&pAE->pCtrlRegister->CC), CC.AsUlong); } Thanks, Alex ________________________________ From:nvmewin-bounces at lists.openfabrics.org[mailto:nvmewin-bounces at lists.openfabrics.org]On Behalf Of Luse, Paul E Sent: Wednesday, August 15, 2012 8:12 AM To: nvmewin at lists.openfabrics.org Subject: [nvmewin] patch review status Just checking in... any questions to date and how are folks tracking for completing their review by this Fri? Thx Paul ____________________________________ Paul Luse Sr. Staff Engineer PCG Server Software Engineering Desk: 480.554.3688, Mobile: 480.334.4630 -------------- next part -------------- An HTML attachment was scrubbed... URL: From Karthik_Rajagopalan at Dell.com Wed Aug 15 16:11:08 2012 From: Karthik_Rajagopalan at Dell.com (Karthik_Rajagopalan at Dell.com) Date: Wed, 15 Aug 2012 18:11:08 -0500 Subject: [nvmewin] patch review status In-Reply-To: <82C9F782B054C94B9FC04A331649C77A07B2B9EB@FMSMSX106.amr.corp.intel.com> References: <82C9F782B054C94B9FC04A331649C77A07B2B0C9@FMSMSX106.amr.corp.intel.com> <82C9F782B054C94B9FC04A331649C77A07B2B9EB@FMSMSX106.amr.corp.intel.com> Message-ID: <718182ECAE1D7149A1BF0C789EAC6909049B647C47@AUSX7MCPC110.AMER.DELL.COM> Why not unconditionally set CC.EN to 0 as soon as the driver loads ? - There will be an issue with this iff someone before (opROM, UEFI ?) had set CC.EN to 1 & somehow STS.RDY never went to 1... but things still continued (unlikely) i.e., /* Disable / Reset */ CC.EN = 0; StorPortWriteRegisterUlong(pAE, (PULONG)(&pAE->pCtrlRegister->CC), CC.AsUlong); /* Set EN to 1 (all others zero) */ CC.EN = 1; StorPortWriteRegisterUlong(pAE, (PULONG)(&pAE->pCtrlRegister->CC), CC.AsUlong); /* Wait for RDY */ From: nvmewin-bounces at lists.openfabrics.org [mailto:nvmewin-bounces at lists.openfabrics.org] On Behalf Of Luse, Paul E Sent: Wednesday, August 15, 2012 5:59 PM To: Verma, Vishal L; Chang, Alex; nvmewin at lists.openfabrics.org Subject: Re: [nvmewin] patch review status So maybe a better proposal for this section of code... go back to what we had before (sorta) but include a wait on RDY after setting EN to 1 and only set EN to 1 if its zero to start. This is the most generic soln and since the behavior of the HW at this detailed level isn't specified we can call this the least common denominator and if IHVs want to optimize for their implementations they can. If this first RDY wait files, then we fail driver init. This code assures we always reset and we never touch EN after setting it to 1 without waiting for RDY IF EN == 0 { /* Set EN to 1 (all others zero) */ CC.EN = 1; StorPortWriteRegisterUlong(pAE, (PULONG)(&pAE->pCtrlRegister->CC), CC.AsUlong); WAIT ON RDY } /* Now reset */ CC.EN = 0; StorPortWriteRegisterUlong(pAE, (PULONG)(&pAE->pCtrlRegister->CC), CC.AsUlong); From: Luse, Paul E Sent: Wednesday, August 15, 2012 3:21 PM To: Verma, Vishal L; Chang, Alex; nvmewin at lists.openfabrics.org Subject: RE: [nvmewin] patch review status I was thinking about that too... If EN was already 1 and we caught RDY before it was 0 then we would indeed set EN to 1 again (possibly before, during or after RDY is set). The spec, of course, does not dictate behavior in this case but one would assume it would be treated as a NOP and our init state machine would properly wait for RDY. I'm not a HW guy though, anyone else have a comment/concern on setting EN to 1 if tis already 1? From: Verma, Vishal L Sent: Wednesday, August 15, 2012 1:46 PM To: Luse, Paul E; Chang, Alex; nvmewin at lists.openfabrics.org Subject: Re: [nvmewin] patch review status Paul, Is this correct though? Making CC.EN 0 implies that RDY will become 0, but RDY being 0 does not guarantee that CC.EN is 0. If what I'm thinking is right, we could be in a state where CC.EN was 1 when the driver comes up, but the controller never became ready. In this case we will think everything is alright, and we will try to make CC.EN 1 again - Which is not something we'd want to do, right? Vishal From: , Paul E > Date: Wednesday, August 15, 2012 11:43 AM To: "Chang, Alex" >, "nvmewin at lists.openfabrics.org" > Subject: Re: [nvmewin] patch review status OK, I think I can get behind that. I was thinking STS might not be valid when EN is 0 but looking at the spec I think this is fine Ready (RDY): This field is set to '1' when the controller is ready to process commands after CC.EN is set to '1'. This field shall be cleared to '0' when CC.EN is cleared to '0'. Commands shall not be issued to the controller until this field is set to '1' after the CC.EN bit is set to '1'. Failure to follow this requirement produces undefined results. Host software shall wait a minimum of CAP.TO seconds for this field to be set to '1' after setting CC.EN to '1' from a previous value of '0'. So I'll just chance the check below for EN == 1 to RDY == 1 then Thanks! Paul From: Chang, Alex [mailto:Alex.Chang at idt.com] Sent: Wednesday, August 15, 2012 10:21 AM To: Luse, Paul E; nvmewin at lists.openfabrics.org Subject: RE: patch review status Absolutely. Now, the tricky part is when CC.EN=1 and STS.RDY never becomes 1. I'd say simply check STS.RDY only. If it's 1, then reset the device, otherwise, skip resetting it. What do you think? Thanks, Alex ________________________________ From: Luse, Paul E [mailto:paul.e.luse at intel.com] Sent: Wednesday, August 15, 2012 9:56 AM To: Chang, Alex; nvmewin at lists.openfabrics.org Subject: RE: patch review status OK, I see what you're saying I think. We don't want to check for RDY without first looking at EN so does this make sense: if (CC.EN == 1) { -- now wait for RDY - /* Now reset */ CC.EN = 0; StorPortWriteRegisterUlong(pAE, (PULONG)(&pAE->pCtrlRegister->CC), CC.AsUlong); } From: Chang, Alex [mailto:Alex.Chang at idt.com] Sent: Wednesday, August 15, 2012 9:52 AM To: Luse, Paul E; nvmewin at lists.openfabrics.org Subject: RE: patch review status Hi Paul, I agree with the new logic on #3. However, instead of checking CC.EN, should we check STS.RDY or both? Thanks, Alex ________________________________ From: Luse, Paul E [mailto:paul.e.luse at intel.com] Sent: Wednesday, August 15, 2012 9:03 AM To: Chang, Alex; nvmewin at lists.openfabrics.org Subject: RE: patch review status Thanks Alex, see below. From: Chang, Alex [mailto:Alex.Chang at idt.com] Sent: Wednesday, August 15, 2012 8:51 AM To: Luse, Paul E; nvmewin at lists.openfabrics.org Subject: RE: patch review status Hi Paul, Some questions after browsing the difference between current source and your changes: 1. Why you removed "CC.SHN = 1;" in NVMeNormalShutdown routine? PL> Good eye, that was a mistake. I'll add it back before pushing 2. In nvme.h, why you changed the opcode of NVM_DATASET_MANAGEMENT from 0x06 to 0x09? PL> 6 was wrong, 9 is correct per spec 3. Is there any specific reason(s), you added the following in NVMeInitialize, and then immediately clear EN without checking RDY = 1? /* Set EN to 1 (all others zero) */ CC.EN = 1; StorPortWriteRegisterUlong(pAE, (PULONG)(&pAE->pCtrlRegister->CC), CC.AsUlong); PL> I wrote this up in the patch submission and I can't find that email right now. I didn't add code, I changed code. Before, we use to set EN to 1, read it back, and then set it to 0 to kick off a reset and then later we wait for RDY in the init state machine. We had an IHV complain that their implementation required waiting on RDY after setting EN 1 to even before clearing EN. So, our old implementation was a problem. So what I did now is to not force a reset unless EN is set to 1 when we come in. If its set to 0 then either (a) we're coming from power on or (b) the pre_OS reset the card when it was done. Either way if its already 0 then we're in a known state so we do nothing and then later we set to 1 and wait on RDY. If its 1 when we enter here we set to 0 and then later set to 1 and wait on RDY. Make sense? This assures we never enable without waiting for RDY ***** WAS ******* /* Set EN to 1 (all others zero) */ CC.EN = 1; StorPortWriteRegisterUlong(pAE, (PULONG)(&pAE->pCtrlRegister->CC), CC.AsUlong); /* Read it back */ CC.AsUlong = StorPortReadRegisterUlong(pAE, (PULONG)(&pAE->pCtrlRegister->CC)); /* Now reset */ CC.EN = 0; StorPortWriteRegisterUlong(pAE, (PULONG)(&pAE->pCtrlRegister->CC), CC.AsUlong); CC.AsUlong = StorPortReadRegisterUlong(pAE, (PULONG)(&pAE->pCtrlRegister->CC)); *** NOW IS ******* if (CC.EN == 1) { /* Now reset */ CC.EN = 0; StorPortWriteRegisterUlong(pAE, (PULONG)(&pAE->pCtrlRegister->CC), CC.AsUlong); } Thanks, Alex ________________________________ From:nvmewin-bounces at lists.openfabrics.org[mailto:nvmewin-bounces at lists.openfabrics.org]On Behalf Of Luse, Paul E Sent: Wednesday, August 15, 2012 8:12 AM To: nvmewin at lists.openfabrics.org Subject: [nvmewin] patch review status Just checking in... any questions to date and how are folks tracking for completing their review by this Fri? Thx Paul ____________________________________ Paul Luse Sr. Staff Engineer PCG Server Software Engineering Desk: 480.554.3688, Mobile: 480.334.4630 -------------- next part -------------- An HTML attachment was scrubbed... URL: From paul.e.luse at intel.com Wed Aug 15 16:21:58 2012 From: paul.e.luse at intel.com (Luse, Paul E) Date: Wed, 15 Aug 2012 23:21:58 +0000 Subject: [nvmewin] patch review status In-Reply-To: <718182ECAE1D7149A1BF0C789EAC6909049B647C47@AUSX7MCPC110.AMER.DELL.COM> References: <82C9F782B054C94B9FC04A331649C77A07B2B0C9@FMSMSX106.amr.corp.intel.com> <82C9F782B054C94B9FC04A331649C77A07B2B9EB@FMSMSX106.amr.corp.intel.com> <718182ECAE1D7149A1BF0C789EAC6909049B647C47@AUSX7MCPC110.AMER.DELL.COM> Message-ID: <82C9F782B054C94B9FC04A331649C77A07B2BAA5@FMSMSX106.amr.corp.intel.com> 0 doesn't reset unless it was previously 1... that's why the initial implementation set it to 1, then 0 to force reset then init happened. The question came in about setting it to 0 right after setting it to 1, anytime you transition to 1 you need to wait for RDY so the code I proposed last will always reset but only for a 2nd wait on RDY (there's one after we enable as part of init) if the controller wasn't disabled when we started. Hard to believe something so simple can be so convoluted isn't it? From: Karthik_Rajagopalan at Dell.com [mailto:Karthik_Rajagopalan at Dell.com] Sent: Wednesday, August 15, 2012 4:11 PM To: Luse, Paul E; Verma, Vishal L; Alex.Chang at idt.com; nvmewin at lists.openfabrics.org Subject: RE: [nvmewin] patch review status Why not unconditionally set CC.EN to 0 as soon as the driver loads ? - There will be an issue with this iff someone before (opROM, UEFI ?) had set CC.EN to 1 & somehow STS.RDY never went to 1... but things still continued (unlikely) i.e., /* Disable / Reset */ CC.EN = 0; StorPortWriteRegisterUlong(pAE, (PULONG)(&pAE->pCtrlRegister->CC), CC.AsUlong); /* Set EN to 1 (all others zero) */ CC.EN = 1; StorPortWriteRegisterUlong(pAE, (PULONG)(&pAE->pCtrlRegister->CC), CC.AsUlong); /* Wait for RDY */ From: nvmewin-bounces at lists.openfabrics.org [mailto:nvmewin-bounces at lists.openfabrics.org] On Behalf Of Luse, Paul E Sent: Wednesday, August 15, 2012 5:59 PM To: Verma, Vishal L; Chang, Alex; nvmewin at lists.openfabrics.org Subject: Re: [nvmewin] patch review status So maybe a better proposal for this section of code... go back to what we had before (sorta) but include a wait on RDY after setting EN to 1 and only set EN to 1 if its zero to start. This is the most generic soln and since the behavior of the HW at this detailed level isn't specified we can call this the least common denominator and if IHVs want to optimize for their implementations they can. If this first RDY wait files, then we fail driver init. This code assures we always reset and we never touch EN after setting it to 1 without waiting for RDY IF EN == 0 { /* Set EN to 1 (all others zero) */ CC.EN = 1; StorPortWriteRegisterUlong(pAE, (PULONG)(&pAE->pCtrlRegister->CC), CC.AsUlong); WAIT ON RDY } /* Now reset */ CC.EN = 0; StorPortWriteRegisterUlong(pAE, (PULONG)(&pAE->pCtrlRegister->CC), CC.AsUlong); From: Luse, Paul E Sent: Wednesday, August 15, 2012 3:21 PM To: Verma, Vishal L; Chang, Alex; nvmewin at lists.openfabrics.org Subject: RE: [nvmewin] patch review status I was thinking about that too... If EN was already 1 and we caught RDY before it was 0 then we would indeed set EN to 1 again (possibly before, during or after RDY is set). The spec, of course, does not dictate behavior in this case but one would assume it would be treated as a NOP and our init state machine would properly wait for RDY. I'm not a HW guy though, anyone else have a comment/concern on setting EN to 1 if tis already 1? From: Verma, Vishal L Sent: Wednesday, August 15, 2012 1:46 PM To: Luse, Paul E; Chang, Alex; nvmewin at lists.openfabrics.org Subject: Re: [nvmewin] patch review status Paul, Is this correct though? Making CC.EN 0 implies that RDY will become 0, but RDY being 0 does not guarantee that CC.EN is 0. If what I'm thinking is right, we could be in a state where CC.EN was 1 when the driver comes up, but the controller never became ready. In this case we will think everything is alright, and we will try to make CC.EN 1 again - Which is not something we'd want to do, right? Vishal From: , Paul E > Date: Wednesday, August 15, 2012 11:43 AM To: "Chang, Alex" >, "nvmewin at lists.openfabrics.org" > Subject: Re: [nvmewin] patch review status OK, I think I can get behind that. I was thinking STS might not be valid when EN is 0 but looking at the spec I think this is fine Ready (RDY): This field is set to '1' when the controller is ready to process commands after CC.EN is set to '1'. This field shall be cleared to '0' when CC.EN is cleared to '0'. Commands shall not be issued to the controller until this field is set to '1' after the CC.EN bit is set to '1'. Failure to follow this requirement produces undefined results. Host software shall wait a minimum of CAP.TO seconds for this field to be set to '1' after setting CC.EN to '1' from a previous value of '0'. So I'll just chance the check below for EN == 1 to RDY == 1 then Thanks! Paul From: Chang, Alex [mailto:Alex.Chang at idt.com] Sent: Wednesday, August 15, 2012 10:21 AM To: Luse, Paul E; nvmewin at lists.openfabrics.org Subject: RE: patch review status Absolutely. Now, the tricky part is when CC.EN=1 and STS.RDY never becomes 1. I'd say simply check STS.RDY only. If it's 1, then reset the device, otherwise, skip resetting it. What do you think? Thanks, Alex ________________________________ From: Luse, Paul E [mailto:paul.e.luse at intel.com] Sent: Wednesday, August 15, 2012 9:56 AM To: Chang, Alex; nvmewin at lists.openfabrics.org Subject: RE: patch review status OK, I see what you're saying I think. We don't want to check for RDY without first looking at EN so does this make sense: if (CC.EN == 1) { -- now wait for RDY - /* Now reset */ CC.EN = 0; StorPortWriteRegisterUlong(pAE, (PULONG)(&pAE->pCtrlRegister->CC), CC.AsUlong); } From: Chang, Alex [mailto:Alex.Chang at idt.com] Sent: Wednesday, August 15, 2012 9:52 AM To: Luse, Paul E; nvmewin at lists.openfabrics.org Subject: RE: patch review status Hi Paul, I agree with the new logic on #3. However, instead of checking CC.EN, should we check STS.RDY or both? Thanks, Alex ________________________________ From: Luse, Paul E [mailto:paul.e.luse at intel.com] Sent: Wednesday, August 15, 2012 9:03 AM To: Chang, Alex; nvmewin at lists.openfabrics.org Subject: RE: patch review status Thanks Alex, see below. From: Chang, Alex [mailto:Alex.Chang at idt.com] Sent: Wednesday, August 15, 2012 8:51 AM To: Luse, Paul E; nvmewin at lists.openfabrics.org Subject: RE: patch review status Hi Paul, Some questions after browsing the difference between current source and your changes: 1. Why you removed "CC.SHN = 1;" in NVMeNormalShutdown routine? PL> Good eye, that was a mistake. I'll add it back before pushing 2. In nvme.h, why you changed the opcode of NVM_DATASET_MANAGEMENT from 0x06 to 0x09? PL> 6 was wrong, 9 is correct per spec 3. Is there any specific reason(s), you added the following in NVMeInitialize, and then immediately clear EN without checking RDY = 1? /* Set EN to 1 (all others zero) */ CC.EN = 1; StorPortWriteRegisterUlong(pAE, (PULONG)(&pAE->pCtrlRegister->CC), CC.AsUlong); PL> I wrote this up in the patch submission and I can't find that email right now. I didn't add code, I changed code. Before, we use to set EN to 1, read it back, and then set it to 0 to kick off a reset and then later we wait for RDY in the init state machine. We had an IHV complain that their implementation required waiting on RDY after setting EN 1 to even before clearing EN. So, our old implementation was a problem. So what I did now is to not force a reset unless EN is set to 1 when we come in. If its set to 0 then either (a) we're coming from power on or (b) the pre_OS reset the card when it was done. Either way if its already 0 then we're in a known state so we do nothing and then later we set to 1 and wait on RDY. If its 1 when we enter here we set to 0 and then later set to 1 and wait on RDY. Make sense? This assures we never enable without waiting for RDY ***** WAS ******* /* Set EN to 1 (all others zero) */ CC.EN = 1; StorPortWriteRegisterUlong(pAE, (PULONG)(&pAE->pCtrlRegister->CC), CC.AsUlong); /* Read it back */ CC.AsUlong = StorPortReadRegisterUlong(pAE, (PULONG)(&pAE->pCtrlRegister->CC)); /* Now reset */ CC.EN = 0; StorPortWriteRegisterUlong(pAE, (PULONG)(&pAE->pCtrlRegister->CC), CC.AsUlong); CC.AsUlong = StorPortReadRegisterUlong(pAE, (PULONG)(&pAE->pCtrlRegister->CC)); *** NOW IS ******* if (CC.EN == 1) { /* Now reset */ CC.EN = 0; StorPortWriteRegisterUlong(pAE, (PULONG)(&pAE->pCtrlRegister->CC), CC.AsUlong); } Thanks, Alex ________________________________ From:nvmewin-bounces at lists.openfabrics.org[mailto:nvmewin-bounces at lists.openfabrics.org]On Behalf Of Luse, Paul E Sent: Wednesday, August 15, 2012 8:12 AM To: nvmewin at lists.openfabrics.org Subject: [nvmewin] patch review status Just checking in... any questions to date and how are folks tracking for completing their review by this Fri? Thx Paul ____________________________________ Paul Luse Sr. Staff Engineer PCG Server Software Engineering Desk: 480.554.3688, Mobile: 480.334.4630 -------------- next part -------------- An HTML attachment was scrubbed... URL: From Karthik_Rajagopalan at Dell.com Wed Aug 15 16:32:06 2012 From: Karthik_Rajagopalan at Dell.com (Karthik_Rajagopalan at Dell.com) Date: Wed, 15 Aug 2012 18:32:06 -0500 Subject: [nvmewin] patch review status In-Reply-To: <82C9F782B054C94B9FC04A331649C77A07B2BAA5@FMSMSX106.amr.corp.intel.com> References: <82C9F782B054C94B9FC04A331649C77A07B2B0C9@FMSMSX106.amr.corp.intel.com> <82C9F782B054C94B9FC04A331649C77A07B2B9EB@FMSMSX106.amr.corp.intel.com> <718182ECAE1D7149A1BF0C789EAC6909049B647C47@AUSX7MCPC110.AMER.DELL.COM> <82C9F782B054C94B9FC04A331649C77A07B2BAA5@FMSMSX106.amr.corp.intel.com> Message-ID: <718182ECAE1D7149A1BF0C789EAC6909049B647C4F@AUSX7MCPC110.AMER.DELL.COM> Yep :) but I think it doesn't have to be so - all we need to do is an unconditional (attempt) to reset.. we don't set CC.EN to 1 until we're ready to enable the controller. Here are the possibilities when we try to reset: 1. CC.EN is 0 to begin with => no need to reset; but setting it again to 0 should be a NOP 2. CC.EN is 1 to begin with => need to reset; we didn't wiggle this yet, so STS.RDY should be 1; setting EN to 0 will actually reset a. The original issue you mention could still happen if STS.RDY stayed 0 (bad controller IMHO); still goodness in trying to reset Here's a patch in case you want to try this approach... From: Luse, Paul E [mailto:paul.e.luse at intel.com] Sent: Wednesday, August 15, 2012 6:22 PM To: Rajagopalan, Karthik; Verma, Vishal L; Alex.Chang at idt.com; nvmewin at lists.openfabrics.org Subject: RE: [nvmewin] patch review status 0 doesn't reset unless it was previously 1... that's why the initial implementation set it to 1, then 0 to force reset then init happened. The question came in about setting it to 0 right after setting it to 1, anytime you transition to 1 you need to wait for RDY so the code I proposed last will always reset but only for a 2nd wait on RDY (there's one after we enable as part of init) if the controller wasn't disabled when we started. Hard to believe something so simple can be so convoluted isn't it? From: Karthik_Rajagopalan at Dell.com [mailto:Karthik_Rajagopalan at Dell.com] Sent: Wednesday, August 15, 2012 4:11 PM To: Luse, Paul E; Verma, Vishal L; Alex.Chang at idt.com; nvmewin at lists.openfabrics.org Subject: RE: [nvmewin] patch review status Why not unconditionally set CC.EN to 0 as soon as the driver loads ? - There will be an issue with this iff someone before (opROM, UEFI ?) had set CC.EN to 1 & somehow STS.RDY never went to 1... but things still continued (unlikely) i.e., /* Disable / Reset */ CC.EN = 0; StorPortWriteRegisterUlong(pAE, (PULONG)(&pAE->pCtrlRegister->CC), CC.AsUlong); /* Set EN to 1 (all others zero) */ CC.EN = 1; StorPortWriteRegisterUlong(pAE, (PULONG)(&pAE->pCtrlRegister->CC), CC.AsUlong); /* Wait for RDY */ From: nvmewin-bounces at lists.openfabrics.org [mailto:nvmewin-bounces at lists.openfabrics.org] On Behalf Of Luse, Paul E Sent: Wednesday, August 15, 2012 5:59 PM To: Verma, Vishal L; Chang, Alex; nvmewin at lists.openfabrics.org Subject: Re: [nvmewin] patch review status So maybe a better proposal for this section of code... go back to what we had before (sorta) but include a wait on RDY after setting EN to 1 and only set EN to 1 if its zero to start. This is the most generic soln and since the behavior of the HW at this detailed level isn't specified we can call this the least common denominator and if IHVs want to optimize for their implementations they can. If this first RDY wait files, then we fail driver init. This code assures we always reset and we never touch EN after setting it to 1 without waiting for RDY IF EN == 0 { /* Set EN to 1 (all others zero) */ CC.EN = 1; StorPortWriteRegisterUlong(pAE, (PULONG)(&pAE->pCtrlRegister->CC), CC.AsUlong); WAIT ON RDY } /* Now reset */ CC.EN = 0; StorPortWriteRegisterUlong(pAE, (PULONG)(&pAE->pCtrlRegister->CC), CC.AsUlong); From: Luse, Paul E Sent: Wednesday, August 15, 2012 3:21 PM To: Verma, Vishal L; Chang, Alex; nvmewin at lists.openfabrics.org Subject: RE: [nvmewin] patch review status I was thinking about that too... If EN was already 1 and we caught RDY before it was 0 then we would indeed set EN to 1 again (possibly before, during or after RDY is set). The spec, of course, does not dictate behavior in this case but one would assume it would be treated as a NOP and our init state machine would properly wait for RDY. I'm not a HW guy though, anyone else have a comment/concern on setting EN to 1 if tis already 1? From: Verma, Vishal L Sent: Wednesday, August 15, 2012 1:46 PM To: Luse, Paul E; Chang, Alex; nvmewin at lists.openfabrics.org Subject: Re: [nvmewin] patch review status Paul, Is this correct though? Making CC.EN 0 implies that RDY will become 0, but RDY being 0 does not guarantee that CC.EN is 0. If what I'm thinking is right, we could be in a state where CC.EN was 1 when the driver comes up, but the controller never became ready. In this case we will think everything is alright, and we will try to make CC.EN 1 again - Which is not something we'd want to do, right? Vishal From: , Paul E > Date: Wednesday, August 15, 2012 11:43 AM To: "Chang, Alex" >, "nvmewin at lists.openfabrics.org" > Subject: Re: [nvmewin] patch review status OK, I think I can get behind that. I was thinking STS might not be valid when EN is 0 but looking at the spec I think this is fine Ready (RDY): This field is set to '1' when the controller is ready to process commands after CC.EN is set to '1'. This field shall be cleared to '0' when CC.EN is cleared to '0'. Commands shall not be issued to the controller until this field is set to '1' after the CC.EN bit is set to '1'. Failure to follow this requirement produces undefined results. Host software shall wait a minimum of CAP.TO seconds for this field to be set to '1' after setting CC.EN to '1' from a previous value of '0'. So I'll just chance the check below for EN == 1 to RDY == 1 then Thanks! Paul From: Chang, Alex [mailto:Alex.Chang at idt.com] Sent: Wednesday, August 15, 2012 10:21 AM To: Luse, Paul E; nvmewin at lists.openfabrics.org Subject: RE: patch review status Absolutely. Now, the tricky part is when CC.EN=1 and STS.RDY never becomes 1. I'd say simply check STS.RDY only. If it's 1, then reset the device, otherwise, skip resetting it. What do you think? Thanks, Alex ________________________________ From: Luse, Paul E [mailto:paul.e.luse at intel.com] Sent: Wednesday, August 15, 2012 9:56 AM To: Chang, Alex; nvmewin at lists.openfabrics.org Subject: RE: patch review status OK, I see what you're saying I think. We don't want to check for RDY without first looking at EN so does this make sense: if (CC.EN == 1) { -- now wait for RDY - /* Now reset */ CC.EN = 0; StorPortWriteRegisterUlong(pAE, (PULONG)(&pAE->pCtrlRegister->CC), CC.AsUlong); } From: Chang, Alex [mailto:Alex.Chang at idt.com] Sent: Wednesday, August 15, 2012 9:52 AM To: Luse, Paul E; nvmewin at lists.openfabrics.org Subject: RE: patch review status Hi Paul, I agree with the new logic on #3. However, instead of checking CC.EN, should we check STS.RDY or both? Thanks, Alex ________________________________ From: Luse, Paul E [mailto:paul.e.luse at intel.com] Sent: Wednesday, August 15, 2012 9:03 AM To: Chang, Alex; nvmewin at lists.openfabrics.org Subject: RE: patch review status Thanks Alex, see below. From: Chang, Alex [mailto:Alex.Chang at idt.com] Sent: Wednesday, August 15, 2012 8:51 AM To: Luse, Paul E; nvmewin at lists.openfabrics.org Subject: RE: patch review status Hi Paul, Some questions after browsing the difference between current source and your changes: 1. Why you removed "CC.SHN = 1;" in NVMeNormalShutdown routine? PL> Good eye, that was a mistake. I'll add it back before pushing 2. In nvme.h, why you changed the opcode of NVM_DATASET_MANAGEMENT from 0x06 to 0x09? PL> 6 was wrong, 9 is correct per spec 3. Is there any specific reason(s), you added the following in NVMeInitialize, and then immediately clear EN without checking RDY = 1? /* Set EN to 1 (all others zero) */ CC.EN = 1; StorPortWriteRegisterUlong(pAE, (PULONG)(&pAE->pCtrlRegister->CC), CC.AsUlong); PL> I wrote this up in the patch submission and I can't find that email right now. I didn't add code, I changed code. Before, we use to set EN to 1, read it back, and then set it to 0 to kick off a reset and then later we wait for RDY in the init state machine. We had an IHV complain that their implementation required waiting on RDY after setting EN 1 to even before clearing EN. So, our old implementation was a problem. So what I did now is to not force a reset unless EN is set to 1 when we come in. If its set to 0 then either (a) we're coming from power on or (b) the pre_OS reset the card when it was done. Either way if its already 0 then we're in a known state so we do nothing and then later we set to 1 and wait on RDY. If its 1 when we enter here we set to 0 and then later set to 1 and wait on RDY. Make sense? This assures we never enable without waiting for RDY ***** WAS ******* /* Set EN to 1 (all others zero) */ CC.EN = 1; StorPortWriteRegisterUlong(pAE, (PULONG)(&pAE->pCtrlRegister->CC), CC.AsUlong); /* Read it back */ CC.AsUlong = StorPortReadRegisterUlong(pAE, (PULONG)(&pAE->pCtrlRegister->CC)); /* Now reset */ CC.EN = 0; StorPortWriteRegisterUlong(pAE, (PULONG)(&pAE->pCtrlRegister->CC), CC.AsUlong); CC.AsUlong = StorPortReadRegisterUlong(pAE, (PULONG)(&pAE->pCtrlRegister->CC)); *** NOW IS ******* if (CC.EN == 1) { /* Now reset */ CC.EN = 0; StorPortWriteRegisterUlong(pAE, (PULONG)(&pAE->pCtrlRegister->CC), CC.AsUlong); } Thanks, Alex ________________________________ From:nvmewin-bounces at lists.openfabrics.org[mailto:nvmewin-bounces at lists.openfabrics.org]On Behalf Of Luse, Paul E Sent: Wednesday, August 15, 2012 8:12 AM To: nvmewin at lists.openfabrics.org Subject: [nvmewin] patch review status Just checking in... any questions to date and how are folks tracking for completing their review by this Fri? Thx Paul ____________________________________ Paul Luse Sr. Staff Engineer PCG Server Software Engineering Desk: 480.554.3688, Mobile: 480.334.4630 -------------- next part -------------- An HTML attachment was scrubbed... URL: -------------- next part -------------- A non-text attachment was scrubbed... Name: cc-en.diff Type: application/octet-stream Size: 640 bytes Desc: cc-en.diff URL: From Karthik_Rajagopalan at Dell.com Wed Aug 15 16:43:15 2012 From: Karthik_Rajagopalan at Dell.com (Karthik_Rajagopalan at Dell.com) Date: Wed, 15 Aug 2012 18:43:15 -0500 Subject: [nvmewin] patch review status References: <82C9F782B054C94B9FC04A331649C77A07B2B0C9@FMSMSX106.amr.corp.intel.com> <82C9F782B054C94B9FC04A331649C77A07B2B9EB@FMSMSX106.amr.corp.intel.com> <718182ECAE1D7149A1BF0C789EAC6909049B647C47@AUSX7MCPC110.AMER.DELL.COM> <82C9F782B054C94B9FC04A331649C77A07B2BAA5@FMSMSX106.amr.corp.intel.com> Message-ID: <718182ECAE1D7149A1BF0C789EAC6909049B647C52@AUSX7MCPC110.AMER.DELL.COM> BTW, here's what the spec says for CC.EN When this field is set to '1', the controller sets CSTS.RDY to '1' when it is ready to process commands. I read it as "set CC.EN to 1 after setting up the admin-queues, otherwise RDY will *not* be 1" From: Rajagopalan, Karthik Sent: Wednesday, August 15, 2012 6:32 PM To: 'Luse, Paul E'; Verma, Vishal L; Alex.Chang at idt.com; nvmewin at lists.openfabrics.org Subject: RE: [nvmewin] patch review status Yep :) but I think it doesn't have to be so - all we need to do is an unconditional (attempt) to reset.. we don't set CC.EN to 1 until we're ready to enable the controller. Here are the possibilities when we try to reset: 1. CC.EN is 0 to begin with => no need to reset; but setting it again to 0 should be a NOP 2. CC.EN is 1 to begin with => need to reset; we didn't wiggle this yet, so STS.RDY should be 1; setting EN to 0 will actually reset a. The original issue you mention could still happen if STS.RDY stayed 0 (bad controller IMHO); still goodness in trying to reset Here's a patch in case you want to try this approach... From: Luse, Paul E [mailto:paul.e.luse at intel.com] Sent: Wednesday, August 15, 2012 6:22 PM To: Rajagopalan, Karthik; Verma, Vishal L; Alex.Chang at idt.com; nvmewin at lists.openfabrics.org Subject: RE: [nvmewin] patch review status 0 doesn't reset unless it was previously 1... that's why the initial implementation set it to 1, then 0 to force reset then init happened. The question came in about setting it to 0 right after setting it to 1, anytime you transition to 1 you need to wait for RDY so the code I proposed last will always reset but only for a 2nd wait on RDY (there's one after we enable as part of init) if the controller wasn't disabled when we started. Hard to believe something so simple can be so convoluted isn't it? From: Karthik_Rajagopalan at Dell.com [mailto:Karthik_Rajagopalan at Dell.com] Sent: Wednesday, August 15, 2012 4:11 PM To: Luse, Paul E; Verma, Vishal L; Alex.Chang at idt.com; nvmewin at lists.openfabrics.org Subject: RE: [nvmewin] patch review status Why not unconditionally set CC.EN to 0 as soon as the driver loads ? - There will be an issue with this iff someone before (opROM, UEFI ?) had set CC.EN to 1 & somehow STS.RDY never went to 1... but things still continued (unlikely) i.e., /* Disable / Reset */ CC.EN = 0; StorPortWriteRegisterUlong(pAE, (PULONG)(&pAE->pCtrlRegister->CC), CC.AsUlong); /* Set EN to 1 (all others zero) */ CC.EN = 1; StorPortWriteRegisterUlong(pAE, (PULONG)(&pAE->pCtrlRegister->CC), CC.AsUlong); /* Wait for RDY */ From: nvmewin-bounces at lists.openfabrics.org [mailto:nvmewin-bounces at lists.openfabrics.org] On Behalf Of Luse, Paul E Sent: Wednesday, August 15, 2012 5:59 PM To: Verma, Vishal L; Chang, Alex; nvmewin at lists.openfabrics.org Subject: Re: [nvmewin] patch review status So maybe a better proposal for this section of code... go back to what we had before (sorta) but include a wait on RDY after setting EN to 1 and only set EN to 1 if its zero to start. This is the most generic soln and since the behavior of the HW at this detailed level isn't specified we can call this the least common denominator and if IHVs want to optimize for their implementations they can. If this first RDY wait files, then we fail driver init. This code assures we always reset and we never touch EN after setting it to 1 without waiting for RDY IF EN == 0 { /* Set EN to 1 (all others zero) */ CC.EN = 1; StorPortWriteRegisterUlong(pAE, (PULONG)(&pAE->pCtrlRegister->CC), CC.AsUlong); WAIT ON RDY } /* Now reset */ CC.EN = 0; StorPortWriteRegisterUlong(pAE, (PULONG)(&pAE->pCtrlRegister->CC), CC.AsUlong); From: Luse, Paul E Sent: Wednesday, August 15, 2012 3:21 PM To: Verma, Vishal L; Chang, Alex; nvmewin at lists.openfabrics.org Subject: RE: [nvmewin] patch review status I was thinking about that too... If EN was already 1 and we caught RDY before it was 0 then we would indeed set EN to 1 again (possibly before, during or after RDY is set). The spec, of course, does not dictate behavior in this case but one would assume it would be treated as a NOP and our init state machine would properly wait for RDY. I'm not a HW guy though, anyone else have a comment/concern on setting EN to 1 if tis already 1? From: Verma, Vishal L Sent: Wednesday, August 15, 2012 1:46 PM To: Luse, Paul E; Chang, Alex; nvmewin at lists.openfabrics.org Subject: Re: [nvmewin] patch review status Paul, Is this correct though? Making CC.EN 0 implies that RDY will become 0, but RDY being 0 does not guarantee that CC.EN is 0. If what I'm thinking is right, we could be in a state where CC.EN was 1 when the driver comes up, but the controller never became ready. In this case we will think everything is alright, and we will try to make CC.EN 1 again - Which is not something we'd want to do, right? Vishal From: , Paul E > Date: Wednesday, August 15, 2012 11:43 AM To: "Chang, Alex" >, "nvmewin at lists.openfabrics.org" > Subject: Re: [nvmewin] patch review status OK, I think I can get behind that. I was thinking STS might not be valid when EN is 0 but looking at the spec I think this is fine Ready (RDY): This field is set to '1' when the controller is ready to process commands after CC.EN is set to '1'. This field shall be cleared to '0' when CC.EN is cleared to '0'. Commands shall not be issued to the controller until this field is set to '1' after the CC.EN bit is set to '1'. Failure to follow this requirement produces undefined results. Host software shall wait a minimum of CAP.TO seconds for this field to be set to '1' after setting CC.EN to '1' from a previous value of '0'. So I'll just chance the check below for EN == 1 to RDY == 1 then Thanks! Paul From: Chang, Alex [mailto:Alex.Chang at idt.com] Sent: Wednesday, August 15, 2012 10:21 AM To: Luse, Paul E; nvmewin at lists.openfabrics.org Subject: RE: patch review status Absolutely. Now, the tricky part is when CC.EN=1 and STS.RDY never becomes 1. I'd say simply check STS.RDY only. If it's 1, then reset the device, otherwise, skip resetting it. What do you think? Thanks, Alex ________________________________ From: Luse, Paul E [mailto:paul.e.luse at intel.com] Sent: Wednesday, August 15, 2012 9:56 AM To: Chang, Alex; nvmewin at lists.openfabrics.org Subject: RE: patch review status OK, I see what you're saying I think. We don't want to check for RDY without first looking at EN so does this make sense: if (CC.EN == 1) { -- now wait for RDY - /* Now reset */ CC.EN = 0; StorPortWriteRegisterUlong(pAE, (PULONG)(&pAE->pCtrlRegister->CC), CC.AsUlong); } From: Chang, Alex [mailto:Alex.Chang at idt.com] Sent: Wednesday, August 15, 2012 9:52 AM To: Luse, Paul E; nvmewin at lists.openfabrics.org Subject: RE: patch review status Hi Paul, I agree with the new logic on #3. However, instead of checking CC.EN, should we check STS.RDY or both? Thanks, Alex ________________________________ From: Luse, Paul E [mailto:paul.e.luse at intel.com] Sent: Wednesday, August 15, 2012 9:03 AM To: Chang, Alex; nvmewin at lists.openfabrics.org Subject: RE: patch review status Thanks Alex, see below. From: Chang, Alex [mailto:Alex.Chang at idt.com] Sent: Wednesday, August 15, 2012 8:51 AM To: Luse, Paul E; nvmewin at lists.openfabrics.org Subject: RE: patch review status Hi Paul, Some questions after browsing the difference between current source and your changes: 1. Why you removed "CC.SHN = 1;" in NVMeNormalShutdown routine? PL> Good eye, that was a mistake. I'll add it back before pushing 2. In nvme.h, why you changed the opcode of NVM_DATASET_MANAGEMENT from 0x06 to 0x09? PL> 6 was wrong, 9 is correct per spec 3. Is there any specific reason(s), you added the following in NVMeInitialize, and then immediately clear EN without checking RDY = 1? /* Set EN to 1 (all others zero) */ CC.EN = 1; StorPortWriteRegisterUlong(pAE, (PULONG)(&pAE->pCtrlRegister->CC), CC.AsUlong); PL> I wrote this up in the patch submission and I can't find that email right now. I didn't add code, I changed code. Before, we use to set EN to 1, read it back, and then set it to 0 to kick off a reset and then later we wait for RDY in the init state machine. We had an IHV complain that their implementation required waiting on RDY after setting EN 1 to even before clearing EN. So, our old implementation was a problem. So what I did now is to not force a reset unless EN is set to 1 when we come in. If its set to 0 then either (a) we're coming from power on or (b) the pre_OS reset the card when it was done. Either way if its already 0 then we're in a known state so we do nothing and then later we set to 1 and wait on RDY. If its 1 when we enter here we set to 0 and then later set to 1 and wait on RDY. Make sense? This assures we never enable without waiting for RDY ***** WAS ******* /* Set EN to 1 (all others zero) */ CC.EN = 1; StorPortWriteRegisterUlong(pAE, (PULONG)(&pAE->pCtrlRegister->CC), CC.AsUlong); /* Read it back */ CC.AsUlong = StorPortReadRegisterUlong(pAE, (PULONG)(&pAE->pCtrlRegister->CC)); /* Now reset */ CC.EN = 0; StorPortWriteRegisterUlong(pAE, (PULONG)(&pAE->pCtrlRegister->CC), CC.AsUlong); CC.AsUlong = StorPortReadRegisterUlong(pAE, (PULONG)(&pAE->pCtrlRegister->CC)); *** NOW IS ******* if (CC.EN == 1) { /* Now reset */ CC.EN = 0; StorPortWriteRegisterUlong(pAE, (PULONG)(&pAE->pCtrlRegister->CC), CC.AsUlong); } Thanks, Alex ________________________________ From:nvmewin-bounces at lists.openfabrics.org[mailto:nvmewin-bounces at lists.openfabrics.org]On Behalf Of Luse, Paul E Sent: Wednesday, August 15, 2012 8:12 AM To: nvmewin at lists.openfabrics.org Subject: [nvmewin] patch review status Just checking in... any questions to date and how are folks tracking for completing their review by this Fri? Thx Paul ____________________________________ Paul Luse Sr. Staff Engineer PCG Server Software Engineering Desk: 480.554.3688, Mobile: 480.334.4630 -------------- next part -------------- An HTML attachment was scrubbed... URL: From paul.e.luse at intel.com Wed Aug 15 16:47:17 2012 From: paul.e.luse at intel.com (Luse, Paul E) Date: Wed, 15 Aug 2012 23:47:17 +0000 Subject: [nvmewin] patch review status In-Reply-To: <718182ECAE1D7149A1BF0C789EAC6909049B647C52@AUSX7MCPC110.AMER.DELL.COM> References: <82C9F782B054C94B9FC04A331649C77A07B2B0C9@FMSMSX106.amr.corp.intel.com> <82C9F782B054C94B9FC04A331649C77A07B2B9EB@FMSMSX106.amr.corp.intel.com> <718182ECAE1D7149A1BF0C789EAC6909049B647C47@AUSX7MCPC110.AMER.DELL.COM> <82C9F782B054C94B9FC04A331649C77A07B2BAA5@FMSMSX106.amr.corp.intel.com> <718182ECAE1D7149A1BF0C789EAC6909049B647C52@AUSX7MCPC110.AMER.DELL.COM> Message-ID: <82C9F782B054C94B9FC04A331649C77A07B2BB67@FMSMSX106.amr.corp.intel.com> To your immediate email: I don't know what you're trying to say To your email before that: You realize that's what's in the patch I sent out initially right? We've come full circle here, Alex suggested that I add a wait for RDY just before setting to 0 as we don't know if its safe to set to 0 yet and then further suggested the RDY check instead of EN to avoid having to deal with what we do if RDY never comes up. Additionally, this approach does not reset the card at startup. If its 0 to being with, we simply move it to 1 which you would *assume* it OK but I have at least one IHV report that its not good enough, a 1?0 transition is needed at init. So, I still think this is the most generic and safest: IF EN == 0 { /* Set EN to 1 (all others zero) */ CC.EN = 1; StorPortWriteRegisterUlong(pAE, (PULONG)(&pAE->pCtrlRegister->CC), CC.AsUlong); WAIT ON RDY } /* Now reset */ CC.EN = 0; StorPortWriteRegisterUlong(pAE, (PULONG)(&pAE->pCtrlRegister->CC), CC.AsUlong); From: Karthik_Rajagopalan at Dell.com [mailto:Karthik_Rajagopalan at Dell.com] Sent: Wednesday, August 15, 2012 4:43 PM To: Luse, Paul E; Verma, Vishal L; Alex.Chang at idt.com; nvmewin at lists.openfabrics.org Subject: RE: [nvmewin] patch review status BTW, here's what the spec says for CC.EN When this field is set to '1', the controller sets CSTS.RDY to '1' when it is ready to process commands. I read it as "set CC.EN to 1 after setting up the admin-queues, otherwise RDY will *not* be 1" From: Rajagopalan, Karthik Sent: Wednesday, August 15, 2012 6:32 PM To: 'Luse, Paul E'; Verma, Vishal L; Alex.Chang at idt.com; nvmewin at lists.openfabrics.org Subject: RE: [nvmewin] patch review status Yep :) but I think it doesn't have to be so - all we need to do is an unconditional (attempt) to reset.. we don't set CC.EN to 1 until we're ready to enable the controller. Here are the possibilities when we try to reset: 1. CC.EN is 0 to begin with => no need to reset; but setting it again to 0 should be a NOP 2. CC.EN is 1 to begin with => need to reset; we didn't wiggle this yet, so STS.RDY should be 1; setting EN to 0 will actually reset a. The original issue you mention could still happen if STS.RDY stayed 0 (bad controller IMHO); still goodness in trying to reset Here's a patch in case you want to try this approach... From: Luse, Paul E [mailto:paul.e.luse at intel.com] Sent: Wednesday, August 15, 2012 6:22 PM To: Rajagopalan, Karthik; Verma, Vishal L; Alex.Chang at idt.com; nvmewin at lists.openfabrics.org Subject: RE: [nvmewin] patch review status 0 doesn't reset unless it was previously 1... that's why the initial implementation set it to 1, then 0 to force reset then init happened. The question came in about setting it to 0 right after setting it to 1, anytime you transition to 1 you need to wait for RDY so the code I proposed last will always reset but only for a 2nd wait on RDY (there's one after we enable as part of init) if the controller wasn't disabled when we started. Hard to believe something so simple can be so convoluted isn't it? From: Karthik_Rajagopalan at Dell.com [mailto:Karthik_Rajagopalan at Dell.com] Sent: Wednesday, August 15, 2012 4:11 PM To: Luse, Paul E; Verma, Vishal L; Alex.Chang at idt.com; nvmewin at lists.openfabrics.org Subject: RE: [nvmewin] patch review status Why not unconditionally set CC.EN to 0 as soon as the driver loads ? - There will be an issue with this iff someone before (opROM, UEFI ?) had set CC.EN to 1 & somehow STS.RDY never went to 1... but things still continued (unlikely) i.e., /* Disable / Reset */ CC.EN = 0; StorPortWriteRegisterUlong(pAE, (PULONG)(&pAE->pCtrlRegister->CC), CC.AsUlong); /* Set EN to 1 (all others zero) */ CC.EN = 1; StorPortWriteRegisterUlong(pAE, (PULONG)(&pAE->pCtrlRegister->CC), CC.AsUlong); /* Wait for RDY */ From: nvmewin-bounces at lists.openfabrics.org [mailto:nvmewin-bounces at lists.openfabrics.org] On Behalf Of Luse, Paul E Sent: Wednesday, August 15, 2012 5:59 PM To: Verma, Vishal L; Chang, Alex; nvmewin at lists.openfabrics.org Subject: Re: [nvmewin] patch review status So maybe a better proposal for this section of code... go back to what we had before (sorta) but include a wait on RDY after setting EN to 1 and only set EN to 1 if its zero to start. This is the most generic soln and since the behavior of the HW at this detailed level isn't specified we can call this the least common denominator and if IHVs want to optimize for their implementations they can. If this first RDY wait files, then we fail driver init. This code assures we always reset and we never touch EN after setting it to 1 without waiting for RDY IF EN == 0 { /* Set EN to 1 (all others zero) */ CC.EN = 1; StorPortWriteRegisterUlong(pAE, (PULONG)(&pAE->pCtrlRegister->CC), CC.AsUlong); WAIT ON RDY } /* Now reset */ CC.EN = 0; StorPortWriteRegisterUlong(pAE, (PULONG)(&pAE->pCtrlRegister->CC), CC.AsUlong); From: Luse, Paul E Sent: Wednesday, August 15, 2012 3:21 PM To: Verma, Vishal L; Chang, Alex; nvmewin at lists.openfabrics.org Subject: RE: [nvmewin] patch review status I was thinking about that too... If EN was already 1 and we caught RDY before it was 0 then we would indeed set EN to 1 again (possibly before, during or after RDY is set). The spec, of course, does not dictate behavior in this case but one would assume it would be treated as a NOP and our init state machine would properly wait for RDY. I'm not a HW guy though, anyone else have a comment/concern on setting EN to 1 if tis already 1? From: Verma, Vishal L Sent: Wednesday, August 15, 2012 1:46 PM To: Luse, Paul E; Chang, Alex; nvmewin at lists.openfabrics.org Subject: Re: [nvmewin] patch review status Paul, Is this correct though? Making CC.EN 0 implies that RDY will become 0, but RDY being 0 does not guarantee that CC.EN is 0. If what I'm thinking is right, we could be in a state where CC.EN was 1 when the driver comes up, but the controller never became ready. In this case we will think everything is alright, and we will try to make CC.EN 1 again - Which is not something we'd want to do, right? Vishal From: , Paul E > Date: Wednesday, August 15, 2012 11:43 AM To: "Chang, Alex" >, "nvmewin at lists.openfabrics.org" > Subject: Re: [nvmewin] patch review status OK, I think I can get behind that. I was thinking STS might not be valid when EN is 0 but looking at the spec I think this is fine Ready (RDY): This field is set to '1' when the controller is ready to process commands after CC.EN is set to '1'. This field shall be cleared to '0' when CC.EN is cleared to '0'. Commands shall not be issued to the controller until this field is set to '1' after the CC.EN bit is set to '1'. Failure to follow this requirement produces undefined results. Host software shall wait a minimum of CAP.TO seconds for this field to be set to '1' after setting CC.EN to '1' from a previous value of '0'. So I'll just chance the check below for EN == 1 to RDY == 1 then Thanks! Paul From: Chang, Alex [mailto:Alex.Chang at idt.com] Sent: Wednesday, August 15, 2012 10:21 AM To: Luse, Paul E; nvmewin at lists.openfabrics.org Subject: RE: patch review status Absolutely. Now, the tricky part is when CC.EN=1 and STS.RDY never becomes 1. I'd say simply check STS.RDY only. If it's 1, then reset the device, otherwise, skip resetting it. What do you think? Thanks, Alex ________________________________ From: Luse, Paul E [mailto:paul.e.luse at intel.com] Sent: Wednesday, August 15, 2012 9:56 AM To: Chang, Alex; nvmewin at lists.openfabrics.org Subject: RE: patch review status OK, I see what you're saying I think. We don't want to check for RDY without first looking at EN so does this make sense: if (CC.EN == 1) { -- now wait for RDY - /* Now reset */ CC.EN = 0; StorPortWriteRegisterUlong(pAE, (PULONG)(&pAE->pCtrlRegister->CC), CC.AsUlong); } From: Chang, Alex [mailto:Alex.Chang at idt.com] Sent: Wednesday, August 15, 2012 9:52 AM To: Luse, Paul E; nvmewin at lists.openfabrics.org Subject: RE: patch review status Hi Paul, I agree with the new logic on #3. However, instead of checking CC.EN, should we check STS.RDY or both? Thanks, Alex ________________________________ From: Luse, Paul E [mailto:paul.e.luse at intel.com] Sent: Wednesday, August 15, 2012 9:03 AM To: Chang, Alex; nvmewin at lists.openfabrics.org Subject: RE: patch review status Thanks Alex, see below. From: Chang, Alex [mailto:Alex.Chang at idt.com] Sent: Wednesday, August 15, 2012 8:51 AM To: Luse, Paul E; nvmewin at lists.openfabrics.org Subject: RE: patch review status Hi Paul, Some questions after browsing the difference between current source and your changes: 1. Why you removed "CC.SHN = 1;" in NVMeNormalShutdown routine? PL> Good eye, that was a mistake. I'll add it back before pushing 2. In nvme.h, why you changed the opcode of NVM_DATASET_MANAGEMENT from 0x06 to 0x09? PL> 6 was wrong, 9 is correct per spec 3. Is there any specific reason(s), you added the following in NVMeInitialize, and then immediately clear EN without checking RDY = 1? /* Set EN to 1 (all others zero) */ CC.EN = 1; StorPortWriteRegisterUlong(pAE, (PULONG)(&pAE->pCtrlRegister->CC), CC.AsUlong); PL> I wrote this up in the patch submission and I can't find that email right now. I didn't add code, I changed code. Before, we use to set EN to 1, read it back, and then set it to 0 to kick off a reset and then later we wait for RDY in the init state machine. We had an IHV complain that their implementation required waiting on RDY after setting EN 1 to even before clearing EN. So, our old implementation was a problem. So what I did now is to not force a reset unless EN is set to 1 when we come in. If its set to 0 then either (a) we're coming from power on or (b) the pre_OS reset the card when it was done. Either way if its already 0 then we're in a known state so we do nothing and then later we set to 1 and wait on RDY. If its 1 when we enter here we set to 0 and then later set to 1 and wait on RDY. Make sense? This assures we never enable without waiting for RDY ***** WAS ******* /* Set EN to 1 (all others zero) */ CC.EN = 1; StorPortWriteRegisterUlong(pAE, (PULONG)(&pAE->pCtrlRegister->CC), CC.AsUlong); /* Read it back */ CC.AsUlong = StorPortReadRegisterUlong(pAE, (PULONG)(&pAE->pCtrlRegister->CC)); /* Now reset */ CC.EN = 0; StorPortWriteRegisterUlong(pAE, (PULONG)(&pAE->pCtrlRegister->CC), CC.AsUlong); CC.AsUlong = StorPortReadRegisterUlong(pAE, (PULONG)(&pAE->pCtrlRegister->CC)); *** NOW IS ******* if (CC.EN == 1) { /* Now reset */ CC.EN = 0; StorPortWriteRegisterUlong(pAE, (PULONG)(&pAE->pCtrlRegister->CC), CC.AsUlong); } Thanks, Alex ________________________________ From:nvmewin-bounces at lists.openfabrics.org[mailto:nvmewin-bounces at lists.openfabrics.org]On Behalf Of Luse, Paul E Sent: Wednesday, August 15, 2012 8:12 AM To: nvmewin at lists.openfabrics.org Subject: [nvmewin] patch review status Just checking in... any questions to date and how are folks tracking for completing their review by this Fri? Thx Paul ____________________________________ Paul Luse Sr. Staff Engineer PCG Server Software Engineering Desk: 480.554.3688, Mobile: 480.334.4630 -------------- next part -------------- An HTML attachment was scrubbed... URL: From Karthik_Rajagopalan at Dell.com Wed Aug 15 16:58:14 2012 From: Karthik_Rajagopalan at Dell.com (Karthik_Rajagopalan at Dell.com) Date: Wed, 15 Aug 2012 18:58:14 -0500 Subject: [nvmewin] patch review status In-Reply-To: <82C9F782B054C94B9FC04A331649C77A07B2BB67@FMSMSX106.amr.corp.intel.com> References: <82C9F782B054C94B9FC04A331649C77A07B2B0C9@FMSMSX106.amr.corp.intel.com> <82C9F782B054C94B9FC04A331649C77A07B2B9EB@FMSMSX106.amr.corp.intel.com> <718182ECAE1D7149A1BF0C789EAC6909049B647C47@AUSX7MCPC110.AMER.DELL.COM> <82C9F782B054C94B9FC04A331649C77A07B2BAA5@FMSMSX106.amr.corp.intel.com> <718182ECAE1D7149A1BF0C789EAC6909049B647C52@AUSX7MCPC110.AMER.DELL.COM> <82C9F782B054C94B9FC04A331649C77A07B2BB67@FMSMSX106.amr.corp.intel.com> Message-ID: <718182ECAE1D7149A1BF0C789EAC6909049B647C5B@AUSX7MCPC110.AMER.DELL.COM> Setting CC.EN to 0 will satisfy the 1-->0 transition. >To your immediate email: I don't know what you're trying to say What I've removed in the patch is the 0-->1 transition... per the spec it can happen only after we've setup the AQA, ASQ, ACQ entries in MMIO (EnableAdapter). - Otherwise, the controller doesn't have to set STS.RDY - so we'll in fact be waiting forever for RDY From: Luse, Paul E [mailto:paul.e.luse at intel.com] Sent: Wednesday, August 15, 2012 6:47 PM To: Rajagopalan, Karthik; Verma, Vishal L; Alex.Chang at idt.com; nvmewin at lists.openfabrics.org Subject: RE: [nvmewin] patch review status To your immediate email: I don't know what you're trying to say To your email before that: You realize that's what's in the patch I sent out initially right? We've come full circle here, Alex suggested that I add a wait for RDY just before setting to 0 as we don't know if its safe to set to 0 yet and then further suggested the RDY check instead of EN to avoid having to deal with what we do if RDY never comes up. Additionally, this approach does not reset the card at startup. If its 0 to being with, we simply move it to 1 which you would *assume* it OK but I have at least one IHV report that its not good enough, a 1-->0 transition is needed at init. So, I still think this is the most generic and safest: IF EN == 0 { /* Set EN to 1 (all others zero) */ CC.EN = 1; StorPortWriteRegisterUlong(pAE, (PULONG)(&pAE->pCtrlRegister->CC), CC.AsUlong); WAIT ON RDY } /* Now reset */ CC.EN = 0; StorPortWriteRegisterUlong(pAE, (PULONG)(&pAE->pCtrlRegister->CC), CC.AsUlong); From: Karthik_Rajagopalan at Dell.com [mailto:Karthik_Rajagopalan at Dell.com] Sent: Wednesday, August 15, 2012 4:43 PM To: Luse, Paul E; Verma, Vishal L; Alex.Chang at idt.com; nvmewin at lists.openfabrics.org Subject: RE: [nvmewin] patch review status BTW, here's what the spec says for CC.EN When this field is set to '1', the controller sets CSTS.RDY to '1' when it is ready to process commands. I read it as "set CC.EN to 1 after setting up the admin-queues, otherwise RDY will *not* be 1" From: Rajagopalan, Karthik Sent: Wednesday, August 15, 2012 6:32 PM To: 'Luse, Paul E'; Verma, Vishal L; Alex.Chang at idt.com; nvmewin at lists.openfabrics.org Subject: RE: [nvmewin] patch review status Yep :) but I think it doesn't have to be so - all we need to do is an unconditional (attempt) to reset.. we don't set CC.EN to 1 until we're ready to enable the controller. Here are the possibilities when we try to reset: 1. CC.EN is 0 to begin with => no need to reset; but setting it again to 0 should be a NOP 2. CC.EN is 1 to begin with => need to reset; we didn't wiggle this yet, so STS.RDY should be 1; setting EN to 0 will actually reset a. The original issue you mention could still happen if STS.RDY stayed 0 (bad controller IMHO); still goodness in trying to reset Here's a patch in case you want to try this approach... From: Luse, Paul E [mailto:paul.e.luse at intel.com] Sent: Wednesday, August 15, 2012 6:22 PM To: Rajagopalan, Karthik; Verma, Vishal L; Alex.Chang at idt.com; nvmewin at lists.openfabrics.org Subject: RE: [nvmewin] patch review status 0 doesn't reset unless it was previously 1... that's why the initial implementation set it to 1, then 0 to force reset then init happened. The question came in about setting it to 0 right after setting it to 1, anytime you transition to 1 you need to wait for RDY so the code I proposed last will always reset but only for a 2nd wait on RDY (there's one after we enable as part of init) if the controller wasn't disabled when we started. Hard to believe something so simple can be so convoluted isn't it? From: Karthik_Rajagopalan at Dell.com [mailto:Karthik_Rajagopalan at Dell.com] Sent: Wednesday, August 15, 2012 4:11 PM To: Luse, Paul E; Verma, Vishal L; Alex.Chang at idt.com; nvmewin at lists.openfabrics.org Subject: RE: [nvmewin] patch review status Why not unconditionally set CC.EN to 0 as soon as the driver loads ? - There will be an issue with this iff someone before (opROM, UEFI ?) had set CC.EN to 1 & somehow STS.RDY never went to 1... but things still continued (unlikely) i.e., /* Disable / Reset */ CC.EN = 0; StorPortWriteRegisterUlong(pAE, (PULONG)(&pAE->pCtrlRegister->CC), CC.AsUlong); /* Set EN to 1 (all others zero) */ CC.EN = 1; StorPortWriteRegisterUlong(pAE, (PULONG)(&pAE->pCtrlRegister->CC), CC.AsUlong); /* Wait for RDY */ From: nvmewin-bounces at lists.openfabrics.org [mailto:nvmewin-bounces at lists.openfabrics.org] On Behalf Of Luse, Paul E Sent: Wednesday, August 15, 2012 5:59 PM To: Verma, Vishal L; Chang, Alex; nvmewin at lists.openfabrics.org Subject: Re: [nvmewin] patch review status So maybe a better proposal for this section of code... go back to what we had before (sorta) but include a wait on RDY after setting EN to 1 and only set EN to 1 if its zero to start. This is the most generic soln and since the behavior of the HW at this detailed level isn't specified we can call this the least common denominator and if IHVs want to optimize for their implementations they can. If this first RDY wait files, then we fail driver init. This code assures we always reset and we never touch EN after setting it to 1 without waiting for RDY IF EN == 0 { /* Set EN to 1 (all others zero) */ CC.EN = 1; StorPortWriteRegisterUlong(pAE, (PULONG)(&pAE->pCtrlRegister->CC), CC.AsUlong); WAIT ON RDY } /* Now reset */ CC.EN = 0; StorPortWriteRegisterUlong(pAE, (PULONG)(&pAE->pCtrlRegister->CC), CC.AsUlong); From: Luse, Paul E Sent: Wednesday, August 15, 2012 3:21 PM To: Verma, Vishal L; Chang, Alex; nvmewin at lists.openfabrics.org Subject: RE: [nvmewin] patch review status I was thinking about that too... If EN was already 1 and we caught RDY before it was 0 then we would indeed set EN to 1 again (possibly before, during or after RDY is set). The spec, of course, does not dictate behavior in this case but one would assume it would be treated as a NOP and our init state machine would properly wait for RDY. I'm not a HW guy though, anyone else have a comment/concern on setting EN to 1 if tis already 1? From: Verma, Vishal L Sent: Wednesday, August 15, 2012 1:46 PM To: Luse, Paul E; Chang, Alex; nvmewin at lists.openfabrics.org Subject: Re: [nvmewin] patch review status Paul, Is this correct though? Making CC.EN 0 implies that RDY will become 0, but RDY being 0 does not guarantee that CC.EN is 0. If what I'm thinking is right, we could be in a state where CC.EN was 1 when the driver comes up, but the controller never became ready. In this case we will think everything is alright, and we will try to make CC.EN 1 again - Which is not something we'd want to do, right? Vishal From: , Paul E > Date: Wednesday, August 15, 2012 11:43 AM To: "Chang, Alex" >, "nvmewin at lists.openfabrics.org" > Subject: Re: [nvmewin] patch review status OK, I think I can get behind that. I was thinking STS might not be valid when EN is 0 but looking at the spec I think this is fine Ready (RDY): This field is set to '1' when the controller is ready to process commands after CC.EN is set to '1'. This field shall be cleared to '0' when CC.EN is cleared to '0'. Commands shall not be issued to the controller until this field is set to '1' after the CC.EN bit is set to '1'. Failure to follow this requirement produces undefined results. Host software shall wait a minimum of CAP.TO seconds for this field to be set to '1' after setting CC.EN to '1' from a previous value of '0'. So I'll just chance the check below for EN == 1 to RDY == 1 then Thanks! Paul From: Chang, Alex [mailto:Alex.Chang at idt.com] Sent: Wednesday, August 15, 2012 10:21 AM To: Luse, Paul E; nvmewin at lists.openfabrics.org Subject: RE: patch review status Absolutely. Now, the tricky part is when CC.EN=1 and STS.RDY never becomes 1. I'd say simply check STS.RDY only. If it's 1, then reset the device, otherwise, skip resetting it. What do you think? Thanks, Alex ________________________________ From: Luse, Paul E [mailto:paul.e.luse at intel.com] Sent: Wednesday, August 15, 2012 9:56 AM To: Chang, Alex; nvmewin at lists.openfabrics.org Subject: RE: patch review status OK, I see what you're saying I think. We don't want to check for RDY without first looking at EN so does this make sense: if (CC.EN == 1) { -- now wait for RDY - /* Now reset */ CC.EN = 0; StorPortWriteRegisterUlong(pAE, (PULONG)(&pAE->pCtrlRegister->CC), CC.AsUlong); } From: Chang, Alex [mailto:Alex.Chang at idt.com] Sent: Wednesday, August 15, 2012 9:52 AM To: Luse, Paul E; nvmewin at lists.openfabrics.org Subject: RE: patch review status Hi Paul, I agree with the new logic on #3. However, instead of checking CC.EN, should we check STS.RDY or both? Thanks, Alex ________________________________ From: Luse, Paul E [mailto:paul.e.luse at intel.com] Sent: Wednesday, August 15, 2012 9:03 AM To: Chang, Alex; nvmewin at lists.openfabrics.org Subject: RE: patch review status Thanks Alex, see below. From: Chang, Alex [mailto:Alex.Chang at idt.com] Sent: Wednesday, August 15, 2012 8:51 AM To: Luse, Paul E; nvmewin at lists.openfabrics.org Subject: RE: patch review status Hi Paul, Some questions after browsing the difference between current source and your changes: 1. Why you removed "CC.SHN = 1;" in NVMeNormalShutdown routine? PL> Good eye, that was a mistake. I'll add it back before pushing 2. In nvme.h, why you changed the opcode of NVM_DATASET_MANAGEMENT from 0x06 to 0x09? PL> 6 was wrong, 9 is correct per spec 3. Is there any specific reason(s), you added the following in NVMeInitialize, and then immediately clear EN without checking RDY = 1? /* Set EN to 1 (all others zero) */ CC.EN = 1; StorPortWriteRegisterUlong(pAE, (PULONG)(&pAE->pCtrlRegister->CC), CC.AsUlong); PL> I wrote this up in the patch submission and I can't find that email right now. I didn't add code, I changed code. Before, we use to set EN to 1, read it back, and then set it to 0 to kick off a reset and then later we wait for RDY in the init state machine. We had an IHV complain that their implementation required waiting on RDY after setting EN 1 to even before clearing EN. So, our old implementation was a problem. So what I did now is to not force a reset unless EN is set to 1 when we come in. If its set to 0 then either (a) we're coming from power on or (b) the pre_OS reset the card when it was done. Either way if its already 0 then we're in a known state so we do nothing and then later we set to 1 and wait on RDY. If its 1 when we enter here we set to 0 and then later set to 1 and wait on RDY. Make sense? This assures we never enable without waiting for RDY ***** WAS ******* /* Set EN to 1 (all others zero) */ CC.EN = 1; StorPortWriteRegisterUlong(pAE, (PULONG)(&pAE->pCtrlRegister->CC), CC.AsUlong); /* Read it back */ CC.AsUlong = StorPortReadRegisterUlong(pAE, (PULONG)(&pAE->pCtrlRegister->CC)); /* Now reset */ CC.EN = 0; StorPortWriteRegisterUlong(pAE, (PULONG)(&pAE->pCtrlRegister->CC), CC.AsUlong); CC.AsUlong = StorPortReadRegisterUlong(pAE, (PULONG)(&pAE->pCtrlRegister->CC)); *** NOW IS ******* if (CC.EN == 1) { /* Now reset */ CC.EN = 0; StorPortWriteRegisterUlong(pAE, (PULONG)(&pAE->pCtrlRegister->CC), CC.AsUlong); } Thanks, Alex ________________________________ From:nvmewin-bounces at lists.openfabrics.org[mailto:nvmewin-bounces at lists.openfabrics.org]On Behalf Of Luse, Paul E Sent: Wednesday, August 15, 2012 8:12 AM To: nvmewin at lists.openfabrics.org Subject: [nvmewin] patch review status Just checking in... any questions to date and how are folks tracking for completing their review by this Fri? Thx Paul ____________________________________ Paul Luse Sr. Staff Engineer PCG Server Software Engineering Desk: 480.554.3688, Mobile: 480.334.4630 -------------- next part -------------- An HTML attachment was scrubbed... URL: From paul.e.luse at intel.com Wed Aug 15 17:11:18 2012 From: paul.e.luse at intel.com (Luse, Paul E) Date: Thu, 16 Aug 2012 00:11:18 +0000 Subject: [nvmewin] patch review status In-Reply-To: <718182ECAE1D7149A1BF0C789EAC6909049B647C5B@AUSX7MCPC110.AMER.DELL.COM> References: <82C9F782B054C94B9FC04A331649C77A07B2B0C9@FMSMSX106.amr.corp.intel.com> <82C9F782B054C94B9FC04A331649C77A07B2B9EB@FMSMSX106.amr.corp.intel.com> <718182ECAE1D7149A1BF0C789EAC6909049B647C47@AUSX7MCPC110.AMER.DELL.COM> <82C9F782B054C94B9FC04A331649C77A07B2BAA5@FMSMSX106.amr.corp.intel.com> <718182ECAE1D7149A1BF0C789EAC6909049B647C52@AUSX7MCPC110.AMER.DELL.COM> <82C9F782B054C94B9FC04A331649C77A07B2BB67@FMSMSX106.amr.corp.intel.com> <718182ECAE1D7149A1BF0C789EAC6909049B647C5B@AUSX7MCPC110.AMER.DELL.COM> Message-ID: <82C9F782B054C94B9FC04A331649C77A07B2BC46@FMSMSX106.amr.corp.intel.com> OK, now I understand what you are saying but I don't think the spec is clear enough either way to make a black and white decision (and I bet of the 3 IHV boards I have access to, there's at least 1 that behaves differently than the other 2 in this regard). I have a plane to catch so will think about this overnight. I'd welcome other comments/opinions as well and thanks everyone so far for the inpits! Paul From: Karthik_Rajagopalan at Dell.com [mailto:Karthik_Rajagopalan at Dell.com] Sent: Wednesday, August 15, 2012 4:58 PM To: Luse, Paul E; Verma, Vishal L; Alex.Chang at idt.com; nvmewin at lists.openfabrics.org Subject: RE: [nvmewin] patch review status Setting CC.EN to 0 will satisfy the 1-->0 transition. >To your immediate email: I don't know what you're trying to say What I've removed in the patch is the 0-->1 transition... per the spec it can happen only after we've setup the AQA, ASQ, ACQ entries in MMIO (EnableAdapter). - Otherwise, the controller doesn't have to set STS.RDY - so we'll in fact be waiting forever for RDY From: Luse, Paul E [mailto:paul.e.luse at intel.com] Sent: Wednesday, August 15, 2012 6:47 PM To: Rajagopalan, Karthik; Verma, Vishal L; Alex.Chang at idt.com; nvmewin at lists.openfabrics.org Subject: RE: [nvmewin] patch review status To your immediate email: I don't know what you're trying to say To your email before that: You realize that's what's in the patch I sent out initially right? We've come full circle here, Alex suggested that I add a wait for RDY just before setting to 0 as we don't know if its safe to set to 0 yet and then further suggested the RDY check instead of EN to avoid having to deal with what we do if RDY never comes up. Additionally, this approach does not reset the card at startup. If its 0 to being with, we simply move it to 1 which you would *assume* it OK but I have at least one IHV report that its not good enough, a 1-->0 transition is needed at init. So, I still think this is the most generic and safest: IF EN == 0 { /* Set EN to 1 (all others zero) */ CC.EN = 1; StorPortWriteRegisterUlong(pAE, (PULONG)(&pAE->pCtrlRegister->CC), CC.AsUlong); WAIT ON RDY } /* Now reset */ CC.EN = 0; StorPortWriteRegisterUlong(pAE, (PULONG)(&pAE->pCtrlRegister->CC), CC.AsUlong); From: Karthik_Rajagopalan at Dell.com [mailto:Karthik_Rajagopalan at Dell.com] Sent: Wednesday, August 15, 2012 4:43 PM To: Luse, Paul E; Verma, Vishal L; Alex.Chang at idt.com; nvmewin at lists.openfabrics.org Subject: RE: [nvmewin] patch review status BTW, here's what the spec says for CC.EN When this field is set to '1', the controller sets CSTS.RDY to '1' when it is ready to process commands. I read it as "set CC.EN to 1 after setting up the admin-queues, otherwise RDY will *not* be 1" From: Rajagopalan, Karthik Sent: Wednesday, August 15, 2012 6:32 PM To: 'Luse, Paul E'; Verma, Vishal L; Alex.Chang at idt.com; nvmewin at lists.openfabrics.org Subject: RE: [nvmewin] patch review status Yep :) but I think it doesn't have to be so - all we need to do is an unconditional (attempt) to reset.. we don't set CC.EN to 1 until we're ready to enable the controller. Here are the possibilities when we try to reset: 1. CC.EN is 0 to begin with => no need to reset; but setting it again to 0 should be a NOP 2. CC.EN is 1 to begin with => need to reset; we didn't wiggle this yet, so STS.RDY should be 1; setting EN to 0 will actually reset a. The original issue you mention could still happen if STS.RDY stayed 0 (bad controller IMHO); still goodness in trying to reset Here's a patch in case you want to try this approach... From: Luse, Paul E [mailto:paul.e.luse at intel.com] Sent: Wednesday, August 15, 2012 6:22 PM To: Rajagopalan, Karthik; Verma, Vishal L; Alex.Chang at idt.com; nvmewin at lists.openfabrics.org Subject: RE: [nvmewin] patch review status 0 doesn't reset unless it was previously 1... that's why the initial implementation set it to 1, then 0 to force reset then init happened. The question came in about setting it to 0 right after setting it to 1, anytime you transition to 1 you need to wait for RDY so the code I proposed last will always reset but only for a 2nd wait on RDY (there's one after we enable as part of init) if the controller wasn't disabled when we started. Hard to believe something so simple can be so convoluted isn't it? From: Karthik_Rajagopalan at Dell.com [mailto:Karthik_Rajagopalan at Dell.com] Sent: Wednesday, August 15, 2012 4:11 PM To: Luse, Paul E; Verma, Vishal L; Alex.Chang at idt.com; nvmewin at lists.openfabrics.org Subject: RE: [nvmewin] patch review status Why not unconditionally set CC.EN to 0 as soon as the driver loads ? - There will be an issue with this iff someone before (opROM, UEFI ?) had set CC.EN to 1 & somehow STS.RDY never went to 1... but things still continued (unlikely) i.e., /* Disable / Reset */ CC.EN = 0; StorPortWriteRegisterUlong(pAE, (PULONG)(&pAE->pCtrlRegister->CC), CC.AsUlong); /* Set EN to 1 (all others zero) */ CC.EN = 1; StorPortWriteRegisterUlong(pAE, (PULONG)(&pAE->pCtrlRegister->CC), CC.AsUlong); /* Wait for RDY */ From: nvmewin-bounces at lists.openfabrics.org [mailto:nvmewin-bounces at lists.openfabrics.org] On Behalf Of Luse, Paul E Sent: Wednesday, August 15, 2012 5:59 PM To: Verma, Vishal L; Chang, Alex; nvmewin at lists.openfabrics.org Subject: Re: [nvmewin] patch review status So maybe a better proposal for this section of code... go back to what we had before (sorta) but include a wait on RDY after setting EN to 1 and only set EN to 1 if its zero to start. This is the most generic soln and since the behavior of the HW at this detailed level isn't specified we can call this the least common denominator and if IHVs want to optimize for their implementations they can. If this first RDY wait files, then we fail driver init. This code assures we always reset and we never touch EN after setting it to 1 without waiting for RDY IF EN == 0 { /* Set EN to 1 (all others zero) */ CC.EN = 1; StorPortWriteRegisterUlong(pAE, (PULONG)(&pAE->pCtrlRegister->CC), CC.AsUlong); WAIT ON RDY } /* Now reset */ CC.EN = 0; StorPortWriteRegisterUlong(pAE, (PULONG)(&pAE->pCtrlRegister->CC), CC.AsUlong); From: Luse, Paul E Sent: Wednesday, August 15, 2012 3:21 PM To: Verma, Vishal L; Chang, Alex; nvmewin at lists.openfabrics.org Subject: RE: [nvmewin] patch review status I was thinking about that too... If EN was already 1 and we caught RDY before it was 0 then we would indeed set EN to 1 again (possibly before, during or after RDY is set). The spec, of course, does not dictate behavior in this case but one would assume it would be treated as a NOP and our init state machine would properly wait for RDY. I'm not a HW guy though, anyone else have a comment/concern on setting EN to 1 if tis already 1? From: Verma, Vishal L Sent: Wednesday, August 15, 2012 1:46 PM To: Luse, Paul E; Chang, Alex; nvmewin at lists.openfabrics.org Subject: Re: [nvmewin] patch review status Paul, Is this correct though? Making CC.EN 0 implies that RDY will become 0, but RDY being 0 does not guarantee that CC.EN is 0. If what I'm thinking is right, we could be in a state where CC.EN was 1 when the driver comes up, but the controller never became ready. In this case we will think everything is alright, and we will try to make CC.EN 1 again - Which is not something we'd want to do, right? Vishal From: , Paul E > Date: Wednesday, August 15, 2012 11:43 AM To: "Chang, Alex" >, "nvmewin at lists.openfabrics.org" > Subject: Re: [nvmewin] patch review status OK, I think I can get behind that. I was thinking STS might not be valid when EN is 0 but looking at the spec I think this is fine Ready (RDY): This field is set to '1' when the controller is ready to process commands after CC.EN is set to '1'. This field shall be cleared to '0' when CC.EN is cleared to '0'. Commands shall not be issued to the controller until this field is set to '1' after the CC.EN bit is set to '1'. Failure to follow this requirement produces undefined results. Host software shall wait a minimum of CAP.TO seconds for this field to be set to '1' after setting CC.EN to '1' from a previous value of '0'. So I'll just chance the check below for EN == 1 to RDY == 1 then Thanks! Paul From: Chang, Alex [mailto:Alex.Chang at idt.com] Sent: Wednesday, August 15, 2012 10:21 AM To: Luse, Paul E; nvmewin at lists.openfabrics.org Subject: RE: patch review status Absolutely. Now, the tricky part is when CC.EN=1 and STS.RDY never becomes 1. I'd say simply check STS.RDY only. If it's 1, then reset the device, otherwise, skip resetting it. What do you think? Thanks, Alex ________________________________ From: Luse, Paul E [mailto:paul.e.luse at intel.com] Sent: Wednesday, August 15, 2012 9:56 AM To: Chang, Alex; nvmewin at lists.openfabrics.org Subject: RE: patch review status OK, I see what you're saying I think. We don't want to check for RDY without first looking at EN so does this make sense: if (CC.EN == 1) { -- now wait for RDY - /* Now reset */ CC.EN = 0; StorPortWriteRegisterUlong(pAE, (PULONG)(&pAE->pCtrlRegister->CC), CC.AsUlong); } From: Chang, Alex [mailto:Alex.Chang at idt.com] Sent: Wednesday, August 15, 2012 9:52 AM To: Luse, Paul E; nvmewin at lists.openfabrics.org Subject: RE: patch review status Hi Paul, I agree with the new logic on #3. However, instead of checking CC.EN, should we check STS.RDY or both? Thanks, Alex ________________________________ From: Luse, Paul E [mailto:paul.e.luse at intel.com] Sent: Wednesday, August 15, 2012 9:03 AM To: Chang, Alex; nvmewin at lists.openfabrics.org Subject: RE: patch review status Thanks Alex, see below. From: Chang, Alex [mailto:Alex.Chang at idt.com] Sent: Wednesday, August 15, 2012 8:51 AM To: Luse, Paul E; nvmewin at lists.openfabrics.org Subject: RE: patch review status Hi Paul, Some questions after browsing the difference between current source and your changes: 1. Why you removed "CC.SHN = 1;" in NVMeNormalShutdown routine? PL> Good eye, that was a mistake. I'll add it back before pushing 2. In nvme.h, why you changed the opcode of NVM_DATASET_MANAGEMENT from 0x06 to 0x09? PL> 6 was wrong, 9 is correct per spec 3. Is there any specific reason(s), you added the following in NVMeInitialize, and then immediately clear EN without checking RDY = 1? /* Set EN to 1 (all others zero) */ CC.EN = 1; StorPortWriteRegisterUlong(pAE, (PULONG)(&pAE->pCtrlRegister->CC), CC.AsUlong); PL> I wrote this up in the patch submission and I can't find that email right now. I didn't add code, I changed code. Before, we use to set EN to 1, read it back, and then set it to 0 to kick off a reset and then later we wait for RDY in the init state machine. We had an IHV complain that their implementation required waiting on RDY after setting EN 1 to even before clearing EN. So, our old implementation was a problem. So what I did now is to not force a reset unless EN is set to 1 when we come in. If its set to 0 then either (a) we're coming from power on or (b) the pre_OS reset the card when it was done. Either way if its already 0 then we're in a known state so we do nothing and then later we set to 1 and wait on RDY. If its 1 when we enter here we set to 0 and then later set to 1 and wait on RDY. Make sense? This assures we never enable without waiting for RDY ***** WAS ******* /* Set EN to 1 (all others zero) */ CC.EN = 1; StorPortWriteRegisterUlong(pAE, (PULONG)(&pAE->pCtrlRegister->CC), CC.AsUlong); /* Read it back */ CC.AsUlong = StorPortReadRegisterUlong(pAE, (PULONG)(&pAE->pCtrlRegister->CC)); /* Now reset */ CC.EN = 0; StorPortWriteRegisterUlong(pAE, (PULONG)(&pAE->pCtrlRegister->CC), CC.AsUlong); CC.AsUlong = StorPortReadRegisterUlong(pAE, (PULONG)(&pAE->pCtrlRegister->CC)); *** NOW IS ******* if (CC.EN == 1) { /* Now reset */ CC.EN = 0; StorPortWriteRegisterUlong(pAE, (PULONG)(&pAE->pCtrlRegister->CC), CC.AsUlong); } Thanks, Alex ________________________________ From:nvmewin-bounces at lists.openfabrics.org[mailto:nvmewin-bounces at lists.openfabrics.org]On Behalf Of Luse, Paul E Sent: Wednesday, August 15, 2012 8:12 AM To: nvmewin at lists.openfabrics.org Subject: [nvmewin] patch review status Just checking in... any questions to date and how are folks tracking for completing their review by this Fri? Thx Paul ____________________________________ Paul Luse Sr. Staff Engineer PCG Server Software Engineering Desk: 480.554.3688, Mobile: 480.334.4630 -------------- next part -------------- An HTML attachment was scrubbed... URL: From Karthik_Rajagopalan at Dell.com Wed Aug 15 17:45:18 2012 From: Karthik_Rajagopalan at Dell.com (Karthik_Rajagopalan at Dell.com) Date: Wed, 15 Aug 2012 19:45:18 -0500 Subject: [nvmewin] patch review status In-Reply-To: <82C9F782B054C94B9FC04A331649C77A07B2BC46@FMSMSX106.amr.corp.intel.com> References: <82C9F782B054C94B9FC04A331649C77A07B2B0C9@FMSMSX106.amr.corp.intel.com> <82C9F782B054C94B9FC04A331649C77A07B2B9EB@FMSMSX106.amr.corp.intel.com> <718182ECAE1D7149A1BF0C789EAC6909049B647C47@AUSX7MCPC110.AMER.DELL.COM> <82C9F782B054C94B9FC04A331649C77A07B2BAA5@FMSMSX106.amr.corp.intel.com> <718182ECAE1D7149A1BF0C789EAC6909049B647C52@AUSX7MCPC110.AMER.DELL.COM> <82C9F782B054C94B9FC04A331649C77A07B2BB67@FMSMSX106.amr.corp.intel.com> <718182ECAE1D7149A1BF0C789EAC6909049B647C5B@AUSX7MCPC110.AMER.DELL.COM> <82C9F782B054C94B9FC04A331649C77A07B2BC46@FMSMSX106.amr.corp.intel.com> Message-ID: <718182ECAE1D7149A1BF0C789EAC6909049B647C67@AUSX7MCPC110.AMER.DELL.COM> I'd say the patch you sent is the safest, but can be optimized further to avoid the MMIO read (a non-posted transaction) to get CC.EN. I agree with your assessment of the spec for 1-->0 transition without correct admin-queue setup. - But, on the other hand, any h/w that doesn't get enabled correctly on 0-->1 is broken... it cannot expect a 0-->1-->0-->1 to be enabled. The Linux driver certainly isn't doing it. - 1-->0-->1 transition (when driver is loaded) is fair & will happen with both our patches (the Linux driver just writes 0 to CC - same as my patch) Consider this too: If we set CC.EN to 1, we need to wait at least 500 ms (CAP.TO) for STS.RDY - in NvmeInitialize, we're in DIRQL time & really should not linger unnecessarily (or give up earlier than what the spec says). From: Luse, Paul E [mailto:paul.e.luse at intel.com] Sent: Wednesday, August 15, 2012 7:11 PM To: Rajagopalan, Karthik; Verma, Vishal L; Alex.Chang at idt.com; nvmewin at lists.openfabrics.org Subject: RE: [nvmewin] patch review status OK, now I understand what you are saying but I don't think the spec is clear enough either way to make a black and white decision (and I bet of the 3 IHV boards I have access to, there's at least 1 that behaves differently than the other 2 in this regard). I have a plane to catch so will think about this overnight. I'd welcome other comments/opinions as well and thanks everyone so far for the inpits! Paul From: Karthik_Rajagopalan at Dell.com [mailto:Karthik_Rajagopalan at Dell.com] Sent: Wednesday, August 15, 2012 4:58 PM To: Luse, Paul E; Verma, Vishal L; Alex.Chang at idt.com; nvmewin at lists.openfabrics.org Subject: RE: [nvmewin] patch review status Setting CC.EN to 0 will satisfy the 1-->0 transition. >To your immediate email: I don't know what you're trying to say What I've removed in the patch is the 0-->1 transition... per the spec it can happen only after we've setup the AQA, ASQ, ACQ entries in MMIO (EnableAdapter). - Otherwise, the controller doesn't have to set STS.RDY - so we'll in fact be waiting forever for RDY From: Luse, Paul E [mailto:paul.e.luse at intel.com] Sent: Wednesday, August 15, 2012 6:47 PM To: Rajagopalan, Karthik; Verma, Vishal L; Alex.Chang at idt.com; nvmewin at lists.openfabrics.org Subject: RE: [nvmewin] patch review status To your immediate email: I don't know what you're trying to say To your email before that: You realize that's what's in the patch I sent out initially right? We've come full circle here, Alex suggested that I add a wait for RDY just before setting to 0 as we don't know if its safe to set to 0 yet and then further suggested the RDY check instead of EN to avoid having to deal with what we do if RDY never comes up. Additionally, this approach does not reset the card at startup. If its 0 to being with, we simply move it to 1 which you would *assume* it OK but I have at least one IHV report that its not good enough, a 1-->0 transition is needed at init. So, I still think this is the most generic and safest: IF EN == 0 { /* Set EN to 1 (all others zero) */ CC.EN = 1; StorPortWriteRegisterUlong(pAE, (PULONG)(&pAE->pCtrlRegister->CC), CC.AsUlong); WAIT ON RDY } /* Now reset */ CC.EN = 0; StorPortWriteRegisterUlong(pAE, (PULONG)(&pAE->pCtrlRegister->CC), CC.AsUlong); From: Karthik_Rajagopalan at Dell.com [mailto:Karthik_Rajagopalan at Dell.com] Sent: Wednesday, August 15, 2012 4:43 PM To: Luse, Paul E; Verma, Vishal L; Alex.Chang at idt.com; nvmewin at lists.openfabrics.org Subject: RE: [nvmewin] patch review status BTW, here's what the spec says for CC.EN When this field is set to '1', the controller sets CSTS.RDY to '1' when it is ready to process commands. I read it as "set CC.EN to 1 after setting up the admin-queues, otherwise RDY will *not* be 1" From: Rajagopalan, Karthik Sent: Wednesday, August 15, 2012 6:32 PM To: 'Luse, Paul E'; Verma, Vishal L; Alex.Chang at idt.com; nvmewin at lists.openfabrics.org Subject: RE: [nvmewin] patch review status Yep :) but I think it doesn't have to be so - all we need to do is an unconditional (attempt) to reset.. we don't set CC.EN to 1 until we're ready to enable the controller. Here are the possibilities when we try to reset: 1. CC.EN is 0 to begin with => no need to reset; but setting it again to 0 should be a NOP 2. CC.EN is 1 to begin with => need to reset; we didn't wiggle this yet, so STS.RDY should be 1; setting EN to 0 will actually reset a. The original issue you mention could still happen if STS.RDY stayed 0 (bad controller IMHO); still goodness in trying to reset Here's a patch in case you want to try this approach... From: Luse, Paul E [mailto:paul.e.luse at intel.com] Sent: Wednesday, August 15, 2012 6:22 PM To: Rajagopalan, Karthik; Verma, Vishal L; Alex.Chang at idt.com; nvmewin at lists.openfabrics.org Subject: RE: [nvmewin] patch review status 0 doesn't reset unless it was previously 1... that's why the initial implementation set it to 1, then 0 to force reset then init happened. The question came in about setting it to 0 right after setting it to 1, anytime you transition to 1 you need to wait for RDY so the code I proposed last will always reset but only for a 2nd wait on RDY (there's one after we enable as part of init) if the controller wasn't disabled when we started. Hard to believe something so simple can be so convoluted isn't it? From: Karthik_Rajagopalan at Dell.com [mailto:Karthik_Rajagopalan at Dell.com] Sent: Wednesday, August 15, 2012 4:11 PM To: Luse, Paul E; Verma, Vishal L; Alex.Chang at idt.com; nvmewin at lists.openfabrics.org Subject: RE: [nvmewin] patch review status Why not unconditionally set CC.EN to 0 as soon as the driver loads ? - There will be an issue with this iff someone before (opROM, UEFI ?) had set CC.EN to 1 & somehow STS.RDY never went to 1... but things still continued (unlikely) i.e., /* Disable / Reset */ CC.EN = 0; StorPortWriteRegisterUlong(pAE, (PULONG)(&pAE->pCtrlRegister->CC), CC.AsUlong); /* Set EN to 1 (all others zero) */ CC.EN = 1; StorPortWriteRegisterUlong(pAE, (PULONG)(&pAE->pCtrlRegister->CC), CC.AsUlong); /* Wait for RDY */ From: nvmewin-bounces at lists.openfabrics.org [mailto:nvmewin-bounces at lists.openfabrics.org] On Behalf Of Luse, Paul E Sent: Wednesday, August 15, 2012 5:59 PM To: Verma, Vishal L; Chang, Alex; nvmewin at lists.openfabrics.org Subject: Re: [nvmewin] patch review status So maybe a better proposal for this section of code... go back to what we had before (sorta) but include a wait on RDY after setting EN to 1 and only set EN to 1 if its zero to start. This is the most generic soln and since the behavior of the HW at this detailed level isn't specified we can call this the least common denominator and if IHVs want to optimize for their implementations they can. If this first RDY wait files, then we fail driver init. This code assures we always reset and we never touch EN after setting it to 1 without waiting for RDY IF EN == 0 { /* Set EN to 1 (all others zero) */ CC.EN = 1; StorPortWriteRegisterUlong(pAE, (PULONG)(&pAE->pCtrlRegister->CC), CC.AsUlong); WAIT ON RDY } /* Now reset */ CC.EN = 0; StorPortWriteRegisterUlong(pAE, (PULONG)(&pAE->pCtrlRegister->CC), CC.AsUlong); From: Luse, Paul E Sent: Wednesday, August 15, 2012 3:21 PM To: Verma, Vishal L; Chang, Alex; nvmewin at lists.openfabrics.org Subject: RE: [nvmewin] patch review status I was thinking about that too... If EN was already 1 and we caught RDY before it was 0 then we would indeed set EN to 1 again (possibly before, during or after RDY is set). The spec, of course, does not dictate behavior in this case but one would assume it would be treated as a NOP and our init state machine would properly wait for RDY. I'm not a HW guy though, anyone else have a comment/concern on setting EN to 1 if tis already 1? From: Verma, Vishal L Sent: Wednesday, August 15, 2012 1:46 PM To: Luse, Paul E; Chang, Alex; nvmewin at lists.openfabrics.org Subject: Re: [nvmewin] patch review status Paul, Is this correct though? Making CC.EN 0 implies that RDY will become 0, but RDY being 0 does not guarantee that CC.EN is 0. If what I'm thinking is right, we could be in a state where CC.EN was 1 when the driver comes up, but the controller never became ready. In this case we will think everything is alright, and we will try to make CC.EN 1 again - Which is not something we'd want to do, right? Vishal From: , Paul E > Date: Wednesday, August 15, 2012 11:43 AM To: "Chang, Alex" >, "nvmewin at lists.openfabrics.org" > Subject: Re: [nvmewin] patch review status OK, I think I can get behind that. I was thinking STS might not be valid when EN is 0 but looking at the spec I think this is fine Ready (RDY): This field is set to '1' when the controller is ready to process commands after CC.EN is set to '1'. This field shall be cleared to '0' when CC.EN is cleared to '0'. Commands shall not be issued to the controller until this field is set to '1' after the CC.EN bit is set to '1'. Failure to follow this requirement produces undefined results. Host software shall wait a minimum of CAP.TO seconds for this field to be set to '1' after setting CC.EN to '1' from a previous value of '0'. So I'll just chance the check below for EN == 1 to RDY == 1 then Thanks! Paul From: Chang, Alex [mailto:Alex.Chang at idt.com] Sent: Wednesday, August 15, 2012 10:21 AM To: Luse, Paul E; nvmewin at lists.openfabrics.org Subject: RE: patch review status Absolutely. Now, the tricky part is when CC.EN=1 and STS.RDY never becomes 1. I'd say simply check STS.RDY only. If it's 1, then reset the device, otherwise, skip resetting it. What do you think? Thanks, Alex ________________________________ From: Luse, Paul E [mailto:paul.e.luse at intel.com] Sent: Wednesday, August 15, 2012 9:56 AM To: Chang, Alex; nvmewin at lists.openfabrics.org Subject: RE: patch review status OK, I see what you're saying I think. We don't want to check for RDY without first looking at EN so does this make sense: if (CC.EN == 1) { -- now wait for RDY - /* Now reset */ CC.EN = 0; StorPortWriteRegisterUlong(pAE, (PULONG)(&pAE->pCtrlRegister->CC), CC.AsUlong); } From: Chang, Alex [mailto:Alex.Chang at idt.com] Sent: Wednesday, August 15, 2012 9:52 AM To: Luse, Paul E; nvmewin at lists.openfabrics.org Subject: RE: patch review status Hi Paul, I agree with the new logic on #3. However, instead of checking CC.EN, should we check STS.RDY or both? Thanks, Alex ________________________________ From: Luse, Paul E [mailto:paul.e.luse at intel.com] Sent: Wednesday, August 15, 2012 9:03 AM To: Chang, Alex; nvmewin at lists.openfabrics.org Subject: RE: patch review status Thanks Alex, see below. From: Chang, Alex [mailto:Alex.Chang at idt.com] Sent: Wednesday, August 15, 2012 8:51 AM To: Luse, Paul E; nvmewin at lists.openfabrics.org Subject: RE: patch review status Hi Paul, Some questions after browsing the difference between current source and your changes: 1. Why you removed "CC.SHN = 1;" in NVMeNormalShutdown routine? PL> Good eye, that was a mistake. I'll add it back before pushing 2. In nvme.h, why you changed the opcode of NVM_DATASET_MANAGEMENT from 0x06 to 0x09? PL> 6 was wrong, 9 is correct per spec 3. Is there any specific reason(s), you added the following in NVMeInitialize, and then immediately clear EN without checking RDY = 1? /* Set EN to 1 (all others zero) */ CC.EN = 1; StorPortWriteRegisterUlong(pAE, (PULONG)(&pAE->pCtrlRegister->CC), CC.AsUlong); PL> I wrote this up in the patch submission and I can't find that email right now. I didn't add code, I changed code. Before, we use to set EN to 1, read it back, and then set it to 0 to kick off a reset and then later we wait for RDY in the init state machine. We had an IHV complain that their implementation required waiting on RDY after setting EN 1 to even before clearing EN. So, our old implementation was a problem. So what I did now is to not force a reset unless EN is set to 1 when we come in. If its set to 0 then either (a) we're coming from power on or (b) the pre_OS reset the card when it was done. Either way if its already 0 then we're in a known state so we do nothing and then later we set to 1 and wait on RDY. If its 1 when we enter here we set to 0 and then later set to 1 and wait on RDY. Make sense? This assures we never enable without waiting for RDY ***** WAS ******* /* Set EN to 1 (all others zero) */ CC.EN = 1; StorPortWriteRegisterUlong(pAE, (PULONG)(&pAE->pCtrlRegister->CC), CC.AsUlong); /* Read it back */ CC.AsUlong = StorPortReadRegisterUlong(pAE, (PULONG)(&pAE->pCtrlRegister->CC)); /* Now reset */ CC.EN = 0; StorPortWriteRegisterUlong(pAE, (PULONG)(&pAE->pCtrlRegister->CC), CC.AsUlong); CC.AsUlong = StorPortReadRegisterUlong(pAE, (PULONG)(&pAE->pCtrlRegister->CC)); *** NOW IS ******* if (CC.EN == 1) { /* Now reset */ CC.EN = 0; StorPortWriteRegisterUlong(pAE, (PULONG)(&pAE->pCtrlRegister->CC), CC.AsUlong); } Thanks, Alex ________________________________ From:nvmewin-bounces at lists.openfabrics.org[mailto:nvmewin-bounces at lists.openfabrics.org]On Behalf Of Luse, Paul E Sent: Wednesday, August 15, 2012 8:12 AM To: nvmewin at lists.openfabrics.org Subject: [nvmewin] patch review status Just checking in... any questions to date and how are folks tracking for completing their review by this Fri? Thx Paul ____________________________________ Paul Luse Sr. Staff Engineer PCG Server Software Engineering Desk: 480.554.3688, Mobile: 480.334.4630 -------------- next part -------------- An HTML attachment was scrubbed... URL: From Karthik_Rajagopalan at Dell.com Wed Aug 15 17:48:43 2012 From: Karthik_Rajagopalan at Dell.com (Karthik_Rajagopalan at Dell.com) Date: Wed, 15 Aug 2012 19:48:43 -0500 Subject: [nvmewin] patch review status References: <82C9F782B054C94B9FC04A331649C77A07B2B0C9@FMSMSX106.amr.corp.intel.com> <82C9F782B054C94B9FC04A331649C77A07B2B9EB@FMSMSX106.amr.corp.intel.com> <718182ECAE1D7149A1BF0C789EAC6909049B647C47@AUSX7MCPC110.AMER.DELL.COM> <82C9F782B054C94B9FC04A331649C77A07B2BAA5@FMSMSX106.amr.corp.intel.com> <718182ECAE1D7149A1BF0C789EAC6909049B647C52@AUSX7MCPC110.AMER.DELL.COM> <82C9F782B054C94B9FC04A331649C77A07B2BB67@FMSMSX106.amr.corp.intel.com> <718182ECAE1D7149A1BF0C789EAC6909049B647C5B@AUSX7MCPC110.AMER.DELL.COM> <82C9F782B054C94B9FC04A331649C77A07B2BC46@FMSMSX106.amr.corp.intel.com> Message-ID: <718182ECAE1D7149A1BF0C789EAC6909049B647C68@AUSX7MCPC110.AMER.DELL.COM> Correction of typo below: I agree with your assessment of the spec for 0-->1 transition without correct admin-queue setup. From: Rajagopalan, Karthik Sent: Wednesday, August 15, 2012 7:45 PM To: 'Luse, Paul E'; Verma, Vishal L; Alex.Chang at idt.com; nvmewin at lists.openfabrics.org Subject: RE: [nvmewin] patch review status I'd say the patch you sent is the safest, but can be optimized further to avoid the MMIO read (a non-posted transaction) to get CC.EN. I agree with your assessment of the spec for 1-->0 transition without correct admin-queue setup. - But, on the other hand, any h/w that doesn't get enabled correctly on 0-->1 is broken... it cannot expect a 0-->1-->0-->1 to be enabled. The Linux driver certainly isn't doing it. - 1-->0-->1 transition (when driver is loaded) is fair & will happen with both our patches (the Linux driver just writes 0 to CC - same as my patch) Consider this too: If we set CC.EN to 1, we need to wait at least 500 ms (CAP.TO) for STS.RDY - in NvmeInitialize, we're in DIRQL time & really should not linger unnecessarily (or give up earlier than what the spec says). From: Luse, Paul E [mailto:paul.e.luse at intel.com] Sent: Wednesday, August 15, 2012 7:11 PM To: Rajagopalan, Karthik; Verma, Vishal L; Alex.Chang at idt.com; nvmewin at lists.openfabrics.org Subject: RE: [nvmewin] patch review status OK, now I understand what you are saying but I don't think the spec is clear enough either way to make a black and white decision (and I bet of the 3 IHV boards I have access to, there's at least 1 that behaves differently than the other 2 in this regard). I have a plane to catch so will think about this overnight. I'd welcome other comments/opinions as well and thanks everyone so far for the inpits! Paul From: Karthik_Rajagopalan at Dell.com [mailto:Karthik_Rajagopalan at Dell.com] Sent: Wednesday, August 15, 2012 4:58 PM To: Luse, Paul E; Verma, Vishal L; Alex.Chang at idt.com; nvmewin at lists.openfabrics.org Subject: RE: [nvmewin] patch review status Setting CC.EN to 0 will satisfy the 1-->0 transition. >To your immediate email: I don't know what you're trying to say What I've removed in the patch is the 0-->1 transition... per the spec it can happen only after we've setup the AQA, ASQ, ACQ entries in MMIO (EnableAdapter). - Otherwise, the controller doesn't have to set STS.RDY - so we'll in fact be waiting forever for RDY From: Luse, Paul E [mailto:paul.e.luse at intel.com] Sent: Wednesday, August 15, 2012 6:47 PM To: Rajagopalan, Karthik; Verma, Vishal L; Alex.Chang at idt.com; nvmewin at lists.openfabrics.org Subject: RE: [nvmewin] patch review status To your immediate email: I don't know what you're trying to say To your email before that: You realize that's what's in the patch I sent out initially right? We've come full circle here, Alex suggested that I add a wait for RDY just before setting to 0 as we don't know if its safe to set to 0 yet and then further suggested the RDY check instead of EN to avoid having to deal with what we do if RDY never comes up. Additionally, this approach does not reset the card at startup. If its 0 to being with, we simply move it to 1 which you would *assume* it OK but I have at least one IHV report that its not good enough, a 1-->0 transition is needed at init. So, I still think this is the most generic and safest: IF EN == 0 { /* Set EN to 1 (all others zero) */ CC.EN = 1; StorPortWriteRegisterUlong(pAE, (PULONG)(&pAE->pCtrlRegister->CC), CC.AsUlong); WAIT ON RDY } /* Now reset */ CC.EN = 0; StorPortWriteRegisterUlong(pAE, (PULONG)(&pAE->pCtrlRegister->CC), CC.AsUlong); From: Karthik_Rajagopalan at Dell.com [mailto:Karthik_Rajagopalan at Dell.com] Sent: Wednesday, August 15, 2012 4:43 PM To: Luse, Paul E; Verma, Vishal L; Alex.Chang at idt.com; nvmewin at lists.openfabrics.org Subject: RE: [nvmewin] patch review status BTW, here's what the spec says for CC.EN When this field is set to '1', the controller sets CSTS.RDY to '1' when it is ready to process commands. I read it as "set CC.EN to 1 after setting up the admin-queues, otherwise RDY will *not* be 1" From: Rajagopalan, Karthik Sent: Wednesday, August 15, 2012 6:32 PM To: 'Luse, Paul E'; Verma, Vishal L; Alex.Chang at idt.com; nvmewin at lists.openfabrics.org Subject: RE: [nvmewin] patch review status Yep :) but I think it doesn't have to be so - all we need to do is an unconditional (attempt) to reset.. we don't set CC.EN to 1 until we're ready to enable the controller. Here are the possibilities when we try to reset: 1. CC.EN is 0 to begin with => no need to reset; but setting it again to 0 should be a NOP 2. CC.EN is 1 to begin with => need to reset; we didn't wiggle this yet, so STS.RDY should be 1; setting EN to 0 will actually reset a. The original issue you mention could still happen if STS.RDY stayed 0 (bad controller IMHO); still goodness in trying to reset Here's a patch in case you want to try this approach... From: Luse, Paul E [mailto:paul.e.luse at intel.com] Sent: Wednesday, August 15, 2012 6:22 PM To: Rajagopalan, Karthik; Verma, Vishal L; Alex.Chang at idt.com; nvmewin at lists.openfabrics.org Subject: RE: [nvmewin] patch review status 0 doesn't reset unless it was previously 1... that's why the initial implementation set it to 1, then 0 to force reset then init happened. The question came in about setting it to 0 right after setting it to 1, anytime you transition to 1 you need to wait for RDY so the code I proposed last will always reset but only for a 2nd wait on RDY (there's one after we enable as part of init) if the controller wasn't disabled when we started. Hard to believe something so simple can be so convoluted isn't it? From: Karthik_Rajagopalan at Dell.com [mailto:Karthik_Rajagopalan at Dell.com] Sent: Wednesday, August 15, 2012 4:11 PM To: Luse, Paul E; Verma, Vishal L; Alex.Chang at idt.com; nvmewin at lists.openfabrics.org Subject: RE: [nvmewin] patch review status Why not unconditionally set CC.EN to 0 as soon as the driver loads ? - There will be an issue with this iff someone before (opROM, UEFI ?) had set CC.EN to 1 & somehow STS.RDY never went to 1... but things still continued (unlikely) i.e., /* Disable / Reset */ CC.EN = 0; StorPortWriteRegisterUlong(pAE, (PULONG)(&pAE->pCtrlRegister->CC), CC.AsUlong); /* Set EN to 1 (all others zero) */ CC.EN = 1; StorPortWriteRegisterUlong(pAE, (PULONG)(&pAE->pCtrlRegister->CC), CC.AsUlong); /* Wait for RDY */ From: nvmewin-bounces at lists.openfabrics.org [mailto:nvmewin-bounces at lists.openfabrics.org] On Behalf Of Luse, Paul E Sent: Wednesday, August 15, 2012 5:59 PM To: Verma, Vishal L; Chang, Alex; nvmewin at lists.openfabrics.org Subject: Re: [nvmewin] patch review status So maybe a better proposal for this section of code... go back to what we had before (sorta) but include a wait on RDY after setting EN to 1 and only set EN to 1 if its zero to start. This is the most generic soln and since the behavior of the HW at this detailed level isn't specified we can call this the least common denominator and if IHVs want to optimize for their implementations they can. If this first RDY wait files, then we fail driver init. This code assures we always reset and we never touch EN after setting it to 1 without waiting for RDY IF EN == 0 { /* Set EN to 1 (all others zero) */ CC.EN = 1; StorPortWriteRegisterUlong(pAE, (PULONG)(&pAE->pCtrlRegister->CC), CC.AsUlong); WAIT ON RDY } /* Now reset */ CC.EN = 0; StorPortWriteRegisterUlong(pAE, (PULONG)(&pAE->pCtrlRegister->CC), CC.AsUlong); From: Luse, Paul E Sent: Wednesday, August 15, 2012 3:21 PM To: Verma, Vishal L; Chang, Alex; nvmewin at lists.openfabrics.org Subject: RE: [nvmewin] patch review status I was thinking about that too... If EN was already 1 and we caught RDY before it was 0 then we would indeed set EN to 1 again (possibly before, during or after RDY is set). The spec, of course, does not dictate behavior in this case but one would assume it would be treated as a NOP and our init state machine would properly wait for RDY. I'm not a HW guy though, anyone else have a comment/concern on setting EN to 1 if tis already 1? From: Verma, Vishal L Sent: Wednesday, August 15, 2012 1:46 PM To: Luse, Paul E; Chang, Alex; nvmewin at lists.openfabrics.org Subject: Re: [nvmewin] patch review status Paul, Is this correct though? Making CC.EN 0 implies that RDY will become 0, but RDY being 0 does not guarantee that CC.EN is 0. If what I'm thinking is right, we could be in a state where CC.EN was 1 when the driver comes up, but the controller never became ready. In this case we will think everything is alright, and we will try to make CC.EN 1 again - Which is not something we'd want to do, right? Vishal From: , Paul E > Date: Wednesday, August 15, 2012 11:43 AM To: "Chang, Alex" >, "nvmewin at lists.openfabrics.org" > Subject: Re: [nvmewin] patch review status OK, I think I can get behind that. I was thinking STS might not be valid when EN is 0 but looking at the spec I think this is fine Ready (RDY): This field is set to '1' when the controller is ready to process commands after CC.EN is set to '1'. This field shall be cleared to '0' when CC.EN is cleared to '0'. Commands shall not be issued to the controller until this field is set to '1' after the CC.EN bit is set to '1'. Failure to follow this requirement produces undefined results. Host software shall wait a minimum of CAP.TO seconds for this field to be set to '1' after setting CC.EN to '1' from a previous value of '0'. So I'll just chance the check below for EN == 1 to RDY == 1 then Thanks! Paul From: Chang, Alex [mailto:Alex.Chang at idt.com] Sent: Wednesday, August 15, 2012 10:21 AM To: Luse, Paul E; nvmewin at lists.openfabrics.org Subject: RE: patch review status Absolutely. Now, the tricky part is when CC.EN=1 and STS.RDY never becomes 1. I'd say simply check STS.RDY only. If it's 1, then reset the device, otherwise, skip resetting it. What do you think? Thanks, Alex ________________________________ From: Luse, Paul E [mailto:paul.e.luse at intel.com] Sent: Wednesday, August 15, 2012 9:56 AM To: Chang, Alex; nvmewin at lists.openfabrics.org Subject: RE: patch review status OK, I see what you're saying I think. We don't want to check for RDY without first looking at EN so does this make sense: if (CC.EN == 1) { -- now wait for RDY - /* Now reset */ CC.EN = 0; StorPortWriteRegisterUlong(pAE, (PULONG)(&pAE->pCtrlRegister->CC), CC.AsUlong); } From: Chang, Alex [mailto:Alex.Chang at idt.com] Sent: Wednesday, August 15, 2012 9:52 AM To: Luse, Paul E; nvmewin at lists.openfabrics.org Subject: RE: patch review status Hi Paul, I agree with the new logic on #3. However, instead of checking CC.EN, should we check STS.RDY or both? Thanks, Alex ________________________________ From: Luse, Paul E [mailto:paul.e.luse at intel.com] Sent: Wednesday, August 15, 2012 9:03 AM To: Chang, Alex; nvmewin at lists.openfabrics.org Subject: RE: patch review status Thanks Alex, see below. From: Chang, Alex [mailto:Alex.Chang at idt.com] Sent: Wednesday, August 15, 2012 8:51 AM To: Luse, Paul E; nvmewin at lists.openfabrics.org Subject: RE: patch review status Hi Paul, Some questions after browsing the difference between current source and your changes: 1. Why you removed "CC.SHN = 1;" in NVMeNormalShutdown routine? PL> Good eye, that was a mistake. I'll add it back before pushing 2. In nvme.h, why you changed the opcode of NVM_DATASET_MANAGEMENT from 0x06 to 0x09? PL> 6 was wrong, 9 is correct per spec 3. Is there any specific reason(s), you added the following in NVMeInitialize, and then immediately clear EN without checking RDY = 1? /* Set EN to 1 (all others zero) */ CC.EN = 1; StorPortWriteRegisterUlong(pAE, (PULONG)(&pAE->pCtrlRegister->CC), CC.AsUlong); PL> I wrote this up in the patch submission and I can't find that email right now. I didn't add code, I changed code. Before, we use to set EN to 1, read it back, and then set it to 0 to kick off a reset and then later we wait for RDY in the init state machine. We had an IHV complain that their implementation required waiting on RDY after setting EN 1 to even before clearing EN. So, our old implementation was a problem. So what I did now is to not force a reset unless EN is set to 1 when we come in. If its set to 0 then either (a) we're coming from power on or (b) the pre_OS reset the card when it was done. Either way if its already 0 then we're in a known state so we do nothing and then later we set to 1 and wait on RDY. If its 1 when we enter here we set to 0 and then later set to 1 and wait on RDY. Make sense? This assures we never enable without waiting for RDY ***** WAS ******* /* Set EN to 1 (all others zero) */ CC.EN = 1; StorPortWriteRegisterUlong(pAE, (PULONG)(&pAE->pCtrlRegister->CC), CC.AsUlong); /* Read it back */ CC.AsUlong = StorPortReadRegisterUlong(pAE, (PULONG)(&pAE->pCtrlRegister->CC)); /* Now reset */ CC.EN = 0; StorPortWriteRegisterUlong(pAE, (PULONG)(&pAE->pCtrlRegister->CC), CC.AsUlong); CC.AsUlong = StorPortReadRegisterUlong(pAE, (PULONG)(&pAE->pCtrlRegister->CC)); *** NOW IS ******* if (CC.EN == 1) { /* Now reset */ CC.EN = 0; StorPortWriteRegisterUlong(pAE, (PULONG)(&pAE->pCtrlRegister->CC), CC.AsUlong); } Thanks, Alex ________________________________ From:nvmewin-bounces at lists.openfabrics.org[mailto:nvmewin-bounces at lists.openfabrics.org]On Behalf Of Luse, Paul E Sent: Wednesday, August 15, 2012 8:12 AM To: nvmewin at lists.openfabrics.org Subject: [nvmewin] patch review status Just checking in... any questions to date and how are folks tracking for completing their review by this Fri? Thx Paul ____________________________________ Paul Luse Sr. Staff Engineer PCG Server Software Engineering Desk: 480.554.3688, Mobile: 480.334.4630 -------------- next part -------------- An HTML attachment was scrubbed... URL: From paul.e.luse at intel.com Wed Aug 15 21:33:58 2012 From: paul.e.luse at intel.com (Luse, Paul E) Date: Thu, 16 Aug 2012 04:33:58 +0000 Subject: [nvmewin] patch review status In-Reply-To: <718182ECAE1D7149A1BF0C789EAC6909049B647C68@AUSX7MCPC110.AMER.DELL.COM> References: <82C9F782B054C94B9FC04A331649C77A07B2B0C9@FMSMSX106.amr.corp.intel.com> <82C9F782B054C94B9FC04A331649C77A07B2B9EB@FMSMSX106.amr.corp.intel.com> <718182ECAE1D7149A1BF0C789EAC6909049B647C47@AUSX7MCPC110.AMER.DELL.COM> <82C9F782B054C94B9FC04A331649C77A07B2BAA5@FMSMSX106.amr.corp.intel.com> <718182ECAE1D7149A1BF0C789EAC6909049B647C52@AUSX7MCPC110.AMER.DELL.COM> <82C9F782B054C94B9FC04A331649C77A07B2BB67@FMSMSX106.amr.corp.intel.com> <718182ECAE1D7149A1BF0C789EAC6909049B647C5B@AUSX7MCPC110.AMER.DELL.COM> <82C9F782B054C94B9FC04A331649C77A07B2BC46@FMSMSX106.amr.corp.intel.com> <718182ECAE1D7149A1BF0C789EAC6909049B647C68@AUSX7MCPC110.AMER.DELL.COM> Message-ID: <82C9F782B054C94B9FC04A331649C77A07B2BD77@FMSMSX106.amr.corp.intel.com> FYI the linux guys were going this direction as well, it just hasn't made it out there yet.... Yes, I'm aware of the negative implications of waiting on RDY and don't like it. I'll push the last agreed upon 'safest' patch then and I think we can all feel comfortable knowing that on the windows side this really is a reference implementation. You can't take a pre-WHQL'd binary so if this doesn't work for some specific implementation then that IHV can change it. And, if/when Msft does a native NVMe driver the HW that doesn't work with that interpretation of the spec will have challenges for sure J Thanks again for the inputs Paul From: Karthik_Rajagopalan at Dell.com [mailto:Karthik_Rajagopalan at Dell.com] Sent: Wednesday, August 15, 2012 5:49 PM To: Luse, Paul E; Verma, Vishal L; Alex.Chang at idt.com; nvmewin at lists.openfabrics.org Subject: RE: [nvmewin] patch review status Correction of typo below: I agree with your assessment of the spec for 0-->1 transition without correct admin-queue setup. From: Rajagopalan, Karthik Sent: Wednesday, August 15, 2012 7:45 PM To: 'Luse, Paul E'; Verma, Vishal L; Alex.Chang at idt.com; nvmewin at lists.openfabrics.org Subject: RE: [nvmewin] patch review status I'd say the patch you sent is the safest, but can be optimized further to avoid the MMIO read (a non-posted transaction) to get CC.EN. I agree with your assessment of the spec for 1-->0 transition without correct admin-queue setup. - But, on the other hand, any h/w that doesn't get enabled correctly on 0-->1 is broken... it cannot expect a 0-->1-->0-->1 to be enabled. The Linux driver certainly isn't doing it. - 1-->0-->1 transition (when driver is loaded) is fair & will happen with both our patches (the Linux driver just writes 0 to CC - same as my patch) Consider this too: If we set CC.EN to 1, we need to wait at least 500 ms (CAP.TO) for STS.RDY - in NvmeInitialize, we're in DIRQL time & really should not linger unnecessarily (or give up earlier than what the spec says). From: Luse, Paul E [mailto:paul.e.luse at intel.com] Sent: Wednesday, August 15, 2012 7:11 PM To: Rajagopalan, Karthik; Verma, Vishal L; Alex.Chang at idt.com; nvmewin at lists.openfabrics.org Subject: RE: [nvmewin] patch review status OK, now I understand what you are saying but I don't think the spec is clear enough either way to make a black and white decision (and I bet of the 3 IHV boards I have access to, there's at least 1 that behaves differently than the other 2 in this regard). I have a plane to catch so will think about this overnight. I'd welcome other comments/opinions as well and thanks everyone so far for the inpits! Paul From: Karthik_Rajagopalan at Dell.com [mailto:Karthik_Rajagopalan at Dell.com] Sent: Wednesday, August 15, 2012 4:58 PM To: Luse, Paul E; Verma, Vishal L; Alex.Chang at idt.com; nvmewin at lists.openfabrics.org Subject: RE: [nvmewin] patch review status Setting CC.EN to 0 will satisfy the 1-->0 transition. >To your immediate email: I don't know what you're trying to say What I've removed in the patch is the 0-->1 transition... per the spec it can happen only after we've setup the AQA, ASQ, ACQ entries in MMIO (EnableAdapter). - Otherwise, the controller doesn't have to set STS.RDY - so we'll in fact be waiting forever for RDY From: Luse, Paul E [mailto:paul.e.luse at intel.com] Sent: Wednesday, August 15, 2012 6:47 PM To: Rajagopalan, Karthik; Verma, Vishal L; Alex.Chang at idt.com; nvmewin at lists.openfabrics.org Subject: RE: [nvmewin] patch review status To your immediate email: I don't know what you're trying to say To your email before that: You realize that's what's in the patch I sent out initially right? We've come full circle here, Alex suggested that I add a wait for RDY just before setting to 0 as we don't know if its safe to set to 0 yet and then further suggested the RDY check instead of EN to avoid having to deal with what we do if RDY never comes up. Additionally, this approach does not reset the card at startup. If its 0 to being with, we simply move it to 1 which you would *assume* it OK but I have at least one IHV report that its not good enough, a 1-->0 transition is needed at init. So, I still think this is the most generic and safest: IF EN == 0 { /* Set EN to 1 (all others zero) */ CC.EN = 1; StorPortWriteRegisterUlong(pAE, (PULONG)(&pAE->pCtrlRegister->CC), CC.AsUlong); WAIT ON RDY } /* Now reset */ CC.EN = 0; StorPortWriteRegisterUlong(pAE, (PULONG)(&pAE->pCtrlRegister->CC), CC.AsUlong); From: Karthik_Rajagopalan at Dell.com [mailto:Karthik_Rajagopalan at Dell.com] Sent: Wednesday, August 15, 2012 4:43 PM To: Luse, Paul E; Verma, Vishal L; Alex.Chang at idt.com; nvmewin at lists.openfabrics.org Subject: RE: [nvmewin] patch review status BTW, here's what the spec says for CC.EN When this field is set to '1', the controller sets CSTS.RDY to '1' when it is ready to process commands. I read it as "set CC.EN to 1 after setting up the admin-queues, otherwise RDY will *not* be 1" From: Rajagopalan, Karthik Sent: Wednesday, August 15, 2012 6:32 PM To: 'Luse, Paul E'; Verma, Vishal L; Alex.Chang at idt.com; nvmewin at lists.openfabrics.org Subject: RE: [nvmewin] patch review status Yep :) but I think it doesn't have to be so - all we need to do is an unconditional (attempt) to reset.. we don't set CC.EN to 1 until we're ready to enable the controller. Here are the possibilities when we try to reset: 1. CC.EN is 0 to begin with => no need to reset; but setting it again to 0 should be a NOP 2. CC.EN is 1 to begin with => need to reset; we didn't wiggle this yet, so STS.RDY should be 1; setting EN to 0 will actually reset a. The original issue you mention could still happen if STS.RDY stayed 0 (bad controller IMHO); still goodness in trying to reset Here's a patch in case you want to try this approach... From: Luse, Paul E [mailto:paul.e.luse at intel.com] Sent: Wednesday, August 15, 2012 6:22 PM To: Rajagopalan, Karthik; Verma, Vishal L; Alex.Chang at idt.com; nvmewin at lists.openfabrics.org Subject: RE: [nvmewin] patch review status 0 doesn't reset unless it was previously 1... that's why the initial implementation set it to 1, then 0 to force reset then init happened. The question came in about setting it to 0 right after setting it to 1, anytime you transition to 1 you need to wait for RDY so the code I proposed last will always reset but only for a 2nd wait on RDY (there's one after we enable as part of init) if the controller wasn't disabled when we started. Hard to believe something so simple can be so convoluted isn't it? From: Karthik_Rajagopalan at Dell.com [mailto:Karthik_Rajagopalan at Dell.com] Sent: Wednesday, August 15, 2012 4:11 PM To: Luse, Paul E; Verma, Vishal L; Alex.Chang at idt.com; nvmewin at lists.openfabrics.org Subject: RE: [nvmewin] patch review status Why not unconditionally set CC.EN to 0 as soon as the driver loads ? - There will be an issue with this iff someone before (opROM, UEFI ?) had set CC.EN to 1 & somehow STS.RDY never went to 1... but things still continued (unlikely) i.e., /* Disable / Reset */ CC.EN = 0; StorPortWriteRegisterUlong(pAE, (PULONG)(&pAE->pCtrlRegister->CC), CC.AsUlong); /* Set EN to 1 (all others zero) */ CC.EN = 1; StorPortWriteRegisterUlong(pAE, (PULONG)(&pAE->pCtrlRegister->CC), CC.AsUlong); /* Wait for RDY */ From: nvmewin-bounces at lists.openfabrics.org [mailto:nvmewin-bounces at lists.openfabrics.org] On Behalf Of Luse, Paul E Sent: Wednesday, August 15, 2012 5:59 PM To: Verma, Vishal L; Chang, Alex; nvmewin at lists.openfabrics.org Subject: Re: [nvmewin] patch review status So maybe a better proposal for this section of code... go back to what we had before (sorta) but include a wait on RDY after setting EN to 1 and only set EN to 1 if its zero to start. This is the most generic soln and since the behavior of the HW at this detailed level isn't specified we can call this the least common denominator and if IHVs want to optimize for their implementations they can. If this first RDY wait files, then we fail driver init. This code assures we always reset and we never touch EN after setting it to 1 without waiting for RDY IF EN == 0 { /* Set EN to 1 (all others zero) */ CC.EN = 1; StorPortWriteRegisterUlong(pAE, (PULONG)(&pAE->pCtrlRegister->CC), CC.AsUlong); WAIT ON RDY } /* Now reset */ CC.EN = 0; StorPortWriteRegisterUlong(pAE, (PULONG)(&pAE->pCtrlRegister->CC), CC.AsUlong); From: Luse, Paul E Sent: Wednesday, August 15, 2012 3:21 PM To: Verma, Vishal L; Chang, Alex; nvmewin at lists.openfabrics.org Subject: RE: [nvmewin] patch review status I was thinking about that too... If EN was already 1 and we caught RDY before it was 0 then we would indeed set EN to 1 again (possibly before, during or after RDY is set). The spec, of course, does not dictate behavior in this case but one would assume it would be treated as a NOP and our init state machine would properly wait for RDY. I'm not a HW guy though, anyone else have a comment/concern on setting EN to 1 if tis already 1? From: Verma, Vishal L Sent: Wednesday, August 15, 2012 1:46 PM To: Luse, Paul E; Chang, Alex; nvmewin at lists.openfabrics.org Subject: Re: [nvmewin] patch review status Paul, Is this correct though? Making CC.EN 0 implies that RDY will become 0, but RDY being 0 does not guarantee that CC.EN is 0. If what I'm thinking is right, we could be in a state where CC.EN was 1 when the driver comes up, but the controller never became ready. In this case we will think everything is alright, and we will try to make CC.EN 1 again - Which is not something we'd want to do, right? Vishal From: , Paul E > Date: Wednesday, August 15, 2012 11:43 AM To: "Chang, Alex" >, "nvmewin at lists.openfabrics.org" > Subject: Re: [nvmewin] patch review status OK, I think I can get behind that. I was thinking STS might not be valid when EN is 0 but looking at the spec I think this is fine Ready (RDY): This field is set to '1' when the controller is ready to process commands after CC.EN is set to '1'. This field shall be cleared to '0' when CC.EN is cleared to '0'. Commands shall not be issued to the controller until this field is set to '1' after the CC.EN bit is set to '1'. Failure to follow this requirement produces undefined results. Host software shall wait a minimum of CAP.TO seconds for this field to be set to '1' after setting CC.EN to '1' from a previous value of '0'. So I'll just chance the check below for EN == 1 to RDY == 1 then Thanks! Paul From: Chang, Alex [mailto:Alex.Chang at idt.com] Sent: Wednesday, August 15, 2012 10:21 AM To: Luse, Paul E; nvmewin at lists.openfabrics.org Subject: RE: patch review status Absolutely. Now, the tricky part is when CC.EN=1 and STS.RDY never becomes 1. I'd say simply check STS.RDY only. If it's 1, then reset the device, otherwise, skip resetting it. What do you think? Thanks, Alex ________________________________ From: Luse, Paul E [mailto:paul.e.luse at intel.com] Sent: Wednesday, August 15, 2012 9:56 AM To: Chang, Alex; nvmewin at lists.openfabrics.org Subject: RE: patch review status OK, I see what you're saying I think. We don't want to check for RDY without first looking at EN so does this make sense: if (CC.EN == 1) { -- now wait for RDY - /* Now reset */ CC.EN = 0; StorPortWriteRegisterUlong(pAE, (PULONG)(&pAE->pCtrlRegister->CC), CC.AsUlong); } From: Chang, Alex [mailto:Alex.Chang at idt.com] Sent: Wednesday, August 15, 2012 9:52 AM To: Luse, Paul E; nvmewin at lists.openfabrics.org Subject: RE: patch review status Hi Paul, I agree with the new logic on #3. However, instead of checking CC.EN, should we check STS.RDY or both? Thanks, Alex ________________________________ From: Luse, Paul E [mailto:paul.e.luse at intel.com] Sent: Wednesday, August 15, 2012 9:03 AM To: Chang, Alex; nvmewin at lists.openfabrics.org Subject: RE: patch review status Thanks Alex, see below. From: Chang, Alex [mailto:Alex.Chang at idt.com] Sent: Wednesday, August 15, 2012 8:51 AM To: Luse, Paul E; nvmewin at lists.openfabrics.org Subject: RE: patch review status Hi Paul, Some questions after browsing the difference between current source and your changes: 1. Why you removed "CC.SHN = 1;" in NVMeNormalShutdown routine? PL> Good eye, that was a mistake. I'll add it back before pushing 2. In nvme.h, why you changed the opcode of NVM_DATASET_MANAGEMENT from 0x06 to 0x09? PL> 6 was wrong, 9 is correct per spec 3. Is there any specific reason(s), you added the following in NVMeInitialize, and then immediately clear EN without checking RDY = 1? /* Set EN to 1 (all others zero) */ CC.EN = 1; StorPortWriteRegisterUlong(pAE, (PULONG)(&pAE->pCtrlRegister->CC), CC.AsUlong); PL> I wrote this up in the patch submission and I can't find that email right now. I didn't add code, I changed code. Before, we use to set EN to 1, read it back, and then set it to 0 to kick off a reset and then later we wait for RDY in the init state machine. We had an IHV complain that their implementation required waiting on RDY after setting EN 1 to even before clearing EN. So, our old implementation was a problem. So what I did now is to not force a reset unless EN is set to 1 when we come in. If its set to 0 then either (a) we're coming from power on or (b) the pre_OS reset the card when it was done. Either way if its already 0 then we're in a known state so we do nothing and then later we set to 1 and wait on RDY. If its 1 when we enter here we set to 0 and then later set to 1 and wait on RDY. Make sense? This assures we never enable without waiting for RDY ***** WAS ******* /* Set EN to 1 (all others zero) */ CC.EN = 1; StorPortWriteRegisterUlong(pAE, (PULONG)(&pAE->pCtrlRegister->CC), CC.AsUlong); /* Read it back */ CC.AsUlong = StorPortReadRegisterUlong(pAE, (PULONG)(&pAE->pCtrlRegister->CC)); /* Now reset */ CC.EN = 0; StorPortWriteRegisterUlong(pAE, (PULONG)(&pAE->pCtrlRegister->CC), CC.AsUlong); CC.AsUlong = StorPortReadRegisterUlong(pAE, (PULONG)(&pAE->pCtrlRegister->CC)); *** NOW IS ******* if (CC.EN == 1) { /* Now reset */ CC.EN = 0; StorPortWriteRegisterUlong(pAE, (PULONG)(&pAE->pCtrlRegister->CC), CC.AsUlong); } Thanks, Alex ________________________________ From:nvmewin-bounces at lists.openfabrics.org[mailto:nvmewin-bounces at lists.openfabrics.org]On Behalf Of Luse, Paul E Sent: Wednesday, August 15, 2012 8:12 AM To: nvmewin at lists.openfabrics.org Subject: [nvmewin] patch review status Just checking in... any questions to date and how are folks tracking for completing their review by this Fri? Thx Paul ____________________________________ Paul Luse Sr. Staff Engineer PCG Server Software Engineering Desk: 480.554.3688, Mobile: 480.334.4630 -------------- next part -------------- An HTML attachment was scrubbed... URL: From paul.e.luse at intel.com Fri Aug 17 08:20:44 2012 From: paul.e.luse at intel.com (Luse, Paul E) Date: Fri, 17 Aug 2012 15:20:44 +0000 Subject: [nvmewin] final block for NVMeInitialize Message-ID: <82C9F782B054C94B9FC04A331649C77A07B390CC@FMSMSX106.amr.corp.intel.com> I'll pass this along to Ray later today, wanted to give everyone one last chance to review... CC.AsUlong = StorPortReadRegisterUlong(pAE, (PULONG)(&pAE->pCtrlRegister->CC)); if (CC.EN == 1) { NVMe_CONTROLLER_STATUS CSTS = {0}; BOOLEAN timeOut = TRUE; ULONG time = 0; /* * Before we transition to 0, make sure the ctrl is actually RDY * NOTE: Some HW implementations may not require this wait and * if not then it could be removed as waiting at this IRQL is * not recommended. The spec is not clear on whether we need * to wait for RDY to transition EN back to 0 or not. */ while (CSTS.RDY != 1) { CSTS.AsUlong = StorPortReadRegisterUlong(pAE, &pAE->pCtrlRegister->CSTS.AsUlong); NVMeCrashDelay(STORPORT_TIMER_CB_us); time += STORPORT_TIMER_CB_us; if (time > pAE->uSecCrtlTimeout) { return (FALSE); } }; /* Now reset */ CC.EN = 0; StorPortWriteRegisterUlong(pAE, (PULONG)(&pAE->pCtrlRegister->CC), CC.AsUlong); } ____________________________________ Paul Luse Sr. Staff Engineer PCG Server Software Engineering Desk: 480.554.3688, Mobile: 480.334.4630 -------------- next part -------------- An HTML attachment was scrubbed... URL: From paul.e.luse at intel.com Fri Aug 17 08:23:37 2012 From: paul.e.luse at intel.com (Luse, Paul E) Date: Fri, 17 Aug 2012 15:23:37 +0000 Subject: [nvmewin] final block for NVMeInitialize Message-ID: <82C9F782B054C94B9FC04A331649C77A07B390EE@FMSMSX106.amr.corp.intel.com> Sorry, didn't send the final one... corrected below From: Luse, Paul E Sent: Friday, August 17, 2012 8:21 AM To: nvmewin at lists.openfabrics.org Subject: final block for NVMeInitialize I'll pass this along to Ray later today, wanted to give everyone one last chance to review... CC.AsUlong = StorPortReadRegisterUlong(pAE, (PULONG)(&pAE->pCtrlRegister->CC)); if (CC.EN == 1) { NVMe_CONTROLLER_STATUS CSTS = {0}; BOOLEAN timeOut = TRUE; ULONG time = 0; /* * Before we transition to 0, make sure the ctrl is actually RDY * NOTE: Some HW implementations may not require this wait and * if not then it could be removed as waiting at this IRQL is * not recommended. The spec is not clear on whether we need * to wait for RDY to transition EN back to 0 or not. */ CSTS.AsUlong = StorPortReadRegisterUlong(pAE, &pAE->pCtrlRegister->CSTS.AsUlong); while (CSTS.RDY != 1) { NVMeCrashDelay(STORPORT_TIMER_CB_us); time += STORPORT_TIMER_CB_us; if (time > pAE->uSecCrtlTimeout) { return (FALSE); } CSTS.AsUlong = StorPortReadRegisterUlong(pAE, &pAE->pCtrlRegister->CSTS.AsUlong); }; /* Now reset */ CC.EN = 0; StorPortWriteRegisterUlong(pAE, (PULONG)(&pAE->pCtrlRegister->CC), CC.AsUlong); } ____________________________________ Paul Luse Sr. Staff Engineer PCG Server Software Engineering Desk: 480.554.3688, Mobile: 480.334.4630 -------------- next part -------------- An HTML attachment was scrubbed... URL: From paul.e.luse at intel.com Fri Aug 17 08:40:40 2012 From: paul.e.luse at intel.com (Luse, Paul E) Date: Fri, 17 Aug 2012 15:40:40 +0000 Subject: [nvmewin] final block for NVMeInitialize In-Reply-To: <82C9F782B054C94B9FC04A331649C77A07B390EE@FMSMSX106.amr.corp.intel.com> References: <82C9F782B054C94B9FC04A331649C77A07B390EE@FMSMSX106.amr.corp.intel.com> Message-ID: <82C9F782B054C94B9FC04A331649C77A07B391DB@FMSMSX106.amr.corp.intel.com> And I won't check in the extra timeOut var either (deleted below) J From: nvmewin-bounces at lists.openfabrics.org [mailto:nvmewin-bounces at lists.openfabrics.org] On Behalf Of Luse, Paul E Sent: Friday, August 17, 2012 8:24 AM To: nvmewin at lists.openfabrics.org Subject: Re: [nvmewin] final block for NVMeInitialize Sorry, didn't send the final one... corrected below From: Luse, Paul E Sent: Friday, August 17, 2012 8:21 AM To: nvmewin at lists.openfabrics.org Subject: final block for NVMeInitialize I'll pass this along to Ray later today, wanted to give everyone one last chance to review... CC.AsUlong = StorPortReadRegisterUlong(pAE, (PULONG)(&pAE->pCtrlRegister->CC)); if (CC.EN == 1) { NVMe_CONTROLLER_STATUS CSTS = {0}; ULONG time = 0; /* * Before we transition to 0, make sure the ctrl is actually RDY * NOTE: Some HW implementations may not require this wait and * if not then it could be removed as waiting at this IRQL is * not recommended. The spec is not clear on whether we need * to wait for RDY to transition EN back to 0 or not. */ CSTS.AsUlong = StorPortReadRegisterUlong(pAE, &pAE->pCtrlRegister->CSTS.AsUlong); while (CSTS.RDY != 1) { NVMeCrashDelay(STORPORT_TIMER_CB_us); time += STORPORT_TIMER_CB_us; if (time > pAE->uSecCrtlTimeout) { return (FALSE); } CSTS.AsUlong = StorPortReadRegisterUlong(pAE, &pAE->pCtrlRegister->CSTS.AsUlong); }; /* Now reset */ CC.EN = 0; StorPortWriteRegisterUlong(pAE, (PULONG)(&pAE->pCtrlRegister->CC), CC.AsUlong); } ____________________________________ Paul Luse Sr. Staff Engineer PCG Server Software Engineering Desk: 480.554.3688, Mobile: 480.334.4630 -------------- next part -------------- An HTML attachment was scrubbed... URL: From Alex.Chang at idt.com Fri Aug 17 09:19:14 2012 From: Alex.Chang at idt.com (Chang, Alex) Date: Fri, 17 Aug 2012 16:19:14 +0000 Subject: [nvmewin] final block for NVMeInitialize In-Reply-To: <82C9F782B054C94B9FC04A331649C77A07B391DB@FMSMSX106.amr.corp.intel.com> References: <82C9F782B054C94B9FC04A331649C77A07B390EE@FMSMSX106.amr.corp.intel.com> <82C9F782B054C94B9FC04A331649C77A07B391DB@FMSMSX106.amr.corp.intel.com> Message-ID: <548C5470AAD9DA4A85D259B663190D36039A1C@corpmail1.na.ads.idt.com> Hi Paul, It makes sense to me. Thanks, Alex ________________________________ From: nvmewin-bounces at lists.openfabrics.org [mailto:nvmewin-bounces at lists.openfabrics.org] On Behalf Of Luse, Paul E Sent: Friday, August 17, 2012 8:41 AM To: Luse, Paul E; nvmewin at lists.openfabrics.org Subject: Re: [nvmewin] final block for NVMeInitialize And I won't check in the extra timeOut var either (deleted below) J From: nvmewin-bounces at lists.openfabrics.org [mailto:nvmewin-bounces at lists.openfabrics.org] On Behalf Of Luse, Paul E Sent: Friday, August 17, 2012 8:24 AM To: nvmewin at lists.openfabrics.org Subject: Re: [nvmewin] final block for NVMeInitialize Sorry, didn't send the final one... corrected below From: Luse, Paul E Sent: Friday, August 17, 2012 8:21 AM To: nvmewin at lists.openfabrics.org Subject: final block for NVMeInitialize I'll pass this along to Ray later today, wanted to give everyone one last chance to review... CC.AsUlong = StorPortReadRegisterUlong(pAE, (PULONG)(&pAE->pCtrlRegister->CC)); if (CC.EN == 1) { NVMe_CONTROLLER_STATUS CSTS = {0}; ULONG time = 0; /* * Before we transition to 0, make sure the ctrl is actually RDY * NOTE: Some HW implementations may not require this wait and * if not then it could be removed as waiting at this IRQL is * not recommended. The spec is not clear on whether we need * to wait for RDY to transition EN back to 0 or not. */ CSTS.AsUlong = StorPortReadRegisterUlong(pAE, &pAE->pCtrlRegister->CSTS.AsUlong); while (CSTS.RDY != 1) { NVMeCrashDelay(STORPORT_TIMER_CB_us); time += STORPORT_TIMER_CB_us; if (time > pAE->uSecCrtlTimeout) { return (FALSE); } CSTS.AsUlong = StorPortReadRegisterUlong(pAE, &pAE->pCtrlRegister->CSTS.AsUlong); }; /* Now reset */ CC.EN = 0; StorPortWriteRegisterUlong(pAE, (PULONG)(&pAE->pCtrlRegister->CC), CC.AsUlong); } ____________________________________ Paul Luse Sr. Staff Engineer PCG Server Software Engineering Desk: 480.554.3688, Mobile: 480.334.4630 -------------- next part -------------- An HTML attachment was scrubbed... URL: From raymond.c.robles at intel.com Fri Aug 17 15:14:13 2012 From: raymond.c.robles at intel.com (Robles, Raymond C) Date: Fri, 17 Aug 2012 22:14:13 +0000 Subject: [nvmewin] NVMe Windows DB is LOCKED - Pushing latest patch from Paul Luse (IOCTL fixes, minor reset fixs, minor init fixes) Message-ID: <49158E750348AA499168FD41D889836018070209@FMSMSX105.amr.corp.intel.com> Locking the NVMe Windows DB. Thanks, Ray [Description: cid:image001.png at 01CB3870.4BB88E70] Raymond C. Robles Attached Platform Storage Software Datacenter Software Division Intel Corporation Desk: 480.554.2600 Mobile: 480.399.0645 -------------- next part -------------- An HTML attachment was scrubbed... URL: -------------- next part -------------- A non-text attachment was scrubbed... Name: image001.png Type: image/png Size: 1756 bytes Desc: image001.png URL: From raymond.c.robles at intel.com Fri Aug 17 15:30:27 2012 From: raymond.c.robles at intel.com (Robles, Raymond C) Date: Fri, 17 Aug 2012 22:30:27 +0000 Subject: [nvmewin] NVMe Windows DB is UNLOCKED - Pushing latest patch from Paul Luse (IOCTL fixes, minor reset fixs, minor init fixes) Message-ID: <49158E750348AA499168FD41D889836018070236@FMSMSX105.amr.corp.intel.com> Latest patch by Intel has been pushed to the trunk. And, as always, I've created a tag for the latest push (ioctl_fix) If anyone has any questions, please free to contact me. Thanks, Ray From: nvmewin-bounces at lists.openfabrics.org [mailto:nvmewin-bounces at lists.openfabrics.org] On Behalf Of Robles, Raymond C Sent: Friday, August 17, 2012 3:14 PM To: nvmewin at lists.openfabrics.org Subject: [nvmewin] NVMe Windows DB is LOCKED - Pushing latest patch from Paul Luse (IOCTL fixes, minor reset fixs, minor init fixes) Locking the NVMe Windows DB. Thanks, Ray [Description: cid:image001.png at 01CB3870.4BB88E70] Raymond C. Robles Attached Platform Storage Software Datacenter Software Division Intel Corporation Desk: 480.554.2600 Mobile: 480.399.0645 -------------- next part -------------- An HTML attachment was scrubbed... URL: -------------- next part -------------- A non-text attachment was scrubbed... Name: image001.png Type: image/png Size: 1756 bytes Desc: image001.png URL: From Alex.Chang at idt.com Tue Aug 28 16:40:29 2012 From: Alex.Chang at idt.com (Chang, Alex) Date: Tue, 28 Aug 2012 23:40:29 +0000 Subject: [nvmewin] ***UNCHECKED*** Aug 28 - Patch (NVMe 1.0c Compliance and others) In-Reply-To: <82C9F782B054C94B9FC04A331649C77A07B24E46@FMSMSX106.amr.corp.intel.com> References: <82C9F782B054C94B9FC04A331649C77A07B24E46@FMSMSX106.amr.corp.intel.com> Message-ID: <548C5470AAD9DA4A85D259B663190D3602B7E213@corpmail1.na.ads.idt.com> Hi all, Here's the next patch (password is idt123). Main changes related to NVMe 1.0c compliance, INTx race condition fix, etc. I'll be looking for feedback/approval by end of next week (September 7). Please let me know if you need more time. Tested as per our guidelines, and no issues. NVMe 1.0c compliance: - Added two functions, NVMeIoctlTxDataToDev and NVMeIoctlTxDataToHost, to handle data transferring via IOCTL vendor defined command - Ensure transfer size is not zero when issuing IOCTL vendor unique command to transfer data. - Added one new IOCTL code NVME_RESET_DEVICE to allow controller reset request from applications if necessary, such as after Firmware Activate command completed successfully. - Changed NUMD to be 0's based for Firmware Image Download and Get Log Page commands. - Changed LID field of Get Log Page command to be 8 bit in length. - Changed Status field of Error Information Log Entry. - Added checking Admin/NVM Vendor Specific Command Configuration bits before processing vendor-defined commands. - Changed Length field of Security Send/Receive commands from 16 to 32 bits. - Added supporting Admin/NVM Vendor Specific command format - Fixed other errors in NVMe related structures INTx race condition fix: - Return TRUE only after confirming the ownership of the interrupt. - Masking interrupt only after confirming the ownership of the interrupt. - Moved interrupt unmasking within the DPC lock, just like masking. Added ensuring all queues are deleted before shutting down the controller normally. Added RDY bit checking whenever EN bit of CC is programmed. Thanks, Alex -------------- next part -------------- An HTML attachment was scrubbed... URL: -------------- next part -------------- A non-text attachment was scrubbed... Name: sources_10c.zip Type: application/x-zip-compressed Size: 167818 bytes Desc: sources_10c.zip URL: From paul.e.luse at intel.com Tue Aug 28 17:34:45 2012 From: paul.e.luse at intel.com (Luse, Paul E) Date: Wed, 29 Aug 2012 00:34:45 +0000 Subject: [nvmewin] ***UNCHECKED*** Aug 28 - Patch (NVMe 1.0c Compliance and others) In-Reply-To: <548C5470AAD9DA4A85D259B663190D3602B7E213@corpmail1.na.ads.idt.com> References: <82C9F782B054C94B9FC04A331649C77A07B24E46@FMSMSX106.amr.corp.intel.com> <548C5470AAD9DA4A85D259B663190D3602B7E213@corpmail1.na.ads.idt.com> Message-ID: <82C9F782B054C94B9FC04A331649C77A07B481A3@FMSMSX106.amr.corp.intel.com> Great, thanks Alex! We'll take a look this week for sure Thx Paul From: nvmewin-bounces at lists.openfabrics.org [mailto:nvmewin-bounces at lists.openfabrics.org] On Behalf Of Chang, Alex Sent: Tuesday, August 28, 2012 4:40 PM To: nvmewin at lists.openfabrics.org Subject: [nvmewin] ***UNCHECKED*** Aug 28 - Patch (NVMe 1.0c Compliance and others) Hi all, Here's the next patch (password is idt123). Main changes related to NVMe 1.0c compliance, INTx race condition fix, etc. I'll be looking for feedback/approval by end of next week (September 7). Please let me know if you need more time. Tested as per our guidelines, and no issues. NVMe 1.0c compliance: - Added two functions, NVMeIoctlTxDataToDev and NVMeIoctlTxDataToHost, to handle data transferring via IOCTL vendor defined command - Ensure transfer size is not zero when issuing IOCTL vendor unique command to transfer data. - Added one new IOCTL code NVME_RESET_DEVICE to allow controller reset request from applications if necessary, such as after Firmware Activate command completed successfully. - Changed NUMD to be 0's based for Firmware Image Download and Get Log Page commands. - Changed LID field of Get Log Page command to be 8 bit in length. - Changed Status field of Error Information Log Entry. - Added checking Admin/NVM Vendor Specific Command Configuration bits before processing vendor-defined commands. - Changed Length field of Security Send/Receive commands from 16 to 32 bits. - Added supporting Admin/NVM Vendor Specific command format - Fixed other errors in NVMe related structures INTx race condition fix: - Return TRUE only after confirming the ownership of the interrupt. - Masking interrupt only after confirming the ownership of the interrupt. - Moved interrupt unmasking within the DPC lock, just like masking. Added ensuring all queues are deleted before shutting down the controller normally. Added RDY bit checking whenever EN bit of CC is programmed. Thanks, Alex -------------- next part -------------- An HTML attachment was scrubbed... URL: