From iuliur at google.com Tue Jun 2 15:35:56 2015 From: iuliur at google.com (Iuliu Rus) Date: Tue, 2 Jun 2015 15:35:56 -0700 Subject: [nvmewin] OFA Release Plan: Release #1 2015 -- Call for patch volunteers In-Reply-To: <49158E750348AA499168FD41D88983606B669C18@fmsmsx117.amr.corp.intel.com> References: <49158E750348AA499168FD41D88983606B63477F@fmsmsx117.amr.corp.intel.com> <49158E750348AA499168FD41D88983606B656163@fmsmsx117.amr.corp.intel.com> <49158E750348AA499168FD41D88983606B6697AD@fmsmsx117.amr.corp.intel.com> <49158E750348AA499168FD41D88983606B669C18@fmsmsx117.amr.corp.intel.com> Message-ID: Raymond, given this, project we are going to wait for QEMU to accept the virtualization extensions first ( http://git.infradead.org/users/kbusch/qemu-nvme.git/commit/162ec6479447abbd76308f831f36703aa3acc485), then submit our payload which includes what we previously discussed and the windows implementation for the nvme doorbell extensions. On Fri, May 15, 2015 at 11:38 AM, Robles, Raymond C < raymond.c.robles at intel.com> wrote: > Hi Iuliu, > > > > The process for submitting a patch is quite simple (by design). The steps > are as follows: > > > > 1. Download the current latest trunk source via an SVN client (i.e. > TortoiseSVN). I assume you’ve already done this. > > 2. In the Releases folder, under a specific release, there are > readme.txt files that explain the build environment needed. Insure you can > compile the trunk with the recommended tool/build set. > > 3. Port your changes over to the trunk. > > 4. Unit test your changes on the trunk source. > > 5. Run the required testing outlined in the file attached in this > email and insure all tests are passing. > > 6. Send an email with the trunk source zipped (with your ported > changes) to this email distribution list. > > 7. The three reviewing companies will review your changes and also > run the tests in the file. > > 8. If all looks good, then the source maintainer will merge your > changes to the trunk, and create a tag. > > 9. If there are any question/comments that require a modifications > to you patch, then go back to step 4 after you’ve made any changes and > complete the process again. > > > > Let me know if you have any further questions. Feel free to add both > patches into a single submission. > > > > You should be able to download all tools necessary outlined in the file. > Let us know if you need assistance. > > > > Thanks, > > Ray > > > > *From:* Iuliu Rus [mailto:iuliur at google.com] > *Sent:* Thursday, May 14, 2015 11:02 PM > > *To:* Robles, Raymond C > *Cc:* nvmewin at lists.openfabrics.org > *Subject:* Re: [nvmewin] OFA Release Plan: Release #1 2015 -- Call for > patch volunteers > > > > Can you please point me to the instructions on how to send the patch? > > As for the free q list issue, we enabled driver verifier and under stress > it bugchecked in RtlpCheckListEntry. > > > > On Thu, May 14, 2015 at 10:52 PM, Robles, Raymond C < > raymond.c.robles at intel.com> wrote: > > Hello, > > > > We always welcome companies to submit patches for candidate features. No > other company has volunteered for concurrent channels at this time. We > would gladly welcome your patch at your earliest convenience. The first > release for 2015 is planned for ~August/September... would you like to > volunteer for this patch prior to this release? > > > > As for a race condition synchronization fix for FreeQList, could you > please describe the issue you were seeing and how you addressed it? > > > > Thanks, > > Ray > > > > *From:* Iuliu Rus [mailto:iuliur at google.com] > *Sent:* Thursday, May 14, 2015 7:43 PM > *To:* Robles, Raymond C > *Cc:* nvmewin at lists.openfabrics.org > *Subject:* Re: [nvmewin] OFA Release Plan: Release #1 2015 -- Call for > patch volunteers > > > > Hello, > > We are the google compute engine team and we already have Concurrent > channels working. We're still figuring out how we can integrate between our > internal repo and svn. > > Do you still need this feature? > > We also have a synchronization fix for a race that corrupted FreeQList. > > > > On Thu, May 7, 2015 at 3:46 PM, Robles, Raymond C < > raymond.c.robles at intel.com> wrote: > > All, > > > > I just wanted to send out a quick update on the release plans for 2015. > We’ve had two more companies volunteer to submit patches this year: Samsung > (Suman) and HGST (Tom). Thank you to both companies for volunteering! > > > > Here is the updated release plan for 2015. Unfortunately, some of the > patch submissions will be a little later than originally planned. > Therefore, I would like to ask the question: *is everyone ok if we push > the first release out until end of August to allow for more patches to be > submitted?* You can response publicly or just to me. > > > > Other alternatives would include just having smaller releases at different > cadences. I’m open to any ideas since there are still many requests for > features that have claimed for patch submission. > > > > > > *2015 Release #1 (Q2/Q3 – 2015)* > > > > *Feature* > > *Description* > > *Submitter* > > *Date* > > Concurrent Channels > > Start I/O Concurrent Channels allows StartIo to be called in parallel > resulting in improved IOPS. > > > > > > Perf Opts > > Instead of learning cores, use perf opts. Shave off 2 seconds in init > code… using Storport initialize perf opts. No need for source core > interrupt steering code in driver. > > Samsung (Suman) > > End of June > > Win 8.1 Timers > > Storport Notification usage. Hot plug and IOCTLs, this does not work, for > Win8.1… StorportRequestTimer() function needed. > > Samsung (Suman) > > End of July > > CFS Bit Monitoring > > CFS bit monitoring and handling. Look into adding additional handling code > just to monitor CSTS.CFS. > > > > > > EOL - Read Only > > Read only support for devices at EOL. Detection at init, hot plug, or > run-time. > > Samsung (Suman) > > End of August > > Namespace Mgmt > > Namespace management - creation and deletion of namespaces. > > Intel (Carolyn) > > End of June > > WHQL > > WHQL test suite run (any bug fixes). > > > > > > > > > > *2015 Release #2 (Q4 – 2015)* > > > > *Feature* > > *Description* > > *Submitter* > > *Date* > > Multi-Path > > Multipath feature set from 1.1 spec (Multi-path I/O, Namespace Sharing, > Reservations) > > HGST (Tom) > > Mid-November > > RTD3 > > Active/idle RTD3 > > > > > > Temp Thresholds > > Temperature thresholds (Get/Set Feature and AER trigger) > > > > > > Live FW Update > > Live firmware update > > > > > > Atomicity > > Atomicity Enhancements > > > > > > Win 10 > > Win 10 support (push to Q4 release) > > > > > > SNTL > > SNTL 1.5 doc updates > > HGST (Tom) > > End of August > > WHQL > > WHQL test suite run (any bug fixes) - for each release of OFA > > > > > > > > Thanks, > > Ray > > > > *From:* nvmewin-bounces at lists.openfabrics.org [mailto: > nvmewin-bounces at lists.openfabrics.org] *On Behalf Of *Robles, Raymond C > *Sent:* Thursday, April 09, 2015 5:29 PM > *To:* nvmewin at lists.openfabrics.org > *Subject:* [nvmewin] OFA Release Plan: Release #1 2015 -- Call for patch > volunteers > > > > All, > > > > As we enter Q2, I’m happy to announce that we a have a very good feature > plan in place for both Q2 and Q4 releases in 2015 of the OFA driver. The > list of requested features is included below for final review. > > > > I would like to formally ask for volunteers for patch submissions. As of > today, there are no volunteers for any of the below patches. > > > > Intel would like to start with the volunteering… we will volunteer to > provide the namespace management patch. > > > > *2015 Release #1 (Q2/Q3 – 2015)* > > > > *Feature* > > *Description* > > *Submitter* > > *Date* > > Concurrent Channels > > Start I/O Concurrent Channels allows StartIo to be called in parallel > resulting in improved IOPS. > > > > > > Perf Opts > > Instead of learning cores, use perf opts. Shave off 2 seconds in init > code… using Storport initialize perf opts. No need for source core > interrupt steering code in driver. > > Samsung (Suman) > > End of June > > Win 8.1 Timers > > Storport Notification usage. Hot plug and IOCTLs, this does not work, for > Win8.1… StorportRequestTimer() function needed. > > Samsung (Suman) > > End of July > > CFS Bit Monitoring > > CFS bit monitoring and handling. Look into adding additional handling code > just to monitor CSTS.CFS. > > > > > > EOL - Read Only > > Read only support for devices at EOL. Detection at init, hot plug, or > run-time. > > Samsung (Suman) > > End of August > > Namespace Mgmt > > Namespace management - creation and deletion of namespaces. > > Intel (Carolyn) > > End of June > > WHQL > > WHQL test suite run (any bug fixes). > > > > > > > > > > *2015 Release #2 (Q4 – 2015)* > > > > *Feature* > > *Description* > > *Submitter* > > *Date* > > Multi-Path > > Multipath feature set from 1.1 spec (Multi-path I/O, Namespace Sharing, > Reservations) > > HGST (Tom) > > Mid-November > > RTD3 > > Active/idle RTD3 > > > > > > Temp Thresholds > > Temperature thresholds (Get/Set Feature and AER trigger) > > > > > > Live FW Update > > Live firmware update > > > > > > Atomicity > > Atomicity Enhancements > > > > > > Win 10 > > Win 10 support (push to Q4 release) > > > > > > SNTL > > SNTL 1.5 doc updates > > HGST (Tom) > > End of August > > WHQL > > WHQL test suite run (any bug fixes) - for each release of OFA > > > > > > > > > > Thanks, > > Ray > > > > *Raymond C. Robles* > > NSG ISE Host Storage Software > > Intel Corporation > > Office: 480-554-2600 > > Mobile: 480-399-0645 > > > > > _______________________________________________ > nvmewin mailing list > nvmewin at lists.openfabrics.org > http://lists.openfabrics.org/mailman/listinfo/nvmewin > > > > > -------------- next part -------------- An HTML attachment was scrubbed... URL: From suman.p at samsung.com Mon Jun 15 10:36:11 2015 From: suman.p at samsung.com (SUMAN PRAKASH B) Date: Mon, 15 Jun 2015 17:36:11 +0000 (GMT) Subject: [nvmewin] regarding incorrect core-queue-msix mapping in 1.4 OFA code Message-ID: <49.F6.18557.B0D0F755@epcpsbgx3.samsung.com> An HTML attachment was scrubbed... URL: -------------- next part -------------- A non-text attachment was scrubbed... Name: 201506152308025_BSL8PYMC.gif Type: image/gif Size: 32398 bytes Desc: not available URL: -------------- next part -------------- A non-text attachment was scrubbed... Name: 201506152308043_4XEV4D4T.gif Type: image/gif Size: 31273 bytes Desc: not available URL: From wuwq85 at gmail.com Mon Jun 15 15:32:20 2015 From: wuwq85 at gmail.com (Wenqian Wu) Date: Mon, 15 Jun 2015 15:32:20 -0700 Subject: [nvmewin] nvmewin Digest, Vol 38, Issue 9 In-Reply-To: <49158E750348AA499168FD41D88983606B5EC770@fmsmsx117.amr.corp.intel.com> References: <49158E750348AA499168FD41D88983606B5EC770@fmsmsx117.amr.corp.intel.com> Message-ID: Hi Ray, I brought up this question again due to we still found the issue using Windows OFA driver. Basically the controller reports supports 32 entries per queue, but the driver will do some calculation and try to create a IO queue with 64 entries per queue. Then controller will report error. Linux/Windows inbox driver didn't see such issue. NVme spec only states the queue base address needs to be page aligned, but not the queue size(number of entries). Please reconsider it. Thanks! [image: Inline image 1] Thanks, Wenqian On Tue, Feb 17, 2015 at 4:17 PM, Robles, Raymond C < raymond.c.robles at intel.com> wrote: > Hi, > > > > The code below simply rounds up the memory allocated to the next page > boundary. The NVMe controller expects this memory (queue size) to be host > page aligned. Queue size will still be within the limits of what the > controller communicates. > > > > Thanks, > > Ray > > > > *From:* nvmewin-bounces at lists.openfabrics.org [mailto: > nvmewin-bounces at lists.openfabrics.org] *On Behalf Of *Wenqian Wu > *Sent:* Sunday, February 15, 2015 3:43 PM > *To:* nvmewin at lists.openfabrics.org > *Subject:* Re: [nvmewin] nvmewin Digest, Vol 38, Issue 9 > > > > Hi Ray, > > > > Thanks for your reply. I understand your saying that before calling NVMeAllocQueues, > the CAP.MQES has been checked and passed in as 3rd parameter(QEntries). But > if you look further in NVMeAllocQueues function, QEntires will be modified. > > > > if ((QEntries % SysPageSizeInSubEntries) != 0) > > QEntries = (QEntries + SysPageSizeInSubEntries) & > > ~(SysPageSizeInSubEntries - 1); > > > > And finally QEntires will be used to create the Queue. But QEntries may > already bigger the CAP.MQES after the above round up. > > > > [image: Inline image 1] > > > > A simple example is if controller reports CAP.MQES is 32, host memory page > size is 4K. The round up procedure will change QEntries to 64 (4 * 1024 / > 64), and used to create the queue. Controller will returen Invalid Queue > Size and failed the command. > > > > In summary, I think the alignment requirement is for host memory only. > Driver can still allocate 4K(page aligned) for 32 entries for the above > example, just keeping the second half unused. But when creating the queue, > the entries should be 32, instead of 64. > > > > Please let me know if this makes sense. > > > > Thanks, > > Wenqian > > > > > > On Fri, Feb 13, 2015 at 12:00 PM, > wrote: > > Send nvmewin mailing list submissions to > nvmewin at lists.openfabrics.org > > To subscribe or unsubscribe via the World Wide Web, visit > http://lists.openfabrics.org/mailman/listinfo/nvmewin > or, via email, send a message with subject or body 'help' to > nvmewin-request at lists.openfabrics.org > > You can reach the person managing the list at > nvmewin-owner at lists.openfabrics.org > > When replying, please edit your Subject line so it is more specific > than "Re: Contents of nvmewin digest..." > > > Today's Topics: > > 1. Re: NVMe Queue entry number (Robles, Raymond C) > > > ---------------------------------------------------------------------- > > Message: 1 > Date: Fri, 13 Feb 2015 04:42:37 +0000 > From: "Robles, Raymond C" > To: 'Wenqian Wu' , "nvmewin at lists.openfabrics.org" > > Subject: Re: [nvmewin] NVMe Queue entry number > Message-ID: > < > 49158E750348AA499168FD41D88983606B5DA191 at fmsmsx117.amr.corp.intel.com> > > Content-Type: text/plain; charset="utf-8" > > Hi Wenqian, > > Thank you for your inquiry. In the FindAdapter routine (in nvmeStd.c), the > driver checks CAP.MQES and saves the value in pAE->InitInfo.IoQEntries (if > the default or register override is smaller than CAP.MQES, it is saved > instead). Within the function NVMeAllocIoQueues, there is loop that will > iterate to create queue pairs based on the number of cpu cores. > > In this loop, a call to NVMeAllocQueues is made. Just prior to this call, > the value saved in pAE->InitInfo.IoQEntires is retrieved (stack variable > ?QEntres?) and passed in as the 3rd parameter. So, once the allocation is > taking place where you mention below, the 3rd parameter of the function has > already been checked against CAP.MQES. Also, per the NVMe spec sections 5.3 > and 5.4 ? Figures 33 and 37 (create completion queue and submission queue > PRP 1), all queue memory must be ?physically contiguous and memory page > aligned?. > > Thanks > Ray > > From: nvmewin-bounces at lists.openfabrics.org [mailto: > nvmewin-bounces at lists.openfabrics.org] On Behalf Of Wenqian Wu > Sent: Wednesday, February 11, 2015 4:58 PM > To: nvmewin at lists.openfabrics.org > Subject: [nvmewin] NVMe Queue entry number > > Hi OFA driver member, > > I have one question regarding the Queue entry size. The driver will > allocate number of entries aligned to memory page (line877, nvmeInit.c), > instead of the actual queue size controller supports (CAP.MQES). Controller > can return error if host request more entries while ignoring controller's > capability. I think as long as the base address is page aligned, there is > no reason to make the entries number aligned to page boundary. Can this be > considered a driver bug or is there any particular consideration? > > Thanks, > Wenqian > -------------- next part -------------- > An HTML attachment was scrubbed... > URL: < > http://lists.openfabrics.org/pipermail/nvmewin/attachments/20150213/edd26ac2/attachment-0001.html > > > > ------------------------------ > > _______________________________________________ > nvmewin mailing list > nvmewin at lists.openfabrics.org > http://lists.openfabrics.org/mailman/listinfo/nvmewin > > > End of nvmewin Digest, Vol 38, Issue 9 > ************************************** > > > -------------- next part -------------- An HTML attachment was scrubbed... URL: -------------- next part -------------- A non-text attachment was scrubbed... Name: image.png Type: image/png Size: 146097 bytes Desc: not available URL: -------------- next part -------------- A non-text attachment was scrubbed... Name: image001.png Type: image/png Size: 94158 bytes Desc: not available URL: From carolyn.d.foster at intel.com Mon Jun 15 17:20:58 2015 From: carolyn.d.foster at intel.com (Foster, Carolyn D) Date: Tue, 16 Jun 2015 00:20:58 +0000 Subject: [nvmewin] regarding incorrect core-queue-msix mapping in 1.4 OFA code In-Reply-To: <49.F6.18557.B0D0F755@epcpsbgx3.samsung.com> References: <49.F6.18557.B0D0F755@epcpsbgx3.samsung.com> Message-ID: Hi Suman, There are two important boundary conditions to consider with analyzing the queue to core and MSI-X mapping. These boundary conditions include systems with more than 32 cores and also more than 64 cores. On a system with more than 32 cores, in your example below, it's possible some cores won't get mapped to the appropriate MSI-X vectors. The other boundary condition is when the system has more than 64 cores. In this case, the core numbering may not be consecutive. In both cases, without re-creating the queues, there are likely to be problems. All that being said, I do see areas for improvements in terms of memory allocation per NUMA node and the core to queue pair assignment. Is it possible for you to do more investigation and to come back with a proposal for a fix and potentially a patch? Thanks, Carolyn From: nvmewin-bounces at lists.openfabrics.org [mailto:nvmewin-bounces at lists.openfabrics.org] On Behalf Of SUMAN PRAKASH B Sent: Monday, June 15, 2015 10:36 AM To: nvmewin at lists.openfabrics.org Subject: [nvmewin] regarding incorrect core-queue-msix mapping in 1.4 OFA code Dear All, The core-queue-msix mapping(NUMA implementation) in the 1.4 OFA code doesn't look correct. During learning-cores, after the driver sends a read command on each queue to get the Core to MSIx mapping, the driver updates the following: a) MSIx vector corresponding to core. b) Queue assignment with corresponding core. Because of b, the queue assignment is changed but the memory allocation remains unchanged. Then when the queues are deleted and re-created, we see that the queues are created on memory from other nodes, which should not be the case, as this will introduce remote node memory access which might impact the performance. We tested on a 32 logical processor server with 2 NUMA nodes and following is the mapping after learning cores - [cid:image001.gif at 01D0A785.DFC282E0] Ideally, during learning-cores, after the driver sends a read command on each queue to get the Core to MSIx mapping, the driver should update only the following: a) MSIx vector corresponding to core. With only a), following is the core-queue-msix mapping: [cid:image002.gif at 01D0A785.DFC282E0] Any comments? Thanks, Suman [Image removed by sender.] -------------- next part -------------- An HTML attachment was scrubbed... URL: -------------- next part -------------- A non-text attachment was scrubbed... Name: ~WRD000.jpg Type: image/jpeg Size: 823 bytes Desc: ~WRD000.jpg URL: -------------- next part -------------- A non-text attachment was scrubbed... Name: image001.gif Type: image/gif Size: 32398 bytes Desc: image001.gif URL: -------------- next part -------------- A non-text attachment was scrubbed... Name: image002.gif Type: image/gif Size: 31273 bytes Desc: image002.gif URL: From cheng.peng at memblaze.com Thu Jun 18 02:13:20 2015 From: cheng.peng at memblaze.com (Cheng Peng) Date: Thu, 18 Jun 2015 09:13:20 +0000 Subject: [nvmewin] send ADMIN_FIRMWARE_IMAGE_DOWNLOAD command fail Message-ID: Hi all When I send ADMIN_FIRMWARE_IMAGE_DOWNLOAD command to driver by DeviceIoControl, I get error code 1450 (ERROR_NO_SYSTEM_RESOURCES) I find the driver don't receive the ioctl command. It is as if the size of firmware image is larger than pPCI->MaximumTransferLength. How to resolve the problem? Thank you -------------- next part -------------- An HTML attachment was scrubbed... URL: From raymond.c.robles at intel.com Thu Jun 18 17:59:57 2015 From: raymond.c.robles at intel.com (Robles, Raymond C) Date: Fri, 19 Jun 2015 00:59:57 +0000 Subject: [nvmewin] nvmewin Digest, Vol 38, Issue 9 In-Reply-To: References: <49158E750348AA499168FD41D88983606B5EC770@fmsmsx117.amr.corp.intel.com> Message-ID: <49158E750348AA499168FD41D88983606B6A9F27@fmsmsx117.amr.corp.intel.com> Hi Wenqian, I understand what you are stating. If you are observing behavior in the OFA driver that is allocating more entries than is defined via CAP.MQES, then this is a bug. Would you be able to supply a patch to the OFA driver and send it out for review (to this mailing list)? Thanks, Ray From: Wenqian Wu [mailto:wuwq85 at gmail.com] Sent: Monday, June 15, 2015 3:32 PM To: Robles, Raymond C; nancyx at marvell.com; renlei at marvell.com; Joyce Yang Cc: nvmewin at lists.openfabrics.org Subject: Re: [nvmewin] nvmewin Digest, Vol 38, Issue 9 Hi Ray, I brought up this question again due to we still found the issue using Windows OFA driver. Basically the controller reports supports 32 entries per queue, but the driver will do some calculation and try to create a IO queue with 64 entries per queue. Then controller will report error. Linux/Windows inbox driver didn't see such issue. NVme spec only states the queue base address needs to be page aligned, but not the queue size(number of entries). Please reconsider it. Thanks! [Inline image 1] Thanks, Wenqian On Tue, Feb 17, 2015 at 4:17 PM, Robles, Raymond C > wrote: Hi, The code below simply rounds up the memory allocated to the next page boundary. The NVMe controller expects this memory (queue size) to be host page aligned. Queue size will still be within the limits of what the controller communicates. Thanks, Ray From: nvmewin-bounces at lists.openfabrics.org [mailto:nvmewin-bounces at lists.openfabrics.org] On Behalf Of Wenqian Wu Sent: Sunday, February 15, 2015 3:43 PM To: nvmewin at lists.openfabrics.org Subject: Re: [nvmewin] nvmewin Digest, Vol 38, Issue 9 Hi Ray, Thanks for your reply. I understand your saying that before calling NVMeAllocQueues, the CAP.MQES has been checked and passed in as 3rd parameter(QEntries). But if you look further in NVMeAllocQueues function, QEntires will be modified. if ((QEntries % SysPageSizeInSubEntries) != 0) QEntries = (QEntries + SysPageSizeInSubEntries) & ~(SysPageSizeInSubEntries - 1); And finally QEntires will be used to create the Queue. But QEntries may already bigger the CAP.MQES after the above round up. [Inline image 1] A simple example is if controller reports CAP.MQES is 32, host memory page size is 4K. The round up procedure will change QEntries to 64 (4 * 1024 / 64), and used to create the queue. Controller will returen Invalid Queue Size and failed the command. In summary, I think the alignment requirement is for host memory only. Driver can still allocate 4K(page aligned) for 32 entries for the above example, just keeping the second half unused. But when creating the queue, the entries should be 32, instead of 64. Please let me know if this makes sense. Thanks, Wenqian On Fri, Feb 13, 2015 at 12:00 PM, > wrote: Send nvmewin mailing list submissions to nvmewin at lists.openfabrics.org To subscribe or unsubscribe via the World Wide Web, visit http://lists.openfabrics.org/mailman/listinfo/nvmewin or, via email, send a message with subject or body 'help' to nvmewin-request at lists.openfabrics.org You can reach the person managing the list at nvmewin-owner at lists.openfabrics.org When replying, please edit your Subject line so it is more specific than "Re: Contents of nvmewin digest..." Today's Topics: 1. Re: NVMe Queue entry number (Robles, Raymond C) ---------------------------------------------------------------------- Message: 1 Date: Fri, 13 Feb 2015 04:42:37 +0000 From: "Robles, Raymond C" > To: 'Wenqian Wu' >, "nvmewin at lists.openfabrics.org" > Subject: Re: [nvmewin] NVMe Queue entry number Message-ID: <49158E750348AA499168FD41D88983606B5DA191 at fmsmsx117.amr.corp.intel.com> Content-Type: text/plain; charset="utf-8" Hi Wenqian, Thank you for your inquiry. In the FindAdapter routine (in nvmeStd.c), the driver checks CAP.MQES and saves the value in pAE->InitInfo.IoQEntries (if the default or register override is smaller than CAP.MQES, it is saved instead). Within the function NVMeAllocIoQueues, there is loop that will iterate to create queue pairs based on the number of cpu cores. In this loop, a call to NVMeAllocQueues is made. Just prior to this call, the value saved in pAE->InitInfo.IoQEntires is retrieved (stack variable ?QEntres?) and passed in as the 3rd parameter. So, once the allocation is taking place where you mention below, the 3rd parameter of the function has already been checked against CAP.MQES. Also, per the NVMe spec sections 5.3 and 5.4 ? Figures 33 and 37 (create completion queue and submission queue PRP 1), all queue memory must be ?physically contiguous and memory page aligned?. Thanks Ray From: nvmewin-bounces at lists.openfabrics.org [mailto:nvmewin-bounces at lists.openfabrics.org] On Behalf Of Wenqian Wu Sent: Wednesday, February 11, 2015 4:58 PM To: nvmewin at lists.openfabrics.org Subject: [nvmewin] NVMe Queue entry number Hi OFA driver member, I have one question regarding the Queue entry size. The driver will allocate number of entries aligned to memory page (line877, nvmeInit.c), instead of the actual queue size controller supports (CAP.MQES). Controller can return error if host request more entries while ignoring controller's capability. I think as long as the base address is page aligned, there is no reason to make the entries number aligned to page boundary. Can this be considered a driver bug or is there any particular consideration? Thanks, Wenqian -------------- next part -------------- An HTML attachment was scrubbed... URL: ------------------------------ _______________________________________________ nvmewin mailing list nvmewin at lists.openfabrics.org http://lists.openfabrics.org/mailman/listinfo/nvmewin End of nvmewin Digest, Vol 38, Issue 9 ************************************** -------------- next part -------------- An HTML attachment was scrubbed... URL: -------------- next part -------------- A non-text attachment was scrubbed... Name: image002.png Type: image/png Size: 174905 bytes Desc: image002.png URL: -------------- next part -------------- A non-text attachment was scrubbed... Name: image003.png Type: image/png Size: 94158 bytes Desc: image003.png URL: From raymond.c.robles at intel.com Mon Jun 22 14:33:26 2015 From: raymond.c.robles at intel.com (Robles, Raymond C) Date: Mon, 22 Jun 2015 21:33:26 +0000 Subject: [nvmewin] send ADMIN_FIRMWARE_IMAGE_DOWNLOAD command fail In-Reply-To: References: Message-ID: <49158E750348AA499168FD41D88983606B6AB59A@fmsmsx117.amr.corp.intel.com> Hi Cheng, Could you please provide contents of the pass through IOCTL structure you create? Also, any of the sample user space code you use to create the IOCTL would be helpful. In addition, have you referenced the PT IOCTL document on the nvmewin repo? This document has sample code on how to construct pass through IOCTL's. Thanks, Ray From: nvmewin-bounces at lists.openfabrics.org [mailto:nvmewin-bounces at lists.openfabrics.org] On Behalf Of Cheng Peng Sent: Thursday, June 18, 2015 2:13 AM To: nvmewin at lists.openfabrics.org Subject: [nvmewin] send ADMIN_FIRMWARE_IMAGE_DOWNLOAD command fail Hi all When I send ADMIN_FIRMWARE_IMAGE_DOWNLOAD command to driver by DeviceIoControl, I get error code 1450 (ERROR_NO_SYSTEM_RESOURCES) I find the driver don't receive the ioctl command. It is as if the size of firmware image is larger than pPCI->MaximumTransferLength. How to resolve the problem? Thank you -------------- next part -------------- An HTML attachment was scrubbed... URL: From cheng.peng at memblaze.com Mon Jun 22 20:05:42 2015 From: cheng.peng at memblaze.com (Cheng Peng) Date: Tue, 23 Jun 2015 03:05:42 +0000 Subject: [nvmewin] =?gb2312?b?tPC4tDogc2VuZCBBRE1JTl9GSVJNV0FSRV9JTUFHRV9E?= =?gb2312?b?T1dOTE9BRCBjb21tYW5kIGZhaWw=?= In-Reply-To: <49158E750348AA499168FD41D88983606B6AB59A@fmsmsx117.amr.corp.intel.com> References: <49158E750348AA499168FD41D88983606B6AB59A@fmsmsx117.amr.corp.intel.com> Message-ID: Hi Ray Thanks for your reply. I have referenced the PT IOCTL document on the nvmewin repo I find a related thread: https://www.osronline.com/showthread.cfm?link=249023 The code is following: int nvme_fwdownload(int argc, char **argv) { int c, option_index = 0, fw_image = 0; int error, slot_used = 0, total_slots = 0, slot; int activate = 1, device_index = 0; HANDLE fd, mapobj, ssd; void *mapped_file_addr; unsigned long size, len; PNVME_PASS_THROUGH_IOCTL ioctl_buffer = NULL; PNVMe_COMMAND nvme_cmd = NULL; PADMIN_FIRMWARE_IMAGE_DOWNLOAD_COMMAND_DW10 cdw10; /* * firmware transfer works in units of Dwords, * so we do check here to find the special case. */ if (size % 4) { printf("Fwdownload: size of the image file %llu bytes\n", size); exit(1); } mapobj = CreateFileMapping(fd, NULL, PAGE_READONLY, 0, 0, NULL); if (mapobj == NULL) { error = GetLastError(); printf("Fwdownload: CreateFileMapping fail, error %d\n", error); exit(1); } mapped_file_addr = MapViewOfFile(mapobj, FILE_MAP_READ, 0, 0, size); if (mapped_file_addr == NULL) { error = GetLastError(); printf("Fwdownload: MapViewofFile fail, error %d\n", error); exit(1); } len = sizeof(NVME_PASS_THROUGH_IOCTL) + size - 1; ioctl_buffer = (PNVME_PASS_THROUGH_IOCTL)malloc(len); memset(ioctl_buffer, 0, len); memcpy(ioctl_buffer->DataBuffer, mapped_file_addr, size); /* * unmap it immediately after copying the fw to databuffer. * In case there is somewhere modify the fw. */ UnmapViewOfFile(mapped_file_addr); mapped_file_addr = NULL; CloseHandle(mapobj); CloseHandle(fd); ioctl_buffer->SrbIoCtrl.HeaderLength = sizeof(SRB_IO_CONTROL); memcpy(ioctl_buffer->SrbIoCtrl.Signature, NVME_SIG_STR, sizeof(NVME_SIG_STR)); ioctl_buffer->SrbIoCtrl.Timeout= 30; ioctl_buffer->SrbIoCtrl.ControlCode = (ULONG)NVME_PASS_THROUGH_SRB_IO_CODE; ioctl_buffer->SrbIoCtrl.Length = len - sizeof(SRB_IO_CONTROL); nvme_cmd = (PNVMe_COMMAND)ioctl_buffer->NVMeCmd; nvme_cmd->CDW0.OPC = ADMIN_FIRMWARE_IMAGE_DOWNLOAD; nvme_cmd->NSID = 0; cdw10 = (PADMIN_FIRMWARE_IMAGE_DOWNLOAD_COMMAND_DW10)&nvme_cmd->CDW10; cdw10->NUMD = size / 4 - 1; ioctl_buffer->QueueId = 0; /* Admin queue */ ioctl_buffer->DataBufferLen = size; ioctl_buffer->Direction = NVME_FROM_HOST_TO_DEV; ioctl_buffer->ReturnBufferLen = len; printf("firmware image size %llu\n", size); if(nvme_ioctl(ssd, ioctl_buffer, len, 0)) exit(1); if (activate) { printf("auto-activate this firmware...\n"); nvme_get_current_slot(&slot_used, &ssd); printf("slot used: %d total_slots: %d\n", slot_used, total_slots); slot = (slot_used + 1 > total_slots ? 2 : ++slot_used); activate_slot_action(&ssd, slot, 1); } if (ioctl_buffer) { free(ioctl_buffer); ioctl_buffer = NULL; } nvme_close_device(&ssd); ssd = NULL; return 0; } 发件人: Robles, Raymond C [mailto:raymond.c.robles at intel.com] 发送时间: Tuesday, June 23, 2015 5:33 AM 收件人: Cheng Peng; nvmewin at lists.openfabrics.org 主题: RE: send ADMIN_FIRMWARE_IMAGE_DOWNLOAD command fail Hi Cheng, Could you please provide contents of the pass through IOCTL structure you create? Also, any of the sample user space code you use to create the IOCTL would be helpful. In addition, have you referenced the PT IOCTL document on the nvmewin repo? This document has sample code on how to construct pass through IOCTL’s. Thanks, Ray From: nvmewin-bounces at lists.openfabrics.org [mailto:nvmewin-bounces at lists.openfabrics.org] On Behalf Of Cheng Peng Sent: Thursday, June 18, 2015 2:13 AM To: nvmewin at lists.openfabrics.org Subject: [nvmewin] send ADMIN_FIRMWARE_IMAGE_DOWNLOAD command fail Hi all When I send ADMIN_FIRMWARE_IMAGE_DOWNLOAD command to driver by DeviceIoControl, I get error code 1450 (ERROR_NO_SYSTEM_RESOURCES) I find the driver don’t receive the ioctl command. It is as if the size of firmware image is larger than pPCI->MaximumTransferLength. How to resolve the problem? Thank you -------------- next part -------------- An HTML attachment was scrubbed... URL: From iuliur at google.com Thu Jun 25 12:05:50 2015 From: iuliur at google.com (Iuliu Rus) Date: Thu, 25 Jun 2015 12:05:50 -0700 Subject: [nvmewin] OFA Release Plan: Release #1 2015 -- Call for patch volunteers In-Reply-To: References: <49158E750348AA499168FD41D88983606B63477F@fmsmsx117.amr.corp.intel.com> <49158E750348AA499168FD41D88983606B656163@fmsmsx117.amr.corp.intel.com> <49158E750348AA499168FD41D88983606B6697AD@fmsmsx117.amr.corp.intel.com> <49158E750348AA499168FD41D88983606B669C18@fmsmsx117.amr.corp.intel.com> Message-ID: Raymond, we are running the windows HCK fuzz test on the driver and it exposes several crashes. Also, on SCSIOP_START_STOP_UNIT it looks like SntiTransitionPowerState builds an io which it never submits, causing Windows to eventually reset the adapter. Given that some of these fuzz crashes are potential security issues, we are getting ready to submit the patch next week. However it is rather big and the current patch submission process doesn't work that well. We would like to break it into separate changes, but with zipped thrunk this is not possible. Is there any way to switch this over to github so we can use pull requests instead? Or, should we do separate submissions for now? On Tue, Jun 2, 2015 at 3:35 PM, Iuliu Rus wrote: > Raymond, given this, project we are going to wait for QEMU to accept the > virtualization extensions first ( > http://git.infradead.org/users/kbusch/qemu-nvme.git/commit/162ec6479447abbd76308f831f36703aa3acc485), > then submit our payload which includes what we previously discussed and the > windows implementation for the nvme doorbell extensions. > > On Fri, May 15, 2015 at 11:38 AM, Robles, Raymond C < > raymond.c.robles at intel.com> wrote: > >> Hi Iuliu, >> >> >> >> The process for submitting a patch is quite simple (by design). The steps >> are as follows: >> >> >> >> 1. Download the current latest trunk source via an SVN client >> (i.e. TortoiseSVN). I assume you’ve already done this. >> >> 2. In the Releases folder, under a specific release, there are >> readme.txt files that explain the build environment needed. Insure you can >> compile the trunk with the recommended tool/build set. >> >> 3. Port your changes over to the trunk. >> >> 4. Unit test your changes on the trunk source. >> >> 5. Run the required testing outlined in the file attached in this >> email and insure all tests are passing. >> >> 6. Send an email with the trunk source zipped (with your ported >> changes) to this email distribution list. >> >> 7. The three reviewing companies will review your changes and also >> run the tests in the file. >> >> 8. If all looks good, then the source maintainer will merge your >> changes to the trunk, and create a tag. >> >> 9. If there are any question/comments that require a modifications >> to you patch, then go back to step 4 after you’ve made any changes and >> complete the process again. >> >> >> >> Let me know if you have any further questions. Feel free to add both >> patches into a single submission. >> >> >> >> You should be able to download all tools necessary outlined in the file. >> Let us know if you need assistance. >> >> >> >> Thanks, >> >> Ray >> >> >> >> *From:* Iuliu Rus [mailto:iuliur at google.com] >> *Sent:* Thursday, May 14, 2015 11:02 PM >> >> *To:* Robles, Raymond C >> *Cc:* nvmewin at lists.openfabrics.org >> *Subject:* Re: [nvmewin] OFA Release Plan: Release #1 2015 -- Call for >> patch volunteers >> >> >> >> Can you please point me to the instructions on how to send the patch? >> >> As for the free q list issue, we enabled driver verifier and under stress >> it bugchecked in RtlpCheckListEntry. >> >> >> >> On Thu, May 14, 2015 at 10:52 PM, Robles, Raymond C < >> raymond.c.robles at intel.com> wrote: >> >> Hello, >> >> >> >> We always welcome companies to submit patches for candidate features. No >> other company has volunteered for concurrent channels at this time. We >> would gladly welcome your patch at your earliest convenience. The first >> release for 2015 is planned for ~August/September... would you like to >> volunteer for this patch prior to this release? >> >> >> >> As for a race condition synchronization fix for FreeQList, could you >> please describe the issue you were seeing and how you addressed it? >> >> >> >> Thanks, >> >> Ray >> >> >> >> *From:* Iuliu Rus [mailto:iuliur at google.com] >> *Sent:* Thursday, May 14, 2015 7:43 PM >> *To:* Robles, Raymond C >> *Cc:* nvmewin at lists.openfabrics.org >> *Subject:* Re: [nvmewin] OFA Release Plan: Release #1 2015 -- Call for >> patch volunteers >> >> >> >> Hello, >> >> We are the google compute engine team and we already have Concurrent >> channels working. We're still figuring out how we can integrate between our >> internal repo and svn. >> >> Do you still need this feature? >> >> We also have a synchronization fix for a race that corrupted FreeQList. >> >> >> >> On Thu, May 7, 2015 at 3:46 PM, Robles, Raymond C < >> raymond.c.robles at intel.com> wrote: >> >> All, >> >> >> >> I just wanted to send out a quick update on the release plans for 2015. >> We’ve had two more companies volunteer to submit patches this year: Samsung >> (Suman) and HGST (Tom). Thank you to both companies for volunteering! >> >> >> >> Here is the updated release plan for 2015. Unfortunately, some of the >> patch submissions will be a little later than originally planned. >> Therefore, I would like to ask the question: *is everyone ok if we push >> the first release out until end of August to allow for more patches to be >> submitted?* You can response publicly or just to me. >> >> >> >> Other alternatives would include just having smaller releases at >> different cadences. I’m open to any ideas since there are still many >> requests for features that have claimed for patch submission. >> >> >> >> >> >> *2015 Release #1 (Q2/Q3 – 2015)* >> >> >> >> *Feature* >> >> *Description* >> >> *Submitter* >> >> *Date* >> >> Concurrent Channels >> >> Start I/O Concurrent Channels allows StartIo to be called in parallel >> resulting in improved IOPS. >> >> >> >> >> >> Perf Opts >> >> Instead of learning cores, use perf opts. Shave off 2 seconds in init >> code… using Storport initialize perf opts. No need for source core >> interrupt steering code in driver. >> >> Samsung (Suman) >> >> End of June >> >> Win 8.1 Timers >> >> Storport Notification usage. Hot plug and IOCTLs, this does not work, for >> Win8.1… StorportRequestTimer() function needed. >> >> Samsung (Suman) >> >> End of July >> >> CFS Bit Monitoring >> >> CFS bit monitoring and handling. Look into adding additional handling >> code just to monitor CSTS.CFS. >> >> >> >> >> >> EOL - Read Only >> >> Read only support for devices at EOL. Detection at init, hot plug, or >> run-time. >> >> Samsung (Suman) >> >> End of August >> >> Namespace Mgmt >> >> Namespace management - creation and deletion of namespaces. >> >> Intel (Carolyn) >> >> End of June >> >> WHQL >> >> WHQL test suite run (any bug fixes). >> >> >> >> >> >> >> >> >> >> *2015 Release #2 (Q4 – 2015)* >> >> >> >> *Feature* >> >> *Description* >> >> *Submitter* >> >> *Date* >> >> Multi-Path >> >> Multipath feature set from 1.1 spec (Multi-path I/O, Namespace Sharing, >> Reservations) >> >> HGST (Tom) >> >> Mid-November >> >> RTD3 >> >> Active/idle RTD3 >> >> >> >> >> >> Temp Thresholds >> >> Temperature thresholds (Get/Set Feature and AER trigger) >> >> >> >> >> >> Live FW Update >> >> Live firmware update >> >> >> >> >> >> Atomicity >> >> Atomicity Enhancements >> >> >> >> >> >> Win 10 >> >> Win 10 support (push to Q4 release) >> >> >> >> >> >> SNTL >> >> SNTL 1.5 doc updates >> >> HGST (Tom) >> >> End of August >> >> WHQL >> >> WHQL test suite run (any bug fixes) - for each release of OFA >> >> >> >> >> >> >> >> Thanks, >> >> Ray >> >> >> >> *From:* nvmewin-bounces at lists.openfabrics.org [mailto: >> nvmewin-bounces at lists.openfabrics.org] *On Behalf Of *Robles, Raymond C >> *Sent:* Thursday, April 09, 2015 5:29 PM >> *To:* nvmewin at lists.openfabrics.org >> *Subject:* [nvmewin] OFA Release Plan: Release #1 2015 -- Call for patch >> volunteers >> >> >> >> All, >> >> >> >> As we enter Q2, I’m happy to announce that we a have a very good feature >> plan in place for both Q2 and Q4 releases in 2015 of the OFA driver. The >> list of requested features is included below for final review. >> >> >> >> I would like to formally ask for volunteers for patch submissions. As of >> today, there are no volunteers for any of the below patches. >> >> >> >> Intel would like to start with the volunteering… we will volunteer to >> provide the namespace management patch. >> >> >> >> *2015 Release #1 (Q2/Q3 – 2015)* >> >> >> >> *Feature* >> >> *Description* >> >> *Submitter* >> >> *Date* >> >> Concurrent Channels >> >> Start I/O Concurrent Channels allows StartIo to be called in parallel >> resulting in improved IOPS. >> >> >> >> >> >> Perf Opts >> >> Instead of learning cores, use perf opts. Shave off 2 seconds in init >> code… using Storport initialize perf opts. No need for source core >> interrupt steering code in driver. >> >> Samsung (Suman) >> >> End of June >> >> Win 8.1 Timers >> >> Storport Notification usage. Hot plug and IOCTLs, this does not work, for >> Win8.1… StorportRequestTimer() function needed. >> >> Samsung (Suman) >> >> End of July >> >> CFS Bit Monitoring >> >> CFS bit monitoring and handling. Look into adding additional handling >> code just to monitor CSTS.CFS. >> >> >> >> >> >> EOL - Read Only >> >> Read only support for devices at EOL. Detection at init, hot plug, or >> run-time. >> >> Samsung (Suman) >> >> End of August >> >> Namespace Mgmt >> >> Namespace management - creation and deletion of namespaces. >> >> Intel (Carolyn) >> >> End of June >> >> WHQL >> >> WHQL test suite run (any bug fixes). >> >> >> >> >> >> >> >> >> >> *2015 Release #2 (Q4 – 2015)* >> >> >> >> *Feature* >> >> *Description* >> >> *Submitter* >> >> *Date* >> >> Multi-Path >> >> Multipath feature set from 1.1 spec (Multi-path I/O, Namespace Sharing, >> Reservations) >> >> HGST (Tom) >> >> Mid-November >> >> RTD3 >> >> Active/idle RTD3 >> >> >> >> >> >> Temp Thresholds >> >> Temperature thresholds (Get/Set Feature and AER trigger) >> >> >> >> >> >> Live FW Update >> >> Live firmware update >> >> >> >> >> >> Atomicity >> >> Atomicity Enhancements >> >> >> >> >> >> Win 10 >> >> Win 10 support (push to Q4 release) >> >> >> >> >> >> SNTL >> >> SNTL 1.5 doc updates >> >> HGST (Tom) >> >> End of August >> >> WHQL >> >> WHQL test suite run (any bug fixes) - for each release of OFA >> >> >> >> >> >> >> >> >> >> Thanks, >> >> Ray >> >> >> >> *Raymond C. Robles* >> >> NSG ISE Host Storage Software >> >> Intel Corporation >> >> Office: 480-554-2600 >> >> Mobile: 480-399-0645 >> >> >> >> >> _______________________________________________ >> nvmewin mailing list >> nvmewin at lists.openfabrics.org >> http://lists.openfabrics.org/mailman/listinfo/nvmewin >> >> >> >> >> > > -------------- next part -------------- An HTML attachment was scrubbed... URL: