From raymond.c.robles at intel.com Fri Sep 9 10:47:27 2016 From: raymond.c.robles at intel.com (Robles, Raymond C) Date: Fri, 9 Sep 2016 17:47:27 +0000 Subject: [nvmewin] OFA reference driver release assistance Message-ID: <49158E750348AA499168FD41D88983607C63713C@fmsmsx117.amr.corp.intel.com> Hello, In order to officially release the OFA reference driver, there are some tests that need to be run. I'm asking for volunteers for run the following tests: * WHQL/WHCK Certification Tests (no submission to MS required... just need test results) * Normal Unit Testing Test Suite The only requirements for devices on which to run the above tests is that the NVMe SSD be 1.2 compliant. As additional info, here is a list of features that we've added since our last release. Please review and sanity check my list. Recent patches include * Performance improvement & robustness * NVMe Spec rev 1.2 feature compliant * Supports MS Windows 10, 8.1, 7, Server 2012 R2, 2012 and 2008 R2 * Supports both 32 & 64-bit * What is new since last release: o Namespace Management (Create, Delete, Attach, Detach) o EOL Read Only Support o Win 8.1 Timers o Surprise Removal Support in IOCTL Path o Disk Initialization Performance Optimization o Storage Request Block Support o StorPort Performance Options o StorPort DPC Redirection o Misc. Bug Fixes o Security Send/Receive with Zero Data Length o SNTI updates for SCSI to NVMe Translation Thanks... Ray Raymond C. Robles NSG ISE Host Storage Software Intel Corporation Office: 480-554-2600 Mobile: 480-399-0645 raymond.c.robles at intel.com [cid:image001.gif at 01CB9B29.EA8D14F0] -------------- next part -------------- An HTML attachment was scrubbed... URL: -------------- next part -------------- A non-text attachment was scrubbed... Name: image003.jpg Type: image/jpeg Size: 1568 bytes Desc: image003.jpg URL: From thomas.freeman at hgst.com Mon Sep 12 09:49:31 2016 From: thomas.freeman at hgst.com (Tom Freeman) Date: Mon, 12 Sep 2016 16:49:31 +0000 Subject: [nvmewin] OFA reference driver release assistance In-Reply-To: <49158E750348AA499168FD41D88983607C63713C@fmsmsx117.amr.corp.intel.com> References: <49158E750348AA499168FD41D88983607C63713C@fmsmsx117.amr.corp.intel.com> Message-ID: I can run the Unit testing test for 64-bit on 2008R2, 2012 and 2012R2 These are the unit tests I'll run: - 1 hour of data integrity testing using sdstress (Microsoft Tool) - 1 hour of heavy stress testing using IOMETER covering, at least, 512B, 4KB and 128KB ranging from 1 OIO to 64 OIO both sequential and random - Quick and slow format of both MBR and GPT partitioning - Microsoft SCSI Compliance, no failures except (warnings OK) Tom Freeman Software Engineer, Device Manager and Driver Development Western Digital Corporation e. Thomas.freeman at hgst.com o. +1-507-322-2311 [cid:image003.jpg at 01D20CEB.B7D3E1D0] From: nvmewin [mailto:nvmewin-bounces at lists.openfabrics.org] On Behalf Of Robles, Raymond C Sent: Friday, September 9, 2016 12:47 PM To: nvmewin Cc: Huffman, Amber Subject: [nvmewin] OFA reference driver release assistance Hello, In order to officially release the OFA reference driver, there are some tests that need to be run. I'm asking for volunteers for run the following tests: * WHQL/WHCK Certification Tests (no submission to MS required... just need test results) * Normal Unit Testing Test Suite The only requirements for devices on which to run the above tests is that the NVMe SSD be 1.2 compliant. As additional info, here is a list of features that we've added since our last release. Please review and sanity check my list. Recent patches include * Performance improvement & robustness * NVMe Spec rev 1.2 feature compliant * Supports MS Windows 10, 8.1, 7, Server 2012 R2, 2012 and 2008 R2 * Supports both 32 & 64-bit * What is new since last release: o Namespace Management (Create, Delete, Attach, Detach) o EOL Read Only Support o Win 8.1 Timers o Surprise Removal Support in IOCTL Path o Disk Initialization Performance Optimization o Storage Request Block Support o StorPort Performance Options o StorPort DPC Redirection o Misc. Bug Fixes o Security Send/Receive with Zero Data Length o SNTI updates for SCSI to NVMe Translation Thanks... Ray Raymond C. Robles NSG ISE Host Storage Software Intel Corporation Office: 480-554-2600 Mobile: 480-399-0645 raymond.c.robles at intel.com [cid:image001.gif at 01CB9B29.EA8D14F0] Western Digital Corporation (and its subsidiaries) E-mail Confidentiality Notice & Disclaimer: This e-mail and any files transmitted with it may contain confidential or legally privileged information of WDC and/or its affiliates, and are intended solely for the use of the individual or entity to which they are addressed. If you are not the intended recipient, any disclosure, copying, distribution or any action taken or omitted to be taken in reliance on it, is prohibited. If you have received this e-mail in error, please notify the sender immediately and delete the e-mail in its entirety from your system. -------------- next part -------------- An HTML attachment was scrubbed... URL: -------------- next part -------------- A non-text attachment was scrubbed... Name: image002.jpg Type: image/jpeg Size: 1568 bytes Desc: image002.jpg URL: -------------- next part -------------- A non-text attachment was scrubbed... Name: image003.jpg Type: image/jpeg Size: 2937 bytes Desc: image003.jpg URL: From raymond.c.robles at intel.com Mon Sep 12 10:07:17 2016 From: raymond.c.robles at intel.com (Robles, Raymond C) Date: Mon, 12 Sep 2016 17:07:17 +0000 Subject: [nvmewin] OFA reference driver release assistance In-Reply-To: References: <49158E750348AA499168FD41D88983607C63713C@fmsmsx117.amr.corp.intel.com> Message-ID: <49158E750348AA499168FD41D88983607C6388A9@fmsmsx117.amr.corp.intel.com> Thanks Tom! From: Tom Freeman [mailto:thomas.freeman at hgst.com] Sent: Monday, September 12, 2016 9:50 AM To: Robles, Raymond C ; nvmewin Cc: Huffman, Amber Subject: RE: OFA reference driver release assistance I can run the Unit testing test for 64-bit on 2008R2, 2012 and 2012R2 These are the unit tests I'll run: - 1 hour of data integrity testing using sdstress (Microsoft Tool) - 1 hour of heavy stress testing using IOMETER covering, at least, 512B, 4KB and 128KB ranging from 1 OIO to 64 OIO both sequential and random - Quick and slow format of both MBR and GPT partitioning - Microsoft SCSI Compliance, no failures except (warnings OK) Tom Freeman Software Engineer, Device Manager and Driver Development Western Digital Corporation e. Thomas.freeman at hgst.com o. +1-507-322-2311 [cid:image001.jpg at 01D20CDD.6F370C30] From: nvmewin [mailto:nvmewin-bounces at lists.openfabrics.org] On Behalf Of Robles, Raymond C Sent: Friday, September 9, 2016 12:47 PM To: nvmewin > Cc: Huffman, Amber > Subject: [nvmewin] OFA reference driver release assistance Hello, In order to officially release the OFA reference driver, there are some tests that need to be run. I'm asking for volunteers for run the following tests: * WHQL/WHCK Certification Tests (no submission to MS required... just need test results) * Normal Unit Testing Test Suite The only requirements for devices on which to run the above tests is that the NVMe SSD be 1.2 compliant. As additional info, here is a list of features that we've added since our last release. Please review and sanity check my list. Recent patches include * Performance improvement & robustness * NVMe Spec rev 1.2 feature compliant * Supports MS Windows 10, 8.1, 7, Server 2012 R2, 2012 and 2008 R2 * Supports both 32 & 64-bit * What is new since last release: o Namespace Management (Create, Delete, Attach, Detach) o EOL Read Only Support o Win 8.1 Timers o Surprise Removal Support in IOCTL Path o Disk Initialization Performance Optimization o Storage Request Block Support o StorPort Performance Options o StorPort DPC Redirection o Misc. Bug Fixes o Security Send/Receive with Zero Data Length o SNTI updates for SCSI to NVMe Translation Thanks... Ray Raymond C. Robles NSG ISE Host Storage Software Intel Corporation Office: 480-554-2600 Mobile: 480-399-0645 raymond.c.robles at intel.com [cid:image001.gif at 01CB9B29.EA8D14F0] Western Digital Corporation (and its subsidiaries) E-mail Confidentiality Notice & Disclaimer: This e-mail and any files transmitted with it may contain confidential or legally privileged information of WDC and/or its affiliates, and are intended solely for the use of the individual or entity to which they are addressed. If you are not the intended recipient, any disclosure, copying, distribution or any action taken or omitted to be taken in reliance on it, is prohibited. If you have received this e-mail in error, please notify the sender immediately and delete the e-mail in its entirety from your system. -------------- next part -------------- An HTML attachment was scrubbed... URL: -------------- next part -------------- A non-text attachment was scrubbed... Name: image001.jpg Type: image/jpeg Size: 2937 bytes Desc: image001.jpg URL: -------------- next part -------------- A non-text attachment was scrubbed... Name: image002.jpg Type: image/jpeg Size: 1568 bytes Desc: image002.jpg URL: From raymond.c.robles at intel.com Tue Sep 20 10:59:05 2016 From: raymond.c.robles at intel.com (Robles, Raymond C) Date: Tue, 20 Sep 2016 17:59:05 +0000 Subject: [nvmewin] OFA NVMe Windows Driver Source Code Access Message-ID: <49158E750348AA499168FD41D88983607C64EB01@fmsmsx117.amr.corp.intel.com> Hello, I've gotten a few offline requests for getting access to the OFA reference driver to help with the unit testing for our upcoming release. Thanks to those that are offering to help... it is greatly appreciated. Anyone can access the OFA reference driver code. Please see instructions on the website. Below is some helpful information... * First, if you have not already, please subscribe to the OFA reference nvmewin mailing list. You can do this at the following link: o http://lists.openfabrics.org/mailman/listinfo/nvmewin * For general information on the OFA nvmewin group, please follow this link: o https://www.openfabrics.org/index.php/working-groups.html o https://www.openfabrics.org/index.php/working-groups-overview.html#nvmewin * To gain access to the OFA source code, you can use a client like Tortoise SVN and either use the https protocol address or SVN protocol address o https://www.openfabrics.org/svnrepo/nvmewin o svn://svn.openfabrics.org/nvmewin Please let me know if there are any question. If you are offering to help with release testing, please email this distribution and state which set of required tests you will be performing. Thanks! Ray Raymond C. Robles NSG ISE Host Storage Software Intel Corporation Office: 480-554-2600 Mobile: 480-399-0645 raymond.c.robles at intel.com [cid:image001.gif at 01CB9B29.EA8D14F0] -------------- next part -------------- An HTML attachment was scrubbed... URL: -------------- next part -------------- A non-text attachment was scrubbed... Name: image001.jpg Type: image/jpeg Size: 1568 bytes Desc: image001.jpg URL: From thomas.freeman at hgst.com Tue Sep 20 11:59:23 2016 From: thomas.freeman at hgst.com (Tom Freeman) Date: Tue, 20 Sep 2016 18:59:23 +0000 Subject: [nvmewin] OFA reference driver release assistance References: <49158E750348AA499168FD41D88983607C63713C@fmsmsx117.amr.corp.intel.com> Message-ID: I've completed my testing for the 1.5 release All the testing was successful. There was a single failure in the Scsi Compliance (details at the end of the note). Filter 10700 overrides this on Server 2016 HLK testing. I was not able to verify that filter on HCK on 2008R2, 2012 and 2012R2. Testing details: 64-bit on 2008R2, 2012 and 2012R2 1 hour of data integrity testing using sdstress 1 hour of heavy stress testing using IOMETER covering, at least, 512B, 4KB and 128KB ranging from 1 OIO to 64 OIO both sequential and random Quick and slow format of both MBR and GPT partitioning Microsoft SCSI Compliance. Scsi Compliance Failure: Policy: REQUIRED Start: ASSERTION: INQUIRY Attempting Unit Serial Number Page 0x80., TUID= Description: Checking if Unit Serial Number Page 0x80 is supported Reference: SCSI Primary Commands - 4 (SPC-4) Revision 30 (or published) specification Section 7.8.15 http://www.t10.org/ftp/t10/drafts/spc4/spc4r30.pdf Expectation: Device returns meaningful data as defined in SCSI-NVME translation document Sending SCSI command to device. INQUIRY Command: 12 01 80 00 ff 00 DeviceIoControl SUCCEEDED. SCSI status: 0x00 (GOOD) Sense bytes: 0x00 Data bytes: 0x2c 00 04 08 0C --------------------------------------------------------------- 000 00800028 30303030 5F303030 305F3030 ...(0000_0000_00 010 30305F30 3030305F 30303043 5F434130 00_0000_000C_CA0 020 305F3630 31465F42 4230302E 00000000 0_601F_BB00..... == DONE ============================================================== INQUIRY: Unit serial number page is supported. "Peripheral Qualifier" = 0x0 "Peripheral Device Type" = 0x0 "Page Code" = 0x80 Error: 0x0, Error 0x00000000 Unit serial number page length shall be set to either 20 for EUI64 or 36 for NGUID. Actual=40 File=drivers\storage\tests\scsicompliance\lib\nvme\inquiry_nvme.cpp Line=120 Error: 0x0, Error 0x00000000 Logging an error File=drivers\storage\tests\scsicompliance\lib\core\policyscenario.cpp Line=213 End: Fail, ASSERTION: INQUIRY Attempting Unit Serial Number Page 0x80., TUID=, Repro=scsicompliance /device \\.\PhysicalDrive1 /verbosity 4 /scenario NVMe /Operation test /Version 2 Tom Freeman Software Engineer, Device Manager and Driver Development Western Digital Corporation e. Thomas.freeman at hgst.com o. +1-507-322-2311 [cid:image002.jpg at 01D21347.2F77C5B0] From: Tom Freeman Sent: Monday, September 12, 2016 11:49 AM To: 'Robles, Raymond C' ; nvmewin Cc: Huffman, Amber Subject: RE: OFA reference driver release assistance I can run the Unit testing test for 64-bit on 2008R2, 2012 and 2012R2 These are the unit tests I'll run: - 1 hour of data integrity testing using sdstress (Microsoft Tool) - 1 hour of heavy stress testing using IOMETER covering, at least, 512B, 4KB and 128KB ranging from 1 OIO to 64 OIO both sequential and random - Quick and slow format of both MBR and GPT partitioning - Microsoft SCSI Compliance, no failures except (warnings OK) Tom Freeman Software Engineer, Device Manager and Driver Development Western Digital Corporation e. Thomas.freeman at hgst.com o. +1-507-322-2311 [cid:image003.jpg at 01D21347.2F730AC0] From: nvmewin [mailto:nvmewin-bounces at lists.openfabrics.org] On Behalf Of Robles, Raymond C Sent: Friday, September 9, 2016 12:47 PM To: nvmewin > Cc: Huffman, Amber > Subject: [nvmewin] OFA reference driver release assistance Hello, In order to officially release the OFA reference driver, there are some tests that need to be run. I'm asking for volunteers for run the following tests: * WHQL/WHCK Certification Tests (no submission to MS required... just need test results) * Normal Unit Testing Test Suite The only requirements for devices on which to run the above tests is that the NVMe SSD be 1.2 compliant. As additional info, here is a list of features that we've added since our last release. Please review and sanity check my list. Recent patches include * Performance improvement & robustness * NVMe Spec rev 1.2 feature compliant * Supports MS Windows 10, 8.1, 7, Server 2012 R2, 2012 and 2008 R2 * Supports both 32 & 64-bit * What is new since last release: o Namespace Management (Create, Delete, Attach, Detach) o EOL Read Only Support o Win 8.1 Timers o Surprise Removal Support in IOCTL Path o Disk Initialization Performance Optimization o Storage Request Block Support o StorPort Performance Options o StorPort DPC Redirection o Misc. Bug Fixes o Security Send/Receive with Zero Data Length o SNTI updates for SCSI to NVMe Translation Thanks... Ray Raymond C. Robles NSG ISE Host Storage Software Intel Corporation Office: 480-554-2600 Mobile: 480-399-0645 raymond.c.robles at intel.com [cid:image001.gif at 01CB9B29.EA8D14F0] Western Digital Corporation (and its subsidiaries) E-mail Confidentiality Notice & Disclaimer: This e-mail and any files transmitted with it may contain confidential or legally privileged information of WDC and/or its affiliates, and are intended solely for the use of the individual or entity to which they are addressed. If you are not the intended recipient, any disclosure, copying, distribution or any action taken or omitted to be taken in reliance on it, is prohibited. If you have received this e-mail in error, please notify the sender immediately and delete the e-mail in its entirety from your system. -------------- next part -------------- An HTML attachment was scrubbed... URL: -------------- next part -------------- A non-text attachment was scrubbed... Name: image003.jpg Type: image/jpeg Size: 2937 bytes Desc: image003.jpg URL: -------------- next part -------------- A non-text attachment was scrubbed... Name: image004.jpg Type: image/jpeg Size: 1568 bytes Desc: image004.jpg URL: -------------- next part -------------- A non-text attachment was scrubbed... Name: image002.jpg Type: image/jpeg Size: 2934 bytes Desc: image002.jpg URL: From raymond.c.robles at intel.com Tue Sep 20 14:29:14 2016 From: raymond.c.robles at intel.com (Robles, Raymond C) Date: Tue, 20 Sep 2016 21:29:14 +0000 Subject: [nvmewin] OFA reference driver release assistance In-Reply-To: References: <49158E750348AA499168FD41D88983607C63713C@fmsmsx117.amr.corp.intel.com> Message-ID: <49158E750348AA499168FD41D88983607C64EF11@fmsmsx117.amr.corp.intel.com> Hi Tom, This is awesome. Thank you very much for completing this testing. I'm adding it to a test spreadsheet I'm maintaining. On the failure below, I'll log the 40 length expectation by the test. BTW, can you provide the MS SCSI compliance command line you used to run the test? As well as the version? Thanks again! Ray From: Tom Freeman [mailto:thomas.freeman at hgst.com] Sent: Tuesday, September 20, 2016 11:59 AM To: Robles, Raymond C ; nvmewin Cc: Huffman, Amber Subject: RE: OFA reference driver release assistance I've completed my testing for the 1.5 release All the testing was successful. There was a single failure in the Scsi Compliance (details at the end of the note). Filter 10700 overrides this on Server 2016 HLK testing. I was not able to verify that filter on HCK on 2008R2, 2012 and 2012R2. Testing details: 64-bit on 2008R2, 2012 and 2012R2 1 hour of data integrity testing using sdstress 1 hour of heavy stress testing using IOMETER covering, at least, 512B, 4KB and 128KB ranging from 1 OIO to 64 OIO both sequential and random Quick and slow format of both MBR and GPT partitioning Microsoft SCSI Compliance. Scsi Compliance Failure: Policy: REQUIRED Start: ASSERTION: INQUIRY Attempting Unit Serial Number Page 0x80., TUID= Description: Checking if Unit Serial Number Page 0x80 is supported Reference: SCSI Primary Commands - 4 (SPC-4) Revision 30 (or published) specification Section 7.8.15 http://www.t10.org/ftp/t10/drafts/spc4/spc4r30.pdf Expectation: Device returns meaningful data as defined in SCSI-NVME translation document Sending SCSI command to device. INQUIRY Command: 12 01 80 00 ff 00 DeviceIoControl SUCCEEDED. SCSI status: 0x00 (GOOD) Sense bytes: 0x00 Data bytes: 0x2c 00 04 08 0C --------------------------------------------------------------- 000 00800028 30303030 5F303030 305F3030 ...(0000_0000_00 010 30305F30 3030305F 30303043 5F434130 00_0000_000C_CA0 020 305F3630 31465F42 4230302E 00000000 0_601F_BB00..... == DONE ============================================================== INQUIRY: Unit serial number page is supported. "Peripheral Qualifier" = 0x0 "Peripheral Device Type" = 0x0 "Page Code" = 0x80 Error: 0x0, Error 0x00000000 Unit serial number page length shall be set to either 20 for EUI64 or 36 for NGUID. Actual=40 File=drivers\storage\tests\scsicompliance\lib\nvme\inquiry_nvme.cpp Line=120 Error: 0x0, Error 0x00000000 Logging an error File=drivers\storage\tests\scsicompliance\lib\core\policyscenario.cpp Line=213 End: Fail, ASSERTION: INQUIRY Attempting Unit Serial Number Page 0x80., TUID=, Repro=scsicompliance /device \\.\PhysicalDrive1 /verbosity 4 /scenario NVMe /Operation test /Version 2 Tom Freeman Software Engineer, Device Manager and Driver Development Western Digital Corporation e. Thomas.freeman at hgst.com o. +1-507-322-2311 [cid:image001.jpg at 01D2134B.5BA487F0] From: Tom Freeman Sent: Monday, September 12, 2016 11:49 AM To: 'Robles, Raymond C' >; nvmewin > Cc: Huffman, Amber > Subject: RE: OFA reference driver release assistance I can run the Unit testing test for 64-bit on 2008R2, 2012 and 2012R2 These are the unit tests I'll run: - 1 hour of data integrity testing using sdstress (Microsoft Tool) - 1 hour of heavy stress testing using IOMETER covering, at least, 512B, 4KB and 128KB ranging from 1 OIO to 64 OIO both sequential and random - Quick and slow format of both MBR and GPT partitioning - Microsoft SCSI Compliance, no failures except (warnings OK) Tom Freeman Software Engineer, Device Manager and Driver Development Western Digital Corporation e. Thomas.freeman at hgst.com o. +1-507-322-2311 [cid:image002.jpg at 01D2134B.5BA487F0] From: nvmewin [mailto:nvmewin-bounces at lists.openfabrics.org] On Behalf Of Robles, Raymond C Sent: Friday, September 9, 2016 12:47 PM To: nvmewin > Cc: Huffman, Amber > Subject: [nvmewin] OFA reference driver release assistance Hello, In order to officially release the OFA reference driver, there are some tests that need to be run. I'm asking for volunteers for run the following tests: * WHQL/WHCK Certification Tests (no submission to MS required... just need test results) * Normal Unit Testing Test Suite The only requirements for devices on which to run the above tests is that the NVMe SSD be 1.2 compliant. As additional info, here is a list of features that we've added since our last release. Please review and sanity check my list. Recent patches include * Performance improvement & robustness * NVMe Spec rev 1.2 feature compliant * Supports MS Windows 10, 8.1, 7, Server 2012 R2, 2012 and 2008 R2 * Supports both 32 & 64-bit * What is new since last release: o Namespace Management (Create, Delete, Attach, Detach) o EOL Read Only Support o Win 8.1 Timers o Surprise Removal Support in IOCTL Path o Disk Initialization Performance Optimization o Storage Request Block Support o StorPort Performance Options o StorPort DPC Redirection o Misc. Bug Fixes o Security Send/Receive with Zero Data Length o SNTI updates for SCSI to NVMe Translation Thanks... Ray Raymond C. Robles NSG ISE Host Storage Software Intel Corporation Office: 480-554-2600 Mobile: 480-399-0645 raymond.c.robles at intel.com [cid:image001.gif at 01CB9B29.EA8D14F0] Western Digital Corporation (and its subsidiaries) E-mail Confidentiality Notice & Disclaimer: This e-mail and any files transmitted with it may contain confidential or legally privileged information of WDC and/or its affiliates, and are intended solely for the use of the individual or entity to which they are addressed. If you are not the intended recipient, any disclosure, copying, distribution or any action taken or omitted to be taken in reliance on it, is prohibited. If you have received this e-mail in error, please notify the sender immediately and delete the e-mail in its entirety from your system. -------------- next part -------------- An HTML attachment was scrubbed... URL: -------------- next part -------------- A non-text attachment was scrubbed... Name: image001.jpg Type: image/jpeg Size: 2934 bytes Desc: image001.jpg URL: -------------- next part -------------- A non-text attachment was scrubbed... Name: image002.jpg Type: image/jpeg Size: 2937 bytes Desc: image002.jpg URL: -------------- next part -------------- A non-text attachment was scrubbed... Name: image003.jpg Type: image/jpeg Size: 1568 bytes Desc: image003.jpg URL: From l.subrahmanya at mobiveil.co.in Wed Sep 21 02:46:06 2016 From: l.subrahmanya at mobiveil.co.in (subrahmanya l) Date: Wed, 21 Sep 2016 15:16:06 +0530 Subject: [nvmewin] [PATCH] ADMIN_CREATE_IO_SUBMISSION_QUEUE bug fix Message-ID: All, As of today if the CAP.MQES = 0x1F, with windows driver code, NVMeCreateSubQueue function will set QSIZE to 0x3F, which is clearly violation of the NVME specification, This happened because of the following unconditional change of queue entry size in NVMeAllocQueues() ​ SysPageSizeInSubEntries = PAGE_SIZE / sizeof (NVMe_COMMAND); if ((QEntries % SysPageSizeInSubEntries) != 0) QEntries = (QEntries + SysPageSizeInSubEntries) & ~(SysPageSizeInSubEntries - 1); ​Attaching the patch for the revision 1.4 source code, please consider. -- Thanks, ~subbu -------------- next part -------------- An HTML attachment was scrubbed... URL: -------------- next part -------------- A non-text attachment was scrubbed... Name: nvme_QEntries_fix.patch Type: application/octet-stream Size: 3423 bytes Desc: not available URL: From Harlan.Chou at liteon.com Wed Sep 21 03:34:08 2016 From: Harlan.Chou at liteon.com (Harlan Chou) Date: Wed, 21 Sep 2016 10:34:08 +0000 Subject: [nvmewin] OFA NVMe Windows Driver Source Code Access In-Reply-To: <49158E750348AA499168FD41D88983607C64EB01@fmsmsx117.amr.corp.intel.com> References: <49158E750348AA499168FD41D88983607C64EB01@fmsmsx117.amr.corp.intel.com> Message-ID: <63697667A4DABE4ABF5C991A5E70FE462B44FF@MAL002.liteon.com> Hi Ray, It's great to receive your notice. We have finished to subscribe OFA reference mail list and access OFA source code. I will mail alpha-test plan to you after our internal test-plan audit meeting is finished. (About one week) We have very interest to support to do release test. Regards, Harlan Chou 14F Ruey Kuang Rd., Neihu District Tapei City 114, Taiwan, R.O.C. * 886-2-8798-2888#8811| f. 886-2-8798-2822 | * harlan.chou at liteon.com From: Robles, Raymond C [mailto:raymond.c.robles at intel.com] Sent: Wednesday, September 21, 2016 1:59 AM To: nvmewin Cc: Harlan Chou; 'Henrique.Maqueda at powerex.com' Subject: OFA NVMe Windows Driver Source Code Access Hello, I've gotten a few offline requests for getting access to the OFA reference driver to help with the unit testing for our upcoming release. Thanks to those that are offering to help... it is greatly appreciated. Anyone can access the OFA reference driver code. Please see instructions on the website. Below is some helpful information... * First, if you have not already, please subscribe to the OFA reference nvmewin mailing list. You can do this at the following link: o http://lists.openfabrics.org/mailman/listinfo/nvmewin * For general information on the OFA nvmewin group, please follow this link: o https://www.openfabrics.org/index.php/working-groups.html o https://www.openfabrics.org/index.php/working-groups-overview.html#nvmewin * To gain access to the OFA source code, you can use a client like Tortoise SVN and either use the https protocol address or SVN protocol address o https://www.openfabrics.org/svnrepo/nvmewin o svn://svn.openfabrics.org/nvmewin Please let me know if there are any question. If you are offering to help with release testing, please email this distribution and state which set of required tests you will be performing. Thanks! Ray Raymond C. Robles NSG ISE Host Storage Software Intel Corporation Office: 480-554-2600 Mobile: 480-399-0645 raymond.c.robles at intel.com [cid:image001.gif at 01CB9B29.EA8D14F0] Confidential Information:This message is sent to the intended recipient and may contain privileged or confidential information. If you received this transmission in error, please notify the sender with a replying e-mail and delete the message and any attachment.Transmission Caveat and Virus Alert: Internet communications cannot be guaranteed to be timely, secure, error or virus-free. The sender does not accept liability for any errors or omissions. -------------- next part -------------- An HTML attachment was scrubbed... URL: -------------- next part -------------- A non-text attachment was scrubbed... Name: image001.jpg Type: image/jpeg Size: 1568 bytes Desc: image001.jpg URL: From thomas.freeman at hgst.com Wed Sep 21 13:48:18 2016 From: thomas.freeman at hgst.com (Tom Freeman) Date: Wed, 21 Sep 2016 20:48:18 +0000 Subject: [nvmewin] OFA reference driver release assistance In-Reply-To: <49158E750348AA499168FD41D88983607C64EF11@fmsmsx117.amr.corp.intel.com> References: <49158E750348AA499168FD41D88983607C63713C@fmsmsx117.amr.corp.intel.com> <49158E750348AA499168FD41D88983607C64EF11@fmsmsx117.amr.corp.intel.com> Message-ID: In my previous test, I incorrectly used the HLK version of scsicompliance.exe. I reran the test with the HCK version of scsicompliance.exe (version 2.0.20100428) I'm attaching the files with the complete results - password "nvmeOFA15" For 2012 and 2012R2, the command was: scsicompliance /device \\.\physicaldrive1 /verbosity 4 /scenario nvme /operation test /version 2 For 2008R2 the command was: scsicompliance /device \\.\PhysicalDrive2 /verbosity 4 /scenario common /operation test /version 2 2008R2 passed 2012 and 2012r2 had a similar failure as reported with the HLK version of the test. Here are the details of that failure: Policy: REQUIRED Start: ASSERTION: INQUIRY Attempting Unit Serial Number Page 0x80., TUID= Description: Checking if Unit Serial Number Page 0x80 is supported Reference: SCSI Primary Commands - 4 (SPC-4) Revision 30 (or published) specification Section 7.8.15 http://www.t10.org/ftp/t10/drafts/spc4/spc4r30.pdf Expectation: Device returns meaningful data as defined in SCSI-NVME translation document Sending SCSI command to device. INQUIRY Command: 12 01 80 00 ff 00 DeviceIoControl SUCCEEDED. SCSI status: 0x00 (GOOD) Sense bytes: 0x00 Data bytes: 0x2c 00 04 08 0C --------------------------------------------------------------- 000 00800028 30303030 5F303030 305F3030 ...(0000_0000_00 010 30305F30 3030305F 30303043 5F434130 00_0000_000C_CA0 020 305F3630 31465F42 4230302E 00000000 0_601F_BB00..... == DONE ============================================================== INQUIRY: Unit serial number page is supported. "Peripheral Qualifier" = 0x0 "Peripheral Device Type" = 0x0 "Page Code" = 0x80 "Page Length" = 0x28 "Unit serial number page length shall be set to 20" Error: 0x0, Error 0x00000000 Logging an error File=testsrc\driverstest\storage\wdk\scsicompliance\lib\core\policyscenario.cpp Line=213 End: Fail, ASSERTION: INQUIRY Attempting Unit Serial Number Page 0x80., TUID=, Repro=scsicompliance /device \\.\physicaldrive1 /verbosity 4 /scenario nvme /operation test /version 2 Tom Freeman Software Engineer, Device Manager and Driver Development Western Digital Corporation e. Thomas.freeman at hgst.com o. +1-507-322-2311 [cid:image001.jpg at 01D2141F.90AB68E0] From: Robles, Raymond C [mailto:raymond.c.robles at intel.com] Sent: Tuesday, September 20, 2016 4:29 PM To: Tom Freeman ; nvmewin Cc: Huffman, Amber Subject: RE: OFA reference driver release assistance Hi Tom, This is awesome. Thank you very much for completing this testing. I'm adding it to a test spreadsheet I'm maintaining. On the failure below, I'll log the 40 length expectation by the test. BTW, can you provide the MS SCSI compliance command line you used to run the test? As well as the version? Thanks again! Ray From: Tom Freeman [mailto:thomas.freeman at hgst.com] Sent: Tuesday, September 20, 2016 11:59 AM To: Robles, Raymond C >; nvmewin > Cc: Huffman, Amber > Subject: RE: OFA reference driver release assistance I've completed my testing for the 1.5 release All the testing was successful. There was a single failure in the Scsi Compliance (details at the end of the note). Filter 10700 overrides this on Server 2016 HLK testing. I was not able to verify that filter on HCK on 2008R2, 2012 and 2012R2. Testing details: 64-bit on 2008R2, 2012 and 2012R2 1 hour of data integrity testing using sdstress 1 hour of heavy stress testing using IOMETER covering, at least, 512B, 4KB and 128KB ranging from 1 OIO to 64 OIO both sequential and random Quick and slow format of both MBR and GPT partitioning Microsoft SCSI Compliance. Scsi Compliance Failure: Policy: REQUIRED Start: ASSERTION: INQUIRY Attempting Unit Serial Number Page 0x80., TUID= Description: Checking if Unit Serial Number Page 0x80 is supported Reference: SCSI Primary Commands - 4 (SPC-4) Revision 30 (or published) specification Section 7.8.15 http://www.t10.org/ftp/t10/drafts/spc4/spc4r30.pdf Expectation: Device returns meaningful data as defined in SCSI-NVME translation document Sending SCSI command to device. INQUIRY Command: 12 01 80 00 ff 00 DeviceIoControl SUCCEEDED. SCSI status: 0x00 (GOOD) Sense bytes: 0x00 Data bytes: 0x2c 00 04 08 0C --------------------------------------------------------------- 000 00800028 30303030 5F303030 305F3030 ...(0000_0000_00 010 30305F30 3030305F 30303043 5F434130 00_0000_000C_CA0 020 305F3630 31465F42 4230302E 00000000 0_601F_BB00..... == DONE ============================================================== INQUIRY: Unit serial number page is supported. "Peripheral Qualifier" = 0x0 "Peripheral Device Type" = 0x0 "Page Code" = 0x80 Error: 0x0, Error 0x00000000 Unit serial number page length shall be set to either 20 for EUI64 or 36 for NGUID. Actual=40 File=drivers\storage\tests\scsicompliance\lib\nvme\inquiry_nvme.cpp Line=120 Error: 0x0, Error 0x00000000 Logging an error File=drivers\storage\tests\scsicompliance\lib\core\policyscenario.cpp Line=213 End: Fail, ASSERTION: INQUIRY Attempting Unit Serial Number Page 0x80., TUID=, Repro=scsicompliance /device \\.\PhysicalDrive1 /verbosity 4 /scenario NVMe /Operation test /Version 2 Tom Freeman Software Engineer, Device Manager and Driver Development Western Digital Corporation e. Thomas.freeman at hgst.com o. +1-507-322-2311 [cid:image006.jpg at 01D2141E.ACAFEA30] From: Tom Freeman Sent: Monday, September 12, 2016 11:49 AM To: 'Robles, Raymond C' >; nvmewin > Cc: Huffman, Amber > Subject: RE: OFA reference driver release assistance I can run the Unit testing test for 64-bit on 2008R2, 2012 and 2012R2 These are the unit tests I'll run: - 1 hour of data integrity testing using sdstress (Microsoft Tool) - 1 hour of heavy stress testing using IOMETER covering, at least, 512B, 4KB and 128KB ranging from 1 OIO to 64 OIO both sequential and random - Quick and slow format of both MBR and GPT partitioning - Microsoft SCSI Compliance, no failures except (warnings OK) Tom Freeman Software Engineer, Device Manager and Driver Development Western Digital Corporation e. Thomas.freeman at hgst.com o. +1-507-322-2311 [cid:image007.jpg at 01D2141E.ACAFEA30] From: nvmewin [mailto:nvmewin-bounces at lists.openfabrics.org] On Behalf Of Robles, Raymond C Sent: Friday, September 9, 2016 12:47 PM To: nvmewin > Cc: Huffman, Amber > Subject: [nvmewin] OFA reference driver release assistance Hello, In order to officially release the OFA reference driver, there are some tests that need to be run. I'm asking for volunteers for run the following tests: * WHQL/WHCK Certification Tests (no submission to MS required... just need test results) * Normal Unit Testing Test Suite The only requirements for devices on which to run the above tests is that the NVMe SSD be 1.2 compliant. As additional info, here is a list of features that we've added since our last release. Please review and sanity check my list. Recent patches include * Performance improvement & robustness * NVMe Spec rev 1.2 feature compliant * Supports MS Windows 10, 8.1, 7, Server 2012 R2, 2012 and 2008 R2 * Supports both 32 & 64-bit * What is new since last release: o Namespace Management (Create, Delete, Attach, Detach) o EOL Read Only Support o Win 8.1 Timers o Surprise Removal Support in IOCTL Path o Disk Initialization Performance Optimization o Storage Request Block Support o StorPort Performance Options o StorPort DPC Redirection o Misc. Bug Fixes o Security Send/Receive with Zero Data Length o SNTI updates for SCSI to NVMe Translation Thanks... Ray Raymond C. Robles NSG ISE Host Storage Software Intel Corporation Office: 480-554-2600 Mobile: 480-399-0645 raymond.c.robles at intel.com [cid:image001.gif at 01CB9B29.EA8D14F0] Western Digital Corporation (and its subsidiaries) E-mail Confidentiality Notice & Disclaimer: This e-mail and any files transmitted with it may contain confidential or legally privileged information of WDC and/or its affiliates, and are intended solely for the use of the individual or entity to which they are addressed. If you are not the intended recipient, any disclosure, copying, distribution or any action taken or omitted to be taken in reliance on it, is prohibited. If you have received this e-mail in error, please notify the sender immediately and delete the e-mail in its entirety from your system. Western Digital Corporation (and its subsidiaries) E-mail Confidentiality Notice & Disclaimer: This e-mail and any files transmitted with it may contain confidential or legally privileged information of WDC and/or its affiliates, and are intended solely for the use of the individual or entity to which they are addressed. If you are not the intended recipient, any disclosure, copying, distribution or any action taken or omitted to be taken in reliance on it, is prohibited. If you have received this e-mail in error, please notify the sender immediately and delete the e-mail in its entirety from your system. -------------- next part -------------- An HTML attachment was scrubbed... URL: -------------- next part -------------- A non-text attachment was scrubbed... Name: image006.jpg Type: image/jpeg Size: 2934 bytes Desc: image006.jpg URL: -------------- next part -------------- A non-text attachment was scrubbed... Name: image007.jpg Type: image/jpeg Size: 2937 bytes Desc: image007.jpg URL: -------------- next part -------------- A non-text attachment was scrubbed... Name: image008.jpg Type: image/jpeg Size: 1568 bytes Desc: image008.jpg URL: -------------- next part -------------- A non-text attachment was scrubbed... Name: image001.jpg Type: image/jpeg Size: 2938 bytes Desc: image001.jpg URL: -------------- next part -------------- A non-text attachment was scrubbed... Name: ScsiResults.zip Type: application/x-zip-compressed Size: 47878 bytes Desc: ScsiResults.zip URL: From ie895110 at hotmail.com Sun Sep 18 11:16:56 2016 From: ie895110 at hotmail.com (chou cheng-yen) Date: Sun, 18 Sep 2016 18:16:56 +0000 Subject: [nvmewin] Suscribe to NVMEWIN Message-ID: Hi Nvmewin Im NVMe test-driver engineer and I have interest to be assistant to test OFA reference driver. Please agree to post MSG on nvmewin forum. BR. -------------- next part -------------- An HTML attachment was scrubbed... URL: From raymond.c.robles at intel.com Wed Sep 21 14:05:51 2016 From: raymond.c.robles at intel.com (Robles, Raymond C) Date: Wed, 21 Sep 2016 21:05:51 +0000 Subject: [nvmewin] Suscribe to NVMEWIN In-Reply-To: References: Message-ID: <49158E750348AA499168FD41D88983607C64FBCA@fmsmsx117.amr.corp.intel.com> Hello Chou, To subscribe to the nvmewin distribution mailing list, please follow the instructions below: * First, if you have not already, please subscribe to the OFA reference nvmewin mailing list. You can do this at the following link: o http://lists.openfabrics.org/mailman/listinfo/nvmewin * For general information on the OFA nvmewin group, please follow this link: o https://www.openfabrics.org/index.php/working-groups.html o https://www.openfabrics.org/index.php/working-groups-overview.html#nvmewin * To gain access to the OFA source code, you can use a client like Tortoise SVN and either use the https protocol address or SVN protocol address o https://www.openfabrics.org/svnrepo/nvmewin o svn://svn.openfabrics.org/nvmewin Thank you... Ray From: nvmewin [mailto:nvmewin-bounces at lists.openfabrics.org] On Behalf Of chou cheng-yen Sent: Sunday, September 18, 2016 11:17 AM To: nvmewin at lists.openfabrics.org Subject: [nvmewin] Suscribe to NVMEWIN Hi Nvmewin Im NVMe test-driver engineer and I have interest to be assistant to test OFA reference driver. Please agree to post MSG on nvmewin forum. BR. -------------- next part -------------- An HTML attachment was scrubbed... URL: From raymond.c.robles at intel.com Thu Sep 22 16:12:15 2016 From: raymond.c.robles at intel.com (Robles, Raymond C) Date: Thu, 22 Sep 2016 23:12:15 +0000 Subject: [nvmewin] OFA reference driver release assistance In-Reply-To: References: <49158E750348AA499168FD41D88983607C63713C@fmsmsx117.amr.corp.intel.com> <49158E750348AA499168FD41D88983607C64EF11@fmsmsx117.amr.corp.intel.com> Message-ID: <49158E750348AA499168FD41D88983607C6507D2@fmsmsx117.amr.corp.intel.com> Thanks Tom. Question... do you think you can push a fix for this failure? Thanks... Ray From: Tom Freeman [mailto:thomas.freeman at hgst.com] Sent: Wednesday, September 21, 2016 1:48 PM To: Robles, Raymond C ; nvmewin Subject: RE: OFA reference driver release assistance In my previous test, I incorrectly used the HLK version of scsicompliance.exe. I reran the test with the HCK version of scsicompliance.exe (version 2.0.20100428) I'm attaching the files with the complete results - password "nvmeOFA15" For 2012 and 2012R2, the command was: scsicompliance /device \\.\physicaldrive1 /verbosity 4 /scenario nvme /operation test /version 2 For 2008R2 the command was: scsicompliance /device \\.\PhysicalDrive2 /verbosity 4 /scenario common /operation test /version 2 2008R2 passed 2012 and 2012r2 had a similar failure as reported with the HLK version of the test. Here are the details of that failure: Policy: REQUIRED Start: ASSERTION: INQUIRY Attempting Unit Serial Number Page 0x80., TUID= Description: Checking if Unit Serial Number Page 0x80 is supported Reference: SCSI Primary Commands - 4 (SPC-4) Revision 30 (or published) specification Section 7.8.15 http://www.t10.org/ftp/t10/drafts/spc4/spc4r30.pdf Expectation: Device returns meaningful data as defined in SCSI-NVME translation document Sending SCSI command to device. INQUIRY Command: 12 01 80 00 ff 00 DeviceIoControl SUCCEEDED. SCSI status: 0x00 (GOOD) Sense bytes: 0x00 Data bytes: 0x2c 00 04 08 0C --------------------------------------------------------------- 000 00800028 30303030 5F303030 305F3030 ...(0000_0000_00 010 30305F30 3030305F 30303043 5F434130 00_0000_000C_CA0 020 305F3630 31465F42 4230302E 00000000 0_601F_BB00..... == DONE ============================================================== INQUIRY: Unit serial number page is supported. "Peripheral Qualifier" = 0x0 "Peripheral Device Type" = 0x0 "Page Code" = 0x80 "Page Length" = 0x28 "Unit serial number page length shall be set to 20" Error: 0x0, Error 0x00000000 Logging an error File=testsrc\driverstest\storage\wdk\scsicompliance\lib\core\policyscenario.cpp Line=213 End: Fail, ASSERTION: INQUIRY Attempting Unit Serial Number Page 0x80., TUID=, Repro=scsicompliance /device \\.\physicaldrive1 /verbosity 4 /scenario nvme /operation test /version 2 Tom Freeman Software Engineer, Device Manager and Driver Development Western Digital Corporation e. Thomas.freeman at hgst.com o. +1-507-322-2311 [cid:image001.jpg at 01D214EC.13F5D100] From: Robles, Raymond C [mailto:raymond.c.robles at intel.com] Sent: Tuesday, September 20, 2016 4:29 PM To: Tom Freeman >; nvmewin > Cc: Huffman, Amber > Subject: RE: OFA reference driver release assistance Hi Tom, This is awesome. Thank you very much for completing this testing. I'm adding it to a test spreadsheet I'm maintaining. On the failure below, I'll log the 40 length expectation by the test. BTW, can you provide the MS SCSI compliance command line you used to run the test? As well as the version? Thanks again! Ray From: Tom Freeman [mailto:thomas.freeman at hgst.com] Sent: Tuesday, September 20, 2016 11:59 AM To: Robles, Raymond C >; nvmewin > Cc: Huffman, Amber > Subject: RE: OFA reference driver release assistance I've completed my testing for the 1.5 release All the testing was successful. There was a single failure in the Scsi Compliance (details at the end of the note). Filter 10700 overrides this on Server 2016 HLK testing. I was not able to verify that filter on HCK on 2008R2, 2012 and 2012R2. Testing details: 64-bit on 2008R2, 2012 and 2012R2 1 hour of data integrity testing using sdstress 1 hour of heavy stress testing using IOMETER covering, at least, 512B, 4KB and 128KB ranging from 1 OIO to 64 OIO both sequential and random Quick and slow format of both MBR and GPT partitioning Microsoft SCSI Compliance. Scsi Compliance Failure: Policy: REQUIRED Start: ASSERTION: INQUIRY Attempting Unit Serial Number Page 0x80., TUID= Description: Checking if Unit Serial Number Page 0x80 is supported Reference: SCSI Primary Commands - 4 (SPC-4) Revision 30 (or published) specification Section 7.8.15 http://www.t10.org/ftp/t10/drafts/spc4/spc4r30.pdf Expectation: Device returns meaningful data as defined in SCSI-NVME translation document Sending SCSI command to device. INQUIRY Command: 12 01 80 00 ff 00 DeviceIoControl SUCCEEDED. SCSI status: 0x00 (GOOD) Sense bytes: 0x00 Data bytes: 0x2c 00 04 08 0C --------------------------------------------------------------- 000 00800028 30303030 5F303030 305F3030 ...(0000_0000_00 010 30305F30 3030305F 30303043 5F434130 00_0000_000C_CA0 020 305F3630 31465F42 4230302E 00000000 0_601F_BB00..... == DONE ============================================================== INQUIRY: Unit serial number page is supported. "Peripheral Qualifier" = 0x0 "Peripheral Device Type" = 0x0 "Page Code" = 0x80 Error: 0x0, Error 0x00000000 Unit serial number page length shall be set to either 20 for EUI64 or 36 for NGUID. Actual=40 File=drivers\storage\tests\scsicompliance\lib\nvme\inquiry_nvme.cpp Line=120 Error: 0x0, Error 0x00000000 Logging an error File=drivers\storage\tests\scsicompliance\lib\core\policyscenario.cpp Line=213 End: Fail, ASSERTION: INQUIRY Attempting Unit Serial Number Page 0x80., TUID=, Repro=scsicompliance /device \\.\PhysicalDrive1 /verbosity 4 /scenario NVMe /Operation test /Version 2 Tom Freeman Software Engineer, Device Manager and Driver Development Western Digital Corporation e. Thomas.freeman at hgst.com o. +1-507-322-2311 [cid:image006.jpg at 01D2141E.ACAFEA30] From: Tom Freeman Sent: Monday, September 12, 2016 11:49 AM To: 'Robles, Raymond C' >; nvmewin > Cc: Huffman, Amber > Subject: RE: OFA reference driver release assistance I can run the Unit testing test for 64-bit on 2008R2, 2012 and 2012R2 These are the unit tests I'll run: - 1 hour of data integrity testing using sdstress (Microsoft Tool) - 1 hour of heavy stress testing using IOMETER covering, at least, 512B, 4KB and 128KB ranging from 1 OIO to 64 OIO both sequential and random - Quick and slow format of both MBR and GPT partitioning - Microsoft SCSI Compliance, no failures except (warnings OK) Tom Freeman Software Engineer, Device Manager and Driver Development Western Digital Corporation e. Thomas.freeman at hgst.com o. +1-507-322-2311 [cid:image007.jpg at 01D2141E.ACAFEA30] From: nvmewin [mailto:nvmewin-bounces at lists.openfabrics.org] On Behalf Of Robles, Raymond C Sent: Friday, September 9, 2016 12:47 PM To: nvmewin > Cc: Huffman, Amber > Subject: [nvmewin] OFA reference driver release assistance Hello, In order to officially release the OFA reference driver, there are some tests that need to be run. I'm asking for volunteers for run the following tests: * WHQL/WHCK Certification Tests (no submission to MS required... just need test results) * Normal Unit Testing Test Suite The only requirements for devices on which to run the above tests is that the NVMe SSD be 1.2 compliant. As additional info, here is a list of features that we've added since our last release. Please review and sanity check my list. Recent patches include * Performance improvement & robustness * NVMe Spec rev 1.2 feature compliant * Supports MS Windows 10, 8.1, 7, Server 2012 R2, 2012 and 2008 R2 * Supports both 32 & 64-bit * What is new since last release: o Namespace Management (Create, Delete, Attach, Detach) o EOL Read Only Support o Win 8.1 Timers o Surprise Removal Support in IOCTL Path o Disk Initialization Performance Optimization o Storage Request Block Support o StorPort Performance Options o StorPort DPC Redirection o Misc. Bug Fixes o Security Send/Receive with Zero Data Length o SNTI updates for SCSI to NVMe Translation Thanks... Ray Raymond C. Robles NSG ISE Host Storage Software Intel Corporation Office: 480-554-2600 Mobile: 480-399-0645 raymond.c.robles at intel.com [cid:image001.gif at 01CB9B29.EA8D14F0] Western Digital Corporation (and its subsidiaries) E-mail Confidentiality Notice & Disclaimer: This e-mail and any files transmitted with it may contain confidential or legally privileged information of WDC and/or its affiliates, and are intended solely for the use of the individual or entity to which they are addressed. If you are not the intended recipient, any disclosure, copying, distribution or any action taken or omitted to be taken in reliance on it, is prohibited. If you have received this e-mail in error, please notify the sender immediately and delete the e-mail in its entirety from your system. Western Digital Corporation (and its subsidiaries) E-mail Confidentiality Notice & Disclaimer: This e-mail and any files transmitted with it may contain confidential or legally privileged information of WDC and/or its affiliates, and are intended solely for the use of the individual or entity to which they are addressed. If you are not the intended recipient, any disclosure, copying, distribution or any action taken or omitted to be taken in reliance on it, is prohibited. If you have received this e-mail in error, please notify the sender immediately and delete the e-mail in its entirety from your system. -------------- next part -------------- An HTML attachment was scrubbed... URL: -------------- next part -------------- A non-text attachment was scrubbed... Name: image001.jpg Type: image/jpeg Size: 2938 bytes Desc: image001.jpg URL: -------------- next part -------------- A non-text attachment was scrubbed... Name: image002.jpg Type: image/jpeg Size: 2934 bytes Desc: image002.jpg URL: -------------- next part -------------- A non-text attachment was scrubbed... Name: image003.jpg Type: image/jpeg Size: 2937 bytes Desc: image003.jpg URL: -------------- next part -------------- A non-text attachment was scrubbed... Name: image004.jpg Type: image/jpeg Size: 1568 bytes Desc: image004.jpg URL: From Harlan.Chou at liteon.com Fri Sep 23 03:15:35 2016 From: Harlan.Chou at liteon.com (Harlan Chou) Date: Fri, 23 Sep 2016 10:15:35 +0000 Subject: [nvmewin] OFA NVMe Windows Driver Source Code Access References: <49158E750348AA499168FD41D88983607C64EB01@fmsmsx117.amr.corp.intel.com> Message-ID: <63697667A4DABE4ABF5C991A5E70FE462B4C1C@MAL002.liteon.com> Hi Ray We will cover following test, especially for these modification (EOL Read Only Support, SorPort Performance Options, Compatibility Windows 10 32bit/64bit and Surprise Removal Support in IOCTL Path) l IO Test l Compatibility Test l Power Management Test l Performance Test Regards, Harlan Chou 14F Ruey Kuang Rd., Neihu District Tapei City 114, Taiwan, R.O.C. * 886-2-8798-2888#8811| f. 886-2-8798-2822 | * harlan.chou at liteon.com From: Harlan Chou Sent: Wednesday, September 21, 2016 6:34 PM To: 'Robles, Raymond C'; nvmewin Cc: 'Henrique.Maqueda at powerex.com' Subject: RE: OFA NVMe Windows Driver Source Code Access Hi Ray, It's great to receive your notice. We have finished to subscribe OFA reference mail list and access OFA source code. I will mail alpha-test plan to you after our internal test-plan audit meeting is finished. (About one week) We have very interest to support to do release test. Regards, Harlan Chou 14F Ruey Kuang Rd., Neihu District Tapei City 114, Taiwan, R.O.C. * 886-2-8798-2888#8811| f. 886-2-8798-2822 | * harlan.chou at liteon.com From: Robles, Raymond C [mailto:raymond.c.robles at intel.com] Sent: Wednesday, September 21, 2016 1:59 AM To: nvmewin Cc: Harlan Chou; 'Henrique.Maqueda at powerex.com' Subject: OFA NVMe Windows Driver Source Code Access Hello, I've gotten a few offline requests for getting access to the OFA reference driver to help with the unit testing for our upcoming release. Thanks to those that are offering to help... it is greatly appreciated. Anyone can access the OFA reference driver code. Please see instructions on the website. Below is some helpful information... * First, if you have not already, please subscribe to the OFA reference nvmewin mailing list. You can do this at the following link: o http://lists.openfabrics.org/mailman/listinfo/nvmewin * For general information on the OFA nvmewin group, please follow this link: o https://www.openfabrics.org/index.php/working-groups.html o https://www.openfabrics.org/index.php/working-groups-overview.html#nvmewin * To gain access to the OFA source code, you can use a client like Tortoise SVN and either use the https protocol address or SVN protocol address o https://www.openfabrics.org/svnrepo/nvmewin o svn://svn.openfabrics.org/nvmewin Please let me know if there are any question. If you are offering to help with release testing, please email this distribution and state which set of required tests you will be performing. Thanks! Ray Raymond C. Robles NSG ISE Host Storage Software Intel Corporation Office: 480-554-2600 Mobile: 480-399-0645 raymond.c.robles at intel.com [cid:image001.gif at 01CB9B29.EA8D14F0] Confidential Information:This message is sent to the intended recipient and may contain privileged or confidential information. If you received this transmission in error, please notify the sender with a replying e-mail and delete the message and any attachment.Transmission Caveat and Virus Alert: Internet communications cannot be guaranteed to be timely, secure, error or virus-free. The sender does not accept liability for any errors or omissions. -------------- next part -------------- An HTML attachment was scrubbed... URL: -------------- next part -------------- A non-text attachment was scrubbed... Name: image001.jpg Type: image/jpeg Size: 1568 bytes Desc: image001.jpg URL: From thomas.freeman at hgst.com Fri Sep 23 06:15:21 2016 From: thomas.freeman at hgst.com (Tom Freeman) Date: Fri, 23 Sep 2016 13:15:21 +0000 Subject: [nvmewin] OFA reference driver release assistance In-Reply-To: <49158E750348AA499168FD41D88983607C6507D2@fmsmsx117.amr.corp.intel.com> References: <49158E750348AA499168FD41D88983607C63713C@fmsmsx117.amr.corp.intel.com> <49158E750348AA499168FD41D88983607C64EF11@fmsmsx117.amr.corp.intel.com> <49158E750348AA499168FD41D88983607C6507D2@fmsmsx117.amr.corp.intel.com> Message-ID: Ray, I don't think this is an issue with the driver. I think this is an HCK issue. I've run the HLK and see there is a filter for this error - 10700. I haven't run the HCK to see if the same filter applies there. >From the Scsi Translation Spec (1.5), section 6.1.3.1.1 NGUID-based translation "PAGE LENGTH shall be set to 40 indicating the page length in bytes." Tom Freeman Software Engineer, Device Manager and Driver Development Western Digital Corporation e. Thomas.freeman at hgst.com o. +1-507-322-2311 [cid:image001.jpg at 01D21572.9E451BF0] From: Robles, Raymond C [mailto:raymond.c.robles at intel.com] Sent: Thursday, September 22, 2016 6:12 PM To: Tom Freeman ; nvmewin Subject: RE: OFA reference driver release assistance Thanks Tom. Question... do you think you can push a fix for this failure? Thanks... Ray From: Tom Freeman [mailto:thomas.freeman at hgst.com] Sent: Wednesday, September 21, 2016 1:48 PM To: Robles, Raymond C >; nvmewin > Subject: RE: OFA reference driver release assistance In my previous test, I incorrectly used the HLK version of scsicompliance.exe. I reran the test with the HCK version of scsicompliance.exe (version 2.0.20100428) I'm attaching the files with the complete results - password "nvmeOFA15" For 2012 and 2012R2, the command was: scsicompliance /device \\.\physicaldrive1 /verbosity 4 /scenario nvme /operation test /version 2 For 2008R2 the command was: scsicompliance /device \\.\PhysicalDrive2 /verbosity 4 /scenario common /operation test /version 2 2008R2 passed 2012 and 2012r2 had a similar failure as reported with the HLK version of the test. Here are the details of that failure: Policy: REQUIRED Start: ASSERTION: INQUIRY Attempting Unit Serial Number Page 0x80., TUID= Description: Checking if Unit Serial Number Page 0x80 is supported Reference: SCSI Primary Commands - 4 (SPC-4) Revision 30 (or published) specification Section 7.8.15 http://www.t10.org/ftp/t10/drafts/spc4/spc4r30.pdf Expectation: Device returns meaningful data as defined in SCSI-NVME translation document Sending SCSI command to device. INQUIRY Command: 12 01 80 00 ff 00 DeviceIoControl SUCCEEDED. SCSI status: 0x00 (GOOD) Sense bytes: 0x00 Data bytes: 0x2c 00 04 08 0C --------------------------------------------------------------- 000 00800028 30303030 5F303030 305F3030 ...(0000_0000_00 010 30305F30 3030305F 30303043 5F434130 00_0000_000C_CA0 020 305F3630 31465F42 4230302E 00000000 0_601F_BB00..... == DONE ============================================================== INQUIRY: Unit serial number page is supported. "Peripheral Qualifier" = 0x0 "Peripheral Device Type" = 0x0 "Page Code" = 0x80 "Page Length" = 0x28 "Unit serial number page length shall be set to 20" Error: 0x0, Error 0x00000000 Logging an error File=testsrc\driverstest\storage\wdk\scsicompliance\lib\core\policyscenario.cpp Line=213 End: Fail, ASSERTION: INQUIRY Attempting Unit Serial Number Page 0x80., TUID=, Repro=scsicompliance /device \\.\physicaldrive1 /verbosity 4 /scenario nvme /operation test /version 2 Tom Freeman Software Engineer, Device Manager and Driver Development Western Digital Corporation e. Thomas.freeman at hgst.com o. +1-507-322-2311 [cid:image006.jpg at 01D21572.9E3C9070] From: Robles, Raymond C [mailto:raymond.c.robles at intel.com] Sent: Tuesday, September 20, 2016 4:29 PM To: Tom Freeman >; nvmewin > Cc: Huffman, Amber > Subject: RE: OFA reference driver release assistance Hi Tom, This is awesome. Thank you very much for completing this testing. I'm adding it to a test spreadsheet I'm maintaining. On the failure below, I'll log the 40 length expectation by the test. BTW, can you provide the MS SCSI compliance command line you used to run the test? As well as the version? Thanks again! Ray From: Tom Freeman [mailto:thomas.freeman at hgst.com] Sent: Tuesday, September 20, 2016 11:59 AM To: Robles, Raymond C >; nvmewin > Cc: Huffman, Amber > Subject: RE: OFA reference driver release assistance I've completed my testing for the 1.5 release All the testing was successful. There was a single failure in the Scsi Compliance (details at the end of the note). Filter 10700 overrides this on Server 2016 HLK testing. I was not able to verify that filter on HCK on 2008R2, 2012 and 2012R2. Testing details: 64-bit on 2008R2, 2012 and 2012R2 1 hour of data integrity testing using sdstress 1 hour of heavy stress testing using IOMETER covering, at least, 512B, 4KB and 128KB ranging from 1 OIO to 64 OIO both sequential and random Quick and slow format of both MBR and GPT partitioning Microsoft SCSI Compliance. Scsi Compliance Failure: Policy: REQUIRED Start: ASSERTION: INQUIRY Attempting Unit Serial Number Page 0x80., TUID= Description: Checking if Unit Serial Number Page 0x80 is supported Reference: SCSI Primary Commands - 4 (SPC-4) Revision 30 (or published) specification Section 7.8.15 http://www.t10.org/ftp/t10/drafts/spc4/spc4r30.pdf Expectation: Device returns meaningful data as defined in SCSI-NVME translation document Sending SCSI command to device. INQUIRY Command: 12 01 80 00 ff 00 DeviceIoControl SUCCEEDED. SCSI status: 0x00 (GOOD) Sense bytes: 0x00 Data bytes: 0x2c 00 04 08 0C --------------------------------------------------------------- 000 00800028 30303030 5F303030 305F3030 ...(0000_0000_00 010 30305F30 3030305F 30303043 5F434130 00_0000_000C_CA0 020 305F3630 31465F42 4230302E 00000000 0_601F_BB00..... == DONE ============================================================== INQUIRY: Unit serial number page is supported. "Peripheral Qualifier" = 0x0 "Peripheral Device Type" = 0x0 "Page Code" = 0x80 Error: 0x0, Error 0x00000000 Unit serial number page length shall be set to either 20 for EUI64 or 36 for NGUID. Actual=40 File=drivers\storage\tests\scsicompliance\lib\nvme\inquiry_nvme.cpp Line=120 Error: 0x0, Error 0x00000000 Logging an error File=drivers\storage\tests\scsicompliance\lib\core\policyscenario.cpp Line=213 End: Fail, ASSERTION: INQUIRY Attempting Unit Serial Number Page 0x80., TUID=, Repro=scsicompliance /device \\.\PhysicalDrive1 /verbosity 4 /scenario NVMe /Operation test /Version 2 Tom Freeman Software Engineer, Device Manager and Driver Development Western Digital Corporation e. Thomas.freeman at hgst.com o. +1-507-322-2311 [cid:image006.jpg at 01D2141E.ACAFEA30] From: Tom Freeman Sent: Monday, September 12, 2016 11:49 AM To: 'Robles, Raymond C' >; nvmewin > Cc: Huffman, Amber > Subject: RE: OFA reference driver release assistance I can run the Unit testing test for 64-bit on 2008R2, 2012 and 2012R2 These are the unit tests I'll run: - 1 hour of data integrity testing using sdstress (Microsoft Tool) - 1 hour of heavy stress testing using IOMETER covering, at least, 512B, 4KB and 128KB ranging from 1 OIO to 64 OIO both sequential and random - Quick and slow format of both MBR and GPT partitioning - Microsoft SCSI Compliance, no failures except (warnings OK) Tom Freeman Software Engineer, Device Manager and Driver Development Western Digital Corporation e. Thomas.freeman at hgst.com o. +1-507-322-2311 [cid:image007.jpg at 01D2141E.ACAFEA30] From: nvmewin [mailto:nvmewin-bounces at lists.openfabrics.org] On Behalf Of Robles, Raymond C Sent: Friday, September 9, 2016 12:47 PM To: nvmewin > Cc: Huffman, Amber > Subject: [nvmewin] OFA reference driver release assistance Hello, In order to officially release the OFA reference driver, there are some tests that need to be run. I'm asking for volunteers for run the following tests: * WHQL/WHCK Certification Tests (no submission to MS required... just need test results) * Normal Unit Testing Test Suite The only requirements for devices on which to run the above tests is that the NVMe SSD be 1.2 compliant. As additional info, here is a list of features that we've added since our last release. Please review and sanity check my list. Recent patches include * Performance improvement & robustness * NVMe Spec rev 1.2 feature compliant * Supports MS Windows 10, 8.1, 7, Server 2012 R2, 2012 and 2008 R2 * Supports both 32 & 64-bit * What is new since last release: o Namespace Management (Create, Delete, Attach, Detach) o EOL Read Only Support o Win 8.1 Timers o Surprise Removal Support in IOCTL Path o Disk Initialization Performance Optimization o Storage Request Block Support o StorPort Performance Options o StorPort DPC Redirection o Misc. Bug Fixes o Security Send/Receive with Zero Data Length o SNTI updates for SCSI to NVMe Translation Thanks... Ray Raymond C. Robles NSG ISE Host Storage Software Intel Corporation Office: 480-554-2600 Mobile: 480-399-0645 raymond.c.robles at intel.com [cid:image001.gif at 01CB9B29.EA8D14F0] Western Digital Corporation (and its subsidiaries) E-mail Confidentiality Notice & Disclaimer: This e-mail and any files transmitted with it may contain confidential or legally privileged information of WDC and/or its affiliates, and are intended solely for the use of the individual or entity to which they are addressed. If you are not the intended recipient, any disclosure, copying, distribution or any action taken or omitted to be taken in reliance on it, is prohibited. If you have received this e-mail in error, please notify the sender immediately and delete the e-mail in its entirety from your system. Western Digital Corporation (and its subsidiaries) E-mail Confidentiality Notice & Disclaimer: This e-mail and any files transmitted with it may contain confidential or legally privileged information of WDC and/or its affiliates, and are intended solely for the use of the individual or entity to which they are addressed. If you are not the intended recipient, any disclosure, copying, distribution or any action taken or omitted to be taken in reliance on it, is prohibited. If you have received this e-mail in error, please notify the sender immediately and delete the e-mail in its entirety from your system. Western Digital Corporation (and its subsidiaries) E-mail Confidentiality Notice & Disclaimer: This e-mail and any files transmitted with it may contain confidential or legally privileged information of WDC and/or its affiliates, and are intended solely for the use of the individual or entity to which they are addressed. If you are not the intended recipient, any disclosure, copying, distribution or any action taken or omitted to be taken in reliance on it, is prohibited. If you have received this e-mail in error, please notify the sender immediately and delete the e-mail in its entirety from your system. -------------- next part -------------- An HTML attachment was scrubbed... URL: -------------- next part -------------- A non-text attachment was scrubbed... Name: image006.jpg Type: image/jpeg Size: 2938 bytes Desc: image006.jpg URL: -------------- next part -------------- A non-text attachment was scrubbed... Name: image007.jpg Type: image/jpeg Size: 2934 bytes Desc: image007.jpg URL: -------------- next part -------------- A non-text attachment was scrubbed... Name: image008.jpg Type: image/jpeg Size: 2937 bytes Desc: image008.jpg URL: -------------- next part -------------- A non-text attachment was scrubbed... Name: image009.jpg Type: image/jpeg Size: 1568 bytes Desc: image009.jpg URL: -------------- next part -------------- A non-text attachment was scrubbed... Name: image001.jpg Type: image/jpeg Size: 2934 bytes Desc: image001.jpg URL: From raymond.c.robles at intel.com Fri Sep 23 11:14:11 2016 From: raymond.c.robles at intel.com (Robles, Raymond C) Date: Fri, 23 Sep 2016 18:14:11 +0000 Subject: [nvmewin] OFA NVMe Windows Driver Source Code Access In-Reply-To: <53F7355564791145BAF24BA7C569C846953833@FPTPWXMBX1.bchydro.adroot.bchydro.bc.ca> References: <49158E750348AA499168FD41D88983607C64EB01@fmsmsx117.amr.corp.intel.com> <63697667A4DABE4ABF5C991A5E70FE462B4C1C@MAL002.liteon.com> <53F7355564791145BAF24BA7C569C846953833@FPTPWXMBX1.bchydro.adroot.bchydro.bc.ca> Message-ID: <49158E750348AA499168FD41D88983607C6510C8@fmsmsx117.amr.corp.intel.com> Hello, The OFA reference driver must be built from the source code (and VS project files provided). To compile for Win10, you just need to install the WDK 10.0, SDK 10.0, and Visutal Studio 2015 (you can download a free trial version price is an issue). The OFA reference community does not post binaries of unreleased code. You can find legacy binaries for the 1.2, 1.3, and 1.4 release. Once we release, I will post the binaries for the current 1.5 release. In the meantime, you will have to build the driver yourself in order to test the current trunk source. Thanks... Ray From: Maqueda, Henrique [mailto:Henrique.Maqueda at powerex.com] Sent: Friday, September 23, 2016 10:46 AM To: Harlan Chou Cc: nvmewin ; Robles, Raymond C Subject: RE: OFA NVMe Windows Driver Source Code Access Hi Harlan, Where did you find get the Win10 driver? Is there a compiled version that we can just install and run some tests? Thanks. From: Harlan Chou [mailto:Harlan.Chou at liteon.com] Sent: September-23-16 3:16 AM To: Robles, Raymond C; nvmewin Cc: Maqueda, Henrique Subject: RE: OFA NVMe Windows Driver Source Code Access Hi Ray We will cover following test, especially for these modification (EOL Read Only Support, SorPort Performance Options, Compatibility Windows 10 32bit/64bit and Surprise Removal Support in IOCTL Path) l IO Test l Compatibility Test l Power Management Test l Performance Test Regards, Harlan Chou 14F Ruey Kuang Rd., Neihu District Tapei City 114, Taiwan, R.O.C. * 886-2-8798-2888#8811| f. 886-2-8798-2822 | * harlan.chou at liteon.com From: Harlan Chou Sent: Wednesday, September 21, 2016 6:34 PM To: 'Robles, Raymond C'; nvmewin Cc: 'Henrique.Maqueda at powerex.com' Subject: RE: OFA NVMe Windows Driver Source Code Access Hi Ray, It's great to receive your notice. We have finished to subscribe OFA reference mail list and access OFA source code. I will mail alpha-test plan to you after our internal test-plan audit meeting is finished. (About one week) We have very interest to support to do release test. Regards, Harlan Chou 14F Ruey Kuang Rd., Neihu District Tapei City 114, Taiwan, R.O.C. * 886-2-8798-2888#8811| f. 886-2-8798-2822 | * harlan.chou at liteon.com From: Robles, Raymond C [mailto:raymond.c.robles at intel.com] Sent: Wednesday, September 21, 2016 1:59 AM To: nvmewin Cc: Harlan Chou; 'Henrique.Maqueda at powerex.com' Subject: OFA NVMe Windows Driver Source Code Access Hello, I've gotten a few offline requests for getting access to the OFA reference driver to help with the unit testing for our upcoming release. Thanks to those that are offering to help... it is greatly appreciated. Anyone can access the OFA reference driver code. Please see instructions on the website. Below is some helpful information... * First, if you have not already, please subscribe to the OFA reference nvmewin mailing list. You can do this at the following link: o http://lists.openfabrics.org/mailman/listinfo/nvmewin * For general information on the OFA nvmewin group, please follow this link: o https://www.openfabrics.org/index.php/working-groups.html o https://www.openfabrics.org/index.php/working-groups-overview.html#nvmewin * To gain access to the OFA source code, you can use a client like Tortoise SVN and either use the https protocol address or SVN protocol address o https://www.openfabrics.org/svnrepo/nvmewin o svn://svn.openfabrics.org/nvmewin Please let me know if there are any question. If you are offering to help with release testing, please email this distribution and state which set of required tests you will be performing. Thanks! Ray Raymond C. Robles NSG ISE Host Storage Software Intel Corporation Office: 480-554-2600 Mobile: 480-399-0645 raymond.c.robles at intel.com [cid:image001.gif at 01CB9B29.EA8D14F0] Confidential Information:This message is sent to the intended recipient and may contain privileged or confidential information. If you received this transmission in error, please notify the sender with a replying e-mail and delete the message and any attachment.Transmission Caveat and Virus Alert: Internet communications cannot be guaranteed to be timely, secure, error or virus-free. The sender does not accept liability for any errors or omissions. ________________________________ This email may contain information that is confidential. If you are not the intended recipient you may not distribute or copy this email or any attachments. If you have received this email in error, please notify the sender and delete this email and attachments from your system immediately. Thank you. -PowerexDisclaimerID5.2.8.1541 -------------- next part -------------- An HTML attachment was scrubbed... URL: -------------- next part -------------- A non-text attachment was scrubbed... Name: image001.jpg Type: image/jpeg Size: 1568 bytes Desc: image001.jpg URL: From raymond.c.robles at intel.com Mon Sep 26 13:20:54 2016 From: raymond.c.robles at intel.com (Robles, Raymond C) Date: Mon, 26 Sep 2016 20:20:54 +0000 Subject: [nvmewin] OFA NVMe Windows Driver Source Code Access In-Reply-To: <63697667A4DABE4ABF5C991A5E70FE462B4C1C@MAL002.liteon.com> References: <49158E750348AA499168FD41D88983607C64EB01@fmsmsx117.amr.corp.intel.com> <63697667A4DABE4ABF5C991A5E70FE462B4C1C@MAL002.liteon.com> Message-ID: <49158E750348AA499168FD41D88983607C65E9BD@fmsmsx117.amr.corp.intel.com> Hi Harlan, Thank you for volunteering. Any additional testing the community perform on the driver, the better! Thank you... Ray From: Harlan Chou [mailto:Harlan.Chou at liteon.com] Sent: Friday, September 23, 2016 3:16 AM To: Robles, Raymond C ; nvmewin Cc: 'Henrique.Maqueda at powerex.com' Subject: RE: OFA NVMe Windows Driver Source Code Access Hi Ray We will cover following test, especially for these modification (EOL Read Only Support, SorPort Performance Options, Compatibility Windows 10 32bit/64bit and Surprise Removal Support in IOCTL Path) l IO Test l Compatibility Test l Power Management Test l Performance Test Regards, Harlan Chou 14F Ruey Kuang Rd., Neihu District Tapei City 114, Taiwan, R.O.C. * 886-2-8798-2888#8811| f. 886-2-8798-2822 | * harlan.chou at liteon.com From: Harlan Chou Sent: Wednesday, September 21, 2016 6:34 PM To: 'Robles, Raymond C'; nvmewin Cc: 'Henrique.Maqueda at powerex.com' Subject: RE: OFA NVMe Windows Driver Source Code Access Hi Ray, It's great to receive your notice. We have finished to subscribe OFA reference mail list and access OFA source code. I will mail alpha-test plan to you after our internal test-plan audit meeting is finished. (About one week) We have very interest to support to do release test. Regards, Harlan Chou 14F Ruey Kuang Rd., Neihu District Tapei City 114, Taiwan, R.O.C. * 886-2-8798-2888#8811| f. 886-2-8798-2822 | * harlan.chou at liteon.com From: Robles, Raymond C [mailto:raymond.c.robles at intel.com] Sent: Wednesday, September 21, 2016 1:59 AM To: nvmewin Cc: Harlan Chou; 'Henrique.Maqueda at powerex.com' Subject: OFA NVMe Windows Driver Source Code Access Hello, I've gotten a few offline requests for getting access to the OFA reference driver to help with the unit testing for our upcoming release. Thanks to those that are offering to help... it is greatly appreciated. Anyone can access the OFA reference driver code. Please see instructions on the website. Below is some helpful information... * First, if you have not already, please subscribe to the OFA reference nvmewin mailing list. You can do this at the following link: o http://lists.openfabrics.org/mailman/listinfo/nvmewin * For general information on the OFA nvmewin group, please follow this link: o https://www.openfabrics.org/index.php/working-groups.html o https://www.openfabrics.org/index.php/working-groups-overview.html#nvmewin * To gain access to the OFA source code, you can use a client like Tortoise SVN and either use the https protocol address or SVN protocol address o https://www.openfabrics.org/svnrepo/nvmewin o svn://svn.openfabrics.org/nvmewin Please let me know if there are any question. If you are offering to help with release testing, please email this distribution and state which set of required tests you will be performing. Thanks! Ray Raymond C. Robles NSG ISE Host Storage Software Intel Corporation Office: 480-554-2600 Mobile: 480-399-0645 raymond.c.robles at intel.com [cid:image001.gif at 01CB9B29.EA8D14F0] Confidential Information:This message is sent to the intended recipient and may contain privileged or confidential information. If you received this transmission in error, please notify the sender with a replying e-mail and delete the message and any attachment.Transmission Caveat and Virus Alert: Internet communications cannot be guaranteed to be timely, secure, error or virus-free. The sender does not accept liability for any errors or omissions. -------------- next part -------------- An HTML attachment was scrubbed... URL: -------------- next part -------------- A non-text attachment was scrubbed... Name: image001.jpg Type: image/jpeg Size: 1568 bytes Desc: image001.jpg URL: From Henrique.Maqueda at powerex.com Fri Sep 23 10:45:39 2016 From: Henrique.Maqueda at powerex.com (Maqueda, Henrique) Date: Fri, 23 Sep 2016 17:45:39 +0000 Subject: [nvmewin] OFA NVMe Windows Driver Source Code Access In-Reply-To: <63697667A4DABE4ABF5C991A5E70FE462B4C1C@MAL002.liteon.com> References: <49158E750348AA499168FD41D88983607C64EB01@fmsmsx117.amr.corp.intel.com> <63697667A4DABE4ABF5C991A5E70FE462B4C1C@MAL002.liteon.com> Message-ID: <53F7355564791145BAF24BA7C569C846953833@FPTPWXMBX1.bchydro.adroot.bchydro.bc.ca> Hi Harlan, Where did you find get the Win10 driver? Is there a compiled version that we can just install and run some tests? Thanks. From: Harlan Chou [mailto:Harlan.Chou at liteon.com] Sent: September-23-16 3:16 AM To: Robles, Raymond C; nvmewin Cc: Maqueda, Henrique Subject: RE: OFA NVMe Windows Driver Source Code Access Hi Ray We will cover following test, especially for these modification (EOL Read Only Support, SorPort Performance Options, Compatibility Windows 10 32bit/64bit and Surprise Removal Support in IOCTL Path) l IO Test l Compatibility Test l Power Management Test l Performance Test Regards, Harlan Chou 14F Ruey Kuang Rd., Neihu District Tapei City 114, Taiwan, R.O.C. * 886-2-8798-2888#8811| f. 886-2-8798-2822 | * harlan.chou at liteon.com From: Harlan Chou Sent: Wednesday, September 21, 2016 6:34 PM To: 'Robles, Raymond C'; nvmewin Cc: 'Henrique.Maqueda at powerex.com' Subject: RE: OFA NVMe Windows Driver Source Code Access Hi Ray, It's great to receive your notice. We have finished to subscribe OFA reference mail list and access OFA source code. I will mail alpha-test plan to you after our internal test-plan audit meeting is finished. (About one week) We have very interest to support to do release test. Regards, Harlan Chou 14F Ruey Kuang Rd., Neihu District Tapei City 114, Taiwan, R.O.C. * 886-2-8798-2888#8811| f. 886-2-8798-2822 | * harlan.chou at liteon.com From: Robles, Raymond C [mailto:raymond.c.robles at intel.com] Sent: Wednesday, September 21, 2016 1:59 AM To: nvmewin Cc: Harlan Chou; 'Henrique.Maqueda at powerex.com' Subject: OFA NVMe Windows Driver Source Code Access Hello, I've gotten a few offline requests for getting access to the OFA reference driver to help with the unit testing for our upcoming release. Thanks to those that are offering to help... it is greatly appreciated. Anyone can access the OFA reference driver code. Please see instructions on the website. Below is some helpful information... * First, if you have not already, please subscribe to the OFA reference nvmewin mailing list. You can do this at the following link: o http://lists.openfabrics.org/mailman/listinfo/nvmewin * For general information on the OFA nvmewin group, please follow this link: o https://www.openfabrics.org/index.php/working-groups.html o https://www.openfabrics.org/index.php/working-groups-overview.html#nvmewin * To gain access to the OFA source code, you can use a client like Tortoise SVN and either use the https protocol address or SVN protocol address o https://www.openfabrics.org/svnrepo/nvmewin o svn://svn.openfabrics.org/nvmewin Please let me know if there are any question. If you are offering to help with release testing, please email this distribution and state which set of required tests you will be performing. Thanks! Ray Raymond C. Robles NSG ISE Host Storage Software Intel Corporation Office: 480-554-2600 Mobile: 480-399-0645 raymond.c.robles at intel.com [cid:image001.gif at 01CB9B29.EA8D14F0] Confidential Information:This message is sent to the intended recipient and may contain privileged or confidential information. If you received this transmission in error, please notify the sender with a replying e-mail and delete the message and any attachment.Transmission Caveat and Virus Alert: Internet communications cannot be guaranteed to be timely, secure, error or virus-free. The sender does not accept liability for any errors or omissions. ________________________________ This email may contain information that is confidential. If you are not the intended recipient you may not distribute or copy this email or any attachments. If you have received this email in error, please notify the sender and delete this email and attachments from your system immediately. Thank you. -PowerexDisclaimerID5.2.8.1541 -------------- next part -------------- An HTML attachment was scrubbed... URL: -------------- next part -------------- A non-text attachment was scrubbed... Name: image001.jpg Type: image/jpeg Size: 1568 bytes Desc: image001.jpg URL: From raymond.c.robles at intel.com Mon Sep 26 14:00:18 2016 From: raymond.c.robles at intel.com (Robles, Raymond C) Date: Mon, 26 Sep 2016 21:00:18 +0000 Subject: [nvmewin] OFA NVMe Windows Driver Source Code Access In-Reply-To: <53F7355564791145BAF24BA7C569C846953833@FPTPWXMBX1.bchydro.adroot.bchydro.bc.ca> References: <49158E750348AA499168FD41D88983607C64EB01@fmsmsx117.amr.corp.intel.com> <63697667A4DABE4ABF5C991A5E70FE462B4C1C@MAL002.liteon.com> <53F7355564791145BAF24BA7C569C846953833@FPTPWXMBX1.bchydro.adroot.bchydro.bc.ca> Message-ID: <49158E750348AA499168FD41D88983607C65EA75@fmsmsx117.amr.corp.intel.com> Hi Henrique, Before sending any more messages to the nvmewin distribution list, please subscribe to our mailing list (http://lists.openfabrics.org/mailman/listinfo/nvmewin). All of your emails you are sending to this list are getting blocked and then bounced to me... and I have to approve them individually to be sent to the distribution. As for the OFA reference driver binaries, we only publish binaries for released drivers. Since we are currently working on a release, there are no binaries for the current source code. You must build the driver yourself. This is a very simple process. You just need to download and install (in order) the WDK 10, SDK 10, and Visual Studio 2015. You can get the WDK and SDK for free... and a free trial version is available of Visual Studio 2015 from MSDN. Thank you... Ray From: Maqueda, Henrique [mailto:Henrique.Maqueda at powerex.com] Sent: Friday, September 23, 2016 10:46 AM To: Harlan Chou Cc: nvmewin ; Robles, Raymond C Subject: RE: OFA NVMe Windows Driver Source Code Access Hi Harlan, Where did you find get the Win10 driver? Is there a compiled version that we can just install and run some tests? Thanks. From: Harlan Chou [mailto:Harlan.Chou at liteon.com] Sent: September-23-16 3:16 AM To: Robles, Raymond C; nvmewin Cc: Maqueda, Henrique Subject: RE: OFA NVMe Windows Driver Source Code Access Hi Ray We will cover following test, especially for these modification (EOL Read Only Support, SorPort Performance Options, Compatibility Windows 10 32bit/64bit and Surprise Removal Support in IOCTL Path) l IO Test l Compatibility Test l Power Management Test l Performance Test Regards, Harlan Chou 14F Ruey Kuang Rd., Neihu District Tapei City 114, Taiwan, R.O.C. * 886-2-8798-2888#8811| f. 886-2-8798-2822 | * harlan.chou at liteon.com From: Harlan Chou Sent: Wednesday, September 21, 2016 6:34 PM To: 'Robles, Raymond C'; nvmewin Cc: 'Henrique.Maqueda at powerex.com' Subject: RE: OFA NVMe Windows Driver Source Code Access Hi Ray, It's great to receive your notice. We have finished to subscribe OFA reference mail list and access OFA source code. I will mail alpha-test plan to you after our internal test-plan audit meeting is finished. (About one week) We have very interest to support to do release test. Regards, Harlan Chou 14F Ruey Kuang Rd., Neihu District Tapei City 114, Taiwan, R.O.C. * 886-2-8798-2888#8811| f. 886-2-8798-2822 | * harlan.chou at liteon.com From: Robles, Raymond C [mailto:raymond.c.robles at intel.com] Sent: Wednesday, September 21, 2016 1:59 AM To: nvmewin Cc: Harlan Chou; 'Henrique.Maqueda at powerex.com' Subject: OFA NVMe Windows Driver Source Code Access Hello, I've gotten a few offline requests for getting access to the OFA reference driver to help with the unit testing for our upcoming release. Thanks to those that are offering to help... it is greatly appreciated. Anyone can access the OFA reference driver code. Please see instructions on the website. Below is some helpful information... * First, if you have not already, please subscribe to the OFA reference nvmewin mailing list. You can do this at the following link: o http://lists.openfabrics.org/mailman/listinfo/nvmewin * For general information on the OFA nvmewin group, please follow this link: o https://www.openfabrics.org/index.php/working-groups.html o https://www.openfabrics.org/index.php/working-groups-overview.html#nvmewin * To gain access to the OFA source code, you can use a client like Tortoise SVN and either use the https protocol address or SVN protocol address o https://www.openfabrics.org/svnrepo/nvmewin o svn://svn.openfabrics.org/nvmewin Please let me know if there are any question. If you are offering to help with release testing, please email this distribution and state which set of required tests you will be performing. Thanks! Ray Raymond C. Robles NSG ISE Host Storage Software Intel Corporation Office: 480-554-2600 Mobile: 480-399-0645 raymond.c.robles at intel.com [cid:image001.gif at 01CB9B29.EA8D14F0] Confidential Information:This message is sent to the intended recipient and may contain privileged or confidential information. If you received this transmission in error, please notify the sender with a replying e-mail and delete the message and any attachment.Transmission Caveat and Virus Alert: Internet communications cannot be guaranteed to be timely, secure, error or virus-free. The sender does not accept liability for any errors or omissions. ________________________________ This email may contain information that is confidential. If you are not the intended recipient you may not distribute or copy this email or any attachments. If you have received this email in error, please notify the sender and delete this email and attachments from your system immediately. Thank you. -PowerexDisclaimerID5.2.8.1541 -------------- next part -------------- An HTML attachment was scrubbed... URL: -------------- next part -------------- A non-text attachment was scrubbed... Name: image001.jpg Type: image/jpeg Size: 1568 bytes Desc: image001.jpg URL: From raymond.c.robles at intel.com Fri Sep 30 09:52:25 2016 From: raymond.c.robles at intel.com (Robles, Raymond C) Date: Fri, 30 Sep 2016 16:52:25 +0000 Subject: [nvmewin] OFA reference driver release assistance In-Reply-To: References: <49158E750348AA499168FD41D88983607C63713C@fmsmsx117.amr.corp.intel.com> <49158E750348AA499168FD41D88983607C64EF11@fmsmsx117.amr.corp.intel.com> <49158E750348AA499168FD41D88983607C6507D2@fmsmsx117.amr.corp.intel.com> Message-ID: <49158E750348AA499168FD41D88983607C668668@fmsmsx117.amr.corp.intel.com> Hello, As we continue to test for our next release, Intel was originally planning on running the WHCK tests for the release. Unfortunately, we are fairly bandwidth constrained and cannot perform the WHCK tests until about 1 month out. I'd like to ask for a volunteer to the run the WHCK test on the OFA reference driver (on an actively release NVMe SSD). Please keep in mind we do not need to submit results to MSFT, but simply just need the results of the WHCK tests. Any volunteers? Thanks... Ray From: Tom Freeman [mailto:thomas.freeman at hgst.com] Sent: Friday, September 23, 2016 6:15 AM To: Robles, Raymond C ; nvmewin Subject: RE: OFA reference driver release assistance Ray, I don't think this is an issue with the driver. I think this is an HCK issue. I've run the HLK and see there is a filter for this error - 10700. I haven't run the HCK to see if the same filter applies there. >From the Scsi Translation Spec (1.5), section 6.1.3.1.1 NGUID-based translation "PAGE LENGTH shall be set to 40 indicating the page length in bytes." Tom Freeman Software Engineer, Device Manager and Driver Development Western Digital Corporation e. Thomas.freeman at hgst.com o. +1-507-322-2311 [cid:image001.jpg at 01D21B00.57BD7630] From: Robles, Raymond C [mailto:raymond.c.robles at intel.com] Sent: Thursday, September 22, 2016 6:12 PM To: Tom Freeman >; nvmewin > Subject: RE: OFA reference driver release assistance Thanks Tom. Question... do you think you can push a fix for this failure? Thanks... Ray From: Tom Freeman [mailto:thomas.freeman at hgst.com] Sent: Wednesday, September 21, 2016 1:48 PM To: Robles, Raymond C >; nvmewin > Subject: RE: OFA reference driver release assistance In my previous test, I incorrectly used the HLK version of scsicompliance.exe. I reran the test with the HCK version of scsicompliance.exe (version 2.0.20100428) I'm attaching the files with the complete results - password "nvmeOFA15" For 2012 and 2012R2, the command was: scsicompliance /device \\.\physicaldrive1 /verbosity 4 /scenario nvme /operation test /version 2 For 2008R2 the command was: scsicompliance /device \\.\PhysicalDrive2 /verbosity 4 /scenario common /operation test /version 2 2008R2 passed 2012 and 2012r2 had a similar failure as reported with the HLK version of the test. Here are the details of that failure: Policy: REQUIRED Start: ASSERTION: INQUIRY Attempting Unit Serial Number Page 0x80., TUID= Description: Checking if Unit Serial Number Page 0x80 is supported Reference: SCSI Primary Commands - 4 (SPC-4) Revision 30 (or published) specification Section 7.8.15 http://www.t10.org/ftp/t10/drafts/spc4/spc4r30.pdf Expectation: Device returns meaningful data as defined in SCSI-NVME translation document Sending SCSI command to device. INQUIRY Command: 12 01 80 00 ff 00 DeviceIoControl SUCCEEDED. SCSI status: 0x00 (GOOD) Sense bytes: 0x00 Data bytes: 0x2c 00 04 08 0C --------------------------------------------------------------- 000 00800028 30303030 5F303030 305F3030 ...(0000_0000_00 010 30305F30 3030305F 30303043 5F434130 00_0000_000C_CA0 020 305F3630 31465F42 4230302E 00000000 0_601F_BB00..... == DONE ============================================================== INQUIRY: Unit serial number page is supported. "Peripheral Qualifier" = 0x0 "Peripheral Device Type" = 0x0 "Page Code" = 0x80 "Page Length" = 0x28 "Unit serial number page length shall be set to 20" Error: 0x0, Error 0x00000000 Logging an error File=testsrc\driverstest\storage\wdk\scsicompliance\lib\core\policyscenario.cpp Line=213 End: Fail, ASSERTION: INQUIRY Attempting Unit Serial Number Page 0x80., TUID=, Repro=scsicompliance /device \\.\physicaldrive1 /verbosity 4 /scenario nvme /operation test /version 2 Tom Freeman Software Engineer, Device Manager and Driver Development Western Digital Corporation e. Thomas.freeman at hgst.com o. +1-507-322-2311 [cid:image002.jpg at 01D21B00.57BD7630] From: Robles, Raymond C [mailto:raymond.c.robles at intel.com] Sent: Tuesday, September 20, 2016 4:29 PM To: Tom Freeman >; nvmewin > Cc: Huffman, Amber > Subject: RE: OFA reference driver release assistance Hi Tom, This is awesome. Thank you very much for completing this testing. I'm adding it to a test spreadsheet I'm maintaining. On the failure below, I'll log the 40 length expectation by the test. BTW, can you provide the MS SCSI compliance command line you used to run the test? As well as the version? Thanks again! Ray From: Tom Freeman [mailto:thomas.freeman at hgst.com] Sent: Tuesday, September 20, 2016 11:59 AM To: Robles, Raymond C >; nvmewin > Cc: Huffman, Amber > Subject: RE: OFA reference driver release assistance I've completed my testing for the 1.5 release All the testing was successful. There was a single failure in the Scsi Compliance (details at the end of the note). Filter 10700 overrides this on Server 2016 HLK testing. I was not able to verify that filter on HCK on 2008R2, 2012 and 2012R2. Testing details: 64-bit on 2008R2, 2012 and 2012R2 1 hour of data integrity testing using sdstress 1 hour of heavy stress testing using IOMETER covering, at least, 512B, 4KB and 128KB ranging from 1 OIO to 64 OIO both sequential and random Quick and slow format of both MBR and GPT partitioning Microsoft SCSI Compliance. Scsi Compliance Failure: Policy: REQUIRED Start: ASSERTION: INQUIRY Attempting Unit Serial Number Page 0x80., TUID= Description: Checking if Unit Serial Number Page 0x80 is supported Reference: SCSI Primary Commands - 4 (SPC-4) Revision 30 (or published) specification Section 7.8.15 http://www.t10.org/ftp/t10/drafts/spc4/spc4r30.pdf Expectation: Device returns meaningful data as defined in SCSI-NVME translation document Sending SCSI command to device. INQUIRY Command: 12 01 80 00 ff 00 DeviceIoControl SUCCEEDED. SCSI status: 0x00 (GOOD) Sense bytes: 0x00 Data bytes: 0x2c 00 04 08 0C --------------------------------------------------------------- 000 00800028 30303030 5F303030 305F3030 ...(0000_0000_00 010 30305F30 3030305F 30303043 5F434130 00_0000_000C_CA0 020 305F3630 31465F42 4230302E 00000000 0_601F_BB00..... == DONE ============================================================== INQUIRY: Unit serial number page is supported. "Peripheral Qualifier" = 0x0 "Peripheral Device Type" = 0x0 "Page Code" = 0x80 Error: 0x0, Error 0x00000000 Unit serial number page length shall be set to either 20 for EUI64 or 36 for NGUID. Actual=40 File=drivers\storage\tests\scsicompliance\lib\nvme\inquiry_nvme.cpp Line=120 Error: 0x0, Error 0x00000000 Logging an error File=drivers\storage\tests\scsicompliance\lib\core\policyscenario.cpp Line=213 End: Fail, ASSERTION: INQUIRY Attempting Unit Serial Number Page 0x80., TUID=, Repro=scsicompliance /device \\.\PhysicalDrive1 /verbosity 4 /scenario NVMe /Operation test /Version 2 Tom Freeman Software Engineer, Device Manager and Driver Development Western Digital Corporation e. Thomas.freeman at hgst.com o. +1-507-322-2311 [cid:image006.jpg at 01D2141E.ACAFEA30] From: Tom Freeman Sent: Monday, September 12, 2016 11:49 AM To: 'Robles, Raymond C' >; nvmewin > Cc: Huffman, Amber > Subject: RE: OFA reference driver release assistance I can run the Unit testing test for 64-bit on 2008R2, 2012 and 2012R2 These are the unit tests I'll run: - 1 hour of data integrity testing using sdstress (Microsoft Tool) - 1 hour of heavy stress testing using IOMETER covering, at least, 512B, 4KB and 128KB ranging from 1 OIO to 64 OIO both sequential and random - Quick and slow format of both MBR and GPT partitioning - Microsoft SCSI Compliance, no failures except (warnings OK) Tom Freeman Software Engineer, Device Manager and Driver Development Western Digital Corporation e. Thomas.freeman at hgst.com o. +1-507-322-2311 [cid:image007.jpg at 01D2141E.ACAFEA30] From: nvmewin [mailto:nvmewin-bounces at lists.openfabrics.org] On Behalf Of Robles, Raymond C Sent: Friday, September 9, 2016 12:47 PM To: nvmewin > Cc: Huffman, Amber > Subject: [nvmewin] OFA reference driver release assistance Hello, In order to officially release the OFA reference driver, there are some tests that need to be run. I'm asking for volunteers for run the following tests: * WHQL/WHCK Certification Tests (no submission to MS required... just need test results) * Normal Unit Testing Test Suite The only requirements for devices on which to run the above tests is that the NVMe SSD be 1.2 compliant. As additional info, here is a list of features that we've added since our last release. Please review and sanity check my list. Recent patches include * Performance improvement & robustness * NVMe Spec rev 1.2 feature compliant * Supports MS Windows 10, 8.1, 7, Server 2012 R2, 2012 and 2008 R2 * Supports both 32 & 64-bit * What is new since last release: o Namespace Management (Create, Delete, Attach, Detach) o EOL Read Only Support o Win 8.1 Timers o Surprise Removal Support in IOCTL Path o Disk Initialization Performance Optimization o Storage Request Block Support o StorPort Performance Options o StorPort DPC Redirection o Misc. Bug Fixes o Security Send/Receive with Zero Data Length o SNTI updates for SCSI to NVMe Translation Thanks... Ray Raymond C. Robles NSG ISE Host Storage Software Intel Corporation Office: 480-554-2600 Mobile: 480-399-0645 raymond.c.robles at intel.com [cid:image001.gif at 01CB9B29.EA8D14F0] Western Digital Corporation (and its subsidiaries) E-mail Confidentiality Notice & Disclaimer: This e-mail and any files transmitted with it may contain confidential or legally privileged information of WDC and/or its affiliates, and are intended solely for the use of the individual or entity to which they are addressed. If you are not the intended recipient, any disclosure, copying, distribution or any action taken or omitted to be taken in reliance on it, is prohibited. If you have received this e-mail in error, please notify the sender immediately and delete the e-mail in its entirety from your system. Western Digital Corporation (and its subsidiaries) E-mail Confidentiality Notice & Disclaimer: This e-mail and any files transmitted with it may contain confidential or legally privileged information of WDC and/or its affiliates, and are intended solely for the use of the individual or entity to which they are addressed. If you are not the intended recipient, any disclosure, copying, distribution or any action taken or omitted to be taken in reliance on it, is prohibited. If you have received this e-mail in error, please notify the sender immediately and delete the e-mail in its entirety from your system. Western Digital Corporation (and its subsidiaries) E-mail Confidentiality Notice & Disclaimer: This e-mail and any files transmitted with it may contain confidential or legally privileged information of WDC and/or its affiliates, and are intended solely for the use of the individual or entity to which they are addressed. If you are not the intended recipient, any disclosure, copying, distribution or any action taken or omitted to be taken in reliance on it, is prohibited. If you have received this e-mail in error, please notify the sender immediately and delete the e-mail in its entirety from your system. -------------- next part -------------- An HTML attachment was scrubbed... URL: -------------- next part -------------- A non-text attachment was scrubbed... Name: image001.jpg Type: image/jpeg Size: 2934 bytes Desc: image001.jpg URL: -------------- next part -------------- A non-text attachment was scrubbed... Name: image002.jpg Type: image/jpeg Size: 2938 bytes Desc: image002.jpg URL: -------------- next part -------------- A non-text attachment was scrubbed... Name: image003.jpg Type: image/jpeg Size: 2934 bytes Desc: image003.jpg URL: -------------- next part -------------- A non-text attachment was scrubbed... Name: image004.jpg Type: image/jpeg Size: 2937 bytes Desc: image004.jpg URL: -------------- next part -------------- A non-text attachment was scrubbed... Name: image005.jpg Type: image/jpeg Size: 1568 bytes Desc: image005.jpg URL: