<HTML xmlns:v = "urn:schemas-microsoft-com:vml" xmlns:o = "urn:schemas-microsoft-com:office:office" xmlns:w = "urn:schemas-microsoft-com:office:word" xmlns:m = "http://schemas.microsoft.com/office/2004/12/omml"><HEAD>
<META content="text/html; charset=utf-8" http-equiv=Content-Type>
<STYLE id=mysingle_style type=text/css>P {
MARGIN-BOTTOM: 5px; FONT-SIZE: 9pt; FONT-FAMILY: Arial, arial; MARGIN-TOP: 5px
}
TD {
MARGIN-BOTTOM: 5px; FONT-SIZE: 9pt; FONT-FAMILY: Arial, arial; MARGIN-TOP: 5px
}
LI {
MARGIN-BOTTOM: 5px; FONT-SIZE: 9pt; FONT-FAMILY: Arial, arial; MARGIN-TOP: 5px
}
BODY {
FONT-SIZE: 9pt; FONT-FAMILY: Arial, arial; MARGIN: 10px; LINE-HEIGHT: 1.4
}
</STYLE>
<META content=IE=5 http-equiv=X-UA-Compatible>
<META name=GENERATOR content="MSHTML 11.00.9600.18212"></HEAD>
<BODY>
<P><SPAN style="FONT-SIZE: 11pt; FONT-FAMILY: Calibri">Hi Ray,</SPAN></P>
<P><SPAN style="FONT-SIZE: 11pt; FONT-FAMILY: Calibri"></SPAN> </P>
<P><SPAN style="FONT-SIZE: 11pt; FONT-FAMILY: Calibri">Samsung submitted 3 patches. Revision 134 is first patch, Revision 136 and 137 is second patch.</SPAN></P>
<P><SPAN style="FONT-SIZE: 11pt; FONT-FAMILY: Calibri">We submitted third patch for Handling device Surprise removal in IOCTL path which was approved by HGST and Intel on 1st June. Please find attached the mail thread for reference.</SPAN></P>
<P><SPAN style="FONT-SIZE: 11pt; FONT-FAMILY: Calibri">The third patch is not yet checked-in to trunk.</SPAN></P>
<P><SPAN style="FONT-SIZE: 11pt; FONT-FAMILY: Calibri"></SPAN> </P>
<P><SPAN style="FONT-SIZE: 11pt; FONT-FAMILY: Calibri">Thanks,</SPAN></P>
<P><SPAN style="FONT-SIZE: 11pt; FONT-FAMILY: Calibri">Suman</SPAN></P>
<P> </P>
<P>------- <B>Original Message</B> -------</P>
<P><B>Sender</B> : Robles, Raymond C<raymond.c.robles@intel.com></P>
<P><B>Date</B> : Jun 27, 2016 12:19 (GMT+05:30)</P>
<P><B>Title</B> : Re: [nvmewin] review comments for patch from Google</P>
<P> </P>
<META name=Generator content=ActiveSquare>
<STYLE>v\:* {
BEHAVIOR: url(#default#VML)
}
o\:* {
BEHAVIOR: url(#default#VML)
}
w\:* {
BEHAVIOR: url(#default#VML)
}
.shape {
BEHAVIOR: url(#default#VML)
}
</STYLE>
<STYLE>@font-face {
font-family: Cambria Math;
}
@font-face {
font-family: Calibri;
}
@font-face {
font-family: Tahoma;
}
@font-face {
font-family: Intel Clear;
}
@page WordSection1 {size: 8.5in 11.0in; margin: 1.0in 1.0in 1.0in 1.0in; }
P.MsoNormal {
FONT-SIZE: 12pt; FONT-FAMILY: "Times New Roman",serif; MARGIN: 0in 0in 0pt
}
LI.MsoNormal {
FONT-SIZE: 12pt; FONT-FAMILY: "Times New Roman",serif; MARGIN: 0in 0in 0pt
}
DIV.MsoNormal {
FONT-SIZE: 12pt; FONT-FAMILY: "Times New Roman",serif; MARGIN: 0in 0in 0pt
}
A:link {
TEXT-DECORATION: underline; COLOR: blue; mso-style-priority: 99
}
SPAN.MsoHyperlink {
TEXT-DECORATION: underline; COLOR: blue; mso-style-priority: 99
}
A:visited {
TEXT-DECORATION: underline; COLOR: purple; mso-style-priority: 99
}
SPAN.MsoHyperlinkFollowed {
TEXT-DECORATION: underline; COLOR: purple; mso-style-priority: 99
}
P {
FONT-SIZE: 12pt; FONT-FAMILY: "Times New Roman",serif; MARGIN-LEFT: 0in; MARGIN-RIGHT: 0in; mso-style-priority: 99; mso-margin-top-alt: auto; mso-margin-bottom-alt: auto
}
SPAN.EmailStyle18 {
FONT-FAMILY: "Calibri",sans-serif; FONT-WEIGHT: normal; COLOR: #002060; FONT-STYLE: normal; mso-style-type: personal
}
SPAN.EmailStyle19 {
TEXT-DECORATION: none; FONT-FAMILY: "Calibri",sans-serif; FONT-WEIGHT: normal; COLOR: #002060; FONT-STYLE: normal; mso-style-type: personal-reply
}
.MsoChpDefault {
FONT-SIZE: 10pt; mso-style-type: export-only
}
DIV.WordSection1 {
page: WordSection1
}
</STYLE>
<META content=IE=5 http-equiv=X-UA-Compatible><X-BODY lang=EN-US link="blue" vlink="purple">
<DIV class=WordSection1>
<P class=MsoNormal><SPAN style='FONT-SIZE: 11pt; FONT-FAMILY: "Calibri",sans-serif; COLOR: #002060'>Hi Iuliu,<o:p></o:p></SPAN></P>
<P class=MsoNormal><SPAN style='FONT-SIZE: 11pt; FONT-FAMILY: "Calibri",sans-serif; COLOR: #002060'><o:p> </o:p></SPAN></P>
<P class=MsoNormal><SPAN style='FONT-SIZE: 11pt; FONT-FAMILY: "Calibri",sans-serif; COLOR: #002060'>I apologize for the delay in responding. I was performing a manual diff of the last several merges to complete the investigation. Here is what I found. <o:p></o:p></SPAN></P>
<P class=MsoNormal><SPAN style='FONT-SIZE: 11pt; FONT-FAMILY: "Calibri",sans-serif; COLOR: #002060'><o:p> </o:p></SPAN></P>
<P class=MsoNormal><SPAN style='FONT-SIZE: 11pt; FONT-FAMILY: "Calibri",sans-serif; COLOR: #002060'>The latest revision on the OFA source trunk is actually 137 (refer to my screenshot below). I’ve manually gone back and made sure that all the patches submitted are in this latest revision. I found no errors or issues. The latest trunk is up to date with all of the patches. If you were trying to apply your patch yourself, you cannot. Only myself (OFA co-chair) and Carolyn Foster (OFA source maintainer) have write permissions to the repo.<o:p></o:p></SPAN></P>
<P class=MsoNormal><SPAN style='FONT-SIZE: 11pt; FONT-FAMILY: "Calibri",sans-serif; COLOR: #002060'><o:p> </o:p></SPAN></P>
<P class=MsoNormal><SPAN style='FONT-SIZE: 11pt; FONT-FAMILY: "Calibri",sans-serif; COLOR: #002060'>Each patch listed below is applied to trunk. The only think you need to insure is that your latest concurrent channels modifications are based off the 137 revision. If you have any questions, please me know. <o:p></o:p></SPAN></P>
<P class=MsoNormal><SPAN style='FONT-SIZE: 11pt; FONT-FAMILY: "Calibri",sans-serif; COLOR: #002060'><o:p> </o:p></SPAN></P>
<P class=MsoNormal><SPAN style='FONT-SIZE: 11pt; FONT-FAMILY: "Calibri",sans-serif; COLOR: #002060'>Here is the screenshot of all the patches I’ve checked…<o:p></o:p></SPAN></P>
<P class=MsoNormal><SPAN style='FONT-SIZE: 11pt; FONT-FAMILY: "Calibri",sans-serif; COLOR: #002060'><o:p> </o:p></SPAN></P>
<P class=MsoNormal><SPAN style='FONT-SIZE: 11pt; FONT-FAMILY: "Calibri",sans-serif; COLOR: #002060'><o:p> </o:p></SPAN></P>
<P class=MsoNormal><SPAN style='FONT-SIZE: 11pt; FONT-FAMILY: "Calibri",sans-serif; COLOR: #002060'><IMG id=Picture_x0020_1 src="cid:QVYHLCGFY2AH@namo.co.kr" width=1082 height=449></SPAN><SPAN style='FONT-SIZE: 11pt; FONT-FAMILY: "Calibri",sans-serif; COLOR: #002060'><o:p></o:p></SPAN></P>
<P class=MsoNormal><SPAN style='FONT-SIZE: 11pt; FONT-FAMILY: "Calibri",sans-serif; COLOR: #002060'><o:p> </o:p></SPAN></P>
<P class=MsoNormal><SPAN style='FONT-SIZE: 11pt; FONT-FAMILY: "Calibri",sans-serif; COLOR: #002060'><o:p> </o:p></SPAN></P>
<P class=MsoNormal><SPAN style='FONT-SIZE: 11pt; FONT-FAMILY: "Calibri",sans-serif; COLOR: #002060'>Thank you…<o:p></o:p></SPAN></P>
<P class=MsoNormal><SPAN style='FONT-SIZE: 11pt; FONT-FAMILY: "Calibri",sans-serif; COLOR: #002060'>Ray<o:p></o:p></SPAN></P>
<P class=MsoNormal><A name=_MailEndCompose><SPAN style='FONT-SIZE: 11pt; FONT-FAMILY: "Calibri",sans-serif; COLOR: #002060'><o:p> </o:p></SPAN></A></P>
<DIV>
<DIV style="BORDER-TOP: #e1e1e1 1pt solid; BORDER-RIGHT: medium none; BORDER-BOTTOM: medium none; PADDING-BOTTOM: 0in; PADDING-TOP: 3pt; PADDING-LEFT: 0in; BORDER-LEFT: medium none; PADDING-RIGHT: 0in">
<P class=MsoNormal><B><SPAN style='FONT-SIZE: 11pt; FONT-FAMILY: "Calibri",sans-serif'>From:</SPAN></B><SPAN style='FONT-SIZE: 11pt; FONT-FAMILY: "Calibri",sans-serif'> nvmewin [mailto:nvmewin-bounces@lists.openfabrics.org] <B>On Behalf Of </B>Robles, Raymond C<BR><B>Sent:</B> Friday, June 17, 2016 12:08 PM<BR><B>To:</B> Iuliu Rus <iuliur@google.com><BR><B>Cc:</B> nvmewin <nvmewin@lists.openfabrics.org><BR><B>Subject:</B> Re: [nvmewin] review comments for patch from Google<o:p></o:p></SPAN></P></DIV></DIV>
<P class=MsoNormal><o:p> </o:p></P>
<P class=MsoNormal><SPAN style='FONT-SIZE: 11pt; FONT-FAMILY: "Calibri",sans-serif; COLOR: #002060'>Thanks Iuliu. I’m currently investigating the issue. Will let everyone know when the trunk is updated. <o:p></o:p></SPAN></P>
<P class=MsoNormal><SPAN style='FONT-SIZE: 11pt; FONT-FAMILY: "Calibri",sans-serif; COLOR: #002060'><o:p> </o:p></SPAN></P>
<P class=MsoNormal><SPAN style='FONT-SIZE: 11pt; FONT-FAMILY: "Calibri",sans-serif; COLOR: #002060'>Thank you…<o:p></o:p></SPAN></P>
<P class=MsoNormal><SPAN style='FONT-SIZE: 11pt; FONT-FAMILY: "Calibri",sans-serif; COLOR: #002060'>Ray<o:p></o:p></SPAN></P>
<P class=MsoNormal><SPAN style='FONT-SIZE: 11pt; FONT-FAMILY: "Calibri",sans-serif; COLOR: #002060'><o:p> </o:p></SPAN></P>
<P class=MsoNormal><B><SPAN style='FONT-SIZE: 11pt; FONT-FAMILY: "Calibri",sans-serif'>From:</SPAN></B><SPAN style='FONT-SIZE: 11pt; FONT-FAMILY: "Calibri",sans-serif'> Iuliu Rus [<A href="mailto:iuliur@google.com">mailto:iuliur@google.com</A>] <BR><B>Sent:</B> Friday, June 17, 2016 10:29 AM<BR><B>To:</B> Robles, Raymond C <<A href="mailto:raymond.c.robles@intel.com">raymond.c.robles@intel.com</A>><BR><B>Cc:</B> Thomas Freeman <<A href="mailto:thomas.freeman@hgst.com">thomas.freeman@hgst.com</A>>; nvmewin <<A href="mailto:nvmewin@lists.openfabrics.org">nvmewin@lists.openfabrics.org</A>><BR><B>Subject:</B> Re: [nvmewin] review comments for patch from Google<o:p></o:p></SPAN></P>
<P class=MsoNormal><o:p> </o:p></P>
<DIV>
<P class=MsoNormal>I don't see any changes when i try to update. My svn says : completed at revision 137.<o:p></o:p></P></DIV>
<DIV>
<P class=MsoNormal><o:p> </o:p></P>
<DIV>
<P class=MsoNormal>On Thu, Jun 16, 2016 at 2:42 PM, Robles, Raymond C <<A href="mailto:raymond.c.robles@intel.com" target=_blank>raymond.c.robles@intel.com</A>> wrote:<o:p></o:p></P>
<BLOCKQUOTE style="BORDER-TOP: medium none; BORDER-RIGHT: medium none; BORDER-BOTTOM: medium none; PADDING-BOTTOM: 0in; PADDING-TOP: 0in; PADDING-LEFT: 6pt; MARGIN: 5pt 0in 5pt 4.8pt; BORDER-LEFT: #cccccc 1pt solid; PADDING-RIGHT: 0in">
<DIV>
<DIV>
<P class=MsoNormal style="mso-margin-top-alt: auto; mso-margin-bottom-alt: auto"><SPAN style='FONT-SIZE: 11pt; FONT-FAMILY: "Calibri",sans-serif; COLOR: #002060'>Hi Iuliu- Please rebase your patch with the latest OFA trunk source. If you do not see any updates during your rebase, please let me know and I’ll investigate. Thanks!</SPAN><o:p></o:p></P>
<P class=MsoNormal style="mso-margin-top-alt: auto; mso-margin-bottom-alt: auto"><SPAN style='FONT-SIZE: 11pt; FONT-FAMILY: "Calibri",sans-serif; COLOR: #002060'> </SPAN><o:p></o:p></P>
<P class=MsoNormal style="mso-margin-top-alt: auto; mso-margin-bottom-alt: auto"><B><SPAN style='FONT-SIZE: 11pt; FONT-FAMILY: "Calibri",sans-serif'>From:</SPAN></B><SPAN style='FONT-SIZE: 11pt; FONT-FAMILY: "Calibri",sans-serif'> Iuliu Rus [mailto:<A href="mailto:iuliur@google.com" target=_blank>iuliur@google.com</A>] <BR><B>Sent:</B> Thursday, June 16, 2016 2:15 PM<BR><B>To:</B> Robles, Raymond C <<A href="mailto:raymond.c.robles@intel.com" target=_blank>raymond.c.robles@intel.com</A>><BR><B>Cc:</B> Thomas Freeman <<A href="mailto:thomas.freeman@hgst.com" target=_blank>thomas.freeman@hgst.com</A>>; nvmewin <<A href="mailto:nvmewin@lists.openfabrics.org" target=_blank>nvmewin@lists.openfabrics.org</A>></SPAN><o:p></o:p></P>
<DIV>
<DIV>
<P class=MsoNormal><BR><B>Subject:</B> Re: [nvmewin] review comments for patch from Google<o:p></o:p></P></DIV></DIV>
<DIV>
<DIV>
<P class=MsoNormal style="mso-margin-top-alt: auto; mso-margin-bottom-alt: auto"> <o:p></o:p></P>
<DIV>
<P class=MsoNormal style="mso-margin-top-alt: auto; mso-margin-bottom-alt: auto">Let me know if i have to sync to the latest and re-do the patch.<o:p></o:p></P></DIV>
<DIV>
<P class=MsoNormal style="mso-margin-top-alt: auto; mso-margin-bottom-alt: auto"> <o:p></o:p></P>
<DIV>
<P class=MsoNormal style="mso-margin-top-alt: auto; mso-margin-bottom-alt: auto">On Thu, Jun 16, 2016 at 2:10 PM, Iuliu Rus <<A href="mailto:iuliur@google.com" target=_blank>iuliur@google.com</A>> wrote:<o:p></o:p></P>
<BLOCKQUOTE style="BORDER-TOP: medium none; BORDER-RIGHT: medium none; BORDER-BOTTOM: medium none; PADDING-BOTTOM: 0in; PADDING-TOP: 0in; PADDING-LEFT: 6pt; MARGIN: 5pt 0in 5pt 4.8pt; BORDER-LEFT: #cccccc 1pt solid; PADDING-RIGHT: 0in">
<DIV>
<P class=MsoNormal style="mso-margin-top-alt: auto; mso-margin-bottom-alt: auto">We already have a pool of locks - the dpc locks, which is what i'm using in my latest patch. You cannot create spinlocks directly in a storport driver, the only option is to use either the StartIoLock, the dpc lock, the interrupt lock or one of the MSI locks.<o:p></o:p></P></DIV>
<DIV>
<DIV>
<DIV>
<P class=MsoNormal style="mso-margin-top-alt: auto; mso-margin-bottom-alt: auto"> <o:p></o:p></P>
<DIV>
<P class=MsoNormal style="mso-margin-top-alt: auto; mso-margin-bottom-alt: auto">On Thu, Jun 16, 2016 at 2:04 PM, Robles, Raymond C <<A href="mailto:raymond.c.robles@intel.com" target=_blank>raymond.c.robles@intel.com</A>> wrote:<o:p></o:p></P>
<BLOCKQUOTE style="BORDER-TOP: medium none; BORDER-RIGHT: medium none; BORDER-BOTTOM: medium none; PADDING-BOTTOM: 0in; PADDING-TOP: 0in; PADDING-LEFT: 6pt; MARGIN: 5pt 0in 5pt 4.8pt; BORDER-LEFT: #cccccc 1pt solid; PADDING-RIGHT: 0in">
<DIV>
<DIV>
<P class=MsoNormal style="mso-margin-top-alt: auto; mso-margin-bottom-alt: auto"><SPAN style='FONT-SIZE: 11pt; FONT-FAMILY: "Calibri",sans-serif; COLOR: #002060'>Hello,</SPAN><o:p></o:p></P>
<P class=MsoNormal style="mso-margin-top-alt: auto; mso-margin-bottom-alt: auto"><SPAN style='FONT-SIZE: 11pt; FONT-FAMILY: "Calibri",sans-serif; COLOR: #002060'> </SPAN><o:p></o:p></P>
<P class=MsoNormal style="mso-margin-top-alt: auto; mso-margin-bottom-alt: auto"><SPAN style='FONT-SIZE: 11pt; FONT-FAMILY: "Calibri",sans-serif; COLOR: #002060'>Here are my thoughts… </SPAN><o:p></o:p></P>
<P class=MsoNormal style="mso-margin-top-alt: auto; mso-margin-bottom-alt: auto"><SPAN style='FONT-SIZE: 11pt; FONT-FAMILY: "Calibri",sans-serif; COLOR: #002060'> </SPAN><o:p></o:p></P>
<P class=MsoNormal style="mso-margin-top-alt: auto; mso-margin-bottom-alt: auto"><SPAN style='FONT-SIZE: 11pt; FONT-FAMILY: "Calibri",sans-serif; COLOR: #002060'>In short, I agree with Thomas’s assessment. </SPAN><o:p></o:p></P>
<P class=MsoNormal style="mso-margin-top-alt: auto; mso-margin-bottom-alt: auto"><SPAN style='FONT-SIZE: 11pt; FONT-FAMILY: "Calibri",sans-serif; COLOR: #002060'> </SPAN><o:p></o:p></P>
<P class=MsoNormal style="mso-margin-top-alt: auto; mso-margin-bottom-alt: auto"><SPAN style='FONT-SIZE: 11pt; FONT-FAMILY: "Calibri",sans-serif; COLOR: #002060'>Concurrent channels allows Storport to call StartIo concurrently… and with no locks. Therefore the locking of critical sections is now the responsibility of the miniport.</SPAN><o:p></o:p></P>
<P class=MsoNormal style="mso-margin-top-alt: auto; mso-margin-bottom-alt: auto"><SPAN style='FONT-SIZE: 11pt; FONT-FAMILY: "Calibri",sans-serif; COLOR: #002060'> </SPAN><o:p></o:p></P>
<P class=MsoNormal style="mso-margin-top-alt: auto; mso-margin-bottom-alt: auto"><SPAN style='FONT-SIZE: 11pt; FONT-FAMILY: "Calibri",sans-serif; COLOR: #002060'>Any solution proposed should protect critical resources (cmd id pool, cmd entries, doorbells) with a lock that is granular enough for our driver, but not at the visibility of the OS or Storport. Acquiring the StartIo lock to protect resources in our driver only is too broad of a lock. My suggestion would be to create a pool of locks (one for every SQ/CQ pair) that is maintained within our driver. Again, crashdump mode is a non-issue because it is single threaded.</SPAN><o:p></o:p></P>
<P class=MsoNormal style="mso-margin-top-alt: auto; mso-margin-bottom-alt: auto"><SPAN style='FONT-SIZE: 11pt; FONT-FAMILY: "Calibri",sans-serif; COLOR: #002060'> </SPAN><o:p></o:p></P>
<P class=MsoNormal style="mso-margin-top-alt: auto; mso-margin-bottom-alt: auto"><SPAN style='FONT-SIZE: 11pt; FONT-FAMILY: "Calibri",sans-serif; COLOR: #002060'>Thank you…</SPAN><o:p></o:p></P>
<P class=MsoNormal style="mso-margin-top-alt: auto; mso-margin-bottom-alt: auto"><SPAN style='FONT-SIZE: 11pt; FONT-FAMILY: "Calibri",sans-serif; COLOR: #002060'>Ray</SPAN><o:p></o:p></P>
<P class=MsoNormal style="mso-margin-top-alt: auto; mso-margin-bottom-alt: auto"><SPAN style='FONT-SIZE: 11pt; FONT-FAMILY: "Calibri",sans-serif; COLOR: #002060'> </SPAN><o:p></o:p></P>
<DIV>
<DIV style="BORDER-TOP: #e1e1e1 1pt solid; BORDER-RIGHT: medium none; BORDER-BOTTOM: medium none; PADDING-BOTTOM: 0in; PADDING-TOP: 3pt; PADDING-LEFT: 0in; BORDER-LEFT: medium none; PADDING-RIGHT: 0in">
<P class=MsoNormal style="mso-margin-top-alt: auto; mso-margin-bottom-alt: auto"><B><SPAN style='FONT-SIZE: 11pt; FONT-FAMILY: "Calibri",sans-serif'>From:</SPAN></B><SPAN style='FONT-SIZE: 11pt; FONT-FAMILY: "Calibri",sans-serif'> Thomas Freeman [mailto:<A href="mailto:thomas.freeman@hgst.com" target=_blank>thomas.freeman@hgst.com</A>] <BR><B>Sent:</B> Thursday, June 16, 2016 1:51 PM<BR><B>To:</B> Iuliu Rus <<A href="mailto:iuliur@google.com" target=_blank>iuliur@google.com</A>><BR><B>Cc:</B> Robles, Raymond C <<A href="mailto:raymond.c.robles@intel.com" target=_blank>raymond.c.robles@intel.com</A>>; nvmewin <<A href="mailto:nvmewin@lists.openfabrics.org" target=_blank>nvmewin@lists.openfabrics.org</A>><BR><B>Subject:</B> RE: [nvmewin] review comments for patch from Google</SPAN><o:p></o:p></P></DIV></DIV>
<DIV>
<DIV>
<P class=MsoNormal style="mso-margin-top-alt: auto; mso-margin-bottom-alt: auto"> <o:p></o:p></P>
<P class=MsoNormal style="mso-margin-top-alt: auto; mso-margin-bottom-alt: auto"><SPAN style='FONT-SIZE: 11pt; FONT-FAMILY: "Calibri",sans-serif; COLOR: #1f497d'>Hi Iuliu,</SPAN><o:p></o:p></P>
<P class=MsoNormal style="mso-margin-top-alt: auto; mso-margin-bottom-alt: auto"><SPAN style='FONT-SIZE: 11pt; FONT-FAMILY: "Calibri",sans-serif; COLOR: #1f497d'>Here my view, I hope someone else can chime in to verify or correct what I'm saying</SPAN><o:p></o:p></P>
<P class=MsoNormal style="mso-margin-top-alt: auto; mso-margin-bottom-alt: auto"><SPAN style='FONT-SIZE: 11pt; FONT-FAMILY: "Calibri",sans-serif; COLOR: #1f497d'> </SPAN><o:p></o:p></P>
<P class=MsoNormal style="mso-margin-top-alt: auto; mso-margin-bottom-alt: auto"><SPAN style='FONT-SIZE: 11pt; FONT-FAMILY: "Calibri",sans-serif; COLOR: #1f497d'>The purpose of the concurrent channels change is to prevent Storport from acquiring the StartIo lock before calling Startio. This allows multiple simultaneous calls to StartIo. If you require ProcessIo get the StartIo lock, you are replicating what Storport used to - it defeats the point of concurrent channels. </SPAN><o:p></o:p></P>
<P class=MsoNormal style="mso-margin-top-alt: auto; mso-margin-bottom-alt: auto"><SPAN style='FONT-SIZE: 11pt; FONT-FAMILY: "Calibri",sans-serif; COLOR: #1f497d'> </SPAN><o:p></o:p></P>
<P class=MsoNormal style="mso-margin-top-alt: auto; mso-margin-bottom-alt: auto"><SPAN style='FONT-SIZE: 11pt; FONT-FAMILY: "Calibri",sans-serif; COLOR: #1f497d'>Here is my suggestion for what needs to be changed</SPAN><o:p></o:p></P>
<P class=MsoNormal style="mso-margin-top-alt: auto; mso-margin-bottom-alt: auto"><SPAN style='FONT-SIZE: 11pt; FONT-FAMILY: "Calibri",sans-serif; COLOR: #1f497d'>1. There is no need for StartIo Locks - remove the requests to acquire them</SPAN><o:p></o:p></P>
<P class=MsoNormal style="mso-margin-top-alt: auto; mso-margin-bottom-alt: auto"><SPAN style='FONT-SIZE: 11pt; FONT-FAMILY: "Calibri",sans-serif; COLOR: #1f497d'>2. Acquire the Dpc locks in the following situations</SPAN><o:p></o:p></P>
<P class=MsoNormal style="mso-margin-top-alt: auto; mso-margin-bottom-alt: auto"><SPAN style='FONT-SIZE: 11pt; FONT-FAMILY: "Calibri",sans-serif; COLOR: #1f497d'> -ProcessIo, before calling GetCmdEntry - if there is queue sharing</SPAN><o:p></o:p></P>
<P class=MsoNormal style="mso-margin-top-alt: auto; mso-margin-bottom-alt: auto"><SPAN style='FONT-SIZE: 11pt; FONT-FAMILY: "Calibri",sans-serif; COLOR: #1f497d'> -ProcessIo, before calling IssueCmd - always</SPAN><o:p></o:p></P>
<P class=MsoNormal style="mso-margin-top-alt: auto; mso-margin-bottom-alt: auto"><SPAN style='FONT-SIZE: 11pt; FONT-FAMILY: "Calibri",sans-serif; COLOR: #1f497d'> -IoCompletionRoutine, before calling NVMeCompleteCmd - if there is queue sharing</SPAN><o:p></o:p></P>
<P class=MsoNormal style="mso-margin-top-alt: auto; mso-margin-bottom-alt: auto"><SPAN style='FONT-SIZE: 11pt; FONT-FAMILY: "Calibri",sans-serif; COLOR: #1f497d'> </SPAN><o:p></o:p></P>
<P class=MsoNormal style="mso-margin-top-alt: auto; mso-margin-bottom-alt: auto"><SPAN style='FONT-SIZE: 11pt; FONT-FAMILY: "Calibri",sans-serif; COLOR: #1f497d'> -recoveryDpcRoutine - get all of the Dpc locks</SPAN><o:p></o:p></P>
<P class=MsoNormal style="mso-margin-top-alt: auto; mso-margin-bottom-alt: auto"><SPAN style='FONT-SIZE: 11pt; FONT-FAMILY: "Calibri",sans-serif; COLOR: #1f497d'> </SPAN><o:p></o:p></P>
<P class=MsoNormal style="mso-margin-top-alt: auto; mso-margin-bottom-alt: auto"><SPAN style='FONT-SIZE: 11pt; FONT-FAMILY: "Calibri",sans-serif; COLOR: #1f497d'> </SPAN><o:p></o:p></P>
<P class=MsoNormal style="mso-margin-top-alt: auto; mso-margin-bottom-alt: auto"><SPAN style='FONT-SIZE: 11pt; FONT-FAMILY: "Calibri",sans-serif; COLOR: #1f497d'>Here is my reasoning</SPAN><o:p></o:p></P>
<P class=MsoNormal style="mso-margin-top-alt: auto; mso-margin-bottom-alt: auto"><SPAN style='FONT-SIZE: 11pt; FONT-FAMILY: "Calibri",sans-serif; COLOR: #1f497d'>-GetCmdEntry and NVMeCompleteCmd both access the FreeQList. With queue sharing, these could run simultaneously and have a collision when they update the list. </SPAN><o:p></o:p></P>
<P class=MsoNormal style="mso-margin-top-alt: auto; mso-margin-bottom-alt: auto"><SPAN style='FONT-SIZE: 11pt; FONT-FAMILY: "Calibri",sans-serif; COLOR: #1f497d'>-IssueCmd. With queue sharing, different cpu's could attempt to put entries on the queue simultaneously. But, to make RecoveryDpcRoutine work, this must be protected by the dpcLock whether or not there is queue sharing. </SPAN><o:p></o:p></P>
<P class=MsoNormal style="mso-margin-top-alt: auto; mso-margin-bottom-alt: auto"><SPAN style='FONT-SIZE: 11pt; FONT-FAMILY: "Calibri",sans-serif; COLOR: #1f497d'> </SPAN><o:p></o:p></P>
<P class=MsoNormal style="mso-margin-top-alt: auto; mso-margin-bottom-alt: auto"><SPAN style='FONT-SIZE: 11pt; FONT-FAMILY: "Calibri",sans-serif; COLOR: #1f497d'>I think this will address the deadlock problem I reported also.</SPAN><o:p></o:p></P>
<P class=MsoNormal style="mso-margin-top-alt: auto; mso-margin-bottom-alt: auto"><SPAN style='FONT-SIZE: 11pt; FONT-FAMILY: "Calibri",sans-serif; COLOR: #1f497d'> </SPAN><o:p></o:p></P>
<P class=MsoNormal style="mso-margin-top-alt: auto; mso-margin-bottom-alt: auto"><SPAN style='FONT-SIZE: 11pt; FONT-FAMILY: "Calibri",sans-serif; COLOR: #1f497d'>Regards,</SPAN><o:p></o:p></P>
<P class=MsoNormal style="mso-margin-top-alt: auto; mso-margin-bottom-alt: auto"><B><SPAN style='FONT-SIZE: 10pt; FONT-FAMILY: "Arial",sans-serif; COLOR: #1f497d'>Tom Freeman</SPAN></B><o:p></o:p></P>
<P class=MsoNormal style="mso-margin-top-alt: auto; mso-margin-bottom-alt: auto"><B><SPAN style='FONT-SIZE: 10pt; FONT-FAMILY: "Arial",sans-serif; COLOR: #1f497d'>Software Engineer, Device Manager and Driver Development</SPAN></B><o:p></o:p></P>
<P class=MsoNormal style="mso-margin-top-alt: auto; mso-margin-bottom-alt: auto"><SPAN style='FONT-SIZE: 10pt; FONT-FAMILY: "Arial",sans-serif; COLOR: #1f497d'>Western Digital Corporation</SPAN><o:p></o:p></P>
<P class=MsoNormal style="mso-margin-top-alt: auto; mso-margin-bottom-alt: auto"><SPAN style='FONT-SIZE: 10pt; FONT-FAMILY: "Arial",sans-serif; COLOR: #1f497d'>e. <A href="mailto:Thomas.freeman@hgst.com" target=_blank>Thomas.freeman@hgst.com</A></SPAN><o:p></o:p></P>
<P class=MsoNormal style="mso-margin-top-alt: auto; mso-margin-bottom-alt: auto"><SPAN style='FONT-SIZE: 10pt; FONT-FAMILY: "Arial",sans-serif; COLOR: #1f497d'>o. <A href="tel:%2B1-507-322-2311" target=_blank>+1-507-322-2311</A></SPAN><o:p></o:p></P>
<P class=MsoNormal style="mso-margin-top-alt: auto; mso-margin-bottom-alt: auto"><SPAN style='FONT-SIZE: 11pt; FONT-FAMILY: "Calibri",sans-serif; COLOR: #1f497d'> </SPAN><o:p></o:p></P>
<P class=MsoNormal style="mso-margin-top-alt: auto; mso-margin-bottom-alt: auto"><SPAN style='FONT-SIZE: 11pt; FONT-FAMILY: "Calibri",sans-serif; COLOR: #1f497d'><IMG id=_x0000_i1025 border=0 src="cid:DPBBX1FI5HHV@namo.co.kr" width=150 height=47></SPAN><o:p></o:p></P>
<P class=MsoNormal style="mso-margin-top-alt: auto; mso-margin-bottom-alt: auto"><SPAN style='FONT-SIZE: 11pt; FONT-FAMILY: "Calibri",sans-serif; COLOR: #1f497d'> </SPAN><o:p></o:p></P>
<P class=MsoNormal style="mso-margin-top-alt: auto; mso-margin-bottom-alt: auto"><B><SPAN style='FONT-SIZE: 11pt; FONT-FAMILY: "Calibri",sans-serif'>From:</SPAN></B><SPAN style='FONT-SIZE: 11pt; FONT-FAMILY: "Calibri",sans-serif'> Iuliu Rus [<A href="mailto:iuliur@google.com" target=_blank>mailto:iuliur@google.com</A>] <BR><B>Sent:</B> Thursday, June 16, 2016 2:42 PM<BR><B>To:</B> Thomas Freeman <<A href="mailto:thomas.freeman@hgst.com" target=_blank>thomas.freeman@hgst.com</A>><BR><B>Cc:</B> Robles, Raymond C <<A href="mailto:raymond.c.robles@intel.com" target=_blank>raymond.c.robles@intel.com</A>>; nvmewin <<A href="mailto:nvmewin@lists.openfabrics.org" target=_blank>nvmewin@lists.openfabrics.org</A>><BR><B>Subject:</B> Re: [nvmewin] review comments for patch from Google</SPAN><o:p></o:p></P>
<P class=MsoNormal style="mso-margin-top-alt: auto; mso-margin-bottom-alt: auto"> <o:p></o:p></P>
<DIV>
<P class=MsoNormal style="mso-margin-top-alt: auto; mso-margin-bottom-alt: auto">Here's my revised patch for concurrent channels. The synchronization rules are now as follows:<o:p></o:p></P>
<DIV>
<P class=MsoNormal style="mso-margin-top-alt: auto; mso-margin-bottom-alt: auto">1. Init is single threaded so it doesn't matter if acquirelock is true or false<o:p></o:p></P></DIV>
<DIV>
<P class=MsoNormal style="mso-margin-top-alt: auto; mso-margin-bottom-alt: auto">2. Anything called from the DPC has to call ProcessIo with false, since the dpc spinlock is already acquired.<o:p></o:p></P></DIV>
<DIV>
<P class=MsoNormal style="mso-margin-top-alt: auto; mso-margin-bottom-alt: auto">3. Anything called from StartIo has to call ProcessIo with true, since now the startio is not automatically acquired.<o:p></o:p></P></DIV>
<DIV>
<P class=MsoNormal style="mso-margin-top-alt: auto; mso-margin-bottom-alt: auto"> <o:p></o:p></P></DIV>
<DIV>
<P class=MsoNormal style="mso-margin-top-alt: auto; mso-margin-bottom-alt: auto">Thomas, i think the bug that you found is because the ProcessIo was incorrectly called with TRUE instead of FALSE.<o:p></o:p></P></DIV>
<DIV>
<P class=MsoNormal style="mso-margin-top-alt: auto; mso-margin-bottom-alt: auto">Let me know what are your thoughts on this.<o:p></o:p></P></DIV></DIV>
<DIV>
<P class=MsoNormal style="mso-margin-top-alt: auto; mso-margin-bottom-alt: auto"> <o:p></o:p></P>
<DIV>
<P class=MsoNormal style="mso-margin-top-alt: auto; mso-margin-bottom-alt: auto">On Thu, Jun 16, 2016 at 10:41 AM, Iuliu Rus <<A href="mailto:iuliur@google.com" target=_blank>iuliur@google.com</A>> wrote:<o:p></o:p></P>
<BLOCKQUOTE style="BORDER-TOP: medium none; BORDER-RIGHT: medium none; BORDER-BOTTOM: medium none; PADDING-BOTTOM: 0in; PADDING-TOP: 0in; PADDING-LEFT: 6pt; MARGIN: 5pt 0in 5pt 4.8pt; BORDER-LEFT: #cccccc 1pt solid; PADDING-RIGHT: 0in">
<DIV>
<P class=MsoNormal style="mso-margin-top-alt: auto; mso-margin-bottom-alt: auto">Guys, am i OK continuing with the patch on the current code base or do you want me to wait until you pull in some more changes?<o:p></o:p></P></DIV>
<DIV>
<P class=MsoNormal style="mso-margin-top-alt: auto; mso-margin-bottom-alt: auto"> <o:p></o:p></P>
<DIV>
<DIV>
<DIV>
<P class=MsoNormal style="mso-margin-top-alt: auto; mso-margin-bottom-alt: auto">On Wed, Jun 15, 2016 at 8:37 AM, Thomas Freeman <<A href="mailto:thomas.freeman@hgst.com" target=_blank>thomas.freeman@hgst.com</A>> wrote:<o:p></o:p></P></DIV></DIV>
<BLOCKQUOTE style="BORDER-TOP: medium none; BORDER-RIGHT: medium none; BORDER-BOTTOM: medium none; PADDING-BOTTOM: 0in; PADDING-TOP: 0in; PADDING-LEFT: 6pt; MARGIN: 5pt 0in 5pt 4.8pt; BORDER-LEFT: #cccccc 1pt solid; PADDING-RIGHT: 0in">
<DIV>
<DIV>
<DIV>
<DIV>
<P class=MsoNormal style="mso-margin-top-alt: auto; mso-margin-bottom-alt: auto"><SPAN style='FONT-SIZE: 11pt; FONT-FAMILY: "Calibri",sans-serif; COLOR: #1f497d'>Ray,</SPAN><o:p></o:p></P>
<P class=MsoNormal style="mso-margin-top-alt: auto; mso-margin-bottom-alt: auto"><SPAN style='FONT-SIZE: 11pt; FONT-FAMILY: "Calibri",sans-serif; COLOR: #1f497d'>It appears that the latest patch file from Iuliu is applied to revision 137. I don’t see Suman’s changes in that revision.</SPAN><o:p></o:p></P>
<P class=MsoNormal style="mso-margin-top-alt: auto; mso-margin-bottom-alt: auto"><SPAN style='FONT-SIZE: 11pt; FONT-FAMILY: "Calibri",sans-serif; COLOR: #1f497d'> </SPAN><o:p></o:p></P>
<P class=MsoNormal style="mso-margin-top-alt: auto; mso-margin-bottom-alt: auto"><SPAN style='FONT-SIZE: 11pt; FONT-FAMILY: "Calibri",sans-serif; COLOR: #1f497d'>As an example, the use of StorPortRequestTimer is not in nvmeStat.c.</SPAN><o:p></o:p></P>
<P class=MsoNormal style="mso-margin-top-alt: auto; mso-margin-bottom-alt: auto"><SPAN style='FONT-SIZE: 11pt; FONT-FAMILY: "Calibri",sans-serif; COLOR: #1f497d'> </SPAN><o:p></o:p></P>
<P class=MsoNormal style="mso-margin-top-alt: auto; mso-margin-bottom-alt: auto"><SPAN style='FONT-SIZE: 11pt; FONT-FAMILY: "Calibri",sans-serif; COLOR: #1f497d'>I thought there were some newer revisions than 137, but 137 is the latest I’m able to download.</SPAN><o:p></o:p></P>
<P class=MsoNormal style="mso-margin-top-alt: auto; mso-margin-bottom-alt: auto"><SPAN style='FONT-SIZE: 11pt; FONT-FAMILY: "Calibri",sans-serif; COLOR: #1f497d'>137 is also the revision that appears when I go to <A href="https://svn.openfabrics.org/svnrepo/nvmewin/" target=_blank>https://svn.openfabrics.org/svnrepo/nvmewin/</A></SPAN><o:p></o:p></P>
<P class=MsoNormal style="mso-margin-top-alt: auto; mso-margin-bottom-alt: auto"><SPAN style='FONT-SIZE: 11pt; FONT-FAMILY: "Calibri",sans-serif; COLOR: #1f497d'> </SPAN><o:p></o:p></P>
<P class=MsoNormal style="mso-margin-top-alt: auto; mso-margin-bottom-alt: auto"><SPAN style='FONT-SIZE: 11pt; FONT-FAMILY: "Calibri",sans-serif; COLOR: #1f497d'>SVN is giving me some troubles, so I can’t tell for sure if this is my environment or a server issue.</SPAN><o:p></o:p></P>
<P class=MsoNormal style="mso-margin-top-alt: auto; mso-margin-bottom-alt: auto"><SPAN style='FONT-SIZE: 11pt; FONT-FAMILY: "Calibri",sans-serif; COLOR: #1f497d'> </SPAN><o:p></o:p></P>
<DIV>
<P class=MsoNormal style="mso-margin-top-alt: auto; mso-margin-bottom-alt: auto"><B><SPAN style='FONT-SIZE: 10pt; FONT-FAMILY: "Arial",sans-serif; COLOR: #1f497d'>Tom Freeman</SPAN></B><o:p></o:p></P>
<P class=MsoNormal style="mso-margin-top-alt: auto; mso-margin-bottom-alt: auto"><B><SPAN style='FONT-SIZE: 10pt; FONT-FAMILY: "Arial",sans-serif; COLOR: #1f497d'>Software Engineer, Device Manager and Driver Development</SPAN></B><o:p></o:p></P>
<P class=MsoNormal style="mso-margin-top-alt: auto; mso-margin-bottom-alt: auto"><SPAN style='FONT-SIZE: 10pt; FONT-FAMILY: "Arial",sans-serif; COLOR: #1f497d'>Western Digital Corporation</SPAN><o:p></o:p></P>
<P class=MsoNormal style="mso-margin-top-alt: auto; mso-margin-bottom-alt: auto"><SPAN style='FONT-SIZE: 10pt; FONT-FAMILY: "Arial",sans-serif; COLOR: #1f497d'>e. <A href="mailto:Thomas.freeman@hgst.com" target=_blank>Thomas.freeman@hgst.com</A></SPAN><o:p></o:p></P>
<P class=MsoNormal style="mso-margin-top-alt: auto; mso-margin-bottom-alt: auto"><SPAN style='FONT-SIZE: 10pt; FONT-FAMILY: "Arial",sans-serif; COLOR: #1f497d'>o. <A href="tel:%2B1-507-322-2311" target=_blank>+1-507-322-2311</A></SPAN><o:p></o:p></P>
<P class=MsoNormal style="mso-margin-top-alt: auto; mso-margin-bottom-alt: auto"><SPAN style='FONT-SIZE: 11pt; FONT-FAMILY: "Calibri",sans-serif; COLOR: #1f497d'> </SPAN><o:p></o:p></P>
<P class=MsoNormal style="mso-margin-top-alt: auto; mso-margin-bottom-alt: auto"><SPAN style='FONT-SIZE: 11pt; FONT-FAMILY: "Calibri",sans-serif; COLOR: #1f497d'><IMG id=_x0000_i1026 border=0 src="cid:ESYT2XYNX48N@namo.co.kr" width=150 height=47></SPAN><o:p></o:p></P></DIV>
<P class=MsoNormal style="mso-margin-top-alt: auto; mso-margin-bottom-alt: auto"><SPAN style='FONT-SIZE: 11pt; FONT-FAMILY: "Calibri",sans-serif; COLOR: #1f497d'> </SPAN><o:p></o:p></P>
<DIV>
<DIV style="BORDER-TOP: #e1e1e1 1pt solid; BORDER-RIGHT: medium none; BORDER-BOTTOM: medium none; PADDING-BOTTOM: 0in; PADDING-TOP: 3pt; PADDING-LEFT: 0in; BORDER-LEFT: medium none; PADDING-RIGHT: 0in">
<P class=MsoNormal style="mso-margin-top-alt: auto; mso-margin-bottom-alt: auto"><B><SPAN style='FONT-SIZE: 11pt; FONT-FAMILY: "Calibri",sans-serif'>From:</SPAN></B><SPAN style='FONT-SIZE: 11pt; FONT-FAMILY: "Calibri",sans-serif'> nvmewin [mailto:<A href="mailto:nvmewin-bounces@lists.openfabrics.org" target=_blank>nvmewin-bounces@lists.openfabrics.org</A>] <B>On Behalf Of </B>Robles, Raymond C<BR><B>Sent:</B> Friday, June 10, 2016 12:23 PM<BR><B>To:</B> Iuliu Rus <<A href="mailto:iuliur@google.com" target=_blank>iuliur@google.com</A>>; <A href="mailto:suman.p@samsung.com" target=_blank>suman.p@samsung.com</A><BR><B>Cc:</B> nvmewin <<A href="mailto:nvmewin@lists.openfabrics.org" target=_blank>nvmewin@lists.openfabrics.org</A>></SPAN><o:p></o:p></P>
<DIV>
<DIV>
<P class=MsoNormal style="mso-margin-top-alt: auto; mso-margin-bottom-alt: auto"><BR><B>Subject:</B> Re: [nvmewin] review comments for patch from Google<o:p></o:p></P></DIV></DIV></DIV></DIV>
<DIV>
<DIV>
<P class=MsoNormal style="mso-margin-top-alt: auto; mso-margin-bottom-alt: auto"> <o:p></o:p></P>
<P class=MsoNormal style="mso-margin-top-alt: auto; mso-margin-bottom-alt: auto"><SPAN style='FONT-SIZE: 11pt; FONT-FAMILY: "Calibri",sans-serif; COLOR: #002060'>Hi Iuliu,</SPAN><o:p></o:p></P>
<P class=MsoNormal style="mso-margin-top-alt: auto; mso-margin-bottom-alt: auto"><SPAN style='FONT-SIZE: 11pt; FONT-FAMILY: "Calibri",sans-serif; COLOR: #002060'> </SPAN><o:p></o:p></P>
<P class=MsoNormal style="mso-margin-top-alt: auto; mso-margin-bottom-alt: auto"><SPAN style='FONT-SIZE: 11pt; FONT-FAMILY: "Calibri",sans-serif; COLOR: #002060'>If you send out the updated patch with the protection in place for both list entries and doorbells, I can run it on an Intel NVMe SSD (P3700) and see if there is any performance impact (up or down).</SPAN><o:p></o:p></P>
<P class=MsoNormal style="mso-margin-top-alt: auto; mso-margin-bottom-alt: auto"><SPAN style='FONT-SIZE: 11pt; FONT-FAMILY: "Calibri",sans-serif; COLOR: #002060'> </SPAN><o:p></o:p></P>
<P class=MsoNormal style="mso-margin-top-alt: auto; mso-margin-bottom-alt: auto"><SPAN style='FONT-SIZE: 11pt; FONT-FAMILY: "Calibri",sans-serif; COLOR: #002060'>Thank you…</SPAN><o:p></o:p></P>
<P class=MsoNormal style="mso-margin-top-alt: auto; mso-margin-bottom-alt: auto"><SPAN style='FONT-SIZE: 11pt; FONT-FAMILY: "Calibri",sans-serif; COLOR: #002060'>Ray</SPAN><o:p></o:p></P>
<P class=MsoNormal style="mso-margin-top-alt: auto; mso-margin-bottom-alt: auto"><SPAN style='FONT-SIZE: 11pt; FONT-FAMILY: "Calibri",sans-serif; COLOR: #002060'> </SPAN><o:p></o:p></P>
<P class=MsoNormal style="mso-margin-top-alt: auto; mso-margin-bottom-alt: auto"><B><SPAN style='FONT-SIZE: 11pt; FONT-FAMILY: "Calibri",sans-serif'>From:</SPAN></B><SPAN style='FONT-SIZE: 11pt; FONT-FAMILY: "Calibri",sans-serif'> Iuliu Rus [</SPAN><A href="mailto:iuliur@google.com" target=_blank><SPAN style='FONT-SIZE: 11pt; FONT-FAMILY: "Calibri",sans-serif'>mailto:iuliur@google.com</SPAN></A><SPAN style='FONT-SIZE: 11pt; FONT-FAMILY: "Calibri",sans-serif'>] <BR><B>Sent:</B> Friday, June 10, 2016 10:17 AM<BR><B>To:</B> </SPAN><A href="mailto:suman.p@samsung.com" target=_blank><SPAN style='FONT-SIZE: 11pt; FONT-FAMILY: "Calibri",sans-serif'>suman.p@samsung.com</SPAN></A><SPAN style='FONT-SIZE: 11pt; FONT-FAMILY: "Calibri",sans-serif'><BR><B>Cc:</B> Robles, Raymond C <</SPAN><A href="mailto:raymond.c.robles@intel.com" target=_blank><SPAN style='FONT-SIZE: 11pt; FONT-FAMILY: "Calibri",sans-serif'>raymond.c.robles@intel.com</SPAN></A><SPAN style='FONT-SIZE: 11pt; FONT-FAMILY: "Calibri",sans-serif'>>; Judy Brock <</SPAN><A href="mailto:judy.brock@samsung.com" target=_blank><SPAN style='FONT-SIZE: 11pt; FONT-FAMILY: "Calibri",sans-serif'>judy.brock@samsung.com</SPAN></A><SPAN style='FONT-SIZE: 11pt; FONT-FAMILY: "Calibri",sans-serif'>>; nvmewin <</SPAN><A href="mailto:nvmewin@lists.openfabrics.org" target=_blank><SPAN style='FONT-SIZE: 11pt; FONT-FAMILY: "Calibri",sans-serif'>nvmewin@lists.openfabrics.org</SPAN></A><SPAN style='FONT-SIZE: 11pt; FONT-FAMILY: "Calibri",sans-serif'>><BR><B>Subject:</B> Re: RE: review comments for patch from Google</SPAN><o:p></o:p></P>
<P class=MsoNormal style="mso-margin-top-alt: auto; mso-margin-bottom-alt: auto"> <o:p></o:p></P>
<DIV>
<P class=MsoNormal style="mso-margin-top-alt: auto; mso-margin-bottom-alt: auto">we are running this in a virtual machine, and we do see a perf difference with concurrent channels. We did not observe any performance drop with synchronized access to the list entry.<o:p></o:p></P></DIV></DIV></DIV>
<DIV>
<P class=MsoNormal style="mso-margin-top-alt: auto; mso-margin-bottom-alt: auto"> <o:p></o:p></P>
<DIV>
<DIV>
<DIV>
<P class=MsoNormal style="mso-margin-top-alt: auto; mso-margin-bottom-alt: auto">On Fri, Jun 10, 2016 at 7:15 AM, SUMAN PRAKASH B <<A href="mailto:suman.p@samsung.com" target=_blank>suman.p@samsung.com</A>> wrote:<o:p></o:p></P></DIV></DIV>
<BLOCKQUOTE style="BORDER-TOP: medium none; BORDER-RIGHT: medium none; BORDER-BOTTOM: medium none; PADDING-BOTTOM: 0in; PADDING-TOP: 0in; PADDING-LEFT: 6pt; MARGIN: 5pt 0in 5pt 4.8pt; BORDER-LEFT: #cccccc 1pt solid; PADDING-RIGHT: 0in">
<DIV>
<DIV>
<DIV>
<P><SPAN style='FONT-SIZE: 11pt; FONT-FAMILY: "Calibri",sans-serif'>Hi Iuliu,</SPAN><o:p></o:p></P>
<P> <o:p></o:p></P>
<P><SPAN style='FONT-SIZE: 11pt; FONT-FAMILY: "Calibri",sans-serif'>As you have the implementation in place, can you let us know if there is any performance improvement when STOR_PERF_CONCURRENT_CHANNELS is set in case of #core and #queues not matching.</SPAN><o:p></o:p></P>
<P><SPAN style='FONT-SIZE: 11pt; FONT-FAMILY: "Calibri",sans-serif'>Also, if there is any drop in performance if the list entry and doorbell is protected after setting STOR_PERF_CONCURRENT_CHANNELS.</SPAN><o:p></o:p></P>
<P> <o:p></o:p></P>
<P><SPAN style='FONT-SIZE: 11pt; FONT-FAMILY: "Calibri",sans-serif'>Perhaps a bit off topic:</SPAN><o:p></o:p></P>
<P><SPAN style='FONT-SIZE: 11pt; FONT-FAMILY: "Calibri",sans-serif'>Even in case when #cores and #queues are same and when STOR_PERF_CONCURRENT_CHANNELS is set, we did not observe any performance gain. Maybe we are hitting the max performance from device even without setting STOR_PERF_CONCURRENT_CHANNELS. Can you please let us know your observation?</SPAN><o:p></o:p></P>
<P> <o:p></o:p></P>
<P><SPAN style='FONT-SIZE: 11pt; FONT-FAMILY: "Calibri",sans-serif'>Thanks,</SPAN><o:p></o:p></P>
<P><SPAN style='FONT-SIZE: 11pt; FONT-FAMILY: "Calibri",sans-serif'>Suman</SPAN><o:p></o:p></P>
<P> <o:p></o:p></P>
<P>------- <B>Original Message</B> -------<o:p></o:p></P>
<P><B>Sender</B> : Robles, Raymond C<<A href="mailto:raymond.c.robles@intel.com" target=_blank>raymond.c.robles@intel.com</A>><o:p></o:p></P>
<P><B>Date</B> : Jun 10, 2016 08:30 (GMT+05:30)<o:p></o:p></P>
<P><B>Title</B> : RE: review comments for patch from Google<o:p></o:p></P></DIV></DIV>
<DIV>
<DIV>
<P> <o:p></o:p></P>
<DIV>
<DIV>
<DIV>
<P class=MsoNormal style="mso-margin-top-alt: auto; mso-margin-bottom-alt: auto"><SPAN style='FONT-SIZE: 11pt; FONT-FAMILY: "Calibri",sans-serif; COLOR: #002060'>Hi Judy, </SPAN><o:p></o:p></P>
<P class=MsoNormal style="mso-margin-top-alt: auto; mso-margin-bottom-alt: auto"><SPAN style='FONT-SIZE: 11pt; FONT-FAMILY: "Calibri",sans-serif; COLOR: #002060'> </SPAN><o:p></o:p></P>
<P class=MsoNormal style="mso-margin-top-alt: auto; mso-margin-bottom-alt: auto"><SPAN style='FONT-SIZE: 11pt; FONT-FAMILY: "Calibri",sans-serif; COLOR: #002060'>Thanks for the additional info. I agree with the assessment. My statement was directed at the general direction of the solution… and the fact that I think option 1 is not as robust as option 2.</SPAN><o:p></o:p></P>
<P class=MsoNormal style="mso-margin-top-alt: auto; mso-margin-bottom-alt: auto"><SPAN style='FONT-SIZE: 11pt; FONT-FAMILY: "Calibri",sans-serif; COLOR: #002060'> </SPAN><o:p></o:p></P>
<P class=MsoNormal style="mso-margin-top-alt: auto; mso-margin-bottom-alt: auto"><SPAN style='FONT-SIZE: 11pt; FONT-FAMILY: "Calibri",sans-serif; COLOR: #002060'>Thank you…</SPAN><o:p></o:p></P>
<P class=MsoNormal style="mso-margin-top-alt: auto; mso-margin-bottom-alt: auto"><SPAN style='FONT-SIZE: 11pt; FONT-FAMILY: "Calibri",sans-serif; COLOR: #002060'>Ray</SPAN><o:p></o:p></P>
<P class=MsoNormal style="mso-margin-top-alt: auto; mso-margin-bottom-alt: auto"><A name=m_-8021989622238516106_m_-72961863198882><SPAN style='FONT-SIZE: 11pt; FONT-FAMILY: "Calibri",sans-serif; COLOR: #002060'> </SPAN></A><o:p></o:p></P>
<DIV>
<DIV style="BORDER-TOP: #e1e1e1 1pt solid; BORDER-RIGHT: medium none; BORDER-BOTTOM: medium none; PADDING-BOTTOM: 0in; PADDING-TOP: 3pt; PADDING-LEFT: 0in; BORDER-LEFT: medium none; PADDING-RIGHT: 0in">
<P class=MsoNormal style="mso-margin-top-alt: auto; mso-margin-bottom-alt: auto"><B><SPAN style='FONT-SIZE: 11pt; FONT-FAMILY: "Calibri",sans-serif'>From:</SPAN></B><SPAN style='FONT-SIZE: 11pt; FONT-FAMILY: "Calibri",sans-serif'> Judy Brock [mailto:</SPAN><A href="mailto:judy.brock@samsung.com" target=_blank><SPAN style='FONT-SIZE: 11pt; FONT-FAMILY: "Calibri",sans-serif'>judy.brock@samsung.com</SPAN></A><SPAN style='FONT-SIZE: 11pt; FONT-FAMILY: "Calibri",sans-serif'>] <BR><B>Sent:</B> Thursday, June 09, 2016 5:14 PM<BR><B>To:</B> Robles, Raymond C <</SPAN><A href="mailto:raymond.c.robles@intel.com" target=_blank><SPAN style='FONT-SIZE: 11pt; FONT-FAMILY: "Calibri",sans-serif'>raymond.c.robles@intel.com</SPAN></A><SPAN style='FONT-SIZE: 11pt; FONT-FAMILY: "Calibri",sans-serif'>>; Iuliu Rus <</SPAN><A href="mailto:iuliur@google.com" target=_blank><SPAN style='FONT-SIZE: 11pt; FONT-FAMILY: "Calibri",sans-serif'>iuliur@google.com</SPAN></A><SPAN style='FONT-SIZE: 11pt; FONT-FAMILY: "Calibri",sans-serif'>>; </SPAN><A href="mailto:suman.p@samsung.com" target=_blank><SPAN style='FONT-SIZE: 11pt; FONT-FAMILY: "Calibri",sans-serif'>suman.p@samsung.com</SPAN></A><SPAN style='FONT-SIZE: 11pt; FONT-FAMILY: "Calibri",sans-serif'><BR><B>Cc:</B> </SPAN><A href="mailto:nvmewin@lists.openfabrics.org" target=_blank><SPAN style='FONT-SIZE: 11pt; FONT-FAMILY: "Calibri",sans-serif'>nvmewin@lists.openfabrics.org</SPAN></A><SPAN style='FONT-SIZE: 11pt; FONT-FAMILY: "Calibri",sans-serif'><BR><B>Subject:</B> RE: review comments for patch from Google</SPAN><o:p></o:p></P></DIV></DIV>
<P class=MsoNormal style="mso-margin-top-alt: auto; mso-margin-bottom-alt: auto"> <o:p></o:p></P>
<P class=MsoNormal style="mso-margin-top-alt: auto; mso-margin-bottom-alt: auto"><SPAN style='FONT-SIZE: 11pt; FONT-FAMILY: "Calibri",sans-serif; COLOR: #1f497d'>Hi Ray et al,</SPAN><o:p></o:p></P>
<P class=MsoNormal style="MARGIN-LEFT: 0.5in; TEXT-INDENT: 0.5in; mso-margin-top-alt: auto; mso-margin-bottom-alt: auto"><SPAN style='FONT-SIZE: 11pt; FONT-FAMILY: "Calibri",sans-serif'>[Suman wrote] >> Protect the listentry and doorbell before access/update</SPAN><o:p></o:p></P>
<P class=MsoNormal style="MARGIN-LEFT: 0.5in; TEXT-INDENT: 0.5in; mso-margin-top-alt: auto; mso-margin-bottom-alt: auto"><SPAN style='FONT-SIZE: 11pt; FONT-FAMILY: "Calibri",sans-serif'> >>Do not set STOR_PERF_CONCURRENT_CHANNELS when number of cores and queues are not same.</SPAN><o:p></o:p></P>
<P class=MsoNormal style="MARGIN-LEFT: 0.5in; TEXT-INDENT: 0.5in; mso-margin-top-alt: auto; mso-margin-bottom-alt: auto"><SPAN style='FONT-SIZE: 11pt; FONT-FAMILY: "Calibri",sans-serif; COLOR: #002060'> </SPAN><o:p></o:p></P>
<P class=MsoNormal style="MARGIN-LEFT: 0.5in; TEXT-INDENT: 0.5in; mso-margin-top-alt: auto; mso-margin-bottom-alt: auto"><SPAN style='FONT-SIZE: 11pt; FONT-FAMILY: "Calibri",sans-serif; COLOR: #002060'>Ray wrote] >>In the case Suman points out, concurrent channels should still be enabled even if the number of cores and queues do not match. This is especially true for large core count systems (>128 logical cpu cores) but devices may only support 32 queues or less</SPAN><o:p></o:p></P>
<P class=MsoNormal style="mso-margin-top-alt: auto; mso-margin-bottom-alt: auto"><SPAN style='FONT-SIZE: 11pt; FONT-FAMILY: "Calibri",sans-serif; COLOR: #1f497d'> </SPAN><o:p></o:p></P>
<P class=MsoNormal style="mso-margin-top-alt: auto; mso-margin-bottom-alt: auto"><SPAN style='FONT-SIZE: 11pt; FONT-FAMILY: "Calibri",sans-serif; COLOR: #1f497d'>I think Suman was saying, we have two options: </SPAN><o:p></o:p></P>
<P><SPAN style='FONT-SIZE: 11pt; FONT-FAMILY: "Calibri",sans-serif; COLOR: #1f497d'>1.</SPAN><SPAN style="FONT-SIZE: 7pt; COLOR: #1f497d"> </SPAN><SPAN style='FONT-SIZE: 11pt; FONT-FAMILY: "Calibri",sans-serif; COLOR: #1f497d'>either don’t enable it at all if # cores != #queues or </SPAN><o:p></o:p></P>
<P><SPAN style='FONT-SIZE: 11pt; FONT-FAMILY: "Calibri",sans-serif; COLOR: #1f497d'>2.</SPAN><SPAN style="FONT-SIZE: 7pt; COLOR: #1f497d"> </SPAN><SPAN style='FONT-SIZE: 11pt; FONT-FAMILY: "Calibri",sans-serif; COLOR: #1f497d'>protect both the list entries and doorbell writes before any accesses/updates if #cores != #queues. </SPAN><o:p></o:p></P>
<P class=MsoNormal style="mso-margin-top-alt: auto; mso-margin-bottom-alt: auto"><SPAN style='FONT-SIZE: 11pt; FONT-FAMILY: "Calibri",sans-serif; COLOR: #1f497d'> </SPAN><o:p></o:p></P>
<P class=MsoNormal style="mso-margin-top-alt: auto; mso-margin-bottom-alt: auto"><SPAN style='FONT-SIZE: 11pt; FONT-FAMILY: "Calibri",sans-serif; COLOR: #1f497d'>So we will need to check to see if we need to use locks or not in the performance path based on whether #cores and # queues DO match or not if we want to enable concurrent channels no matter what. It’s probably better than option 1) above if this is a configuration that can come up frequently.</SPAN><o:p></o:p></P>
<P class=MsoNormal style="mso-margin-top-alt: auto; mso-margin-bottom-alt: auto"><SPAN style='FONT-SIZE: 11pt; FONT-FAMILY: "Calibri",sans-serif; COLOR: #1f497d'> </SPAN><o:p></o:p></P>
<P class=MsoNormal style="mso-margin-top-alt: auto; mso-margin-bottom-alt: auto"><SPAN style='FONT-SIZE: 11pt; FONT-FAMILY: "Calibri",sans-serif; COLOR: #1f497d'>Thanks,</SPAN><o:p></o:p></P>
<P class=MsoNormal style="mso-margin-top-alt: auto; mso-margin-bottom-alt: auto"><SPAN style='FONT-SIZE: 11pt; FONT-FAMILY: "Calibri",sans-serif; COLOR: #1f497d'>Judy</SPAN><o:p></o:p></P>
<P class=MsoNormal style="mso-margin-top-alt: auto; mso-margin-bottom-alt: auto"><SPAN style='FONT-SIZE: 11pt; FONT-FAMILY: "Calibri",sans-serif; COLOR: #1f497d'> </SPAN><o:p></o:p></P>
<P class=MsoNormal style="mso-margin-top-alt: auto; mso-margin-bottom-alt: auto"><SPAN style='FONT-SIZE: 11pt; FONT-FAMILY: "Calibri",sans-serif; COLOR: #1f497d'> </SPAN><o:p></o:p></P>
<DIV>
<DIV style="BORDER-TOP: #b5c4df 1pt solid; BORDER-RIGHT: medium none; BORDER-BOTTOM: medium none; PADDING-BOTTOM: 0in; PADDING-TOP: 3pt; PADDING-LEFT: 0in; BORDER-LEFT: medium none; PADDING-RIGHT: 0in">
<P class=MsoNormal style="mso-margin-top-alt: auto; mso-margin-bottom-alt: auto"><B><SPAN style='FONT-SIZE: 10pt; FONT-FAMILY: "Tahoma",sans-serif'>From:</SPAN></B><SPAN style='FONT-SIZE: 10pt; FONT-FAMILY: "Tahoma",sans-serif'> nvmewin [</SPAN><A href="mailto:nvmewin-bounces@lists.openfabrics.org" target=_blank><SPAN style='FONT-SIZE: 10pt; FONT-FAMILY: "Tahoma",sans-serif'>mailto:nvmewin-bounces@lists.openfabrics.org</SPAN></A><SPAN style='FONT-SIZE: 10pt; FONT-FAMILY: "Tahoma",sans-serif'>] <B>On Behalf Of </B>Robles, Raymond C<BR><B>Sent:</B> Thursday, June 09, 2016 2:32 PM<BR><B>To:</B> Iuliu Rus; </SPAN><A href="mailto:suman.p@samsung.com" target=_blank><SPAN style='FONT-SIZE: 10pt; FONT-FAMILY: "Tahoma",sans-serif'>suman.p@samsung.com</SPAN></A><SPAN style='FONT-SIZE: 10pt; FONT-FAMILY: "Tahoma",sans-serif'><BR><B>Cc:</B> </SPAN><A href="mailto:nvmewin@lists.openfabrics.org" target=_blank><SPAN style='FONT-SIZE: 10pt; FONT-FAMILY: "Tahoma",sans-serif'>nvmewin@lists.openfabrics.org</SPAN></A><SPAN style='FONT-SIZE: 10pt; FONT-FAMILY: "Tahoma",sans-serif'><BR><B>Subject:</B> Re: [nvmewin] review comments for patch from Google</SPAN><o:p></o:p></P></DIV></DIV>
<P class=MsoNormal style="mso-margin-top-alt: auto; mso-margin-bottom-alt: auto"> <o:p></o:p></P>
<P class=MsoNormal style="mso-margin-top-alt: auto; mso-margin-bottom-alt: auto"><SPAN style='FONT-SIZE: 11pt; FONT-FAMILY: "Calibri",sans-serif; COLOR: #002060'>Hi Iuliu/Suman,</SPAN><o:p></o:p></P>
<P class=MsoNormal style="mso-margin-top-alt: auto; mso-margin-bottom-alt: auto"><SPAN style='FONT-SIZE: 11pt; FONT-FAMILY: "Calibri",sans-serif; COLOR: #002060'> </SPAN><o:p></o:p></P>
<P class=MsoNormal style="mso-margin-top-alt: auto; mso-margin-bottom-alt: auto"><SPAN style='FONT-SIZE: 11pt; FONT-FAMILY: "Calibri",sans-serif; COLOR: #002060'>I’m of the opinion that any solution provided should be as robust as possible. In the case Suman points out, concurrent channels should still be enabled even if the number of cores and queues do not match. This is especially true for large core count systems (>128 logical cpu cores) but devices may only support 32 queues or less. </SPAN><o:p></o:p></P>
<P class=MsoNormal style="mso-margin-top-alt: auto; mso-margin-bottom-alt: auto"><SPAN style='FONT-SIZE: 11pt; FONT-FAMILY: "Calibri",sans-serif; COLOR: #002060'> </SPAN><o:p></o:p></P>
<P class=MsoNormal style="mso-margin-top-alt: auto; mso-margin-bottom-alt: auto"><SPAN style='FONT-SIZE: 11pt; FONT-FAMILY: "Calibri",sans-serif; COLOR: #002060'>Thank you…</SPAN><o:p></o:p></P>
<P class=MsoNormal style="mso-margin-top-alt: auto; mso-margin-bottom-alt: auto"><SPAN style='FONT-SIZE: 11pt; FONT-FAMILY: "Calibri",sans-serif; COLOR: #002060'>Ray</SPAN><o:p></o:p></P>
<P class=MsoNormal style="mso-margin-top-alt: auto; mso-margin-bottom-alt: auto"><SPAN style='FONT-SIZE: 11pt; FONT-FAMILY: "Calibri",sans-serif; COLOR: #002060'> </SPAN><o:p></o:p></P>
<P class=MsoNormal style="mso-margin-top-alt: auto; mso-margin-bottom-alt: auto"><B><SPAN style='FONT-SIZE: 11pt; FONT-FAMILY: "Calibri",sans-serif'>From:</SPAN></B><SPAN style='FONT-SIZE: 11pt; FONT-FAMILY: "Calibri",sans-serif'> Iuliu Rus </SPAN><A href="mailto:[mailto:iuliur@google.com]" target=_blank><SPAN style='FONT-SIZE: 11pt; FONT-FAMILY: "Calibri",sans-serif'>[mailto:iuliur@google.com]</SPAN></A><SPAN style='FONT-SIZE: 11pt; FONT-FAMILY: "Calibri",sans-serif'> <BR><B>Sent:</B> Thursday, June 09, 2016 2:09 PM<BR><B>To:</B> </SPAN><A href="mailto:suman.p@samsung.com" target=_blank><SPAN style='FONT-SIZE: 11pt; FONT-FAMILY: "Calibri",sans-serif'>suman.p@samsung.com</SPAN></A><SPAN style='FONT-SIZE: 11pt; FONT-FAMILY: "Calibri",sans-serif'><BR><B>Cc:</B> Robles, Raymond C <</SPAN><A href="mailto:raymond.c.robles@intel.com" target=_blank><SPAN style='FONT-SIZE: 11pt; FONT-FAMILY: "Calibri",sans-serif'>raymond.c.robles@intel.com</SPAN></A><SPAN style='FONT-SIZE: 11pt; FONT-FAMILY: "Calibri",sans-serif'>>; </SPAN><A href="mailto:nvmewin@lists.openfabrics.org" target=_blank><SPAN style='FONT-SIZE: 11pt; FONT-FAMILY: "Calibri",sans-serif'>nvmewin@lists.openfabrics.org</SPAN></A><SPAN style='FONT-SIZE: 11pt; FONT-FAMILY: "Calibri",sans-serif'><BR><B>Subject:</B> Re: review comments for patch from Google</SPAN><o:p></o:p></P>
<P class=MsoNormal style="mso-margin-top-alt: auto; mso-margin-bottom-alt: auto"> <o:p></o:p></P>
<DIV>
<P class=MsoNormal style="mso-margin-top-alt: auto; mso-margin-bottom-alt: auto">Which one do you prefer? Our google driver has the synchronization in place already for a different reason. Or, I can follow your second suggestion.<o:p></o:p></P></DIV></DIV></DIV>
<DIV>
<P class=MsoNormal style="mso-margin-top-alt: auto; mso-margin-bottom-alt: auto"> <o:p></o:p></P>
<DIV>
<DIV>
<DIV>
<P class=MsoNormal style="mso-margin-top-alt: auto; mso-margin-bottom-alt: auto">On Thu, Jun 9, 2016 at 7:39 AM, SUMAN PRAKASH B <<A href="mailto:suman.p@samsung.com" target=_blank>suman.p@samsung.com</A>> wrote:<o:p></o:p></P></DIV></DIV>
<BLOCKQUOTE style="BORDER-TOP: medium none; BORDER-RIGHT: medium none; BORDER-BOTTOM: medium none; PADDING-BOTTOM: 0in; PADDING-TOP: 0in; PADDING-LEFT: 6pt; MARGIN: 5pt 0in 5pt 4.8pt; BORDER-LEFT: #cccccc 1pt solid; PADDING-RIGHT: 0in">
<DIV>
<DIV>
<DIV>
<P><SPAN style='FONT-SIZE: 11pt; FONT-FAMILY: "Calibri",sans-serif'>Hi Iuliu,</SPAN><o:p></o:p></P>
<P> <o:p></o:p></P>
<P><SPAN style='FONT-SIZE: 11pt; FONT-FAMILY: "Calibri",sans-serif'>Please find my review comments below:</SPAN><o:p></o:p></P>
<P> <o:p></o:p></P>
<P><SPAN style='FONT-SIZE: 11pt; FONT-FAMILY: "Calibri",sans-serif'>1. If the number of Cores and Queues supported by device are not same, then setting STOR_PERF_CONCURRENT_CHANNELS will corrupt the submission queue. For example, if device has 4 queues and connected to target PC which has 8 logical processors, and if STOR_PERF_CONCURRENT_CHANNELS is set, all the 8 logical processors will get startIo calls concurrently. As per OFA driver NUMA design, the IO Queue 1 will be shared by core 0 and core 4. There is possibility that core 0 and core 4 will get startio() calls concurrently and will access/update the listenty and queue 1 doorbell register at same time, which will lead to listentry and doorbell register corruption. <BR>This problem can be address by following ways:<BR>a. Protect the listentry and doorbell before access/update.<BR>b. Do not set STOR_PERF_CONCURRENT_CHANNELS when number of cores and queues are not same.</SPAN><o:p></o:p></P>
<P> <o:p></o:p></P>
<P><SPAN style='FONT-SIZE: 11pt; FONT-FAMILY: "Calibri",sans-serif'>2. We can use pRMT->NumActiveCores which is updated in NVMeEnumNumaCores() instead of using KeQueryActiveProcessorCountEx/KeQueryActiveProcessorCount.</SPAN><o:p></o:p></P>
<P> <o:p></o:p></P>
<P><SPAN style='FONT-SIZE: 11pt; FONT-FAMILY: "Calibri",sans-serif'>Please let us know your opinion.</SPAN><o:p></o:p></P>
<P> <o:p></o:p></P>
<P><SPAN style='FONT-SIZE: 11pt; FONT-FAMILY: "Calibri",sans-serif'>Thanks,<BR>Suman</SPAN><o:p></o:p></P>
<P> <o:p></o:p></P>
<P>------- <B>Original Message</B> -------<o:p></o:p></P>
<P><B>Sender</B> : Robles, Raymond C<<A href="mailto:raymond.c.robles@intel.com" target=_blank>raymond.c.robles@intel.com</A>><o:p></o:p></P>
<P><B>Date</B> : Jun 09, 2016 03:56 (GMT+05:30)<o:p></o:p></P>
<P><B>Title</B> : Re: [nvmewin] Happy New Year... and status update<o:p></o:p></P>
<P> <o:p></o:p></P>
<DIV>
<P class=MsoNormal style="mso-margin-top-alt: auto; mso-margin-bottom-alt: auto"><SPAN style='FONT-SIZE: 11pt; FONT-FAMILY: "Calibri",sans-serif; COLOR: #002060'>Thank you Iuliu!</SPAN><o:p></o:p></P>
<P class=MsoNormal style="mso-margin-top-alt: auto; mso-margin-bottom-alt: auto"><SPAN style='FONT-SIZE: 11pt; FONT-FAMILY: "Calibri",sans-serif; COLOR: #002060'> </SPAN><o:p></o:p></P>
<P class=MsoNormal style="mso-margin-top-alt: auto; mso-margin-bottom-alt: auto"><SPAN style='FONT-SIZE: 11pt; FONT-FAMILY: "Calibri",sans-serif; BACKGROUND: yellow; COLOR: #002060'>Reviewing companies, please try to target the review completion for 2 weeks from today (6/22/16).</SPAN><SPAN style='FONT-SIZE: 11pt; FONT-FAMILY: "Calibri",sans-serif; COLOR: #002060'> </SPAN><o:p></o:p></P>
<P class=MsoNormal style="mso-margin-top-alt: auto; mso-margin-bottom-alt: auto"><SPAN style='FONT-SIZE: 11pt; FONT-FAMILY: "Calibri",sans-serif; COLOR: #002060'> </SPAN><o:p></o:p></P>
<P class=MsoNormal style="mso-margin-top-alt: auto; mso-margin-bottom-alt: auto"><SPAN style='FONT-SIZE: 11pt; FONT-FAMILY: "Calibri",sans-serif; COLOR: #002060'>Thank you!</SPAN><o:p></o:p></P>
<P class=MsoNormal style="mso-margin-top-alt: auto; mso-margin-bottom-alt: auto"><SPAN style='FONT-SIZE: 11pt; FONT-FAMILY: "Calibri",sans-serif; COLOR: #002060'>Ray</SPAN><o:p></o:p></P>
<P class=MsoNormal style="mso-margin-top-alt: auto; mso-margin-bottom-alt: auto"><SPAN style='FONT-SIZE: 11pt; FONT-FAMILY: "Calibri",sans-serif; COLOR: #002060'> </SPAN><o:p></o:p></P>
<P class=MsoNormal style="mso-margin-top-alt: auto; mso-margin-bottom-alt: auto"><B><SPAN style='FONT-SIZE: 11pt; FONT-FAMILY: "Calibri",sans-serif'>From:</SPAN></B><SPAN style='FONT-SIZE: 11pt; FONT-FAMILY: "Calibri",sans-serif'> Iuliu Rus [mailto:</SPAN><A href="mailto:iuliur@google.com" target=_blank><SPAN style='FONT-SIZE: 11pt; FONT-FAMILY: "Calibri",sans-serif'>iuliur@google.com</SPAN></A><SPAN style='FONT-SIZE: 11pt; FONT-FAMILY: "Calibri",sans-serif'>] <BR><B>Sent:</B> Wednesday, June 08, 2016 12:26 PM<BR><B>To:</B> Robles, Raymond C <</SPAN><A href="mailto:raymond.c.robles@intel.com" target=_blank><SPAN style='FONT-SIZE: 11pt; FONT-FAMILY: "Calibri",sans-serif'>raymond.c.robles@intel.com</SPAN></A><SPAN style='FONT-SIZE: 11pt; FONT-FAMILY: "Calibri",sans-serif'>><BR><B>Cc:</B> </SPAN><A href="mailto:nvmewin@lists.openfabrics.org" target=_blank><SPAN style='FONT-SIZE: 11pt; FONT-FAMILY: "Calibri",sans-serif'>nvmewin@lists.openfabrics.org</SPAN></A><SPAN style='FONT-SIZE: 11pt; FONT-FAMILY: "Calibri",sans-serif'><BR><B>Subject:</B> Re: Happy New Year... and status update</SPAN><o:p></o:p></P>
<P class=MsoNormal style="mso-margin-top-alt: auto; mso-margin-bottom-alt: auto"> <o:p></o:p></P>
<DIV>
<P class=MsoNormal style="mso-margin-top-alt: auto; mso-margin-bottom-alt: auto">I attached our patch. <o:p></o:p></P></DIV>
<DIV>
<P class=MsoNormal style="mso-margin-top-alt: auto; mso-margin-bottom-alt: auto"> <o:p></o:p></P>
<DIV>
<P class=MsoNormal style="mso-margin-top-alt: auto; mso-margin-bottom-alt: auto">On Tue, Jun 7, 2016 at 2:37 PM, Iuliu Rus <<A href="mailto:iuliur@google.com" target=_blank>iuliur@google.com</A>> wrote:<o:p></o:p></P>
<BLOCKQUOTE style="BORDER-TOP: medium none; BORDER-RIGHT: medium none; BORDER-BOTTOM: medium none; PADDING-BOTTOM: 0in; PADDING-TOP: 0in; PADDING-LEFT: 6pt; MARGIN: 5pt 0in 5pt 4.8pt; BORDER-LEFT: #cccccc 1pt solid; PADDING-RIGHT: 0in">
<DIV>
<P class=MsoNormal style="mso-margin-top-alt: auto; mso-margin-bottom-alt: auto">Let me see if i can get it done by tomorrow eod...<o:p></o:p></P></DIV>
<DIV>
<DIV>
<DIV>
<P class=MsoNormal style="mso-margin-top-alt: auto; mso-margin-bottom-alt: auto"> <o:p></o:p></P>
<DIV>
<P class=MsoNormal style="mso-margin-top-alt: auto; mso-margin-bottom-alt: auto">On Tue, Jun 7, 2016 at 2:31 PM, Robles, Raymond C <<A href="mailto:raymond.c.robles@intel.com" target=_blank>raymond.c.robles@intel.com</A>> wrote:<o:p></o:p></P>
<BLOCKQUOTE style="BORDER-TOP: medium none; BORDER-RIGHT: medium none; BORDER-BOTTOM: medium none; PADDING-BOTTOM: 0in; PADDING-TOP: 0in; PADDING-LEFT: 6pt; MARGIN: 5pt 0in 5pt 4.8pt; BORDER-LEFT: #cccccc 1pt solid; PADDING-RIGHT: 0in">
<DIV>
<DIV>
<P class=MsoNormal style="mso-margin-top-alt: auto; mso-margin-bottom-alt: auto"><SPAN style='FONT-SIZE: 11pt; FONT-FAMILY: "Calibri",sans-serif; COLOR: #002060'>Hi Iuliu,</SPAN><o:p></o:p></P>
<P class=MsoNormal style="mso-margin-top-alt: auto; mso-margin-bottom-alt: auto"><SPAN style='FONT-SIZE: 11pt; FONT-FAMILY: "Calibri",sans-serif; COLOR: #002060'> </SPAN><o:p></o:p></P>
<P class=MsoNormal style="mso-margin-top-alt: auto; mso-margin-bottom-alt: auto"><SPAN style='FONT-SIZE: 11pt; FONT-FAMILY: "Calibri",sans-serif; COLOR: #002060'>That’s great… thank you. Yes, the process is still the same (using SVN patches).</SPAN><o:p></o:p></P>
<P class=MsoNormal style="mso-margin-top-alt: auto; mso-margin-bottom-alt: auto"><SPAN style='FONT-SIZE: 11pt; FONT-FAMILY: "Calibri",sans-serif; COLOR: #002060'> </SPAN><o:p></o:p></P>
<P class=MsoNormal style="mso-margin-top-alt: auto; mso-margin-bottom-alt: auto"><SPAN style='FONT-SIZE: 11pt; FONT-FAMILY: "Calibri",sans-serif; COLOR: #002060'>Please let me know if you have any questions/comments. My only question for you is an ETA for the patch to be submitted.</SPAN><o:p></o:p></P>
<P class=MsoNormal style="mso-margin-top-alt: auto; mso-margin-bottom-alt: auto"><SPAN style='FONT-SIZE: 11pt; FONT-FAMILY: "Calibri",sans-serif; COLOR: #002060'> </SPAN><o:p></o:p></P>
<P class=MsoNormal style="mso-margin-top-alt: auto; mso-margin-bottom-alt: auto"><SPAN style='FONT-SIZE: 11pt; FONT-FAMILY: "Calibri",sans-serif; COLOR: #002060'>Thank you…</SPAN><o:p></o:p></P>
<P class=MsoNormal style="mso-margin-top-alt: auto; mso-margin-bottom-alt: auto"><SPAN style='FONT-SIZE: 11pt; FONT-FAMILY: "Calibri",sans-serif; COLOR: #002060'>Ray</SPAN><o:p></o:p></P>
<P class=MsoNormal style="mso-margin-top-alt: auto; mso-margin-bottom-alt: auto"><SPAN style='FONT-SIZE: 11pt; FONT-FAMILY: "Calibri",sans-serif; COLOR: #002060'> </SPAN><o:p></o:p></P>
<P class=MsoNormal style="mso-margin-top-alt: auto; mso-margin-bottom-alt: auto"><B><SPAN style='FONT-SIZE: 11pt; FONT-FAMILY: "Calibri",sans-serif'>From:</SPAN></B><SPAN style='FONT-SIZE: 11pt; FONT-FAMILY: "Calibri",sans-serif'> Iuliu Rus [mailto:</SPAN><A href="mailto:iuliur@google.com" target=_blank><SPAN style='FONT-SIZE: 11pt; FONT-FAMILY: "Calibri",sans-serif'>iuliur@google.com</SPAN></A><SPAN style='FONT-SIZE: 11pt; FONT-FAMILY: "Calibri",sans-serif'>] <BR><B>Sent:</B> Tuesday, June 07, 2016 2:28 PM<BR><B>To:</B> Robles, Raymond C <</SPAN><A href="mailto:raymond.c.robles@intel.com" target=_blank><SPAN style='FONT-SIZE: 11pt; FONT-FAMILY: "Calibri",sans-serif'>raymond.c.robles@intel.com</SPAN></A><SPAN style='FONT-SIZE: 11pt; FONT-FAMILY: "Calibri",sans-serif'>><BR><B>Cc:</B> </SPAN><A href="mailto:nvmewin@lists.openfabrics.org" target=_blank><SPAN style='FONT-SIZE: 11pt; FONT-FAMILY: "Calibri",sans-serif'>nvmewin@lists.openfabrics.org</SPAN></A><SPAN style='FONT-SIZE: 11pt; FONT-FAMILY: "Calibri",sans-serif'><BR><B>Subject:</B> Re: Happy New Year... and status update</SPAN><o:p></o:p></P>
<DIV>
<DIV>
<P class=MsoNormal style="mso-margin-top-alt: auto; mso-margin-bottom-alt: auto"> <o:p></o:p></P>
<DIV>
<P class=MsoNormal style="mso-margin-top-alt: auto; mso-margin-bottom-alt: auto">Yes we can send that to you. Is the process still the same?<o:p></o:p></P></DIV>
<DIV>
<P class=MsoNormal style="mso-margin-top-alt: auto; mso-margin-bottom-alt: auto"> <o:p></o:p></P>
<DIV>
<P class=MsoNormal style="mso-margin-top-alt: auto; mso-margin-bottom-alt: auto">On Tue, Jun 7, 2016 at 1:25 PM, Robles, Raymond C <<A href="mailto:raymond.c.robles@intel.com" target=_blank>raymond.c.robles@intel.com</A>> wrote:<o:p></o:p></P>
<BLOCKQUOTE style="BORDER-TOP: medium none; BORDER-RIGHT: medium none; BORDER-BOTTOM: medium none; PADDING-BOTTOM: 0in; PADDING-TOP: 0in; PADDING-LEFT: 6pt; MARGIN: 5pt 0in 5pt 4.8pt; BORDER-LEFT: #cccccc 1pt solid; PADDING-RIGHT: 0in">
<DIV>
<DIV>
<P class=MsoNormal style="mso-margin-top-alt: auto; mso-margin-bottom-alt: auto"><SPAN style="COLOR: #002060">Hello Iuliu,</SPAN><o:p></o:p></P>
<P class=MsoNormal style="mso-margin-top-alt: auto; mso-margin-bottom-alt: auto"><SPAN style="COLOR: #002060"> </SPAN><o:p></o:p></P>
<P class=MsoNormal style="mso-margin-top-alt: auto; mso-margin-bottom-alt: auto"><SPAN style="COLOR: #002060">As the OFA community is closing on its mid-year release, we have only one more patch that was targeted… concurrent channels. Will Google still be submitting the concurrent channels patch?</SPAN><o:p></o:p></P>
<P class=MsoNormal style="mso-margin-top-alt: auto; mso-margin-bottom-alt: auto"><SPAN style="COLOR: #002060"> </SPAN><o:p></o:p></P>
<P class=MsoNormal style="mso-margin-top-alt: auto; mso-margin-bottom-alt: auto"><SPAN style="COLOR: #002060">Thank you…</SPAN><o:p></o:p></P>
<P class=MsoNormal style="mso-margin-top-alt: auto; mso-margin-bottom-alt: auto"><SPAN style="COLOR: #002060">Ray</SPAN><o:p></o:p></P>
<P class=MsoNormal style="mso-margin-top-alt: auto; mso-margin-bottom-alt: auto"><SPAN style="COLOR: #002060"> </SPAN><o:p></o:p></P>
<DIV>
<DIV style="BORDER-TOP: #e1e1e1 1pt solid; BORDER-RIGHT: medium none; BORDER-BOTTOM: medium none; PADDING-BOTTOM: 0in; PADDING-TOP: 3pt; PADDING-LEFT: 0in; BORDER-LEFT: medium none; PADDING-RIGHT: 0in">
<P class=MsoNormal style="mso-margin-top-alt: auto; mso-margin-bottom-alt: auto"><B>From:</B> <A href="mailto:nvmewin-bounces@lists.openfabrics.org" target=_blank>nvmewin-bounces@lists.openfabrics.org</A> [mailto:<A href="mailto:nvmewin-bounces@lists.openfabrics.org" target=_blank>nvmewin-bounces@lists.openfabrics.org</A>] <B>On Behalf Of </B>Robles, Raymond C<BR><B>Sent:</B> Thursday, January 14, 2016 4:55 PM<BR><B>To:</B> '<A href="mailto:nvmewin@lists.openfabrics.org" target=_blank>nvmewin@lists.openfabrics.org</A>' <<A href="mailto:nvmewin@lists.openfabrics.org" target=_blank>nvmewin@lists.openfabrics.org</A>><BR><B>Subject:</B> [nvmewin] Happy New Year... and status update<o:p></o:p></P></DIV></DIV>
<P class=MsoNormal style="mso-margin-top-alt: auto; mso-margin-bottom-alt: auto"> <o:p></o:p></P>
<P class=MsoNormal style="mso-margin-top-alt: auto; mso-margin-bottom-alt: auto"><SPAN style='FONT-SIZE: 10pt; FONT-FAMILY: "Intel Clear",sans-serif; COLOR: #44546a'>Hello and Happy New Year,</SPAN><o:p></o:p></P>
<DIV>
<DIV>
<P class=MsoNormal style="mso-margin-top-alt: auto; mso-margin-bottom-alt: auto"><SPAN style='FONT-SIZE: 10pt; FONT-FAMILY: "Intel Clear",sans-serif; COLOR: #44546a'> </SPAN><o:p></o:p></P>
<P class=MsoNormal style="mso-margin-top-alt: auto; mso-margin-bottom-alt: auto"><SPAN style='FONT-SIZE: 10pt; FONT-FAMILY: "Intel Clear",sans-serif; COLOR: #44546a'>I hope everyone had a great holiday season and is off to a great start to the new year!</SPAN><o:p></o:p></P>
<P class=MsoNormal style="mso-margin-top-alt: auto; mso-margin-bottom-alt: auto"><SPAN style='FONT-SIZE: 10pt; FONT-FAMILY: "Intel Clear",sans-serif; COLOR: #44546a'> </SPAN><o:p></o:p></P>
<P class=MsoNormal style="mso-margin-top-alt: auto; mso-margin-bottom-alt: auto"><SPAN style='FONT-SIZE: 10pt; FONT-FAMILY: "Intel Clear",sans-serif; COLOR: #44546a'>As communicated last month, the patch from HGST for the SCSI multi-initiator changes has been approved and pushed. The holidays slowed down much of the progress on the OFA driver and there were several patches that did not get pushed prior to the end of the year. The list of patches remained to be pushed are as follows…</SPAN><o:p></o:p></P>
<P class=MsoNormal style="mso-margin-top-alt: auto; mso-margin-bottom-alt: auto"><SPAN style='FONT-SIZE: 10pt; FONT-FAMILY: "Intel Clear",sans-serif; COLOR: #44546a'> </SPAN><o:p></o:p></P>
<P><SPAN style="FONT-SIZE: 10pt; FONT-FAMILY: Symbol; COLOR: #44546a">·</SPAN><SPAN style="FONT-SIZE: 7pt; COLOR: #44546a"> </SPAN><SPAN style='FONT-SIZE: 10pt; FONT-FAMILY: "Intel Clear",sans-serif; COLOR: #44546a'>Namespace Management (Intel)</SPAN><o:p></o:p></P>
<P><SPAN style="FONT-SIZE: 10pt; FONT-FAMILY: Symbol; COLOR: #44546a">·</SPAN><SPAN style="FONT-SIZE: 7pt; COLOR: #44546a"> </SPAN><SPAN style='FONT-SIZE: 10pt; FONT-FAMILY: "Intel Clear",sans-serif; COLOR: #44546a'>Perf Opts (Samsung) </SPAN><o:p></o:p></P>
<P><SPAN style="FONT-SIZE: 10pt; FONT-FAMILY: Symbol; COLOR: #44546a">·</SPAN><SPAN style="FONT-SIZE: 7pt; COLOR: #44546a"> </SPAN><SPAN style='FONT-SIZE: 10pt; FONT-FAMILY: "Intel Clear",sans-serif; COLOR: #44546a'>Win 8.1 Timers (Samsung)</SPAN><o:p></o:p></P>
<P><SPAN style="FONT-SIZE: 10pt; FONT-FAMILY: Symbol; COLOR: #44546a">·</SPAN><SPAN style="FONT-SIZE: 7pt; COLOR: #44546a"> </SPAN><SPAN style='FONT-SIZE: 10pt; FONT-FAMILY: "Intel Clear",sans-serif; COLOR: #44546a'>EOL Read Only (Samsung)</SPAN><o:p></o:p></P>
<P><SPAN style="FONT-SIZE: 10pt; FONT-FAMILY: Symbol; COLOR: #44546a">·</SPAN><SPAN style="FONT-SIZE: 7pt; COLOR: #44546a"> </SPAN><SPAN style='FONT-SIZE: 10pt; FONT-FAMILY: "Intel Clear",sans-serif; COLOR: #44546a'>Concurrent channels (Google)</SPAN><o:p></o:p></P>
<P class=MsoNormal style="mso-margin-top-alt: auto; mso-margin-bottom-alt: auto"><SPAN style='FONT-SIZE: 10pt; FONT-FAMILY: "Intel Clear",sans-serif; COLOR: #44546a'> </SPAN><o:p></o:p></P>
<P class=MsoNormal style="mso-margin-top-alt: auto; mso-margin-bottom-alt: auto"><SPAN style='FONT-SIZE: 10pt; FONT-FAMILY: "Intel Clear",sans-serif; COLOR: #44546a'>The namespace management patch will be sent out for review tomorrow (look for the patch email from Carolyn)… stay tuned!!!</SPAN><o:p></o:p></P>
<P class=MsoNormal style="mso-margin-top-alt: auto; mso-margin-bottom-alt: auto"><SPAN style='FONT-SIZE: 10pt; FONT-FAMILY: "Intel Clear",sans-serif; COLOR: #44546a'> </SPAN><o:p></o:p></P>
<P class=MsoNormal style="mso-margin-top-alt: auto; mso-margin-bottom-alt: auto"><SPAN style='FONT-SIZE: 10pt; FONT-FAMILY: "Intel Clear",sans-serif; COLOR: #44546a'>However, once this patch is resolved, we as a community will have to make a decision on an official release strategy. The patch process and cadence was significantly slower in 2015 which leaves us with a few options. </SPAN><o:p></o:p></P>
<P class=MsoNormal style="mso-margin-top-alt: auto; mso-margin-bottom-alt: auto"><SPAN style='FONT-SIZE: 10pt; FONT-FAMILY: "Intel Clear",sans-serif; COLOR: #44546a'> </SPAN><o:p></o:p></P>
<P><SPAN style='FONT-SIZE: 10pt; FONT-FAMILY: "Intel Clear",sans-serif; COLOR: #44546a'>1.</SPAN><SPAN style="FONT-SIZE: 7pt; COLOR: #44546a"> </SPAN><SPAN style='FONT-SIZE: 10pt; FONT-FAMILY: "Intel Clear",sans-serif; COLOR: #44546a'>Release what is in the trunk today (or after the namespace management patch)… and call that the 2015 release (albeit later than expected)</SPAN><o:p></o:p></P>
<P><SPAN style='FONT-SIZE: 10pt; FONT-FAMILY: "Intel Clear",sans-serif; COLOR: #44546a'>2.</SPAN><SPAN style="FONT-SIZE: 7pt; COLOR: #44546a"> </SPAN><SPAN style='FONT-SIZE: 10pt; FONT-FAMILY: "Intel Clear",sans-serif; COLOR: #44546a'>In lieu of an official 2015 release, we push the remaining patches listed above… and then release in ~Q2 of 2015. Basically skip a 2015 release and go right into the mid-2016 release.</SPAN><o:p></o:p></P>
<P><SPAN style='FONT-SIZE: 10pt; FONT-FAMILY: "Intel Clear",sans-serif; COLOR: #44546a'>3.</SPAN><SPAN style="FONT-SIZE: 7pt; COLOR: #44546a"> </SPAN><SPAN style='FONT-SIZE: 10pt; FONT-FAMILY: "Intel Clear",sans-serif; COLOR: #44546a'>Remove the concept of “official releases” from the OFA Windows NVMe driver and just allow people, companies, and users to pull from the OFA trunk as necessary.</SPAN><o:p></o:p></P>
<P class=MsoNormal style="mso-margin-top-alt: auto; mso-margin-bottom-alt: auto"><SPAN style='FONT-SIZE: 10pt; FONT-FAMILY: "Intel Clear",sans-serif; COLOR: #44546a'> </SPAN><o:p></o:p></P>
<P class=MsoNormal style="mso-margin-top-alt: auto; mso-margin-bottom-alt: auto"><SPAN style='FONT-SIZE: 10pt; FONT-FAMILY: "Intel Clear",sans-serif; COLOR: #44546a'>For #3 above, my thoughts are that because the OFA driver is not a production driver… but a reference and enabling driver, it should/could be managed as just that… a reference baseline driver that any potential user can go and grab the source, or contribute a patch. Nothing more… nothing less.</SPAN><o:p></o:p></P>
<P class=MsoNormal style="mso-margin-top-alt: auto; mso-margin-bottom-alt: auto"><SPAN style='FONT-SIZE: 10pt; FONT-FAMILY: "Intel Clear",sans-serif; COLOR: #44546a'> </SPAN><o:p></o:p></P>
<P class=MsoNormal style="mso-margin-top-alt: auto; mso-margin-bottom-alt: auto"><SPAN style='FONT-SIZE: 10pt; FONT-FAMILY: "Intel Clear",sans-serif; BACKGROUND: yellow; COLOR: #44546a'>For the release decision, I’ll be happy to call a meeting… but we can also handle it via email as well… just let me know. Feedback from all is welcome… but I would request mandatory feedback form the 4 reviewing companies: Samsung, HGST, PMC-Sierra, and Intel. How would you like to proceed?</SPAN><o:p></o:p></P>
<P class=MsoNormal style="mso-margin-top-alt: auto; mso-margin-bottom-alt: auto"><SPAN style='FONT-SIZE: 10pt; FONT-FAMILY: "Intel Clear",sans-serif; COLOR: #44546a'> </SPAN><o:p></o:p></P>
<P class=MsoNormal style="mso-margin-top-alt: auto; mso-margin-bottom-alt: auto"><SPAN style='FONT-SIZE: 10pt; FONT-FAMILY: "Intel Clear",sans-serif; COLOR: #44546a'>Thanks,</SPAN><o:p></o:p></P>
<P class=MsoNormal style="mso-margin-top-alt: auto; mso-margin-bottom-alt: auto"><SPAN style='FONT-SIZE: 10pt; FONT-FAMILY: "Intel Clear",sans-serif; COLOR: #44546a'>Ray</SPAN><o:p></o:p></P>
<P class=MsoNormal style="mso-margin-top-alt: auto; mso-margin-bottom-alt: auto"><SPAN style='FONT-SIZE: 10pt; FONT-FAMILY: "Intel Clear",sans-serif; COLOR: #44546a'> </SPAN><o:p></o:p></P>
<P class=MsoNormal style="mso-margin-top-alt: auto; mso-margin-bottom-alt: auto"><B><SPAN style='FONT-SIZE: 10pt; FONT-FAMILY: "Intel Clear",sans-serif; COLOR: #1f497d'><IMG id=_x0000_i1027 border=0 alt=cid:image001.png@01CB3870.4BB88E70 src="cid:MKNRCZHWKC3T@namo.co.kr" width=54 height=39></SPAN></B><o:p></o:p></P>
<P class=MsoNormal style="mso-margin-top-alt: auto; mso-margin-bottom-alt: auto"><B><I><SPAN style='FONT-FAMILY: "Intel Clear",sans-serif; COLOR: #1f497d'>Raymond C. Robles</SPAN></I></B><o:p></o:p></P>
<P class=MsoNormal style="mso-margin-top-alt: auto; mso-margin-bottom-alt: auto"><SPAN style='FONT-SIZE: 10pt; FONT-FAMILY: "Intel Clear",sans-serif; COLOR: #1f497d'>Non-Volatile Memory Solutions Group</SPAN><o:p></o:p></P>
<P class=MsoNormal style="mso-margin-top-alt: auto; mso-margin-bottom-alt: auto"><SPAN style='FONT-SIZE: 10pt; FONT-FAMILY: "Intel Clear",sans-serif; COLOR: #1f497d'>Intel Corporation</SPAN><o:p></o:p></P>
<P class=MsoNormal style="mso-margin-top-alt: auto; mso-margin-bottom-alt: auto"><SPAN style='FONT-SIZE: 10pt; FONT-FAMILY: "Intel Clear",sans-serif; COLOR: #1f497d'>Office: </SPAN><A href="tel:480-554-2600" target=_blank><SPAN style='FONT-SIZE: 10pt; FONT-FAMILY: "Intel Clear",sans-serif'>480-554-2600</SPAN></A><o:p></o:p></P>
<P class=MsoNormal style="mso-margin-top-alt: auto; mso-margin-bottom-alt: auto"><SPAN style='FONT-SIZE: 10pt; FONT-FAMILY: "Intel Clear",sans-serif; COLOR: #1f497d'>Mobile: </SPAN><A href="tel:480-399-0645" target=_blank><SPAN style='FONT-SIZE: 10pt; FONT-FAMILY: "Intel Clear",sans-serif'>480-399-0645</SPAN></A><o:p></o:p></P>
<P class=MsoNormal style="mso-margin-top-alt: auto; mso-margin-bottom-alt: auto"> <o:p></o:p></P></DIV></DIV></DIV></DIV></BLOCKQUOTE></DIV>
<P class=MsoNormal style="mso-margin-top-alt: auto; mso-margin-bottom-alt: auto"> <o:p></o:p></P></DIV></DIV></DIV></DIV></DIV></BLOCKQUOTE></DIV>
<P class=MsoNormal style="mso-margin-top-alt: auto; mso-margin-bottom-alt: auto"> <o:p></o:p></P></DIV></DIV></DIV></BLOCKQUOTE></DIV>
<P class=MsoNormal style="mso-margin-top-alt: auto; mso-margin-bottom-alt: auto"> <o:p></o:p></P></DIV></DIV>
<P> <o:p></o:p></P>
<P> <o:p></o:p></P></DIV></DIV>
<TABLE class=MsoNormalTable cellPadding=0 border=0>
<TBODY>
<TR>
<TD style="PADDING-BOTTOM: 0.75pt; PADDING-TOP: 0.75pt; PADDING-LEFT: 0.75pt; PADDING-RIGHT: 0.75pt">
<P><IMG id=_x0000_i1028 border=0 alt=cid:image006.gif@01D1C6EF.5D9CDE20 src="cid:VGKDE24SOZ9F@namo.co.kr" width=520 height=144><o:p></o:p></P></TD></TR></TBODY></TABLE></DIV>
<P class=MsoNormal style="mso-margin-top-alt: auto; mso-margin-bottom-alt: auto"><SPAN style="BORDER-TOP: windowtext 1pt solid; BORDER-RIGHT: windowtext 1pt solid; BORDER-BOTTOM: windowtext 1pt solid; PADDING-BOTTOM: 0in; PADDING-TOP: 0in; PADDING-LEFT: 0in; BORDER-LEFT: windowtext 1pt solid; PADDING-RIGHT: 0in"><IMG id=_x0000_i1029 border=0 alt="Image removed by sender. http://ext.samsung.net/mailcheck/SeenTimeChecker?do=9226f2572c3ad117df6a46883b3f3dde1d283838fa336dda194352576c69fe9bfdd53f331c9856e34bea3c1765014a1208cece8541bc14eacf878f9a26ce15a0" src="cid:1LI1H0JVUG24@namo.co.kr" width=32 height=32></SPAN><o:p></o:p></P></BLOCKQUOTE></DIV>
<P class=MsoNormal style="mso-margin-top-alt: auto; mso-margin-bottom-alt: auto"> <o:p></o:p></P></DIV></DIV>
<P> <o:p></o:p></P>
<P> <o:p></o:p></P></DIV></DIV>
<TABLE class=MsoNormalTable cellPadding=0 border=0>
<TBODY>
<TR>
<TD style="PADDING-BOTTOM: 0.75pt; PADDING-TOP: 0.75pt; PADDING-LEFT: 0.75pt; PADDING-RIGHT: 0.75pt">
<P><IMG id=_x0000_i1030 border=0 alt=cid:image006.gif@01D1C6EF.5D9CDE20 src="cid:VGKDE24SOZ9F@namo.co.kr" width=520 height=144><o:p></o:p></P></TD></TR></TBODY></TABLE></DIV>
<P class=MsoNormal style="mso-margin-top-alt: auto; mso-margin-bottom-alt: auto"><SPAN style="BORDER-TOP: windowtext 1pt solid; BORDER-RIGHT: windowtext 1pt solid; BORDER-BOTTOM: windowtext 1pt solid; PADDING-BOTTOM: 0in; PADDING-TOP: 0in; PADDING-LEFT: 0in; BORDER-LEFT: windowtext 1pt solid; PADDING-RIGHT: 0in"><IMG id=_x0000_i1031 border=0 alt="Image removed by sender. http://ext.samsung.net/mailcheck/SeenTimeChecker?do=36e611fbbc499bdb45df2c678ea6c804daf257cfe78672b8194352576c69fe9bfdd53f331c9856e34bea3c1765014a1208cece8541bc14eacf878f9a26ce15a0" src="cid:1LI1H0JVUG24@namo.co.kr" width=32 height=32></SPAN><o:p></o:p></P></BLOCKQUOTE></DIV>
<P class=MsoNormal style="mso-margin-top-alt: auto; mso-margin-bottom-alt: auto"> <o:p></o:p></P></DIV></DIV>
<DIV>
<P class=MsoNormal style="mso-margin-top-alt: auto; mso-margin-bottom-alt: auto">Western Digital Corporation (and its subsidiaries) E-mail Confidentiality Notice & Disclaimer:<o:p></o:p></P>
<DIV>
<P class=MsoNormal style="mso-margin-top-alt: auto; mso-margin-bottom-alt: auto">This e-mail and any files transmitted with it may contain confidential or legally privileged information of WDC and/or its affiliates, and are intended solely for the use of the individual or entity to which they are addressed. If you are not the intended recipient, any disclosure, copying, distribution or any action taken or omitted to be taken in reliance on it, is prohibited. If you have received this e-mail in error, please notify the sender immediately and delete the e-mail in its entirety from your system.<o:p></o:p></P></DIV></DIV></DIV>
<P class=MsoNormal style="mso-margin-top-alt: auto; mso-margin-bottom-alt: auto"> <o:p></o:p></P></DIV></DIV>
<P class=MsoNormal style="MARGIN-BOTTOM: 12pt; mso-margin-top-alt: auto">_______________________________________________<BR>nvmewin mailing list<BR><A href="mailto:nvmewin@lists.openfabrics.org" target=_blank>nvmewin@lists.openfabrics.org</A><BR><A href="http://lists.openfabrics.org/mailman/listinfo/nvmewin" target=_blank>http://lists.openfabrics.org/mailman/listinfo/nvmewin</A><o:p></o:p></P></BLOCKQUOTE></DIV>
<P class=MsoNormal style="mso-margin-top-alt: auto; mso-margin-bottom-alt: auto"> <o:p></o:p></P></DIV></BLOCKQUOTE></DIV>
<P class=MsoNormal style="mso-margin-top-alt: auto; mso-margin-bottom-alt: auto"> <o:p></o:p></P></DIV>
<DIV>
<P class=MsoNormal style="mso-margin-top-alt: auto; mso-margin-bottom-alt: auto">Western Digital Corporation (and its subsidiaries) E-mail Confidentiality Notice & Disclaimer:<o:p></o:p></P>
<DIV>
<P class=MsoNormal style="mso-margin-top-alt: auto; mso-margin-bottom-alt: auto">This e-mail and any files transmitted with it may contain confidential or legally privileged information of WDC and/or its affiliates, and are intended solely for the use of the individual or entity to which they are addressed. If you are not the intended recipient, any disclosure, copying, distribution or any action taken or omitted to be taken in reliance on it, is prohibited. If you have received this e-mail in error, please notify the sender immediately and delete the e-mail in its entirety from your system.<o:p></o:p></P></DIV></DIV></DIV></DIV></DIV></DIV></BLOCKQUOTE></DIV>
<P class=MsoNormal style="mso-margin-top-alt: auto; mso-margin-bottom-alt: auto"> <o:p></o:p></P></DIV></DIV></DIV></BLOCKQUOTE></DIV>
<P class=MsoNormal style="mso-margin-top-alt: auto; mso-margin-bottom-alt: auto"> <o:p></o:p></P></DIV></DIV></DIV></DIV></DIV></BLOCKQUOTE></DIV>
<P class=MsoNormal><o:p> </o:p></P></DIV></DIV></X-BODY>
<P> </P>
<P> </P>
<TABLE id=confidentialsignimg>
<TBODY>
<TR>
<TD NAMO_LOCK>
<P><IMG border=0 src="cid:KMPGOPJRNQ91@namo.co.kr"></P></TD></TR></TBODY></TABLE></BODY></HTML><img src='http://ext.samsung.net/mailcheck/SeenTimeChecker?do=82e1a98ccd90fe044e802b8acba378fd5c47ca882526d76c7d9badbdf7e30042d1afaaba7860cdcd9564217c646641ad61e16949eaa607501b20909a04efd4d2748cfe1d4e847419cf878f9a26ce15a0' border=0 width=0 height=0 style='display:none'>