<HTML xmlns:v = "urn:schemas-microsoft-com:vml" xmlns:o = "urn:schemas-microsoft-com:office:office" xmlns:w = "urn:schemas-microsoft-com:office:word" xmlns:m = "http://schemas.microsoft.com/office/2004/12/omml" xmlns:ns0 = ""><HEAD><TITLE>Samsung Enterprise Portal mySingle</TITLE>
<META content="text/html; charset=utf-8" http-equiv=Content-Type>
<STYLE id=mysingle_style type=text/css>P {
MARGIN-TOP: 5px; FONT-FAMILY: Arial, arial; MARGIN-BOTTOM: 5px; FONT-SIZE: 9pt
}
TD {
MARGIN-TOP: 5px; FONT-FAMILY: Arial, arial; MARGIN-BOTTOM: 5px; FONT-SIZE: 9pt
}
LI {
MARGIN-TOP: 5px; FONT-FAMILY: Arial, arial; MARGIN-BOTTOM: 5px; FONT-SIZE: 9pt
}
BODY {
LINE-HEIGHT: 1.4; MARGIN: 10px; FONT-FAMILY: Arial, arial; FONT-SIZE: 9pt
}
</STYLE>
<META name=GENERATOR content=ActiveSquare></HEAD>
<BODY>
<P class=MsoNormal><SPAN style="FONT-FAMILY: 'Calibri','sans-serif'; COLOR: #000000; FONT-SIZE: 11pt; mso-ascii-theme-font: minor-latin; mso-hansi-theme-font: minor-latin; mso-bidi-font-family: 'Times New Roman'; mso-bidi-theme-font: minor-bidi; mso-themecolor: dark2">Hi Ray,</SPAN><o:p><SPAN style="COLOR: #000000"></SPAN></o:p></P>
<P class=MsoNormal><SPAN style="FONT-FAMILY: 'Calibri','sans-serif'; COLOR: #000000; FONT-SIZE: 11pt; mso-ascii-theme-font: minor-latin; mso-hansi-theme-font: minor-latin; mso-bidi-font-family: 'Times New Roman'; mso-bidi-theme-font: minor-bidi; mso-themecolor: dark2">Attached is the summary of the screen shot.</SPAN><o:p></o:p></P>
<P><IMG style="WIDTH: 882px; HEIGHT: 515px" src="cid:XOK0LK7CT9SZ@namo.co.kr"></P>
<P> </P>
<P>We can discuss in the next WG call.</P>
<P> </P>
<P>Regards</P>
<P>Santosh</P>
<P> </P>
<P>------- <B>Original Message</B> -------</P>
<P><B>Sender</B> : Robles, Raymond C<raymond.c.robles@intel.com></P>
<P><B>Date</B> : Mar 22, 2013 05:17 (GMT+09:00)</P>
<P><B>Title</B> : RE: RE: RE: NVMe 1.1 : Clarification on PRP2 Entry list</P>
<P> </P>
<META name=Generator content=ActiveSquare>
<STYLE>v\:* {
BEHAVIOR: url(#default#VML)
}
o\:* {
BEHAVIOR: url(#default#VML)
}
w\:* {
BEHAVIOR: url(#default#VML)
}
.shape {
BEHAVIOR: url(#default#VML)
}
</STYLE>
<STYLE>@font-face {
font-family: Calibri;
}
@font-face {
font-family: Tahoma;
}
@page WordSection1 {size: 8.5in 11.0in; margin: 1.0in 1.0in 1.0in 1.0in; }
P.MsoNormal {
FONT-FAMILY: "Times New Roman","serif"; MARGIN-LEFT: 0in; FONT-SIZE: 12pt; MARGIN-RIGHT: 0in; mso-margin-top-alt: auto; mso-margin-bottom-alt: auto; mso-believe-normal-left: yes
}
LI.MsoNormal {
FONT-FAMILY: "Times New Roman","serif"; MARGIN-LEFT: 0in; FONT-SIZE: 12pt; MARGIN-RIGHT: 0in; mso-margin-top-alt: auto; mso-margin-bottom-alt: auto; mso-believe-normal-left: yes
}
DIV.MsoNormal {
FONT-FAMILY: "Times New Roman","serif"; MARGIN-LEFT: 0in; FONT-SIZE: 12pt; MARGIN-RIGHT: 0in; mso-margin-top-alt: auto; mso-margin-bottom-alt: auto; mso-believe-normal-left: yes
}
A:link {
COLOR: blue; TEXT-DECORATION: underline; mso-style-priority: 99
}
SPAN.MsoHyperlink {
COLOR: blue; TEXT-DECORATION: underline; mso-style-priority: 99
}
A:visited {
COLOR: purple; TEXT-DECORATION: underline; mso-style-priority: 99
}
SPAN.MsoHyperlinkFollowed {
COLOR: purple; TEXT-DECORATION: underline; mso-style-priority: 99
}
P {
MARGIN: 3.75pt 0in; FONT-FAMILY: "Arial","sans-serif"; FONT-SIZE: 9pt; mso-style-priority: 99
}
P.MsoAcetate {
FONT-FAMILY: "Tahoma","sans-serif"; MARGIN-LEFT: 0in; FONT-SIZE: 8pt; MARGIN-RIGHT: 0in; mso-margin-top-alt: auto; mso-margin-bottom-alt: auto; mso-style-priority: 99; mso-style-link: "Balloon Text Char"
}
LI.MsoAcetate {
FONT-FAMILY: "Tahoma","sans-serif"; MARGIN-LEFT: 0in; FONT-SIZE: 8pt; MARGIN-RIGHT: 0in; mso-margin-top-alt: auto; mso-margin-bottom-alt: auto; mso-style-priority: 99; mso-style-link: "Balloon Text Char"
}
DIV.MsoAcetate {
FONT-FAMILY: "Tahoma","sans-serif"; MARGIN-LEFT: 0in; FONT-SIZE: 8pt; MARGIN-RIGHT: 0in; mso-margin-top-alt: auto; mso-margin-bottom-alt: auto; mso-style-priority: 99; mso-style-link: "Balloon Text Char"
}
SPAN.BalloonTextChar {
FONT-FAMILY: "Tahoma","sans-serif"; mso-style-priority: 99; mso-style-link: "Balloon Text"; mso-style-name: "Balloon Text Char"
}
SPAN.EmailStyle20 {
FONT-FAMILY: "Calibri","sans-serif"; COLOR: #1f497d; mso-style-type: personal
}
SPAN.EmailStyle21 {
FONT-STYLE: normal; FONT-FAMILY: "Times New Roman","serif"; COLOR: #1f497d; FONT-WEIGHT: normal; mso-style-type: personal-reply
}
.MsoChpDefault {
FONT-SIZE: 10pt; mso-style-type: export-only
}
DIV.WordSection1 {
page: WordSection1
}
</STYLE>
<X-BODY style="MARGIN: 7.5pt" lang=EN-US link="blue" vlink="purple">
<DIV class=WordSection1>
<P class=MsoNormal><SPAN style="COLOR: #1f497d; FONT-SIZE: 10pt">Hi Santosh,<o:p></o:p></SPAN></P>
<P class=MsoNormal><SPAN style="COLOR: #1f497d; FONT-SIZE: 10pt">I am one of the original authors of the OFA Windows NVMe driver (sorry, I’m late to this thread). What do you believe is the problem? The Windows OFA driver constructs PRP lists per the NVMe spec. We’ve run for several days with numerous data integrity testing tools (without error).<o:p></o:p></SPAN></P>
<P class=MsoNormal><SPAN style="COLOR: #1f497d; FONT-SIZE: 10pt">Do you believe that the PRP list is incorrectly constructed? Based on the screen shot you sent out, the second PRP entry in the submission queue entry points to a PRP list… it should not contain the 2<SUP>nd</SUP> PRP entry. This is per the NVMe spec.<o:p></o:p></SPAN></P>
<P class=MsoNormal><SPAN style="COLOR: #1f497d; FONT-SIZE: 10pt">Thanks,<o:p></o:p></SPAN></P>
<P class=MsoNormal><SPAN style="COLOR: #1f497d; FONT-SIZE: 10pt">Ray<o:p></o:p></SPAN></P>
<P class=MsoNormal><SPAN style="COLOR: #1f497d; FONT-SIZE: 10pt"><o:p> </o:p></SPAN></P>
<DIV>
<DIV style="BORDER-BOTTOM: medium none; BORDER-LEFT: medium none; PADDING-BOTTOM: 0in; PADDING-LEFT: 0in; PADDING-RIGHT: 0in; BORDER-TOP: #b5c4df 1pt solid; BORDER-RIGHT: medium none; PADDING-TOP: 3pt">
<P style="MARGIN: 0in 0in 0pt" class=MsoNormal><B><SPAN style="FONT-FAMILY: 'Tahoma','sans-serif'; FONT-SIZE: 10pt">From:</SPAN></B><SPAN style="FONT-FAMILY: 'Tahoma','sans-serif'; FONT-SIZE: 10pt"> Kong, Kwok [mailto:Kwok.Kong@idt.com] <BR><B>Sent:</B> Thursday, March 21, 2013 11:14 AM<BR><B>To:</B> santosh.s2@samsung.com<BR><B>Cc:</B> technical@nvmexpress.org; Onufryk, Peter; Wilcox, Matthew R<BR><B>Subject:</B> RE: RE: RE: NVMe 1.1 : Clarification on PRP2 Entry list<o:p></o:p></SPAN></P></DIV></DIV>
<P class=MsoNormal><o:p> </o:p></P>
<P class=MsoNormal><SPAN style="FONT-FAMILY: 'Calibri','sans-serif'; COLOR: #1f497d; FONT-SIZE: 11pt">Santosh,<o:p></o:p></SPAN></P>
<P class=MsoNormal><SPAN style="FONT-FAMILY: 'Calibri','sans-serif'; COLOR: #1f497d; FONT-SIZE: 11pt">The specification is very clear on how the PRP list should be constructed. If you see a problem with any driver, then it is a driver problem and not a specification problem.<o:p></o:p></SPAN></P>
<P class=MsoNormal><SPAN style="FONT-FAMILY: 'Calibri','sans-serif'; COLOR: #1f497d; FONT-SIZE: 11pt">Please send your question to <A href="mailto:nvmewin@lists.openfabrics.org">nvmewin@lists.openfabrics.org</A> if you believe there is a driver bug.<o:p></o:p></SPAN></P>
<P class=MsoNormal><SPAN style="FONT-FAMILY: 'Calibri','sans-serif'; COLOR: #1f497d; FONT-SIZE: 11pt">What is the LBA size for your testing ? 512B or 4KB ?<o:p></o:p></SPAN></P>
<P class=MsoNormal><SPAN style="FONT-FAMILY: 'Calibri','sans-serif'; COLOR: #1f497d; FONT-SIZE: 11pt">Thanks<o:p></o:p></SPAN></P>
<P class=MsoNormal><SPAN style="FONT-FAMILY: 'Calibri','sans-serif'; COLOR: #1f497d; FONT-SIZE: 11pt">-Kwok<o:p></o:p></SPAN></P>
<P class=MsoNormal><SPAN style="FONT-FAMILY: 'Calibri','sans-serif'; COLOR: #1f497d; FONT-SIZE: 11pt"><o:p> </o:p></SPAN></P>
<DIV>
<DIV style="BORDER-BOTTOM: medium none; BORDER-LEFT: medium none; PADDING-BOTTOM: 0in; PADDING-LEFT: 0in; PADDING-RIGHT: 0in; BORDER-TOP: #b5c4df 1pt solid; BORDER-RIGHT: medium none; PADDING-TOP: 3pt">
<P style="MARGIN: 0in 0in 0pt" class=MsoNormal><B><SPAN style="FONT-FAMILY: 'Tahoma','sans-serif'; FONT-SIZE: 10pt">From:</SPAN></B><SPAN style="FONT-FAMILY: 'Tahoma','sans-serif'; FONT-SIZE: 10pt"> SANTOSH SINGH [<A href="mailto:santosh.s2@samsung.com">mailto:santosh.s2@samsung.com</A>] <BR><B>Sent:</B> Thursday, March 21, 2013 3:55 AM<BR><B>To:</B> Kong, Kwok<BR><B>Cc:</B> <A href="mailto:technical@nvmexpress.org">technical@nvmexpress.org</A>; Onufryk, Peter; 'Wilcox, Matthew R'<BR><B>Subject:</B> Re: RE: RE: NVMe 1.1 : Clarification on PRP2 Entry list<o:p></o:p></SPAN></P></DIV></DIV>
<P class=MsoNormal><o:p> </o:p></P>
<P>Sorry some how the attachement is missing. Resending.<o:p></o:p></P>
<P> <o:p></o:p></P>
<P><IMG id=_x0000_i1025 border=0 src="cid:PYMC4CBBEM6S@namo.co.kr" width=1600 height=860><o:p></o:p></P>
<P> <o:p></o:p></P>
<P>Regards<o:p></o:p></P>
<P>Santosh<o:p></o:p></P>
<P> <o:p></o:p></P>
<P> <o:p></o:p></P>
<P>------- <B>Original Message</B> -------<o:p></o:p></P>
<P><B>Sender</B> : SANTOSH SINGH<<A href="mailto:santosh.s2@samsung.com">santosh.s2@samsung.com</A>> Senior Chief Engineer/SRI-Bangalore-SSD Solutions/Samsung Electronics<o:p></o:p></P>
<P><B>Date</B> : Mar 21, 2013 19:45 (GMT+09:00)<o:p></o:p></P>
<P><B>Title</B> : RE: RE: NVMe 1.1 : Clarification on PRP2 Entry list<o:p></o:p></P>
<P> <o:p></o:p></P>
<P style="MARGIN-BOTTOM: 12pt; MARGIN-LEFT: 0in; MARGIN-RIGHT: 0in; mso-margin-top-alt: 0in" class=MsoNormal><SPAN style="FONT-FAMILY: 'Arial','sans-serif'; FONT-SIZE: 9pt">Hi Kwok,<BR><BR>I got the scenario reproduced again, while issuing the FS format command.<BR>Following are the debug details.<BR><BR>Page size 4k<BR>Data Transfer size was 16 LBA<BR>PRP1 Entry 0x3e8fd060<BR>PRP2 Entry list 0xbe166ff0<BR>Total no. of PRP entries 17<BR><BR>The 16 PRP entries should fit in single page. But the PRP2 offset<BR>0xbe166ff0(PRP2) is not correct and it has run off the page(from 0xbe166ff0<BR>to 0xbe167000) which is the next page in continuity. Attached is the<BR>snapshot of the debug window for the detailed analysis.<BR><BR><BR><BR><BR><BR><BR>Regards<BR>Santosh<BR><BR><BR>-----Original Message-----<BR>From: Kong, Kwok [<A href="mailto:Kwok.Kong@idt.com">mailto:Kwok.Kong@idt.com</A>] <BR>Sent: Thursday, March 21, 2013 1:22 AM<BR>To: Wilcox, Matthew R; <A href="mailto:santosh.s2@samsung.com">santosh.s2@samsung.com</A><BR>Cc: Onufryk, Peter; <A href="mailto:technical@nvmexpress.org">technical@nvmexpress.org</A><BR>Subject: RE: RE: NVMe 1.1 : Clarification on PRP2 Entry list<BR><BR>Santosh,<BR><BR>Your verification on the OFA driver that it uses PRP list format as in<BR>figure 2 is incorrect.<BR><BR>The OFA driver does neither figure 1, nor figure 2.<BR><BR>By default, the max request size that the mini-port supports is 128KB. If<BR>the request size is bigger than 128KB, the port driver sends multiple 128KB<BR>requests to the OFA mini-port driver.<BR><BR>The OFA mini-port driver pre-allocates the PRP list buffers during<BR>initialization. The PRP entries (a max of 32 entries for 128KB request<BR>size) never run off the end of a page as shown in figure 1 or figure 2. All<BR>PRP entries are guaranteed to fit within a single Memory Page.<BR><BR>Thanks<BR><BR>-Kwok<BR><BR><BR><BR>-----Original Message-----<BR>From: Wilcox, Matthew R [<A href="mailto:matthew.r.wilcox@intel.com">mailto:matthew.r.wilcox@intel.com</A>] <BR>Sent: Wednesday, March 20, 2013 8:26 AM<BR>To: <A href="mailto:santosh.s2@samsung.com">santosh.s2@samsung.com</A>; Kong, Kwok<BR>Cc: Onufryk, Peter; <A href="mailto:technical@nvmexpress.org">technical@nvmexpress.org</A><BR>Subject: RE: RE: NVMe 1.1 : Clarification on PRP2 Entry list<BR><BR>The Linux driver does neither figure 1, nor figure 2.<BR><BR>If the number of PRP entries requires more than one page, it starts at the<BR>beginning of a page. If it requires less than a page, it may start in the<BR>middle of a page, but will never run off the end of a page as shown in<BR>figure 2.<BR><BR>I have not reviewed the OFA driver to see what it does.<BR><BR>________________________________<BR>From: SANTOSH SINGH [santosh.s2@samsung.com]<BR>Sent: March 19, 2013 8:01 PM<BR>To: Kong, Kwok; Wilcox, Matthew R<BR>Cc: Onufryk, Peter; <A href="mailto:technical@nvmexpress.org">technical@nvmexpress.org</A><BR>Subject: Re: RE: NVMe 1.1 : Clarification on PRP2 Entry list<BR><BR><BR>Hi Kwok, Matthew,<BR><BR><BR><BR>I verified the OFA driver too and that prepares the PRP List entry as in<BR>fig-2.<BR><BR>Any reason why both the drivers(Linux and OFA) prepares the PRP2 list like<BR>fig-2. Will it change in the later version of drivers as fig-1.<BR><BR><BR><BR>Regards<BR><BR>Santosh<BR><BR><BR><BR><BR><BR><BR><BR>------- Original Message -------<BR><BR>Sender : Kong, Kwok<BR><BR>Date : Mar 20, 2013 01:01 (GMT+09:00)<BR><BR>Title : RE: NVMe 1.1 : Clarification on PRP2 Entry list<BR><BR><BR>I believe the specification is very clear to indicate that figure 1 is<BR>correct.<BR><BR>"The last entry within a memory page, as indicated by the memory page size<BR>in the CC.MPS field, shall be a PRP List pointer if there is more than a<BR>single memory page of data to be transferred.".<BR><BR><BR>Thanks<BR><BR>-Kwok<BR><BR><BR><BR>From: Onufryk, Peter [<A href="mailto:Peter.Onufryk@idt.com">mailto:Peter.Onufryk@idt.com</A>]<BR>Sent: Tuesday, March 19, 2013 7:22 AM<BR>To: Santosh Singh; <A href="mailto:technical@nvmexpress.org">technical@nvmexpress.org</A><BR>Subject: RE: NVMe 1.1 : Clarification on PRP2 Entry list<BR><BR>Santosh,<BR><BR>John and I discussed this and we believe that Figure 1 is correct and that<BR>the best way to clarify this is by adding a figure showing it to the spec.<BR>This will be the first item on the agenda in this week's calls.<BR><BR>Regards,<BR>Peter<BR><BR><BR>From: Santosh Singh [<A href="mailto:santosh.s2@samsung.com">mailto:santosh.s2@samsung.com</A>]<BR>Sent: Tuesday, March 19, 2013 5:49 AM<BR>To: <A href="mailto:technical@nvmexpress.org">technical@nvmexpress.org</A><ns0:technical_x0040_nvmexpress.org><BR>Subject: NVMe 1.1 : Clarification on PRP2 Entry list<BR><BR>Hi All,<BR><BR><BR>I got the query on PRP2 , when it is entry list and not memory page<BR>aligned from a design engineer .<BR><BR>The below paragraph is from section 4.3 'Physical Region Page Entry and<BR>List' of Spec 1.1.<BR><BR>[<A href="cid:Z5JE7EUABGFC@namo.co.kr">cid:Z5JE7EUABGFC@namo.co.kr</A>]<BR><BR>PRP entry 2, when pointing to a list may also have a non-zero offset within<BR>a memory page, means that is not memory page aligned.<BR>The last entry within a memory page shall be a list pointer. There are<BR>following two understandings for this:<BR><BR>1000<BR><BR>h<BR><BR>1<BR><BR>FFFh<BR><BR>PRP entry<BR><BR>2<BR><BR>pointing to a list<BR><BR>Entry<BR><BR>1<BR><BR>Entry<BR><BR>2<BR><BR>Entry<BR><BR>3<BR><BR>Entry<BR><BR>4<BR><BR>Address of PRP list<BR><BR>5000<BR><BR>h<BR><BR>5<BR><BR>FFFh<BR><BR>Entry<BR><BR>5<BR><BR>Entry<BR><BR>6<BR><BR>Entry<BR><BR>7<BR><BR>Entry<BR><BR>8<BR><BR>1000<BR><BR>h<BR><BR>1<BR><BR>FFFh<BR><BR>PRP entry<BR><BR>2<BR><BR>pointing to a list<BR><BR>Entry<BR><BR>1<BR><BR>Entry<BR><BR>2<BR><BR>Entry<BR><BR>3<BR><BR>Entry<BR><BR>4<BR><BR>2<BR><BR>FFFh<BR><BR>Entry<BR><BR>5<BR><BR>Entry<BR><BR>6<BR><BR>Entry<BR><BR>7<BR><BR>Entry<BR><BR>8<BR><BR>Entry<BR><BR>511<BR><BR>Entry<BR><BR>510<BR><BR>Address of PRP list<BR><BR>5000<BR><BR>h<BR><BR>5<BR><BR>FFFh<BR><BR>Entry<BR><BR>512<BR><BR>Entry<BR><BR>513<BR><BR>Entry<BR><BR>514<BR><BR>Fig:1<BR><BR>Fig:2<BR><BR>Page boundary<BR><BR><BR>So just want to verify with others that as per the line in section 4.3 'A<BR>physical region page list (PRP List) is a set of PRP entries in a single<BR>page of contiguous memory'<BR>fig:2 is the correct understanding.<BR><BR><BR>Thanks & Regards<BR>Santosh<BR><BR><BR><BR><BR><BR><BR><BR><BR><BR><BR>[<A href="cid:LK7CT9SZN3WZ@namo.co.kr">cid:LK7CT9SZN3WZ@namo.co.kr</A>]<BR><BR><BR></ns0:technical_x0040_nvmexpress.org><o:p></o:p></SPAN></P>
<P> <o:p></o:p></P>
<P> <o:p></o:p></P>
<P> <o:p></o:p></P>
<TABLE id=confidentialsignimg class=MsoNormalTable border=0 cellPadding=0>
<TBODY>
<TR>
<TD style="PADDING-BOTTOM: 0.75pt; PADDING-LEFT: 0.75pt; PADDING-RIGHT: 0.75pt; PADDING-TOP: 0.75pt">
<P><IMG id=_x0000_i1026 border=0 src="cid:4D4TEW9BEWXX@namo.co.kr" width=520 height=144><o:p></o:p></P></TD></TR></TBODY></TABLE>
<P style="MARGIN: 0in 0in 0pt" class=MsoNormal><IMG id=_x0000_i1027 border=0 src="http://ext.samsung.net/mailcheck/SeenTimeChecker?do=ba69d47c78c3acc08d47d8c18e24da0151171515984b9d550ad7d0699a0799098adfa564d3c39ac365b186a42a35dd3e259756a7cc35ba77326bbdfb2ea96a2fcf878f9a26ce15a0"><o:p></o:p></P></DIV></X-BODY>
<P> </P>
<P> </P><!--SP:santosh.s2--><!--santosh.s2:EP--></BODY></HTML><img src='http://ext.samsung.net/mailcheck/SeenTimeChecker?do=ba69d47c78c3acc0b6872ba05ecac647cb20e510c9bc8be1c883f74d3027808b94d548315d7b78b3f676ccf8e4fcda078aa631650c9c0a6c62e1ac75b522795a07805447a154a46fcf878f9a26ce15a0' border=0 width=0 height=0 style='display:none'>