<html xmlns:v="urn:schemas-microsoft-com:vml" xmlns:o="urn:schemas-microsoft-com:office:office" xmlns:w="urn:schemas-microsoft-com:office:word" xmlns:m="http://schemas.microsoft.com/office/2004/12/omml" xmlns="http://www.w3.org/TR/REC-html40">
<head>
<meta http-equiv="Content-Type" content="text/html; charset=utf-8">
<meta name="Generator" content="Microsoft Word 15 (filtered medium)">
<!--[if !mso]><style>v\:* {behavior:url(#default#VML);}
o\:* {behavior:url(#default#VML);}
w\:* {behavior:url(#default#VML);}
.shape {behavior:url(#default#VML);}
</style><![endif]--><style><!--
/* Font Definitions */
@font-face
{font-family:"Cambria Math";
panose-1:2 4 5 3 5 4 6 3 2 4;}
@font-face
{font-family:Calibri;
panose-1:2 15 5 2 2 2 4 3 2 4;}
/* Style Definitions */
p.MsoNormal, li.MsoNormal, div.MsoNormal
{margin:0in;
margin-bottom:.0001pt;
font-size:12.0pt;
font-family:"Times New Roman",serif;}
a:link, span.MsoHyperlink
{mso-style-priority:99;
color:blue;
text-decoration:underline;}
a:visited, span.MsoHyperlinkFollowed
{mso-style-priority:99;
color:purple;
text-decoration:underline;}
span.EmailStyle17
{mso-style-type:personal-reply;
font-family:"Calibri",sans-serif;
color:#1F497D;
font-weight:normal;
font-style:normal;}
.MsoChpDefault
{mso-style-type:export-only;
font-family:"Calibri",sans-serif;}
@page WordSection1
{size:8.5in 11.0in;
margin:1.0in 1.0in 1.0in 1.0in;}
div.WordSection1
{page:WordSection1;}
--></style><!--[if gte mso 9]><xml>
<o:shapedefaults v:ext="edit" spidmax="1026" />
</xml><![endif]--><!--[if gte mso 9]><xml>
<o:shapelayout v:ext="edit">
<o:idmap v:ext="edit" data="1" />
</o:shapelayout></xml><![endif]-->
</head>
<body lang="EN-US" link="blue" vlink="purple">
<div class="WordSection1">
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:"Calibri",sans-serif;color:#1F497D">Hi Wenqian,<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:"Calibri",sans-serif;color:#1F497D"><o:p> </o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:"Calibri",sans-serif;color:#1F497D">I understand what you are stating. If you are observing behavior in the OFA driver that is allocating more entries than is defined via CAP.MQES, then this is
a bug. <o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:"Calibri",sans-serif;color:#1F497D"><o:p> </o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:"Calibri",sans-serif;color:#1F497D">Would you be able to supply a patch to the OFA driver and send it out for review (to this mailing list)?<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:"Calibri",sans-serif;color:#1F497D"><o:p> </o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:"Calibri",sans-serif;color:#1F497D">Thanks,<br>
Ray<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:"Calibri",sans-serif;color:#1F497D"><o:p> </o:p></span></p>
<p class="MsoNormal"><b><span style="font-size:11.0pt;font-family:"Calibri",sans-serif">From:</span></b><span style="font-size:11.0pt;font-family:"Calibri",sans-serif"> Wenqian Wu [mailto:wuwq85@gmail.com]
<br>
<b>Sent:</b> Monday, June 15, 2015 3:32 PM<br>
<b>To:</b> Robles, Raymond C; nancyx@marvell.com; renlei@marvell.com; Joyce Yang<br>
<b>Cc:</b> nvmewin@lists.openfabrics.org<br>
<b>Subject:</b> Re: [nvmewin] nvmewin Digest, Vol 38, Issue 9<o:p></o:p></span></p>
<p class="MsoNormal"><o:p> </o:p></p>
<div>
<p class="MsoNormal">Hi Ray,<o:p></o:p></p>
<div>
<p class="MsoNormal"><o:p> </o:p></p>
</div>
<div>
<p class="MsoNormal">I brought up this question again due to we still found the issue using Windows OFA driver. Basically the controller reports supports 32 entries per queue, but the driver will do some calculation and try to create a IO queue with 64 entries
per queue. Then controller will report error. <o:p></o:p></p>
</div>
<div>
<p class="MsoNormal"><o:p> </o:p></p>
</div>
<div>
<p class="MsoNormal">Linux/Windows inbox driver didn't see such issue.<o:p></o:p></p>
</div>
<div>
<p class="MsoNormal"><o:p> </o:p></p>
</div>
<div>
<p class="MsoNormal">NVme spec only states the queue base address needs to be page aligned, but not the queue size(number of entries). Please reconsider it. Thanks!<o:p></o:p></p>
</div>
<div>
<p class="MsoNormal"><img width="563" height="300" id="_x0000_i1025" src="cid:image002.png@01D0A9F0.95B09FD0" alt="Inline image 1"><o:p></o:p></p>
</div>
<div>
<p class="MsoNormal"><o:p> </o:p></p>
</div>
<div>
<p class="MsoNormal">Thanks,<o:p></o:p></p>
</div>
<div>
<p class="MsoNormal">Wenqian<o:p></o:p></p>
</div>
</div>
<div>
<p class="MsoNormal"><o:p> </o:p></p>
<div>
<p class="MsoNormal">On Tue, Feb 17, 2015 at 4:17 PM, Robles, Raymond C <<a href="mailto:raymond.c.robles@intel.com" target="_blank">raymond.c.robles@intel.com</a>> wrote:<o:p></o:p></p>
<blockquote style="border:none;border-left:solid #CCCCCC 1.0pt;padding:0in 0in 0in 6.0pt;margin-left:4.8pt;margin-right:0in">
<div>
<div>
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto"><span style="font-size:10.0pt;font-family:"Calibri",sans-serif;color:#1F497D">Hi,</span><o:p></o:p></p>
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto"><span style="font-size:10.0pt;font-family:"Calibri",sans-serif;color:#1F497D"> </span><o:p></o:p></p>
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto"><span style="font-size:10.0pt;font-family:"Calibri",sans-serif;color:#1F497D">The code below simply rounds up the memory allocated to the next page boundary. The NVMe controller
expects this memory (queue size) to be host page aligned. Queue size will still be within the limits of what the controller communicates.</span><o:p></o:p></p>
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto"><span style="font-size:10.0pt;font-family:"Calibri",sans-serif;color:#1F497D"> </span><o:p></o:p></p>
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto"><span style="font-size:10.0pt;font-family:"Calibri",sans-serif;color:#1F497D">Thanks,</span><o:p></o:p></p>
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto"><span style="font-size:10.0pt;font-family:"Calibri",sans-serif;color:#1F497D">Ray</span><o:p></o:p></p>
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto"><span style="font-size:10.0pt;font-family:"Calibri",sans-serif;color:#1F497D"> </span><o:p></o:p></p>
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto"><b><span style="font-size:11.0pt;font-family:"Calibri",sans-serif">From:</span></b><span style="font-size:11.0pt;font-family:"Calibri",sans-serif">
<a href="mailto:nvmewin-bounces@lists.openfabrics.org" target="_blank">nvmewin-bounces@lists.openfabrics.org</a> [mailto:<a href="mailto:nvmewin-bounces@lists.openfabrics.org" target="_blank">nvmewin-bounces@lists.openfabrics.org</a>]
<b>On Behalf Of </b>Wenqian Wu<br>
<b>Sent:</b> Sunday, February 15, 2015 3:43 PM<br>
<b>To:</b> <a href="mailto:nvmewin@lists.openfabrics.org" target="_blank">nvmewin@lists.openfabrics.org</a><br>
<b>Subject:</b> Re: [nvmewin] nvmewin Digest, Vol 38, Issue 9</span><o:p></o:p></p>
<div>
<div>
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto"> <o:p></o:p></p>
<div>
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto">Hi Ray,<o:p></o:p></p>
<div>
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto"> <o:p></o:p></p>
</div>
<div>
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto">Thanks for your reply. I understand your saying that before calling <span style="font-size:9.5pt">NVMeAllocQueues, the CAP.MQES has been checked and passed in as 3rd parameter(QEntries).
But if you look further in NVMeAllocQueues function, QEntires will be modified.</span><o:p></o:p></p>
</div>
<div>
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto"> <o:p></o:p></p>
</div>
<div>
<div>
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto"><span style="font-size:9.5pt"> if ((QEntries % SysPageSizeInSubEntries) != 0)</span><o:p></o:p></p>
</div>
<div>
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto"><span style="font-size:9.5pt"> QEntries = (QEntries + SysPageSizeInSubEntries) &</span><o:p></o:p></p>
</div>
<div>
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto"><span style="font-size:9.5pt"> ~(SysPageSizeInSubEntries - 1);</span><o:p></o:p></p>
</div>
<div>
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto"><span style="font-size:9.5pt"> </span><o:p></o:p></p>
</div>
</div>
<div>
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto"><span style="font-size:9.5pt">And finally QEntires will be used to create the Queue. But QEntries may already bigger the CAP.MQES after the above round up.</span><o:p></o:p></p>
</div>
<div>
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto"><span style="font-size:9.5pt"> </span><o:p></o:p></p>
</div>
<div>
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto"><span style="font-size:9.5pt"><img border="0" width="472" height="150" id="_x0000_i1026" src="cid:image003.png@01D0A9F0.95B09FD0" alt="Inline image 1"></span><o:p></o:p></p>
</div>
<div>
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto"><span style="font-size:9.5pt"> </span><o:p></o:p></p>
</div>
<div>
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto"><span style="font-size:9.5pt">A simple example is if controller reports CAP.MQES is 32, host memory page size is 4K. The round up procedure will change QEntries to 64 (4 * 1024
/ 64), and used to create the queue. Controller will returen Invalid Queue Size and failed the command.</span><o:p></o:p></p>
</div>
<div>
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto"><span style="font-size:9.5pt"> </span><o:p></o:p></p>
</div>
<div>
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto"><span style="font-size:9.5pt">In summary, I think the alignment requirement is for host memory only. Driver can still allocate 4K(page aligned) for 32 entries for the above example,
just keeping the second half unused. But when creating the queue, the entries should be 32, instead of 64.</span><o:p></o:p></p>
</div>
<div>
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto"><span style="font-size:9.5pt"> </span><o:p></o:p></p>
</div>
<div>
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto"><span style="font-size:9.5pt">Please let me know if this makes sense.</span><o:p></o:p></p>
</div>
<div>
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto"><span style="font-size:9.5pt"> </span><o:p></o:p></p>
</div>
<div>
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto"><span style="font-size:9.5pt">Thanks,</span><o:p></o:p></p>
</div>
<div>
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto"><span style="font-size:9.5pt">Wenqian</span><o:p></o:p></p>
</div>
<div>
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto"><span style="font-size:9.5pt"> </span><o:p></o:p></p>
</div>
</div>
<div>
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto"> <o:p></o:p></p>
<div>
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto">On Fri, Feb 13, 2015 at 12:00 PM, <<a href="mailto:nvmewin-request@lists.openfabrics.org" target="_blank">nvmewin-request@lists.openfabrics.org</a>> wrote:<o:p></o:p></p>
<blockquote style="border:none;border-left:solid #CCCCCC 1.0pt;padding:0in 0in 0in 6.0pt;margin-left:4.8pt;margin-top:5.0pt;margin-right:0in;margin-bottom:5.0pt">
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto">Send nvmewin mailing list submissions to<br>
<a href="mailto:nvmewin@lists.openfabrics.org" target="_blank">nvmewin@lists.openfabrics.org</a><br>
<br>
To subscribe or unsubscribe via the World Wide Web, visit<br>
<a href="http://lists.openfabrics.org/mailman/listinfo/nvmewin" target="_blank">
http://lists.openfabrics.org/mailman/listinfo/nvmewin</a><br>
or, via email, send a message with subject or body 'help' to<br>
<a href="mailto:nvmewin-request@lists.openfabrics.org" target="_blank">nvmewin-request@lists.openfabrics.org</a><br>
<br>
You can reach the person managing the list at<br>
<a href="mailto:nvmewin-owner@lists.openfabrics.org" target="_blank">nvmewin-owner@lists.openfabrics.org</a><br>
<br>
When replying, please edit your Subject line so it is more specific<br>
than "Re: Contents of nvmewin digest..."<br>
<br>
<br>
Today's Topics:<br>
<br>
1. Re: NVMe Queue entry number (Robles, Raymond C)<br>
<br>
<br>
----------------------------------------------------------------------<br>
<br>
Message: 1<br>
Date: Fri, 13 Feb 2015 04:42:37 +0000<br>
From: "Robles, Raymond C" <<a href="mailto:raymond.c.robles@intel.com" target="_blank">raymond.c.robles@intel.com</a>><br>
To: 'Wenqian Wu' <<a href="mailto:wuwq85@gmail.com" target="_blank">wuwq85@gmail.com</a>>, "<a href="mailto:nvmewin@lists.openfabrics.org" target="_blank">nvmewin@lists.openfabrics.org</a>"<br>
<<a href="mailto:nvmewin@lists.openfabrics.org" target="_blank">nvmewin@lists.openfabrics.org</a>><br>
Subject: Re: [nvmewin] NVMe Queue entry number<br>
Message-ID:<br>
<<a href="mailto:49158E750348AA499168FD41D88983606B5DA191@fmsmsx117.amr.corp.intel.com" target="_blank">49158E750348AA499168FD41D88983606B5DA191@fmsmsx117.amr.corp.intel.com</a>><br>
<br>
Content-Type: text/plain; charset="utf-8"<br>
<br>
Hi Wenqian,<br>
<br>
Thank you for your inquiry. In the FindAdapter routine (in nvmeStd.c), the driver checks CAP.MQES and saves the value in pAE->InitInfo.IoQEntries (if the default or register override is smaller than CAP.MQES, it is saved instead). Within the function NVMeAllocIoQueues,
there is loop that will iterate to create queue pairs based on the number of cpu cores.<br>
<br>
In this loop, a call to NVMeAllocQueues is made. Just prior to this call, the value saved in pAE->InitInfo.IoQEntires is retrieved (stack variable ?QEntres?) and passed in as the 3rd parameter. So, once the allocation is taking place where you mention below,
the 3rd parameter of the function has already been checked against CAP.MQES. Also, per the NVMe spec sections 5.3 and 5.4 ? Figures 33 and 37 (create completion queue and submission queue PRP 1), all queue memory must be ?physically contiguous and memory page
aligned?.<br>
<br>
Thanks<br>
Ray<br>
<br>
From: <a href="mailto:nvmewin-bounces@lists.openfabrics.org" target="_blank">nvmewin-bounces@lists.openfabrics.org</a> [mailto:<a href="mailto:nvmewin-bounces@lists.openfabrics.org" target="_blank">nvmewin-bounces@lists.openfabrics.org</a>] On Behalf Of Wenqian
Wu<br>
Sent: Wednesday, February 11, 2015 4:58 PM<br>
To: <a href="mailto:nvmewin@lists.openfabrics.org" target="_blank">nvmewin@lists.openfabrics.org</a><br>
Subject: [nvmewin] NVMe Queue entry number<br>
<br>
Hi OFA driver member,<br>
<br>
I have one question regarding the Queue entry size. The driver will allocate number of entries aligned to memory page (line877, nvmeInit.c), instead of the actual queue size controller supports (CAP.MQES). Controller can return error if host request more entries
while ignoring controller's capability. I think as long as the base address is page aligned, there is no reason to make the entries number aligned to page boundary. Can this be considered a driver bug or is there any particular consideration?<br>
<br>
Thanks,<br>
Wenqian<br>
-------------- next part --------------<br>
An HTML attachment was scrubbed...<br>
URL: <<a href="http://lists.openfabrics.org/pipermail/nvmewin/attachments/20150213/edd26ac2/attachment-0001.html" target="_blank">http://lists.openfabrics.org/pipermail/nvmewin/attachments/20150213/edd26ac2/attachment-0001.html</a>><br>
<br>
------------------------------<br>
<br>
_______________________________________________<br>
nvmewin mailing list<br>
<a href="mailto:nvmewin@lists.openfabrics.org" target="_blank">nvmewin@lists.openfabrics.org</a><br>
<a href="http://lists.openfabrics.org/mailman/listinfo/nvmewin" target="_blank">http://lists.openfabrics.org/mailman/listinfo/nvmewin</a><br>
<br>
<br>
End of nvmewin Digest, Vol 38, Issue 9<br>
**************************************<o:p></o:p></p>
</blockquote>
</div>
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto"> <o:p></o:p></p>
</div>
</div>
</div>
</div>
</div>
</blockquote>
</div>
<p class="MsoNormal"><o:p> </o:p></p>
</div>
</div>
</body>
</html>