<html xmlns:v="urn:schemas-microsoft-com:vml" xmlns:o="urn:schemas-microsoft-com:office:office" xmlns:w="urn:schemas-microsoft-com:office:word" xmlns:m="http://schemas.microsoft.com/office/2004/12/omml" xmlns="http://www.w3.org/TR/REC-html40"><head><meta http-equiv=Content-Type content="text/html; charset=iso-8859-1"><meta name=Generator content="Microsoft Word 12 (filtered medium)"><!--[if !mso]><style>v\:* {behavior:url(#default#VML);}
o\:* {behavior:url(#default#VML);}
w\:* {behavior:url(#default#VML);}
.shape {behavior:url(#default#VML);}
</style><![endif]--><style><!--
/* Font Definitions */
@font-face
{font-family:Wingdings;
panose-1:5 0 0 0 0 0 0 0 0 0;}
@font-face
{font-family:"Cambria Math";
panose-1:2 4 5 3 5 4 6 3 2 4;}
@font-face
{font-family:Calibri;
panose-1:2 15 5 2 2 2 4 3 2 4;}
@font-face
{font-family:Tahoma;
panose-1:2 11 6 4 3 5 4 4 2 4;}
/* Style Definitions */
p.MsoNormal, li.MsoNormal, div.MsoNormal
{margin:0in;
margin-bottom:.0001pt;
font-size:11.0pt;
font-family:"Calibri","sans-serif";}
a:link, span.MsoHyperlink
{mso-style-priority:99;
color:blue;
text-decoration:underline;}
a:visited, span.MsoHyperlinkFollowed
{mso-style-priority:99;
color:purple;
text-decoration:underline;}
p.MsoAcetate, li.MsoAcetate, div.MsoAcetate
{mso-style-priority:99;
mso-style-link:"Balloon Text Char";
margin:0in;
margin-bottom:.0001pt;
font-size:8.0pt;
font-family:"Tahoma","sans-serif";}
span.BalloonTextChar
{mso-style-name:"Balloon Text Char";
mso-style-priority:99;
mso-style-link:"Balloon Text";
font-family:"Tahoma","sans-serif";}
span.EmailStyle19
{mso-style-type:personal;
font-family:"Calibri","sans-serif";
color:windowtext;
font-weight:normal;
font-style:normal;
text-decoration:none none;}
span.EmailStyle20
{mso-style-type:personal;
font-family:"Calibri","sans-serif";
color:#1F497D;
font-weight:normal;
font-style:normal;
text-decoration:none none;}
span.EmailStyle21
{mso-style-type:personal-reply;
font-family:"Calibri","sans-serif";
color:#1F497D;}
.MsoChpDefault
{mso-style-type:export-only;
font-size:10.0pt;}
@page WordSection1
{size:8.5in 11.0in;
margin:1.0in 1.0in 1.0in 1.0in;}
div.WordSection1
{page:WordSection1;}
--></style><!--[if gte mso 9]><xml>
<o:shapedefaults v:ext="edit" spidmax="1026" />
</xml><![endif]--><!--[if gte mso 9]><xml>
<o:shapelayout v:ext="edit">
<o:idmap v:ext="edit" data="1" />
</o:shapelayout></xml><![endif]--></head><body lang=EN-US link=blue vlink=purple><div class=WordSection1><p class=MsoNormal><span style='color:#1F497D'>Hi Paul,<o:p></o:p></span></p><p class=MsoNormal><span style='color:#1F497D'>I just reviewed the patch. The patch looks good to us. <o:p></o:p></span></p><p class=MsoNormal><span style='color:#1F497D'>Thanks <o:p></o:p></span></p><p class=MsoNormal><span style='color:#1F497D'>Arpit <o:p></o:p></span></p><p class=MsoNormal><span style='color:#1F497D'><o:p> </o:p></span></p><div><div style='border:none;border-top:solid #B5C4DF 1.0pt;padding:3.0pt 0in 0in 0in'><p class=MsoNormal><b><span style='font-size:10.0pt;font-family:"Tahoma","sans-serif"'>From:</span></b><span style='font-size:10.0pt;font-family:"Tahoma","sans-serif"'> nvmewin-bounces@lists.openfabrics.org [mailto:nvmewin-bounces@lists.openfabrics.org] <b>On Behalf Of </b>Luse, Paul E<br><b>Sent:</b> Friday, October 05, 2012 3:24 PM<br><b>To:</b> Chang, Alex; nvmewin@lists.openfabrics.org<br><b>Subject:</b> Re: [nvmewin] MSID 0 share w/IO queue patch<o:p></o:p></span></p></div></div><p class=MsoNormal><o:p> </o:p></p><p class=MsoNormal><span style='font-size:10.0pt;color:#1F497D'>Thanks Alex. Rick or Arpit, have you guys had a chance to review yet?<o:p></o:p></span></p><p class=MsoNormal><span style='font-size:10.0pt;color:#1F497D'><o:p> </o:p></span></p><p class=MsoNormal><span style='font-size:10.0pt;color:#1F497D'>Thx<br>Paul<o:p></o:p></span></p><p class=MsoNormal><span style='font-size:10.0pt;color:#1F497D'><o:p> </o:p></span></p><div><div style='border:none;border-top:solid #B5C4DF 1.0pt;padding:3.0pt 0in 0in 0in'><p class=MsoNormal><b><span style='font-size:10.0pt;font-family:"Tahoma","sans-serif"'>From:</span></b><span style='font-size:10.0pt;font-family:"Tahoma","sans-serif"'> Chang, Alex [<a href="mailto:Alex.Chang@idt.com">mailto:Alex.Chang@idt.com</a>] <br><b>Sent:</b> Friday, October 05, 2012 3:19 PM<br><b>To:</b> Luse, Paul E; <a href="mailto:nvmewin@lists.openfabrics.org">nvmewin@lists.openfabrics.org</a><br><b>Subject:</b> RE: MSID 0 share w/IO queue patch<o:p></o:p></span></p></div></div><p class=MsoNormal><o:p> </o:p></p><p class=MsoNormal><span style='font-size:10.0pt;font-family:"Arial","sans-serif";color:blue'>Hi Paul,</span><span style='font-size:12.0pt;font-family:"Times New Roman","serif"'><o:p></o:p></span></p><p class=MsoNormal><span style='font-size:12.0pt;font-family:"Times New Roman","serif"'> <o:p></o:p></span></p><p class=MsoNormal><span style='font-size:10.0pt;font-family:"Arial","sans-serif";color:blue'>We are good with the patch. Thanks a lot for the effort.</span><span style='font-size:12.0pt;font-family:"Times New Roman","serif"'><o:p></o:p></span></p><p class=MsoNormal><span style='font-size:12.0pt;font-family:"Times New Roman","serif"'> <o:p></o:p></span></p><p class=MsoNormal><span style='font-size:10.0pt;font-family:"Arial","sans-serif";color:blue'>Regards,</span><span style='font-size:12.0pt;font-family:"Times New Roman","serif"'><o:p></o:p></span></p><p class=MsoNormal><span style='font-size:10.0pt;font-family:"Arial","sans-serif";color:blue'>Alex</span><span style='font-size:12.0pt;font-family:"Times New Roman","serif"'><o:p></o:p></span></p><p class=MsoNormal><span style='font-size:12.0pt;font-family:"Times New Roman","serif"'><o:p> </o:p></span></p><div class=MsoNormal align=center style='text-align:center'><span style='font-size:12.0pt;font-family:"Times New Roman","serif"'><hr size=3 width="100%" align=center></span></div><p class=MsoNormal style='margin-bottom:12.0pt'><b><span style='font-size:10.0pt;font-family:"Tahoma","sans-serif"'>From:</span></b><span style='font-size:10.0pt;font-family:"Tahoma","sans-serif"'> <a href="mailto:nvmewin-bounces@lists.openfabrics.org">nvmewin-bounces@lists.openfabrics.org</a> <a href="mailto:[mailto:nvmewin-bounces@lists.openfabrics.org]">[mailto:nvmewin-bounces@lists.openfabrics.org]</a> <b>On Behalf Of </b>Luse, Paul E<br><b>Sent:</b> Monday, October 01, 2012 4:03 PM<br><b>To:</b> <a href="mailto:nvmewin@lists.openfabrics.org">nvmewin@lists.openfabrics.org</a><br><b>Subject:</b> [nvmewin] ***UNCHECKED*** MSID 0 share w/IO queue patch</span><span style='font-size:12.0pt;font-family:"Times New Roman","serif"'><o:p></o:p></span></p><p class=MsoNormal><span style='font-size:10.0pt'>All-<o:p></o:p></span></p><p class=MsoNormal><span style='font-size:10.0pt'><o:p> </o:p></span></p><p class=MsoNormal><span style='font-size:10.0pt'>Here is the final patch for admin queue sharing and a few other misc cleanup items. Fully tested and I also reviewed w/Alex and Kwok F2F last week but will let Alex approve for himself. PW is intel123<o:p></o:p></span></p><p class=MsoNormal><span style='font-size:10.0pt'><o:p> </o:p></span></p><p class=MsoNormal><span style='font-size:10.0pt'>Please let me know if there are any questions<o:p></o:p></span></p><p class=MsoNormal><span style='font-size:10.0pt'><o:p> </o:p></span></p><p class=MsoNormal><span style='font-size:10.0pt'>Thx<o:p></o:p></span></p><p class=MsoNormal><span style='font-size:10.0pt'>Paul<o:p></o:p></span></p><p class=MsoNormal><span style='font-size:10.0pt'><o:p> </o:p></span></p><p class=MsoNormal><span style='font-size:10.0pt'><o:p> </o:p></span></p><p class=MsoNormal><span style='font-size:10.0pt'>nvmeInit.c:<o:p></o:p></span></p><p class=MsoNormal><span style='font-size:10.0pt'>- changes in loops and conditions to account for the number of cores now being the same as the number of vectors requested<o:p></o:p></span></p><p class=MsoNormal><span style='font-size:10.0pt'>- removed unused element pMsiMsgTbl->CoreNum<o:p></o:p></span></p><p class=MsoNormal><span style='font-size:10.0pt'>- removed unused element pMMT->CoreNum<o:p></o:p></span></p><p class=MsoNormal><span style='font-size:10.0pt'>- in NVMeMsiMapCores(), now init the MsgId in the core table to the CQ number -1 (same as core #)<o:p></o:p></span></p><p class=MsoNormal><span style='font-size:10.0pt'>- in DBG mode, track when learning is complete to support an assertion to make sure learning is always working<o:p></o:p></span></p><p class=MsoNormal><span style='font-size:10.0pt'>- for the identify commands during the init state machine, we were DMA’ing directly into elements within devExt structures with no assurance of alignment. To address this I changed the target address of the xfer to use the driver state machine data buffer and copy it over on completion<o:p></o:p></span></p><p class=MsoNormal><span style='font-size:10.0pt'>- NVMeAllocIoQueues(): fix to use an index that counts up through both core and NUMA loops (what Alex saw) instead of the inner loop. This val is used to index into the coreTable, the QueuId value will continue to behave as before and wrap at the number of queues<o:p></o:p></span></p><p class=MsoNormal><span style='font-size:10.0pt'><o:p> </o:p></span></p><p class=MsoNormal><span style='font-size:10.0pt'>nvmeIo.c:<o:p></o:p></span></p><p class=MsoNormal><span style='font-size:10.0pt'>- two changes both wrapped in DBG, one to print PRP details and one to init the core # element in the srbExt used to make sure learning continues to do its job (check submitting core & completing core)<o:p></o:p></span></p><p class=MsoNormal><span style='font-size:10.0pt'><o:p> </o:p></span></p><p class=MsoNormal><span style='font-size:10.0pt'>nvmeStat.c<o:p></o:p></span></p><p class=MsoNormal><span style='font-size:10.0pt'>- changes in loops and conditions to account for the number of cores now being the same as the number of vectors requested<o:p></o:p></span></p><p class=MsoNormal><span style='font-size:10.0pt'>- init of the debug var used for learning mode<o:p></o:p></span></p><p class=MsoNormal><span style='font-size:10.0pt'><o:p> </o:p></span></p><p class=MsoNormal><span style='font-size:10.0pt'>nvmeStd.c<o:p></o:p></span></p><p class=MsoNormal><span style='font-size:10.0pt'>- alloc one less entry for the MsgTable<o:p></o:p></span></p><p class=MsoNormal><span style='font-size:10.0pt'>- in the completion routines (either DPC or ISR):<o:p></o:p></span></p><p class=MsoNormal><span style='font-size:10.0pt'>-- switched the logic to check for the common case first (shared == FALSE)<o:p></o:p></span></p><p class=MsoNormal><span style='font-size:10.0pt'>-- got rid of the learningMode var as we detect now based on startState as follows:<o:p></o:p></span></p><p class=MsoNormal><span style='font-size:10.0pt'>If we’re done with the init State machine:<o:p></o:p></span></p><p class=MsoNormal><span style='font-size:10.0pt;font-family:Wingdings'>à</span><span style='font-size:10.0pt'> use the msg table to figure out which queue to look in<o:p></o:p></span></p><p class=MsoNormal><span style='font-size:10.0pt'>Else if we’re in learning mode:<o:p></o:p></span></p><p class=MsoNormal><span style='font-size:10.0pt;font-family:Wingdings'>à</span><span style='font-size:10.0pt'> use the msgId + 1. Recall that when we alloc’d the queues we setup the CQ’s such that QP 1 would use MSID0, QP 2 MSID 1, etc. Learning mode will loop through all of the QPs by walking the core table 0..number of cores. In the event that there are fewer QPs than cores because of an HBA limitation, this still works we just learn each queue more than once which does not hurt anything. Clearly when this happens things will not be optimal, they can’t be without enough QPs, however we’ll still fully utilize all of the available queue pairs<o:p></o:p></span></p><p class=MsoNormal><span style='font-size:10.0pt'>- the loop has changed as the previous for loop didn’t have the flexibility to check 2 non back-back queues. The QP that shared MSIX0 could be any of the other queues. I reworked the loops to be 2 do while loops and for an actual admin queue request we’ll just check the admin queue, for the shared IO queue however we have to always check the admin queue as well. This logic is at the bottom of the loop and is fairly straightforward<o:p></o:p></span></p><p class=MsoNormal><span style='font-size:10.0pt'><o:p> </o:p></span></p><p class=MsoNormal><span style='font-size:10.0pt'>nvmeStd.h:<o:p></o:p></span></p><p class=MsoNormal><span style='font-size:10.0pt'>- a few supporting changes - obvious <o:p></o:p></span></p><p class=MsoNormal><span style='font-size:10.0pt'><o:p> </o:p></span></p><p class=MsoNormal><span style='font-size:10.0pt'>Also made a few changes enar the end following review w/IDT:<o:p></o:p></span></p><p class=MsoNormal><span style='font-size:10.0pt'>- replaced Rtl copy commands with storport copy commands<o:p></o:p></span></p><p class=MsoNormal><span style='font-size:10.0pt'>- replaced Rtl zero mem commands with memset<o:p></o:p></span></p><p class=MsoNormal><span style='font-size:10.0pt'>- added print at the end of learning mode to see updated mappings (initial mappings still print)<o:p></o:p></span></p><p class=MsoNormal><span style='font-size:10.0pt'><o:p> </o:p></span></p><p class=MsoNormal><span style='font-size:10.0pt'><o:p> </o:p></span></p><p class=MsoNormal><span style='font-size:10.0pt'><o:p> </o:p></span></p><p class=MsoNormal><span style='font-size:10.0pt;font-family:"Tahoma","sans-serif"'>____________________________________</span><span style='font-size:12.0pt;font-family:"Arial","sans-serif"'><o:p></o:p></span></p><p class=MsoNormal><span style='font-size:10.0pt;font-family:"Tahoma","sans-serif"'>Paul Luse<br>Sr. Staff Engineer<br>PCG Server Software Engineering <br>Desk: 480.554.3688, Mobile: 480.334.4630</span><span style='font-size:12.0pt;font-family:"Arial","sans-serif"'><o:p></o:p></span></p><p class=MsoNormal><o:p> </o:p></p></div></body></html>