<html>
<head>
<meta http-equiv="Content-Type" content="text/html; charset=us-ascii">
<meta name="Generator" content="Microsoft Exchange Server">
<!-- converted from rtf -->
<style><!-- .EmailQuote { margin-left: 1pt; padding-left: 4pt; border-left: #800000 2px solid; } --></style>
</head>
<body>
<font face="Arial" size="2"><span style="font-size:10pt;">
<div>Hi Paul,</div>
<div> </div>
<div>Sure, I would like to try it out your changes later…</div>
<div> </div>
<div>Thanks,</div>
<div>Alex</div>
<div> </div>
<div>-----Original Message-----</div>
<div>From: Luse, Paul E [<a href="mailto:paul.e.luse@intel.com"><font color="blue"><u>mailto:paul.e.luse@intel.com</u></font></a>] </div>
<div>Sent: Monday, September 24, 2012 2:19 PM</div>
<div>To: Chang, Alex</div>
<div>Cc: nvmewin@lists.openfabrics.org</div>
<div>Subject: RE: [nvmewin] Issues Found</div>
<div> </div>
<div>I'm not sure what you are seeing Alex, but the MSID does not depend on the mapping, it depends on what vector we put in the CQ when we created it. I'm not sure what you think you fixed but lets you and I grab some time and review my patch before doing
much more. I have a feeling that you are seeing the impact of your change - if you just delete that queueid=0 line then you haven't done enough, that's not a complete fix. My change to share MSID0 with a queue addresses what you are seeing simply by how its
implemented.</div>
<div> </div>
<div>I'm actually in San Jose this week, might be able to whip over later this week if that's easier. I'll send out the new code this eve.</div>
<div> </div>
<div>-----Original Message-----</div>
<div>From: Chang, Alex [<a href="mailto:Alex.Chang@idt.com"><font color="blue"><u>mailto:Alex.Chang@idt.com</u></font></a>]</div>
<div>Sent: Monday, September 24, 2012 2:11 PM</div>
<div>To: Luse, Paul E</div>
<div>Cc: nvmewin@lists.openfabrics.org</div>
<div>Subject: RE: [nvmewin] Issues Found</div>
<div> </div>
<div>Hi Paul,</div>
<div> </div>
<div>As for the 2nd item I brought up, I believe we have set up a mapping between cores and queues before learning:</div>
<div>Cores# Queues#</div>
<div>0 1</div>
<div>1 2</div>
<div>2 3</div>
<div>...</div>
<div>While learning, we decide which queue to use based on the above mappings. When commands complete, the value of MsgID depends on the APIC settings, which is the purpose of learning. In other words, MsgId is not necessarily equal to QID. After fixing Item#
1, I have seen the failure of Driver State machine with Dbg build driver due to timeout in the learning state.</div>
<div> </div>
<div>Alex</div>
<div> </div>
<div> </div>
<div>-----Original Message-----</div>
<div>From: Luse, Paul E [<a href="mailto:paul.e.luse@intel.com"><font color="blue"><u>mailto:paul.e.luse@intel.com</u></font></a>]</div>
<div>Sent: Monday, September 24, 2012 1:46 PM</div>
<div>To: Luse, Paul E; Chang, Alex</div>
<div>Cc: nvmewin@lists.openfabrics.org</div>
<div>Subject: RE: [nvmewin] Issues Found</div>
<div> </div>
<div>So I found myself with some time here and was able to prepare my next patch, I won't send it out til I have a chance to test it but I wanted to run through all the changes real quick before I responded.</div>
<div> </div>
<div>The changes I'll be sending out are primarily focused around sharing the admin queue MSIX with one other queue (which ones depends on learning). This came about because of a bug report from someone running on a 32 core system - in this case we ask for
33 vectors and when we don't get them we end up dropping to 1 and sharing everything. This, of course, works but under heavy IO it causes DPC watchdog timeouts simply due to the amount of time we spend looking through all the queues processing IOs. The load
in question was 32 workers (iometer) and 64 IO depth with 512B reads. There are several different ways we could address this but the one I'm suggesting as a generic improvement is to have the admin queue share with another queue so that we require an even
number of vectors and can readily support 32 cores which is a pretty common config.</div>
<div> </div>
<div>In the process of putting this together I ran into the item you mention below Alex so have already fixed that. Had not previously tested on a system with multiple NUMA nodes but clearly with that LOC in there, we don't init enough queues, we setup 32
allright but we do the same set of 16 twice. So, this is fixed in my patch.</div>
<div> </div>
<div>On your second question, good question BYW, this is one of the reasons why learning mode works. We know which queue to look in only because we are still in learning mode and we set the queues up so that we can count on QID==MsgId. Remember, we're learning
the association between MSIX vector and completing core, then updating the tables and deleting/recreating the CQ so once learning is done we use the table but before we count on how we set things up.</div>
<div> </div>
<div>On your 3rd question, we didn't write or test that code, I forget who added it but I would consider it untested and a prime candidate for anyone wanting to contribute :) We at Intel will be looking more closely at that code in the coming months.</div>
<div> </div>
<div>Anyway, hope that answers your questions and I'll send out my patch either tonight that includes the fix the first item below, some additional debug prints (via compile switch) to dump our PRP info as you go, a few additional assert, etc. Its not very
big. After that we'll be coming with a series of AER fixes.</div>
<div> </div>
<div>Thx</div>
<div>Paul</div>
<div> </div>
<div> </div>
<div>-----Original Message-----</div>
<div>From: nvmewin-bounces@lists.openfabrics.org [<a href="mailto:nvmewin-bounces@lists.openfabrics.org"><font color="blue"><u>mailto:nvmewin-bounces@lists.openfabrics.org</u></font></a>] On Behalf Of Luse, Paul E</div>
<div>Sent: Monday, September 24, 2012 11:59 AM</div>
<div>To: Chang, Alex</div>
<div>Cc: nvmewin@lists.openfabrics.org</div>
<div>Subject: Re: [nvmewin] Issues Found</div>
<div> </div>
<div>Alex</div>
<div> </div>
<div>I'll cover your questions this evening plus have some non bug fix changes in some of these areas anyways.</div>
<div> </div>
<div>Thx</div>
<div>Paul</div>
<div> </div>
<div>Sent from my iPhone</div>
<div> </div>
<div>On Sep 24, 2012, at 9:46 AM, "Chang, Alex" <Alex.Chang@idt.com<mailto:Alex.Chang@idt.com>> wrote:</div>
<div> </div>
<div>Hi Paul,</div>
<div> </div>
<div>When testing the latest patch I added, I came across couple issues in the driver:</div>
<div>1. In the patch you sent out on July 13 (later tagged as misc_bug_fixes_and_enum), within NVMeAllocIoQueues function, you reset the QueueID for each NUMA node loop as below:</div>
<div> for (Node = 0; Node < pRMT->NumNumaNodes; Node++) {</div>
<div> pNNT = pRMT->pNumaNodeTbl + Node;</div>
<div> QueueID = 0;</div>
<div> for (Core = pNNT->FirstCoreNum; Core <= pNNT->LastCoreNum; Core++) { It turns out only allocating the number of cores of a given NUMA node for the entire system. I wonder why?</div>
<div> </div>
<div>2. When the driver is in learning phase where it tries to find out the mappings between cores and MSI vectors, in IoCompletionDpcRoutine, the driver limits the pending completion entry checking based on MsgID:</div>
<div> if (!learning) {</div>
<div> firstCheckQueue = lastCheckQueue = pMMT->CplQueueNum;</div>
<div> } else {</div>
<div> firstCheckQueue = lastCheckQueue = (USHORT)MsgID;</div>
<div> }</div>
<div>Since it's still in learning phase, shouldn't it look up every created completion queue to find out the mapping?</div>
<div> </div>
<div>3. In NVMeInitialize, the driver call StorPortInitializePerfOpts in both normal and Crashdump/Hibernation cases. The routine returns failure and I wonder if it makes sense to call it in Crashdump/Hibernation case.</div>
<div> </div>
<div>Thanks,</div>
<div>Alex</div>
<div> </div>
<div>_______________________________________________</div>
<div>nvmewin mailing list</div>
<div>nvmewin@lists.openfabrics.org<mailto:nvmewin@lists.openfabrics.org></div>
<div><a href="http://lists.openfabrics.org/cgi-bin/mailman/listinfo/nvmewin"><font color="blue"><u>http://lists.openfabrics.org/cgi-bin/mailman/listinfo/nvmewin</u></font></a></div>
<div>_______________________________________________</div>
<div>nvmewin mailing list</div>
<div>nvmewin@lists.openfabrics.org</div>
<div><a href="http://lists.openfabrics.org/cgi-bin/mailman/listinfo/nvmewin"><font color="blue"><u>http://lists.openfabrics.org/cgi-bin/mailman/listinfo/nvmewin</u></font></a></div>
<div> </div>
</span></font>
</body>
</html>