<html xmlns:v="urn:schemas-microsoft-com:vml" xmlns:o="urn:schemas-microsoft-com:office:office" xmlns:w="urn:schemas-microsoft-com:office:word" xmlns:m="http://schemas.microsoft.com/office/2004/12/omml" xmlns="http://www.w3.org/TR/REC-html40">
<head>
<meta http-equiv="Content-Type" content="text/html; charset=iso-8859-1">
<meta name="Generator" content="Microsoft Word 12 (filtered medium)">
<style><!--
/* Font Definitions */
@font-face
{font-family:Wingdings;
panose-1:5 0 0 0 0 0 0 0 0 0;}
@font-face
{font-family:"Cambria Math";
panose-1:2 4 5 3 5 4 6 3 2 4;}
@font-face
{font-family:Calibri;
panose-1:2 15 5 2 2 2 4 3 2 4;}
/* Style Definitions */
p.MsoNormal, li.MsoNormal, div.MsoNormal
{margin:0in;
margin-bottom:.0001pt;
font-size:12.0pt;
font-family:"Times New Roman","serif";}
a:link, span.MsoHyperlink
{mso-style-priority:99;
color:blue;
text-decoration:underline;}
a:visited, span.MsoHyperlinkFollowed
{mso-style-priority:99;
color:purple;
text-decoration:underline;}
p.MsoListParagraph, li.MsoListParagraph, div.MsoListParagraph
{mso-style-priority:34;
margin-top:0in;
margin-right:0in;
margin-bottom:0in;
margin-left:.5in;
margin-bottom:.0001pt;
font-size:12.0pt;
font-family:"Times New Roman","serif";}
p.emailquote, li.emailquote, div.emailquote
{mso-style-name:emailquote;
mso-margin-top-alt:auto;
margin-right:0in;
mso-margin-bottom-alt:auto;
margin-left:1.0pt;
font-size:12.0pt;
font-family:"Times New Roman","serif";}
span.EmailStyle18
{mso-style-type:personal-reply;
font-family:"Calibri","sans-serif";
color:#1F497D;}
.MsoChpDefault
{mso-style-type:export-only;
font-size:10.0pt;}
@page WordSection1
{size:8.5in 11.0in;
margin:1.0in 1.0in 1.0in 1.0in;}
div.WordSection1
{page:WordSection1;}
/* List Definitions */
@list l0
{mso-list-id:1234311374;
mso-list-type:hybrid;
mso-list-template-ids:-1744388740 67698709 67698713 67698715 67698703 67698713 67698715 67698703 67698713 67698715;}
@list l0:level1
{mso-level-number-format:alpha-upper;
mso-level-tab-stop:none;
mso-level-number-position:left;
text-indent:-.25in;}
ol
{margin-bottom:0in;}
ul
{margin-bottom:0in;}
--></style><!--[if gte mso 9]><xml>
<o:shapedefaults v:ext="edit" spidmax="1026" />
</xml><![endif]--><!--[if gte mso 9]><xml>
<o:shapelayout v:ext="edit">
<o:idmap v:ext="edit" data="1" />
</o:shapelayout></xml><![endif]-->
</head>
<body lang="EN-US" link="blue" vlink="purple">
<div class="WordSection1">
<div>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:"Calibri","sans-serif";color:#1F497D">All,
<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:"Calibri","sans-serif";color:#1F497D"><o:p> </o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:"Calibri","sans-serif";color:black">Under Windows Server 2012, I’ve seen a crash where NVMeStartIo() gets interrupted by our ISR at a time when it’s in the middle of manipulating a linked list
critical data structure which the ISR then goes on to attempt to manipulate also - which results in a crash. Below is the call stack – see where I’ve inserted the comment</span><span style="font-size:11.0pt;font-family:"Calibri","sans-serif";color:#1F497D"> “</span><b><span style="font-size:9.0pt;font-family:Wingdings;color:red">ß</span></b><b><span style="font-size:9.0pt;font-family:"Calibri","sans-serif";color:red">-STARTIO
PATH GETS CLOBBERED BY OUR INTERRUPT HANDLER BECAUSE WE AREN’T HOLDING THE INTERRUPT SPIN LOCK”<o:p></o:p></span></b></p>
<p class="MsoNormal"><b><span style="font-size:9.0pt;font-family:"Calibri","sans-serif";color:red"><o:p> </o:p></span></b></p>
<p class="MsoNormal" style="margin-left:.5in"><span style="font-size:9.0pt;font-family:"Calibri","sans-serif";color:#1F497D">2: kd> kc<o:p></o:p></span></p>
<p class="MsoNormal" style="margin-left:.5in"><span style="font-size:9.0pt;font-family:"Calibri","sans-serif";color:#1F497D">Call Site<o:p></o:p></span></p>
<p class="MsoNormal" style="margin-left:.5in"><span style="font-size:9.0pt;font-family:"Calibri","sans-serif";color:#1F497D">nt!RtlpBreakWithStatusInstruction<o:p></o:p></span></p>
<p class="MsoNormal" style="margin-left:.5in"><span style="font-size:9.0pt;font-family:"Calibri","sans-serif";color:#1F497D">nt!KiBugCheckDebugBreak<o:p></o:p></span></p>
<p class="MsoNormal" style="margin-left:.5in"><span style="font-size:9.0pt;font-family:"Calibri","sans-serif";color:#1F497D">nt!KeBugCheck2<o:p></o:p></span></p>
<p class="MsoNormal" style="margin-left:.5in"><span style="font-size:9.0pt;font-family:"Calibri","sans-serif";color:#1F497D">nt!KeBugCheckEx<o:p></o:p></span></p>
<p class="MsoNormal" style="margin-left:.5in"><span style="font-size:9.0pt;font-family:"Calibri","sans-serif";color:#1F497D">nt!KiBugCheckDispatch<o:p></o:p></span></p>
<p class="MsoNormal" style="margin-left:.5in"><span style="font-size:9.0pt;font-family:"Calibri","sans-serif";color:#1F497D">nt!KiFastFailDispatch<o:p></o:p></span></p>
<p class="MsoNormal" style="margin-left:.5in"><span style="font-size:9.0pt;font-family:"Calibri","sans-serif";color:#1F497D">nt!KiRaiseSecurityCheckFailure<o:p></o:p></span></p>
<p class="MsoNormal" style="margin-left:.5in"><span style="font-size:9.0pt;font-family:"Calibri","sans-serif";color:#1F497D">nvme!RtlFailFast<o:p></o:p></span></p>
<p class="MsoNormal" style="margin-left:.5in"><span style="font-size:9.0pt;font-family:"Calibri","sans-serif";color:#1F497D">nvme!FatalListEntryError<o:p></o:p></span></p>
<p class="MsoNormal" style="margin-left:.5in"><span style="font-size:9.0pt;font-family:"Calibri","sans-serif";color:#1F497D">nvme!RtlpCheckListEntry<o:p></o:p></span></p>
<p class="MsoNormal" style="margin-left:.5in"><span style="font-size:9.0pt;font-family:"Calibri","sans-serif";color:#1F497D">nvme!InsertTailList<o:p></o:p></span></p>
<p class="MsoNormal" style="margin-left:.5in"><span style="font-size:9.0pt;font-family:"Calibri","sans-serif";color:#1F497D">nvme!NVMeCompleteCmd<o:p></o:p></span></p>
<p class="MsoNormal" style="margin-left:.5in"><span style="font-size:9.0pt;font-family:"Calibri","sans-serif";color:#1F497D">nvme!NVMeIsrMsix<o:p></o:p></span></p>
<p class="MsoNormal" style="margin-left:.5in"><span style="font-size:9.0pt;font-family:"Calibri","sans-serif";color:#1F497D">nt!KiInterruptDispatch
</span><b><span style="font-size:9.0pt;font-family:"Calibri","sans-serif";color:red"> </span></b><b><span style="font-size:9.0pt;font-family:Wingdings;color:red">ß</span></b><b><span style="font-size:9.0pt;font-family:"Calibri","sans-serif";color:red">-STARTIO
PATH GETS CLOBBERED BY OUR INTERRUPT HANDLER BECAUSE WE AREN’T HOLDING THE INTERRUPT SPIN LOCK<o:p></o:p></span></b></p>
<p class="MsoNormal" style="margin-left:.5in"><span style="font-size:9.0pt;font-family:"Calibri","sans-serif";color:#1F497D">nvme!RemoveHeadList<o:p></o:p></span></p>
<p class="MsoNormal" style="margin-left:.5in"><span style="font-size:9.0pt;font-family:"Calibri","sans-serif";color:#1F497D">nvme!NVMeGetCmdEntry<o:p></o:p></span></p>
<p class="MsoNormal" style="margin-left:.5in"><span style="font-size:9.0pt;font-family:"Calibri","sans-serif";color:#1F497D">nvme!ProcessIo<o:p></o:p></span></p>
<p class="MsoNormal" style="margin-left:.5in"><span style="font-size:9.0pt;font-family:"Calibri","sans-serif";color:#1F497D">nvme!NVMeStartIo<o:p></o:p></span></p>
<p class="MsoNormal" style="margin-left:.5in"><span style="font-size:9.0pt;font-family:"Calibri","sans-serif";color:#1F497D">storport!RaidpAdapterContinueScatterGather<o:p></o:p></span></p>
<p class="MsoNormal" style="margin-left:.5in"><span style="font-size:9.0pt;font-family:"Calibri","sans-serif";color:#1F497D">hal!HalpAllocateAdapterCallbackV2<o:p></o:p></span></p>
<p class="MsoNormal" style="margin-left:.5in"><span style="font-size:9.0pt;font-family:"Calibri","sans-serif";color:#1F497D">hal!IoFreeAdapterChannelV2<o:p></o:p></span></p>
<p class="MsoNormal" style="margin-left:.5in"><span style="font-size:9.0pt;font-family:"Calibri","sans-serif";color:#1F497D">hal!HalAllocateAdapterChannelV2<o:p></o:p></span></p>
<p class="MsoNormal" style="margin-left:.5in"><span style="font-size:9.0pt;font-family:"Calibri","sans-serif";color:#1F497D">hal!HalBuildScatterGatherListV2<o:p></o:p></span></p>
<p class="MsoNormal" style="margin-left:.5in"><span style="font-size:9.0pt;font-family:"Calibri","sans-serif";color:#1F497D">storport!RaUnitStartIo<o:p></o:p></span></p>
<p class="MsoNormal" style="margin-left:.5in"><span style="font-size:9.0pt;font-family:"Calibri","sans-serif";color:#1F497D">storport!RaidUnitCompleteRequest<o:p></o:p></span></p>
<p class="MsoNormal" style="margin-left:.5in"><span style="font-size:9.0pt;font-family:"Calibri","sans-serif";color:#1F497D">storport!RaidpAdapterRedirectDpcRoutine<o:p></o:p></span></p>
<p class="MsoNormal" style="margin-left:.5in"><span style="font-size:9.0pt;font-family:"Calibri","sans-serif";color:#1F497D">nt!KiExecuteAllDpcs<o:p></o:p></span></p>
<p class="MsoNormal" style="margin-left:.5in"><span style="font-size:9.0pt;font-family:"Calibri","sans-serif";color:#1F497D">nt!KiRetireDpcList</span><span style="font-size:11.0pt;font-family:"Calibri","sans-serif";color:#1F497D"><o:p></o:p></span></p>
<p class="MsoNormal"><b><span style="font-size:9.0pt;font-family:"Calibri","sans-serif";color:red"><o:p> </o:p></span></b></p>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:"Calibri","sans-serif";color:black">I looked through the code and noticed we never call StorPortAcquireSpinLock to acquire the InterruptLock to protect us from such pre-emption. Another way to achieve
this would be to indicate we run at half-duplex rather than full-duplex but that would degrade the general performance of the driver.
<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:"Calibri","sans-serif";color:black"> <o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:"Calibri","sans-serif";color:black">I’m not sure why we didn’t run into this way before now – is there some other re-entrance protection algorithm besides the two above that others are aware of?
If not, I believe we need to fix this asap.</span><span style="font-size:11.0pt;font-family:"Calibri","sans-serif";color:black"> Suggestions:<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:"Calibri","sans-serif";color:#1F497D"><o:p> </o:p></span></p>
<p class="MsoListParagraph" style="text-indent:-.25in;mso-list:l0 level1 lfo1"><![if !supportLists]><span style="font-size:11.0pt;font-family:"Calibri","sans-serif";color:black"><span style="mso-list:Ignore">A.<span style="font:7.0pt "Times New Roman"">
</span></span></span><![endif]><span style="font-size:11.0pt;font-family:"Calibri","sans-serif";color:black">Simplest approach is to lock down all of NVMeStartIo as per below (not tested yet) but we almost may as well run half-duplex if we do this:<o:p></o:p></span></p>
<p class="MsoNormal" style="margin-left:.5in"><span style="font-size:9.0pt;font-family:"Calibri","sans-serif";color:black"><o:p> </o:p></span></p>
<p class="MsoNormal" style="margin-left:.5in"><span style="font-size:9.0pt;font-family:"Calibri","sans-serif";color:black">1 . At the very the top of NVMeStartIo:<o:p></o:p></span></p>
<p class="MsoNormal" style="margin-left:.5in"><span style="font-size:9.0pt;font-family:"Calibri","sans-serif";color:black"><o:p> </o:p></span></p>
<p class="MsoNormal" style="margin-left:.5in;text-autospace:none"><span style="font-size:9.0pt;font-family:"Courier New";color:green">/* we should never be holding the interrupt lock upon entry to NVMeStartIo.<o:p></o:p></span></p>
<p class="MsoNormal" style="margin-left:.5in;text-autospace:none"><span style="font-size:9.0pt;font-family:"Courier New";color:green"> * Acquire the Interrupt Spin Lock to protect against getting hit by our ISR.
<o:p></o:p></span></p>
<p class="MsoNormal" style="margin-left:.5in;text-autospace:none"><span style="font-size:9.0pt;font-family:"Courier New";color:green"> */<o:p></o:p></span></p>
<p class="MsoNormal" style="margin-left:.5in;text-autospace:none"><span style="font-size:9.0pt;font-family:"Courier New"">
<o:p></o:p></span></p>
<p class="MsoNormal" style="margin-left:.5in;text-autospace:none"><span style="font-size:9.0pt;font-family:"Courier New"">
<span style="color:blue">if</span> (NULL == pAdapterExtension->hInterruptLock) {<o:p></o:p></span></p>
<p class="MsoNormal" style="margin-left:.5in;text-autospace:none"><span style="font-size:9.0pt;font-family:"Courier New""> (StorPortAcquireSpinLock(pAdapterExtension,<o:p></o:p></span></p>
<p class="MsoNormal" style="margin-left:.5in;text-autospace:none"><span style="font-size:9.0pt;font-family:"Courier New""> InterruptLock,<o:p></o:p></span></p>
<p class="MsoNormal" style="margin-left:.5in;text-autospace:none"><span style="font-size:9.0pt;font-family:"Courier New""> NULL,<o:p></o:p></span></p>
<p class="MsoNormal" style="margin-left:.5in;text-autospace:none"><span style="font-size:9.0pt;font-family:"Courier New""> &pAdapterExtension->hInterruptLock);<o:p></o:p></span></p>
<p class="MsoNormal" style="margin-left:.5in;text-autospace:none"><span style="font-size:9.0pt;font-family:"Courier New""> }<o:p></o:p></span></p>
<p class="MsoNormal" style="margin-left:.5in;text-autospace:none"><span style="font-size:9.0pt;font-family:"Courier New"">
<span style="color:blue">else</span> {<o:p></o:p></span></p>
<p class="MsoNormal" style="margin-left:.5in;text-autospace:none"><span style="font-size:9.0pt;font-family:"Courier New""> ASSERT(FALSE);<o:p></o:p></span></p>
<p class="MsoNormal" style="margin-left:.5in"><span style="font-size:9.0pt;font-family:"Courier New""> }<o:p></o:p></span></p>
<p class="MsoNormal" style="margin-left:.5in"><span style="font-size:9.0pt;font-family:"Courier New""><o:p> </o:p></span></p>
<p class="MsoNormal" style="margin-left:.5in"><span style="font-size:9.0pt;font-family:"Courier New"">2. </span><span style="font-size:9.0pt;font-family:"Calibri","sans-serif";color:black">At the very the top of
</span><span style="font-size:9.0pt;font-family:"Courier New"">IO_StorPortNotification</span><span style="font-size:9.0pt;font-family:"Calibri","sans-serif";color:black"><o:p></o:p></span></p>
<p class="MsoNormal" style="margin-left:.5in;text-autospace:none"><span style="font-size:9.0pt;font-family:"Courier New""><o:p> </o:p></span></p>
<p class="MsoNormal" style="margin-left:.5in;text-autospace:none"><span style="font-size:9.0pt;font-family:"Courier New""> PNVME_DEVICE_EXTENSION pAE = (PNVME_DEVICE_EXTENSION) pHwDeviceExtension;<o:p></o:p></span></p>
<p class="MsoNormal" style="margin-left:.5in;text-autospace:none"><span style="font-size:9.0pt;font-family:"Courier New""><o:p> </o:p></span></p>
<p class="MsoNormal" style="margin-left:.5in;text-autospace:none"><span style="font-size:9.0pt;font-family:"Courier New"">
<span style="color:green">/* if we got here from NvmeStartIo we need to release the interrupt lock */<o:p></o:p></span></span></p>
<p class="MsoNormal" style="margin-left:.5in;text-autospace:none"><span style="font-size:9.0pt;font-family:"Courier New"">
<span style="color:blue">if</span> (NULL != pAE->hInterruptLock) {<o:p></o:p></span></p>
<p class="MsoNormal" style="margin-left:.5in;text-autospace:none"><span style="font-size:9.0pt;font-family:"Courier New""> STOR_LOCK_HANDLE hInterruptLockCopy = pAE->hInterruptLock;<o:p></o:p></span></p>
<p class="MsoNormal" style="margin-left:.5in;text-autospace:none"><span style="font-size:9.0pt;font-family:"Courier New""> pAE->hInterruptLock = NULL;<o:p></o:p></span></p>
<p class="MsoNormal" style="margin-left:.5in;text-autospace:none"><span style="font-size:9.0pt;font-family:"Courier New""> StorPortReleaseSpinLock(pAE, &hInterruptLockCopy);<o:p></o:p></span></p>
<p class="MsoNormal" style="margin-left:.5in;text-autospace:none"><span style="font-size:9.0pt;font-family:"Courier New""> }<o:p></o:p></span></p>
<p class="MsoNormal" style="margin-left:.5in;text-autospace:none"><span style="font-size:9.0pt;font-family:"Courier New""><o:p> </o:p></span></p>
<p class="MsoNormal" style="margin-left:.5in"><span style="font-size:9.0pt;font-family:"Courier New""><o:p> </o:p></span></p>
<p class="MsoNormal" style="margin-left:.5in"><span style="font-size:9.0pt;font-family:"Courier New"">3. At the very bottom of NVMeStartIo:<o:p></o:p></span></p>
<p class="MsoNormal" style="margin-left:.5in;text-autospace:none"><span style="font-size:9.0pt;font-family:"Courier New""> <o:p></o:p></span></p>
<p class="MsoNormal" style="margin-left:.5in;text-autospace:none"><span style="font-size:9.0pt;font-family:"Courier New"">
<span style="color:green">/* if we didn't release the Interrupt Lock in one of the calls to
<o:p></o:p></span></span></p>
<p class="MsoNormal" style="margin-left:.5in;text-autospace:none"><span style="font-size:9.0pt;font-family:"Courier New";color:green"> * IO_StorPortNotification above we need to release before we exit NVMEStartIo<o:p></o:p></span></p>
<p class="MsoNormal" style="margin-left:.5in;text-autospace:none"><span style="font-size:9.0pt;font-family:"Courier New";color:green"> */<o:p></o:p></span></p>
<p class="MsoNormal" style="margin-left:.5in;text-autospace:none"><span style="font-size:9.0pt;font-family:"Courier New";color:green"><o:p> </o:p></span></p>
<p class="MsoNormal" style="margin-left:.5in;text-autospace:none"><span style="font-size:9.0pt;font-family:"Courier New"">
<span style="color:blue">if</span> (NULL != pAE->hInterruptLock) {<o:p></o:p></span></p>
<p class="MsoNormal" style="margin-left:.5in;text-autospace:none"><span style="font-size:9.0pt;font-family:"Courier New""> STOR_LOCK_HANDLE hInterruptLockCopy = pAE->hInterruptLock;<o:p></o:p></span></p>
<p class="MsoNormal" style="margin-left:.5in;text-autospace:none"><span style="font-size:9.0pt;font-family:"Courier New""> pAE->hInterruptLock = NULL;<o:p></o:p></span></p>
<p class="MsoNormal" style="margin-left:.5in;text-autospace:none"><span style="font-size:9.0pt;font-family:"Courier New""> StorPortReleaseSpinLock(pAE, &hInterruptLockCopy);<o:p></o:p></span></p>
<p class="MsoNormal" style="margin-left:.5in;text-autospace:none"><span style="font-size:9.0pt;font-family:"Courier New""> }<o:p></o:p></span></p>
<p class="MsoNormal" style="margin-left:.5in;text-autospace:none"><span style="font-size:9.0pt;font-family:"Courier New""><o:p> </o:p></span></p>
<p class="MsoNormal" style="margin-left:.5in;text-autospace:none"><span style="font-size:9.0pt;font-family:"Courier New"">
<span style="color:blue">return</span> TRUE;<o:p></o:p></span></p>
<p class="MsoNormal" style="margin-left:.5in;text-autospace:none"><span style="font-size:9.0pt;font-family:"Courier New"">}
<span style="color:green">/* NVMeStartIo */</span></span><span style="font-size:10.0pt;font-family:"Courier New";color:green"><o:p></o:p></span></p>
<p class="MsoNormal" style="margin-left:.5in;text-autospace:none"><span style="font-size:10.0pt;font-family:"Courier New";color:green"><o:p> </o:p></span></p>
<p class="MsoListParagraph" style="text-indent:-.25in;mso-list:l0 level1 lfo1"><![if !supportLists]><span style="font-size:11.0pt;font-family:"Calibri","sans-serif";color:black"><span style="mso-list:Ignore">B.<span style="font:7.0pt "Times New Roman"">
</span></span></span><![endif]><span style="font-size:11.0pt;font-family:"Calibri","sans-serif";color:black">Better approach is to just lock ProcessIo(). But code exists in that routine which acquires the StartIo lock – we can’t take locks out of order or
we’ll cause deadlock. Right now that code never gets invoked – what was it for? Do we still need it? Can ProcessIo() get called from non-StartIo Paths? Can it get called multiple times? Not having been involved in the initial development of this driver, I
would need to study the flow to make sure to respect the StorPort lock acquiring/releasing hierarchy rules at all times. If those conversant in the overall developmental history and architecture of this driver could share their thoughts, that would be great.<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:"Calibri","sans-serif";color:black"><o:p> </o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:"Calibri","sans-serif";color:black">Thanks,<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:"Calibri","sans-serif";color:black"><o:p> </o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:"Calibri","sans-serif";color:black">Judy<o:p></o:p></span></p>
<p class="MsoNormal" style="margin-left:.5in"><span style="font-size:11.0pt;font-family:"Calibri","sans-serif";color:#1F497D"><o:p> </o:p></span></p>
</div>
</div>
</body>
</html>