<html xmlns:o="urn:schemas-microsoft-com:office:office" xmlns:w="urn:schemas-microsoft-com:office:word" xmlns:x="urn:schemas-microsoft-com:office:excel" xmlns="http://www.w3.org/TR/REC-html40">
<head>
<meta http-equiv=Content-Type content="text/html; charset=us-ascii">
<meta name=Generator content="Microsoft Word 11 (filtered medium)">
<style>
<!--
/* Style Definitions */
p.MsoNormal, li.MsoNormal, div.MsoNormal
{margin:0in;
margin-bottom:.0001pt;
font-size:12.0pt;
font-family:"Times New Roman";}
a:link, span.MsoHyperlink
{color:blue;
text-decoration:underline;}
a:visited, span.MsoHyperlinkFollowed
{color:purple;
text-decoration:underline;}
span.EmailStyle17
{mso-style-type:personal-compose;
font-family:Arial;
color:windowtext;}
@page Section1
{size:8.5in 11.0in;
margin:1.0in 1.25in 1.0in 1.25in;}
div.Section1
{page:Section1;}
-->
</style>
</head>
<body lang=EN-US link=blue vlink=purple>
<div class=Section1>
<p class=MsoNormal><font size=2 face=Arial><span style='font-size:10.0pt;
font-family:Arial'>Hi,<o:p></o:p></span></font></p>
<p class=MsoNormal><font size=2 face=Arial><span style='font-size:10.0pt;
font-family:Arial'><o:p> </o:p></span></font></p>
<p class=MsoNormal><font size=2 face=Arial><span style='font-size:10.0pt;
font-family:Arial'>I’ve been debugging an IB kernel driver and see that
sometimes we get a stuck send operation. I believe what happens is the send
actually happens but we don’t get a CQ completion callback. I’ve
been trying to track down the CORRECT programming semantics for CQ polling and
rearming. <o:p></o:p></span></font></p>
<p class=MsoNormal><font size=2 face=Arial><span style='font-size:10.0pt;
font-family:Arial'><o:p> </o:p></span></font></p>
<p class=MsoNormal><font size=2 face=Arial><span style='font-size:10.0pt;
font-family:Arial'>Looking in the Windows IB stack, I see in same cases were in
the completion callback routine, the CQ is rearmed BEFORE the CQ entries are
polled (like in the base mad processing code). In other places (like the IPoIB driver)
I see where it polls first, in a loop until no CQ entries are returned, and
then it rearms the CQ. I also found a document from 2003 from Intel called the
IB verb implementers guide (at infiniband.sourceforge.net/HWDrivers/HCA_DDK/VIG_SF.pdf),
and it very clearly states in section 8.3 you need to use what look like edge
triggered interrupt semantics to handle the race condition of polling and
rearming the CQ. Assuming the Intel document is correct, then the IB stack may
be getting stuck completions on occasion. <o:p></o:p></span></font></p>
<p class=MsoNormal><font size=2 face=Arial><span style='font-size:10.0pt;
font-family:Arial'><o:p> </o:p></span></font></p>
<p class=MsoNormal><font size=2 face=Arial><span style='font-size:10.0pt;
font-family:Arial'>Can anybody give a definite answer if the CQ trigger has
edge or level semantics, and what I need to do to assure CQ entries are always
processed without a delay? The docs for ib_rearm_cq seem to say something
different than the docs for ib_rearm_n_cq, so the docs aren’t much help
either.<o:p></o:p></span></font></p>
<p class=MsoNormal><font size=2 face=Arial><span style='font-size:10.0pt;
font-family:Arial'><o:p> </o:p></span></font></p>
<p class=MsoNormal><font size=2 face=Arial><span style='font-size:10.0pt;
font-family:Arial'>- Jan<o:p></o:p></span></font></p>
</div>
</body>
</html>