<html xmlns:v="urn:schemas-microsoft-com:vml" xmlns:o="urn:schemas-microsoft-com:office:office" xmlns:w="urn:schemas-microsoft-com:office:word" xmlns:m="http://schemas.microsoft.com/office/2004/12/omml" xmlns="http://www.w3.org/TR/REC-html40">
<head>
<meta http-equiv=Content-Type content="text/html; charset=us-ascii">
<meta name=Generator content="Microsoft Word 12 (filtered medium)">
<style>
<!--
/* Font Definitions */
@font-face
{font-family:"Cambria Math";
panose-1:2 4 5 3 5 4 6 3 2 4;}
@font-face
{font-family:Calibri;
panose-1:2 15 5 2 2 2 4 3 2 4;}
/* Style Definitions */
p.MsoNormal, li.MsoNormal, div.MsoNormal
{margin:0in;
margin-bottom:.0001pt;
font-size:12.0pt;
font-family:"Times New Roman","serif";}
a:link, span.MsoHyperlink
{mso-style-priority:99;
color:blue;
text-decoration:underline;}
a:visited, span.MsoHyperlinkFollowed
{mso-style-priority:99;
color:purple;
text-decoration:underline;}
p
{mso-style-priority:99;
mso-margin-top-alt:auto;
margin-right:0in;
mso-margin-bottom-alt:auto;
margin-left:0in;
font-size:12.0pt;
font-family:"Times New Roman","serif";}
span.EmailStyle18
{mso-style-type:personal-reply;
font-family:"Calibri","sans-serif";
color:#1F497D;}
.MsoChpDefault
{mso-style-type:export-only;
font-size:10.0pt;}
@page Section1
{size:8.5in 11.0in;
margin:1.0in 1.0in 1.0in 1.0in;}
div.Section1
{page:Section1;}
-->
</style>
<!--[if gte mso 9]><xml>
<o:shapedefaults v:ext="edit" spidmax="1026" />
</xml><![endif]--><!--[if gte mso 9]><xml>
<o:shapelayout v:ext="edit">
<o:idmap v:ext="edit" data="1" />
</o:shapelayout></xml><![endif]-->
</head>
<body lang=EN-US link=blue vlink=purple>
<div class=Section1>
<p class=MsoNormal><span style='font-size:11.0pt;font-family:"Calibri","sans-serif";
color:#1F497D'>Hi Yossi,<o:p></o:p></span></p>
<p class=MsoNormal><span style='font-size:11.0pt;font-family:"Calibri","sans-serif";
color:#1F497D'><o:p> </o:p></span></p>
<p class=MsoNormal><span style='font-size:11.0pt;font-family:"Calibri","sans-serif";
color:#1F497D'>A question about r538:<o:p></o:p></span></p>
<p><span style='font-size:10.0pt;font-family:"Arial","sans-serif"'>------------------------------------------------------------------------<br>
r538 | sleybo | 2006-11-07 08:54:25 +0200 (Tue, 07 Nov 2006) | 3 lines<o:p></o:p></span></p>
<p class=MsoNormal><span style='font-size:10.0pt;font-family:"Arial","sans-serif"'>[IBAL]
Compliance tests<br>
1. pass switch_info to the HCA - compliance test C13-026<br>
2. Not use AL cashe for node_description node_info to force Mkey check
-compliance test C14-018<br>
------------------------------------------------------------------------<br>
<br>
</span><span style='font-size:11.0pt;font-family:"Calibri","sans-serif";
color:#1F497D'><o:p></o:p></span></p>
<p class=MsoNormal><span style='font-size:11.0pt;font-family:"Calibri","sans-serif";
color:#1F497D'>Have you tested to see what the effects of removing the cache
for node description and node info are on SM sweeps when the system is busy?<o:p></o:p></span></p>
<p class=MsoNormal><span style='font-size:11.0pt;font-family:"Calibri","sans-serif";
color:#1F497D'><o:p> </o:p></span></p>
<p class=MsoNormal><span style='font-size:11.0pt;font-family:"Calibri","sans-serif";
color:#1F497D'>I initially added the cache for these so that the response could
be issued in the context of the CQ callback for the special QP (thus at
DISPATCH_LEVEL). Without the cache processing requires a call to the
local MAD verb, which has to be scheduled on a passive-level thread. If
the system is very busy doing I/O (i.e. lots of small packets in Iometer over
IPoIB), I have seen cases where the local MAD thread does not run fast enough
so the response time for the MAD is too long and the SM declares the node as
having failed and removes it from the fabric. This is pretty nasty, as
suddenly all IB multicast group memberships are lost, but there’s no
indication to the host that things went awry.<o:p></o:p></span></p>
<p class=MsoNormal><span style='font-size:11.0pt;font-family:"Calibri","sans-serif";
color:#1F497D'><o:p> </o:p></span></p>
<p class=MsoNormal><span style='font-size:11.0pt;font-family:"Calibri","sans-serif";
color:#1F497D'>There are two solutions for this, one is more of a temporary fix
than the other IMO. First, the temporary fix: perform the MKey check in
software, so that the MAD response for as many MADs can be generated at
DISPATCH_LEVEL from the context of the special QP’s CQ callback.
This should maintain compliance while also keeping response times for MADs as
short as possible.<o:p></o:p></span></p>
<p class=MsoNormal><span style='font-size:11.0pt;font-family:"Calibri","sans-serif";
color:#1F497D'><o:p> </o:p></span></p>
<p class=MsoNormal><span style='font-size:11.0pt;font-family:"Calibri","sans-serif";
color:#1F497D'>The second solution is to make the local MAD verb
asynchronous. The HCA handles the command asynchronously anyway, so this
is a more natural fit given the HW design. This would mean the local MAD
verb would be called directly from the CQ callback (at DISPATCH_LEVEL), and
would return pending. When the local MAD is processed and the HCA
generates the response to the EQ, the driver could invoke a callback to
indicate completion (again at DISPATCH_LEVEL) which would send out the
response. This solution eliminates the thread scheduling issues
associated with handling local MAD requests in a passive-level thread.<o:p></o:p></span></p>
<p class=MsoNormal><span style='font-size:11.0pt;font-family:"Calibri","sans-serif";
color:#1F497D'><o:p> </o:p></span></p>
<p class=MsoNormal><span style='font-size:11.0pt;font-family:"Calibri","sans-serif";
color:#1F497D'>We should make sure that systems aren’t susceptible to
Denial of Service attacks from someone flooding them with IPoIB traffic (which
gets handled at DISPATCH_LEVEL in IPoIB’s CQ callback). It’s
bad if an application on one host can cause another host to be removed from the
fabric – there will be no port down events, no notification to the SM
when the host is responsive again, and the host will not be able to participate
properly in the fabric until the next SM sweep.<o:p></o:p></span></p>
<p class=MsoNormal><span style='font-size:11.0pt;font-family:"Calibri","sans-serif";
color:#1F497D'> <o:p></o:p></span></p>
<p class=MsoNormal><span style='font-size:11.0pt;font-family:"Calibri","sans-serif";
color:#1F497D'>-Fab</span><span style='font-size:10.0pt;font-family:"Arial","sans-serif"'><o:p></o:p></span></p>
</div>
</body>
</html>