<html xmlns:v="urn:schemas-microsoft-com:vml" xmlns:o="urn:schemas-microsoft-com:office:office" xmlns:w="urn:schemas-microsoft-com:office:word" xmlns:x="urn:schemas-microsoft-com:office:excel" xmlns:p="urn:schemas-microsoft-com:office:powerpoint" xmlns:a="urn:schemas-microsoft-com:office:access" xmlns:dt="uuid:C2F41010-65B3-11d1-A29F-00AA00C14882" xmlns:s="uuid:BDC6E3F0-6DA3-11d1-A2A3-00AA00C14882" xmlns:rs="urn:schemas-microsoft-com:rowset" xmlns:z="#RowsetSchema" xmlns:b="urn:schemas-microsoft-com:office:publisher" xmlns:ss="urn:schemas-microsoft-com:office:spreadsheet" xmlns:oa="urn:schemas-microsoft-com:office:activation" xmlns:html="http://www.w3.org/TR/REC-html40" xmlns:q="http://schemas.xmlsoap.org/soap/envelope/" xmlns:D="DAV:" xmlns:x2="http://schemas.microsoft.com/office/excel/2003/xml" xmlns:ois="http://schemas.microsoft.com/sharepoint/soap/ois/" xmlns:dir="http://schemas.microsoft.com/sharepoint/soap/directory/" xmlns:ds="http://www.w3.org/2000/09/xmldsig#" xmlns:dsp="http://schemas.microsoft.com/sharepoint/dsp" xmlns:udc="http://schemas.microsoft.com/data/udc" xmlns:xsd="http://www.w3.org/2001/XMLSchema" xmlns:sps="http://schemas.microsoft.com/sharepoint/soap/" xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance" xmlns:udcxf="http://schemas.microsoft.com/data/udc/xmlfile" xmlns:wf="http://schemas.microsoft.com/sharepoint/soap/workflow/" xmlns:mver="http://schemas.openxmlformats.org/markup-compatibility/2006" xmlns:m="http://schemas.microsoft.com/office/2004/12/omml" xmlns:ex12t="http://schemas.microsoft.com/exchange/services/2006/types" xmlns="http://www.w3.org/TR/REC-html40">
<head>
<META HTTP-EQUIV="Content-Type" CONTENT="text/html; charset=us-ascii">
<meta name=Generator content="Microsoft Word 12 (filtered medium)">
<!--[if !mso]>
<style>
v\:* {behavior:url(#default#VML);}
o\:* {behavior:url(#default#VML);}
w\:* {behavior:url(#default#VML);}
.shape {behavior:url(#default#VML);}
</style>
<![endif]-->
<style>
<!--
/* Font Definitions */
@font-face
{font-family:"Cambria Math";
panose-1:2 4 5 3 5 4 6 3 2 4;}
@font-face
{font-family:Calibri;
panose-1:2 15 5 2 2 2 4 3 2 4;}
@font-face
{font-family:Tahoma;
panose-1:2 11 6 4 3 5 4 4 2 4;}
/* Style Definitions */
p.MsoNormal, li.MsoNormal, div.MsoNormal
{margin:0in;
margin-bottom:.0001pt;
font-size:12.0pt;
font-family:"Times New Roman","serif";}
a:link, span.MsoHyperlink
{mso-style-priority:99;
color:blue;
text-decoration:underline;}
a:visited, span.MsoHyperlinkFollowed
{mso-style-priority:99;
color:purple;
text-decoration:underline;}
p
{mso-style-priority:99;
mso-margin-top-alt:auto;
margin-right:0in;
mso-margin-bottom-alt:auto;
margin-left:0in;
font-size:12.0pt;
font-family:"Times New Roman","serif";}
span.EmailStyle18
{mso-style-type:personal;
font-family:"Calibri","sans-serif";
color:#1F497D;}
span.EmailStyle19
{mso-style-type:personal-reply;
font-family:"Calibri","sans-serif";
color:#1F497D;}
.MsoChpDefault
{mso-style-type:export-only;
font-size:10.0pt;}
@page Section1
{size:8.5in 11.0in;
margin:1.0in 1.0in 1.0in 1.0in;}
div.Section1
{page:Section1;}
-->
</style>
<!--[if gte mso 9]><xml>
<o:shapedefaults v:ext="edit" spidmax="1026" />
</xml><![endif]--><!--[if gte mso 9]><xml>
<o:shapelayout v:ext="edit">
<o:idmap v:ext="edit" data="1" />
</o:shapelayout></xml><![endif]-->
</head>
<body lang=EN-US link=blue vlink=purple>
<div class=Section1>
<p class=MsoNormal><span style='font-size:11.0pt;font-family:"Calibri","sans-serif";
color:#1F497D'>You should ideally be able to cache PKey and GID tables. The
HCA driver’s SMP cache is setup to do that, but I don’t remember why I didn’t
set it up to actually do it.<o:p></o:p></span></p>
<p class=MsoNormal><span style='font-size:11.0pt;font-family:"Calibri","sans-serif";
color:#1F497D'><o:p> </o:p></span></p>
<p class=MsoNormal><span style='font-size:11.0pt;font-family:"Calibri","sans-serif";
color:#1F497D'>Note however that the SMP cache in the HCA driver is still
invoked from the passive level thread, so it doesn’t quite solve the problem. I
don’t know if the cache should be in IBAL. If implemented in the HCA driver, I
think using a passive thread (or even just a work item could provide async
local MAD functionality even if the underlying HCA driver implementation blocks,
allowing the elimination of the local mad complexity in IBAL.<o:p></o:p></span></p>
<p class=MsoNormal><span style='font-size:11.0pt;font-family:"Calibri","sans-serif";
color:#1F497D'><o:p> </o:p></span></p>
<p class=MsoNormal><span style='font-size:11.0pt;font-family:"Calibri","sans-serif";
color:#1F497D'>-Fab<o:p></o:p></span></p>
<p class=MsoNormal><span style='font-size:11.0pt;font-family:"Calibri","sans-serif";
color:#1F497D'><o:p> </o:p></span></p>
<div>
<div style='border:none;border-top:solid #B5C4DF 1.0pt;padding:3.0pt 0in 0in 0in'>
<p class=MsoNormal><b><span style='font-size:10.0pt;font-family:"Tahoma","sans-serif"'>From:</span></b><span
style='font-size:10.0pt;font-family:"Tahoma","sans-serif"'> Yossi Leybovich
[mailto:sleybo@dev.mellanox.co.il] <br>
<b>Sent:</b> Tuesday, January 30, 2007 6:47 AM<br>
<b>To:</b> 'Yossi Leybovich'; Fab Tillier; openib-windows@openib.org<br>
<b>Subject:</b> RE: [Openib-windows] [ANNOUNCE] Build 1.0.0.566 posted<o:p></o:p></span></p>
</div>
</div>
<p class=MsoNormal><o:p> </o:p></p>
<div>
<p class=MsoNormal><span style='font-size:10.0pt;font-family:"Arial","sans-serif";
color:blue'>Two more things.</span><o:p></o:p></p>
</div>
<div>
<p class=MsoNormal> <o:p></o:p></p>
</div>
<div>
<p class=MsoNormal><span style='font-size:10.0pt;font-family:"Arial","sans-serif";
color:blue'>1. pls note that even in case that the simple m-key check is good
we still need to reset m-key lease timer (which is used by HW/FW ) </span><o:p></o:p></p>
</div>
<div>
<p class=MsoNormal><span style='font-size:10.0pt;font-family:"Arial","sans-serif";
color:blue'>so we still need to forward the first good packet to the FW.</span><o:p></o:p></p>
</div>
<div>
<p class=MsoNormal><span style='font-size:10.0pt;font-family:"Arial","sans-serif";
color:blue'>2. Does there any reason why we not keep GID/Pkey tables in cache ?
Cant we answer this packets from cache ?</span><o:p></o:p></p>
</div>
<div>
<p class=MsoNormal> <o:p></o:p></p>
</div>
<div>
<p class=MsoNormal><span style='font-size:10.0pt;font-family:"Arial","sans-serif";
color:blue'>Yossi </span><o:p></o:p></p>
</div>
<div>
<p class=MsoNormal> <o:p></o:p></p>
</div>
<blockquote style='border:none;border-left:solid blue 1.5pt;padding:0in 0in 0in 4.0pt;
margin-left:3.75pt;margin-top:5.0pt;margin-right:0in;margin-bottom:5.0pt'>
<p class=MsoNormal><o:p> </o:p></p>
<div class=MsoNormal align=center style='text-align:center'>
<hr size=2 width="100%" align=center>
</div>
<p class=MsoNormal style='margin-bottom:12.0pt'><b><span style='font-size:10.0pt;
font-family:"Tahoma","sans-serif"'>From:</span></b><span style='font-size:10.0pt;
font-family:"Tahoma","sans-serif"'> openib-windows-bounces@openib.org
[mailto:openib-windows-bounces@openib.org] <b>On Behalf Of </b>Yossi Leybovich<br>
<b>Sent:</b> Monday, January 29, 2007 4:29 PM<br>
<b>To:</b> 'Fab Tillier'; openib-windows@openib.org<br>
<b>Subject:</b> Re: [Openib-windows] [ANNOUNCE] Build 1.0.0.566 posted</span><o:p></o:p></p>
<div>
<p class=MsoNormal><span style='font-size:10.0pt;font-family:"Arial","sans-serif";
color:blue'>see my comments below.</span><o:p></o:p></p>
</div>
<blockquote style='border:none;border-left:solid blue 1.5pt;padding:0in 0in 0in 4.0pt;
margin-left:3.75pt;margin-top:5.0pt;margin-right:0in;margin-bottom:5.0pt'>
<p class=MsoNormal><o:p> </o:p></p>
<div class=MsoNormal align=center style='text-align:center'>
<hr size=2 width="100%" align=center>
</div>
<p class=MsoNormal style='margin-bottom:12.0pt'><b><span style='font-size:10.0pt;
font-family:"Tahoma","sans-serif"'>From:</span></b><span style='font-size:10.0pt;
font-family:"Tahoma","sans-serif"'> openib-windows-bounces@openib.org
[mailto:openib-windows-bounces@openib.org] <b>On Behalf Of </b>Fab Tillier<br>
<b>Sent:</b> Thursday, January 25, 2007 7:34 PM<br>
<b>To:</b> Yossi Leybovich; openib-windows@openib.org<br>
<b>Subject:</b> Re: [Openib-windows] [ANNOUNCE] Build 1.0.0.566 posted</span><o:p></o:p></p>
<p class=MsoNormal><span style='font-size:11.0pt;font-family:"Calibri","sans-serif";
color:#1F497D'>Hi Yossi,<o:p></o:p></span></p>
<p class=MsoNormal><span style='font-size:11.0pt;font-family:"Calibri","sans-serif";
color:#1F497D'><o:p> </o:p></span></p>
<p class=MsoNormal><span style='font-size:11.0pt;font-family:"Calibri","sans-serif";
color:#1F497D'>A question about r538:<o:p></o:p></span></p>
<p><span style='font-size:10.0pt;font-family:"Arial","sans-serif"'>------------------------------------------------------------------------<br>
r538 | sleybo | 2006-11-07 08:54:25 +0200 (Tue, 07 Nov 2006) | 3 lines<o:p></o:p></span></p>
<p class=MsoNormal style='margin-bottom:12.0pt'><span style='font-size:10.0pt;
font-family:"Arial","sans-serif"'>[IBAL] Compliance tests<br>
1. pass switch_info to the HCA - compliance test C13-026<br>
2. Not use AL cashe for node_description node_info to force Mkey check
-compliance test C14-018<br>
------------------------------------------------------------------------</span><span
style='font-size:11.0pt;font-family:"Calibri","sans-serif";color:#1F497D'><o:p></o:p></span></p>
<p class=MsoNormal><span style='font-size:11.0pt;font-family:"Calibri","sans-serif";
color:#1F497D'>Have you tested to see what the effects of removing the cache
for node description and node info are on SM sweeps when the system is busy?<o:p></o:p></span></p>
<p class=MsoNormal><span style='font-size:11.0pt;font-family:"Calibri","sans-serif";
color:#1F497D'><o:p> </o:p></span></p>
<p class=MsoNormal><span style='font-size:11.0pt;font-family:"Calibri","sans-serif";
color:#1F497D'>I initially added the cache for these so that the response could
be issued in the context of the CQ callback for the special QP (thus at
DISPATCH_LEVEL). Without the cache processing requires a call to the
local MAD verb, which has to be scheduled on a passive-level thread. If
the system is very busy doing I/O (i.e. lots of small packets in Iometer over
IPoIB), I have seen cases where the local MAD thread does not run fast enough
so the response time for the MAD is too long and the SM declares the node as
having failed and removes it from the fabric. This is pretty nasty, as
suddenly all IB multicast group memberships are lost, but there’s no indication
to the host that things went awry.<o:p></o:p></span></p>
<p class=MsoNormal><span style='font-size:11.0pt;font-family:"Calibri","sans-serif";
color:#1F497D'><o:p> </o:p></span></p>
<p class=MsoNormal><span style='font-size:11.0pt;font-family:"Calibri","sans-serif";
color:#1F497D'>There are two solutions for this, one is more of a temporary fix
than the other IMO. First, the temporary fix: perform the MKey check in
software, so that the MAD response for as many MADs can be generated at
DISPATCH_LEVEL from the context of the special QP’s CQ callback. This
should maintain compliance while also keeping response times for MADs as short
as possible.<br>
</span><span style='font-size:10.0pt;font-family:"Calibri","sans-serif";
color:blue'>[Yossi Leybovich] To solve the problem of denial of service I
will add simple m-key check . In any case of error (or not trivial m-key check
(i.e m-key =0) I will disable the cache and move the MAD to the FW</span><o:p></o:p></p>
<p class=MsoNormal><span style='font-size:10.0pt;font-family:"Calibri","sans-serif";
color:blue'>( I don't want to count the m key violation and of course not
to add code that generate traps).</span><o:p></o:p></p>
<p class=MsoNormal><span style='font-size:10.0pt;font-family:"Calibri","sans-serif";
color:blue'>This will reduce the handling of good flow packets.</span><o:p></o:p></p>
<p class=MsoNormal><span style='font-size:11.0pt;font-family:"Calibri","sans-serif";
color:#1F497D'><br>
The second solution is to make the local MAD verb asynchronous. The HCA
handles the command asynchronously anyway, so this is a more natural fit given
the HW design. This would mean the local MAD verb would be called
directly from the CQ callback (at DISPATCH_LEVEL), and would return
pending. When the local MAD is processed and the HCA generates the
response to the EQ, the driver could invoke a callback to indicate completion
(again at DISPATCH_LEVEL) which would send out the response. This
solution eliminates the thread scheduling issues associated with handling local
MAD requests in a passive-level thread.<br>
</span><span style='font-size:10.0pt;font-family:"Calibri","sans-serif";
color:blue'>[Yossi Leybovich] </span><o:p></o:p></p>
<p class=MsoNormal><span style='font-size:10.0pt;font-family:"Calibri","sans-serif";
color:blue'>This will require to test our driver with async commands ( I think
that Leonid does not fully support it Leonid?) I don't think we will have the
time to do that in the short future</span><span style='font-size:11.0pt;
font-family:"Calibri","sans-serif";color:#1F497D'> <o:p></o:p></span></p>
<p class=MsoNormal><span style='font-size:11.0pt;font-family:"Calibri","sans-serif";
color:#1F497D'><o:p> </o:p></span></p>
<p class=MsoNormal><span style='font-size:11.0pt;font-family:"Calibri","sans-serif";
color:#1F497D'>We should make sure that systems aren’t susceptible to Denial of
Service attacks from someone flooding them with IPoIB traffic (which gets
handled at DISPATCH_LEVEL in IPoIB’s CQ callback). It’s bad if an
application on one host can cause another host to be removed from the fabric –
there will be no port down events, no notification to the SM when the host is responsive
again, and the host will not be able to participate properly in the fabric
until the next SM sweep.<o:p></o:p></span></p>
<p class=MsoNormal><span style='font-size:11.0pt;font-family:"Calibri","sans-serif";
color:#1F497D'> <o:p></o:p></span></p>
<p class=MsoNormal><span style='font-size:11.0pt;font-family:"Calibri","sans-serif";
color:#1F497D'>-Fab</span><span style='font-size:10.0pt;font-family:"Arial","sans-serif"'><o:p></o:p></span></p>
</blockquote>
</blockquote>
</div>
</body>
</html>