<html xmlns:o="urn:schemas-microsoft-com:office:office" xmlns:w="urn:schemas-microsoft-com:office:word" xmlns:st1="urn:schemas-microsoft-com:office:smarttags" xmlns="http://www.w3.org/TR/REC-html40">
<head>
<meta http-equiv=Content-Type content="text/html; charset=us-ascii">
<meta name=Generator content="Microsoft Word 11 (filtered medium)">
<o:SmartTagType namespaceuri="urn:schemas-microsoft-com:office:smarttags"
name="City"/>
<o:SmartTagType namespaceuri="urn:schemas-microsoft-com:office:smarttags"
name="place"/>
<!--[if !mso]>
<style>
st1\:*{behavior:url(#default#ieooui) }
</style>
<![endif]-->
<style>
<!--
/* Style Definitions */
p.MsoNormal, li.MsoNormal, div.MsoNormal
{margin:0in;
margin-bottom:.0001pt;
font-size:12.0pt;
font-family:"Times New Roman";}
a:link, span.MsoHyperlink
{color:blue;
text-decoration:underline;}
a:visited, span.MsoHyperlinkFollowed
{color:purple;
text-decoration:underline;}
span.EmailStyle17
{mso-style-type:personal-compose;
font-family:Arial;
color:windowtext;}
@page Section1
{size:8.5in 11.0in;
margin:1.0in 1.25in 1.0in 1.25in;}
div.Section1
{page:Section1;}
/* List Definitions */
@list l0
{mso-list-id:89398009;
mso-list-type:hybrid;
mso-list-template-ids:-217262562 67698703 67698713 67698715 67698703 67698713 67698715 67698703 67698713 67698715;}
@list l0:level1
{mso-level-tab-stop:.5in;
mso-level-number-position:left;
text-indent:-.25in;}
ol
{margin-bottom:0in;}
ul
{margin-bottom:0in;}
-->
</style>
</head>
<body lang=EN-US link=blue vlink=purple>
<div class=Section1>
<p class=MsoNormal><font size=2 face=Arial><span style='font-size:10.0pt;
font-family:Arial'><o:p> </o:p></span></font></p>
<p class=MsoNormal><font size=2 face=Arial><span style='font-size:10.0pt;
font-family:Arial'>Hi Fabian and all,<o:p></o:p></span></font></p>
<p class=MsoNormal><font size=2 face=Arial><span style='font-size:10.0pt;
font-family:Arial'><o:p> </o:p></span></font></p>
<p class=MsoNormal><font size=2 face=Arial><span style='font-size:10.0pt;
font-family:Arial'>While testing the Pallas reduce_scatter over IPoIB, with 3
hosts using the following command line:<o:p></o:p></span></font></p>
<p class=MsoNormal><font size=2 face=Arial><span style='font-size:10.0pt;
font-family:Arial'><o:p> </o:p></span></font></p>
<p class=MsoNormal><font size=2 face=Arial><span style='font-size:10.0pt;
font-family:Arial'>mpiexec.exe -env MPICH_NETMASK 10.0.0.0/255.255.255.0 -hosts
3 10.0.0.1 10.0.0.2 10.0.0.3 PMB_MPI1.exe reduce_scatter<o:p></o:p></span></font></p>
<p class=MsoNormal><font size=2 face=Arial><span style='font-size:10.0pt;
font-family:Arial'><o:p> </o:p></span></font></p>
<p class=MsoNormal><font size=2 face=Arial><span style='font-size:10.0pt;
font-family:Arial'>I’ve discovered that the receive queue (default to 128
packets) is being exhausted at the second host (10.0.0.2).<o:p></o:p></span></font></p>
<p class=MsoNormal><font size=2 face=Arial><span style='font-size:10.0pt;
font-family:Arial'><o:p> </o:p></span></font></p>
<p class=MsoNormal><font size=2 face=Arial><span style='font-size:10.0pt;
font-family:Arial'>The TCP/IP stack is holding all the 128 packets, and the
only way to regain connectivity with this host (even ping are not working, of course)
is to kill the smpd application (or wait enough time for TCP timeouts to
expired – although I didn’t actually test it).<o:p></o:p></span></font></p>
<p class=MsoNormal><font size=2 face=Arial><span style='font-size:10.0pt;
font-family:Arial'><o:p> </o:p></span></font></p>
<p class=MsoNormal><font size=2 face=Arial><span style='font-size:10.0pt;
font-family:Arial'>When setting the receive queue to 1024 packets, the problem
didn’t occur.<o:p></o:p></span></font></p>
<p class=MsoNormal><font size=2 face=Arial><span style='font-size:10.0pt;
font-family:Arial'><o:p> </o:p></span></font></p>
<p class=MsoNormal><font size=2 face=Arial><span style='font-size:10.0pt;
font-family:Arial'>All my machines are 2-way SMPs. When running with /ONECPU
boot.ini parameter, the problem occurred, but less frequently.<o:p></o:p></span></font></p>
<p class=MsoNormal><font size=2 face=Arial><span style='font-size:10.0pt;
font-family:Arial'><o:p> </o:p></span></font></p>
<p class=MsoNormal><font size=2 face=Arial><span style='font-size:10.0pt;
font-family:Arial'>My questions:<o:p></o:p></span></font></p>
<ol style='margin-top:0in' start=1 type=1>
<li class=MsoNormal style='mso-list:l0 level1 lfo1'><font size=2 face=Arial><span
style='font-size:10.0pt;font-family:Arial'>Have you encounter similar situations?<o:p></o:p></span></font></li>
<li class=MsoNormal style='mso-list:l0 level1 lfo1'><font size=2 face=Arial><span
style='font-size:10.0pt;font-family:Arial'>I’ve noticed the “Receive
Pool Growth” parameter – but it doesn’t seem to be “connected”.
Why ? If I would like to “connect” it (i.e. write the
appropriate code to handle queue growth) what should be done and where?<o:p></o:p></span></font></li>
<li class=MsoNormal style='mso-list:l0 level1 lfo1'><font size=2 face=Arial><span
style='font-size:10.0pt;font-family:Arial'>And I really don’t know
if someone can answer this: Why does the Windows TCP/IP stack behave in
such a way ? Why it doesn’t copy the packets in case of extreme
situations like the above ?<o:p></o:p></span></font></li>
</ol>
<p class=MsoNormal><font size=2 face=Arial><span style='font-size:10.0pt;
font-family:Arial'> <o:p></o:p></span></font></p>
<p class=MsoNormal><font size=2 face=Arial><span style='font-size:10.0pt;
font-family:Arial'>Thanks,<o:p></o:p></span></font></p>
<p class=MsoNormal><font size=2 face=Arial><span style='font-size:10.0pt;
font-family:Arial'><o:p> </o:p></span></font></p>
<p class=MsoNormal><font size=2 face=Arial><span style='font-size:10.0pt;
font-family:Arial'>Guy Corem<br>
Windows Developer<br>
Voltaire Ltd.<br>
<st1:City w:st="on"><st1:place w:st="on">Mobile</st1:place></st1:City>:
+972-50-7321946<br>
Tel: +972-9-9717672<br>
Fax: +972-9-9717660<br>
Operator: +972-9-9717666<br>
email: <a href="mailto:guyc@voltaire.com">guyc@voltaire.com</a><br>
web: <a href="http://www.voltaire.com">http://www.voltaire.com</a></span></font><font
size=2 face=Arial><span style='font-size:10.0pt;font-family:Arial'><o:p></o:p></span></font></p>
<p class=MsoNormal><font size=3 face="Times New Roman"><span style='font-size:
12.0pt'><o:p> </o:p></span></font></p>
</div>
</body>
</html>