<html xmlns:v="urn:schemas-microsoft-com:vml" xmlns:o="urn:schemas-microsoft-com:office:office" xmlns:w="urn:schemas-microsoft-com:office:word" xmlns:m="http://schemas.microsoft.com/office/2004/12/omml" xmlns="http://www.w3.org/TR/REC-html40">
<head>
<meta http-equiv=Content-Type content="text/html; charset=us-ascii">
<meta name=Generator content="Microsoft Word 12 (filtered medium)">
<!--[if !mso]>
<style>
v\:* {behavior:url(#default#VML);}
o\:* {behavior:url(#default#VML);}
w\:* {behavior:url(#default#VML);}
.shape {behavior:url(#default#VML);}
</style>
<![endif]-->
<style>
<!--
/* Font Definitions */
@font-face
{font-family:"Cambria Math";
panose-1:2 4 5 3 5 4 6 3 2 4;}
@font-face
{font-family:Calibri;
panose-1:2 15 5 2 2 2 4 3 2 4;}
@font-face
{font-family:Tahoma;
panose-1:2 11 6 4 3 5 4 4 2 4;}
/* Style Definitions */
p.MsoNormal, li.MsoNormal, div.MsoNormal
{margin:0in;
margin-bottom:.0001pt;
font-size:12.0pt;
font-family:"Times New Roman","serif";
color:black;}
a:link, span.MsoHyperlink
{mso-style-priority:99;
color:blue;
text-decoration:underline;}
a:visited, span.MsoHyperlinkFollowed
{mso-style-priority:99;
color:purple;
text-decoration:underline;}
p.MsoPlainText, li.MsoPlainText, div.MsoPlainText
{mso-style-priority:99;
mso-style-link:"Plain Text Char";
margin:0in;
margin-bottom:.0001pt;
font-size:12.0pt;
font-family:"Times New Roman","serif";
color:black;}
pre
{mso-style-priority:99;
mso-style-link:"HTML Preformatted Char";
margin:0in;
margin-bottom:.0001pt;
font-size:10.0pt;
font-family:"Courier New";}
span.HTMLPreformattedChar
{mso-style-name:"HTML Preformatted Char";
mso-style-priority:99;
mso-style-link:"HTML Preformatted";
font-family:"Courier New";}
span.PlainTextChar
{mso-style-name:"Plain Text Char";
mso-style-priority:99;
mso-style-link:"Plain Text";
font-family:"Calibri","sans-serif";
color:black;}
span.EmailStyle21
{mso-style-type:personal;
font-family:"Times New Roman","serif";
color:#1F497D;}
span.EmailStyle22
{mso-style-type:personal-reply;
font-family:"Times New Roman","serif";
color:#1F497D;}
.MsoChpDefault
{mso-style-type:export-only;
font-size:10.0pt;}
@page Section1
{size:8.5in 11.0in;
margin:1.0in 1.0in 1.0in 1.0in;}
div.Section1
{page:Section1;}
-->
</style>
<!--[if gte mso 9]><xml>
<o:shapedefaults v:ext="edit" spidmax="1026" />
</xml><![endif]--><!--[if gte mso 9]><xml>
<o:shapelayout v:ext="edit">
<o:idmap v:ext="edit" data="1" />
</o:shapelayout></xml><![endif]-->
</head>
<body bgcolor=white lang=EN-US link=blue vlink=purple>
<div class=Section1>
<p class=MsoNormal><span style='color:#1F497D'>I am talking about Linux but
since there is a lot of shared base code I thought this information might be
useful.<o:p></o:p></span></p>
<p class=MsoNormal><span style='color:#1F497D'><o:p> </o:p></span></p>
<div>
<div style='border:none;border-top:solid #B5C4DF 1.0pt;padding:3.0pt 0in 0in 0in'>
<p class=MsoNormal><b><span style='font-size:10.0pt;font-family:"Tahoma","sans-serif";
color:windowtext'>From:</span></b><span style='font-size:10.0pt;font-family:
"Tahoma","sans-serif";color:windowtext'> Tzachi Dar
[mailto:tzachid@mellanox.co.il] <br>
<b>Sent:</b> Thursday, July 02, 2009 2:52 AM<br>
<b>To:</b> Rupert Dance; David Brean; ofw@lists.openfabrics.org<br>
<b>Subject:</b> RE: [ofw] ping on WinOF<o:p></o:p></span></p>
</div>
</div>
<p class=MsoNormal><o:p> </o:p></p>
<div>
<p class=MsoNormal><span style='font-size:10.0pt;font-family:"Arial","sans-serif";
color:blue'>Thanks for your info - </span><strong><span style='font-size:
10.0pt;font-family:"Tahoma","sans-serif"'> </span></strong><span
style='font-size:10.0pt;font-family:"Tahoma","sans-serif"'>Rupert Dance.</span><span
style='color:windowtext'><o:p></o:p></span></p>
</div>
<div>
<p class=MsoNormal><span style='color:windowtext'> <o:p></o:p></span></p>
</div>
<div>
<p class=MsoNormal><span style='font-size:10.0pt;font-family:"Tahoma","sans-serif"'>However
it seems you are talking about Linux, while we were talking about windows, o r
am I missing something?</span><span style='color:windowtext'><o:p></o:p></span></p>
</div>
<div>
<p class=MsoNormal><span style='color:windowtext'> <o:p></o:p></span></p>
</div>
<div>
<p class=MsoNormal><span style='font-size:10.0pt;font-family:"Tahoma","sans-serif"'>Thanks</span><span
style='color:windowtext'><o:p></o:p></span></p>
</div>
<div>
<p class=MsoNormal><span style='font-size:10.0pt;font-family:"Tahoma","sans-serif"'>Tzachi</span><span
style='color:windowtext'><o:p></o:p></span></p>
</div>
<blockquote style='border:none;border-left:solid blue 1.5pt;padding:0in 0in 0in 4.0pt;
margin-left:3.75pt;margin-top:5.0pt;margin-right:0in;margin-bottom:5.0pt'>
<p class=MsoNormal><span style='color:windowtext'><o:p> </o:p></span></p>
<div class=MsoNormal align=center style='text-align:center'><span
style='color:windowtext'>
<hr size=2 width="100%" align=center>
</span></div>
<p class=MsoNormal style='margin-bottom:12.0pt'><b><span style='font-size:10.0pt;
font-family:"Tahoma","sans-serif";color:windowtext'>From:</span></b><span
style='font-size:10.0pt;font-family:"Tahoma","sans-serif";color:windowtext'>
ofw-bounces@lists.openfabrics.org [mailto:ofw-bounces@lists.openfabrics.org] <b>On
Behalf Of </b>Rupert Dance<br>
<b>Sent:</b> Wednesday, July 01, 2009 7:27 PM<br>
<b>To:</b> 'David Brean'; ofw@lists.openfabrics.org<br>
<b>Subject:</b> RE: [ofw] ping on WinOF</span><span style='color:windowtext'><o:p></o:p></span></p>
<p class=MsoNormal><span style='color:#1F497D'>We have seen issues with IPoIB
in datagram mode particularly when you use a large size (8192 and greater).
This was reported to the OFA <a
href="https://bugs.openfabrics.org/show_bug.cgi?id=1287">Bugzilla Bug # 1287</a>.
Yosef Etigin looked into this and suggested a workaround that did affect the
first packet drop. Here is his comment:<o:p></o:p></span></p>
<p class=MsoNormal><span style='color:#1F497D'><o:p> </o:p></span></p>
<p class=MsoNormal><span style='font-size:10.0pt'>It is a network stack
limitation and not related ipoib in particular.<o:p></o:p></span></p>
<p class=MsoNormal><span style='font-size:10.0pt'>There's a limit (default = 3)
on number of pending skb's before a neighbour is<o:p></o:p></span></p>
<p class=MsoNormal><span style='font-size:10.0pt'>resolved. You can increase it
with sysctl net.ipv4.neigh.ib0.unres_qlen.<o:p></o:p></span></p>
<p class=MsoNormal><span style='font-size:10.0pt'>Obviously, same thing happens
with Ethernet interface.<o:p></o:p></span></p>
<p class=MsoNormal><span style='color:#1F497D'><o:p> </o:p></span></p>
<p class=MsoNormal><span style='color:#1F497D'>When testing at UNH-IOL for the
Logo program, this is what we did:<o:p></o:p></span></p>
<p class=MsoNormal><span style='color:#1F497D'><o:p> </o:p></span></p>
<p class=MsoPlainText>After working with Sasha Khapyorsky on this issue we have
a working fix. To further explain the situation, the large packet sizes we are
using are overflowing the buffers so there is no room to append the arp request
on to the beginning of the cmd. This results in a dropped packet because the
system doesn't know how to get to the destination due to an empty arp table.
The fix, increase the buffer size via:<o:p></o:p></p>
<p class=MsoPlainText><o:p> </o:p></p>
<p class=MsoPlainText>sysctl net.ipv4.neigh.ib0.unres_qlen=17 # default is the
value 3<o:p></o:p></p>
<p class=MsoNormal><span style='color:#1F497D'><o:p> </o:p></span></p>
<p class=MsoNormal><span style='color:#1F497D'>Thanks<o:p></o:p></span></p>
<p class=MsoNormal><span style='color:#1F497D'><o:p> </o:p></span></p>
<p class=MsoNormal><span style='color:#1F497D'>Rupert Dance<o:p></o:p></span></p>
<p class=MsoNormal><span style='color:#1F497D'><o:p> </o:p></span></p>
<div>
<div style='border:none;border-top:solid #B5C4DF 1.0pt;padding:3.0pt 0in 0in 0in'>
<p class=MsoNormal><b><span style='font-size:10.0pt'>From:</span></b><span
style='font-size:10.0pt'> ofw-bounces@lists.openfabrics.org
[mailto:ofw-bounces@lists.openfabrics.org] <b>On Behalf Of </b>David Brean<br>
<b>Sent:</b> Wednesday, July 01, 2009 11:39 AM<br>
<b>To:</b> ofw@lists.openfabrics.org<br>
<b>Subject:</b> [ofw] ping on WinOF<o:p></o:p></span></p>
</div>
</div>
<p class=MsoNormal><o:p> </o:p></p>
<p class=MsoNormal style='margin-bottom:12.0pt'>Hello,<br>
<br>
An internal customer is using WinOF 2.0.X and has reported to me the following
behavior related to IPoIB and ping:<br>
<br>
<i>Do you have any ideas on why windows 2008 client with HCA may first timeout
ping to other clients on the fabric?<br>
<br>
Initially ping fails but then starts working.<br>
<br>
Example : Ping is invoked three times successfully.<br>
<br>
C:\GRITS>ping -a 192.168.100.235<br>
<br>
Pinging 192.168.100.235 with 32 bytes of data:<br>
Request timed out.<br>
Request timed out.<br>
Request timed out.<br>
Request timed out.<br>
<br>
Ping statistics for 192.168.100.235:<br>
Packets: Sent = 4, Received = 0, Lost = 4 (100% loss),<br>
<br>
C:\GRITS>ping -a 192.168.100.235<br>
<br>
Pinging 192.168.100.235 with 32 bytes of data:<br>
Request timed out.<br>
Request timed out.<br>
Reply from 192.168.100.235: bytes=32 time<1ms TTL=255<br>
Reply from 192.168.100.235: bytes=32 time<1ms TTL=255<br>
<br>
Ping statistics for 192.168.100.235:<br>
Packets: Sent = 4, Received = 2, Lost = 2 (50% loss),<br>
Approximate round trip times in milli-seconds:<br>
Minimum = 0ms, Maximum = 0ms, Average = 0ms<br>
<br>
C:\GRITS>ping -a 192.168.100.235<br>
<br>
Pinging 192.168.100.235 with 32 bytes of data:<br>
Reply from 192.168.100.235: bytes=32 time<1ms TTL=255<br>
Reply from 192.168.100.235: bytes=32 time<1ms TTL=255<br>
Reply from 192.168.100.235: bytes=32 time<1ms TTL=255<br>
Reply from 192.168.100.235: bytes=32 time<1ms TTL=255<br>
<br>
Ping statistics for 192.168.100.235:<br>
Packets: Sent = 4, Received = 4, Lost = 0 (0% loss),<br>
Approximate round trip times in milli-seconds:<br>
Minimum = 0ms, Maximum = 0ms, Average = 0ms<br>
<br>
Then we are good for sometime before this starts again if network is idle on
the fabric.<br>
</i><br>
Has this sort of behavior been observed before? The Linux and Solaris
nodes sharing the same IP subnet appear to be behaving normally. Windows
server is the "out-of-the-box" configuration with Voltaire switch configured
with only the default partition (0xFFFF).<br>
<br>
-David<o:p></o:p></p>
</blockquote>
</div>
</body>
</html>