<!DOCTYPE HTML PUBLIC "-//W3C//DTD HTML 4.0 Transitional//EN">
<HTML xmlns="http://www.w3.org/TR/REC-html40" xmlns:v =
"urn:schemas-microsoft-com:vml" xmlns:o =
"urn:schemas-microsoft-com:office:office" xmlns:w =
"urn:schemas-microsoft-com:office:word" xmlns:m =
"http://schemas.microsoft.com/office/2004/12/omml"><HEAD>
<META content="text/html; charset=us-ascii" http-equiv=Content-Type>
<META name=GENERATOR content="MSHTML 8.00.6001.18783"><!--[if !mso]>
<STYLE>v\:* {
BEHAVIOR: url(#default#VML)
}
o\:* {
BEHAVIOR: url(#default#VML)
}
w\:* {
BEHAVIOR: url(#default#VML)
}
.shape {
BEHAVIOR: url(#default#VML)
}
</STYLE>
<![endif]-->
<STYLE>@font-face {
font-family: Cambria Math;
}
@font-face {
font-family: Calibri;
}
@font-face {
font-family: Tahoma;
}
@page Section1 {size: 8.5in 11.0in; margin: 1.0in 1.0in 1.0in 1.0in; }
P.MsoNormal {
MARGIN: 0in 0in 0pt; FONT-FAMILY: "Times New Roman","serif"; COLOR: black; FONT-SIZE: 12pt
}
LI.MsoNormal {
MARGIN: 0in 0in 0pt; FONT-FAMILY: "Times New Roman","serif"; COLOR: black; FONT-SIZE: 12pt
}
DIV.MsoNormal {
MARGIN: 0in 0in 0pt; FONT-FAMILY: "Times New Roman","serif"; COLOR: black; FONT-SIZE: 12pt
}
A:link {
COLOR: blue; TEXT-DECORATION: underline; mso-style-priority: 99
}
SPAN.MsoHyperlink {
COLOR: blue; TEXT-DECORATION: underline; mso-style-priority: 99
}
A:visited {
COLOR: purple; TEXT-DECORATION: underline; mso-style-priority: 99
}
SPAN.MsoHyperlinkFollowed {
COLOR: purple; TEXT-DECORATION: underline; mso-style-priority: 99
}
P.MsoPlainText {
MARGIN: 0in 0in 0pt; FONT-FAMILY: "Times New Roman","serif"; COLOR: black; FONT-SIZE: 12pt; mso-style-priority: 99; mso-style-link: "Plain Text Char"
}
LI.MsoPlainText {
MARGIN: 0in 0in 0pt; FONT-FAMILY: "Times New Roman","serif"; COLOR: black; FONT-SIZE: 12pt; mso-style-priority: 99; mso-style-link: "Plain Text Char"
}
DIV.MsoPlainText {
MARGIN: 0in 0in 0pt; FONT-FAMILY: "Times New Roman","serif"; COLOR: black; FONT-SIZE: 12pt; mso-style-priority: 99; mso-style-link: "Plain Text Char"
}
PRE {
MARGIN: 0in 0in 0pt; FONT-FAMILY: "Courier New"; FONT-SIZE: 10pt; mso-style-priority: 99; mso-style-link: "HTML Preformatted Char"
}
SPAN.HTMLPreformattedChar {
FONT-FAMILY: "Courier New"; mso-style-priority: 99; mso-style-link: "HTML Preformatted"; mso-style-name: "HTML Preformatted Char"
}
SPAN.PlainTextChar {
FONT-FAMILY: "Calibri","sans-serif"; COLOR: black; mso-style-priority: 99; mso-style-link: "Plain Text"; mso-style-name: "Plain Text Char"
}
SPAN.EmailStyle21 {
FONT-FAMILY: "Times New Roman","serif"; COLOR: #1f497d; mso-style-type: personal
}
SPAN.EmailStyle22 {
FONT-FAMILY: "Times New Roman","serif"; COLOR: #1f497d; mso-style-type: personal-reply
}
.MsoChpDefault {
FONT-SIZE: 10pt; mso-style-type: export-only
}
DIV.Section1 {
page: Section1
}
</STYLE>
<!--[if gte mso 9]><xml>
<o:shapedefaults v:ext="edit" spidmax="1026" />
</xml><![endif]--><!--[if gte mso 9]><xml>
<o:shapelayout v:ext="edit">
<o:idmap v:ext="edit" data="1" />
</o:shapelayout></xml><![endif]--></HEAD>
<BODY lang=EN-US link=blue bgColor=white vLink=purple>
<DIV dir=ltr align=left>
<HR tabIndex=-1>
<FONT size=2 face=Tahoma><B>From:</B> ofw-bounces@lists.openfabrics.org
[mailto:ofw-bounces@lists.openfabrics.org] <B>On Behalf Of </B>Rupert
Dance<BR><B>Sent:</B> Thursday, July 02, 2009 5:53 AM<BR><B>To:</B> 'Tzachi
Dar'; 'David Brean'; ofw@lists.openfabrics.org<BR><B>Subject:</B> RE: [ofw] ping
on WinOF<BR></FONT><BR></DIV>
<DIV></DIV>
<DIV class=Section1>
<P class=MsoNormal><SPAN style="COLOR: #1f497d"><SPAN
class=892404216-02072009><FONT color=#0000ff size=2
face=Arial> >> </FONT></SPAN>I am talking about Linux but since
there is a lot of shared base code I thought this information might be
useful.<SPAN class=892404216-02072009><FONT color=#0000ff size=2
face=Arial> </FONT></SPAN></SPAN></P>
<P class=MsoNormal><SPAN style="COLOR: #1f497d"><SPAN
class=892404216-02072009></SPAN></SPAN> </P>
<P class=MsoNormal><SPAN style="COLOR: #1f497d"><SPAN
class=892404216-02072009><FONT color=#0000ff size=2 face=Arial>At
the driver level, there is no common code between Linux/OFED and Windows
IPoIB; some similar algorithms.</FONT></SPAN></SPAN></P>
<P class=MsoNormal><SPAN style="COLOR: #1f497d"><SPAN
class=892404216-02072009><FONT color=#0000ff size=2 face=Arial>The shared base
code is at user-level and only recently been available in the SVN src tree; read
that as technology preview in WinOF 2.0.2.</FONT></SPAN></SPAN></P>
<P class=MsoNormal><SPAN style="COLOR: #1f497d"><SPAN
class=892404216-02072009><FONT color=#0000ff size=2 face=Arial>That said,
windows HCA driver code is based on OFED HCA driver code with a good amount of
code added/modified for the Windows environment.</FONT></SPAN></SPAN></P>
<P class=MsoNormal><SPAN style="COLOR: #1f497d"><SPAN
class=892404216-02072009><FONT color=#0000ff size=2
face=Arial></FONT></SPAN></SPAN> </P>
<P class=MsoNormal><SPAN style="COLOR: #1f497d"><SPAN
class=892404216-02072009><FONT color=#0000ff size=2
face=Arial>stan.</FONT></SPAN></SPAN></P>
<P class=MsoNormal><SPAN style="COLOR: #1f497d"><o:p> </o:p></SPAN></P>
<DIV>
<DIV
style="BORDER-BOTTOM: medium none; BORDER-LEFT: medium none; PADDING-BOTTOM: 0in; PADDING-LEFT: 0in; PADDING-RIGHT: 0in; BORDER-TOP: #b5c4df 1pt solid; BORDER-RIGHT: medium none; PADDING-TOP: 3pt">
<P class=MsoNormal><B><SPAN
style="FONT-FAMILY: 'Tahoma','sans-serif'; COLOR: windowtext; FONT-SIZE: 10pt">From:</SPAN></B><SPAN
style="FONT-FAMILY: 'Tahoma','sans-serif'; COLOR: windowtext; FONT-SIZE: 10pt">
Tzachi Dar [mailto:tzachid@mellanox.co.il] <BR><B>Sent:</B> Thursday, July 02,
2009 2:52 AM<BR><B>To:</B> Rupert Dance; David Brean;
ofw@lists.openfabrics.org<BR><B>Subject:</B> RE: [ofw] ping on
WinOF<o:p></o:p></SPAN></P></DIV></DIV>
<P class=MsoNormal><o:p> </o:p></P>
<DIV>
<P class=MsoNormal><SPAN
style="FONT-FAMILY: 'Arial','sans-serif'; COLOR: blue; FONT-SIZE: 10pt">Thanks
for your info - </SPAN><STRONG><SPAN
style="FONT-FAMILY: 'Tahoma','sans-serif'; FONT-SIZE: 10pt">
</SPAN></STRONG><SPAN
style="FONT-FAMILY: 'Tahoma','sans-serif'; FONT-SIZE: 10pt">Rupert
Dance.</SPAN><SPAN style="COLOR: windowtext"><o:p></o:p></SPAN></P></DIV>
<DIV>
<P class=MsoNormal><SPAN
style="COLOR: windowtext"> <o:p></o:p></SPAN></P></DIV>
<DIV>
<P class=MsoNormal><SPAN
style="FONT-FAMILY: 'Tahoma','sans-serif'; FONT-SIZE: 10pt">However it seems you
are talking about Linux, while we were talking about windows, o r am I missing
something?</SPAN><SPAN style="COLOR: windowtext"><o:p></o:p></SPAN></P></DIV>
<DIV>
<P class=MsoNormal><SPAN
style="COLOR: windowtext"> <o:p></o:p></SPAN></P></DIV>
<DIV>
<P class=MsoNormal><SPAN
style="FONT-FAMILY: 'Tahoma','sans-serif'; FONT-SIZE: 10pt">Thanks</SPAN><SPAN
style="COLOR: windowtext"><o:p></o:p></SPAN></P></DIV>
<DIV>
<P class=MsoNormal><SPAN
style="FONT-FAMILY: 'Tahoma','sans-serif'; FONT-SIZE: 10pt">Tzachi</SPAN><SPAN
style="COLOR: windowtext"><o:p></o:p></SPAN></P></DIV>
<BLOCKQUOTE
style="BORDER-BOTTOM: medium none; BORDER-LEFT: blue 1.5pt solid; PADDING-BOTTOM: 0in; MARGIN: 5pt 0in 5pt 3.75pt; PADDING-LEFT: 4pt; PADDING-RIGHT: 0in; BORDER-TOP: medium none; BORDER-RIGHT: medium none; PADDING-TOP: 0in">
<P class=MsoNormal><SPAN
style="COLOR: windowtext"><o:p> </o:p></SPAN></P>
<DIV style="TEXT-ALIGN: center" class=MsoNormal align=center><SPAN
style="COLOR: windowtext">
<HR align=center SIZE=2 width="100%">
</SPAN></DIV>
<P style="MARGIN-BOTTOM: 12pt" class=MsoNormal><B><SPAN
style="FONT-FAMILY: 'Tahoma','sans-serif'; COLOR: windowtext; FONT-SIZE: 10pt">From:</SPAN></B><SPAN
style="FONT-FAMILY: 'Tahoma','sans-serif'; COLOR: windowtext; FONT-SIZE: 10pt">
ofw-bounces@lists.openfabrics.org [mailto:ofw-bounces@lists.openfabrics.org]
<B>On Behalf Of </B>Rupert Dance<BR><B>Sent:</B> Wednesday, July 01, 2009 7:27
PM<BR><B>To:</B> 'David Brean'; ofw@lists.openfabrics.org<BR><B>Subject:</B>
RE: [ofw] ping on WinOF</SPAN><SPAN
style="COLOR: windowtext"><o:p></o:p></SPAN></P>
<P class=MsoNormal><SPAN style="COLOR: #1f497d">We have seen issues with IPoIB
in datagram mode particularly when you use a large size (8192 and greater).
This was reported to the OFA <A
href="https://bugs.openfabrics.org/show_bug.cgi?id=1287">Bugzilla Bug #
1287</A>. Yosef Etigin looked into this and suggested a workaround that did
affect the first packet drop. Here is his comment:<o:p></o:p></SPAN></P>
<P class=MsoNormal><SPAN style="COLOR: #1f497d"><o:p> </o:p></SPAN></P>
<P class=MsoNormal><SPAN style="FONT-SIZE: 10pt">It is a network stack
limitation and not related ipoib in particular.<o:p></o:p></SPAN></P>
<P class=MsoNormal><SPAN style="FONT-SIZE: 10pt">There's a limit (default = 3)
on number of pending skb's before a neighbour is<o:p></o:p></SPAN></P>
<P class=MsoNormal><SPAN style="FONT-SIZE: 10pt">resolved. You can increase it
with sysctl net.ipv4.neigh.ib0.unres_qlen.<o:p></o:p></SPAN></P>
<P class=MsoNormal><SPAN style="FONT-SIZE: 10pt">Obviously, same thing happens
with Ethernet interface.<o:p></o:p></SPAN></P>
<P class=MsoNormal><SPAN style="COLOR: #1f497d"><o:p> </o:p></SPAN></P>
<P class=MsoNormal><SPAN style="COLOR: #1f497d">When testing at UNH-IOL for
the Logo program, this is what we did:<o:p></o:p></SPAN></P>
<P class=MsoNormal><SPAN style="COLOR: #1f497d"><o:p> </o:p></SPAN></P>
<P class=MsoPlainText>After working with Sasha Khapyorsky on this issue we
have a working fix. To further explain the situation, the large packet sizes
we are using are overflowing the buffers so there is no room to append the arp
request on to the beginning of the cmd. This results in a dropped packet
because the system doesn't know how to get to the destination due to an empty
arp table. The fix, increase the buffer size via:<o:p></o:p></P>
<P class=MsoPlainText><o:p> </o:p></P>
<P class=MsoPlainText>sysctl net.ipv4.neigh.ib0.unres_qlen=17 # default is the
value 3<o:p></o:p></P>
<P class=MsoNormal><SPAN style="COLOR: #1f497d"><o:p> </o:p></SPAN></P>
<P class=MsoNormal><SPAN style="COLOR: #1f497d">Thanks<o:p></o:p></SPAN></P>
<P class=MsoNormal><SPAN style="COLOR: #1f497d"><o:p> </o:p></SPAN></P>
<P class=MsoNormal><SPAN style="COLOR: #1f497d">Rupert
Dance<o:p></o:p></SPAN></P>
<P class=MsoNormal><SPAN style="COLOR: #1f497d"><o:p> </o:p></SPAN></P>
<DIV>
<DIV
style="BORDER-BOTTOM: medium none; BORDER-LEFT: medium none; PADDING-BOTTOM: 0in; PADDING-LEFT: 0in; PADDING-RIGHT: 0in; BORDER-TOP: #b5c4df 1pt solid; BORDER-RIGHT: medium none; PADDING-TOP: 3pt">
<P class=MsoNormal><B><SPAN style="FONT-SIZE: 10pt">From:</SPAN></B><SPAN
style="FONT-SIZE: 10pt"> ofw-bounces@lists.openfabrics.org
[mailto:ofw-bounces@lists.openfabrics.org] <B>On Behalf Of </B>David
Brean<BR><B>Sent:</B> Wednesday, July 01, 2009 11:39 AM<BR><B>To:</B>
ofw@lists.openfabrics.org<BR><B>Subject:</B> [ofw] ping on
WinOF<o:p></o:p></SPAN></P></DIV></DIV>
<P class=MsoNormal><o:p> </o:p></P>
<P style="MARGIN-BOTTOM: 12pt" class=MsoNormal>Hello,<BR><BR>An internal
customer is using WinOF 2.0.X and has reported to me the following behavior
related to IPoIB and ping:<BR><BR><I>Do you have any ideas on why windows 2008
client with HCA may first timeout ping to other clients on the
fabric?<BR><BR>Initially ping fails but then starts working.<BR><BR>Example
: Ping is invoked three times successfully.<BR><BR>C:\GRITS>ping -a
192.168.100.235<BR><BR>Pinging 192.168.100.235 with 32 bytes of
data:<BR>Request timed out.<BR>Request timed out.<BR>Request timed
out.<BR>Request timed out.<BR><BR>Ping statistics for
192.168.100.235:<BR> Packets: Sent = 4, Received = 0, Lost = 4
(100% loss),<BR><BR>C:\GRITS>ping -a 192.168.100.235<BR><BR>Pinging
192.168.100.235 with 32 bytes of data:<BR>Request timed out.<BR>Request timed
out.<BR>Reply from 192.168.100.235: bytes=32 time<1ms TTL=255<BR>Reply from
192.168.100.235: bytes=32 time<1ms TTL=255<BR><BR>Ping statistics for
192.168.100.235:<BR> Packets: Sent = 4, Received = 2, Lost = 2
(50% loss),<BR>Approximate round trip times in milli-seconds:<BR>
Minimum = 0ms, Maximum = 0ms, Average = 0ms<BR><BR>C:\GRITS>ping -a
192.168.100.235<BR><BR>Pinging 192.168.100.235 with 32 bytes of data:<BR>Reply
from 192.168.100.235: bytes=32 time<1ms TTL=255<BR>Reply from
192.168.100.235: bytes=32 time<1ms TTL=255<BR>Reply from 192.168.100.235:
bytes=32 time<1ms TTL=255<BR>Reply from 192.168.100.235: bytes=32
time<1ms TTL=255<BR><BR>Ping statistics for
192.168.100.235:<BR> Packets: Sent = 4, Received = 4, Lost = 0 (0%
loss),<BR>Approximate round trip times in milli-seconds:<BR>
Minimum = 0ms, Maximum = 0ms, Average = 0ms<BR><BR>Then we are good for
sometime before this starts again if network is idle on the
fabric.<BR></I><BR>Has this sort of behavior been observed before? The
Linux and Solaris nodes sharing the same IP subnet appear to be behaving
normally. Windows server is the "out-of-the-box" configuration with
Voltaire switch configured with only the default partition
(0xFFFF).<BR><BR>-David<o:p></o:p></P></BLOCKQUOTE></DIV></BODY></HTML>