<html xmlns:v="urn:schemas-microsoft-com:vml" xmlns:o="urn:schemas-microsoft-com:office:office" xmlns:w="urn:schemas-microsoft-com:office:word" xmlns:m="http://schemas.microsoft.com/office/2004/12/omml" xmlns="http://www.w3.org/TR/REC-html40">
<head>
<meta http-equiv="Content-Type" content="text/html; charset=us-ascii">
<meta name="Generator" content="Microsoft Word 14 (filtered medium)">
<style><!--
/* Font Definitions */
@font-face
{font-family:"Cambria Math";
panose-1:2 4 5 3 5 4 6 3 2 4;}
@font-face
{font-family:Calibri;
panose-1:2 15 5 2 2 2 4 3 2 4;}
@font-face
{font-family:Tahoma;
panose-1:2 11 6 4 3 5 4 4 2 4;}
/* Style Definitions */
p.MsoNormal, li.MsoNormal, div.MsoNormal
{margin:0in;
margin-bottom:.0001pt;
font-size:12.0pt;
font-family:"Times New Roman","serif";}
a:link, span.MsoHyperlink
{mso-style-priority:99;
color:blue;
text-decoration:underline;}
a:visited, span.MsoHyperlinkFollowed
{mso-style-priority:99;
color:purple;
text-decoration:underline;}
p.MsoAcetate, li.MsoAcetate, div.MsoAcetate
{mso-style-priority:99;
mso-style-link:"Balloon Text Char";
margin:0in;
margin-bottom:.0001pt;
font-size:8.0pt;
font-family:"Tahoma","sans-serif";}
span.EmailStyle17
{mso-style-type:personal;
font-family:"Calibri","sans-serif";
color:#1F497D;}
span.BalloonTextChar
{mso-style-name:"Balloon Text Char";
mso-style-priority:99;
mso-style-link:"Balloon Text";
font-family:"Tahoma","sans-serif";}
span.EmailStyle20
{mso-style-type:personal-reply;
font-family:"Calibri","sans-serif";
color:#1F497D;}
.MsoChpDefault
{mso-style-type:export-only;
font-size:10.0pt;}
@page WordSection1
{size:8.5in 11.0in;
margin:1.0in 1.0in 1.0in 1.0in;}
div.WordSection1
{page:WordSection1;}
--></style><!--[if gte mso 9]><xml>
<o:shapedefaults v:ext="edit" spidmax="1026" />
</xml><![endif]--><!--[if gte mso 9]><xml>
<o:shapelayout v:ext="edit">
<o:idmap v:ext="edit" data="1" />
</o:shapelayout></xml><![endif]-->
</head>
<body lang="EN-US" link="blue" vlink="purple">
<div class="WordSection1">
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:"Calibri","sans-serif";color:#1F497D">I understand you are mainly using IPoIB.<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:"Calibri","sans-serif";color:#1F497D">We had some latency issue of IPoIB in WinOF 4.55.<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:"Calibri","sans-serif";color:#1F497D">We plan 4.60 GA release soon where this issue is resolved.<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:"Calibri","sans-serif";color:#1F497D"><o:p> </o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:"Calibri","sans-serif";color:#1F497D">So I suggest you wait for this release.<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:"Calibri","sans-serif";color:#1F497D"><o:p> </o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:"Calibri","sans-serif";color:#1F497D">Tziporet<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:"Calibri","sans-serif";color:#1F497D"><o:p> </o:p></span></p>
<div>
<div style="border:none;border-top:solid #B5C4DF 1.0pt;padding:3.0pt 0in 0in 0in">
<p class="MsoNormal"><b><span style="font-size:10.0pt;font-family:"Tahoma","sans-serif"">From:</span></b><span style="font-size:10.0pt;font-family:"Tahoma","sans-serif""> ofw-bounces@lists.openfabrics.org [mailto:ofw-bounces@lists.openfabrics.org]
<b>On Behalf Of </b>nicholas ferguso<br>
<b>Sent:</b> Monday, December 30, 2013 5:07 PM<br>
<b>To:</b> ofw@lists.openfabrics.org<br>
<b>Subject:</b> Re: [ofw] Unstable operation of WinOFED 4.40.0 with HCA fw 2.30.3200 on Win 2012 Server<o:p></o:p></span></p>
</div>
</div>
<p class="MsoNormal"><o:p> </o:p></p>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:"Calibri","sans-serif";color:#1F497D">Have you researched ICMP policies?<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:"Calibri","sans-serif";color:#1F497D"><o:p> </o:p></span></p>
<div style="border:none;border-top:solid #B5C4DF 1.0pt;padding:3.0pt 0in 0in 0in">
<p class="MsoNormal"><b><span style="font-size:10.0pt;font-family:"Tahoma","sans-serif"">From:</span></b><span style="font-size:10.0pt;font-family:"Tahoma","sans-serif"">
<a href="mailto:ofw-bounces@lists.openfabrics.org">ofw-bounces@lists.openfabrics.org</a> [<a href="mailto:ofw-bounces@lists.openfabrics.org">mailto:ofw-bounces@lists.openfabrics.org</a>]
<b>On Behalf Of </b>Alexey Novozhilov<br>
<b>Sent:</b> Monday, December 30, 2013 12:29 AM<br>
<b>To:</b> <a href="mailto:ofw@lists.openfabrics.org">ofw@lists.openfabrics.org</a><br>
<b>Subject:</b> [ofw] Unstable operation of WinOFED 4.40.0 with HCA fw 2.30.3200 on Win 2012 Server<o:p></o:p></span></p>
</div>
<p class="MsoNormal"><o:p> </o:p></p>
<div>
<div>
<p class="MsoNormal"><span style="font-size:10.0pt;font-family:"Arial","sans-serif"">Hi,<o:p></o:p></span></p>
</div>
<div>
<p class="MsoNormal"><span style="font-size:10.0pt;font-family:"Arial","sans-serif""><o:p> </o:p></span></p>
</div>
<div>
<p class="MsoNormal"><span style="font-size:10.0pt;font-family:"Arial","sans-serif"">I've encountered an issue of unstable operation of WinOFED 4.40.0 with HCA firmware 2.30.3200 on Windows 2012 Server.<o:p></o:p></span></p>
</div>
<div>
<p class="MsoNormal"><span style="font-size:10.0pt;font-family:"Arial","sans-serif"">Last week I'm going mad trying to get it working well.<o:p></o:p></span></p>
</div>
<div>
<p class="MsoNormal"><span style="font-size:10.0pt;font-family:"Arial","sans-serif""><o:p> </o:p></span></p>
</div>
<div>
<p class="MsoNormal"><span style="font-size:10.0pt;font-family:"Arial","sans-serif"">I have a system of two Mellanox Infiniscale-IV IS5023 switches, three hosts running Windows 2012 Server / Windows 2012 Server R2 and two hosts running Ubuntu Linux 12.04 LTS.
All hosts are equiped with Connect-X3 VPI Mellanox network cards ( MCX354A-QCBT ). Each host is plugged into both switches, swithes are connected to each other and use shared fabric. All hosts are based on SuperMicro X9DRW-7TPF mainboards with Intel Xeon E5-2667
v2 CPUs and DDR3-1866 memory.<o:p></o:p></span></p>
</div>
<div>
<p class="MsoNormal"><span style="font-size:10.0pt;font-family:"Arial","sans-serif"">WinOFED 4.40.0 in installed on Win Server 2012, WinOFED 4.55 on Win Server 2012R2. Both linux hosts are routers MLNX_OFED_LINUX-2.0-3.0.0-ubuntu12.04-x86_64 packet installed,
IB interfaces are joined into active-backup bond by ifenslave means. Following modules are loaded on Linux routers: mlx4_core, mlx4_ib, b_umad, ib_mad, ib_ipoib, ib_uverbs. All HCAs are burned with 2.30.3200 firmware.<o:p></o:p></span></p>
</div>
<div>
<p class="MsoNormal"><span style="font-size:10.0pt;font-family:"Arial","sans-serif"">As test I use L3 icmp ping. In case of Linux - Linux communications all is fine. I'm doing flood ping through Infiniband network with amazing results: rtt min/avg/max/mdev
= 0.011/0.013/1.682/0.002 ms. But I see different when it goes to Windows.<o:p></o:p></span></p>
</div>
<div>
<p class="MsoNormal"><span style="font-size:10.0pt;font-family:"Arial","sans-serif"">Flood ping from linux host to w2012 gives almost same good latency numbers (rtt min/avg/max/mdev = 0.022/0.024/2.492/0.021 ms), but packet loss rate is always about 1-2%. At
same time, IBping shows no packet loss at all and ibdiagnet on Linux show no warnings or errors, so I conclude IB works good and issue exist higher than L2.<o:p></o:p></span></p>
</div>
<div>
<p class="MsoNormal"><span style="font-size:10.0pt;font-family:"Arial","sans-serif"">So I've decided to try Win2012R2 with 4.55 OFED version. It resolved issue with packet loss, but also gave latency growth: rtt min/avg/max/mdev = 0.093/0.102/17.550/0.101 ms.
Digging this issue I've found that in other system I have no issues like it, and the difference is a firmware version of HCA. Win2012 with OFED 4.40.0 goes fine with firmware 2.11.500. But downgrading to fw 2.11.500 on my servers didn't help, with all same
versions of fw and software still I see packet loss.<o:p></o:p></span></p>
</div>
<div>
<p class="MsoNormal"><span style="font-size:10.0pt;font-family:"Arial","sans-serif"">Still I want it all together - low latency, no packet loss, and latest software and firmware versions.<o:p></o:p></span></p>
</div>
<div>
<p class="MsoNormal"><span style="font-size:10.0pt;font-family:"Arial","sans-serif""><o:p> </o:p></span></p>
</div>
<div>
<p class="MsoNormal"><span style="font-size:10.0pt;font-family:"Arial","sans-serif"">Running out of ideas about it, any comments and advises are appreciated.<o:p></o:p></span></p>
</div>
<div>
<p class="MsoNormal"><span style="font-size:10.0pt;font-family:"Arial","sans-serif""><o:p> </o:p></span></p>
</div>
<div>
<p class="MsoNormal"><span style="font-size:10.0pt;font-family:"Arial","sans-serif"">--<o:p></o:p></span></p>
</div>
<div>
<p class="MsoNormal"><span style="font-size:10.0pt;font-family:"Arial","sans-serif"">Regards,<o:p></o:p></span></p>
</div>
<div>
<p class="MsoNormal"><span style="font-size:10.0pt;font-family:"Arial","sans-serif"">Alexey<o:p></o:p></span></p>
</div>
</div>
</div>
</body>
</html>