<html xmlns:v="urn:schemas-microsoft-com:vml" xmlns:o="urn:schemas-microsoft-com:office:office" xmlns:w="urn:schemas-microsoft-com:office:word" xmlns:m="http://schemas.microsoft.com/office/2004/12/omml" xmlns="http://www.w3.org/TR/REC-html40">
<head>
<meta http-equiv="Content-Type" content="text/html; charset=utf-8">
<meta name="Generator" content="Microsoft Word 15 (filtered medium)">
<style><!--
/* Font Definitions */
@font-face
{font-family:"Cambria Math";
panose-1:2 4 5 3 5 4 6 3 2 4;}
@font-face
{font-family:Calibri;
panose-1:2 15 5 2 2 2 4 3 2 4;}
@font-face
{font-family:Consolas;
panose-1:2 11 6 9 2 2 4 3 2 4;}
@font-face
{font-family:"Times New Roman \,serif";}
/* Style Definitions */
p.MsoNormal, li.MsoNormal, div.MsoNormal
{margin:0in;
margin-bottom:.0001pt;
font-size:11.0pt;
font-family:"Calibri",sans-serif;
color:black;}
a:link, span.MsoHyperlink
{mso-style-priority:99;
color:#0563C1;
text-decoration:underline;}
a:visited, span.MsoHyperlinkFollowed
{mso-style-priority:99;
color:#954F72;
text-decoration:underline;}
p.MsoPlainText, li.MsoPlainText, div.MsoPlainText
{mso-style-priority:99;
mso-style-link:"Plain Text Char";
margin:0in;
margin-bottom:.0001pt;
font-size:11.0pt;
font-family:"Calibri",sans-serif;}
p
{mso-style-priority:99;
mso-margin-top-alt:auto;
margin-right:0in;
mso-margin-bottom-alt:auto;
margin-left:0in;
font-size:12.0pt;
font-family:"Times New Roman",serif;}
pre
{mso-style-priority:99;
mso-style-link:"HTML Preformatted Char";
margin:0in;
margin-bottom:.0001pt;
font-size:10.0pt;
font-family:"Courier New";
color:black;}
span.HTMLPreformattedChar
{mso-style-name:"HTML Preformatted Char";
mso-style-priority:99;
mso-style-link:"HTML Preformatted";
font-family:"Consolas",serif;
color:black;}
p.msonormal0, li.msonormal0, div.msonormal0
{mso-style-name:msonormal;
mso-style-priority:99;
mso-margin-top-alt:auto;
margin-right:0in;
mso-margin-bottom-alt:auto;
margin-left:0in;
font-size:12.0pt;
font-family:"Times New Roman",serif;
color:black;}
span.PlainTextChar
{mso-style-name:"Plain Text Char";
mso-style-priority:99;
mso-style-link:"Plain Text";
font-family:"Calibri",sans-serif;}
span.EmailStyle23
{mso-style-type:personal;
font-family:"Calibri",sans-serif;
color:windowtext;}
span.EmailStyle24
{mso-style-type:personal;
font-family:"Calibri",sans-serif;
color:#1F497D;}
span.EmailStyle25
{mso-style-type:personal;
font-family:"Calibri",sans-serif;
color:#1F497D;}
span.EmailStyle26
{mso-style-type:personal;
font-family:"Calibri",sans-serif;
color:#1F497D;}
span.apple-converted-space
{mso-style-name:apple-converted-space;}
span.EmailStyle28
{mso-style-type:personal;
font-family:"Calibri",sans-serif;
font-variant:normal !important;
color:windowtext;
text-transform:none;
font-weight:normal;
font-style:normal;
text-decoration:none none;
vertical-align:baseline;}
span.EmailStyle29
{mso-style-type:personal;
font-family:"Calibri",sans-serif;
color:#1F497D;}
span.EmailStyle30
{mso-style-type:personal;
font-family:"Calibri",sans-serif;
color:#1F497D;}
span.EmailStyle31
{mso-style-type:personal;
font-family:"Calibri",sans-serif;
color:#1F497D;}
span.EmailStyle32
{mso-style-type:personal;
font-family:"Calibri",sans-serif;
color:#1F497D;}
span.EmailStyle33
{mso-style-type:personal;
font-family:"Calibri",sans-serif;
color:#1F497D;}
span.EmailStyle34
{mso-style-type:personal;
font-family:"Calibri",sans-serif;
color:#1F497D;}
span.EmailStyle35
{mso-style-type:personal;
font-family:"Calibri",sans-serif;
color:#1F497D;}
span.EmailStyle36
{mso-style-type:personal;
font-family:"Calibri",sans-serif;
color:windowtext;}
span.EmailStyle37
{mso-style-type:personal;
font-family:"Calibri",sans-serif;
color:#1F497D;}
span.EmailStyle38
{mso-style-type:personal;
font-family:"Calibri",sans-serif;
color:#1F497D;}
span.EmailStyle39
{mso-style-type:personal;
font-family:"Calibri",sans-serif;
color:#1F497D;}
span.EmailStyle40
{mso-style-type:personal;
font-family:"Calibri",sans-serif;
color:#1F497D;}
span.EmailStyle41
{mso-style-type:personal;
font-family:"Calibri",sans-serif;
color:#1F497D;}
span.EmailStyle42
{mso-style-type:personal;
font-family:"Calibri",sans-serif;
color:#1F497D;}
span.EmailStyle44
{mso-style-type:personal-reply;
font-family:"Calibri",sans-serif;
color:#1F497D;}
.MsoChpDefault
{mso-style-type:export-only;
font-size:10.0pt;}
@page WordSection1
{size:8.5in 11.0in;
margin:70.85pt 70.85pt 70.85pt 70.85pt;}
div.WordSection1
{page:WordSection1;}
--></style><!--[if gte mso 9]><xml>
<o:shapedefaults v:ext="edit" spidmax="1026" />
</xml><![endif]--><!--[if gte mso 9]><xml>
<o:shapelayout v:ext="edit">
<o:idmap v:ext="edit" data="1" />
</o:shapelayout></xml><![endif]-->
</head>
<body bgcolor="white" lang="EN-US" link="#0563C1" vlink="#954F72">
<div class="WordSection1">
<p class="MsoNormal"><span style="color:#1F497D">The upstream test was for Bug 2631, SRP Reject Issue. The Base OS was SL 7.3 (not SL 7.2) w/ OFED 4.8-rc4 and the ib_srp_backport_4_11. That combination worked properly.<o:p></o:p></span></p>
<p class="MsoNormal"><span style="color:#1F497D"><o:p> </o:p></span></p>
<div>
<div style="border:none;border-top:solid #E1E1E1 1.0pt;padding:3.0pt 0in 0in 0in">
<p class="MsoNormal"><b><span style="color:windowtext">From:</span></b><span style="color:windowtext"> Davis, Arlin R [mailto:arlin.r.davis@intel.com]
<br>
<b>Sent:</b> Monday, June 19, 2017 4:35 PM<br>
<b>To:</b> Marty Schlining <mschlining@ddn.com>; Woodruff, Robert J <robert.j.woodruff@intel.com>; RSD@SFI <rsdance@soft-forge.com>; 'Vladimir Sokolovsky' <vlad@dev.mellanox.co.il><br>
<b>Cc:</b> bart.vanassche@gmail.com; ewg@lists.openfabrics.org; Cedric Fernandes <cfernandes@ddn.com>; Mike Davis <mdavis@ddn.com><br>
<b>Subject:</b> RE: [ewg] [GIT PULL compat-rdma] qib for OFED 4.8<o:p></o:p></span></p>
</div>
</div>
<p class="MsoNormal"><o:p> </o:p></p>
<p class="MsoNormal"><span style="color:#1F497D">I tested SL 7.2 with OFED 4.8-rc4. Scanning SRP targets yielded the following response:<o:p></o:p></span></p>
<p class="MsoNormal"><span style="color:#1F497D"><o:p> </o:p></span></p>
<p class="MsoNormal"><span style="color:windowtext"> Ok, so this is a kernel.org 4.8 base SRP driver backported to 3.10.
<o:p></o:p></span></p>
<p class="MsoNormal"><span style="color:windowtext"> Can someone tell me if the same problem exists with upstream SRP driver in kernel 4.8?<o:p></o:p></span></p>
<p class="MsoNormal"><span style="color:windowtext"> This will tell me if it is an OFED 4.8 backport issue or if it is a kernel 4.8 SRP driver issue.<o:p></o:p></span></p>
<p class="MsoNormal"><span style="color:#1F497D"><o:p> </o:p></span></p>
<p class="MsoNormal"><span style="color:#1F497D">ib_srp: Sending CM DREQ failed and the SRP targets were never scanned. (See Bug 2632.
<a href="http://bugs.openfabrics.org/show_bug.cgi?id=2632">http://bugs.openfabrics.org/show_bug.cgi?id=2632</a>)<o:p></o:p></span></p>
<p class="MsoNormal"><span style="color:#1F497D"><o:p> </o:p></span></p>
<p class="MsoNormal"><span style="color:#1F497D">I attempted to use Bart Van Assche’s ib_srp_backport driver on top of SL 7.2 and OFED 4.8-rc4 according to his instructions. The ib_srp_backport driver failed to compile (see attached text file). The ib_srp_backport
driver will not compile with OFED 4.8-rc4 installed. I was able to uninstall OFED 4.8-rc4, then compile the driver. However, when OFED 4.8-rc4 was reinstalled, the ib_srp_backport driver was not compatible.<o:p></o:p></span></p>
<p class="MsoNormal"><span style="color:#1F497D"><o:p> </o:p></span></p>
<p class="MsoNormal"><span style="color:windowtext"> Ok, I think I understand.
<o:p></o:p></span></p>
<p class="MsoNormal"><span style="color:windowtext"> This is the upstream test that you said passed? A SL 7.2+infiniband+ib_srp_backport_from_4.11?
<o:p></o:p></span></p>
<p class="MsoNormal"><span style="color:windowtext"> Correct?<o:p></o:p></span></p>
<p class="MsoNormal"><span style="color:#1F497D"><o:p> </o:p></span></p>
<p class="MsoNormal"><span style="color:#1F497D"> </span><span style="color:windowtext"> -arlin<o:p></o:p></span></p>
<p class="MsoNormal"><span style="color:#1F497D"><o:p> </o:p></span></p>
<p class="MsoNormal"><span style="color:#1F497D">That’s all of the testing , so far.<o:p></o:p></span></p>
<p class="MsoNormal"><span style="color:#1F497D"><o:p> </o:p></span></p>
<p class="MsoNormal"><span style="color:#1F497D">-Marty<o:p></o:p></span></p>
<p class="MsoNormal"><span style="color:#1F497D"><o:p> </o:p></span></p>
<div>
<div style="border:none;border-top:solid #E1E1E1 1.0pt;padding:3.0pt 0in 0in 0in">
<p class="MsoNormal"><b><span style="color:windowtext">From:</span></b><span style="color:windowtext"> Davis, Arlin R [<a href="mailto:arlin.r.davis@intel.com">mailto:arlin.r.davis@intel.com</a>]
<br>
<b>Sent:</b> Monday, June 19, 2017 4:02 PM<br>
<b>To:</b> Marty Schlining <<a href="mailto:mschlining@ddn.com">mschlining@ddn.com</a>>; Woodruff, Robert J <<a href="mailto:robert.j.woodruff@intel.com">robert.j.woodruff@intel.com</a>>; RSD@SFI <<a href="mailto:rsdance@soft-forge.com">rsdance@soft-forge.com</a>>;
'Vladimir Sokolovsky' <<a href="mailto:vlad@dev.mellanox.co.il">vlad@dev.mellanox.co.il</a>><br>
<b>Cc:</b> <a href="mailto:bart.vanassche@gmail.com">bart.vanassche@gmail.com</a>;
<a href="mailto:ewg@lists.openfabrics.org">ewg@lists.openfabrics.org</a>; Cedric Fernandes <<a href="mailto:cfernandes@ddn.com">cfernandes@ddn.com</a>>; Mike Davis <<a href="mailto:mdavis@ddn.com">mdavis@ddn.com</a>><br>
<b>Subject:</b> RE: [ewg] [GIT PULL compat-rdma] qib for OFED 4.8<o:p></o:p></span></p>
</div>
</div>
<p class="MsoNormal"><o:p> </o:p></p>
<p class="MsoNormal"><span style="color:#1F497D">Sorry, I am still trying to understand what is working and what is needed to fix this for OFED 4.8.
<o:p></o:p></span></p>
<p class="MsoNormal"><span style="color:#1F497D"><o:p> </o:p></span></p>
<p class="MsoNormal"><span style="color:#1F497D">When you said you tested the upstream kernel, was it SL 7.2 kernel plus the backports from Bart’s 4.11 drop? Do you need the 4.11 kernel base driver set for SRP to work properly?<o:p></o:p></span></p>
<p class="MsoNormal"><span style="color:#1F497D"><o:p> </o:p></span></p>
<p class="MsoNormal"><span style="color:#1F497D">-arlin<o:p></o:p></span></p>
<p class="MsoNormal"><span style="color:#1F497D"><o:p> </o:p></span></p>
<div style="border:none;border-left:solid blue 1.5pt;padding:0in 0in 0in 4.0pt">
<div>
<div style="border:none;border-top:solid #E1E1E1 1.0pt;padding:3.0pt 0in 0in 0in">
<p class="MsoNormal"><b><span style="color:windowtext">From:</span></b><span style="color:windowtext"> Marty Schlining [<a href="mailto:mschlining@ddn.com">mailto:mschlining@ddn.com</a>]
<br>
<b>Sent:</b> Monday, June 19, 2017 9:42 AM<br>
<b>To:</b> Davis, Arlin R <<a href="mailto:arlin.r.davis@intel.com">arlin.r.davis@intel.com</a>>; Woodruff, Robert J <<a href="mailto:robert.j.woodruff@intel.com">robert.j.woodruff@intel.com</a>>; RSD@SFI <<a href="mailto:rsdance@soft-forge.com">rsdance@soft-forge.com</a>>;
'Vladimir Sokolovsky' <<a href="mailto:vlad@dev.mellanox.co.il">vlad@dev.mellanox.co.il</a>><br>
<b>Cc:</b> <a href="mailto:bart.vanassche@gmail.com">bart.vanassche@gmail.com</a>;
<a href="mailto:ewg@lists.openfabrics.org">ewg@lists.openfabrics.org</a>; Cedric Fernandes <<a href="mailto:cfernandes@ddn.com">cfernandes@ddn.com</a>>; Mike Davis <<a href="mailto:mdavis@ddn.com">mdavis@ddn.com</a>><br>
<b>Subject:</b> RE: [ewg] [GIT PULL compat-rdma] qib for OFED 4.8<o:p></o:p></span></p>
</div>
</div>
<p class="MsoNormal"><o:p> </o:p></p>
<p class="MsoNormal"><span style="color:#1F497D">No upstream kernel. SL 7.2 (3.10.0-327.el7.x86_64), OFED-4.8-rc4. Was unable to build the backport ib_srp backport driver on this platform with OFED 4.8-rc4 installed. Multiple build errors. However, if I uninstall
OFED-4.8-rc4, I am able to build the upstream ib_srp RPMs on SL 7.2<o:p></o:p></span></p>
<p class="MsoNormal"><span style="color:#1F497D"><o:p> </o:p></span></p>
<p class="MsoNormal"><span style="color:#1F497D">Backport ib_srp driver obtained by the following mechanism:<o:p></o:p></span></p>
<p class="MsoNormal"><span style="color:#1F497D"><o:p> </o:p></span></p>
<p class="MsoNormal"><span style="color:#1F497D">git clone </span><a href="https://github.com/bvanassche/ib_srp-backport.git">https://github.com/bvanassche/ib_srp-backport.git</a><span style="color:#1F497D"><o:p></o:p></span></p>
<p class="MsoNormal"><span style="color:#1F497D"><o:p> </o:p></span></p>
<p class="MsoNormal"><span style="color:#1F497D">-Marty<o:p></o:p></span></p>
<p class="MsoNormal"><span style="color:#1F497D"><o:p> </o:p></span></p>
<p class="MsoNormal"><span style="color:#1F497D"><o:p> </o:p></span></p>
<div>
<div style="border:none;border-top:solid #E1E1E1 1.0pt;padding:3.0pt 0in 0in 0in">
<p class="MsoNormal"><b><span style="color:windowtext">From:</span></b><span style="color:windowtext"> Davis, Arlin R [</span><a href="mailto:arlin.r.davis@intel.com">mailto:arlin.r.davis@intel.com</a><span style="color:windowtext">]
<br>
<b>Sent:</b> Monday, June 19, 2017 12:36 PM<br>
<b>To:</b> Marty Schlining <</span><a href="mailto:mschlining@ddn.com">mschlining@ddn.com</a><span style="color:windowtext">>; Woodruff, Robert J <</span><a href="mailto:robert.j.woodruff@intel.com">robert.j.woodruff@intel.com</a><span style="color:windowtext">>;
RSD@SFI <</span><a href="mailto:rsdance@soft-forge.com">rsdance@soft-forge.com</a><span style="color:windowtext">>; 'Vladimir Sokolovsky' <</span><a href="mailto:vlad@dev.mellanox.co.il">vlad@dev.mellanox.co.il</a><span style="color:windowtext">><br>
<b>Cc:</b> </span><a href="mailto:bart.vanassche@gmail.com">bart.vanassche@gmail.com</a><span style="color:windowtext">;
</span><a href="mailto:ewg@lists.openfabrics.org">ewg@lists.openfabrics.org</a><span style="color:windowtext">; Cedric Fernandes <</span><a href="mailto:cfernandes@ddn.com">cfernandes@ddn.com</a><span style="color:windowtext">>; Mike Davis <</span><a href="mailto:mdavis@ddn.com">mdavis@ddn.com</a><span style="color:windowtext">><br>
<b>Subject:</b> RE: [ewg] [GIT PULL compat-rdma] qib for OFED 4.8<o:p></o:p></span></p>
</div>
</div>
<p class="MsoNormal"><o:p> </o:p></p>
<p class="MsoNormal"><span style="color:#1F497D">Marty,<o:p></o:p></span></p>
<p class="MsoNormal"><span style="color:#1F497D"><o:p> </o:p></span></p>
<p class="MsoNormal"><span style="color:#1F497D">What upstream kernel did you use to verify SRP? Was it on a 4.8 base or something newer? If newer, maybe we are missing some key SRP fixes in our OFED 4.8 kernel base.<o:p></o:p></span></p>
<p class="MsoNormal"><span style="color:#1F497D"><o:p> </o:p></span></p>
<p class="MsoNormal"><span style="color:#1F497D">Thanks, Arlin<o:p></o:p></span></p>
<p class="MsoNormal"><span style="color:#1F497D"><o:p> </o:p></span></p>
<div style="border:none;border-left:solid blue 1.5pt;padding:0in 0in 0in 4.0pt">
<div>
<div style="border:none;border-top:solid #E1E1E1 1.0pt;padding:3.0pt 0in 0in 0in">
<p class="MsoNormal"><b><span style="color:windowtext">From:</span></b><span style="color:windowtext"> Marty Schlining [</span><a href="mailto:mschlining@ddn.com">mailto:mschlining@ddn.com</a><span style="color:windowtext">]
<br>
<b>Sent:</b> Friday, June 16, 2017 1:00 PM<br>
<b>To:</b> Davis, Arlin R <</span><a href="mailto:arlin.r.davis@intel.com">arlin.r.davis@intel.com</a><span style="color:windowtext">>; Woodruff, Robert J <</span><a href="mailto:robert.j.woodruff@intel.com">robert.j.woodruff@intel.com</a><span style="color:windowtext">>;
RSD@SFI <</span><a href="mailto:rsdance@soft-forge.com">rsdance@soft-forge.com</a><span style="color:windowtext">>; 'Vladimir Sokolovsky' <</span><a href="mailto:vlad@dev.mellanox.co.il">vlad@dev.mellanox.co.il</a><span style="color:windowtext">><br>
<b>Cc:</b> </span><a href="mailto:bart.vanassche@gmail.com">bart.vanassche@gmail.com</a><span style="color:windowtext">;
</span><a href="mailto:ewg@lists.openfabrics.org">ewg@lists.openfabrics.org</a><span style="color:windowtext">; Cedric Fernandes <</span><a href="mailto:cfernandes@ddn.com">cfernandes@ddn.com</a><span style="color:windowtext">>; Mike Davis <</span><a href="mailto:mdavis@ddn.com">mdavis@ddn.com</a><span style="color:windowtext">><br>
<b>Subject:</b> RE: [ewg] [GIT PULL compat-rdma] qib for OFED 4.8<o:p></o:p></span></p>
</div>
</div>
<p class="MsoNormal"><o:p> </o:p></p>
<p class="MsoNormal"><span style="color:#1F497D">I can’t say for sure. I am attempting to build the upstream ib_srp driver for this platform, but I am running into build issues. Bart, I will send you a separate email on that subject.<o:p></o:p></span></p>
<p class="MsoNormal"><span style="color:#1F497D"><o:p> </o:p></span></p>
<div>
<div style="border:none;border-top:solid #E1E1E1 1.0pt;padding:3.0pt 0in 0in 0in">
<p class="MsoNormal"><b><span style="color:windowtext">From:</span></b><span style="color:windowtext"> Davis, Arlin R [</span><a href="mailto:arlin.r.davis@intel.com">mailto:arlin.r.davis@intel.com</a><span style="color:windowtext">]
<br>
<b>Sent:</b> Friday, June 16, 2017 3:58 PM<br>
<b>To:</b> Woodruff, Robert J <</span><a href="mailto:robert.j.woodruff@intel.com">robert.j.woodruff@intel.com</a><span style="color:windowtext">>; Marty Schlining <</span><a href="mailto:mschlining@ddn.com">mschlining@ddn.com</a><span style="color:windowtext">>;
RSD@SFI <</span><a href="mailto:rsdance@soft-forge.com">rsdance@soft-forge.com</a><span style="color:windowtext">>; 'Vladimir Sokolovsky' <</span><a href="mailto:vlad@dev.mellanox.co.il">vlad@dev.mellanox.co.il</a><span style="color:windowtext">><br>
<b>Cc:</b> </span><a href="mailto:bart.vanassche@gmail.com">bart.vanassche@gmail.com</a><span style="color:windowtext">;
</span><a href="mailto:ewg@lists.openfabrics.org">ewg@lists.openfabrics.org</a><span style="color:windowtext">; Cedric Fernandes <</span><a href="mailto:cfernandes@ddn.com">cfernandes@ddn.com</a><span style="color:windowtext">>; Mike Davis <</span><a href="mailto:mdavis@ddn.com">mdavis@ddn.com</a><span style="color:windowtext">><br>
<b>Subject:</b> RE: [ewg] [GIT PULL compat-rdma] qib for OFED 4.8<o:p></o:p></span></p>
</div>
</div>
<p class="MsoNormal"><o:p> </o:p></p>
<p class="MsoNormal"><span style="color:windowtext">Another SRP critical issue just opened. Is this also a backport issue?
<o:p></o:p></span></p>
<p class="MsoNormal"><span style="color:#1F497D"><o:p> </o:p></span></p>
<p class="MsoNormal"><a href="http://bugs.openfabrics.org/show_bug.cgi?id=2632"><b><span style="color:#3D4A68;text-decoration:none">Bug 2632</span></b></a><span class="apple-converted-space"><b> -
</b></span>SRP Login failure for SL 7.2 and OFED 4.8-rc4 (ib_srp: Sending CM DREQ failed)<span style="color:#1F497D"><o:p></o:p></span></p>
<p class="MsoNormal"><span style="color:#1F497D"><o:p> </o:p></span></p>
<p class="MsoNormal"><span style="color:#1F497D"><o:p> </o:p></span></p>
<div style="border:none;border-left:solid blue 1.5pt;padding:0in 0in 0in 4.0pt">
<div>
<div style="border:none;border-top:solid #E1E1E1 1.0pt;padding:3.0pt 0in 0in 0in">
<p class="MsoNormal"><b><span style="color:windowtext">From:</span></b><span style="color:windowtext"> Woodruff, Robert J
<br>
<b>Sent:</b> Friday, June 16, 2017 10:24 AM<br>
<b>To:</b> Marty Schlining <</span><a href="mailto:mschlining@ddn.com">mschlining@ddn.com</a><span style="color:windowtext">>; Davis, Arlin R <</span><a href="mailto:arlin.r.davis@intel.com">arlin.r.davis@intel.com</a><span style="color:windowtext">>; RSD@SFI
<</span><a href="mailto:rsdance@soft-forge.com">rsdance@soft-forge.com</a><span style="color:windowtext">>; 'Vladimir Sokolovsky' <</span><a href="mailto:vlad@dev.mellanox.co.il">vlad@dev.mellanox.co.il</a><span style="color:windowtext">><br>
<b>Cc:</b> </span><a href="mailto:bart.vanassche@gmail.com">bart.vanassche@gmail.com</a><span style="color:windowtext">;
</span><a href="mailto:ewg@lists.openfabrics.org">ewg@lists.openfabrics.org</a><span style="color:windowtext">; Cedric Fernandes <</span><a href="mailto:cfernandes@ddn.com">cfernandes@ddn.com</a><span style="color:windowtext">>; Mike Davis <</span><a href="mailto:mdavis@ddn.com">mdavis@ddn.com</a><span style="color:windowtext">><br>
<b>Subject:</b> RE: [ewg] [GIT PULL compat-rdma] qib for OFED 4.8<o:p></o:p></span></p>
</div>
</div>
<p class="MsoNormal"><o:p> </o:p></p>
<p class="MsoNormal"><span style="color:#1F497D">Ok, then it sound like it is important enough to hold the release until we have a fix. I assume this is something that you can look at Vlad ?<o:p></o:p></span></p>
<p class="MsoNormal"><span style="color:#1F497D"><o:p> </o:p></span></p>
<div>
<div style="border:none;border-top:solid #E1E1E1 1.0pt;padding:3.0pt 0in 0in 0in">
<p class="MsoNormal"><b><span style="color:windowtext">From:</span></b><span style="color:windowtext"> Marty Schlining [</span><a href="mailto:mschlining@ddn.com">mailto:mschlining@ddn.com</a><span style="color:windowtext">]
<br>
<b>Sent:</b> Friday, June 16, 2017 10:17 AM<br>
<b>To:</b> Woodruff, Robert J <</span><a href="mailto:robert.j.woodruff@intel.com">robert.j.woodruff@intel.com</a><span style="color:windowtext">>; Davis, Arlin R <</span><a href="mailto:arlin.r.davis@intel.com">arlin.r.davis@intel.com</a><span style="color:windowtext">>;
RSD@SFI <</span><a href="mailto:rsdance@soft-forge.com">rsdance@soft-forge.com</a><span style="color:windowtext">>; 'Vladimir Sokolovsky' <</span><a href="mailto:vlad@dev.mellanox.co.il">vlad@dev.mellanox.co.il</a><span style="color:windowtext">><br>
<b>Cc:</b> </span><a href="mailto:bart.vanassche@gmail.com">bart.vanassche@gmail.com</a><span style="color:windowtext">;
</span><a href="mailto:ewg@lists.openfabrics.org">ewg@lists.openfabrics.org</a><span style="color:windowtext">; Cedric Fernandes <</span><a href="mailto:cfernandes@ddn.com">cfernandes@ddn.com</a><span style="color:windowtext">>; Mike Davis <</span><a href="mailto:mdavis@ddn.com">mdavis@ddn.com</a><span style="color:windowtext">><br>
<b>Subject:</b> RE: [ewg] [GIT PULL compat-rdma] qib for OFED 4.8<o:p></o:p></span></p>
</div>
</div>
<p class="MsoNormal"><o:p> </o:p></p>
<p class="MsoNormal"><span style="color:#1F497D">I tested the upstream driver from Bart VanAssche and it does not have the same issue as the OFED-4.8-rc4 ib_srp driver. The upstream driver handles the SRP_Reject as expected without crashing the SL 7.3 kernel.
That is also detailed in the defect.<o:p></o:p></span></p>
<p class="MsoNormal"><span style="color:#1F497D"><o:p> </o:p></span></p>
<div>
<div style="border:none;border-top:solid #E1E1E1 1.0pt;padding:3.0pt 0in 0in 0in">
<p class="MsoNormal"><b><span style="color:windowtext">From:</span></b><span style="color:windowtext"> Woodruff, Robert J [</span><a href="mailto:robert.j.woodruff@intel.com">mailto:robert.j.woodruff@intel.com</a><span style="color:windowtext">]
<br>
<b>Sent:</b> Friday, June 16, 2017 1:14 PM<br>
<b>To:</b> Marty Schlining <</span><a href="mailto:mschlining@ddn.com">mschlining@ddn.com</a><span style="color:windowtext">>; Davis, Arlin R <</span><a href="mailto:arlin.r.davis@intel.com">arlin.r.davis@intel.com</a><span style="color:windowtext">>; RSD@SFI
<</span><a href="mailto:rsdance@soft-forge.com">rsdance@soft-forge.com</a><span style="color:windowtext">>; 'Vladimir Sokolovsky' <</span><a href="mailto:vlad@dev.mellanox.co.il">vlad@dev.mellanox.co.il</a><span style="color:windowtext">><br>
<b>Cc:</b> </span><a href="mailto:bart.vanassche@gmail.com">bart.vanassche@gmail.com</a><span style="color:windowtext">;
</span><a href="mailto:ewg@lists.openfabrics.org">ewg@lists.openfabrics.org</a><span style="color:windowtext">; Cedric Fernandes <</span><a href="mailto:cfernandes@ddn.com">cfernandes@ddn.com</a><span style="color:windowtext">>; Mike Davis <</span><a href="mailto:mdavis@ddn.com">mdavis@ddn.com</a><span style="color:windowtext">><br>
<b>Subject:</b> RE: [ewg] [GIT PULL compat-rdma] qib for OFED 4.8<o:p></o:p></span></p>
</div>
</div>
<p class="MsoNormal"><o:p> </o:p></p>
<p class="MsoNormal"><span style="color:#1F497D">So I guess the next question is, does anyone have a fix that can be included ? Also, do you know if this is a problem with the backport or is it also broken in the upstream kernel ?<o:p></o:p></span></p>
<p class="MsoNormal"><span style="color:#1F497D"><o:p> </o:p></span></p>
<div>
<div style="border:none;border-top:solid #E1E1E1 1.0pt;padding:3.0pt 0in 0in 0in">
<p class="MsoNormal"><b><span style="color:windowtext">From:</span></b><span style="color:windowtext"> Marty Schlining [</span><a href="mailto:mschlining@ddn.com">mailto:mschlining@ddn.com</a><span style="color:windowtext">]
<br>
<b>Sent:</b> Friday, June 16, 2017 10:03 AM<br>
<b>To:</b> Woodruff, Robert J <</span><a href="mailto:robert.j.woodruff@intel.com">robert.j.woodruff@intel.com</a><span style="color:windowtext">>; Davis, Arlin R <</span><a href="mailto:arlin.r.davis@intel.com">arlin.r.davis@intel.com</a><span style="color:windowtext">>;
RSD@SFI <</span><a href="mailto:rsdance@soft-forge.com">rsdance@soft-forge.com</a><span style="color:windowtext">>; 'Vladimir Sokolovsky' <</span><a href="mailto:vlad@dev.mellanox.co.il">vlad@dev.mellanox.co.il</a><span style="color:windowtext">><br>
<b>Cc:</b> </span><a href="mailto:bart.vanassche@gmail.com">bart.vanassche@gmail.com</a><span style="color:windowtext">;
</span><a href="mailto:ewg@lists.openfabrics.org">ewg@lists.openfabrics.org</a><span style="color:windowtext">; Cedric Fernandes <</span><a href="mailto:cfernandes@ddn.com">cfernandes@ddn.com</a><span style="color:windowtext">>; Mike Davis <</span><a href="mailto:mdavis@ddn.com">mdavis@ddn.com</a><span style="color:windowtext">><br>
<b>Subject:</b> RE: [ewg] [GIT PULL compat-rdma] qib for OFED 4.8<o:p></o:p></span></p>
</div>
</div>
<p class="MsoNormal"><o:p> </o:p></p>
<p class="MsoNormal"><span style="color:#1F497D">I do not think this is an acceptable workaround. An SRP target could have many hosts connected (IO nodes), not just the one that may have been abruptly rebooted (due to a power failure or pulled power cord).
Are you suggesting that customers must reboot all of the IO nodes in their cluster attached to the SRP target and the SRP target as a workaround thereby taking the entire file system offline?<o:p></o:p></span></p>
<p class="MsoNormal"><span style="color:#1F497D"><o:p> </o:p></span></p>
<p class="MsoNormal"><span style="color:#1F497D">The backported ib_srp driver has a defect where it is not handling an SRP_Reject properly leaving a null pointer for the blk_mq driver. Normally, this would just be a note in the log and the srp_daemon could
be setup to attempt a reconnect after 30 seconds. But, that is no longer possible with the current defect. The host would crash and reboot every time a connection is attempted.<o:p></o:p></span></p>
<p class="MsoNormal"><span style="color:#1F497D"><o:p> </o:p></span></p>
<p class="MsoNormal"><span style="color:#1F497D">-Marty Schlining<o:p></o:p></span></p>
<p class="MsoNormal"><span style="color:#1F497D"><o:p> </o:p></span></p>
<div>
<div style="border:none;border-top:solid #E1E1E1 1.0pt;padding:3.0pt 0in 0in 0in">
<p class="MsoNormal"><b><span style="color:windowtext">From:</span></b><span style="color:windowtext"> Woodruff, Robert J [</span><a href="mailto:robert.j.woodruff@intel.com">mailto:robert.j.woodruff@intel.com</a><span style="color:windowtext">]
<br>
<b>Sent:</b> Friday, June 16, 2017 12:18 PM<br>
<b>To:</b> Davis, Arlin R <</span><a href="mailto:arlin.r.davis@intel.com">arlin.r.davis@intel.com</a><span style="color:windowtext">>; RSD@SFI <</span><a href="mailto:rsdance@soft-forge.com">rsdance@soft-forge.com</a><span style="color:windowtext">>; 'Vladimir
Sokolovsky' <</span><a href="mailto:vlad@dev.mellanox.co.il">vlad@dev.mellanox.co.il</a><span style="color:windowtext">><br>
<b>Cc:</b> </span><a href="mailto:bart.vanassche@gmail.com">bart.vanassche@gmail.com</a><span style="color:windowtext">;
</span><a href="mailto:ewg@lists.openfabrics.org">ewg@lists.openfabrics.org</a><span style="color:windowtext">; Marty Schlining <</span><a href="mailto:mschlining@ddn.com">mschlining@ddn.com</a><span style="color:windowtext">><br>
<b>Subject:</b> RE: [ewg] [GIT PULL compat-rdma] qib for OFED 4.8<o:p></o:p></span></p>
</div>
</div>
<p class="MsoNormal"><o:p> </o:p></p>
<p class="MsoNormal"><span style="color:#1F497D">When I read the bug, it looks like the work around is to also reboot the target whenever the host is rebooted. Is this work around acceptable for the short term, and then fix the<br>
issue in OFED-4.8-1, which will be a quick turnaround release since the new functionality being added to OFED-4.8-1 is limited. If people are OK with that, I would recommend moving<br>
to GA for OFED-4.8 and then start work on OFED-4.8-1 right away. OFED-4.8 has been dragging on forever and there are people that will want to start using it.
<o:p></o:p></span></p>
<p class="MsoNormal"><span style="color:#1F497D"><o:p> </o:p></span></p>
<div>
<div style="border:none;border-top:solid #E1E1E1 1.0pt;padding:3.0pt 0in 0in 0in">
<p class="MsoNormal"><b><span style="color:windowtext">From:</span></b><span style="color:windowtext"> ewg [</span><a href="mailto:ewg-bounces@lists.openfabrics.org">mailto:ewg-bounces@lists.openfabrics.org</a><span style="color:windowtext">]
<b>On Behalf Of </b>Davis, Arlin R<br>
<b>Sent:</b> Friday, June 16, 2017 9:13 AM<br>
<b>To:</b> RSD@SFI <</span><a href="mailto:rsdance@soft-forge.com">rsdance@soft-forge.com</a><span style="color:windowtext">>; 'Vladimir Sokolovsky' <</span><a href="mailto:vlad@dev.mellanox.co.il">vlad@dev.mellanox.co.il</a><span style="color:windowtext">><br>
<b>Cc:</b> </span><a href="mailto:bart.vanassche@gmail.com">bart.vanassche@gmail.com</a><span style="color:windowtext">;
</span><a href="mailto:ewg@lists.openfabrics.org">ewg@lists.openfabrics.org</a><span style="color:windowtext">;
</span><a href="mailto:mschlining@ddn.com">mschlining@ddn.com</a><span style="color:windowtext"><br>
<b>Subject:</b> Re: [ewg] [GIT PULL compat-rdma] qib for OFED 4.8<o:p></o:p></span></p>
</div>
</div>
<p class="MsoNormal"><o:p> </o:p></p>
<p class="MsoNormal"><span style="color:#1F497D">What is the visibility and impact of the bug? Is this something that would be seen in normal use cases?<o:p></o:p></span></p>
<p class="MsoNormal"><span style="color:#1F497D">Is this actively being worked? What is the ETA for a fix?
<o:p></o:p></span></p>
<p class="MsoNormal"><span style="color:#1F497D"><o:p> </o:p></span></p>
<div style="border:none;border-left:solid blue 1.5pt;padding:0in 0in 0in 4.0pt">
<div>
<div style="border:none;border-top:solid #E1E1E1 1.0pt;padding:3.0pt 0in 0in 0in">
<p class="MsoNormal"><b><span style="color:windowtext">From:</span></b><span style="color:windowtext"> RSD@SFI [</span><a href="mailto:rsdance@soft-forge.com">mailto:rsdance@soft-forge.com</a><span style="color:windowtext">]
<br>
<b>Sent:</b> Thursday, June 15, 2017 8:02 PM<br>
<b>To:</b> Davis, Arlin R <</span><a href="mailto:arlin.r.davis@intel.com">arlin.r.davis@intel.com</a><span style="color:windowtext">>; 'Vladimir Sokolovsky' <</span><a href="mailto:vlad@dev.mellanox.co.il">vlad@dev.mellanox.co.il</a><span style="color:windowtext">><br>
<b>Cc:</b> </span><a href="mailto:ewg@lists.openfabrics.org">ewg@lists.openfabrics.org</a><span style="color:windowtext"><br>
<b>Subject:</b> RE: [ewg] [GIT PULL compat-rdma] qib for OFED 4.8<o:p></o:p></span></p>
</div>
</div>
<p class="MsoNormal"><o:p> </o:p></p>
<p class="MsoNormal"><span style="color:windowtext">SRP is very important to DDN and they have been a major supporter of IB and the OFA for a long time. Marty always attends the OFA Interop events and has been a big supporter of the OFA Logo program. Therefore
I vote that we wait and find a fix.<o:p></o:p></span></p>
<p class="MsoNormal"><span style="color:windowtext"><o:p> </o:p></span></p>
<div>
<div style="border:none;border-top:solid #E1E1E1 1.0pt;padding:3.0pt 0in 0in 0in">
<p class="MsoNormal"><b><span style="color:windowtext">From:</span></b><span style="color:windowtext"> ewg [</span><a href="mailto:ewg-bounces@lists.openfabrics.org">mailto:ewg-bounces@lists.openfabrics.org</a><span style="color:windowtext">]
<b>On Behalf Of </b>Davis, Arlin R<br>
<b>Sent:</b> Thursday, June 15, 2017 4:20 PM<br>
<b>To:</b> Vladimir Sokolovsky <</span><a href="mailto:vlad@dev.mellanox.co.il">vlad@dev.mellanox.co.il</a><span style="color:windowtext">><br>
<b>Cc:</b> </span><a href="mailto:ewg@lists.openfabrics.org">ewg@lists.openfabrics.org</a><span style="color:windowtext"><br>
<b>Subject:</b> Re: [ewg] [GIT PULL compat-rdma] qib for OFED 4.8<o:p></o:p></span></p>
</div>
</div>
<p class="MsoNormal"><o:p> </o:p></p>
<p class="MsoNormal"><span style="color:#1F497D">I just noticed a new critical SRP bug with RC4. Do we document as “known issues” or hold off on GA for a fix?
<o:p></o:p></span></p>
<p class="MsoNormal"><span style="color:#1F497D"><o:p> </o:p></span></p>
<p class="MsoNormal"><a href="http://bugs.openfabrics.org/bugzilla/show_bug.cgi?id=2631">http://bugs.openfabrics.org/bugzilla/show_bug.cgi?id=2631</a><span style="color:#1F497D"><o:p></o:p></span></p>
<p class="MsoNormal"><span style="color:#1F497D"><o:p> </o:p></span></p>
<p class="MsoNormal"><span style="color:#1F497D"><o:p> </o:p></span></p>
<div style="border:none;border-left:solid blue 1.5pt;padding:0in 0in 0in 4.0pt">
<div>
<div style="border:none;border-top:solid #E1E1E1 1.0pt;padding:3.0pt 0in 0in 0in">
<p class="MsoNormal"><b><span style="color:windowtext">From:</span></b><span style="color:windowtext"> ewg [</span><a href="mailto:ewg-bounces@lists.openfabrics.org">mailto:ewg-bounces@lists.openfabrics.org</a><span style="color:windowtext">]
<b>On Behalf Of </b>Davis, Arlin R<br>
<b>Sent:</b> Thursday, June 15, 2017 10:57 AM<br>
<b>To:</b> Vladimir Sokolovsky <</span><a href="mailto:vlad@dev.mellanox.co.il">vlad@dev.mellanox.co.il</a><span style="color:windowtext">>; Schmidt, William R <</span><a href="mailto:william.r.schmidt@intel.com">william.r.schmidt@intel.com</a><span style="color:windowtext">>;
Schulfer, Pawel <</span><a href="mailto:pawel.schulfer@intel.com">pawel.schulfer@intel.com</a><span style="color:windowtext">><br>
<b>Cc:</b> </span><a href="mailto:ewg@lists.openfabrics.org">ewg@lists.openfabrics.org</a><span style="color:windowtext"><br>
<b>Subject:</b> Re: [ewg] [GIT PULL compat-rdma] qib for OFED 4.8<o:p></o:p></span></p>
</div>
</div>
<p class="MsoNormal"><o:p> </o:p></p>
<p class="MsoPlainText">Last call for release note changes. Please get them to Vlad by end of the day today so we can wrap this up.<o:p></o:p></p>
<p class="MsoPlainText"><o:p> </o:p></p>
<p class="MsoPlainText">Vlad, please roll GA tomorrow with all updated release notes.
<o:p></o:p></p>
<p class="MsoPlainText"><o:p> </o:p></p>
<p class="MsoPlainText">Thanks everyone!<o:p></o:p></p>
<p class="MsoNormal"><span style="color:#1F497D"><o:p> </o:p></span></p>
<p class="MsoNormal"><span style="color:#1F497D"><o:p> </o:p></span></p>
<div style="border:none;border-left:solid blue 1.5pt;padding:0in 0in 0in 4.0pt">
<div>
<div style="border:none;border-top:solid #E1E1E1 1.0pt;padding:3.0pt 0in 0in 0in">
<p class="MsoNormal"><b><span style="color:windowtext">From:</span></b><span style="color:windowtext"> ewg [</span><a href="mailto:ewg-bounces@lists.openfabrics.org">mailto:ewg-bounces@lists.openfabrics.org</a><span style="color:windowtext">]
<b>On Behalf Of </b>Vladimir Sokolovsky<br>
<b>Sent:</b> Thursday, June 15, 2017 8:58 AM<br>
<b>To:</b> Schmidt, William R <</span><a href="mailto:william.r.schmidt@intel.com">william.r.schmidt@intel.com</a><span style="color:windowtext">>; Schulfer, Pawel <</span><a href="mailto:pawel.schulfer@intel.com">pawel.schulfer@intel.com</a><span style="color:windowtext">><br>
<b>Cc:</b> </span><a href="mailto:ewg@lists.openfabrics.org">ewg@lists.openfabrics.org</a><span style="color:windowtext"><br>
<b>Subject:</b> Re: [ewg] [GIT PULL compat-rdma] qib for OFED 4.8<o:p></o:p></span></p>
</div>
</div>
<p class="MsoNormal"><o:p> </o:p></p>
<p class="MsoNormal" style="margin-bottom:12.0pt">This fix will get to GA as it was already applied as I mentioned before and the GA was not built yet.<br>
<br>
Regards,<br>
Vladimir<span style="font-size:12.0pt"><o:p></o:p></span></p>
<div>
<p class="MsoNormal">On 06/15/2017 12:51 AM, Schmidt, William R wrote:<o:p></o:p></p>
</div>
<blockquote style="margin-top:5.0pt;margin-bottom:5.0pt">
<p class="MsoNormal"><span style="color:#1F497D">Vlad,</span><o:p></o:p></p>
<p class="MsoNormal"><span style="color:#1F497D"> </span><o:p></o:p></p>
<p class="MsoNormal"><span style="color:#1F497D">Did this fix get into the GA version of OFED 4.8?</span><o:p></o:p></p>
<p class="MsoNormal"><span style="color:#1F497D"> </span><o:p></o:p></p>
<div>
<div style="border:none;border-top:solid #E1E1E1 1.0pt;padding:3.0pt 0in 0in 0in">
<p class="MsoNormal"><b><span style="color:windowtext">From:</span></b><span style="color:windowtext"> ewg [</span><a href="mailto:ewg-bounces@lists.openfabrics.org">mailto:ewg-bounces@lists.openfabrics.org</a><span style="color:windowtext">]
<b>On Behalf Of </b>Vladimir Sokolovsky<br>
<b>Sent:</b> Thursday, June 8, 2017 12:17 PM<br>
<b>To:</b> Schulfer, Pawel </span><a href="mailto:pawel.schulfer@intel.com"><pawel.schulfer@intel.com></a><span style="color:windowtext"><br>
<b>Cc:</b> </span><a href="mailto:ewg@lists.openfabrics.org">ewg@lists.openfabrics.org</a><span style="color:windowtext"><br>
<b>Subject:</b> Re: [ewg] [GIT PULL compat-rdma] qib for OFED 4.8</span><o:p></o:p></p>
</div>
</div>
<p class="MsoNormal"> <o:p></o:p></p>
<p class="MsoNormal" style="margin-bottom:12.0pt">On 06/07/2017 11:16 PM, Schulfer, Pawel wrote:<o:p></o:p></p>
<blockquote style="margin-top:5.0pt;margin-bottom:5.0pt">
<p class="MsoNormal"><span style="color:#44546A">Hi Vlad,</span><o:p></o:p></p>
<p class="MsoNormal"><span style="color:#44546A"> </span><o:p></o:p></p>
<p class="MsoNormal"><span style="color:#44546A">Please pull from git://flatbed.openfabrics.org/~pschulfer/compat-rdma:for_vlad</span><o:p></o:p></p>
<p class="MsoNormal"><span style="color:#44546A">commit 54931166983d20cf7d12c47f081fd5e86de1cb32 – Support for Kernel 4.8 in QIB on OFED-4.8</span><o:p></o:p></p>
<p class="MsoNormal"><span style="color:#44546A"> </span><o:p></o:p></p>
<p class="MsoNormal"><span style="color:#44546A">Thanks,</span><o:p></o:p></p>
<p class="MsoNormal"><span style="color:#44546A">Pawel</span><o:p></o:p></p>
<p class="MsoNormal"> <o:p></o:p></p>
</blockquote>
<p class="MsoNormal" style="margin-bottom:12.0pt"><span style="font-size:12.0pt;font-family:"Times New Roman \,serif""><br>
Hi Pawel,<br>
Done + build: OFED-4.8-20170608-1100.tgz<br>
<br>
Regards,<br>
Vladimir</span><o:p></o:p></p>
<blockquote style="margin-top:5.0pt;margin-bottom:5.0pt">
<p class="MsoNormal"> <o:p></o:p></p>
<p>---------------------------------------------------------------------<br>
<strong><span style="font-size:9.0pt">Intel Technology Poland sp. z o.o.</span></strong><b><span style="font-size:9.0pt"><br>
</span></b><span style="font-size:9.0pt">ul. Słowackiego 173 | 80-298 Gdańsk | Sąd Rejonowy Gdańsk Północ | VII Wydział Gospodarczy Krajowego Rejestru Sądowego - KRS 101882 | NIP 957-07-52-316 | Kapitał zakładowy 200.000 PLN.</span><o:p></o:p></p>
<p><span style="font-size:8.0pt">Ta wiadomość wraz z załącznikami jest przeznaczona dla określonego adresata i może zawierać informacje poufne. W razie przypadkowego otrzymania tej wiadomości, prosimy o powiadomienie nadawcy oraz trwałe jej usunięcie; jakiekolwiek
przeglądanie lub rozpowszechnianie jest zabronione.<br>
This e-mail and any attachments may contain confidential material for the sole use of the intended recipient(s). If you are not the intended recipient, please contact the sender and delete all copies; any review or distribution by others is strictly prohibited.</span><o:p></o:p></p>
<p class="MsoNormal" style="margin-bottom:12.0pt"><o:p> </o:p></p>
<pre>_______________________________________________<o:p></o:p></pre>
<pre>ewg mailing list<o:p></o:p></pre>
<pre><a href="mailto:ewg@lists.openfabrics.org">ewg@lists.openfabrics.org</a><o:p></o:p></pre>
<pre><a href="http://lists.openfabrics.org/mailman/listinfo/ewg">http://lists.openfabrics.org/mailman/listinfo/ewg</a><o:p></o:p></pre>
</blockquote>
<p class="MsoNormal"><span style="font-size:12.0pt;font-family:"Times New Roman \,serif""> </span><o:p></o:p></p>
</blockquote>
<p class="MsoNormal"><span style="font-size:12.0pt;font-family:"Times New Roman",serif"><o:p> </o:p></span></p>
</div>
</div>
</div>
</div>
</div>
</div>
</div>
</body>
</html>