<html>
<head>
<meta content="text/html; charset=utf-8" http-equiv="Content-Type">
</head>
<body bgcolor="#FFFFFF" text="#000000">
Great, thanks Jess<br>
<br>
-Dave<br>
<br>
<div class="moz-cite-prefix">On 7/22/15 4:23 PM, Calciano, Jess
wrote:<br>
</div>
<blockquote
cite="mid:E079A211AAA8AD48953B67A89DB71C8533169134@fmsmsx104.amr.corp.intel.com"
type="cite">
<meta http-equiv="Content-Type" content="text/html; charset=utf-8">
<meta name="Generator" content="Microsoft Word 15 (filtered
medium)">
<style><!--
/* Font Definitions */
@font-face
{font-family:"Cambria Math";
panose-1:2 4 5 3 5 4 6 3 2 4;}
@font-face
{font-family:Calibri;
panose-1:2 15 5 2 2 2 4 3 2 4;}
/* Style Definitions */
p.MsoNormal, li.MsoNormal, div.MsoNormal
{margin:0in;
margin-bottom:.0001pt;
font-size:12.0pt;
font-family:"Times New Roman",serif;}
a:link, span.MsoHyperlink
{mso-style-priority:99;
color:blue;
text-decoration:underline;}
a:visited, span.MsoHyperlinkFollowed
{mso-style-priority:99;
color:purple;
text-decoration:underline;}
p
{mso-style-priority:99;
mso-margin-top-alt:auto;
margin-right:0in;
mso-margin-bottom-alt:auto;
margin-left:0in;
font-size:12.0pt;
font-family:"Times New Roman",serif;}
span.EmailStyle18
{mso-style-type:personal-reply;
font-family:"Calibri",sans-serif;
color:#1F497D;}
.MsoChpDefault
{mso-style-type:export-only;
font-size:10.0pt;
font-family:"Calibri",sans-serif;}
@page WordSection1
{size:8.5in 11.0in;
margin:1.0in 1.0in 1.0in 1.0in;}
div.WordSection1
{page:WordSection1;}
--></style><!--[if gte mso 9]><xml>
<o:shapedefaults v:ext="edit" spidmax="1026" />
</xml><![endif]--><!--[if gte mso 9]><xml>
<o:shapelayout v:ext="edit">
<o:idmap v:ext="edit" data="1" />
</o:shapelayout></xml><![endif]-->
<div class="WordSection1">
<p class="MsoNormal"><span
style="font-size:11.0pt;font-family:"Calibri",sans-serif;color:#1F497D">Hello
Dave,<o:p></o:p></span></p>
<p class="MsoNormal"><span
style="font-size:11.0pt;font-family:"Calibri",sans-serif;color:#1F497D"><o:p> </o:p></span></p>
<p class="MsoNormal"><span
style="font-size:11.0pt;font-family:"Calibri",sans-serif;color:#1F497D">Both
reports look good and are approved to publish to the Logo
List.<o:p></o:p></span></p>
<p class="MsoNormal"><span
style="font-size:11.0pt;font-family:"Calibri",sans-serif;color:#1F497D"><o:p> </o:p></span></p>
<p class="MsoNormal"><span
style="font-size:11.0pt;font-family:"Calibri",sans-serif;color:#1F497D">--
Jess<o:p></o:p></span></p>
<p class="MsoNormal"><span
style="font-size:11.0pt;font-family:"Calibri",sans-serif;color:#1F497D"><o:p> </o:p></span></p>
<div
style="mso-element:para-border-div;border:none;border-bottom:solid
windowtext 1.0pt;padding:0in 0in 1.0pt 0in">
<p class="MsoNormal" style="border:none;padding:0in"><span
style="font-size:11.0pt;font-family:"Calibri",sans-serif;color:#1F497D"><o:p> </o:p></span></p>
</div>
<p class="MsoNormal"><span
style="font-size:11.0pt;font-family:"Calibri",sans-serif;color:#1F497D"><o:p> </o:p></span></p>
<p class="MsoNormal"><a moz-do-not-send="true"
name="_MailEndCompose"><span
style="font-size:11.0pt;font-family:"Calibri",sans-serif;color:#1F497D"><o:p> </o:p></span></a></p>
<p class="MsoNormal"><b><span
style="font-size:11.0pt;font-family:"Calibri",sans-serif">From:</span></b><span
style="font-size:11.0pt;font-family:"Calibri",sans-serif">
David Wyman [<a class="moz-txt-link-freetext" href="mailto:dwyman@iol.unh.edu">mailto:dwyman@iol.unh.edu</a>]
<br>
<b>Sent:</b> Wednesday, July 22, 2015 12:08 PM<br>
<b>To:</b> Calciano, Jess<br>
<b>Cc:</b> <a class="moz-txt-link-abbreviated" href="mailto:iwg-arbitration-committee@openfabrics.org">iwg-arbitration-committee@openfabrics.org</a>; OFA
Lab Mailing List; Cole, Cliff<br>
<b>Subject:</b> Re: Updated arbitration request for Intel
QLE7340 & QLE7342 HCAs (Jan 2015 OFA Interop Logo Event)<o:p></o:p></span></p>
<p class="MsoNormal"><o:p> </o:p></p>
<div>
<p class="MsoNormal">Hello Jess,<o:p></o:p></p>
<div>
<p class="MsoNormal"><o:p> </o:p></p>
</div>
<div>
<p class="MsoNormal">Your arbitration request regarding the
NFSoRDMA and RSockets issues found in the January 2015
Interop Logo Event has been approved and an updated report
for Intel Infiniband HCAs is attached. I also attached
the Intel Switch Report. Please review both and if you
are satisfied with the content respond with explicit
consent to post these reports to the OpenFabrics
Interoperability Logo List. Please contact me with any
questions or concerns.<o:p></o:p></p>
</div>
<div>
<p class="MsoNormal"><o:p> </o:p></p>
</div>
<div>
<p class="MsoNormal">Thank you,<o:p></o:p></p>
</div>
<div>
<p class="MsoNormal">Dave Wyman<o:p></o:p></p>
</div>
<div>
<p class="MsoNormal">UNH-IOL OpenFabrics Interoperability
Logo Group<o:p></o:p></p>
</div>
<div>
<p class="MsoNormal"><o:p> </o:p></p>
</div>
<div>
<p class="MsoNormal"><o:p> </o:p></p>
</div>
<div>
<p class="MsoNormal"><o:p> </o:p></p>
<div>
<p class="MsoNormal"><o:p> </o:p></p>
<div>
<p class="MsoNormal"><o:p> </o:p></p>
<div>
<p class="MsoNormal">On Wed, Jun 3, 2015 at 3:10 PM,
Calciano, Jess <<a moz-do-not-send="true"
href="mailto:jess.calciano@intel.com"
target="_blank">jess.calciano@intel.com</a>>
wrote:<o:p></o:p></p>
<blockquote style="border:none;border-left:solid
#CCCCCC 1.0pt;padding:0in 0in 0in
6.0pt;margin-left:4.8pt;margin-top:5.0pt;margin-right:0in;margin-bottom:5.0pt">
<div>
<div>
<p class="MsoNormal"
style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto"><span
style="font-size:11.0pt;font-family:"Calibri",sans-serif;color:#1F497D">Hello,</span><o:p></o:p></p>
<p class="MsoNormal"
style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto"><span
style="font-size:11.0pt;font-family:"Calibri",sans-serif;color:#1F497D"> </span><o:p></o:p></p>
<p class="MsoNormal"
style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto"><span
style="font-size:11.0pt;font-family:"Calibri",sans-serif;color:#1F497D">Since
the original arbitration request was
submitted, there’s been some further
discussion about the RSockets failure. With
the fix for librdmacm described in the
original request, rstream ran successfully
for most message sizes, but still hung with
-S 1024.</span><o:p></o:p></p>
<p class="MsoNormal"
style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto"><span
style="font-size:11.0pt;font-family:"Calibri",sans-serif;color:#1F497D"> </span><o:p></o:p></p>
<p class="MsoNormal"
style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto"><span
style="font-size:11.0pt;font-family:"Calibri",sans-serif;color:#1F497D">Additional
investigation traced the new problem to an
incompatibility between the qib driver and
the ibv_create_qp() function. A workaround
(described below) is available for the
current OFED version and a permanent fix to
librdmacm will be included in the next OFED
3.18 release.</span><o:p></o:p></p>
<p class="MsoNormal"
style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto"><span
style="font-size:11.0pt;font-family:"Calibri",sans-serif;color:#1F497D"> </span><o:p></o:p></p>
<p class="MsoNormal"
style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto"><span
style="font-size:11.0pt;font-family:"Calibri",sans-serif;color:#1F497D">Details:</span><o:p></o:p></p>
<p class="MsoNormal"
style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto"><span
style="font-size:11.0pt;font-family:"Calibri",sans-serif;color:#1F497D"> </span><o:p></o:p></p>
<p class="MsoNormal"
style="mso-margin-top-alt:auto;margin-bottom:12.0pt;margin-left:.5in">
The ultimate issue is still related to the qib
driver being non-compliant with the
ibv_create_qp() definition:<br>
<br>
The function ibv_create_qp() will
update the qp_init_attr->cap<br>
struct with the actual QP values
of the QP that was created;<br>
*** the values will be greater
than or equal to the values requested. ***<br>
<br>
Specifically, the qib driver will return an
inline size that is smaller than that
requested. Rsockets has code to trap for
this, but the rsockets code looks like this:<br>
<br>
inline_size = SOME_DEFAULT_LIKE_64<br>
rs_init_bufs(...);<br>
...<br>
rs_create_qp(...);<br>
inline_size =
qp_cap->max_inline_size;<br>
<br>
The issue is that rs_init_bufs(), which
allocates the buffers and registers the
memory, uses the default inline size. The net
result is that rsockets ends up referencing
memory that is outside of the registered
memory region when sending credit updates.
The lost credit update is causing the hang
that you see.<br>
<br>
A quick check shows that I can move the
rs_init_bufs() call after the qp has been
created and have the test work. You should
also be able to override the inline_size by
writing the value 0 into a config file. This
will set the inline_size to 0 as the default.
To do this, you need to write a 0 into
/etc/rdma/rsocket/inline_default. (The actual
path will depend on your configuration, so it
could be under /usr/etc/rdma/... for example.)
Updating the config file should work with the
current version.<br>
<br>
I will provide an update to the librdmacm to
handle this. That update will find its way
into the 3.18 release.<o:p></o:p></p>
<p class="MsoNormal"
style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto"><span
style="font-size:11.0pt;font-family:"Calibri",sans-serif;color:#1F497D">Thanks,</span><o:p></o:p></p>
<p class="MsoNormal"
style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto"><span
style="font-size:11.0pt;font-family:"Calibri",sans-serif;color:#1F497D">Jess
Calciano</span><o:p></o:p></p>
<p class="MsoNormal"
style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto"><span
style="font-size:11.0pt;font-family:"Calibri",sans-serif;color:#1F497D"> </span><o:p></o:p></p>
<p class="MsoNormal"
style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto"><span
style="font-size:11.0pt;font-family:"Calibri",sans-serif;color:#1F497D"> </span><o:p></o:p></p>
<p class="MsoNormal"
style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto"><a
moz-do-not-send="true"
name="14dbad57595896c8__MailEndCompose"><span
style="font-size:11.0pt;font-family:"Calibri",sans-serif;color:#1F497D"> </span></a><o:p></o:p></p>
<div>
<div style="border:none;border-top:solid
#E1E1E1 1.0pt;padding:3.0pt 0in 0in 0in">
<p class="MsoNormal"
style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto"><b><span
style="font-size:11.0pt;font-family:"Calibri",sans-serif">From:</span></b><span
style="font-size:11.0pt;font-family:"Calibri",sans-serif">
Calciano, Jess
<br>
<b>Sent:</b> Wednesday, April 08, 2015
2:39 PM<br>
<b>To:</b> </span><a
moz-do-not-send="true"
href="mailto:iwg-arbitration-committee@openfabrics.org"
target="_blank"><span
style="font-size:11.0pt;font-family:"Calibri",sans-serif">iwg-arbitration-committee@openfabrics.org</span></a><span
style="font-size:11.0pt;font-family:"Calibri",sans-serif"><br>
<b>Cc:</b> OFA Lab Mailing List; Dave
Wyman; Rupert Dance <</span><a
moz-do-not-send="true"
href="mailto:rsdance@soft-forge.com"
target="_blank"><span
style="font-size:11.0pt;font-family:"Calibri",sans-serif">rsdance@soft-forge.com</span></a><span
style="font-size:11.0pt;font-family:"Calibri",sans-serif">>
(</span><a moz-do-not-send="true"
href="mailto:rsdance@soft-forge.com"
target="_blank"><span
style="font-size:11.0pt;font-family:"Calibri",sans-serif">rsdance@soft-forge.com</span></a><span
style="font-size:11.0pt;font-family:"Calibri",sans-serif">);
Cole, Cliff; Mascarenhas, Edward;
Sharma, Karun; Thete, Swapna; Hefty,
Sean; Yan, Philip W; Flores, Jose F<br>
<b>Subject:</b> Arbitration request for
Intel QLE7340 & QLE7342 HCAs (Jan
2015 OFA Interop Logo Event)</span><o:p></o:p></p>
</div>
</div>
<p class="MsoNormal"
style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto"> <o:p></o:p></p>
<p class="MsoNormal"
style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto"><span
style="font-size:11.0pt;font-family:"Calibri",sans-serif;color:#1F497D">Hello,</span><o:p></o:p></p>
<p class="MsoNormal"
style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto"><span
style="font-size:11.0pt;font-family:"Calibri",sans-serif;color:#1F497D"> </span><o:p></o:p></p>
<p class="MsoNormal"
style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto"><span
style="font-size:11.0pt;font-family:"Calibri",sans-serif;color:#1F497D">Intel
would like to file an arbitration request
for the January 2015 OFA Interop Logo Event
results for the Intel QLE7340 and QLE7342
HCAs.</span><o:p></o:p></p>
<p class="MsoNormal"
style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto"><span
style="font-size:11.0pt;font-family:"Calibri",sans-serif;color:#1F497D"> </span><o:p></o:p></p>
<p class="MsoNormal"
style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto"><span
style="font-size:11.0pt;font-family:"Calibri",sans-serif;color:#1F497D">The
provided report (attached for reference)
shows two failing tests:
</span><o:p></o:p></p>
<p><span
style="font-size:11.0pt;font-family:"Calibri",sans-serif;color:#1F497D">1)</span><span
style="font-size:7.0pt;color:#1F497D">
</span><span
style="font-size:11.0pt;font-family:"Calibri",sans-serif;color:#1F497D">TI
NFS over RDMA</span><o:p></o:p></p>
<p><span
style="font-size:11.0pt;font-family:"Calibri",sans-serif;color:#1F497D">2)</span><span
style="font-size:7.0pt;color:#1F497D">
</span><span
style="font-size:11.0pt;font-family:"Calibri",sans-serif;color:#1F497D">TI
RSockets</span><o:p></o:p></p>
<p class="MsoNormal"
style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto"><span
style="font-size:11.0pt;font-family:"Calibri",sans-serif;color:#1F497D"> </span><o:p></o:p></p>
<p class="MsoNormal"
style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto"><span
style="font-size:11.0pt;font-family:"Calibri",sans-serif;color:#1F497D">The
Intel team has investigated these results
and determined that the failures are due to
bugs in non-Intel components.</span><o:p></o:p></p>
<p class="MsoNormal"
style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto"><span
style="font-size:11.0pt;font-family:"Calibri",sans-serif;color:#1F497D"> </span><o:p></o:p></p>
<p class="MsoNormal"
style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto"><span
style="font-size:11.0pt;font-family:"Calibri",sans-serif;color:#1F497D">NFSoRDMA:</span><o:p></o:p></p>
<p class="MsoNormal"
style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto;margin-left:.25in"><span
style="font-size:11.0pt;font-family:"Calibri",sans-serif;color:#1F497D">The
failure is due to a known Connectathon
issue, documented here:</span><o:p></o:p></p>
<p class="MsoNormal"
style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto;margin-left:.25in"><a
moz-do-not-send="true"
href="http://www.spinics.net/lists/linux-nfs/msg16460.html"
target="_blank"><span
style="font-size:11.0pt;font-family:"Calibri",sans-serif">http://www.spinics.net/lists/linux-nfs/msg16460.html</span></a><o:p></o:p></p>
<p class="MsoNormal"
style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto"><span
style="font-size:11.0pt;font-family:"Calibri",sans-serif;color:#1F497D"> </span><o:p></o:p></p>
<p class="MsoNormal"
style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto"><span
style="font-size:11.0pt;font-family:"Calibri",sans-serif;color:#1F497D">RSockets:</span><o:p></o:p></p>
<p class="MsoNormal"
style="mso-margin-top-alt:auto;margin-bottom:12.0pt;margin-left:.25in">
<span
style="font-size:11.0pt;font-family:"Calibri",sans-serif;color:#002060">The
issue is that ibv_modify_qp() is failing.
The problem is that an incorrect bit is set
in the qp_attr_mask, which is returned from
the kernel. With Intel, bit 21 of the
qp_attr_mask is set. This is not the case
for a Mellanox HCA.<br>
<br>
Bit 21 is not defined for userspace.
However, it was defined in the kernel as
IB_QP_SMAC.<br>
<br>
If the librdmacm is modified to mask out
this bit, the call succeeds and rstream runs
successfully.</span><o:p></o:p></p>
<p class="MsoNormal"
style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto"><span
style="font-size:11.0pt;font-family:"Calibri",sans-serif;color:#1F497D"> </span><o:p></o:p></p>
<p class="MsoNormal"
style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto"><span
style="font-size:11.0pt;font-family:"Calibri",sans-serif;color:#1F497D">Please
let me know if the arbitration committee
needs any additional information on the
analysis.</span><o:p></o:p></p>
<p class="MsoNormal"
style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto"><span
style="font-size:11.0pt;font-family:"Calibri",sans-serif;color:#1F497D"> </span><o:p></o:p></p>
<p class="MsoNormal"
style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto"><span
style="font-size:11.0pt;font-family:"Calibri",sans-serif;color:#1F497D"> </span><o:p></o:p></p>
<p class="MsoNormal"
style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto"><span
style="font-size:11.0pt;font-family:"Calibri",sans-serif;color:#1F497D">Thanks,</span><o:p></o:p></p>
<p class="MsoNormal"
style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto"><span
style="font-size:11.0pt;font-family:"Calibri",sans-serif;color:#1F497D">Jess
Calciano</span><o:p></o:p></p>
</div>
</div>
</blockquote>
</div>
<p class="MsoNormal"><o:p> </o:p></p>
</div>
</div>
</div>
</div>
</div>
</blockquote>
<br>
</body>
</html>