<html>
<head>
<meta content="text/html; charset=windows-1252"
http-equiv="Content-Type">
</head>
<body bgcolor="#FFFFFF" text="#000000">
<div class="moz-cite-prefix">Hi all,<br>
<br>
I've reset the cluster to ofed 3.12-1 and rerun rsockets with the
updated librdmacm-1.0.21 installed. Everything passed
heterogeneously. <br>
<br>
I'll now do the same for nfsordma with the hacked together
connectathon plan and move this arbitration request forward. <br>
<br>
Please let me know if you have any questions<br>
<br>
Thanks,<br>
Dave<br>
<br>
<br>
<br>
<br>
<br>
On 6/5/15 4:36 PM, Dave Wyman wrote:<br>
</div>
<blockquote
cite="mid:400F6B2D-1C11-45B0-A0F1-1079D8D2A3A9@iol.unh.edu"
type="cite">
<meta http-equiv="Content-Type" content="text/html;
charset=windows-1252">
<base href="x-msg://2999/">Hello,
<div><br>
</div>
<div>I've done a test run with the updated librdmacm-1.0.21
installed on hosts with QLE7340 and QLE7342 against each other
and the set of Mellanox devices included in the January Logo
Event with each host acting as both server and client. The
installed OFED was 3.18-rc2. All passed per the test plan
(rstream -T [sabn] -S all). This is great news for the upcoming
logo event given the fix has been submitted for 3.18. I plan to
reset the hosts to 3.12-1 and again update the Intel hosts with
librdmacm-1.0.21 and test again to verify this against the
pending arbitration request. </div>
<div><br>
</div>
<div>Thanks,</div>
<div>Dave</div>
<div><br>
</div>
<div><br>
</div>
<div><br>
</div>
<div><br>
</div>
<div><br>
<div>
<div>On Jun 3, 2015, at 3:10 PM, "Calciano, Jess" <<a
moz-do-not-send="true"
href="mailto:jess.calciano@intel.com">jess.calciano@intel.com</a>>
wrote:</div>
<br class="Apple-interchange-newline">
<blockquote type="cite">
<div link="blue" vlink="purple" style="font-family:
Helvetica; font-size: medium; font-style: normal;
font-variant: normal; font-weight: normal; letter-spacing:
normal; line-height: normal; orphans: 2; text-align:
-webkit-auto; text-indent: 0px; text-transform: none;
white-space: normal; widows: 2; word-spacing: 0px;
-webkit-text-size-adjust: auto; -webkit-text-stroke-width:
0px; " lang="EN-US">
<div class="WordSection1" style="page: WordSection1; ">
<div style="margin: 0in 0in 0.0001pt; font-size: 12pt;
font-family: 'Times New Roman', serif; "><span
style="font-size: 11pt; font-family: Calibri,
sans-serif; color: rgb(31, 73, 125); ">Hello,<o:p></o:p></span></div>
<div style="margin: 0in 0in 0.0001pt; font-size: 12pt;
font-family: 'Times New Roman', serif; "><span
style="font-size: 11pt; font-family: Calibri,
sans-serif; color: rgb(31, 73, 125); "> </span></div>
<div style="margin: 0in 0in 0.0001pt; font-size: 12pt;
font-family: 'Times New Roman', serif; "><span
style="font-size: 11pt; font-family: Calibri,
sans-serif; color: rgb(31, 73, 125); ">Since the
original arbitration request was submitted, there’s
been some further discussion about the RSockets
failure. With the fix for librdmacm described in the
original request, rstream ran successfully for most
message sizes, but still hung with -S 1024.<o:p></o:p></span></div>
<div style="margin: 0in 0in 0.0001pt; font-size: 12pt;
font-family: 'Times New Roman', serif; "><span
style="font-size: 11pt; font-family: Calibri,
sans-serif; color: rgb(31, 73, 125); "> </span></div>
<div style="margin: 0in 0in 0.0001pt; font-size: 12pt;
font-family: 'Times New Roman', serif; "><span
style="font-size: 11pt; font-family: Calibri,
sans-serif; color: rgb(31, 73, 125); ">Additional
investigation traced the new problem to an
incompatibility between the qib driver and the
ibv_create_qp() function. A workaround (described
below) is available for the current OFED version and
a permanent fix to librdmacm will be included in the
next OFED 3.18 release.<o:p></o:p></span></div>
<div style="margin: 0in 0in 0.0001pt; font-size: 12pt;
font-family: 'Times New Roman', serif; "><span
style="font-size: 11pt; font-family: Calibri,
sans-serif; color: rgb(31, 73, 125); "> </span></div>
<div style="margin: 0in 0in 0.0001pt; font-size: 12pt;
font-family: 'Times New Roman', serif; "><span
style="font-size: 11pt; font-family: Calibri,
sans-serif; color: rgb(31, 73, 125); ">Details:<o:p></o:p></span></div>
<div style="margin: 0in 0in 0.0001pt; font-size: 12pt;
font-family: 'Times New Roman', serif; "><span
style="font-size: 11pt; font-family: Calibri,
sans-serif; color: rgb(31, 73, 125); "> </span></div>
<div style="margin: 0in 0in 0.0001pt 0.5in; font-size:
12pt; font-family: 'Times New Roman', serif; ">The
ultimate issue is still related to the qib driver
being non-compliant with the ibv_create_qp()
definition:<br>
<br>
<span class="apple-tab-span"> <span
class="Apple-converted-space"> </span></span>The
function ibv_create_qp() will update the
qp_init_attr->cap<br>
<span class="apple-tab-span"> <span
class="Apple-converted-space"> </span></span>struct
with the actual QP values of the QP that was created;<br>
<span class="apple-tab-span"> <span
class="Apple-converted-space"> </span></span>***
the values will be greater than or equal to the values
requested. ***<br>
<span class="apple-tab-span"> <span
class="Apple-converted-space"> </span></span><br>
Specifically, the qib driver will return an inline
size that is smaller than that requested. Rsockets
has code to trap for this, but the rsockets code looks
like this:<br>
<br>
<span class="apple-tab-span"> <span
class="Apple-converted-space"> </span></span>inline_size
= SOME_DEFAULT_LIKE_64<br>
<span class="apple-tab-span"> <span
class="Apple-converted-space"> </span></span>rs_init_bufs(...);<br>
<span class="apple-tab-span"> <span
class="Apple-converted-space"> </span></span>...<br>
<span class="apple-tab-span"> <span
class="Apple-converted-space"> </span></span>rs_create_qp(...);<br>
<span class="apple-tab-span"> <span
class="Apple-converted-space"> </span></span>inline_size
= qp_cap->max_inline_size;<br>
<br>
The issue is that rs_init_bufs(), which allocates the
buffers and registers the memory, uses the default
inline size. The net result is that rsockets ends up
referencing memory that is outside of the registered
memory region when sending credit updates. The lost
credit update is causing the hang that you see.<br>
<br>
A quick check shows that I can move the rs_init_bufs()
call after the qp has been created and have the test
work. You should also be able to override the
inline_size by writing the value 0 into a config file.
This will set the inline_size to 0 as the default.
To do this, you need to write a 0 into
/etc/rdma/rsocket/inline_default. (The actual path
will depend on your configuration, so it could be
under /usr/etc/rdma/... for example.) Updating the
config file should work with the current version.<br>
<br>
I will provide an update to the librdmacm to handle
this. That update will find its way into the 3.18
release.<br>
<br>
<span style="font-size: 11pt; font-family: Calibri,
sans-serif; color: rgb(31, 73, 125); "><o:p></o:p></span></div>
<div style="margin: 0in 0in 0.0001pt; font-size: 12pt;
font-family: 'Times New Roman', serif; "><span
style="font-size: 11pt; font-family: Calibri,
sans-serif; color: rgb(31, 73, 125); ">Thanks,<o:p></o:p></span></div>
<div style="margin: 0in 0in 0.0001pt; font-size: 12pt;
font-family: 'Times New Roman', serif; "><span
style="font-size: 11pt; font-family: Calibri,
sans-serif; color: rgb(31, 73, 125); ">Jess Calciano<o:p></o:p></span></div>
<div style="margin: 0in 0in 0.0001pt; font-size: 12pt;
font-family: 'Times New Roman', serif; "><span
style="font-size: 11pt; font-family: Calibri,
sans-serif; color: rgb(31, 73, 125); "> </span></div>
<div style="margin: 0in 0in 0.0001pt; font-size: 12pt;
font-family: 'Times New Roman', serif; "><span
style="font-size: 11pt; font-family: Calibri,
sans-serif; color: rgb(31, 73, 125); "> </span></div>
<div style="margin: 0in 0in 0.0001pt; font-size: 12pt;
font-family: 'Times New Roman', serif; "><a
moz-do-not-send="true" name="_MailEndCompose"><span
style="font-size: 11pt; font-family: Calibri,
sans-serif; color: rgb(31, 73, 125); "> </span></a></div>
<div>
<div style="border-style: solid none none;
border-top-width: 1pt; border-top-color: rgb(225,
225, 225); padding: 3pt 0in 0in; ">
<div style="margin: 0in 0in 0.0001pt; font-size:
12pt; font-family: 'Times New Roman', serif; "><b><span
style="font-size: 11pt; font-family: Calibri,
sans-serif; ">From:</span></b><span
style="font-size: 11pt; font-family: Calibri,
sans-serif; "><span
class="Apple-converted-space"> </span>Calciano,
Jess<span class="Apple-converted-space"> </span><br>
<b>Sent:</b><span class="Apple-converted-space"> </span>Wednesday,
April 08, 2015 2:39 PM<br>
<b>To:</b><span class="Apple-converted-space"> </span><a
moz-do-not-send="true"
href="mailto:iwg-arbitration-committee@openfabrics.org"
style="color: purple; text-decoration:
underline; ">iwg-arbitration-committee@openfabrics.org</a><br>
<b>Cc:</b><span class="Apple-converted-space"> </span>OFA
Lab Mailing List; Dave Wyman; Rupert Dance <<a
moz-do-not-send="true"
href="mailto:rsdance@soft-forge.com"
style="color: purple; text-decoration:
underline; ">rsdance@soft-forge.com</a>> (<a
moz-do-not-send="true"
href="mailto:rsdance@soft-forge.com"
style="color: purple; text-decoration:
underline; ">rsdance@soft-forge.com</a>);
Cole, Cliff; Mascarenhas, Edward; Sharma, Karun;
Thete, Swapna; Hefty, Sean; Yan, Philip W;
Flores, Jose F<br>
<b>Subject:</b><span
class="Apple-converted-space"> </span>Arbitration
request for Intel QLE7340 & QLE7342 HCAs
(Jan 2015 OFA Interop Logo Event)<o:p></o:p></span></div>
</div>
</div>
<div style="margin: 0in 0in 0.0001pt; font-size: 12pt;
font-family: 'Times New Roman', serif; "><o:p> </o:p></div>
<div style="margin: 0in 0in 0.0001pt; font-size: 12pt;
font-family: 'Times New Roman', serif; "><span
style="font-size: 11pt; font-family: Calibri,
sans-serif; color: rgb(31, 73, 125); ">Hello,<o:p></o:p></span></div>
<div style="margin: 0in 0in 0.0001pt; font-size: 12pt;
font-family: 'Times New Roman', serif; "><span
style="font-size: 11pt; font-family: Calibri,
sans-serif; color: rgb(31, 73, 125); "> </span></div>
<div style="margin: 0in 0in 0.0001pt; font-size: 12pt;
font-family: 'Times New Roman', serif; "><span
style="font-size: 11pt; font-family: Calibri,
sans-serif; color: rgb(31, 73, 125); ">Intel would
like to file an arbitration request for the January
2015 OFA Interop Logo Event results for the Intel
QLE7340 and QLE7342 HCAs.<o:p></o:p></span></div>
<div style="margin: 0in 0in 0.0001pt; font-size: 12pt;
font-family: 'Times New Roman', serif; "><span
style="font-size: 11pt; font-family: Calibri,
sans-serif; color: rgb(31, 73, 125); "> </span></div>
<div style="margin: 0in 0in 0.0001pt; font-size: 12pt;
font-family: 'Times New Roman', serif; "><span
style="font-size: 11pt; font-family: Calibri,
sans-serif; color: rgb(31, 73, 125); ">The provided
report (attached for reference) shows two failing
tests:<o:p></o:p></span></div>
<div style="margin: 0in 0in 0.0001pt 0.5in; font-size:
12pt; font-family: 'Times New Roman', serif;
text-indent: -0.25in; "><span style="font-size: 11pt;
font-family: Calibri, sans-serif; color: rgb(31, 73,
125); "><span>1)<span style="font-style: normal;
font-variant: normal; font-weight: normal;
font-size: 7pt; line-height: normal;
font-family: 'Times New Roman'; "> <span
class="Apple-converted-space"> </span></span></span></span><span
style="font-size: 11pt; font-family: Calibri,
sans-serif; color: rgb(31, 73, 125); ">TI NFS over
RDMA<o:p></o:p></span></div>
<div style="margin: 0in 0in 0.0001pt 0.5in; font-size:
12pt; font-family: 'Times New Roman', serif;
text-indent: -0.25in; "><span style="font-size: 11pt;
font-family: Calibri, sans-serif; color: rgb(31, 73,
125); "><span>2)<span style="font-style: normal;
font-variant: normal; font-weight: normal;
font-size: 7pt; line-height: normal;
font-family: 'Times New Roman'; "> <span
class="Apple-converted-space"> </span></span></span></span><span
style="font-size: 11pt; font-family: Calibri,
sans-serif; color: rgb(31, 73, 125); ">TI RSockets<o:p></o:p></span></div>
<div style="margin: 0in 0in 0.0001pt; font-size: 12pt;
font-family: 'Times New Roman', serif; "><span
style="font-size: 11pt; font-family: Calibri,
sans-serif; color: rgb(31, 73, 125); "> </span></div>
<div style="margin: 0in 0in 0.0001pt; font-size: 12pt;
font-family: 'Times New Roman', serif; "><span
style="font-size: 11pt; font-family: Calibri,
sans-serif; color: rgb(31, 73, 125); ">The Intel
team has investigated these results and determined
that the failures are due to bugs in non-Intel
components.<o:p></o:p></span></div>
<div style="margin: 0in 0in 0.0001pt; font-size: 12pt;
font-family: 'Times New Roman', serif; "><span
style="font-size: 11pt; font-family: Calibri,
sans-serif; color: rgb(31, 73, 125); "> </span></div>
<div style="margin: 0in 0in 0.0001pt; font-size: 12pt;
font-family: 'Times New Roman', serif; "><span
style="font-size: 11pt; font-family: Calibri,
sans-serif; color: rgb(31, 73, 125); ">NFSoRDMA:<o:p></o:p></span></div>
<div style="margin: 0in 0in 0.0001pt 0.25in; font-size:
12pt; font-family: 'Times New Roman', serif; "><span
style="font-size: 11pt; font-family: Calibri,
sans-serif; color: rgb(31, 73, 125); ">The failure
is due to a known Connectathon issue, documented
here:<o:p></o:p></span></div>
<div style="margin: 0in 0in 0.0001pt 0.25in; font-size:
12pt; font-family: 'Times New Roman', serif; "><a
moz-do-not-send="true"
href="http://www.spinics.net/lists/linux-nfs/msg16460.html"
style="color: purple; text-decoration: underline; "><span
style="font-size: 11pt; font-family: Calibri,
sans-serif; ">http://www.spinics.net/lists/linux-nfs/msg16460.html</span></a><span
style="font-size: 11pt; font-family: Calibri,
sans-serif; "><o:p></o:p></span></div>
<div style="margin: 0in 0in 0.0001pt; font-size: 12pt;
font-family: 'Times New Roman', serif; "><span
style="font-size: 11pt; font-family: Calibri,
sans-serif; color: rgb(31, 73, 125); "> </span></div>
<div style="margin: 0in 0in 0.0001pt; font-size: 12pt;
font-family: 'Times New Roman', serif; "><span
style="font-size: 11pt; font-family: Calibri,
sans-serif; color: rgb(31, 73, 125); ">RSockets:<o:p></o:p></span></div>
<p class="MsoNormal" style="margin: 0in 0in 12pt 0.25in;
font-size: 12pt; font-family: 'Times New Roman',
serif; "><span style="font-size: 11pt; font-family:
Calibri, sans-serif; color: rgb(0, 32, 96); ">The
issue is that ibv_modify_qp() is failing. The
problem is that an incorrect bit is set in the
qp_attr_mask, which is returned from the kernel.
With Intel, bit 21 of the qp_attr_mask is set.
This is not the case for a Mellanox HCA.<br>
<br>
Bit 21 is not defined for userspace. However, it
was defined in the kernel as IB_QP_SMAC.<br>
<br>
If the librdmacm is modified to mask out this bit,
the call succeeds and rstream runs successfully.</span><span
style="font-size: 11pt; font-family: Calibri,
sans-serif; color: rgb(31, 73, 125); "><o:p></o:p></span></p>
<div style="margin: 0in 0in 0.0001pt; font-size: 12pt;
font-family: 'Times New Roman', serif; "><span
style="font-size: 11pt; font-family: Calibri,
sans-serif; color: rgb(31, 73, 125); "> </span></div>
<div style="margin: 0in 0in 0.0001pt; font-size: 12pt;
font-family: 'Times New Roman', serif; "><span
style="font-size: 11pt; font-family: Calibri,
sans-serif; color: rgb(31, 73, 125); ">Please let me
know if the arbitration committee needs any
additional information on the analysis.<o:p></o:p></span></div>
<div style="margin: 0in 0in 0.0001pt; font-size: 12pt;
font-family: 'Times New Roman', serif; "><span
style="font-size: 11pt; font-family: Calibri,
sans-serif; color: rgb(31, 73, 125); "> </span></div>
<div style="margin: 0in 0in 0.0001pt; font-size: 12pt;
font-family: 'Times New Roman', serif; "><span
style="font-size: 11pt; font-family: Calibri,
sans-serif; color: rgb(31, 73, 125); "> </span></div>
<div style="margin: 0in 0in 0.0001pt; font-size: 12pt;
font-family: 'Times New Roman', serif; "><span
style="font-size: 11pt; font-family: Calibri,
sans-serif; color: rgb(31, 73, 125); ">Thanks,<o:p></o:p></span></div>
<div style="margin: 0in 0in 0.0001pt; font-size: 12pt;
font-family: 'Times New Roman', serif; "><span
style="font-size: 11pt; font-family: Calibri,
sans-serif; color: rgb(31, 73, 125); ">Jess Calciano<o:p></o:p></span></div>
</div>
</div>
</blockquote>
</div>
<br>
</div>
</blockquote>
<br>
</body>
</html>