<html><head><meta http-equiv="Content-Type" content="text/html charset=windows-1252"><base href="x-msg://2999/"></head><body style="word-wrap: break-word; -webkit-nbsp-mode: space; -webkit-line-break: after-white-space; ">Hello,<div><br></div><div>I've done a test run with the updated librdmacm-1.0.21 installed on hosts with QLE7340 and QLE7342 against each other and the set of Mellanox devices included in the January Logo Event with each host acting as both server and client. The installed OFED was 3.18-rc2. All passed per the test plan (rstream -T [sabn] -S all). This is great news for the upcoming logo event given the fix has been submitted for 3.18. I plan to reset the hosts to 3.12-1 and again update the Intel hosts with librdmacm-1.0.21 and test again to verify this against the pending arbitration request. </div><div><br></div><div>Thanks,</div><div>Dave</div><div><br></div><div><br></div><div><br></div><div><br></div><div><br><div><div>On Jun 3, 2015, at 3:10 PM, "Calciano, Jess" <<a href="mailto:jess.calciano@intel.com">jess.calciano@intel.com</a>> wrote:</div><br class="Apple-interchange-newline"><blockquote type="cite"><div lang="EN-US" link="blue" vlink="purple" style="font-family: Helvetica; font-size: medium; font-style: normal; font-variant: normal; font-weight: normal; letter-spacing: normal; line-height: normal; orphans: 2; text-align: -webkit-auto; text-indent: 0px; text-transform: none; white-space: normal; widows: 2; word-spacing: 0px; -webkit-text-size-adjust: auto; -webkit-text-stroke-width: 0px; "><div class="WordSection1" style="page: WordSection1; "><div style="margin: 0in 0in 0.0001pt; font-size: 12pt; font-family: 'Times New Roman', serif; "><span style="font-size: 11pt; font-family: Calibri, sans-serif; color: rgb(31, 73, 125); ">Hello,<o:p></o:p></span></div><div style="margin: 0in 0in 0.0001pt; font-size: 12pt; font-family: 'Times New Roman', serif; "><span style="font-size: 11pt; font-family: Calibri, sans-serif; color: rgb(31, 73, 125); "> </span></div><div style="margin: 0in 0in 0.0001pt; font-size: 12pt; font-family: 'Times New Roman', serif; "><span style="font-size: 11pt; font-family: Calibri, sans-serif; color: rgb(31, 73, 125); ">Since the original arbitration request was submitted, there’s been some further discussion about the RSockets failure. With the fix for librdmacm described in the original request, rstream ran successfully for most message sizes, but still hung with -S 1024.<o:p></o:p></span></div><div style="margin: 0in 0in 0.0001pt; font-size: 12pt; font-family: 'Times New Roman', serif; "><span style="font-size: 11pt; font-family: Calibri, sans-serif; color: rgb(31, 73, 125); "> </span></div><div style="margin: 0in 0in 0.0001pt; font-size: 12pt; font-family: 'Times New Roman', serif; "><span style="font-size: 11pt; font-family: Calibri, sans-serif; color: rgb(31, 73, 125); ">Additional investigation traced the new problem to an incompatibility between the qib driver and the ibv_create_qp() function. A workaround (described below) is available for the current OFED version and a permanent fix to librdmacm will be included in the next OFED 3.18 release.<o:p></o:p></span></div><div style="margin: 0in 0in 0.0001pt; font-size: 12pt; font-family: 'Times New Roman', serif; "><span style="font-size: 11pt; font-family: Calibri, sans-serif; color: rgb(31, 73, 125); "> </span></div><div style="margin: 0in 0in 0.0001pt; font-size: 12pt; font-family: 'Times New Roman', serif; "><span style="font-size: 11pt; font-family: Calibri, sans-serif; color: rgb(31, 73, 125); ">Details:<o:p></o:p></span></div><div style="margin: 0in 0in 0.0001pt; font-size: 12pt; font-family: 'Times New Roman', serif; "><span style="font-size: 11pt; font-family: Calibri, sans-serif; color: rgb(31, 73, 125); "> </span></div><div style="margin: 0in 0in 0.0001pt 0.5in; font-size: 12pt; font-family: 'Times New Roman', serif; ">The ultimate issue is still related to the qib driver being non-compliant with the ibv_create_qp() definition:<br><br><span class="apple-tab-span"> <span class="Apple-converted-space"> </span></span>The function ibv_create_qp() will update the qp_init_attr->cap<br><span class="apple-tab-span"> <span class="Apple-converted-space"> </span></span>struct with the actual QP values of the QP that was created;<br><span class="apple-tab-span"> <span class="Apple-converted-space"> </span></span>*** the values will be greater than or equal to the values requested. ***<br><span class="apple-tab-span"> <span class="Apple-converted-space"> </span></span><br>Specifically, the qib driver will return an inline size that is smaller than that requested. Rsockets has code to trap for this, but the rsockets code looks like this:<br><br><span class="apple-tab-span"> <span class="Apple-converted-space"> </span></span>inline_size = SOME_DEFAULT_LIKE_64<br><span class="apple-tab-span"> <span class="Apple-converted-space"> </span></span>rs_init_bufs(...);<br><span class="apple-tab-span"> <span class="Apple-converted-space"> </span></span>...<br><span class="apple-tab-span"> <span class="Apple-converted-space"> </span></span>rs_create_qp(...);<br><span class="apple-tab-span"> <span class="Apple-converted-space"> </span></span>inline_size = qp_cap->max_inline_size;<br><br>The issue is that rs_init_bufs(), which allocates the buffers and registers the memory, uses the default inline size. The net result is that rsockets ends up referencing memory that is outside of the registered memory region when sending credit updates. The lost credit update is causing the hang that you see.<br><br>A quick check shows that I can move the rs_init_bufs() call after the qp has been created and have the test work. You should also be able to override the inline_size by writing the value 0 into a config file. This will set the inline_size to 0 as the default. To do this, you need to write a 0 into /etc/rdma/rsocket/inline_default. (The actual path will depend on your configuration, so it could be under /usr/etc/rdma/... for example.) Updating the config file should work with the current version.<br><br>I will provide an update to the librdmacm to handle this. That update will find its way into the 3.18 release.<br><br><span style="font-size: 11pt; font-family: Calibri, sans-serif; color: rgb(31, 73, 125); "><o:p></o:p></span></div><div style="margin: 0in 0in 0.0001pt; font-size: 12pt; font-family: 'Times New Roman', serif; "><span style="font-size: 11pt; font-family: Calibri, sans-serif; color: rgb(31, 73, 125); ">Thanks,<o:p></o:p></span></div><div style="margin: 0in 0in 0.0001pt; font-size: 12pt; font-family: 'Times New Roman', serif; "><span style="font-size: 11pt; font-family: Calibri, sans-serif; color: rgb(31, 73, 125); ">Jess Calciano<o:p></o:p></span></div><div style="margin: 0in 0in 0.0001pt; font-size: 12pt; font-family: 'Times New Roman', serif; "><span style="font-size: 11pt; font-family: Calibri, sans-serif; color: rgb(31, 73, 125); "> </span></div><div style="margin: 0in 0in 0.0001pt; font-size: 12pt; font-family: 'Times New Roman', serif; "><span style="font-size: 11pt; font-family: Calibri, sans-serif; color: rgb(31, 73, 125); "> </span></div><div style="margin: 0in 0in 0.0001pt; font-size: 12pt; font-family: 'Times New Roman', serif; "><a name="_MailEndCompose"><span style="font-size: 11pt; font-family: Calibri, sans-serif; color: rgb(31, 73, 125); "> </span></a></div><div><div style="border-style: solid none none; border-top-width: 1pt; border-top-color: rgb(225, 225, 225); padding: 3pt 0in 0in; "><div style="margin: 0in 0in 0.0001pt; font-size: 12pt; font-family: 'Times New Roman', serif; "><b><span style="font-size: 11pt; font-family: Calibri, sans-serif; ">From:</span></b><span style="font-size: 11pt; font-family: Calibri, sans-serif; "><span class="Apple-converted-space"> </span>Calciano, Jess<span class="Apple-converted-space"> </span><br><b>Sent:</b><span class="Apple-converted-space"> </span>Wednesday, April 08, 2015 2:39 PM<br><b>To:</b><span class="Apple-converted-space"> </span><a href="mailto:iwg-arbitration-committee@openfabrics.org" style="color: purple; text-decoration: underline; ">iwg-arbitration-committee@openfabrics.org</a><br><b>Cc:</b><span class="Apple-converted-space"> </span>OFA Lab Mailing List; Dave Wyman; Rupert Dance <<a href="mailto:rsdance@soft-forge.com" style="color: purple; text-decoration: underline; ">rsdance@soft-forge.com</a>> (<a href="mailto:rsdance@soft-forge.com" style="color: purple; text-decoration: underline; ">rsdance@soft-forge.com</a>); Cole, Cliff; Mascarenhas, Edward; Sharma, Karun; Thete, Swapna; Hefty, Sean; Yan, Philip W; Flores, Jose F<br><b>Subject:</b><span class="Apple-converted-space"> </span>Arbitration request for Intel QLE7340 & QLE7342 HCAs (Jan 2015 OFA Interop Logo Event)<o:p></o:p></span></div></div></div><div style="margin: 0in 0in 0.0001pt; font-size: 12pt; font-family: 'Times New Roman', serif; "><o:p> </o:p></div><div style="margin: 0in 0in 0.0001pt; font-size: 12pt; font-family: 'Times New Roman', serif; "><span style="font-size: 11pt; font-family: Calibri, sans-serif; color: rgb(31, 73, 125); ">Hello,<o:p></o:p></span></div><div style="margin: 0in 0in 0.0001pt; font-size: 12pt; font-family: 'Times New Roman', serif; "><span style="font-size: 11pt; font-family: Calibri, sans-serif; color: rgb(31, 73, 125); "> </span></div><div style="margin: 0in 0in 0.0001pt; font-size: 12pt; font-family: 'Times New Roman', serif; "><span style="font-size: 11pt; font-family: Calibri, sans-serif; color: rgb(31, 73, 125); ">Intel would like to file an arbitration request for the January 2015 OFA Interop Logo Event results for the Intel QLE7340 and QLE7342 HCAs.<o:p></o:p></span></div><div style="margin: 0in 0in 0.0001pt; font-size: 12pt; font-family: 'Times New Roman', serif; "><span style="font-size: 11pt; font-family: Calibri, sans-serif; color: rgb(31, 73, 125); "> </span></div><div style="margin: 0in 0in 0.0001pt; font-size: 12pt; font-family: 'Times New Roman', serif; "><span style="font-size: 11pt; font-family: Calibri, sans-serif; color: rgb(31, 73, 125); ">The provided report (attached for reference) shows two failing tests:<o:p></o:p></span></div><div style="margin: 0in 0in 0.0001pt 0.5in; font-size: 12pt; font-family: 'Times New Roman', serif; text-indent: -0.25in; "><span style="font-size: 11pt; font-family: Calibri, sans-serif; color: rgb(31, 73, 125); "><span>1)<span style="font-style: normal; font-variant: normal; font-weight: normal; font-size: 7pt; line-height: normal; font-family: 'Times New Roman'; "> <span class="Apple-converted-space"> </span></span></span></span><span style="font-size: 11pt; font-family: Calibri, sans-serif; color: rgb(31, 73, 125); ">TI NFS over RDMA<o:p></o:p></span></div><div style="margin: 0in 0in 0.0001pt 0.5in; font-size: 12pt; font-family: 'Times New Roman', serif; text-indent: -0.25in; "><span style="font-size: 11pt; font-family: Calibri, sans-serif; color: rgb(31, 73, 125); "><span>2)<span style="font-style: normal; font-variant: normal; font-weight: normal; font-size: 7pt; line-height: normal; font-family: 'Times New Roman'; "> <span class="Apple-converted-space"> </span></span></span></span><span style="font-size: 11pt; font-family: Calibri, sans-serif; color: rgb(31, 73, 125); ">TI RSockets<o:p></o:p></span></div><div style="margin: 0in 0in 0.0001pt; font-size: 12pt; font-family: 'Times New Roman', serif; "><span style="font-size: 11pt; font-family: Calibri, sans-serif; color: rgb(31, 73, 125); "> </span></div><div style="margin: 0in 0in 0.0001pt; font-size: 12pt; font-family: 'Times New Roman', serif; "><span style="font-size: 11pt; font-family: Calibri, sans-serif; color: rgb(31, 73, 125); ">The Intel team has investigated these results and determined that the failures are due to bugs in non-Intel components.<o:p></o:p></span></div><div style="margin: 0in 0in 0.0001pt; font-size: 12pt; font-family: 'Times New Roman', serif; "><span style="font-size: 11pt; font-family: Calibri, sans-serif; color: rgb(31, 73, 125); "> </span></div><div style="margin: 0in 0in 0.0001pt; font-size: 12pt; font-family: 'Times New Roman', serif; "><span style="font-size: 11pt; font-family: Calibri, sans-serif; color: rgb(31, 73, 125); ">NFSoRDMA:<o:p></o:p></span></div><div style="margin: 0in 0in 0.0001pt 0.25in; font-size: 12pt; font-family: 'Times New Roman', serif; "><span style="font-size: 11pt; font-family: Calibri, sans-serif; color: rgb(31, 73, 125); ">The failure is due to a known Connectathon issue, documented here:<o:p></o:p></span></div><div style="margin: 0in 0in 0.0001pt 0.25in; font-size: 12pt; font-family: 'Times New Roman', serif; "><a href="http://www.spinics.net/lists/linux-nfs/msg16460.html" style="color: purple; text-decoration: underline; "><span style="font-size: 11pt; font-family: Calibri, sans-serif; ">http://www.spinics.net/lists/linux-nfs/msg16460.html</span></a><span style="font-size: 11pt; font-family: Calibri, sans-serif; "><o:p></o:p></span></div><div style="margin: 0in 0in 0.0001pt; font-size: 12pt; font-family: 'Times New Roman', serif; "><span style="font-size: 11pt; font-family: Calibri, sans-serif; color: rgb(31, 73, 125); "> </span></div><div style="margin: 0in 0in 0.0001pt; font-size: 12pt; font-family: 'Times New Roman', serif; "><span style="font-size: 11pt; font-family: Calibri, sans-serif; color: rgb(31, 73, 125); ">RSockets:<o:p></o:p></span></div><p class="MsoNormal" style="margin: 0in 0in 12pt 0.25in; font-size: 12pt; font-family: 'Times New Roman', serif; "><span style="font-size: 11pt; font-family: Calibri, sans-serif; color: rgb(0, 32, 96); ">The issue is that ibv_modify_qp() is failing. The problem is that an incorrect bit is set in the qp_attr_mask, which is returned from the kernel. With Intel, bit 21 of the qp_attr_mask is set. This is not the case for a Mellanox HCA.<br><br>Bit 21 is not defined for userspace. However, it was defined in the kernel as IB_QP_SMAC.<br><br>If the librdmacm is modified to mask out this bit, the call succeeds and rstream runs successfully.</span><span style="font-size: 11pt; font-family: Calibri, sans-serif; color: rgb(31, 73, 125); "><o:p></o:p></span></p><div style="margin: 0in 0in 0.0001pt; font-size: 12pt; font-family: 'Times New Roman', serif; "><span style="font-size: 11pt; font-family: Calibri, sans-serif; color: rgb(31, 73, 125); "> </span></div><div style="margin: 0in 0in 0.0001pt; font-size: 12pt; font-family: 'Times New Roman', serif; "><span style="font-size: 11pt; font-family: Calibri, sans-serif; color: rgb(31, 73, 125); ">Please let me know if the arbitration committee needs any additional information on the analysis.<o:p></o:p></span></div><div style="margin: 0in 0in 0.0001pt; font-size: 12pt; font-family: 'Times New Roman', serif; "><span style="font-size: 11pt; font-family: Calibri, sans-serif; color: rgb(31, 73, 125); "> </span></div><div style="margin: 0in 0in 0.0001pt; font-size: 12pt; font-family: 'Times New Roman', serif; "><span style="font-size: 11pt; font-family: Calibri, sans-serif; color: rgb(31, 73, 125); "> </span></div><div style="margin: 0in 0in 0.0001pt; font-size: 12pt; font-family: 'Times New Roman', serif; "><span style="font-size: 11pt; font-family: Calibri, sans-serif; color: rgb(31, 73, 125); ">Thanks,<o:p></o:p></span></div><div style="margin: 0in 0in 0.0001pt; font-size: 12pt; font-family: 'Times New Roman', serif; "><span style="font-size: 11pt; font-family: Calibri, sans-serif; color: rgb(31, 73, 125); ">Jess Calciano<o:p></o:p></span></div></div></div></blockquote></div><br></div></body></html>