<!DOCTYPE HTML PUBLIC "-//W3C//DTD HTML 3.2//EN">
<HTML>
<HEAD>
<META HTTP-EQUIV="Content-Type" CONTENT="text/html; charset=iso-8859-1">
<META NAME="Generator" CONTENT="MS Exchange Server version 6.5.7226.0">
<TITLE>changing a UC QP to support RDMA Write is not working</TITLE>
</HEAD>
<BODY>
<!-- Converted from text/rtf format -->
<P DIR=LTR><FONT SIZE=2 FACE="Arial">Hi.</FONT></P>
<P DIR=LTR><FONT SIZE=2 FACE="Arial">I'm using Mellanox HCA on a machine with AS4 with kernel 2.6.9-5.ELsmp.</FONT></P>
<P DIR=LTR><FONT SIZE=2 FACE="Arial">I'm working with gen2 driver, svn revision 4032.</FONT></P>
<P DIR=LTR><FONT SIZE=2 FACE="Arial">i wrote a test (2 sides) that have the following flow:</FONT></P>
<P DIR=LTR><FONT SIZE=2 FACE="Arial">common:</FONT></P>
<P DIR=LTR> <FONT SIZE=2 FACE="Arial">create a UC QP</FONT></P>
<P DIR=LTR> <FONT SIZE=2 FACE="Arial">modify the QP to RTS (in RESET->INIT: enable RDMA Read)</FONT></P>
<P DIR=LTR><FONT SIZE=2 FACE="Arial">side A:</FONT></P>
<P DIR=LTR> <FONT SIZE=2 FACE="Arial">post recv request</FONT></P>
<P DIR=LTR><FONT SIZE=2 FACE="Arial">common:</FONT></P>
<P DIR=LTR> <FONT SIZE=2 FACE="Arial">sync with the other side </FONT></P>
<P DIR=LTR><FONT SIZE=2 FACE="Arial">side B:</FONT></P>
<P DIR=LTR> <FONT SIZE=2 FACE="Arial">post send request (RDMA Write with immediate)</FONT></P>
<P DIR=LTR><FONT SIZE=2 FACE="Arial">Side A:</FONT></P>
<P DIR=LTR> <FONT SIZE=2 FACE="Arial">poll CQ (here is the bug: there isn't any completion in the CQ)</FONT></P>
<P DIR=LTR><FONT SIZE=2 FACE="Arial">when i checked the QP context in side A, i noticed that RDMA Write wasn't enabled for this QP.</FONT></P>
<P DIR=LTR><FONT SIZE=2 FACE="Arial">for RC QP, the test passes.</FONT></P>
<P DIR=LTR><FONT SIZE=2 FACE="Arial">(even if side B would have post send request with only RDMA Write (without immediate), there will be a failure: the packet will be dropped in the responder QP).</FONT></P>
<BR>
<P DIR=LTR><FONT SIZE=2 FACE="Arial">i looked in the file mthca_qp.c and i saw the following code:</FONT></P>
<P DIR=LTR><FONT FACE="Times New Roman">if (attr_mask & IB_QP_ACCESS_FLAGS) {<BR>
/*<BR>
* Only enable RDMA/atomics if we have responder<BR>
* resources set to a non-zero value.<BR>
*/<BR>
if (qp->resp_depth) {<BR>
qp_context->params2 |=<BR>
cpu_to_be32(attr->qp_access_flags & IB_ACCESS_REMOTE_WRITE ?<BR>
MTHCA_QP_BIT_RWE : 0);<BR>
qp_context->params2 |=<BR>
cpu_to_be32(attr->qp_access_flags & IB_ACCESS_REMOTE_READ ?<BR>
MTHCA_QP_BIT_RRE : 0);<BR>
qp_context->params2 |=<BR>
cpu_to_be32(attr->qp_access_flags & IB_ACCESS_REMOTE_ATOMIC ?<BR>
MTHCA_QP_BIT_RAE : 0);<BR>
}<BR>
<BR>
qp_param->opt_param_mask |= cpu_to_be32(MTHCA_QP_OPTPAR_RWE |<BR>
MTHCA_QP_OPTPAR_RRE |<BR>
MTHCA_QP_OPTPAR_RAE);<BR>
<BR>
qp->atomic_rd_en = attr->qp_access_flags;<BR>
}<BR>
<BR>
the value of the attribute resp_depth is being changed (starting) the state INIT->RTR,<BR>
but the value of the attribute atomic_rd_en is being changed (starting) the change RESET->INIT.<BR>
</FONT></P>
<P DIR=LTR><FONT SIZE=2 FACE="Arial">did anyone see this behaviour too?</FONT></P>
<P DIR=LTR><FONT SIZE=2 FACE="Arial">thanx</FONT></P>
<P DIR=LTR><B><I><FONT COLOR="#008080" SIZE=6 FACE="Script">Dotan Barak</FONT></I></B></P>
<P DIR=LTR><FONT FACE="Times New Roman">Software Verification Engineer</FONT></P>
<P DIR=LTR><FONT FACE="Times New Roman">Mellanox Technologies LTD</FONT></P>
<P DIR=LTR><FONT FACE="Times New Roman">Tel: +972-4-9097200 Ext: 231 Fax: +972-4-9593245</FONT></P>
<P DIR=LTR><FONT FACE="Times New Roman">P.O. Box 86 Yokneam 20692 ISRAEL.</FONT></P>
<P DIR=LTR><FONT FACE="Times New Roman">Home: +972-77-8841095 Cell: 0</FONT><FONT SIZE=2 FACE="Arial">52-4222383</FONT></P>
<P DIR=LTR><FONT COLOR="#008080" SIZE=2 FACE="Arial">[ May the fork be with you ]</FONT></P>
</BODY>
</HTML>