<html xmlns:o="urn:schemas-microsoft-com:office:office" xmlns:w="urn:schemas-microsoft-com:office:word" xmlns:st1="urn:schemas-microsoft-com:office:smarttags" xmlns="http://www.w3.org/TR/REC-html40">
<head>
<meta http-equiv=Content-Type content="text/html; charset=us-ascii">
<meta name=Generator content="Microsoft Word 11 (filtered medium)">
<o:SmartTagType namespaceuri="urn:schemas-microsoft-com:office:smarttags"
name="PersonName"/>
<!--[if !mso]>
<style>
st1\:*{behavior:url(#default#ieooui) }
</style>
<![endif]-->
<style>
<!--
/* Font Definitions */
@font-face
{font-family:Tahoma;
panose-1:2 11 6 4 3 5 4 4 2 4;}
/* Style Definitions */
p.MsoNormal, li.MsoNormal, div.MsoNormal
{margin:0in;
margin-bottom:.0001pt;
font-size:12.0pt;
font-family:"Times New Roman";}
a:link, span.MsoHyperlink
{color:blue;
text-decoration:underline;}
a:visited, span.MsoHyperlinkFollowed
{color:purple;
text-decoration:underline;}
span.EmailStyle17
{mso-style-type:personal;
font-family:Tahoma;
color:windowtext;
font-weight:normal;
font-style:normal;
text-decoration:none none;}
span.EmailStyle18
{mso-style-type:personal;
font-family:Tahoma;
color:blue;
font-weight:normal;
font-style:normal;
text-decoration:none none;}
span.EmailStyle19
{mso-style-type:personal-reply;
font-family:Arial;
color:navy;}
@page Section1
{size:8.5in 11.0in;
margin:1.0in 1.25in 1.0in 1.25in;}
div.Section1
{page:Section1;}
-->
</style>
</head>
<body lang=EN-US link=blue vlink=purple>
<div class=Section1>
<p class=MsoNormal><font size=2 face=Arial><span style='font-size:10.0pt;
font-family:Arial'>I don’t understand the context value issue yet. I
did test IBAL on mthca with IPoIB and userspace tests. If the context
were incorrect, I would have expected to see an issue, and the context values
into the winverbs driver were correct. Can you provide a little more
details on the problem? I’d like to understand where the issues are
in the assumptions that were made in the 1435 patch.<o:p></o:p></span></font></p>
<p class=MsoNormal><font size=2 face=Arial><span style='font-size:10.0pt;
font-family:Arial'><o:p> </o:p></span></font></p>
<p class=MsoNormal><font size=2 color=navy face=Arial><span style='font-size:
10.0pt;font-family:Arial;color:navy'><o:p> </o:p></span></font></p>
<div style='border:none;border-left:solid blue 1.5pt;padding:0in 0in 0in 4.0pt'>
<div>
<p class=MsoNormal><font size=2 color=blue face=Arial><span style='font-size:
10.0pt;font-family:Arial;color:blue'>Find attached a fix, that will
maybe solve the problem.</span></font><o:p></o:p></p>
</div>
<div>
<p class=MsoNormal><font size=2 color=blue face=Arial><span style='font-size:
10.0pt;font-family:Arial;color:blue'>Maybe, because i haven't time to check it
and this work week (down here) is ended.</span></font><o:p></o:p></p>
</div>
<div>
<p class=MsoNormal><font size=2 color=blue face=Arial><span style='font-size:
10.0pt;font-family:Arial;color:blue'>I'll check the patch next week.</span></font><o:p></o:p></p>
</div>
<div>
<p class=MsoNormal><font size=3 face="Times New Roman"><span style='font-size:
12.0pt'> <o:p></o:p></span></font></p>
</div>
<div>
<p class=MsoNormal><font size=2 color=blue face=Arial><span style='font-size:
10.0pt;font-family:Arial;color:blue'>About the patch.</span></font><o:p></o:p></p>
</div>
<div>
<p class=MsoNormal><font size=3 face="Times New Roman"><span style='font-size:
12.0pt'> <o:p></o:p></span></font></p>
</div>
<div>
<p class=MsoNormal><font size=2 color=blue face=Arial><span style='font-size:
10.0pt;font-family:Arial;color:blue'>First of all, special thanks to Anatoly
for the right pointing to 1435 patch.</span></font><o:p></o:p></p>
</div>
<div>
<p class=MsoNormal><font size=3 face="Times New Roman"><span style='font-size:
12.0pt'> <o:p></o:p></span></font></p>
</div>
<div>
<p class=MsoNormal><font size=2 color=blue face=Arial><span style='font-size:
10.0pt;font-family:Arial;color:blue'>To recall, 1435 patch has improved
event notification mechanism for cq, qp and srq objects.</span></font><o:p></o:p></p>
</div>
<div>
<p class=MsoNormal><font size=2 color=blue face=Arial><span style='font-size:
10.0pt;font-family:Arial;color:blue'>I found one problem in the patch, which
repeats itself for all three objects and for both drivers: new event handlers
get the old (and wrong) context values.</span></font><o:p></o:p></p>
</div>
<div>
<p class=MsoNormal><font size=2 color=blue face=Arial><span style='font-size:
10.0pt;font-family:Arial;color:blue'>The new (and right) context
values are nor used. As a result, IBAL callbacks are called with
wrong handle parameter, which ends up with c<st1:PersonName w:st="on">ras</st1:PersonName>h.</span></font><o:p></o:p></p>
</div>
<div>
<p class=MsoNormal><font size=3 face="Times New Roman"><span style='font-size:
12.0pt'> <o:p></o:p></span></font></p>
</div>
<div>
<p class=MsoNormal><font size=3 face="Times New Roman"><span style='font-size:
12.0pt'> <o:p></o:p></span></font></p>
</div>
<div>
<p class=MsoNormal><font size=2 color=blue face=Arial><span style='font-size:
10.0pt;font-family:Arial;color:blue'>Index: hw/mlx4/kernel/bus/inc/ib_verbs.h<br>
===================================================================<br>
--- hw/mlx4/kernel/bus/inc/ib_verbs.h (revision 1452)<br>
+++ hw/mlx4/kernel/bus/inc/ib_verbs.h (working copy)<br>
@@ -742,7 +742,6 @@<br>
void
*
cq_context;<br>
int
cqe;<br>
atomic_t
usecnt; /* count number of work queues */<br>
- struct ib_cq_ex x;<br>
};<br>
<br>
struct ib_srq {<br>
@@ -752,7 +751,6 @@<br>
void
(*event_handler)(ib_event_rec_t *);<br>
void *srq_context;<br>
atomic_t usecnt;<br>
- struct ib_srq_ex x;<br>
};<br>
<br>
struct ib_qp {<br>
@@ -766,7 +764,6 @@<br>
void *qp_context;<br>
u32 qp_num;<br>
enum ib_qp_type qp_type;<br>
- struct ib_qp_ex x;<br>
};<br>
<br>
struct ib_mr {<br>
Index: hw/mlx4/kernel/bus/inc/ib_verbs_ex.h<br>
===================================================================<br>
--- hw/mlx4/kernel/bus/inc/ib_verbs_ex.h (revision 1452)<br>
+++ hw/mlx4/kernel/bus/inc/ib_verbs_ex.h (working copy)<br>
@@ -73,24 +73,6 @@<br>
int fw_if_open;<br>
};<br>
<br>
-/* extension for ib_cq */<br>
-struct ib_cq_ex <br>
-{<br>
- void * ctx; /* IBAL CQ context */<br>
-};<br>
-<br>
-/* extension for ib_qp */<br>
-struct ib_qp_ex <br>
-{<br>
- void * ctx; /* IBAL QP context */<br>
-};<br>
-<br>
-/* extension for ib_srq */<br>
-struct ib_srq_ex <br>
-{<br>
- void * ctx; /* IBAL SRQ context */<br>
-};<br>
-<br>
/* extension for ib_event */<br>
struct ib_event_ex <br>
{<br>
Index: hw/mlx4/kernel/hca/cq.c<br>
===================================================================<br>
--- hw/mlx4/kernel/hca/cq.c (revision 1452)<br>
+++ hw/mlx4/kernel/hca/cq.c (working copy)<br>
@@ -89,7 +89,7 @@<br>
// allocate cq <br>
p_ib_cq = ibv_create_cq(p_ibdev, <br>
cq_comp_handler, event_handler,<br>
- p_hca, *p_size, p_uctx, p_umv_buf );<br>
+ (void*)cq_context, *p_size, p_uctx, p_umv_buf );<br>
if (IS_ERR(p_ib_cq)) {<br>
err = PTR_ERR(p_ib_cq);<br>
HCA_PRINT (TRACE_LEVEL_ERROR ,HCA_DBG_CQ,
("ibv_create_cq failed (%d)\n", err));<br>
@@ -97,9 +97,6 @@<br>
goto err_create_cq;<br>
}<br>
<br>
- // fill the object<br>
- p_ib_cq->x.ctx = (void*)cq_context;<br>
- <br>
// return the result<br>
*p_size = p_ib_cq->cqe;<br>
<br>
Index: hw/mlx4/kernel/hca/qp.c<br>
===================================================================<br>
--- hw/mlx4/kernel/hca/qp.c (revision 1452)<br>
+++ hw/mlx4/kernel/hca/qp.c (working copy)<br>
@@ -100,8 +100,6 @@<br>
struct ib_qp_init_attr qp_init_attr;<br>
struct ib_ucontext *p_uctx = NULL;<br>
struct ib_pd *p_ib_pd = (struct ib_pd *)h_pd;<br>
- struct ib_device *p_ib_dev = p_ib_pd->device;<br>
- mlnx_hca_t *p_hca = ibdev2hca(p_ib_dev);<br>
struct ibv_create_qp *p_req = NULL;<br>
<br>
HCA_ENTER(HCA_DBG_QP);<br>
@@ -121,7 +119,7 @@<br>
// prepare the parameters<br>
RtlZeroMemory(&qp_init_attr, sizeof(qp_init_attr));<br>
qp_init_attr.event_handler = event_handler;<br>
- qp_init_attr.qp_context = p_hca;<br>
+ qp_init_attr.qp_context = (void*)qp_uctx;<br>
qp_init_attr.send_cq = (struct ib_cq *)p_create_attr->h_sq_cq;<br>
qp_init_attr.recv_cq = (struct ib_cq *)p_create_attr->h_rq_cq;<br>
qp_init_attr.srq = (struct ib_srq *)p_create_attr->h_srq;<br>
@@ -153,9 +151,6 @@<br>
goto err_create_qp;<br>
}<br>
<br>
- // fill the object<br>
- p_ib_qp->x.ctx = (void*)qp_uctx;<br>
-<br>
// Query QP to obtain requested attributes<br>
if (p_qp_attr) {<br>
status = mlnx_query_qp((ib_qp_handle_t)p_ib_qp, p_qp_attr,
p_umv_buf);<br>
Index: hw/mlx4/kernel/hca/srq.c<br>
===================================================================<br>
--- hw/mlx4/kernel/hca/srq.c (revision 1452)<br>
+++ hw/mlx4/kernel/hca/srq.c (working copy)<br>
@@ -55,8 +55,6 @@<br>
struct ib_srq_init_attr srq_init_attr;<br>
struct ib_ucontext *p_uctx = NULL;<br>
struct ib_pd *p_ib_pd = (struct ib_pd *)h_pd;<br>
- struct ib_device *p_ib_dev = p_ib_pd->device;<br>
- mlnx_hca_t *p_hca = ibdev2hca(p_ib_dev);<br>
<br>
HCA_ENTER(HCA_DBG_SRQ);<br>
<br>
@@ -75,7 +73,7 @@<br>
// prepare the parameters<br>
RtlZeroMemory(&srq_init_attr, sizeof(srq_init_attr));<br>
srq_init_attr.event_handler = event_handler;<br>
- srq_init_attr.srq_context = p_hca;<br>
+ srq_init_attr.srq_context = (void*)srq_context;<br>
srq_init_attr.attr.max_wr = p_srq_attr->max_wr;<br>
srq_init_attr.attr.max_sge = p_srq_attr->max_sge;<br>
srq_init_attr.attr.srq_limit = p_srq_attr->srq_limit;<br>
@@ -88,7 +86,6 @@<br>
status = errno_to_iberr(err);<br>
goto err_create_srq;<br>
}<br>
- p_ib_srq->x.ctx = (void*)srq_context;<br>
<br>
// return the result<br>
if (ph_srq) *ph_srq = (ib_srq_handle_t)p_ib_srq;<br>
Index: hw/mthca/kernel/hca_verbs.c<br>
===================================================================<br>
--- hw/mthca/kernel/hca_verbs.c (revision 1452)<br>
+++ hw/mthca/kernel/hca_verbs.c (working copy)<br>
@@ -870,12 +870,10 @@<br>
int err;<br>
ib_api_status_t status;<br>
struct ib_srq *ib_srq_p;<br>
- struct mthca_srq *srq_p;<br>
struct ib_srq_init_attr srq_init_attr;<br>
struct ib_ucontext *p_context = NULL;<br>
struct ib_pd *ib_pd_p = (struct ib_pd *)h_pd;<br>
struct ib_device *ib_dev = ib_pd_p->device;<br>
- mlnx_hob_t *hob_p = HOB_FROM_IBDEV(ib_dev);<br>
<br>
HCA_ENTER(HCA_DBG_SRQ);<br>
<br>
@@ -894,7 +892,7 @@<br>
// prepare the parameters<br>
RtlZeroMemory(&srq_init_attr, sizeof(srq_init_attr));<br>
srq_init_attr.event_handler = event_handler;<br>
- srq_init_attr.srq_context = hob_p;<br>
+ srq_init_attr.srq_context = (void*)srq_context;<br>
srq_init_attr.attr = *p_srq_attr;<br>
<br>
// allocate srq <br>
@@ -906,12 +904,8 @@<br>
goto err_create_srq;<br>
}<br>
<br>
- // fill the object<br>
- srq_p = (struct mthca_srq *)ib_srq_p;<br>
- srq_p->srq_context = (void*)srq_context;<br>
- <br>
// return the result<br>
- if (ph_srq) *ph_srq = (ib_srq_handle_t)srq_p;<br>
+ if (ph_srq) *ph_srq = (ib_srq_handle_t)ib_srq_p;<br>
<br>
status = IB_SUCCESS;<br>
<br>
@@ -1044,7 +1038,6 @@<br>
struct ib_ucontext *p_context = NULL;<br>
struct ib_pd *ib_pd_p = (struct ib_pd *)h_pd;<br>
struct ib_device *ib_dev = ib_pd_p->device;<br>
- mlnx_hob_t *hob_p = HOB_FROM_IBDEV(ib_dev);<br>
<br>
HCA_ENTER(HCA_DBG_QP);<br>
<br>
@@ -1063,7 +1056,7 @@<br>
RtlZeroMemory(&qp_init_attr, sizeof(qp_init_attr));<br>
qp_init_attr.qp_type = p_create_attr->qp_type;<br>
qp_init_attr.event_handler = event_handler;<br>
- qp_init_attr.qp_context = hob_p;<br>
+ qp_init_attr.qp_context = (void*)qp_context;<br>
qp_init_attr.recv_cq = (struct ib_cq *)p_create_attr->h_rq_cq;<br>
qp_init_attr.send_cq = (struct ib_cq *)p_create_attr->h_sq_cq;<br>
qp_init_attr.srq = (struct ib_srq *)p_create_attr->h_srq;<br>
@@ -1087,7 +1080,6 @@<br>
<br>
// fill the object<br>
qp_p = (struct mthca_qp *)ib_qp_p;<br>
- qp_p->qp_context = (void*)qp_context;<br>
qp_p->qp_init_attr = qp_init_attr;<br>
<br>
// Query QP to obtain requested attributes<br>
@@ -1401,7 +1393,6 @@<br>
int err;<br>
ib_api_status_t status;<br>
struct ib_cq *ib_cq_p;<br>
- struct mthca_cq *cq_p;<br>
mlnx_hob_t *hob_p;<br>
struct ib_device *ib_dev;<br>
struct ib_ucontext *p_context;<br>
@@ -1437,7 +1428,7 @@<br>
// allocate cq <br>
ib_cq_p = ibv_create_cq(ib_dev, <br>
cq_comp_handler, event_handler,<br>
- hob_p, *p_size, p_context, p_umv_buf );<br>
+ (void*)cq_context, *p_size, p_context, p_umv_buf );<br>
if (IS_ERR(ib_cq_p)) {<br>
err = PTR_ERR(ib_cq_p);<br>
HCA_PRINT (TRACE_LEVEL_ERROR ,HCA_DBG_CQ,
("ibv_create_cq failed (%d)\n", err));<br>
@@ -1445,15 +1436,11 @@<br>
goto err_create_cq;<br>
}<br>
<br>
- // fill the object<br>
- cq_p = (struct mthca_cq *)ib_cq_p;<br>
- cq_p->cq_context = (void*)cq_context;<br>
- <br>
// return the result<br>
// *p_size = *p_size; // return the same value<br>
*p_size = ib_cq_p->cqe;<br>
<br>
- if (ph_cq) *ph_cq = (ib_cq_handle_t)cq_p;<br>
+ if (ph_cq) *ph_cq = (ib_cq_handle_t)ib_cq_p;<br>
<br>
status = IB_SUCCESS;<br>
<br>
Index: hw/mthca/kernel/mthca_cq.c<br>
===================================================================<br>
--- hw/mthca/kernel/mthca_cq.c (revision 1452)<br>
+++ hw/mthca/kernel/mthca_cq.c (working copy)<br>
@@ -237,7 +237,7 @@<br>
++cq->arm_sn;<br>
}<br>
<br>
- cq->ibcq.comp_handler(cq->cq_context);<br>
+ cq->ibcq.comp_handler(cq->ibcq.cq_context);<br>
}<br>
<br>
void mthca_cq_event(struct mthca_dev *dev, u32 cqn,<br>
Index: hw/mthca/kernel/mthca_provider.h<br>
===================================================================<br>
--- hw/mthca/kernel/mthca_provider.h (revision 1452)<br>
+++ hw/mthca/kernel/mthca_provider.h (working copy)<br>
@@ -189,7 +189,6 @@<br>
<br>
struct mthca_cq {<br>
struct
ib_cq ibcq;<br>
- void *cq_context; // leo: for
IBAL shim<br>
spinlock_t
lock;<br>
atomic_t
refcount;<br>
int
cqn;<br>
@@ -234,7 +233,6 @@<br>
<br>
wait_queue_head_t wait;<br>
KMUTEX mutex;<br>
- void *srq_context; <br>
};<br>
<br>
struct mthca_wq {<br>
@@ -254,7 +252,6 @@<br>
<br>
struct mthca_qp {<br>
struct
ib_qp ibqp;<br>
- void *qp_context; // leo: for
IBAL shim<br>
//TODO: added just because absense of ibv_query_qp<br>
// thereafter it may be worth to be replaced by struct ib_qp_attr
qp_attr;<br>
struct ib_qp_init_attr qp_init_attr; // leo: for query_qp</span></font><o:p></o:p></p>
</div>
<blockquote style='border:none;border-left:solid blue 1.5pt;padding:0in 0in 0in 4.0pt;
margin-left:3.75pt;margin-top:5.0pt;margin-right:0in;margin-bottom:5.0pt'>
<p class=MsoNormal><font size=3 face="Times New Roman"><span style='font-size:
12.0pt'> </span></font><font size=2 face=Tahoma><span style='font-size:
11.0pt;font-family:Tahoma'><o:p></o:p></span></font></p>
<p class=MsoNormal><font size=2 face=Tahoma><span style='font-size:11.0pt;
font-family:Tahoma'><o:p> </o:p></span></font></p>
</blockquote>
</div>
</div>
</body>
</html>