Hi,<br><br> Was there any resolution to this issue? I am seeing the exact behavior<br> where no event is generated after doing a send. There were a few successful sends<br> that got completion events. But it just stops without any error indication.<br>
<br> I am pasting the part of the code which does this operation:<br><br>create_qp ()<br>{<br> struct ibv_qp_init_attr init_attr;<br><br> init_attr.cap.max_send_wr = 20;<br> init_attr.cap.max_recv_wr = 20;<br>
init_attr.cap.max_recv_sge = 1;<br> init_attr.cap.max_send_sge = 1;<br> init_attr.qp_type = IBV_QPT_RC;<br> init_attr.send_cq = send_cq;<br> init_attr.recv_cq = recv_cq;<br> init_attr.sq_sig_all = 0;<br>
<br> qp = ibv_create_qp(pd, &init_attr);<br><br> if (!qp) {<br> return 1;<br> }<br><br> attr.qp_state = IBV_QPS_INIT;<br> attr.pkey_index = 0;<br> attr.port_num = src_port;<br>
attr.qp_access_flags = 0;<br><br> if (ibv_modify_qp(qp, &attr,<br> IBV_QP_STATE |<br> IBV_QP_PKEY_INDEX |<br> IBV_QP_PORT |<br> IBV_QP_ACCESS_FLAGS)) {<br>
return 1;<br> }<br><br><br> attr.qp_state = IBV_QPS_RTR;<br> attr.path_mtu = IBV_MTU_2048;<br> attr.rq_psn = 1;<br> attr.dest_qp_num = dst_qp_num;<br> attr.max_dest_rd_atomic = 1;<br> attr.ah_attr.dlid = dst_lid;<br>
<a href="http://attr.ah_attr.sl">attr.ah_attr.sl</a> = serv_level;<br> attr.ah_attr.port_num = src_port;<br> attr.min_rnr_timer = 12;<br> attr.ah_attr.is_global = 0;<br> attr.ah_attr.src_path_bits = 0;<br>
<br> if (ibv_modify_qp(qp, &attr,<br> IBV_QP_STATE| <br> IBV_QP_PATH_MTU|<br> IBV_QP_RQ_PSN|<br> IBV_QP_DEST_QPN|<br>
IBV_QP_MAX_DEST_RD_ATOMIC| <br> IBV_QP_AV| <br> IBV_QP_MIN_RNR_TIMER)) {<br> return 1;<br> }<br><br> attr.qp_state = IBV_QPS_RTS;<br>
attr.timeout = 10;<br> attr.retry_cnt = 7;<br> attr.rnr_retry = 7;<br> attr.sq_psn = 1;<br> attr.max_rd_atomic = 1;<br> if (ibv_modify_qp(qp, &attr,<br> IBV_QP_STATE |<br> IBV_QP_TIMEOUT |<br>
IBV_QP_RETRY_CNT |<br> IBV_QP_RNR_RETRY |<br> IBV_QP_SQ_PSN |<br> IBV_QP_MAX_QP_RD_ATOMIC)) {<br> return 1;<br> }<br><br>}<br>
<br>send_data(char *buf, int datasz, void *arg)<br>{<br><br> int ret;<br><br> /*<br> * Save the WR-id so that we can compare against this<br> * once tx is done.<br> */<br> sq_wr_id[tail] = global_cnt++;<br>
<br> send_sgl[tail].addr = (u64) (unsigned long) buf;<br> send_sgl[tail].length = datasz;<br> send_sgl[tail].lkey = send_mr->lkey;<br><br> sq_wr[tail].opcode = IBV_WR_SEND;<br> sq_wr[tail].send_flags = IBV_SEND_SIGNALED;<br>
sq_wr[tail].sg_list = &send_sgl[tail];<br> sq_wr[tail].num_sge = 1;<br> send_data[tail] = (u64)buf;<br> send_arg[tail] = arg;<br><br> ret = ibv_post_send(qp, &sq_wr[tail], &bad_wr);<br><br> if (tail == 19) { //max_send_wr -1<br>
tail = 0;<br>
} else {<br>
tail += 1;<br>
}<br> return ret;<br>}<br><br>recv_thread (void *arg)<br>{<br> struct ibv_cq *ev_cq;<br> void *ev_ctx;<br> int ret;<br><br><br> ret = ibv_get_cq_event(comp_channel, &ev_cq, &ev_ctx);<br>
if (ret) {<br> return 1;<br> }<br><br> ibv_ack_cq_events(ev_cq, 1);<br><br> ret = ibv_req_notify_cq(ev_cq, 0);<br> if (ret) {<br> return 1;<br> }<br><br> while ((rv = ibv_poll_cq(cq, 1, &wc)) == 1) {<br>
switch (wc.opcode) {<br> case IBV_WC_SEND: {<br> if (wc.status == IBV_WC_SUCCESS) {<br> if (sq_wr_id[head] != wc.wr_id) {<br> datasz = 0;<br> return 1;<br>
}<br> } else {<br> retuen 1;<br> }<br> buf = (char *)send_data[head];<br> arg = (u64)send_arg[head];<br> sq_wr_id[head] = 0;<br>
if (head == 19) {//max_send_wr -1<br> head = 0;<br> } else {<br> head += 1;<br> }<br> break;<br> }<br> }<br><br>
}<br><br>Thanks<br>Ganesh<br><br><div class="gmail_quote">On Mon, May 28, 2007 at 9:28 PM, Roland Dreier <<a href="mailto:rdreier@cisco.com">rdreier@cisco.com</a>> wrote:<br><blockquote class="gmail_quote" style="border-left: 1px solid rgb(204, 204, 204); margin: 0pt 0pt 0pt 0.8ex; padding-left: 1ex;">
<div class="Ih2E3d"> > Any ideas on why the ibv_get_cq_event() would never see an event<br>
> after a "successful" send requesting a completion event?<br>
<br>
</div>It's either a bug in your code or a bug in the stack below your code.<br>
The best way to debug this would be for you to post your actual code<br>
(in a form that someone else can run), so that we can either point out<br>
what's wrong with your code, or have a test case for the real bug.<br>
<font color="#888888"><br>
- R.<br>
</font><div><div></div><div class="Wj3C7c">_______________________________________________<br>
general mailing list<br>
<a href="mailto:general@lists.openfabrics.org">general@lists.openfabrics.org</a><br>
<a href="http://lists.openfabrics.org/cgi-bin/mailman/listinfo/general" target="_blank">http://lists.openfabrics.org/cgi-bin/mailman/listinfo/general</a><br>
<br>
To unsubscribe, please visit <a href="http://openib.org/mailman/listinfo/openib-general" target="_blank">http://openib.org/mailman/listinfo/openib-general</a><br>
</div></div></blockquote></div><br>