Hi all,<br>I am trying to use the APIs defined in <ts_ib_core_types.h> of IBGD-1.8.0.<br>Could the connection between two QPs be established in the similar way to that in *hca_perf* ? In my concept, tow QPs in *hca_perf* exchange information through a TCP socket then move to RTR, RTS state and the connection is established.
<br>I am wondering this because my calling ib_send() caused the state of QP changed from IB_QP_STATE_RTS to IB_QP_STATE_ERROR, and I guess the reason is that the connection was established in the way like *hca_perf* rather than using the CM tools.
<br>If not, what is the reason?<br><br>Here are the details and any suggestion is appreciated.<br><br>
Modify QP to INIT<br>
=============<br> memset(&qp_attr, 0, sizeof(struct ib_qp_attribute));<br><br> qp_attr.valid_fields |= IB_QP_ATTRIBUTE_STATE;<br> qp_attr.state = IB_QP_STATE_INIT;<br><br> qp_attr.valid_fields |= IB_QP_ATTRIBUTE_RDMA_ATOMIC_ENABLE;
<br> qp_attr.enable_rdma_read = 1;<br> qp_attr.enable_rdma_write = 1;<br><br> qp_attr.valid_fields |= IB_QP_ATTRIBUTE_PORT;<br> qp_attr.port = DEFAULT_PORT_NUM;<br><br> qp_attr.valid_fields |= IB_QP_ATTRIBUTE_PKEY_INDEX;
<br> qp_attr.pkey_index = 0;<br> <br> res = ib_qp_modify(param_p->ib_res.qp_p, &qp_attr);<br><br><br>Modify QP to RTR<br>=============<br> memset(&qp_attr, 0, sizeof(struct ib_qp_attribute));<br>
<br> qp_attr.valid_fields |= IB_QP_ATTRIBUTE_STATE;<br> qp_attr.state = IB_QP_STATE_RTR;<br><br> qp_attr.valid_fields |= IB_QP_ATTRIBUTE_RECEIVE_PSN;<br> qp_attr.receive_psn = START_PSN;<br> <br> qp_attr.valid_fields |= IB_QP_ATTRIBUTE_RESPONDER_RESOURCES;
<br> qp_attr.responder_resources = QP_RESPONDER_RESOURCES;<br> <br> qp_attr.valid_fields |= IB_QP_ATTRIBUTE_DESTINATION_QPN;<br> qp_attr.destination_qpn = param_p->dst_msg.qp_num;<br> <br> qp_attr.valid_fields |= IB_QP_ATTRIBUTE_ADDRESS;
<br> qp_attr.address.service_level = 0;<br> qp_attr.address.static_rate = 2;<br> qp_attr.address.dlid = param_p->dst_msg.lid;<br> qp_attr.address.source_path_bits = 0;<br> qp_attr.address.use_grh = 0/*FALSE*/;
<br><br> qp_attr.valid_fields |= IB_QP_ATTRIBUTE_PATH_MTU;<br> qp_attr.path_mtu = param_p->mtu;<br><br> qp_attr.valid_fields |= IB_QP_ATTRIBUTE_RNR_TIMEOUT;<br> qp_attr.rnr_timeout = IB_RNR_TIMER_002_56; <br>
<br> res = ib_qp_modify(param_p->ib_res.qp_p, &qp_attr);<br><br>Modify QP to RTS<br>
=============<br> memset(&qp_attr, 0, sizeof(struct ib_qp_attribute));<br> qp_attr.valid_fields |= IB_QP_ATTRIBUTE_STATE;<br> qp_attr.state = IB_QP_STATE_RTS;<br><br> qp_attr.valid_fields |= IB_QP_ATTRIBUTE_SEND_PSN;
<br> qp_attr.send_psn = START_PSN;<br> <br> qp_attr.valid_fields |= IB_QP_ATTRIBUTE_INITIATOR_DEPTH;<br> qp_attr.initiator_depth = QP_INITIATOR_DEPTH;<br><br> qp_attr.valid_fields |= IB_QP_ATTRIBUTE_LOCAL_ACK_TIMEOUT;
<br> qp_attr.local_ack_timeout = QP_LOCAL_ACK_TIMEOUT;<br> <br> qp_attr.valid_fields |= IB_QP_ATTRIBUTE_RETRY_COUNT;<br> qp_attr.retry_count = QP_RETRY_COUNT;<br><br> qp_attr.valid_fields |= IB_QP_ATTRIBUTE_RNR_RETRY_COUNT;
<br> qp_attr.rnr_retry_count = QP_RETRY_COUNT;<br> <br> res = ib_qp_modify(param_p->ib_res.qp_p, &qp_attr);<br><br>Send<br>====<br> memset(&u_mr_p->send_param, 0, sizeof(struct ib_send_param));<br>
u_mr_p->send_param.op = IB_OP_SEND;<br> ibsg.address = virt_to_phys(u_mr_p->user_buf);<br> ibsg.length = u_mr_p->user_buf_len;<br> ibsg.key = u_mr_p->lkey;<br> u_mr_p->send_param.work_request_id = (u64)u_mr_p;
<br> u_mr_p->send_param.num_gather_entries = 1;<br> u_mr_p->send_param.gather_list = &ibsg;<br> u_mr_p->send_param.signaled = 1;<br> u_mr_p->send_param.dest_qpn = u_mr_p->dst_qpn;<br><br> PRINT_SEND_PARAM(&u_mr_p->send_param);
<br>#ifdef KVAPITEST_DEBUG<br> show_qp_state(ib_res_p->qp_p);<br>#endif<br> res = ib_send(ib_res_p->qp_p, <br> &u_mr_p->send_param, <br> 1, <br> &actual_wrn);<br>#ifdef KVAPITEST_DEBUG
<br> MY_PRINT("AFTER ib_send\n");<br> show_qp_state(ib_res_p->qp_p);<br>#endif<br><br>Send CQ event handler<br>=================<br>void cq_send_handler(struct ib_cq *cq, struct ib_cq_entry *cq_entry, void *arg)
<br>{<br> if (cq == NULL) {<br> PRINT_ERR("NULL cq\n");<br> return;<br> }<br> if (cq_entry == NULL) {<br> PRINT_ERR("NULL entry of cq: 0x%p\n", cq);<br> return;<br>
}<br><br> PRINT_TRACE("cq: 0x%p, cq_entry: 0x%p\n", cq, cq_entry);<br> PRINT_CQ_ENTRY(cq_entry);<br><br> spin_lock(&send_cq_lock.slock);<br> PRINT_TRACE("set send_cq_lock.state to GOT(%d)\n", CQ_STATE_GOT);
<br> send_cq_lock.state = CQ_STATE_GOT;<br> spin_unlock(&send_cq_lock.slock);<br> wake_up(&send_cq_lock.waitq);<br>}<br><br>Debug infor<br>========<br>Mar 8 21:42:38 linux3 kernel: ==== QP attr to rtr ====
<br>Mar 8 21:42:38 linux3 kernel: .qp_state: 2<br>Mar 8 21:42:38 linux3 kernel: .av.sl: 0<br>Mar 8 21:42:38 linux3 kernel: .av.grh_flag: 0<br>Mar 8 21:42:38 linux3 kernel: .av.dlid: 0x161<br>Mar 8 21:42:38 linux3 kernel: .av.static_rate: 2
<br>Mar 8 21:42:38 linux3 kernel: .av.src_path_bits: 0<br>Mar 8 21:42:38 linux3 kernel: .path_mtu: 4<br>Mar 8 21:42:38 linux3 kernel: .dst_qp_num: 0x70405<br>Mar 8 21:42:38 linux3 kernel: ready_to_trans: move to RTS...
<br>Mar 8 21:42:38 linux3 kernel: qp_move_to_rts: Modified QP to RTS<br>Mar 8 21:42:38 linux3 kernel: ==== QP attr to rts ====<br>Mar 8 21:42:38 linux3 kernel: .qp_state: 3<br>Mar 8 21:42:38 linux3 kernel: test_sr_lat: test 0, [Sender]
<br>Mar 8 21:42:38 linux3 kernel: test_sr_lat: wait to start...<br>Mar 8 21:42:38 linux3 kernel: test_sr_lat: create out_mr<br>Mar 8 21:42:38 linux3 kernel: user_pmr_create: USE_PMR<br>Mar 8 21:42:38 linux3 kernel: user_pmr_create: user buffer 0x00000100271eb000, len: 4096
<br>Mar 8 21:42:38 linux3 kernel: user_pmr_create: register physical memory<br>Mar 8 21:42:38 linux3 kernel: user_pmr_create: PD hndl: 0x0000010033b6f8c0<br>Mar 8 21:42:38 linux3 kernel: user_pmr_create: physical buf start 0x271eb000, size 0x1000
<br>Mar 8 21:42:38 linux3 kernel: ==== user MR ====<br>Mar 8 21:42:38 linux3 kernel: buf start 0x00000100271eb000<br>Mar 8 21:42:38 linux3 kernel: buf len: 4096<br>Mar 8 21:42:38 linux3 kernel: IOVA: 0x0<br>Mar 8 21:42:38 linux3 kernel: IOVA offset: 0x0
<br>Mar 8 21:42:38 linux3 kernel: l_key: 0x413f8b00<br>Mar 8 21:42:38 linux3 kernel: r_key: 0x413f8b00<br>Mar 8 21:42:38 linux3 kernel: PD: 0x0000010033b6f8c0<br>Mar 8 21:42:38 linux3 kernel: IB MR.device: 0x000001003bc896d0
<br>Mar 8 21:42:38 linux3 kernel: IB MR.lkey: 0x413f8b00<br>Mar 8 21:42:38 linux3 kernel: IB MR.rkey: 0x413f8b00<br>Mar 8 21:42:38 linux3 kernel: IB MR.private: 0xa55a5a5a00000003<br>Mar 8 21:42:38 linux3 kernel: test_sr_lat: post_send
<br>Mar 8 21:42:38 linux3 kernel: ==== send param ====<br>Mar 8 21:42:38 linux3 kernel: WR ID: 0x10029649c18<br>Mar 8 21:42:38 linux3 kernel: op type: IB_OP_SEND<br>Mar 8 21:42:38 linux3 kernel: remote addr: 0x0<br>Mar 8 21:42:38 linux3 kernel: r_key: 0x0
<br>Mar 8 21:42:38 linux3 kernel: dest_qpn: 0x70405<br>Mar 8 21:42:38 linux3 kernel: dest_qkey: 0x0<br>Mar 8 21:42:38 linux3 kernel: static_rate: IB_STATIC_RATE_FULL<br>Mar 8 21:42:38 linux3 kernel: num_gather_entries: 1
<br>Mar 8 21:42:38 linux3 kernel: entry[0]: addr 0x271eb000, len 0x1000, key 0x413f8b00<br>Mar 8 21:42:38 linux3 kernel: device_spec: x00000000000000000<br>Mar 8 21:42:38 linux3 kernel: signaled: 1<br>Mar 8 21:42:38 linux3 kernel: === QP[0x0000010029c96c90] ===
<br>Mar 8 21:42:38 linux3 kernel: state: IB_QP_STATE_RTS<br>Mar 8 21:42:38 linux3 kernel: destination_qpn: 0x70405<br>Mar 8 21:42:38 linux3 kernel: ......<br>Mar 8 21:42:38 linux3 kernel: AFTER ib_send<br>Mar 8 21:42:38 linux3 kernel: cq_send_handler: cq: 0x0000010029c96948, cq_entry: 0xffffffff805f0b58
<br>Mar 8 21:42:38 linux3 kernel: ==== CQ entry ====<br>Mar 8 21:42:38 linux3 kernel: device spec. 0x000001003830c038<br>Mar 8 21:42:38 linux3 kernel: WR id: 0x10029649c18<br>Mar 8 21:42:38 linux3 kernel: Operation type: IB_COMPLETION_OP_RECEIVE
<br>Mar 8 21:42:38 linux3 kernel: status: IB_COMPLETION_STATUS_INVALID<br>Mar 8 21:42:38 linux3 kernel: bytes transferred: 0xffffffff805f0ba8<br>Mar 8 21:42:38 linux3 kernel: pkey_index 0x206<br>Mar 8 21:42:38 linux3 kernel: sqpn: 0x805f0c08
<br>Mar 8 21:42:38 linux3 kernel: slid: 0x0<br>Mar 8 21:42:38 linux3 kernel: ......<br>Mar 8 21:42:38 linux3 kernel: cq_send_handler: set send_cq_lock.state to GOT(2)<br>Mar 8 21:42:38 linux3 kernel: === QP[0x0000010029c96c90] ===
<br>Mar 8 21:42:38 linux3 kernel: state: IB_QP_STATE_ERROR<br>Mar 8 21:42:38 linux3 kernel: destination_qpn: 0x70405<br>Mar 8 21:42:38 linux3 kernel: ......<br><br><br clear="all"><br>-- <br>Ian Jiang<br><a href="mailto:ianjiang.ict@gmail.com">
ianjiang.ict@gmail.com</a><br><br>Laboratory of Spatial Information Technology<br>Division of System Architecture<br>Institute of Computing Technology<br>Chinese Academy of Sciences