[openib-general] is my connection establishment correct?
Ian Jiang
ianjiang.ict at gmail.com
Wed Mar 8 06:57:19 PST 2006
Hi all,
I am trying to use the APIs defined in <ts_ib_core_types.h> of IBGD-1.8.0.
Could the connection between two QPs be established in the similar way to
that in *hca_perf* ? In my concept, tow QPs in *hca_perf* exchange
information through a TCP socket then move to RTR, RTS state and the
connection is established.
I am wondering this because my calling ib_send() caused the state of QP
changed from IB_QP_STATE_RTS to IB_QP_STATE_ERROR, and I guess the reason is
that the connection was established in the way like *hca_perf* rather than
using the CM tools.
If not, what is the reason?
Here are the details and any suggestion is appreciated.
Modify QP to INIT
=============
memset(&qp_attr, 0, sizeof(struct ib_qp_attribute));
qp_attr.valid_fields |= IB_QP_ATTRIBUTE_STATE;
qp_attr.state = IB_QP_STATE_INIT;
qp_attr.valid_fields |= IB_QP_ATTRIBUTE_RDMA_ATOMIC_ENABLE;
qp_attr.enable_rdma_read = 1;
qp_attr.enable_rdma_write = 1;
qp_attr.valid_fields |= IB_QP_ATTRIBUTE_PORT;
qp_attr.port = DEFAULT_PORT_NUM;
qp_attr.valid_fields |= IB_QP_ATTRIBUTE_PKEY_INDEX;
qp_attr.pkey_index = 0;
res = ib_qp_modify(param_p->ib_res.qp_p, &qp_attr);
Modify QP to RTR
=============
memset(&qp_attr, 0, sizeof(struct ib_qp_attribute));
qp_attr.valid_fields |= IB_QP_ATTRIBUTE_STATE;
qp_attr.state = IB_QP_STATE_RTR;
qp_attr.valid_fields |= IB_QP_ATTRIBUTE_RECEIVE_PSN;
qp_attr.receive_psn = START_PSN;
qp_attr.valid_fields |= IB_QP_ATTRIBUTE_RESPONDER_RESOURCES;
qp_attr.responder_resources = QP_RESPONDER_RESOURCES;
qp_attr.valid_fields |= IB_QP_ATTRIBUTE_DESTINATION_QPN;
qp_attr.destination_qpn = param_p->dst_msg.qp_num;
qp_attr.valid_fields |= IB_QP_ATTRIBUTE_ADDRESS;
qp_attr.address.service_level = 0;
qp_attr.address.static_rate = 2;
qp_attr.address.dlid = param_p->dst_msg.lid;
qp_attr.address.source_path_bits = 0;
qp_attr.address.use_grh = 0/*FALSE*/;
qp_attr.valid_fields |= IB_QP_ATTRIBUTE_PATH_MTU;
qp_attr.path_mtu = param_p->mtu;
qp_attr.valid_fields |= IB_QP_ATTRIBUTE_RNR_TIMEOUT;
qp_attr.rnr_timeout = IB_RNR_TIMER_002_56;
res = ib_qp_modify(param_p->ib_res.qp_p, &qp_attr);
Modify QP to RTS
=============
memset(&qp_attr, 0, sizeof(struct ib_qp_attribute));
qp_attr.valid_fields |= IB_QP_ATTRIBUTE_STATE;
qp_attr.state = IB_QP_STATE_RTS;
qp_attr.valid_fields |= IB_QP_ATTRIBUTE_SEND_PSN;
qp_attr.send_psn = START_PSN;
qp_attr.valid_fields |= IB_QP_ATTRIBUTE_INITIATOR_DEPTH;
qp_attr.initiator_depth = QP_INITIATOR_DEPTH;
qp_attr.valid_fields |= IB_QP_ATTRIBUTE_LOCAL_ACK_TIMEOUT;
qp_attr.local_ack_timeout = QP_LOCAL_ACK_TIMEOUT;
qp_attr.valid_fields |= IB_QP_ATTRIBUTE_RETRY_COUNT;
qp_attr.retry_count = QP_RETRY_COUNT;
qp_attr.valid_fields |= IB_QP_ATTRIBUTE_RNR_RETRY_COUNT;
qp_attr.rnr_retry_count = QP_RETRY_COUNT;
res = ib_qp_modify(param_p->ib_res.qp_p, &qp_attr);
Send
====
memset(&u_mr_p->send_param, 0, sizeof(struct ib_send_param));
u_mr_p->send_param.op = IB_OP_SEND;
ibsg.address = virt_to_phys(u_mr_p->user_buf);
ibsg.length = u_mr_p->user_buf_len;
ibsg.key = u_mr_p->lkey;
u_mr_p->send_param.work_request_id = (u64)u_mr_p;
u_mr_p->send_param.num_gather_entries = 1;
u_mr_p->send_param.gather_list = &ibsg;
u_mr_p->send_param.signaled = 1;
u_mr_p->send_param.dest_qpn = u_mr_p->dst_qpn;
PRINT_SEND_PARAM(&u_mr_p->send_param);
#ifdef KVAPITEST_DEBUG
show_qp_state(ib_res_p->qp_p);
#endif
res = ib_send(ib_res_p->qp_p,
&u_mr_p->send_param,
1,
&actual_wrn);
#ifdef KVAPITEST_DEBUG
MY_PRINT("AFTER ib_send\n");
show_qp_state(ib_res_p->qp_p);
#endif
Send CQ event handler
=================
void cq_send_handler(struct ib_cq *cq, struct ib_cq_entry *cq_entry, void
*arg)
{
if (cq == NULL) {
PRINT_ERR("NULL cq\n");
return;
}
if (cq_entry == NULL) {
PRINT_ERR("NULL entry of cq: 0x%p\n", cq);
return;
}
PRINT_TRACE("cq: 0x%p, cq_entry: 0x%p\n", cq, cq_entry);
PRINT_CQ_ENTRY(cq_entry);
spin_lock(&send_cq_lock.slock);
PRINT_TRACE("set send_cq_lock.state to GOT(%d)\n", CQ_STATE_GOT);
send_cq_lock.state = CQ_STATE_GOT;
spin_unlock(&send_cq_lock.slock);
wake_up(&send_cq_lock.waitq);
}
Debug infor
========
Mar 8 21:42:38 linux3 kernel: ==== QP attr to rtr ====
Mar 8 21:42:38 linux3 kernel: .qp_state: 2
Mar 8 21:42:38 linux3 kernel: .av.sl: 0
Mar 8 21:42:38 linux3 kernel: .av.grh_flag: 0
Mar 8 21:42:38 linux3 kernel: .av.dlid: 0x161
Mar 8 21:42:38 linux3 kernel: .av.static_rate: 2
Mar 8 21:42:38 linux3 kernel: .av.src_path_bits: 0
Mar 8 21:42:38 linux3 kernel: .path_mtu: 4
Mar 8 21:42:38 linux3 kernel: .dst_qp_num: 0x70405
Mar 8 21:42:38 linux3 kernel: ready_to_trans: move to RTS...
Mar 8 21:42:38 linux3 kernel: qp_move_to_rts: Modified QP to RTS
Mar 8 21:42:38 linux3 kernel: ==== QP attr to rts ====
Mar 8 21:42:38 linux3 kernel: .qp_state: 3
Mar 8 21:42:38 linux3 kernel: test_sr_lat: test 0, [Sender]
Mar 8 21:42:38 linux3 kernel: test_sr_lat: wait to start...
Mar 8 21:42:38 linux3 kernel: test_sr_lat: create out_mr
Mar 8 21:42:38 linux3 kernel: user_pmr_create: USE_PMR
Mar 8 21:42:38 linux3 kernel: user_pmr_create: user buffer
0x00000100271eb000, len: 4096
Mar 8 21:42:38 linux3 kernel: user_pmr_create: register physical memory
Mar 8 21:42:38 linux3 kernel: user_pmr_create: PD hndl:
0x0000010033b6f8c0
Mar 8 21:42:38 linux3 kernel: user_pmr_create: physical buf start
0x271eb000, size 0x1000
Mar 8 21:42:38 linux3 kernel: ==== user MR ====
Mar 8 21:42:38 linux3 kernel: buf start 0x00000100271eb000
Mar 8 21:42:38 linux3 kernel: buf len: 4096
Mar 8 21:42:38 linux3 kernel: IOVA: 0x0
Mar 8 21:42:38 linux3 kernel: IOVA offset: 0x0
Mar 8 21:42:38 linux3 kernel: l_key: 0x413f8b00
Mar 8 21:42:38 linux3 kernel: r_key: 0x413f8b00
Mar 8 21:42:38 linux3 kernel: PD: 0x0000010033b6f8c0
Mar 8 21:42:38 linux3 kernel: IB MR.device: 0x000001003bc896d0
Mar 8 21:42:38 linux3 kernel: IB MR.lkey: 0x413f8b00
Mar 8 21:42:38 linux3 kernel: IB MR.rkey: 0x413f8b00
Mar 8 21:42:38 linux3 kernel: IB MR.private: 0xa55a5a5a00000003
Mar 8 21:42:38 linux3 kernel: test_sr_lat: post_send
Mar 8 21:42:38 linux3 kernel: ==== send param ====
Mar 8 21:42:38 linux3 kernel: WR ID: 0x10029649c18
Mar 8 21:42:38 linux3 kernel: op type: IB_OP_SEND
Mar 8 21:42:38 linux3 kernel: remote addr: 0x0
Mar 8 21:42:38 linux3 kernel: r_key: 0x0
Mar 8 21:42:38 linux3 kernel: dest_qpn: 0x70405
Mar 8 21:42:38 linux3 kernel: dest_qkey: 0x0
Mar 8 21:42:38 linux3 kernel: static_rate: IB_STATIC_RATE_FULL
Mar 8 21:42:38 linux3 kernel: num_gather_entries: 1
Mar 8 21:42:38 linux3 kernel: entry[0]: addr 0x271eb000, len 0x1000, key
0x413f8b00
Mar 8 21:42:38 linux3 kernel: device_spec: x00000000000000000
Mar 8 21:42:38 linux3 kernel: signaled: 1
Mar 8 21:42:38 linux3 kernel: === QP[0x0000010029c96c90] ===
Mar 8 21:42:38 linux3 kernel: state: IB_QP_STATE_RTS
Mar 8 21:42:38 linux3 kernel: destination_qpn: 0x70405
Mar 8 21:42:38 linux3 kernel: ......
Mar 8 21:42:38 linux3 kernel: AFTER ib_send
Mar 8 21:42:38 linux3 kernel: cq_send_handler: cq: 0x0000010029c96948,
cq_entry: 0xffffffff805f0b58
Mar 8 21:42:38 linux3 kernel: ==== CQ entry ====
Mar 8 21:42:38 linux3 kernel: device spec. 0x000001003830c038
Mar 8 21:42:38 linux3 kernel: WR id: 0x10029649c18
Mar 8 21:42:38 linux3 kernel: Operation type: IB_COMPLETION_OP_RECEIVE
Mar 8 21:42:38 linux3 kernel: status: IB_COMPLETION_STATUS_INVALID
Mar 8 21:42:38 linux3 kernel: bytes transferred: 0xffffffff805f0ba8
Mar 8 21:42:38 linux3 kernel: pkey_index 0x206
Mar 8 21:42:38 linux3 kernel: sqpn: 0x805f0c08
Mar 8 21:42:38 linux3 kernel: slid: 0x0
Mar 8 21:42:38 linux3 kernel: ......
Mar 8 21:42:38 linux3 kernel: cq_send_handler: set send_cq_lock.state to
GOT(2)
Mar 8 21:42:38 linux3 kernel: === QP[0x0000010029c96c90] ===
Mar 8 21:42:38 linux3 kernel: state: IB_QP_STATE_ERROR
Mar 8 21:42:38 linux3 kernel: destination_qpn: 0x70405
Mar 8 21:42:38 linux3 kernel: ......
--
Ian Jiang
ianjiang.ict at gmail.com
Laboratory of Spatial Information Technology
Division of System Architecture
Institute of Computing Technology
Chinese Academy of Sciences
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.openfabrics.org/pipermail/general/attachments/20060308/cb4684a4/attachment.html>
More information about the general
mailing list