[ofa-general] librdmacm use problem -- multi qps per cq or a qp per cq?

[Roland Dreier]

 >  I want to use mutli qps referred to a cq, but in librdmacm.
 > First, i create a cm-event-channel using rdma_create_event_channel,
 > which is referred to a context. Then i create a pd, which is also
 > referred to the same context, then i create comp-channel and cq using
 > ibv_create_comp_channel and ibv_create_cq.
 > i create a cm_id for every connection, which as the first param for
 > rdma_create_qp. i create qp per connection using rdma_create_qp, int
 > rdma_create_qp(struct rdma_cm_id *id, struct ibv_pd *pd, struct
 > ibv_qp_init_attr *qp_init_attr). But i cannot identify the param pd,
 > when i put the context-pd here, i would get error-can't create qp,
 > because rdma_create_qp need to check the cm_id->pd and context-pd, they
 > are different, when i create a new pd for qp, i can create qp, but can't
 > transefer data through qp.
 > Alternative, i can create a qp using ibv_create_qp and do sth., but i
 > can't know how to contact qp with cm_id which has created? And i can't
 > use mechanism provided by librdmacm?

I don't really understand your problem.  The sequence that works fine
for me is:

	cm_channel = rdma_create_event_channel();
	rdma_create_id(cm_channel, &cm_id, NULL, RDMA_PS_TCP);
	rdma_resolve_addr(cm_id, NULL, addr, <timeout>);
	rdma_resolve_route(cm_id, <timeout>);
	pd = ibv_alloc_pd(cm_id->verbs);
	comp_chan = ibv_create_comp_channel(cm_id->verbs);
	cq = ibv_create_cq(cm_id->verbs, 2, NULL, comp_chan, 0);
	qp_attr.send_cq		 = cq;
	qp_attr.recv_cq		 = cq;
	rdma_create_qp(cm_id, pd, &qp_attr);

librdmacm internally will use the same libibverbs context for all
cm_id structures attached to the same device, so you should be able to
use the same CQ for multiple QPs.

If you post the code you are trying to run and describe how it fails,
we may be able to give better advice.

 - R.