[ewg] rdma_accept and non-rdma QPs

[Dries Kimpe]

I've done some additional investigation and found the following:

int rdma_accept (called on the passive side with a valid qp_num set in
rdma_conn_param), on line 1535 in current git, there is the following:

[...]
	if (!ucma_is_ud_qp(id->qp_type)) {
		ret = ucma_modify_qp_rtr(id, id_priv->responder_resources);
		if (ret)
			return ret;

		ret = ucma_modify_qp_rts(id, id_priv->initiator_depth);
		if (ret)
			return ret;
	}
[...]



In this case, id->qp_type is IBV_QPT_RC, so ucma_modify_qp_rtr is called.

Unfortunately, ucma_modify_qp_rtr() contains the following on line:


static int ucma_modify_qp_rtr(struct rdma_cm_id *id, uint8_t resp_res)
{
	struct ibv_qp_attr qp_attr;
	int qp_attr_mask, ret;

	if (!id->qp)
		return ERR(EINVAL);

[...]


Since (as explained in my other email), rdma_create_qp() has not been called,
id->qp is NULL, causing ucma_modify_qp_rtr() and subsequently rdma_accept() failure.

Other than directly setting id->qp, I don't see how rdma_accept() can be used
without calling rdma_create_qp() first.

Is the conclusion that rdmacm doesn't support outside QP creation/destruction
(at least not for the passive side) and that the documentation (rdma_accept
manpage) is incorrect?

  Thanks,
  Dries



* 'dries at kimpe.be' (dries.kimpe at gmail.com) <dries.kimpe at gmail.com> [2014-08-20 12:59:47]:

> Hi,
> 
> I've been having some issues in using RDMA CM with QPs created and owned
> by the applications, i.e. created with ibv_create_qp as opposed to
> rdma_create_qp.
> 
> The man pages for both rdma_accept and rdma_connect hint that this is
> possible.
> 
> In my application, whem I'm using rdma_create_qp, everything works.
> If I change to using ibv_create_qp (but everything else the same:
> qp_init_attr, PD, ... -- only change is rdma_create_qp -> ibv_create_qp)
> the call to rdma_accept fails with 'invalid argument'.
> 
> [I do correctly specify the QP number in the accept parameters and
> connect parameters in both cases. The documentation states that rdmacm
> ignores qp_num in case rdma_create_qp was called]
> 
> Some more details:
> - RC connection
> - Ofed version doesn't seem to matter (tried up to 2.1).
> - Using my own protection domain (not created by RDMA CM) but derived
>   from the same context to which the ID is bound.
> - Using outside (i.e. not RDMA CM created) completion queue
> 
> Things I've verified/tried:
> 
> - verify context for RDMA cm id and QP created by ibv_create_qp is the
>   same.
> - verify protection domains belong to the same context.
> - transition QP to INIT on passive side before calling rdma_accept.
>   (I checked the rdma cm source code and noticed that rdma_create_qp
>   also does this)
> - Checked the kernel code to see where an invaliad argument parameter
>   could be returned but nothing stood out.
> - Verified sequence of calls on both active and passive side
>   as described in rdma_cm manpage.
> 
> I wasn't able to find any example code using rdma cm but not
> rdma_create_qp.
> It would be very helpful if somebody could list the differences or point
> me to some code.
> 
>    Thanks,
>    Dries
>