[ofa-general] Question about RDMA CM
jsquyres at cisco.com
Wed Sep 17 10:25:42 PDT 2008
On Sep 17, 2008, at 1:00 PM, Doug Ledford wrote:
> Does head of git not have this snippet in rdma_create_qp():
> if (ucma_is_ud_ps(id->ps))
> ret = ucma_init_ud_qp(id_priv, qp);
> ret = ucma_init_conn_qp(id_priv, qp);
> That's in the librdmacm code I have here, and that happens to be
> *before* the code sets id->qp = qp. And in that call chain,
> ucma_init_*_qp() both end up calling rdma_init_qp_attr() and it's here
> that we write to id->channel->fd (aka, the kernel rdmacm module
> of the verbs module) what we are doing.
And so it does -- I read that code quickly and my eye simply saw what
it wanted to see (ibv_query_qp()), not rdma_init_qp_attr(). I'll flog
the guy who originally wrote this. :-)
> An important factor being that you must use rdma_get_devices() and the
> ib contexts returned there from in your code when you are allocating
> protection domain contexts.
This is what I was afraid of. #@$%@#$%!!
We do not use rdma_get_devices() because our wireup scheme is both
modular (it may use RDMA CM or it may use something else) and separate
from the RDMA device discovery process. Specifically, the PD that we
have allocated is from the context we got back from ibv_open_device(),
I'll have to go tinker around to see if I can make that work.
More information about the general