[ofa-general] Question about RDMA CM

[Jeff Squyres]

On Sep 17, 2008, at 1:00 PM, Doug Ledford wrote:

> Does head of git not have this snippet in rdma_create_qp():
>
>        if (ucma_is_ud_ps(id->ps))
>                ret = ucma_init_ud_qp(id_priv, qp);
>        else
>                ret = ucma_init_conn_qp(id_priv, qp);
>
> That's in the librdmacm code I have here, and that happens to be  
> called
> *before* the code sets id->qp = qp.  And in that call chain,
> ucma_init_*_qp() both end up calling rdma_init_qp_attr() and it's here
> that we write to id->channel->fd (aka, the kernel rdmacm module  
> instead
> of the verbs module) what we are doing.

And so it does -- I read that code quickly and my eye simply saw what  
it wanted to see (ibv_query_qp()), not rdma_init_qp_attr().  I'll flog  
the guy who originally wrote this.  :-)

> An important factor being that you must use rdma_get_devices() and the
> ib contexts returned there from in your code when you are allocating
> protection domain contexts.

This is what I was afraid of.  #@$%@#$%!!

We do not use rdma_get_devices() because our wireup scheme is both  
modular (it may use RDMA CM or it may use something else) and separate  
from the RDMA device discovery process.  Specifically, the PD that we  
have allocated is from the context we got back from ibv_open_device(),  
not rdma_get_devices().

I'll have to go tinker around to see if I can make that work.

-- 
Jeff Squyres
Cisco Systems