[ofa-general] Sharing CQs across multiple connections with librdmacm

[Adrien Guillon]

Hey...

I want to allocate a send CQ and receive CQ for each HCA, to be shared
by all connections using that HCA.  This seems possible according to
the Infiniband standard, but I can't see how to do this in practice
using the ibverbs.  I'm using librdmacm for the actual connections.

My problem is that ibv_create_cq() takes an ibv_context* as an
argument.  With librdmacm, I can get this through rdma_cm_id->verbs.
However it looks like ibv_context objects are associated with
particular connections, not particular HCAs which is what is confusing
me.  It seems to me that ibv_create_cq() should be associated with a
handle to the HCA itself, as the "Infiniband Network Architecture"
book says.  Ideally I would allocate a data structure with HCA
specific data for each device (e.g. PD, CQ, etc.) and use the kernel
name (e.g. mctha0) to lookup the HCA specific data.  That way I can
check the ibv_context to see if I can use existing specific data or
create new.  Whew.

So the question is... how do I do this given that ibv_create_cq()
takes ibv_context* as an argument?  Will it internally just use the
ibv_context to look up the device?  What happens when that ibv_context
is destroyed, but I want the PD to remain open (e.g. connection
destroyed, others still open)?  Can I create CQs and PDs using
ibv_device at initialization time, so I don't have to wait for the
first connection on each device to come in?

Thanks!

AJ