[openib-general] RFC userspace CMA

[Caitlin Bestler]

> -----Original Message-----
> From: openib-general-bounces at openib.org 
> [mailto:openib-general-bounces at openib.org] On Behalf Of Sean Hefty
> Sent: Monday, October 24, 2005 5:06 PM
> To: openib
> Subject: [openib-general] RFC userspace CMA
> 
> I'm soliciting any comments that anyone might have on the 
> general design for the userspace CMA before I get too far 
> into the implementation.
> 
> - The API will match the kernel API for the most part.  The 
> exception is that event handling will match other userspace 
> libraries (get/ack event).
> 
> - There will be a single CMA device exported through 
> /sys/class/infiniband.
> 
> - The kernel CMA will be modified to remove the requirement 
> to use rdma_create_qp().  Users that want to allocate and 
> manage their own QP states will be able to specify QP 
> attributes (qpn, qp_type, srq) through the rdma_conn_param structure.
>

Why?

Every CM interface I've dealt with has had the Consumer create
and configure the QP on each end. On the active side the QP is
supplied with the connect request. On the passive side it is
supplied with the accept. State modifications and other
configuration changes were done by the CM based on the Consumer
having passed in the handle.
 
> - The kernel CMA will expose a new call, rdma_init_qp_attr() 
> to initialize QP attributes used to modify the state of the 
> QP.  The call will be similar to the infiniband CM routine.  
> Use of this call is optional.  The CMA will automatically 
> transition QPs created by rdma_create_qp().
> 


> - The uCMA will open devices for users and return them the 
> device context with related events.  The uCMA will close the 
> device if there are no rdma_cma_id's associated with it.
> 
> - To support device add, the uCMA will need a new verb's call: 
> ibv_open_device_by_guid().  If a connection request occurs 
> for a device that is not yet known by the uCMA, it will open 
> the device.
> 

Why does the uCMA need to open HCAs? Why does it have to be
anything other than a front-end to the kCMA?

I can see where a user-mode daemon might get a connection
request that could only be answered with a QP on an device
that it had not previolusly opened, and that opening that
device based on information in the Connection Request would
be useful -- but that still doesn't have the uCMA opening
the device. More importantly, the problem can be solved 
just as easily by having the listener listen only on rdma
devices that it has already opened.