[openib-general] userspace CM API for per device handling

[Sean Hefty]

Sean Hefty wrote:
> For the userspace portion, I'm still trying to decide what the correct 
> API should be.  I'd like to avoid apps from having to call something 
> like ib_cm_get_devices(), which would mirror the verbs call.  I was 
> thinking of having ib_cm_create_id() still take a struct ibv_context* as 
> input, opening the corresponding CM node, and managing that internally.  
> Thoughts?

To further define this:

The kernel ucm module creates one CM device per physical device, somewhat 
mirroring the work done by uverbs.  (E.g. infiniband_cm/ucm0 references the same 
device as infiniband_verbs/uverbs0).

All CM devices are opened internally by the userspace CM and can be mapped to a 
corresponding ibv_device using a GUID.  This works okay, except for the current 
call:

ib_cm_get_event(**event);

which can now map to multiple fd's.  Some possible solutions are:

1. Add calls similar to ib_cm_get_devices() and ib_cm_open_device(), making the 
CM devices explicit to the user.  ib_cm_get_event() would take a CM device as 
input.  This requires that users manage not only a list of HCAs, but also a 
mirror list of CM devices.

2. Change ib_cm_get_event(struct ibv_context *device_context, **event).  The 
mapping from the device to the corresponding CM fd is performed internally, but 
requires a search based on the GUID.

3. Same as #2, but store the CM fd in the ibv_context to avoid the search.  This 
breaks the encapsulation between the CM and verbs.

4. Have ib_cm_get_event() operate across all CM devices.

- Sean