[openib-general] userspace CM API for per device handling

[Arlin Davis]

Sean Hefty wrote:

> Sean Hefty wrote:
>
>> For the userspace portion, I'm still trying to decide what the 
>> correct API should be.  I'd like to avoid apps from having to call 
>> something like ib_cm_get_devices(), which would mirror the verbs 
>> call.  I was thinking of having ib_cm_create_id() still take a struct 
>> ibv_context* as input, opening the corresponding CM node, and 
>> managing that internally.  Thoughts?
>
>
> To further define this:
>
> The kernel ucm module creates one CM device per physical device, 
> somewhat mirroring the work done by uverbs.  (E.g. infiniband_cm/ucm0 
> references the same device as infiniband_verbs/uverbs0).
>
> All CM devices are opened internally by the userspace CM and can be 
> mapped to a corresponding ibv_device using a GUID.  This works okay, 
> except for the current call:
>
> ib_cm_get_event(**event);
>
> which can now map to multiple fd's.  Some possible solutions are:
>
> 1. Add calls similar to ib_cm_get_devices() and ib_cm_open_device(), 
> making the CM devices explicit to the user.  ib_cm_get_event() would 
> take a CM device as input.  This requires that users manage not only a 
> list of HCAs, but also a mirror list of CM devices.
>
> 2. Change ib_cm_get_event(struct ibv_context *device_context, 
> **event).  The mapping from the device to the corresponding CM fd is 
> performed internally, but requires a search based on the GUID.
>
> 3. Same as #2, but store the CM fd in the ibv_context to avoid the 
> search.  This breaks the encapsulation between the CM and verbs.
>
> 4. Have ib_cm_get_event() operate across all CM devices.

User events are processed (poll/select) with FD's so can we just use the 
FD to get events? This would give the user a direct mapping back to the 
correct device based on the poll or select results.
 
Something like...

5. ib_cm_get_fd( struct ibv_context *device_context)  and  
ib_cm_get_event(int fd, **event).

and maybe consider a verbs change like....

 ibv_get_async_event(int fd, struct ibv_async_event *event);
 ibv_get_cq_event(int fd, struct ibv_cq **cq, void **cq_context);

comments?

-arlin


>
> - Sean
> _______________________________________________
> openib-general mailing list
> openib-general at openib.org
> http://openib.org/mailman/listinfo/openib-general
>
> To unsubscribe, please visit 
> http://openib.org/mailman/listinfo/openib-general
>