[openib-general] [RFC] userspace IB SA support

[Andrew Friedley]

Sean Hefty wrote:
> Today, userspace support for SA related operations is limited to the libibmad
> interface, which supports sending and receiving MADs only.  I've been assigned
> with the task of exposing multicast and informinfo support to userspace.
> Specifically, the following functionality is needed:
> 
> 1. Join a multicast group - needs to use the ib_sa multicast capability.
> 2. Receive notification of multicast errors.
> 3. Leave a multicast group.
> 4. Register to receive SA events - needs to use the ib_sa notice capability.
> 5. Receive notification of events.
> 6. Deregister from SA events.
> 
> Are there any preferences for how this is added?

I'm a user; I can't comment on how this is implemented, but I'm very 
interested in what the API will look like.

I think I discussed this on-list before, but a big feature for me (Open 
MPI) is to be able to request and join an unused multicast group.  The 
exact address (or format, i.e. IP or MGID/MLID) of the group is not 
important as long as I can query it and pass it on out-of-band to peers. 
  This avoids hard-coding of multicast addresses and/or outside (MPI 
user) input, which could result in multiple MPI jobs unknowingly using 
the same multicast group.

The other big concern is control over which hca/port is joined to a 
multicast group -- IIRC this was a problem I had with the RDMA CM.  Open 
MPI specifically tends to open every available network interface 
(multiple IB ports, as well as say GM/IB/TCP together) for bandwidth 
aggregation and failover purposes, so controlling which interfaces data 
goes out over is important.

I hope this is useful information -- let me know if there is more input 
I can provide, or testing I can do.

Andrew