[openib-general] [RFC] userspace IB SA support
afriedle at open-mpi.org
Tue Jan 9 13:23:06 PST 2007
Sean Hefty wrote:
> Today, userspace support for SA related operations is limited to the libibmad
> interface, which supports sending and receiving MADs only. I've been assigned
> with the task of exposing multicast and informinfo support to userspace.
> Specifically, the following functionality is needed:
> 1. Join a multicast group - needs to use the ib_sa multicast capability.
> 2. Receive notification of multicast errors.
> 3. Leave a multicast group.
> 4. Register to receive SA events - needs to use the ib_sa notice capability.
> 5. Receive notification of events.
> 6. Deregister from SA events.
> Are there any preferences for how this is added?
I'm a user; I can't comment on how this is implemented, but I'm very
interested in what the API will look like.
I think I discussed this on-list before, but a big feature for me (Open
MPI) is to be able to request and join an unused multicast group. The
exact address (or format, i.e. IP or MGID/MLID) of the group is not
important as long as I can query it and pass it on out-of-band to peers.
This avoids hard-coding of multicast addresses and/or outside (MPI
user) input, which could result in multiple MPI jobs unknowingly using
the same multicast group.
The other big concern is control over which hca/port is joined to a
multicast group -- IIRC this was a problem I had with the RDMA CM. Open
MPI specifically tends to open every available network interface
(multiple IB ports, as well as say GM/IB/TCP together) for bandwidth
aggregation and failover purposes, so controlling which interfaces data
goes out over is important.
I hope this is useful information -- let me know if there is more input
I can provide, or testing I can do.
More information about the general