[openib-general] FW: [PATCH] [RFC] librdmacm: expose device list to users

[Sean Hefty]

Sean Hefty wrote:
>>One solution (suggested by Matt Leininger) would be for the IB stack
>>(CM?) to hand out multicast addresses.  I'm thinking it would be useful
>>to come up with a header file for a solution before implementing
>>anything.  An 'ideal' API might look something like this:

The IB SA does something like this.  If a user creates a multicast group with an 
MGID of 0, the SA will assign an MGID to the group.  That MGID then somehow 
needs to be distributed to everyone wanting to join that group.

I'm not aware of any equivalent functionality for IP multicast groups, so I'm 
not sure if it makes sense to try to provide this functionality through the RDMA 
CM or expose a raw IB multicast interface.

I've been trying to come up with a raw SA interface for userspace, but a generic 
one that supports RMPP isn't ending up much simpler than the raw MAD interface. 
  And looking at the multicast module in the kernel, I just noticed that it 
doesn't handle the case if the user tries to join a group with an MGID of 0.

>>I would then use ObtainMulticastAddress, then pass the returned address
>>via OOB to all the peers I want to be in that multicast group.  All the
>>peers would then use JoinMulticastGroup.

In order to use what's there, is there any way that the processes can create 
unique addresses to use?  Maybe map the server port numbers into the address?

>>ReturnMulticastAddress could require that all calls to
>>JoinMulticastGroup on the provided address have been paired with
>>matching LeaveMulticastGroup calls.  What would be better though, is if
>>the group were asynchronously released when the last peer in the group
>>calls LeaveMulticastGroup.  This would avoid the need for explicitly
>>fencing between all the LeaveMulticastGroup calls and one peer calling
>>ReturnMulticastAddress.

Your interface looks good.  I just need to think more about the details of what 
approach to use.  I want to find the balance between being easy to use, but 
provide the necessary capabilities.

I like the idea that the group is automatically deleted when the last user 
leaves, since this matches with the IB implementation.  We might also be able to 
remove the ObtainMulticastAddress, by letting JoinMulticastGroup take a wildcard 
address.  Can you take a look at the kernel interface in ib_multicast.h, and let 
me know if exposing that to userspace would work for you?

>>The idea behind ObtainMulticastAddress is that I don't care what the
>>particular address is - just that I have an address, and nobody else is
>>using it.  The address can take any form - does not have to be IP.
>>
>>It's important that everything be non-blocking.  If needed, operations
>>can be done asynchronously (ie completion signalled via an event queue).
>>
>>Is this sort of approach even feasible?

I'm pretty sure that the approach is feasible.  It's just that the code there 
now wasn't written to support a call like ObtainMulticastAddress.

- Sean