[openib-general] multicast

[Andrew Friedley]

Sean Hefty wrote:
> Andrew Friedley wrote:
> 
>> I'm trying to understand how the ibverbs multicast API works, but I'm 
>> not sure how multicast groups are created.  I understand that 
>> ibv_attach_mcast() and ibv_detach_mcast() are used to leave/join a 
>> particular multicast group, but IB architecture spec indicates a group 
>> must be created first.  How do I do this?
>>
>> Actually, is there any multicast sample code anywhere?
> 
> 
> Before you can call ibv_attach_mcast(), you must first join the 
> multicast group by sending a multicast join request to the SA.  The join 
> request may create the multicast group if it doesn't already exist, and 
> the request has the necessary information.
> 
> You can use the rdma_join_multicast() API in the rdma_cm to join a 
> multicast group and attach the group to a QP.  If the group has not 
> already been created, it will be created for you, with the group 
> parameters based on that used by ipoib.  The rdma_join_multicast() call 
> will invoke ibv_attach_mcast() for you once the join operation 
> completes.  There is sample code for joining multicast groups using this 
> API in userspace/librdmacm/examples/mckey.c.

Thanks for the explanation/pointer.  This looks like it will work, 
though I have a concern I'll discuss below.

> If you want more details on multicast group creation, you'll need to see 
> the spec, section 12.2.5.17.  Code to create / join a multicast group 
> can be found in the kernel cma.c module - see cma_join_ib_multicast().
> 
> You may want to see if rdma_join_multicast() meets your needs first.  If 
> it does not, please let me know what additional functionality you need.

I'm concerned about how rdma_cm abstracts HCAs.  It looks like I can use 
the src_addr argument to rdma_resolve_addr() to select which IP 
address/HCA (assuming one IP per HCA), but how can I enumerate the 
available HCAs?

This is important for a number of reasons - one, so that I can pass on 
the available IP addresses to MPI peers out of band.  It's also 
important to know which HCA's are available in the system, and to be 
able to select which HCA to use when connecting to a peer.  This allows 
us to implement things like load balancing and failover.

Matt Leininger suggested looking at the IB CM as an alternative, as it 
gives more low-level control.  Am I missing something, or does the IB CM 
not handle multicast like the RDMA CM?

Andrew