[openib-general] RFC userspace / MPI multicast support

[Hal Rosenstock]

On Thu, 2006-04-20 at 12:58, Sean Hefty wrote:
> >On Wed, 2006-04-19 at 15:05, Sean Hefty wrote:
> >> I'd like to get some feedback regarding the following approach to supporting
> >> multicast groups in userspace, and in particular for MPI.  Based on side
> >> conversations, I need to know if this approach would meet the needs of MPI
> >> developers.
> >>
> >> To join / leave a multicast group,
> >
> >MC groups also need to be created and deleted as well. Creating and
> >deleting the group are assumed under the covers (first joiner, last
> >leaver) so the additional MC parameters for creation need to be
> >available on all adds.
> 
> Creation / deletion would be automatic.  The creation parameters for
> RDMA_PROTO_IP would use the same settings as the ipoib broadcast group.
> 
> >> /* Bind to multicast group. */
> >> mcast_ip = 224.0.0.74.71; /* some fine mcast addr */
> >
> >How are the MGIDs formed from this IP address ? Is the same algorithm as
> >IPoIB used ?
> >
> >Are the MGIDs constrained to use 0x401B in the signature part (and
> >0x601B if this is extended to IPv6) ?
> 
> The MGIDs would be formed using the same algorithm as ipoib.  I hadn't decided
> on whether to use the same signature, or a different one.  My initial thought
> was to use a different signature, but I'm not sure that it's necessary.

Guess it comes down to how much control is needed over the entire MGID
by MPI as well as whether they can share the IPoIB broadcast group
characteristics for all their multicast groups.

Also, is IPoIB always setup when running MPI ?

-- Hal

> >BTW, this example has too many bytes...
> 
> Just a typo...
> 
> >> ip_mreq.imr_multiaddr = mcast_ip.in_addr;
> >> rdma_set_option(id, RDMA_PROTO_IP, IP_ADD_MEMBERSHIP, &ip_mreq,
> >> 		    sizeof(ip_mreq));
> >
> >The API only supports ADD/DROP. It lacks support for JoinStates.
> >(I don't think the IP semantics are rich enough for IB; this was
> >previously pointed out in the context of IP routers quite a while ago).
> 
> Additional join states are IB specific, so would be handled by using the
> RDMA_PROTO_IB option.  As an alternative, we could replace IP_ADD_MEMBERSHIP
> with RDMA_ADD_FULL_MEMBER, RDMA_ADD_SEND_MEMBER, etc.
> 
> >> The multicast group information is created / managed by the rdma_cm.  The
> >> rdma_cm defines the mgid, q_key, p_key, sl, flowlabel, tclass, and joinstate.
> >> Except for mgid, these would most likely match the values used by the ipoib
> >> broadcast group.  The mgid mapping would be similar to that used by ipoib.
> >
> >Does that limit the MGIDs to use IP signatures ?
> 
> Yes - unless the RDMA_PROTO_IB option were used.
> 
> - Sean