[openib-general] RFC userspace / MPI multicast support

[Matt Leininger]

On Thu, 2006-04-20 at 13:00 -0400, Hal Rosenstock wrote:
> On Thu, 2006-04-20 at 12:58, Sean Hefty wrote:
> > >On Wed, 2006-04-19 at 15:05, Sean Hefty wrote:
> > >> I'd like to get some feedback regarding the following approach to supporting
> > >> multicast groups in userspace, and in particular for MPI.  Based on side
> > >> conversations, I need to know if this approach would meet the needs of MPI
> > >> developers.
> > >>
> > >> To join / leave a multicast group,
> > >
> > >MC groups also need to be created and deleted as well. Creating and
> > >deleting the group are assumed under the covers (first joiner, last
> > >leaver) so the additional MC parameters for creation need to be
> > >available on all adds.
> > 
> > Creation / deletion would be automatic.  The creation parameters for
> > RDMA_PROTO_IP would use the same settings as the ipoib broadcast group.
> > 
> > >> /* Bind to multicast group. */
> > >> mcast_ip = 224.0.0.74.71; /* some fine mcast addr */
> > >
> > >How are the MGIDs formed from this IP address ? Is the same algorithm as
> > >IPoIB used ?
> > >
> > >Are the MGIDs constrained to use 0x401B in the signature part (and
> > >0x601B if this is extended to IPv6) ?
> > 
> > The MGIDs would be formed using the same algorithm as ipoib.  I hadn't decided
> > on whether to use the same signature, or a different one.  My initial thought
> > was to use a different signature, but I'm not sure that it's necessary.
> 
> Guess it comes down to how much control is needed over the entire MGID
> by MPI as well as whether they can share the IPoIB broadcast group
> characteristics for all their multicast groups.
> 
> Also, is IPoIB always setup when running MPI ?

  Not always.  For most of the older VAPI based stack we never turned on
IPoIB (or did and it didn't work).  I don't think we want to assume
IPoIB is always set up when MPI is running.

 - Matt

> 
> -- Hal
> 
> > >BTW, this example has too many bytes...
> > 
> > Just a typo...
> > 
> > >> ip_mreq.imr_multiaddr = mcast_ip.in_addr;
> > >> rdma_set_option(id, RDMA_PROTO_IP, IP_ADD_MEMBERSHIP, &ip_mreq,
> > >> 		    sizeof(ip_mreq));
> > >
> > >The API only supports ADD/DROP. It lacks support for JoinStates.
> > >(I don't think the IP semantics are rich enough for IB; this was
> > >previously pointed out in the context of IP routers quite a while ago).
> > 
> > Additional join states are IB specific, so would be handled by using the
> > RDMA_PROTO_IB option.  As an alternative, we could replace IP_ADD_MEMBERSHIP
> > with RDMA_ADD_FULL_MEMBER, RDMA_ADD_SEND_MEMBER, etc.
> > 
> > >> The multicast group information is created / managed by the rdma_cm.  The
> > >> rdma_cm defines the mgid, q_key, p_key, sl, flowlabel, tclass, and joinstate.
> > >> Except for mgid, these would most likely match the values used by the ipoib
> > >> broadcast group.  The mgid mapping would be similar to that used by ipoib.
> > >
> > >Does that limit the MGIDs to use IP signatures ?
> > 
> > Yes - unless the RDMA_PROTO_IB option were used.
> > 
> > - Sean
> 
> _______________________________________________
> openib-general mailing list
> openib-general at openib.org
> http://openib.org/mailman/listinfo/openib-general
> 
> To unsubscribe, please visit http://openib.org/mailman/listinfo/openib-general
>