[openib-general] IPv6 and IPoIB scalability issue

[Hal Rosenstock]

On Fri, 2006-12-01 at 14:42, Todd Rimmer wrote:
> > From: Jason Gunthorpe [mailto:jgunthorpe at obsidianresearch.com]
> > Sent: Friday, December 01, 2006 1:37 PM
> > To: Hal Rosenstock
> > Cc: Todd Rimmer; openib-general at openib.org
> > Subject: Re: [openib-general] IPv6 and IPoIB scalability issue
> > 
> > 
> > Here is another thought.. Is there anything in the spec that says a
> > MGID must map to a MLID? If there is a single subscription why not
> > just do away with the MLID and return a unicast LID of the only
> > subscriber? That would probably solve 90% of the IPv6 issue Todd
> > pointed out. MGID compression would take care of the rest..
> > 
> 
> Summary of alternatives and trade-offs.  Lets assume a 2000 node cluster
> for analysis.
> 
> Option 1 use ALL Nodes Multicast
> Non standard for IPoIB
> small change to IPoIB code only
> Works with all existing SMs
> total of 5 MGIDs in cluster
> 5 Multicast subscriptions per node
> total of 10,000 multicast member records in SA for fabric

IMO if you want to go down this direction, the place to discuss it is on
the ipoib IETF mailing list. It is still active although dormant or very
sleepy.

> Option 2 compress MGID to MLID mapping
> Standard for IPoIB
> modification of SMs required, significant change

Significant in what respect ? The code changes are reasonably simple I
think. Is it from the perspective of upgrading SMs in the field for this
? I think it is a feature for better IPv6 support.

> configuration of MGID space in SM to consider for compression may be
> required
> total of 2005 MGIDs in cluster
> up to 2005 multicast subscriptions per node (sender only for Solicited
> Node initiators)

Does the node subscribe to every IPv6 SN group ?

> total of 2000*2005 (4,010,000) multicast member records in SA for fabric

This is based on the above (which I'm not sure about) and is the worst
theoretical case, not the practical case.

> Option 3 compress MGID to MLID mapping, use Unicast for Solicited Node
> MGIDs
> Standard for IPoIB
> not clear if standard for IB

More issues than this

> modification of SMs required, significant change

At first glance, there are more issues here than option 2 in terms of SM
(and client operation).

> configuration of MGID space in SM to consider for compression may be
> required
> configuration of MGID space in SM to use for unicast may be required
> total of 2005 MGIDs in cluster
> up to 2005 multicast subscriptions per node (sender only for Solicited
> Node initiators)
> total of 2000*2005 (4,010,000) multicast member records in SA for fabric
> 
> Hence thus far, option 2 is most standard, option 3 may be standard,
> option 1 has best scalability for SM.
> 
> It seems worth while to implement option 1 (which should be approx 10-20
> lines of code in IPoIB) and continue to pursue option 2 and 3 as SM
> features.  Then customers can choose which option works best for them.

I think before pursuing option 1 there needs to be a discussion with the
IETF WG involving the RFC authors (Vivek Kashyap, Jerry Chu).

-- Hal

> Todd Rimmer