[openib-general] IPv6 and IPoIB scalability issue

[Todd Rimmer]

> From: Jason Gunthorpe [mailto:jgunthorpe at obsidianresearch.com]
> Sent: Friday, December 01, 2006 1:37 PM
> To: Hal Rosenstock
> Cc: Todd Rimmer; openib-general at openib.org
> Subject: Re: [openib-general] IPv6 and IPoIB scalability issue
> 
> 
> Here is another thought.. Is there anything in the spec that says a
> MGID must map to a MLID? If there is a single subscription why not
> just do away with the MLID and return a unicast LID of the only
> subscriber? That would probably solve 90% of the IPv6 issue Todd
> pointed out. MGID compression would take care of the rest..
> 

Summary of alternatives and trade-offs.  Lets assume a 2000 node cluster
for analysis.

Option 1 use ALL Nodes Multicast
Non standard for IPoIB
small change to IPoIB code only
Works with all existing SMs
total of 5 MGIDs in cluster
5 Multicast subscriptions per node
total of 10,000 multicast member records in SA for fabric

Option 2 compress MGID to MLID mapping
Standard for IPoIB
modification of SMs required, significant change
configuration of MGID space in SM to consider for compression may be
required
total of 2005 MGIDs in cluster
up to 2005 multicast subscriptions per node (sender only for Solicited
Node initiators)
total of 2000*2005 (4,010,000) multicast member records in SA for fabric

Option 3 compress MGID to MLID mapping, use Unicast for Solicited Node
MGIDs
Standard for IPoIB
not clear if standard for IB
modification of SMs required, significant change
configuration of MGID space in SM to consider for compression may be
required
configuration of MGID space in SM to use for unicast may be required
total of 2005 MGIDs in cluster
up to 2005 multicast subscriptions per node (sender only for Solicited
Node initiators)
total of 2000*2005 (4,010,000) multicast member records in SA for fabric

Hence thus far, option 2 is most standard, option 3 may be standard,
option 1 has best scalability for SM.

It seems worth while to implement option 1 (which should be approx 10-20
lines of code in IPoIB) and continue to pursue option 2 and 3 as SM
features.  Then customers can choose which option works best for them.

Todd Rimmer