[openib-general] Scalable Monitoring - RFC

[Eitan Zahavi]

Hi Roland,

Actually using 24 port switches you can only build a cluster of 1728
nodes 
with 3 levels of switches if you insist on CBB ration of 1.

Going to larger size cluster will require more levels.
With 4 levels you can go up to 20736 nodes cluster.

So building a larger cluster (> 20736 nodes) will actually require 5
levels of switching.

Assuming 40000 nodes (keeping 10K LIDs for the switches) with 5 levels
of switching will 
actually require 10*40000  = 400K ports. 

Should I assume from your mail that in your mind reading 300K or 400K
ports is not an issue?

Eitan Zahavi

> -----Original Message-----
> From: Roland Dreier [mailto:rdreier at cisco.com]
> Sent: Monday, November 20, 2006 7:11 PM
> To: Eitan Zahavi
> Cc: openib-general at openib.org
> Subject: Re: [openib-general] Scalable Monitoring - RFC
> 
>  > * scale up to 48k nodes
>  > *  16 ports which gets to about 1,000,000 ports.
>  >     (16 ports per device is average for 32 ports for switch and 1
for
>  > HCA)
> 
> This math seems off -- even with ~48K HCAs and a 3-tier fabric, you
only end
> up with 6 * 48K ports, which is less than 300K.  Why would anyone
sensible
> build a fabric with 1M ports?
>