[openib-general] openib-general Digest, Vol 29, Issue 149

[Bernard King-Smith]

"Eitan Zahavi" <eitan at mellanox.co.il> wrote on Mon, 20 
> Nov 2006 19:28:34 +0200 -----
> 
> To:
> 
> "Roland Dreier" <rdreier at cisco.com>
> 
> cc:
> 
> openib-general at openib.org
> 
> Subject:
> 
> Re: [openib-general] Scalable Monitoring - RFC
> 
> Hi Roland,
> 
> Actually using 24 port switches you can only build a cluster of 1728
> nodes 
> with 3 levels of switches if you insist on CBB ration of 1.
> 

For a 2 stage Fat Tree of 24 port switches I get a total of 13824 nodes 
using 1728 switches. 

K = radix of the switch
p = tree levels

Nodes = K^p
Switches = p*K^(p-1)

> Going to larger size cluster will require more levels.
> With 4 levels you can go up to 20736 nodes cluster.
> 
> So building a larger cluster (> 20736 nodes) will actually require 5
> levels of switching.
> 
> Assuming 40000 nodes (keeping 10K LIDs for the switches) with 5 levels
> of switching will 
> actually require 10*40000  = 400K ports. 

If you assume higher radix switches it cuts down on the number of ports 
and levels in the tree. There are already 96 port and maybe higher IB 
switches out there ( even though there are multiple chip hops in the 
switches ). If you go to 36 ports per switch you can get to 46656 nodes, 
3888 switches and 139968 total ports. This is probably managable and less 
than your 400K ports.

> 
> Should I assume from your mail that in your mind reading 300K or 400K
> ports is not an issue?
> 
> Eitan Zahavi



Bernie King-Smith 
IBM Corporation
Server Group
Cluster System Performance 
wombat2 at us.ibm.com    (845)433-8483
Tie. 293-8483 or wombat2 on NOTES 

"We are not responsible for the world we are born into, only for the world 
we leave when we die.
So we have to accept what has gone before us and work to change the only 
thing we can,
-- The Future." William Shatner
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.openfabrics.org/pipermail/general/attachments/20061121/5a9f8ba8/attachment.html>