[ofa-general] IB switches: managed or not?

[Ira Weiny]

On Tue, 6 Mar 2007 08:47:00 +1100
"Andrew Robbie (GMail)" <andrew.robbie at gmail.com> wrote:

> Hi,
> 
> Apologies if this is the wrong O.F. list to use.
> 
> I am building a small (~16) node cluster with an IB interconnect. I need to
> decide whether I will buy a cheaper, dumb switch and run OpenSM, or get a
> more expensive switch with a built in subnet manager. The largest this
> system would every grow is 32 nodes (two 24 port switches).
>
> Various vendors (integrators, not switch OEMs) have stated to me that
> managed switches are the go, and that OpenSM is (a) buggy, and (b) very time
> consuming to set up. But, a managed name brand switch seems to cost a lot
> more than a non-managed one using the Mellanox reference design kit
> (rebadged, but I suspect made by Flextronics...).
> 

We are using OSM on clusters from 5 nodes all the way up to 1100 nodes.
Granted there have been some issues.  However, the current 1.1 OFED version
seems to work just fine.  (We are pushing to the OFED 1.2 version for the 1100
node cluster at the moment.)

>
> My other query is about diagnostic software. With an ethernet switch it is
> pretty easy to fire up Ethereal (sorry Wireshark, but it is such a silly
> name) or Etherape and get a look at what is going on. If I buy a Cisco or
> Voltaire etc do they come with tools that let me get accurate
> representations of what is going on? Or are their tools really for large IB
> networks?
> 

Their are some issues with diags.  However, I don't know of any product from
any vendor which captures on the wire packets like Wireshark.  (But I could be
wrong and would love to know about it if it was out there.)

Checkout the openib-diags and ibutils packages for diags which are available.

YMMV,
Ira Weiny
weiny2 at llnl.gov