[openib-general] ip over ib throughtput

[Hal Rosenstock]

On Tue, 2005-01-04 at 16:14, Ronald G. Minnich wrote:
> would you all recommend just cutting over to openib at this point? 

Depends on what you are looking to do with your cluster. Is it
production ? What ULPs are you looking to use ?

> I've got a 256-node cluster with hierarchical switches that has never really
> worked well with the vendor stacks, 

If you can live with just IPoIB then OpenIB might be good enough right
now.

> and the opensm has never run more than one day without segv; 
> is openib solid enough that I can run with it? Is
> the opensm in openib pretty solid? 

OpenIB opensm is based on Mellanox Gold 1.6.1. This port is pretty new
and has only been used in small clusters to date.

> Are there tools for tracing the fabric and looking for problems?

There are a number of diagnostic tools currently available:
ibstat/ibstatus, ibnetdiscover (which walks the fabric), smpdump,
sminfo, and mad_test currently. These are all pretty new. There was 
a writeup on these posted some time ago.

If IPoIB is sufficient, then a 2 phase approach to migration might be
possible:

1. Use OpenIB on end nodes but use some vendor's SM
2. Bring up a single or a couple of OpenIB end node(s) and use OpenIB
OpenSM shutting off the other SMs 

If that works, then you can convert over.

-- Hal