[openib-general] OpenSM (again)

[Hal Rosenstock]

On Mon, 2005-04-11 at 12:28, Roland Fehrenbacher wrote: 
> Hi,
> 
> I got gen2 opensm running fine now (there was a problem with a wrong
> include file), and managed to get IP running on a network of
> currently 40 machines (final size will be 144). Performance is pretty
> impressive (initial tests with a simple netpipe): I got a latency of
> 18microsec, and a maximum throughput of approx. 400MB/sec at packet
> size approx. 1MB which then levels of at about 340MB/s for larger
> packets.

That's all good to hear :-)

> One problem and two questions:
> 
> Problem: When I reboot all the 40 nodes (apart from the one the opensm
> is running), the network is non-functional (no pings go through, even
> though ports show status "Active") for quite a while (more than 10
> minutes) after all the nodes have come up. It then recovers without
> intervention. Is this normal? Single node reboots don't affect the
> network operation. osm Log file is appended.

Can you describe your topology ? Is it the following: the SM is
connected to a switch/or switches with the 40 nodes connected off these
switches ?

I'll respond to the log (and these questions) in a separate email
response.

> Question 1: Can I run opensm in a master slave configuration?

Yes. Others are doing this.

>  I noticed
> that there is a priority commandline option, but am not sure how to
> apply this.

SM election occurs per high priority low GUID. So if you don't care
which SM is the master than you don't need to do anything. If you want a
specific order (and it is not in GUID order) then you need to specify
priority.

> Question 2: I plan to run the gen1/Mellanox IBGD drivers on the
> compute nodes (need fast MPI), and gen2 on the control/storage nodes
> (need only IP) with gen2 opensm running on the control nodes. Is there
> any reason why this should not work reliably?

So basically this appears to be an interop question:
1. Will gen2 OpenSM support IBGD nodes ?
2. Will gen2 IPoIB interoperate with IBGD IPoIB ?
I haven't done this but know of no reasons this should not work. Perhaps
others can add to this.

-- Hal 
________________________________________________________________________