[ofa-general] OpenSM?

Tue May 27 08:15:14 PDT 2008

We have a 400 node IB cluster.    We are running an embedded SM in  
failover mode on our TS270/Cisco7008 core switches.    Lately we have  
been seeing problems with LID assignment when rebooting nodes (see log  
messages below).   It is also taking far too long for LIDS to be  
assigned as it takes  on the order of minutes for the ports to  
transition to "ACTIVE".

This seems like a bug to us and we are considering switching to  
OpenSM  on a host.   I'm wondering about experience with running  
OpenSM for medium to large clusters (Fat Tree) and what resources  
(memory/cpu) we should plan on for the host node.

Thanks,

Charlie Taylor
UF HPC Center

May 27 14:14:10 topspin-270sc ib_sm.x[803]: [ib_sm_sweep.c:1914]:
**********************  NEW SWEEP ********************
May 27 14:14:10 topspin-270sc ib_sm.x[803]: [ib_sm_sweep.c:1320]:  
Rediscover
the subnet
May 27 14:14:13 topspin-270sc ib_sm.x[803]: [INFO]: Generate SM  
OUT_OF_SERVICE
trap for GID=fe:80:00:00:00:00:00:00:00:02:c9:02:00:21:4b:59
May 27 14:14:13 topspin-270sc ib_sm.x[803]: [ib_sm_sweep.c:256]: An  
existing IB
node GUID 00:02:c9:02:00:21:4b:59 LID 194 was removed
May 27 14:14:14 topspin-270sc ib_sm.x[803]: [INFO]: Generate SM  
DELETE_MC_GROUP
trap for GID=ff:12:60:1b:ff:ff:00:00:00:00:00:01:ff:21:4b:59
May 27 14:14:14 topspin-270sc ib_sm.x[803]: [ib_sm_sweep.c:1503]:  
Topology
changed
May 27 14:14:14 topspin-270sc ib_sm.x[803]: [INFO]: Configuration  
caused by
discovering removed ports
May 27 14:16:26 topspin-270sc ib_sm.x[803]: [ib_sm_sweep.c:1875]:  
async events
require sweep
May 27 14:16:26 topspin-270sc ib_sm.x[803]: [ib_sm_sweep.c:1914]:
**********************  NEW SWEEP ********************
May 27 14:16:26 topspin-270sc ib_sm.x[803]: [ib_sm_sweep.c:1320]:  
Rediscover
the subnet
May 27 14:16:28 topspin-270sc ib_sm.x[812]: [ib_sm_discovery.c:1009]: no
routing required for port guid 00:02:c9:02:00:21:4b:59, lid 194
May 27 14:16:30 topspin-270sc ib_sm.x[803]: [ib_sm_sweep.c:1503]:  
Topology
changed
May 27 14:16:30 topspin-270sc ib_sm.x[803]: [INFO]: Configuration  
caused by
discovering new ports
May 27 14:16:30 topspin-270sc ib_sm.x[803]: [INFO]: Configuration  
caused by
multicast membership change
May 27 14:16:30 topspin-270sc ib_sm.x[812]: [ib_sm_assign.c:588]:  
Force port to
go down due to LID conflict, node - GUID=00:02:c9:02:00:21:4b:58, port=1
May 27 14:18:42 topspin-270sc ib_sm.x[819]: [ib_sm_bringup.c:562]:  
Program port
state, node=00:02:c9:02:00:21:4b:58, port= 16, current state 2, neighbor
node=00:02:c9:02:00:21:4b:58, port= 1, current state 2
May 27 14:18:42 topspin-270sc ib_sm.x[819]: [ib_sm_bringup.c:733]:  
Failed to
negotiate MTU, op_vl for node=00:02:c9:02:00:21:4b:58, port= 1, mad  
status 0x1c
May 27 14:18:42 topspin-270sc ib_sm.x[803]: [INFO]: Generate SM  
IN_SERVICE trap
for GID=fe:80:00:00:00:00:00:00:00:02:c9:02:00:21:4b:59
May 27 14:18:42 topspin-270sc ib_sm.x[803]: [ib_sm_sweep.c:144]: A new  
IB node
00:02:c9:02:00:21:4b:59 was discovered and assigned LID 0
May 27 14:18:43 topspin-270sc ib_sm.x[803]: [ib_sm_sweep.c:1875]:  
async events
require sweep
May 27 14:18:43 topspin-270sc ib_sm.x[803]: [ib_sm_sweep.c:1914]:
**********************  NEW SWEEP ********************
May 27 14:18:43 topspin-270sc ib_sm.x[803]: [ib_sm_sweep.c:1320]:  
Rediscover
the subnet
May 27 14:18:46 topspin-270sc ib_sm.x[803]: [ib_sm_sweep.c:1516]: No  
topology
change
May 27 14:18:46 topspin-270sc ib_sm.x[803]: [INFO]: Configuration  
caused by
previous GET/SET operation failures
May 27 14:18:46 topspin-270sc ib_sm.x[816]: [ib_sm_assign.c:545]:  
Reassigning
LID, node - GUID=00:02:c9:02:00:21:4b:58, port=1, new LID=411, curr  
LID=0
May 27 14:18:46 topspin-270sc ib_sm.x[816]: [ib_sm_assign.c:588]:  
Force port to
go down due to LID conflict, node - GUID=00:02:c9:02:00:21:4b:58, port=1
May 27 14:18:46 topspin-270sc ib_sm.x[816]: [ib_sm_assign.c:635]:  
Clean up SA
resources for port forced down due to LID conflict, node -
GUID=00:02:c9:02:00:21:4b:58, port=1
May 27 14:18:47 topspin-270sc ib_sm.x[803]: [ib_sm_assign.c:667]:  
cleaning DB
for guid 00:02:c9:02:00:21:4b:59, lid 194
May 27 14:18:47 topspin-270sc ib_sm.x[803]: [ib_sm_routing.c:2936]:
_ib_smAllocSubnet: initRate= 4
May 27 14:18:47 topspin-270sc last message repeated 23 times
May 27 14:18:47 topspin-270sc ib_sm.x[803]: [INFO]: Different capacity  
links
detected in the network
May 27 14:21:01 topspin-270sc ib_sm.x[820]: [ib_sm_bringup.c:516]:  
Active
port(s) now in INIT state node=00:02:c9:02:00:21:4b:58, port=16,  
state=2,
neighbor node=00:02:c9:02:00:21:4b:58, port=1, state=2
May 27 14:21:01 topspin-270sc ib_sm.x[803]: [ib_sm_sweep.c:1875]:  
async events
require sweep
May 27 14:21:01 topspin-270sc ib_sm.x[803]: [ib_sm_sweep.c:1914]:
**********************  NEW SWEEP ********************
May 27 14:21:01 topspin-270sc ib_sm.x[803]: [ib_sm_sweep.c:1320]:  
Rediscover
the subnet
May 27 14:21:05 topspin-270sc ib_sm.x[803]: [ib_sm_sweep.c:1516]: No  
topology
change
May 27 14:21:05 topspin-270sc ib_sm.x[803]: [ib_sm_sweep.c:525]: IB node
00:06:6a:00:d9:00:04:5d port 16 is INIT state
May 27 14:21:05 topspin-270sc ib_sm.x[803]: [INFO]: Configuration  
caused by
some ports in INIT state
May 27 14:21:05 topspin-270sc ib_sm.x[803]: [INFO]: Configuration  
caused by
previous GET/SET operation failures
May 27 14:21:05 topspin-270sc ib_sm.x[803]: [ib_sm_routing.c:2936]:
_ib_smAllocSubnet: initRate= 4
May 27 14:21:05 topspin-270sc last message repeated 23 times
May 27 14:21:05 topspin-270sc ib_sm.x[803]: [INFO]: Different capacity  
links
detected in the network
May 27 14:23:19 topspin-270sc ib_sm.x[817]: [ib_sm_bringup.c:562]:  
Program port
state, node=00:02:c9:02:00:21:4b:58, port= 16, current state 2, neighbor
node=00:02:c9:02:00:21:4b:58, port= 1, current state 2
May 27 14:23:24 topspin-270sc ib_sm.x[823]: [INFO]: Generate SM  
CREATE_MC_GROUP
trap for GID=ff:12:60:1b:ff:ff:00:00:00:00:00:01:ff:21:4b:59
May 27 14:23:24 topspin-270sc ib_sm.x[803]: [ib_sm_sweep.c:1875]:  
async events
require sweep
May 27 14:23:24 topspin-270sc ib_sm.x[803]: [ib_sm_sweep.c:1914]:
**********************  NEW SWEEP ********************
May 27 14:23:26 topspin-270sc ib_sm.x[803]: [ib_sm_sweep.c:1516]: No  
topology
change
May 27 14:23:26 topspin-270sc ib_sm.x[803]: [INFO]: Configuration  
caused by
multicast membership change
May 27 14:23:33 topspin-270sc ib_sm.x[826]: [INFO]: Standby SM guid
00:05:ad:00:00:02:3c:60, is no longer synchronized with Master SM
May 27 14:25:39 topspin-270sc ib_sm.x[826]: [INFO]: Initialize a  
backup session
with Standby SM guid 00:05:ad:00:00:02:3c:60
May 27 14:25:39 topspin-270sc ib_sm.x[803]: [ib_sm_sweep.c:1875]:  
async events
require sweep
May 27 14:25:39 topspin-270sc ib_sm.x[803]: [ib_sm_sweep.c:1914]:
**********************  NEW SWEEP ********************
May 27 14:25:39 topspin-270sc ib_sm.x[826]: [INFO]: Standby SM guid
00:05:ad:00:00:02:3c:60, started synchronizing with Master SM
May 27 14:25:42 topspin-270sc ib_sm.x[803]: [ib_sm_sweep.c:1516]: No  
topology
change
May 27 14:25:42 topspin-270sc ib_sm.x[803]: [INFO]: Configuration  
caused by
multicast membership change
May 27 14:25:43 topspin-270sc ib_sm.x[826]: [INFO]: Master SM DB  
synchronized
with Standby SM guid 00:05:ad:00:00:02:3c:60
May 27 14:25:43 topspin-270sc ib_sm.x[826]: [INFO]: Master SM DB  
synchronized
with all designated backup SMs
May 27 14:28:04 topspin-270sc ib_sm.x[803]: [ib_sm_sweep.c:1914]:
**********************  NEW SWEEP ********************
May 27 14:28:06 topspin-270sc ib_sm.x[803]: [ib_sm_sweep.c:1516]: No  
topology
change

On May 23, 2008, at 2:20 PM, Steve Wise wrote:

> Or Gerlitz wrote:
>> Steve Wise wrote:
>>> Are we sure we need to expose this to the user?
>> I believe this is the way to go if we want to let smart ULPs  
>> generate new rkey/stag per mapping. Simpler ULPs could then just  
>> put the same value for each map associated with the same mr.
>>
>> Or.
>>
>
> How should I add this to the API?
>
> Perhaps we just document the format of an rkey in the struct ib_mr.   
> Thus the app would do this to change the key before posting the  
> fast_reg_mr wr (coded to be explicit, not efficient):
>
> u8 newkey;
> u32 newrkey;
>
> newkey = 0xaa;
> newrkey = (mr->rkey & 0xffffff00) | newkey;
> mr->rkey = newrkey
> wr.wr.fast_reg.mr = mr;
> ...
>
>
> Note, this assumes mr->rkey is in host byte order (I think the linux  
> rdma code assumes this in other places too).
>
>
> Steve.
>
>
>
>
>
>
>
>
>
> _______________________________________________
> general mailing list
> general at lists.openfabrics.org
> http://lists.openfabrics.org/cgi-bin/mailman/listinfo/general
>
> To unsubscribe, please visit http://openib.org/mailman/listinfo/openib-general

-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.openfabrics.org/pipermail/general/attachments/20080527/fee62285/attachment.html>