<html><body style="word-wrap: break-word; -webkit-nbsp-mode: space; -webkit-line-break: after-white-space; "><div><br></div><div>We have a 400 node IB cluster. We are running an embedded SM in failover mode on our TS270/Cisco7008 core switches. Lately we have been seeing problems with LID assignment when rebooting nodes (see log messages below). It is also taking far too long for LIDS to be assigned as it takes on the order of minutes for the ports to transition to "ACTIVE". </div><div><br></div><div>This seems like a bug to us and we are considering switching to OpenSM on a host. I'm wondering about experience with running OpenSM for medium to large clusters (Fat Tree) and what resources (memory/cpu) we should plan on for the host node. </div><div><br></div><div>Thanks,</div><div><br></div><div>Charlie Taylor</div><div>UF HPC Center</div><div><br></div><div><span class="Apple-style-span" style="font-family: Times; font-size: 16px; "><pre id="comment_text_2"><font class="Apple-style-span" face="Courier" size="3"><span class="Apple-style-span" style="font-size: 12px;">May 27 14:14:10 topspin-270sc ib_sm.x[803]: [ib_sm_sweep.c:1914]:
********************** NEW SWEEP ********************
May 27 14:14:10 topspin-270sc ib_sm.x[803]: [ib_sm_sweep.c:1320]: Rediscover
the subnet
May 27 14:14:13 topspin-270sc ib_sm.x[803]: [INFO]: Generate SM OUT_OF_SERVICE
trap for GID=fe:80:00:00:00:00:00:00:00:02:c9:02:00:21:4b:59
May 27 14:14:13 topspin-270sc ib_sm.x[803]: [ib_sm_sweep.c:256]: An existing IB
node GUID 00:02:c9:02:00:21:4b:59 LID 194 was removed
May 27 14:14:14 topspin-270sc ib_sm.x[803]: [INFO]: Generate SM DELETE_MC_GROUP
trap for GID=ff:12:60:1b:ff:ff:00:00:00:00:00:01:ff:21:4b:59
May 27 14:14:14 topspin-270sc ib_sm.x[803]: [ib_sm_sweep.c:1503]: Topology
changed
May 27 14:14:14 topspin-270sc ib_sm.x[803]: [INFO]: Configuration caused by
discovering removed ports
May 27 14:16:26 topspin-270sc ib_sm.x[803]: [ib_sm_sweep.c:1875]: async events
require sweep
May 27 14:16:26 topspin-270sc ib_sm.x[803]: [ib_sm_sweep.c:1914]:
********************** NEW SWEEP ********************
May 27 14:16:26 topspin-270sc ib_sm.x[803]: [ib_sm_sweep.c:1320]: Rediscover
the subnet
May 27 14:16:28 topspin-270sc ib_sm.x[812]: [ib_sm_discovery.c:1009]: no
routing required for port guid 00:02:c9:02:00:21:4b:59, lid 194
May 27 14:16:30 topspin-270sc ib_sm.x[803]: [ib_sm_sweep.c:1503]: Topology
changed
May 27 14:16:30 topspin-270sc ib_sm.x[803]: [INFO]: Configuration caused by
discovering new ports
May 27 14:16:30 topspin-270sc ib_sm.x[803]: [INFO]: Configuration caused by
multicast membership change
May 27 14:16:30 topspin-270sc ib_sm.x[812]: [ib_sm_assign.c:588]: Force port to
go down due to LID conflict, node - GUID=00:02:c9:02:00:21:4b:58, port=1
May 27 14:18:42 topspin-270sc ib_sm.x[819]: [ib_sm_bringup.c:562]: Program port
state, node=00:02:c9:02:00:21:4b:58, port= 16, current state 2, neighbor
node=00:02:c9:02:00:21:4b:58, port= 1, current state 2
May 27 14:18:42 topspin-270sc ib_sm.x[819]: [ib_sm_bringup.c:733]: Failed to
negotiate MTU, op_vl for node=00:02:c9:02:00:21:4b:58, port= 1, mad status 0x1c
May 27 14:18:42 topspin-270sc ib_sm.x[803]: [INFO]: Generate SM IN_SERVICE trap
for GID=fe:80:00:00:00:00:00:00:00:02:c9:02:00:21:4b:59
May 27 14:18:42 topspin-270sc ib_sm.x[803]: [ib_sm_sweep.c:144]: A new IB node
00:02:c9:02:00:21:4b:59 was discovered and assigned LID 0
May 27 14:18:43 topspin-270sc ib_sm.x[803]: [ib_sm_sweep.c:1875]: async events
require sweep
May 27 14:18:43 topspin-270sc ib_sm.x[803]: [ib_sm_sweep.c:1914]:
********************** NEW SWEEP ********************
May 27 14:18:43 topspin-270sc ib_sm.x[803]: [ib_sm_sweep.c:1320]: Rediscover
the subnet
May 27 14:18:46 topspin-270sc ib_sm.x[803]: [ib_sm_sweep.c:1516]: No topology
change
May 27 14:18:46 topspin-270sc ib_sm.x[803]: [INFO]: Configuration caused by
previous GET/SET operation failures
May 27 14:18:46 topspin-270sc ib_sm.x[816]: [ib_sm_assign.c:545]: Reassigning
LID, node - GUID=00:02:c9:02:00:21:4b:58, port=1, new LID=411, curr LID=0
May 27 14:18:46 topspin-270sc ib_sm.x[816]: [ib_sm_assign.c:588]: Force port to
go down due to LID conflict, node - GUID=00:02:c9:02:00:21:4b:58, port=1
May 27 14:18:46 topspin-270sc ib_sm.x[816]: [ib_sm_assign.c:635]: Clean up SA
resources for port forced down due to LID conflict, node -
GUID=00:02:c9:02:00:21:4b:58, port=1
May 27 14:18:47 topspin-270sc ib_sm.x[803]: [ib_sm_assign.c:667]: cleaning DB
for guid 00:02:c9:02:00:21:4b:59, lid 194
May 27 14:18:47 topspin-270sc ib_sm.x[803]: [ib_sm_routing.c:2936]:
_ib_smAllocSubnet: initRate= 4
May 27 14:18:47 topspin-270sc last message repeated 23 times
May 27 14:18:47 topspin-270sc ib_sm.x[803]: [INFO]: Different capacity links
detected in the network
May 27 14:21:01 topspin-270sc ib_sm.x[820]: [ib_sm_bringup.c:516]: Active
port(s) now in INIT state node=00:02:c9:02:00:21:4b:58, port=16, state=2,
neighbor node=00:02:c9:02:00:21:4b:58, port=1, state=2
May 27 14:21:01 topspin-270sc ib_sm.x[803]: [ib_sm_sweep.c:1875]: async events
require sweep
May 27 14:21:01 topspin-270sc ib_sm.x[803]: [ib_sm_sweep.c:1914]:
********************** NEW SWEEP ********************
May 27 14:21:01 topspin-270sc ib_sm.x[803]: [ib_sm_sweep.c:1320]: Rediscover
the subnet
May 27 14:21:05 topspin-270sc ib_sm.x[803]: [ib_sm_sweep.c:1516]: No topology
change
May 27 14:21:05 topspin-270sc ib_sm.x[803]: [ib_sm_sweep.c:525]: IB node
00:06:6a:00:d9:00:04:5d port 16 is INIT state
May 27 14:21:05 topspin-270sc ib_sm.x[803]: [INFO]: Configuration caused by
some ports in INIT state
May 27 14:21:05 topspin-270sc ib_sm.x[803]: [INFO]: Configuration caused by
previous GET/SET operation failures
May 27 14:21:05 topspin-270sc ib_sm.x[803]: [ib_sm_routing.c:2936]:
_ib_smAllocSubnet: initRate= 4
May 27 14:21:05 topspin-270sc last message repeated 23 times
May 27 14:21:05 topspin-270sc ib_sm.x[803]: [INFO]: Different capacity links
detected in the network
May 27 14:23:19 topspin-270sc ib_sm.x[817]: [ib_sm_bringup.c:562]: Program port
state, node=00:02:c9:02:00:21:4b:58, port= 16, current state 2, neighbor
node=00:02:c9:02:00:21:4b:58, port= 1, current state 2
May 27 14:23:24 topspin-270sc ib_sm.x[823]: [INFO]: Generate SM CREATE_MC_GROUP
trap for GID=ff:12:60:1b:ff:ff:00:00:00:00:00:01:ff:21:4b:59
May 27 14:23:24 topspin-270sc ib_sm.x[803]: [ib_sm_sweep.c:1875]: async events
require sweep
May 27 14:23:24 topspin-270sc ib_sm.x[803]: [ib_sm_sweep.c:1914]:
********************** NEW SWEEP ********************
May 27 14:23:26 topspin-270sc ib_sm.x[803]: [ib_sm_sweep.c:1516]: No topology
change
May 27 14:23:26 topspin-270sc ib_sm.x[803]: [INFO]: Configuration caused by
multicast membership change
May 27 14:23:33 topspin-270sc ib_sm.x[826]: [INFO]: Standby SM guid
00:05:ad:00:00:02:3c:60, is no longer synchronized with Master SM
May 27 14:25:39 topspin-270sc ib_sm.x[826]: [INFO]: Initialize a backup session
with Standby SM guid 00:05:ad:00:00:02:3c:60
May 27 14:25:39 topspin-270sc ib_sm.x[803]: [ib_sm_sweep.c:1875]: async events
require sweep
May 27 14:25:39 topspin-270sc ib_sm.x[803]: [ib_sm_sweep.c:1914]:
********************** NEW SWEEP ********************
May 27 14:25:39 topspin-270sc ib_sm.x[826]: [INFO]: Standby SM guid
00:05:ad:00:00:02:3c:60, started synchronizing with Master SM
May 27 14:25:42 topspin-270sc ib_sm.x[803]: [ib_sm_sweep.c:1516]: No topology
change
May 27 14:25:42 topspin-270sc ib_sm.x[803]: [INFO]: Configuration caused by
multicast membership change
May 27 14:25:43 topspin-270sc ib_sm.x[826]: [INFO]: Master SM DB synchronized
with Standby SM guid 00:05:ad:00:00:02:3c:60
May 27 14:25:43 topspin-270sc ib_sm.x[826]: [INFO]: Master SM DB synchronized
with all designated backup SMs
May 27 14:28:04 topspin-270sc ib_sm.x[803]: [ib_sm_sweep.c:1914]:
********************** NEW SWEEP ********************
May 27 14:28:06 topspin-270sc ib_sm.x[803]: [ib_sm_sweep.c:1516]: No topology
change </span></font></pre></span></div><br><div><div>On May 23, 2008, at 2:20 PM, Steve Wise wrote:</div><br class="Apple-interchange-newline"><blockquote type="cite">Or Gerlitz wrote:<br><blockquote type="cite">Steve Wise wrote:<br></blockquote><blockquote type="cite"><blockquote type="cite">Are we sure we need to expose this to the user?<br></blockquote></blockquote><blockquote type="cite">I believe this is the way to go if we want to let smart ULPs generate new rkey/stag per mapping. Simpler ULPs could then just put the same value for each map associated with the same mr.<br></blockquote><blockquote type="cite"><br></blockquote><blockquote type="cite">Or.<br></blockquote><blockquote type="cite"><br></blockquote><br>How should I add this to the API?<br><br>Perhaps we just document the format of an rkey in the struct ib_mr. Thus the app would do this to change the key before posting the fast_reg_mr wr (coded to be explicit, not efficient):<br><br>u8 newkey;<br>u32 newrkey;<br><br>newkey = 0xaa;<br>newrkey = (mr->rkey & 0xffffff00) | newkey;<br>mr->rkey = newrkey<br>wr.wr.fast_reg.mr = mr;<br>...<br><br><br>Note, this assumes mr->rkey is in host byte order (I think the linux rdma code assumes this in other places too).<br><br><br>Steve.<br><br><br><br><br><br><br><br><br><br>_______________________________________________<br>general mailing list<br><a href="mailto:general@lists.openfabrics.org">general@lists.openfabrics.org</a><br>http://lists.openfabrics.org/cgi-bin/mailman/listinfo/general<br><br>To unsubscribe, please visit http://openib.org/mailman/listinfo/openib-general<br></blockquote></div><br></body></html>