<html><body>
<p>Hello<br>
<br>
<br>
<tt>A "Basic Fabric Diagram" at the end. <br>
<br>
<br>
I am working with a customer implementing a large IB fabric and is encountering problems with OpenSM (OFED 1.3) when they added a new 264 node cluster (with its own 288 port IB switch) to their existing cluster. Two more 264 clusters are planned to be added in the near future. They recently moved to SLES 10 SP1 and OFED 1.3 (before adding the new cluster) and had not been experiencing these problems before.</tt><br>
<br>
<tt>Could you help provide answers to the questions listed below? Additional information about the configuration including a basic fabric diagram are provided after the questions.<br>
</tt>
<ol type="1">
<li><tt>What parameters should be set on the non-SM nodes that affect how the Subnet Administrator functions?</tt>
<li><tt>What parameters should be set on the SM node(s) that affect how the Subnet Administrator functions? And, what parameters should be removed from the SM node(s)? (ie. ib_sa paths_per_dest=0x7f)</tt>
<li><tt>How should SM failover be setup? How many failover SM's should be configured? This must happen quickly and transparently or GPFS will die everywhere due to timeouts if this takes too long).</tt>
<li><tt>Are there SA (Subnet Administrator) commands that should not be executed on a large "live" fabric? (ie. "saquery -p")</tt>
<li><tt>Should GPFS be configured "off" on the SM node(s)?</tt>
<li><tt>Do you know of any other OpenSM implementations that have 5 (or more) 288 port IB switches that might have already encountered/resolved some of these issues?</tt>
<li><tt>The following problem that is being encountered may also be SA/SM related. A node (NodeX) may be seen (through IPoIB) by all but a few nodes (NodesA-G). A ping from those node (NodesA-G) to NodeX returns "Destination Host Unreachable". A ping from NodeX to NodesA-G works.<br>
</tt></ol>
<br>
<tt>--------------------------------------------------------------------------------------------------</tt><br>
<br>
<tt>System Information</tt><br>
<br>
<tt>Here is the current opensm.conf file: </tt><i>(See attached file: opensm.conf)</i><br>
<br>
<tt>It is the default configuration from the OFED 1.3 build with "priority" added at the bottom. Note that the /etc/init.d/opensmd sources /etc/sysconfig/opensm not etc/sysconfig/opensm.conf (opensm.conf was just copied to opensm). There are a couple of "proposed" settings that are commented out, that were found them on the web.</tt><br>
<br>
<tt>Following are the present settings that may affect the Fabric:<br>
<br>
/etc/infiniband/openib.conf<br>
SET_IPOIB_CM=no<br>
<br>
/etc/modprobe.conf.local<br>
options ib_ipoib send_queue_size=512 recv_queue_size=512<br>
options ib_sa paths_per_dest=0x7f<br>
<br>
/etc/sysctl.conf<br>
net.ipv4.neigh.ib0.base_reachable_time = 1200<br>
net.ipv4.neigh.default.gc_thresh3 = 3072<br>
net.ipv4.neigh.default.gc_thresh2 = 2500<br>
net.ipv4.neigh.default.gc_thresh1 = 2048<br>
<br>
/etc/sysconfig/opensm<br>
All defaults as supplied with OFED 1.3 OpenSM</tt><br>
<br>
<br>
<tt>-------------------------------------------------------<br>
<br>
<br>
Basic Fabric Diagram<br>
<br>
+----------+<br>
|Top Level |-------------------+ 20 IO nodes<br>
+-----------------| 288 port |----------------+ 16 Viual nodes<br>
| | IB Sw |------------+ | 2 Admin nodes<br>
| +------| |---+ | | (SM nodes)<br>
| | +----------+ | | | 4 Support nodes<br>
| | | | | |<br>
| | | | | |<br>
24 24 24 24 24 24 <--uplinks<br>
| | | | | |<br>
| | | | | +------+<br>
| | | | | |<br>
|(BASE) |(SCU1) |(SCU2) |(SCU3) |(SCU4) |(SCU5)<br>
+--------+ +--------+ +--------+ +--------+ +--------+ +--------+<br>
|288-port| |288-port| |288-port| |288-port| |288-port| |288-port|<br>
| IB Sw | | IB Sw | | IB Sw | | IB Sw | | IB Sw | | IB Sw |<br>
+--------+ +--------+ +--------+ +--------+ +--------+ +--------+<br>
140-nodes 264-nodes 264-nodes 264-nodes 264-nodes 264-nodes<br>
WhiteBox Dell Dell IBM IBM IBM (future)<br>
<br>
NOTE: SCU4 is not currently connected to the Top Level Switch.<br>
We'd like to address these issues before making that connection.<br>
<br>
Subnet Managers are configured on nodes connected to the<br>
Top Leval Switch.</tt><br>
<br>
Let me know if you need any more information.<br>
<br>
Any help you could provide would be most appreciated.<br>
<br>
Thanks.<br>
<br>
Matt Trzyna<br>
IBM Linux Cluster Enablement<br>
3039 Cornwallis Rd.<br>
RTP, NC 27709<br>
e-mail: trzyna@us.ibm.com<br>
Office: (919) 254-9917 Tie Line: 444<br>
</body></html>