[ofa-general] OpenSM Problems/Questions

[Matthew Trzyna]

Hello


A "Basic Fabric Diagram" at the end.


I am working with a customer implementing a large IB fabric and is
encountering problems with OpenSM (OFED 1.3) when they added a new 264 node
cluster (with its own 288 port IB switch) to their existing cluster. Two
more 264 clusters are planned to be added in the near future. They recently
moved to SLES 10 SP1 and OFED 1.3 (before adding the new cluster) and had
not been experiencing these problems before.

Could you help provide answers to the questions listed below? Additional
information about the configuration including a basic fabric diagram are
provided after the questions.

   What parameters should be set on the non-SM nodes that affect how the
   Subnet Administrator functions?
   What parameters should be set on the SM node(s) that affect how the
   Subnet Administrator functions? And, what  parameters should be removed
   from the SM node(s)? (ie.  ib_sa paths_per_dest=0x7f)
   How should SM failover be setup? How many failover SM's should be
   configured? This must happen quickly and transparently or GPFS will die
   everywhere due to timeouts if this takes too long).
   Are there SA (Subnet Administrator) commands that should not be executed
   on a large "live" fabric?            (ie. "saquery -p")
   Should GPFS be configured "off" on the SM node(s)?
   Do you know of any other OpenSM implementations that have 5 (or more)
   288 port IB switches that might have already encountered/resolved some
   of these issues?
   The following problem that is being encountered may also be SA/SM
   related. A node (NodeX) may be seen (through IPoIB) by all but a few
   nodes (NodesA-G). A ping from those node (NodesA-G) to NodeX returns
   "Destination Host Unreachable". A ping from NodeX to NodesA-G works.


--------------------------------------------------------------------------------------------------

System Information

Here is the current opensm.conf file: (See attached file: opensm.conf)

It is the default configuration from the OFED 1.3 build with "priority"
added at the bottom. Note that the /etc/init.d/opensmd
sources /etc/sysconfig/opensm not etc/sysconfig/opensm.conf (opensm.conf
was just copied to opensm). There are a couple of "proposed" settings that
are commented out, that were found them on the web.

Following are the present settings that may affect the Fabric:

/etc/infiniband/openib.conf
SET_IPOIB_CM=no

/etc/modprobe.conf.local
options ib_ipoib send_queue_size=512 recv_queue_size=512
options ib_sa paths_per_dest=0x7f

/etc/sysctl.conf
net.ipv4.neigh.ib0.base_reachable_time = 1200
net.ipv4.neigh.default.gc_thresh3 = 3072
net.ipv4.neigh.default.gc_thresh2 = 2500
net.ipv4.neigh.default.gc_thresh1 = 2048

/etc/sysconfig/opensm
All defaults as supplied with OFED 1.3 OpenSM


-------------------------------------------------------


                    Basic Fabric Diagram

                     +----------+
                     |Top Level |-------------------+ 20 IO nodes
   +-----------------| 288 port |----------------+    16 Viual nodes
   |                 |  IB Sw   |------------+   |     2 Admin nodes
   |          +------|          |---+        |   |       (SM nodes)
   |          |      +----------+   |        |   |     4 Support nodes
   |          |          |          |        |   |
   |          |          |          |        |   |
  24         24         24         24       24  24 <--uplinks
   |          |          |          |        |   |
   |          |          |          |        |   +------+
   |          |          |          |        |          |
   |(BASE)    |(SCU1)    |(SCU2)    |(SCU3)  |(SCU4)    |(SCU5)
+--------+ +--------+ +--------+ +--------+ +--------+ +--------+
|288-port| |288-port| |288-port| |288-port| |288-port| |288-port|
| IB Sw  | | IB Sw  | | IB Sw  | |  IB Sw | |  IB Sw | |  IB Sw |
+--------+ +--------+ +--------+ +--------+ +--------+ +--------+
 140-nodes 264-nodes  264-nodes  264-nodes  264-nodes  264-nodes
 WhiteBox    Dell       Dell       IBM        IBM      IBM (future)

NOTE: SCU4 is not currently connected to the Top Level Switch.
      We'd like to address these issues before making that connection.

      Subnet Managers are configured on nodes connected to the
      Top Leval Switch.

Let me know if you need any more information.

Any help you could provide would be most appreciated.

Thanks.

Matt Trzyna
IBM Linux Cluster Enablement
3039 Cornwallis Rd.
RTP, NC  27709
e-mail: trzyna at us.ibm.com
Office: (919) 254-9917  Tie Line: 444
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.openfabrics.org/pipermail/general/attachments/20080909/1667ec7d/attachment.html>
-------------- next part --------------
A non-text attachment was scrubbed...
Name: opensm.conf
Type: application/octet-stream
Size: 4797 bytes
Desc: not available
URL: <http://lists.openfabrics.org/pipermail/general/attachments/20080909/1667ec7d/attachment.obj>