[ofa-general] [PATCH 3/3] Added documentation for io_guid_file and max_reverse_hop feature

[Nicolas Morey Chaisemartin]

Signed-off-by: Nicolas Morey-Chaisemartin <nicolas.morey-chaisemartin at ext.bull.net>
---
  opensm/doc/current-routing.txt |   32 ++++++++++++++++++++++++++++++++
  opensm/man/opensm.8.in         |   27 +++++++++++++++++++++++++++
  2 files changed, 59 insertions(+), 0 deletions(-)

diff --git a/opensm/doc/current-routing.txt b/opensm/doc/current-routing.txt
index 0034d0e..1302860 100644
--- a/opensm/doc/current-routing.txt
+++ b/opensm/doc/current-routing.txt
@@ -237,6 +237,38 @@ in the same directory where the OpenSM log resides. This ordering file provides
  the CN order that may be used to create efficient communication pattern, that
  will match the routing tables.

+Routing between non-CN nodes
+
+
+The use of the cn_guid_file option allows non-CN nodes to be located on different levels in the fat tree.
+In such case, it is not guaranteed that the Fat Tree algorithm will route between two non-CN nodes.
+In the scheme below, N1, N2 and N3 are non-CN nodes. Although all the CN have routes to and from them,
+there will not necessarily be a route between N1,N2 and N3.
+Such routes would require to use at least one of the Switch the wrong way around
+(In fact, go out of one of the top Switch through a downgoing port while we are supposed to go up).
+
+  Spine1   Spine2    Spine 3
+   / \     /  |  \    /   \
+  /   \   /   |   \  /     \
+ N1  Switch   N2  Switch    N3
+      /|\          /|\
+     / | \        / | \
+    Going down to compute nodes
+
+To solve this problem, a list of non-CN nodes can be specified by \'-G\' or \'--io_guid_file\' option.
+Theses nodes will be allowed to use switches the wrong way around a specific number of times (specified by \'-H\' or \'--max_reverse_hops\'.
+With the proper max_reverse_hops and io_guid_file values, you can ensure full connectivity in the Fat Tree.
+
+In the scheme above, with a max_reverse_hop of 1, routes will be instanciated between N1<->N2 and N2<->N3.
+With a max_reverse_hops value of 2, N1,N2 and N3 will all have routes between them.
+
+Please note that using max_reverse_hops creates routes that use the switch in a counter-stream way.
+This option should never be used to connect nodes with high bandwidth traffic between them ! It should only be used
+to allow connectivity for HA purposes or similar.
+Also having routes the other way around can in theory cause credit loops.
+
+Use these options with extreme care !
+

  Usage:

diff --git a/opensm/man/opensm.8.in b/opensm/man/opensm.8.in
index 7690980..ce14c02 100644
--- a/opensm/man/opensm.8.in
+++ b/opensm/man/opensm.8.in
@@ -22,6 +22,8 @@ opensm \- InfiniBand subnet manager and administration (SM/SA)
  [\-S | \-\-sadb_file <file name>]
  [\-a | \-\-root_guid_file <path to file>]
  [\-u | \-\-cn_guid_file <path to file>]
+[\-G | \-\-io_guid_file <path to file>]
+[\-H | \-\-max_reverse_hops <max reverse hops allowed>]
  [\-X | \-\-guid_routing_order_file <path to file>]
  [\-m | \-\-ids_guid_file <path to file>]
  [\-o(nce)]
@@ -183,6 +185,16 @@ algorithm to the guids provided in the given file (one to a line).
  Set the compute nodes for the Fat-Tree routing algorithm
  to the guids provided in the given file (one to a line).
  .TP
+\fB\-G\fR, \fB\-\-io_guid_file\fR <file name>
+Set the I/O nodes for the Fat-Tree routing algorithm
+to the guids provided in the given file (one to a line).
+I/O nodes are non-CN nodes allowed to use up to max_reverse_hops switches
+the wrong way around to improve connectivity.
+.TP
+\fB\-H\fR, \fB\-\-max_reverse_hops\fR <file name>
+Set the maximum number of reverse hops an I/O node is allowed
+to make. A reverse hop is the use of a switch the wrong way around.
+.TP
  \fB\-m\fR, \fB\-\-ids_guid_file\fR <file name>
  Name of the map file with set of the IDs which will be used
  by Up/Down routing algorithm instead of node GUIDs
@@ -800,6 +812,21 @@ in the same directory where the OpenSM log resides. This ordering file provides
  the CN order that may be used to create efficient communication pattern, that
  will match the routing tables.

+Routing between non-CN nodes
+
+The use of the cn_guid_file option allows non-CN nodes to be located on different levels in the fat tree.
+In such case, it is not guaranteed that the Fat Tree algorithm will route between two non-CN nodes.
+To solve this problem, a list of non-CN nodes can be specified by \'-G\' or \'--io_guid_file\' option.
+Theses nodes will be allowed to use switches the wrong way round a specific number of times (specified by \'-H\' or \'--max_reverse_hops\'.
+With the proper max_reverse_hops and io_guid_file values, you can ensure full connectivity in the Fat Tree.
+
+Please note that using max_reverse_hops creates routes that use the switch in a counter-stream way.
+This option should never be used to connect nodes with high bandwidth traffic between them ! It should only be used
+to allow connectivity for HA purposes or similar.
+Also having routes the other way around can in theory cause credit loops.
+
+Use these options with extreme care !
+
  Activation through OpenSM

  Use '-R ftree' option to activate the fat-tree algorithm.
-- 
1.6.1