[ewg] Re: Please update RN for OFED 1.4.1

[Moni Shoua]

Hi Tziporet,
This is a patch for bonding doc
thanks

diff --git a/ib-bonding.txt b/ib-bonding.txt
index c797cde..c6266f4 100644
--- a/ib-bonding.txt
+++ b/ib-bonding.txt
@@ -15,6 +15,8 @@ IB Bonding
 -------------------------------------------------------------------------------
 ib-bonding is a High Availability solution for IPoIB interfaces. It is based
 on the Linux Ethernet Bonding Driver and was adopted to work with IPoIB.
+However, the support for for IPoIB interfaces is only for the active-backup
+mode, other modes should not be used.
 ib-bonding package contains a bonding driver and a utility called ib-bond to
 manage and control the driver operation.
 
@@ -70,6 +72,7 @@ Update 6 or Update 7) and for Redhat-EL5 and above.
 -----------------------------------------------------------------
 * In the master (bond) interface script add the line:
 TYPE=Bonding
+MTU=<according to the slave's MTU>
 
 Exmaple: for bond0 (master) the file is named /etc/sysconfig/network-scripts/ifcfg-bond0
 with the following text in the file:
@@ -83,11 +86,19 @@ ONBOOT=yes
 BOOTPROTO=none
 USERCTL=no
 TYPE=Bonding
+MTU=65520
+
+Note: 65520 is a valid mtu value only if all IPoIB slaves operate in connected
+mode and are configured with the same value. For IPoIB slaves that work in
+datagram modee, use MTU=2044. If you don't set correct mtu or don't set mtu at
+all (and letting it to be set to the default value), performance of the
+interface might decrease.
 
 * In the slave (ib) interface script put the following lines:
 SLAVE=yes
 MASTER=<bond name>
 TYPE=InfiniBand
+PRIMARY=<yes|no>
 
 Example: the script for ib0 (slave) would be named /etc/sysconfig/network-scripts/ifcfg-ib0
 with the following text in the file:
@@ -99,6 +110,10 @@ MASTER=bond0
 SLAVE=yes
 BOOTPROTO=none
 TYPE=InfiniBand
+PRIMARY=yes
+
+Note: If the slave interface is not primary then the line PRIMARY= is not
+required and can be omitted.
 
 After the configuration is saved, restart the network service by running:
 /etc/init.d/network restart
@@ -108,6 +123,9 @@ After the configuration is saved, restart the network service by running:
 Follow the instructions in 3.1.1 (Writing network scripts under Redhat-AS4)
 with the following changes:
 * In the bondX (master) script - the line TYPE=Bonding is not needed.
+* In the bondX (master) script - you may add to the configuration more options
+with the following line
+BONDING_OPTS=" primary=ib0 updelay=0 downdelay=0"
 * in the ibX (slave) script - the line TYPE=InfiniBand necessary when using 
   bonding over devices configured with partitions ( p_key)
 Example:
@@ -137,6 +155,7 @@ BONDING_MASTER=yes
 BONDING_MODULE_OPTS="mode=active-backup miimon=<value>"
 BONDING_SLAVE0=slave0
 BONDING_SLAVE1=slave1
+MTU=<according to the slave's MTU>
 
 Exmaple: for bond0 (master) the file is named /etc/sysconfig/network/ifcfg-bond0
 with the following text in the file:
@@ -149,9 +168,20 @@ NETWORK="10.0.2.0"
 REMOTE_IPADDR=""
 STARTMODE="onboot"
 BONDING_MASTER="yes"
-BONDING_MODULE_OPTS="mode=active-backup miimon=100"
+BONDING_MODULE_OPTS="mode=active-backup miimon=100 primary=ib0 updelay=0 downdelay=0"
 BONDING_SLAVE0=ib0
 BONDING_SLAVE1=ib1
+MTU=65520
+
+Note: 65520 is a valid mtu value only if all IPoIB slaves operate in connected
+mode and are configured with the same value. For IPoIB slaves that work in
+datagram modee, use MTU=2044. If you don't set correct mtu or don't set mtu at
+all (and letting it to be set to the default value), performance of the
+interface might decrease.
+
+Note: primary, downdelay and updelay is an optional bonding interface
+configuration. You may choose to use them, change them or delete them from the
+configuration script (by editing the line that starts with BONDING_OPTS)
 
 * The slave (ib) interace script should look like this:
 
@@ -167,18 +197,36 @@ After the configuration is saved, restart the network service by running:
 It is not possible to have a mix of Ethernt slaves and IPoIB slaves under the
 same bonding master. It is possible however that a bonding master of Ethernet
 slaves and a bonding master of IPoIB slaves will co-exist in one machne.
-To configure Ethernet slaves under a bonding master use the same instructions
-as for IPoIB slaves (according  to the OS) with one exception. When working
-under Redhat-AS4 do the following when configuring a bonding  master with
-Ethernet slaves
+To configure Ethernet slaves under a bonding master use the following
+instructios (depending  on the OS)
+
+* Under Redhat-AS4
+
+Use the same instructions as for IPoIB slaves with the following exceptions
 
 - In the master configuration file add the line
 SLAVEDEV=1
 - In the slave configuration file leave the line
 TYPE=InfiniBand
+- For Ethernet, it is possible to set parameters of the bonding module in /etc/modprobe.conf
+with the following line for example
+options bonding miimon=100 mode=1 primary=eth0
+Note that alias names for the bonding module (such as bond0) may not work.
+
+* Under Redhat-AS5
+
+No special instructions are required.
+
+* Under SLES10
 
-When using both type of bonding under  SLES-10, it is neccessary to update the
+When using both type of bonding under, it is neccessary to update the
 MANDATORY_DEVICES  environment variable in /etc/sysconfig/network/config with the names
 of the InfiniBand devices ( ib0, ib1, etc. ). Otherwise, bonding devices will be created 
 before InfiniBand devices at boot time.
 
+Note: If there is more than one Ethernet NIC installed then there might be a
+race for the interface name eth0, eth1 etc. This may lead to unexpected
+relation between logical and physical devices which may lead to wrong bonding
+configuration. This issue may be solved by binding a logical device name (e.g.
+eth0) to a physical (hardware) device by specifying the MAC address in the
+ethN configuration file.