[ofa-general] Re: ipoib / bonding and OFED

[Or Gerlitz]

Bob Kossey wrote:
> I copied OR since I think this is related to his OFED HA work, and
> he might have some insights.  A few more questions for Or:
> I was trying to use ipoib bonding with OFED 1.2 rc2 and a 2.6.9 kernel,
> but was not able to get it to work so far.  I saw your Sonoma bonding
> slides, and you mention kernel bonding driver changes were needed.
> 2. Is there a minimum kernel version, with the kernel bonding driver
> changes, that is required to use bonding with OFED ipoib?

Just to have a base line here: to get bonding to work with IPoIB, you 
should use the bonding driver provided with OFED 1.2. This driver is the 
  upstream one (of 2.6.20) being patched to support IPoIB and backported 
to RH5, SLES10 and RH4 U3/4/5, other kernels are not supported.

If you were using the ofed bonding on a system that matches the support 
matrix it should worl. If do have problems under this config, please 
either open a bug at the ofed bugzilla
@ bugs.openfabrics.org assigned to monis at voltaire.com (Moni Shoua) or 
send first report/question to Moni and CC ewg at lists.openfabrics.org

Please note that between RC2 and RC4 (to be released today etc) some 
bugs were fixed, you can search in the bugzilla to see what.

> 3. The bonding driver uses the HWADDR from the underlying ipoib
> devices, how does it obtain the HWADDR?  Does it use the full 20 bytes,
> or some subset?

when enslaving IPoIB devices, the bonding driver uses the full hw 
address of the active slave, it simply looks on the dev_addr field of 
the slave struct netdevice (see include/linux/netdevice.h)

> 4. What use_carrier options for link status detection does OFED ipoib 
> support,
> MII, ETHTOOL or netif_carrier_ok?

the mii/ethertool etc local link detection methods of the bonding driver 
  are somehow deprecated, since nowadays almost any network device 
support the netif_carrier_ok call. The --default-- of the upstream 
bonding driver (eg the one we use in OFED and the 2.6.21 listed below) 
is to set the use_carrier mod param to 1 that is mii is not used anymore.

> author:         Thomas Davis, tadavis at lbl.gov and many others
> description:    Ethernet Channel Bonding Driver, v3.1.2
> version:        3.1.2
> parm:           use_carrier:Use netif_carrier_ok (vs MII ioctls) in miimon; 0 for off, 1 for on (default) (int)
> parm:           miimon:Link check interval in milliseconds (int)

> If you have any good examples of bonding configuration settings that work
> with OFED, I'd appreciate that also.

The bonding RPM provided with OFED is made of a driver, script and some 
help text containing usage examples, please take a look there and let me 
know if you have further questions.

> $ rpm -ql ib-bonding-0.9.0-2.6.9_42.ELsmp
> /lib/modules/2.6.9-42.ELsmp/updates/kernel/drivers/net/bonding/bonding.ko
> /usr/bin/ib-bond
> /usr/share/doc/ib-bonding-0.9.0/ib-bonding.txt

The ofed service (/etc/init.d/openibd) was enhanced to allow for 
--persistent-- bonding configuration, please see the bonding section at
docs/ipoib_release_notes.txt to see how to do it.

Or.