[ofa-general] Re: ipoib / bonding and OFED

[Bob Kossey]

Thanks guys, I'll have to update my bits and try again. 

Another related question.  Does OFED 1.2 now support multiple 
independent IB fabrics
(multiple SMs, etc) connected to multiple HCAs on the same node?  Are 
there any
qualifications about which dimensions are supported with this, such as 
ipoib HA, SRP HA,
other types of failover, etc.?

Thanks,
Bob

Scott Weitzenkamp (sweitzen) wrote:
> Bob, it is now possible to configure IPoIB bonding in
> /etc/infiniband/openib.conf, this configuration file includes the
> following boilerplate.
>
> # Enable the bonding driver on startup
> IPOIBBOND_ENABLE=no
> # Set bond interface names
> #IPOIB_BONDS=bond0,bond1
> # Set specific bond params; address and slaves
> #bond0_IP=10.10.10.1
> #bond0_SLAVES=ib0,ib1
> #bond1_IP=20.10.10.1
> #bond1_SLAVES=ib2,ib3,ib4
>
> Scott Weitzenkamp
> SQA and Release Manager
> Server Virtualization Business Unit
> Cisco Systems
>  
>
>   
>> -----Original Message-----
>> From: general-bounces at lists.openfabrics.org 
>> [mailto:general-bounces at lists.openfabrics.org] On Behalf Of Or Gerlitz
>> Sent: Tuesday, May 29, 2007 12:56 AM
>> To: Bob Kossey
>> Cc: OpenFabrics General
>> Subject: [ofa-general] Re: ipoib / bonding and OFED
>>
>> Bob Kossey wrote:
>>     
>>> I copied OR since I think this is related to his OFED HA work, and
>>> he might have some insights.  A few more questions for Or:
>>> I was trying to use ipoib bonding with OFED 1.2 rc2 and a 
>>>       
>> 2.6.9 kernel,
>>     
>>> but was not able to get it to work so far.  I saw your 
>>>       
>> Sonoma bonding
>>     
>>> slides, and you mention kernel bonding driver changes were needed.
>>> 2. Is there a minimum kernel version, with the kernel bonding driver
>>> changes, that is required to use bonding with OFED ipoib?
>>>       
>> Just to have a base line here: to get bonding to work with IPoIB, you 
>> should use the bonding driver provided with OFED 1.2. This 
>> driver is the 
>>   upstream one (of 2.6.20) being patched to support IPoIB and 
>> backported 
>> to RH5, SLES10 and RH4 U3/4/5, other kernels are not supported.
>>
>> If you were using the ofed bonding on a system that matches 
>> the support 
>> matrix it should worl. If do have problems under this config, please 
>> either open a bug at the ofed bugzilla
>> @ bugs.openfabrics.org assigned to monis at voltaire.com (Moni Shoua) or 
>> send first report/question to Moni and CC ewg at lists.openfabrics.org
>>
>> Please note that between RC2 and RC4 (to be released today etc) some 
>> bugs were fixed, you can search in the bugzilla to see what.
>>
>>     
>>> 3. The bonding driver uses the HWADDR from the underlying ipoib
>>> devices, how does it obtain the HWADDR?  Does it use the 
>>>       
>> full 20 bytes,
>>     
>>> or some subset?
>>>       
>> when enslaving IPoIB devices, the bonding driver uses the full hw 
>> address of the active slave, it simply looks on the dev_addr field of 
>> the slave struct netdevice (see include/linux/netdevice.h)
>>
>>     
>>> 4. What use_carrier options for link status detection does 
>>>       
>> OFED ipoib 
>>     
>>> support,
>>> MII, ETHTOOL or netif_carrier_ok?
>>>       
>> the mii/ethertool etc local link detection methods of the 
>> bonding driver 
>>   are somehow deprecated, since nowadays almost any network device 
>> support the netif_carrier_ok call. The --default-- of the upstream 
>> bonding driver (eg the one we use in OFED and the 2.6.21 
>> listed below) 
>> is to set the use_carrier mod param to 1 that is mii is not 
>> used anymore.
>>
>>     
>>> author:         Thomas Davis, tadavis at lbl.gov and many others
>>> description:    Ethernet Channel Bonding Driver, v3.1.2
>>> version:        3.1.2
>>> parm:           use_carrier:Use netif_carrier_ok (vs MII 
>>>       
>> ioctls) in miimon; 0 for off, 1 for on (default) (int)
>>     
>>> parm:           miimon:Link check interval in milliseconds (int)
>>>       
>>> If you have any good examples of bonding configuration 
>>>       
>> settings that work
>>     
>>> with OFED, I'd appreciate that also.
>>>       
>> The bonding RPM provided with OFED is made of a driver, 
>> script and some 
>> help text containing usage examples, please take a look there 
>> and let me 
>> know if you have further questions.
>>
>>     
>>> $ rpm -ql ib-bonding-0.9.0-2.6.9_42.ELsmp
>>>
>>>       
>> /lib/modules/2.6.9-42.ELsmp/updates/kernel/drivers/net/bonding
>> /bonding.ko
>>     
>>> /usr/bin/ib-bond
>>> /usr/share/doc/ib-bonding-0.9.0/ib-bonding.txt
>>>       
>> The ofed service (/etc/init.d/openibd) was enhanced to allow for 
>> --persistent-- bonding configuration, please see the bonding 
>> section at
>> docs/ipoib_release_notes.txt to see how to do it.
>>
>> Or.
>>
>> _______________________________________________
>> general mailing list
>> general at lists.openfabrics.org
>> http://lists.openfabrics.org/cgi-bin/mailman/listinfo/general
>>
>> To unsubscribe, please visit 
>> http://openib.org/mailman/listinfo/openib-general
>>
>>