[openib-general] RE: ib1 takes ib0 configuration

Tue Mar 7 07:55:43 PST 2006

Moni,

Could you just create an ifcfg-ib1 file with different address
information than ifcfg-ib0, make sure you put this modified file in
/etc/infiniband directory not /etc/sysconfig/network?  Also, can you cat
out the /etc/inifiniband/ifcfg-ib0 and ifcfg-ib1 and send it to us. 
Also send me the openib.conf out of this directory.

We are available to chat with you if necessary.

Regards,
Moiz

>>> "Moni Shoua" <monis at voltaire.com> 3/7/2006 1:25:32 am >>>
Hi Moiz,

Below is the output you asked for.

Also, configuring the interface with ifconfig works fine

thanks
Moni

vacuum20:~ #\ls -l /etc/sysconfig/network
total 84
-rw-r--r-- 1 root root  8950 Feb 27 18:50 config
-rw-r--r-- 1 root root  6707 Feb 28 11:35 dhcp
drwxr-xr-x 2 root root  4096 Feb 27 18:53 if-down.d
drwxr-xr-x 2 root root  4096 Mar  5 11:56 if-up.d
-rw-r--r-- 1 root root   122 Feb 28 12:06 ifcfg-eth0
-rw-r--r-- 1 root root   148 Mar  5 14:40 ifcfg-ib0
-rw-r--r-- 1 root root   141 Feb 21 00:19 ifcfg-lo
-rw-r--r-- 1 root root 25550 Feb 21 00:19 ifcfg.template
-rw-r--r-- 1 root root   239 Feb 21 00:19 ifroute-lo
drwxr-xr-x 2 root root  4096 Feb 27 18:50 ifservices.template
drwx------ 2 root root  4096 Feb 19 18:31 providers
drwxr-xr-x 2 root root  4096 Feb 27 18:53 scripts
vacuum20:~ # cat /etc/sysconfig/network/ifcfg-eth0
DEVICE=eth0
BOOTPROTO='static'
IPADDR='192.168.80.150'
NETMASK='255.255.255.0'
NETWORK='192.168.80.0'
STARTMODE='onboot'
vacuum20:~ # cat /etc/sysconfig/network/ifcfg-ib0
DEVICE=ib0
BOOTPROTO='static'
STARTMODE='onboot'
IPADDR=192.168.20.150
NETWORK=192.168.20.0
NETMASK=255.255.255.0
BROADCAST=192.168.20.255
MTU=1500
vacuum20:~ # cat /etc/sysconfig/network/ifcfg-lo
# Loopback (lo) configuration
IPADDR=127.0.0.1
NETMASK=255.0.0.0
NETWORK=127.0.0.0
BROADCAST=127.255.255.255
STARTMODE=onboot
USERCONTROL=no
vacuum20:~ # cat /etc/udev/rules.d/30-net_persistent_names.rules
# This rules are autogenerated from /lib/udev/rename_netiface.
# But you can modify them, but make sure that you don't use an
interface
name
# twice. Also add such interface name rules only in this rules file.
Otherwise
# rename_netiface will create wrong rules for new interfaces.
# It is safe to delete a rule, as long as you did not disable
automatic
rule
# generation. Only if all interfaces get a rule the renaming will work
# flawlessly. See also /etc/udev/rules.d/31-net_create_names.rules.
#
# Read
/usr/share/doc/packages/sysconfig/README.Persistent_Interface_Names
for
# further information.
#
# Use only a-z, A-Z and 0-9 for interface names!
#
SUBSYSTEM=="net", ACTION=="add", SYSFS{address}=="01:04:23:b7:9e:69",
IMPORT="/lib/udev/rename_netiface %k eth1"
SUBSYSTEM=="net", ACTION=="add", SYSFS{address}=="00:04:23:b7:9e:68",
IMPORT="/lib/udev/rename_netiface %k eth0"

-----Original Message-----
From: Moiz Kohari [mailto:mkohari at novell.com] 
Sent: Monday, March 06, 2006 8:22 PM
To: Moni Shoua
Cc: Moiz Kohari
Subject: Re: ib1 takes ib0 configuration

Moni,

The output of the following commands would be helpful:

 # ls /etc/sysconfig/network/
 # cat /etc/sysconfig/network/ifcfg-*
 # cat /etc/udev/rules.d/30-net_persistent_names.rules

Also, have you tried to assign addresses by hand (ifconfig ib0
10.0.0.x), does this work.  I think you are right and sounds like
somewhere in the configuration we set ib1 to ib0.

Thanks,
Moiz

>>> "Moni Shoua" <monis at voltaire.com> 3/6/2006 8:20:43 am >>>
After building and installing openib stack on SUSE Linux Enterprise
Server 9.90 Beta5  I noticed that ib1 interface has identical IP
configuration to ib0 configuration.

ib0       Link encap:UNSPEC  HWaddr
00-00-04-04-FE-80-00-00-00-00-00-00-00-00-00-00 
          inet addr:192.68.3.238  Bcast:192.68.255.255
Mask:255.255.0.0

ib1       Link encap:UNSPEC  HWaddr
00-00-04-05-FE-80-00-00-00-00-00-00-00-00-00-00 
          inet addr:192.68.3.238  Bcast:192.68.255.255
Mask:255.255.0.0

This  causes problems in the machine's IP configuration even if the
link
of ib1 is down.
The fact that a network script for ib1 exists doesn't make a
difference.
ib1 still takes ib0 configuration.

When trying to query ib1 status (with /sbin/ifstatus ib1) I get this

     ib1       device: Mellanox Technologies MT25208 InfiniHost III Ex
HCA (Tavor compatibility mode) (rev a0)
    ib1       configuration: ib0
ib1 is up
10: ib1: <NO-CARRIER,BROADCAST,MULTICAST,UP> mtu 1500 qdisc pfifo_fast
qlen 128
    link/infiniband
00:00:04:05:fe:80:00:00:00:00:00:00:00:08:f1:04:03:97:08:ea brd
00:ff:ff:ff:ff:12:40:1b:ff:ff:00:00:00:00:00:00:ff:ff:ff:ff
    inet 192.68.3.238/16 brd 192.68.255.255 scope global ib1
    ib1       IP address: 192.68.3.238/16

When looking for the reason to this behavior I found that when the
network is started, ifup calls getcfg like this: /sbin/getcfg -d . -f
ifcfg- -- ib1 and one of the lines in the output is
HWD_CONFIG_0="ib0".
The HWD_CONFIG_0 variable is parsed in ifup and the variable CONFIG is
set to its value.

All the above doesn't happen on SUSE Linux Enterprise Server 9 and I
suspect that the difference is in the version of the rpm sysconfig
(0.50.6 vs. 0.31.0) which packages getcfg.

Does anyone have an idea how to solve this? Is this a bug of getcfg,
bug
in IPoIB or just a wrong IP configuration?

thanks

____________________________________________________________

Moni Shoua |  +972-9-9718630 (o)   |   +072-52-8232979 (m)

SW Engineer, Mainstream IB host stack

Voltaire - The Grid Backbone

 www.voltaire.com <http://www.voltaire.com/>

<mailto:g at voltaire.com>