[ewg] [Q] how to setup /etc/opensm/partitions.conf??

[Hiroyuki Sato]

Hello Hal

Thank you for your information.

I simplified my test environment
The environment is the folloing.

I'm not sure what is wrong Test2 (Partition test)
And Could you please tell me how to check the problem??
(tool, logfile, etc.. )

* Environment

  OS: Scientific Linux6.1
  OFED: 1.5.3.

* Diagram

 +----------+ib0            ib0+----------+
 | Server1  |------------------| Server2  |
 +----------+                  +----------+




1) Test1 (Simple IPoIB no partition Ping test)

  Server1

    a) /sbin/ifconfig ib0 inet 192.168.1.1/24
    b) /sbn/service opensmd start
    3) /sbin/service opensmd start
    * no /etc/opensm/partitions.conf

  Server2
    /sbin/ifconfig ib0 inet 192.168.1.1/24

  Test from server1(192.168.1.1) to server2(192.168.1.2)
    ping 192.168.1.2
    The result was OK.

2) Test2 (Add partition by hand and Ping test)

  reboot Server 1 and Server 2

  Server 1
    1) echo 0x8001 > /sys/class/net/ib0/create_child
    2) /sbin/ifconfig ib0.8001 inet 192.168.0.1/24
    3) /sbin/service opensmd start

    * partitions.conf is the following
      Default=0x7fff,  ipoib : ALL=full
      Net0=0x0001,     ipoib : ALL=full

  Server2
    1) echo 0x8001 > /sys/class/net/ib0/create_child
    2) /sbin/ifconfig ib0.8001 inet 192.168.0.2/24

  Test from server1(ib0.8001/192.168.0.1) to server2(ib0.8001/192.168.0.2)

    ping 192.168.0.2
    no response.

3) Test2 Log

  I enabled debug_log parameter in ib_ipoib modules.

  cd /sys/module/ib_ipoib/parameters
  # cat debug_level
  1

  Could you please tell me what is wrong??

  ib0: enabling connected mode will cause multicast packet drops
  ib0: mtu > 2044 will cause multicast packet drops.
  ib0: mtu > 2044 will cause multicast packet drops.
  ib1: enabling connected mode will cause multicast packet drops
  ib1: mtu > 2044 will cause multicast packet drops.
  ib1: mtu > 2044 will cause multicast packet drops.
  ib0: Event 17 on device mthca0 port 1
  ib0: Not flushing - IPOIB_FLAG_INITIALIZED not set.
  ib0: Event 11 on device mthca0 port 1
  ib0: Not flushing - IPOIB_FLAG_INITIALIZED not set.
  ib0: Event 9 on device mthca0 port 1
  ib0: Not flushing - IPOIB_FLAG_INITIALIZED not set.
  ib0.8001: max_srq_sge=27
  ib0.8001: max_cm_mtu = 0xfff0, num_frags=16
  ib0.8001: bringing up interface
  ib0.8001: IPOIB_FLAG_OPER_UP not set
  ib0.8001: IPOIB_FLAG_OPER_UP not set
  ADDRCONF(NETDEV_UP): ib0.8001: link is not ready
  ib0.8001: IPOIB_FLAG_OPER_UP not set

regards.

2011/10/27 Hal Rosenstock <hal.rosenstock at gmail.com>:
>
>
> On Wed, Oct 26, 2011 at 10:03 AM, Hiroyuki Sato <hiroysato at gmail.com> wrote:
>>
>> Dear members.
>>
>> I have some question about Infiniband Partitions.
>>
>> I would like to build Linux box with IPoIB Router.
>>
>> * Questions
>>
>>  My Question is the following.
>>
>>  a) Can I create the following IPoIB network ??
>>
>>  b) If so, how to setup /etc/opensm/partitions.conf??
>>
>> * Problem.
>>
>>  Ping IB Client to Router : I got the followig error.
>>
>>  Network is unreachable.
>>
>> * Environment
>>
>>  OFED: OFED-1.5.3.2
>>  OS:   Scientific Linux 6.1
>>
>> * Linux box (Router)
>>
>>  1 ethernet interface  (eth0)
>>  1 Infiniband interfae (ib0)
>>  Run subnet manager.
>>
>> * Ethernet setup
>>
>>  a) add VLAN ID 120 eth0.120 (192.168.120.0/24)
>>  b) add VLAN ID 130 eth0.130 (192.168.130.0/24)
>>
>> * Infiniband Setup
>>  a) add subinterface 0 ib0.8000 (192.168.0.0/24)
>>  b) add subinterface 1 ib0.8001 (192.168.1.0/24)
>>  c) Run opensm
>>
>>
>>  I crated subinterface with the following command.
>>
>>  echo 0 > /sys/class/net/ib0/create_child
>>  echo 1 > /sys/class/net/ib0/create_child
>
>
> echo 0x8001 > /sys/class/net/ib0/create_child
> echo 0x8002 > /sys/class/net/ib0/create_child
>
>>
>>  and assign ip address with ifconfig
>>
>> * Logical Diagram
>>
>>                 +----------- 192.168.120.0/24 ---
>>                 |
>>                 |        +---- 192.168.130.0/24
>>         eth0.120|        | eth0.130
>>                +----------+
>>                | Router   |
>>                +----------+
>>         ib0.8000|        |ib0.8001
>>     192.168.0.1 |        +---- 192.168.1.0/24
>>                 |
>>                 +----------- 192.168.0.0/24 --- IB Cllient. (ib0.8000)
>>                                                 (192.168.0.2)
>>
>> * /etc/opensm/partitions.conf
>>
>>  Default=0x7fff   : ALL=full
>>  Net0=0x00, ipoib : ALL=full
>>  Net1=0x01, ipoib : ALL=full
>
>
> Default=0x7fff: ALL=full
> Net0=0x0001, ipoib: ALL=full
> Net1=0x0002, ipoib: ALL=full
>
>
>>
>> Coule you please tell me what is wrong??
>
>
> The one thing in the above I'm not sure of is whether the default partition
> also needs the ipoib flag above so you might need to add that too if the
> above doesn't work.
>
> -- Hal
>
>>
>> Sincerely.
>>
>> --
>> Hiroyuki Sato.
>> _______________________________________________
>> ewg mailing list
>> ewg at lists.openfabrics.org
>> http://lists.openfabrics.org/cgi-bin/mailman/listinfo/ewg
>
>



-- 
Hiroyuki Sato