[ewg] [Q] how to setup /etc/opensm/partitions.conf??
Hiroyuki Sato
hiroysato at gmail.com
Thu Oct 27 04:05:13 PDT 2011
Hello Hal and Richard.
I found what is the problem.
Q: Can I create subinterface with 0 (0x8000)???
My Test result.
partitions.conf
Default=0x7fff, : ALL=full ;
Net0=0x0000, ipoib : ALL=full ;
Net1=0x0001, ipoib : ALL=full ;
Net2=0x0002, ipoib : ALL=full ;
1) Server1 ib.8000 -> Server1 ib.8000
Ping NG
2) Server1 ib.8001 -> Server1 ib.8001
The differece is subinterfae number.
Yesterday, I always tested subinterface 0x8000.
And about previous post,
The partitions.conf which I posted few hous ago,
was semi colon missing.
> * partitions.conf is the following
> Default=0x7fff, ipoib : ALL=full
> Net0=0x0001, ipoib : ALL=full
Should be
Default=0x7fff, ipoib : ALL=full ;
Net0=0x0001, ipoib : ALL=full ;
Thank you again.
2011/10/27 Hiroyuki Sato <hiroysato at gmail.com>:
> Hello Hal
>
> Thank you for your information.
>
> I simplified my test environment
> The environment is the folloing.
>
> I'm not sure what is wrong Test2 (Partition test)
> And Could you please tell me how to check the problem??
> (tool, logfile, etc.. )
>
> * Environment
>
> OS: Scientific Linux6.1
> OFED: 1.5.3.
>
> * Diagram
>
> +----------+ib0 ib0+----------+
> | Server1 |------------------| Server2 |
> +----------+ +----------+
>
>
>
>
> 1) Test1 (Simple IPoIB no partition Ping test)
>
> Server1
>
> a) /sbin/ifconfig ib0 inet 192.168.1.1/24
> b) /sbn/service opensmd start
> 3) /sbin/service opensmd start
> * no /etc/opensm/partitions.conf
>
> Server2
> /sbin/ifconfig ib0 inet 192.168.1.1/24
>
> Test from server1(192.168.1.1) to server2(192.168.1.2)
> ping 192.168.1.2
> The result was OK.
>
> 2) Test2 (Add partition by hand and Ping test)
>
> reboot Server 1 and Server 2
>
> Server 1
> 1) echo 0x8001 > /sys/class/net/ib0/create_child
> 2) /sbin/ifconfig ib0.8001 inet 192.168.0.1/24
> 3) /sbin/service opensmd start
>
> * partitions.conf is the following
> Default=0x7fff, ipoib : ALL=full
> Net0=0x0001, ipoib : ALL=full
>
> Server2
> 1) echo 0x8001 > /sys/class/net/ib0/create_child
> 2) /sbin/ifconfig ib0.8001 inet 192.168.0.2/24
>
> Test from server1(ib0.8001/192.168.0.1) to server2(ib0.8001/192.168.0.2)
>
> ping 192.168.0.2
> no response.
>
> 3) Test2 Log
>
> I enabled debug_log parameter in ib_ipoib modules.
>
> cd /sys/module/ib_ipoib/parameters
> # cat debug_level
> 1
>
> Could you please tell me what is wrong??
>
> ib0: enabling connected mode will cause multicast packet drops
> ib0: mtu > 2044 will cause multicast packet drops.
> ib0: mtu > 2044 will cause multicast packet drops.
> ib1: enabling connected mode will cause multicast packet drops
> ib1: mtu > 2044 will cause multicast packet drops.
> ib1: mtu > 2044 will cause multicast packet drops.
> ib0: Event 17 on device mthca0 port 1
> ib0: Not flushing - IPOIB_FLAG_INITIALIZED not set.
> ib0: Event 11 on device mthca0 port 1
> ib0: Not flushing - IPOIB_FLAG_INITIALIZED not set.
> ib0: Event 9 on device mthca0 port 1
> ib0: Not flushing - IPOIB_FLAG_INITIALIZED not set.
> ib0.8001: max_srq_sge=27
> ib0.8001: max_cm_mtu = 0xfff0, num_frags=16
> ib0.8001: bringing up interface
> ib0.8001: IPOIB_FLAG_OPER_UP not set
> ib0.8001: IPOIB_FLAG_OPER_UP not set
> ADDRCONF(NETDEV_UP): ib0.8001: link is not ready
> ib0.8001: IPOIB_FLAG_OPER_UP not set
>
> regards.
>
> 2011/10/27 Hal Rosenstock <hal.rosenstock at gmail.com>:
>>
>>
>> On Wed, Oct 26, 2011 at 10:03 AM, Hiroyuki Sato <hiroysato at gmail.com> wrote:
>>>
>>> Dear members.
>>>
>>> I have some question about Infiniband Partitions.
>>>
>>> I would like to build Linux box with IPoIB Router.
>>>
>>> * Questions
>>>
>>> My Question is the following.
>>>
>>> a) Can I create the following IPoIB network ??
>>>
>>> b) If so, how to setup /etc/opensm/partitions.conf??
>>>
>>> * Problem.
>>>
>>> Ping IB Client to Router : I got the followig error.
>>>
>>> Network is unreachable.
>>>
>>> * Environment
>>>
>>> OFED: OFED-1.5.3.2
>>> OS: Scientific Linux 6.1
>>>
>>> * Linux box (Router)
>>>
>>> 1 ethernet interface (eth0)
>>> 1 Infiniband interfae (ib0)
>>> Run subnet manager.
>>>
>>> * Ethernet setup
>>>
>>> a) add VLAN ID 120 eth0.120 (192.168.120.0/24)
>>> b) add VLAN ID 130 eth0.130 (192.168.130.0/24)
>>>
>>> * Infiniband Setup
>>> a) add subinterface 0 ib0.8000 (192.168.0.0/24)
>>> b) add subinterface 1 ib0.8001 (192.168.1.0/24)
>>> c) Run opensm
>>>
>>>
>>> I crated subinterface with the following command.
>>>
>>> echo 0 > /sys/class/net/ib0/create_child
>>> echo 1 > /sys/class/net/ib0/create_child
>>
>>
>> echo 0x8001 > /sys/class/net/ib0/create_child
>> echo 0x8002 > /sys/class/net/ib0/create_child
>>
>>>
>>> and assign ip address with ifconfig
>>>
>>> * Logical Diagram
>>>
>>> +----------- 192.168.120.0/24 ---
>>> |
>>> | +---- 192.168.130.0/24
>>> eth0.120| | eth0.130
>>> +----------+
>>> | Router |
>>> +----------+
>>> ib0.8000| |ib0.8001
>>> 192.168.0.1 | +---- 192.168.1.0/24
>>> |
>>> +----------- 192.168.0.0/24 --- IB Cllient. (ib0.8000)
>>> (192.168.0.2)
>>>
>>> * /etc/opensm/partitions.conf
>>>
>>> Default=0x7fff : ALL=full
>>> Net0=0x00, ipoib : ALL=full
>>> Net1=0x01, ipoib : ALL=full
>>
>>
>> Default=0x7fff: ALL=full
>> Net0=0x0001, ipoib: ALL=full
>> Net1=0x0002, ipoib: ALL=full
>>
>>
>>>
>>> Coule you please tell me what is wrong??
>>
>>
>> The one thing in the above I'm not sure of is whether the default partition
>> also needs the ipoib flag above so you might need to add that too if the
>> above doesn't work.
>>
>> -- Hal
>>
>>>
>>> Sincerely.
>>>
>>> --
>>> Hiroyuki Sato.
>>> _______________________________________________
>>> ewg mailing list
>>> ewg at lists.openfabrics.org
>>> http://lists.openfabrics.org/cgi-bin/mailman/listinfo/ewg
>>
>>
>
>
>
> --
> Hiroyuki Sato
>
--
Hiroyuki Sato
More information about the ewg
mailing list