[ewg] [Q] how to setup /etc/opensm/partitions.conf??
Hiroyuki Sato
hiroysato at gmail.com
Thu Oct 27 04:48:33 PDT 2011
Hello Hal
Thank you for your information.
I read this documents. PP 45.
http://www.mellanox.com/related-docs/prod_software/Mellanox_OFED_Linux_user_manual_1_5_1.pdf
Decide on the PKey to be used in the subnet.
Valid values are 0-255.
The actual PKey used is a 16-bit number with
the most significant bit set.
For example, a value of 0 will give a
PKey with the value 0x8000.
Is this wrong??
2011/10/27 Hal Rosenstock <hal.rosenstock at gmail.com>:
>
>
> On Thu, Oct 27, 2011 at 7:05 AM, Hiroyuki Sato <hiroysato at gmail.com> wrote:
>>
>> Hello Hal and Richard.
>>
>> I found what is the problem.
>>
>> Q: Can I create subinterface with 0 (0x8000)???
>
>
> That uses an invalid pkey so this is not a valid configuration.
>
> -- Hal
>
>>
>> My Test result.
>>
>> partitions.conf
>>
>> Default=0x7fff, : ALL=full ;
>> Net0=0x0000, ipoib : ALL=full ;
>> Net1=0x0001, ipoib : ALL=full ;
>> Net2=0x0002, ipoib : ALL=full ;
>>
>> 1) Server1 ib.8000 -> Server1 ib.8000
>> Ping NG
>>
>> 2) Server1 ib.8001 -> Server1 ib.8001
>>
>> The differece is subinterfae number.
>>
>> Yesterday, I always tested subinterface 0x8000.
>>
>>
>>
>> And about previous post,
>>
>> The partitions.conf which I posted few hous ago,
>> was semi colon missing.
>>
>> > * partitions.conf is the following
>> > Default=0x7fff, ipoib : ALL=full
>> > Net0=0x0001, ipoib : ALL=full
>>
>> Should be
>>
>> Default=0x7fff, ipoib : ALL=full ;
>> Net0=0x0001, ipoib : ALL=full ;
>>
>> Thank you again.
>>
>>
>> 2011/10/27 Hiroyuki Sato <hiroysato at gmail.com>:
>> > Hello Hal
>> >
>> > Thank you for your information.
>> >
>> > I simplified my test environment
>> > The environment is the folloing.
>> >
>> > I'm not sure what is wrong Test2 (Partition test)
>> > And Could you please tell me how to check the problem??
>> > (tool, logfile, etc.. )
>> >
>> > * Environment
>> >
>> > OS: Scientific Linux6.1
>> > OFED: 1.5.3.
>> >
>> > * Diagram
>> >
>> > +----------+ib0 ib0+----------+
>> > | Server1 |------------------| Server2 |
>> > +----------+ +----------+
>> >
>> >
>> >
>> >
>> > 1) Test1 (Simple IPoIB no partition Ping test)
>> >
>> > Server1
>> >
>> > a) /sbin/ifconfig ib0 inet 192.168.1.1/24
>> > b) /sbn/service opensmd start
>> > 3) /sbin/service opensmd start
>> > * no /etc/opensm/partitions.conf
>> >
>> > Server2
>> > /sbin/ifconfig ib0 inet 192.168.1.1/24
>> >
>> > Test from server1(192.168.1.1) to server2(192.168.1.2)
>> > ping 192.168.1.2
>> > The result was OK.
>> >
>> > 2) Test2 (Add partition by hand and Ping test)
>> >
>> > reboot Server 1 and Server 2
>> >
>> > Server 1
>> > 1) echo 0x8001 > /sys/class/net/ib0/create_child
>> > 2) /sbin/ifconfig ib0.8001 inet 192.168.0.1/24
>> > 3) /sbin/service opensmd start
>> >
>> > * partitions.conf is the following
>> > Default=0x7fff, ipoib : ALL=full
>> > Net0=0x0001, ipoib : ALL=full
>> >
>> > Server2
>> > 1) echo 0x8001 > /sys/class/net/ib0/create_child
>> > 2) /sbin/ifconfig ib0.8001 inet 192.168.0.2/24
>> >
>> > Test from server1(ib0.8001/192.168.0.1) to
>> > server2(ib0.8001/192.168.0.2)
>> >
>> > ping 192.168.0.2
>> > no response.
>> >
>> > 3) Test2 Log
>> >
>> > I enabled debug_log parameter in ib_ipoib modules.
>> >
>> > cd /sys/module/ib_ipoib/parameters
>> > # cat debug_level
>> > 1
>> >
>> > Could you please tell me what is wrong??
>> >
>> > ib0: enabling connected mode will cause multicast packet drops
>> > ib0: mtu > 2044 will cause multicast packet drops.
>> > ib0: mtu > 2044 will cause multicast packet drops.
>> > ib1: enabling connected mode will cause multicast packet drops
>> > ib1: mtu > 2044 will cause multicast packet drops.
>> > ib1: mtu > 2044 will cause multicast packet drops.
>> > ib0: Event 17 on device mthca0 port 1
>> > ib0: Not flushing - IPOIB_FLAG_INITIALIZED not set.
>> > ib0: Event 11 on device mthca0 port 1
>> > ib0: Not flushing - IPOIB_FLAG_INITIALIZED not set.
>> > ib0: Event 9 on device mthca0 port 1
>> > ib0: Not flushing - IPOIB_FLAG_INITIALIZED not set.
>> > ib0.8001: max_srq_sge=27
>> > ib0.8001: max_cm_mtu = 0xfff0, num_frags=16
>> > ib0.8001: bringing up interface
>> > ib0.8001: IPOIB_FLAG_OPER_UP not set
>> > ib0.8001: IPOIB_FLAG_OPER_UP not set
>> > ADDRCONF(NETDEV_UP): ib0.8001: link is not ready
>> > ib0.8001: IPOIB_FLAG_OPER_UP not set
>> >
>> > regards.
>> >
>> > 2011/10/27 Hal Rosenstock <hal.rosenstock at gmail.com>:
>> >>
>> >>
>> >> On Wed, Oct 26, 2011 at 10:03 AM, Hiroyuki Sato <hiroysato at gmail.com>
>> >> wrote:
>> >>>
>> >>> Dear members.
>> >>>
>> >>> I have some question about Infiniband Partitions.
>> >>>
>> >>> I would like to build Linux box with IPoIB Router.
>> >>>
>> >>> * Questions
>> >>>
>> >>> My Question is the following.
>> >>>
>> >>> a) Can I create the following IPoIB network ??
>> >>>
>> >>> b) If so, how to setup /etc/opensm/partitions.conf??
>> >>>
>> >>> * Problem.
>> >>>
>> >>> Ping IB Client to Router : I got the followig error.
>> >>>
>> >>> Network is unreachable.
>> >>>
>> >>> * Environment
>> >>>
>> >>> OFED: OFED-1.5.3.2
>> >>> OS: Scientific Linux 6.1
>> >>>
>> >>> * Linux box (Router)
>> >>>
>> >>> 1 ethernet interface (eth0)
>> >>> 1 Infiniband interfae (ib0)
>> >>> Run subnet manager.
>> >>>
>> >>> * Ethernet setup
>> >>>
>> >>> a) add VLAN ID 120 eth0.120 (192.168.120.0/24)
>> >>> b) add VLAN ID 130 eth0.130 (192.168.130.0/24)
>> >>>
>> >>> * Infiniband Setup
>> >>> a) add subinterface 0 ib0.8000 (192.168.0.0/24)
>> >>> b) add subinterface 1 ib0.8001 (192.168.1.0/24)
>> >>> c) Run opensm
>> >>>
>> >>>
>> >>> I crated subinterface with the following command.
>> >>>
>> >>> echo 0 > /sys/class/net/ib0/create_child
>> >>> echo 1 > /sys/class/net/ib0/create_child
>> >>
>> >>
>> >> echo 0x8001 > /sys/class/net/ib0/create_child
>> >> echo 0x8002 > /sys/class/net/ib0/create_child
>> >>
>> >>>
>> >>> and assign ip address with ifconfig
>> >>>
>> >>> * Logical Diagram
>> >>>
>> >>> +----------- 192.168.120.0/24 ---
>> >>> |
>> >>> | +---- 192.168.130.0/24
>> >>> eth0.120| | eth0.130
>> >>> +----------+
>> >>> | Router |
>> >>> +----------+
>> >>> ib0.8000| |ib0.8001
>> >>> 192.168.0.1 | +---- 192.168.1.0/24
>> >>> |
>> >>> +----------- 192.168.0.0/24 --- IB Cllient. (ib0.8000)
>> >>> (192.168.0.2)
>> >>>
>> >>> * /etc/opensm/partitions.conf
>> >>>
>> >>> Default=0x7fff : ALL=full
>> >>> Net0=0x00, ipoib : ALL=full
>> >>> Net1=0x01, ipoib : ALL=full
>> >>
>> >>
>> >> Default=0x7fff: ALL=full
>> >> Net0=0x0001, ipoib: ALL=full
>> >> Net1=0x0002, ipoib: ALL=full
>> >>
>> >>
>> >>>
>> >>> Coule you please tell me what is wrong??
>> >>
>> >>
>> >> The one thing in the above I'm not sure of is whether the default
>> >> partition
>> >> also needs the ipoib flag above so you might need to add that too if
>> >> the
>> >> above doesn't work.
>> >>
>> >> -- Hal
>> >>
>> >>>
>> >>> Sincerely.
>> >>>
>> >>> --
>> >>> Hiroyuki Sato.
>> >>> _______________________________________________
>> >>> ewg mailing list
>> >>> ewg at lists.openfabrics.org
>> >>> http://lists.openfabrics.org/cgi-bin/mailman/listinfo/ewg
>> >>
>> >>
>> >
>> >
>> >
>> > --
>> > Hiroyuki Sato
>> >
>>
>>
>>
>> --
>> Hiroyuki Sato
>
>
--
Hiroyuki Sato
More information about the ewg
mailing list