[ewg] [Q] how to setup /etc/opensm/partitions.conf??
Hiroyuki Sato
hiroysato at gmail.com
Thu Oct 27 05:06:18 PDT 2011
Hello Hal.
Thank you for your information.
I'll feedback to Mellanox.
2011/10/27 Hal Rosenstock <hal.rosenstock at gmail.com>:
>
>
> On Thu, Oct 27, 2011 at 7:48 AM, Hiroyuki Sato <hiroysato at gmail.com> wrote:
>>
>> Hello Hal
>>
>> Thank you for your information.
>>
>> I read this documents. PP 45.
>>
>> http://www.mellanox.com/related-docs/prod_software/Mellanox_OFED_Linux_user_manual_1_5_1.pdf
>>
>> Decide on the PKey to be used in the subnet.
>> Valid values are 0-255.
>
>
> Valid values for pkey are 1-32767 (1-0x7ffff).
>
>>
>> The actual PKey used is a 16-bit number with
>> the most significant bit set.
>> For example, a value of 0 will give a
>> PKey with the value 0x8000.
>>
>> Is this wrong??
>
>
> Pkey 0/0x8000 is invalid so this is wrong. What it's showing about turning
> on the full membership bit in the pkey is correct. A better example would be
> showing a value of 1 being transformed into 0x8001 pkey.
>
> -- Hal
>
>>
>> 2011/10/27 Hal Rosenstock <hal.rosenstock at gmail.com>:
>> >
>> >
>> > On Thu, Oct 27, 2011 at 7:05 AM, Hiroyuki Sato <hiroysato at gmail.com>
>> > wrote:
>> >>
>> >> Hello Hal and Richard.
>> >>
>> >> I found what is the problem.
>> >>
>> >> Q: Can I create subinterface with 0 (0x8000)???
>> >
>> >
>> > That uses an invalid pkey so this is not a valid configuration.
>> >
>> > -- Hal
>> >
>> >>
>> >> My Test result.
>> >>
>> >> partitions.conf
>> >>
>> >> Default=0x7fff, : ALL=full ;
>> >> Net0=0x0000, ipoib : ALL=full ;
>> >> Net1=0x0001, ipoib : ALL=full ;
>> >> Net2=0x0002, ipoib : ALL=full ;
>> >>
>> >> 1) Server1 ib.8000 -> Server1 ib.8000
>> >> Ping NG
>> >>
>> >> 2) Server1 ib.8001 -> Server1 ib.8001
>> >>
>> >> The differece is subinterfae number.
>> >>
>> >> Yesterday, I always tested subinterface 0x8000.
>> >>
>> >>
>> >>
>> >> And about previous post,
>> >>
>> >> The partitions.conf which I posted few hous ago,
>> >> was semi colon missing.
>> >>
>> >> > * partitions.conf is the following
>> >> > Default=0x7fff, ipoib : ALL=full
>> >> > Net0=0x0001, ipoib : ALL=full
>> >>
>> >> Should be
>> >>
>> >> Default=0x7fff, ipoib : ALL=full ;
>> >> Net0=0x0001, ipoib : ALL=full ;
>> >>
>> >> Thank you again.
>> >>
>> >>
>> >> 2011/10/27 Hiroyuki Sato <hiroysato at gmail.com>:
>> >> > Hello Hal
>> >> >
>> >> > Thank you for your information.
>> >> >
>> >> > I simplified my test environment
>> >> > The environment is the folloing.
>> >> >
>> >> > I'm not sure what is wrong Test2 (Partition test)
>> >> > And Could you please tell me how to check the problem??
>> >> > (tool, logfile, etc.. )
>> >> >
>> >> > * Environment
>> >> >
>> >> > OS: Scientific Linux6.1
>> >> > OFED: 1.5.3.
>> >> >
>> >> > * Diagram
>> >> >
>> >> > +----------+ib0 ib0+----------+
>> >> > | Server1 |------------------| Server2 |
>> >> > +----------+ +----------+
>> >> >
>> >> >
>> >> >
>> >> >
>> >> > 1) Test1 (Simple IPoIB no partition Ping test)
>> >> >
>> >> > Server1
>> >> >
>> >> > a) /sbin/ifconfig ib0 inet 192.168.1.1/24
>> >> > b) /sbn/service opensmd start
>> >> > 3) /sbin/service opensmd start
>> >> > * no /etc/opensm/partitions.conf
>> >> >
>> >> > Server2
>> >> > /sbin/ifconfig ib0 inet 192.168.1.1/24
>> >> >
>> >> > Test from server1(192.168.1.1) to server2(192.168.1.2)
>> >> > ping 192.168.1.2
>> >> > The result was OK.
>> >> >
>> >> > 2) Test2 (Add partition by hand and Ping test)
>> >> >
>> >> > reboot Server 1 and Server 2
>> >> >
>> >> > Server 1
>> >> > 1) echo 0x8001 > /sys/class/net/ib0/create_child
>> >> > 2) /sbin/ifconfig ib0.8001 inet 192.168.0.1/24
>> >> > 3) /sbin/service opensmd start
>> >> >
>> >> > * partitions.conf is the following
>> >> > Default=0x7fff, ipoib : ALL=full
>> >> > Net0=0x0001, ipoib : ALL=full
>> >> >
>> >> > Server2
>> >> > 1) echo 0x8001 > /sys/class/net/ib0/create_child
>> >> > 2) /sbin/ifconfig ib0.8001 inet 192.168.0.2/24
>> >> >
>> >> > Test from server1(ib0.8001/192.168.0.1) to
>> >> > server2(ib0.8001/192.168.0.2)
>> >> >
>> >> > ping 192.168.0.2
>> >> > no response.
>> >> >
>> >> > 3) Test2 Log
>> >> >
>> >> > I enabled debug_log parameter in ib_ipoib modules.
>> >> >
>> >> > cd /sys/module/ib_ipoib/parameters
>> >> > # cat debug_level
>> >> > 1
>> >> >
>> >> > Could you please tell me what is wrong??
>> >> >
>> >> > ib0: enabling connected mode will cause multicast packet drops
>> >> > ib0: mtu > 2044 will cause multicast packet drops.
>> >> > ib0: mtu > 2044 will cause multicast packet drops.
>> >> > ib1: enabling connected mode will cause multicast packet drops
>> >> > ib1: mtu > 2044 will cause multicast packet drops.
>> >> > ib1: mtu > 2044 will cause multicast packet drops.
>> >> > ib0: Event 17 on device mthca0 port 1
>> >> > ib0: Not flushing - IPOIB_FLAG_INITIALIZED not set.
>> >> > ib0: Event 11 on device mthca0 port 1
>> >> > ib0: Not flushing - IPOIB_FLAG_INITIALIZED not set.
>> >> > ib0: Event 9 on device mthca0 port 1
>> >> > ib0: Not flushing - IPOIB_FLAG_INITIALIZED not set.
>> >> > ib0.8001: max_srq_sge=27
>> >> > ib0.8001: max_cm_mtu = 0xfff0, num_frags=16
>> >> > ib0.8001: bringing up interface
>> >> > ib0.8001: IPOIB_FLAG_OPER_UP not set
>> >> > ib0.8001: IPOIB_FLAG_OPER_UP not set
>> >> > ADDRCONF(NETDEV_UP): ib0.8001: link is not ready
>> >> > ib0.8001: IPOIB_FLAG_OPER_UP not set
>> >> >
>> >> > regards.
>> >> >
>> >> > 2011/10/27 Hal Rosenstock <hal.rosenstock at gmail.com>:
>> >> >>
>> >> >>
>> >> >> On Wed, Oct 26, 2011 at 10:03 AM, Hiroyuki Sato
>> >> >> <hiroysato at gmail.com>
>> >> >> wrote:
>> >> >>>
>> >> >>> Dear members.
>> >> >>>
>> >> >>> I have some question about Infiniband Partitions.
>> >> >>>
>> >> >>> I would like to build Linux box with IPoIB Router.
>> >> >>>
>> >> >>> * Questions
>> >> >>>
>> >> >>> My Question is the following.
>> >> >>>
>> >> >>> a) Can I create the following IPoIB network ??
>> >> >>>
>> >> >>> b) If so, how to setup /etc/opensm/partitions.conf??
>> >> >>>
>> >> >>> * Problem.
>> >> >>>
>> >> >>> Ping IB Client to Router : I got the followig error.
>> >> >>>
>> >> >>> Network is unreachable.
>> >> >>>
>> >> >>> * Environment
>> >> >>>
>> >> >>> OFED: OFED-1.5.3.2
>> >> >>> OS: Scientific Linux 6.1
>> >> >>>
>> >> >>> * Linux box (Router)
>> >> >>>
>> >> >>> 1 ethernet interface (eth0)
>> >> >>> 1 Infiniband interfae (ib0)
>> >> >>> Run subnet manager.
>> >> >>>
>> >> >>> * Ethernet setup
>> >> >>>
>> >> >>> a) add VLAN ID 120 eth0.120 (192.168.120.0/24)
>> >> >>> b) add VLAN ID 130 eth0.130 (192.168.130.0/24)
>> >> >>>
>> >> >>> * Infiniband Setup
>> >> >>> a) add subinterface 0 ib0.8000 (192.168.0.0/24)
>> >> >>> b) add subinterface 1 ib0.8001 (192.168.1.0/24)
>> >> >>> c) Run opensm
>> >> >>>
>> >> >>>
>> >> >>> I crated subinterface with the following command.
>> >> >>>
>> >> >>> echo 0 > /sys/class/net/ib0/create_child
>> >> >>> echo 1 > /sys/class/net/ib0/create_child
>> >> >>
>> >> >>
>> >> >> echo 0x8001 > /sys/class/net/ib0/create_child
>> >> >> echo 0x8002 > /sys/class/net/ib0/create_child
>> >> >>
>> >> >>>
>> >> >>> and assign ip address with ifconfig
>> >> >>>
>> >> >>> * Logical Diagram
>> >> >>>
>> >> >>> +----------- 192.168.120.0/24 ---
>> >> >>> |
>> >> >>> | +---- 192.168.130.0/24
>> >> >>> eth0.120| | eth0.130
>> >> >>> +----------+
>> >> >>> | Router |
>> >> >>> +----------+
>> >> >>> ib0.8000| |ib0.8001
>> >> >>> 192.168.0.1 | +---- 192.168.1.0/24
>> >> >>> |
>> >> >>> +----------- 192.168.0.0/24 --- IB Cllient.
>> >> >>> (ib0.8000)
>> >> >>> (192.168.0.2)
>> >> >>>
>> >> >>> * /etc/opensm/partitions.conf
>> >> >>>
>> >> >>> Default=0x7fff : ALL=full
>> >> >>> Net0=0x00, ipoib : ALL=full
>> >> >>> Net1=0x01, ipoib : ALL=full
>> >> >>
>> >> >>
>> >> >> Default=0x7fff: ALL=full
>> >> >> Net0=0x0001, ipoib: ALL=full
>> >> >> Net1=0x0002, ipoib: ALL=full
>> >> >>
>> >> >>
>> >> >>>
>> >> >>> Coule you please tell me what is wrong??
>> >> >>
>> >> >>
>> >> >> The one thing in the above I'm not sure of is whether the default
>> >> >> partition
>> >> >> also needs the ipoib flag above so you might need to add that too if
>> >> >> the
>> >> >> above doesn't work.
>> >> >>
>> >> >> -- Hal
>> >> >>
>> >> >>>
>> >> >>> Sincerely.
>> >> >>>
>> >> >>> --
>> >> >>> Hiroyuki Sato.
>> >> >>> _______________________________________________
>> >> >>> ewg mailing list
>> >> >>> ewg at lists.openfabrics.org
>> >> >>> http://lists.openfabrics.org/cgi-bin/mailman/listinfo/ewg
>> >> >>
>> >> >>
>> >> >
>> >> >
>> >> >
>> >> > --
>> >> > Hiroyuki Sato
>> >> >
>> >>
>> >>
>> >>
>> >> --
>> >> Hiroyuki Sato
>> >
>> >
>>
>>
>>
>> --
>> Hiroyuki Sato
>
>
--
Hiroyuki Sato
More information about the ewg
mailing list