[ewg] [Q] how to setup /etc/opensm/partitions.conf??
Hal Rosenstock
hal.rosenstock at gmail.com
Thu Oct 27 04:58:38 PDT 2011
On Thu, Oct 27, 2011 at 7:48 AM, Hiroyuki Sato <hiroysato at gmail.com> wrote:
> Hello Hal
>
> Thank you for your information.
>
> I read this documents. PP 45.
>
> http://www.mellanox.com/related-docs/prod_software/Mellanox_OFED_Linux_user_manual_1_5_1.pdf
>
> Decide on the PKey to be used in the subnet.
> Valid values are 0-255.
>
Valid values for pkey are 1-32767 (1-0x7ffff).
> The actual PKey used is a 16-bit number with
> the most significant bit set.
> For example, a value of 0 will give a
> PKey with the value 0x8000.
>
> Is this wrong??
>
Pkey 0/0x8000 is invalid so this is wrong. What it's showing about turning
on the full membership bit in the pkey is correct. A better example would be
showing a value of 1 being transformed into 0x8001 pkey.
-- Hal
>
>
> 2011/10/27 Hal Rosenstock <hal.rosenstock at gmail.com>:
> >
> >
> > On Thu, Oct 27, 2011 at 7:05 AM, Hiroyuki Sato <hiroysato at gmail.com>
> wrote:
> >>
> >> Hello Hal and Richard.
> >>
> >> I found what is the problem.
> >>
> >> Q: Can I create subinterface with 0 (0x8000)???
> >
> >
> > That uses an invalid pkey so this is not a valid configuration.
> >
> > -- Hal
> >
> >>
> >> My Test result.
> >>
> >> partitions.conf
> >>
> >> Default=0x7fff, : ALL=full ;
> >> Net0=0x0000, ipoib : ALL=full ;
> >> Net1=0x0001, ipoib : ALL=full ;
> >> Net2=0x0002, ipoib : ALL=full ;
> >>
> >> 1) Server1 ib.8000 -> Server1 ib.8000
> >> Ping NG
> >>
> >> 2) Server1 ib.8001 -> Server1 ib.8001
> >>
> >> The differece is subinterfae number.
> >>
> >> Yesterday, I always tested subinterface 0x8000.
> >>
> >>
> >>
> >> And about previous post,
> >>
> >> The partitions.conf which I posted few hous ago,
> >> was semi colon missing.
> >>
> >> > * partitions.conf is the following
> >> > Default=0x7fff, ipoib : ALL=full
> >> > Net0=0x0001, ipoib : ALL=full
> >>
> >> Should be
> >>
> >> Default=0x7fff, ipoib : ALL=full ;
> >> Net0=0x0001, ipoib : ALL=full ;
> >>
> >> Thank you again.
> >>
> >>
> >> 2011/10/27 Hiroyuki Sato <hiroysato at gmail.com>:
> >> > Hello Hal
> >> >
> >> > Thank you for your information.
> >> >
> >> > I simplified my test environment
> >> > The environment is the folloing.
> >> >
> >> > I'm not sure what is wrong Test2 (Partition test)
> >> > And Could you please tell me how to check the problem??
> >> > (tool, logfile, etc.. )
> >> >
> >> > * Environment
> >> >
> >> > OS: Scientific Linux6.1
> >> > OFED: 1.5.3.
> >> >
> >> > * Diagram
> >> >
> >> > +----------+ib0 ib0+----------+
> >> > | Server1 |------------------| Server2 |
> >> > +----------+ +----------+
> >> >
> >> >
> >> >
> >> >
> >> > 1) Test1 (Simple IPoIB no partition Ping test)
> >> >
> >> > Server1
> >> >
> >> > a) /sbin/ifconfig ib0 inet 192.168.1.1/24
> >> > b) /sbn/service opensmd start
> >> > 3) /sbin/service opensmd start
> >> > * no /etc/opensm/partitions.conf
> >> >
> >> > Server2
> >> > /sbin/ifconfig ib0 inet 192.168.1.1/24
> >> >
> >> > Test from server1(192.168.1.1) to server2(192.168.1.2)
> >> > ping 192.168.1.2
> >> > The result was OK.
> >> >
> >> > 2) Test2 (Add partition by hand and Ping test)
> >> >
> >> > reboot Server 1 and Server 2
> >> >
> >> > Server 1
> >> > 1) echo 0x8001 > /sys/class/net/ib0/create_child
> >> > 2) /sbin/ifconfig ib0.8001 inet 192.168.0.1/24
> >> > 3) /sbin/service opensmd start
> >> >
> >> > * partitions.conf is the following
> >> > Default=0x7fff, ipoib : ALL=full
> >> > Net0=0x0001, ipoib : ALL=full
> >> >
> >> > Server2
> >> > 1) echo 0x8001 > /sys/class/net/ib0/create_child
> >> > 2) /sbin/ifconfig ib0.8001 inet 192.168.0.2/24
> >> >
> >> > Test from server1(ib0.8001/192.168.0.1) to
> >> > server2(ib0.8001/192.168.0.2)
> >> >
> >> > ping 192.168.0.2
> >> > no response.
> >> >
> >> > 3) Test2 Log
> >> >
> >> > I enabled debug_log parameter in ib_ipoib modules.
> >> >
> >> > cd /sys/module/ib_ipoib/parameters
> >> > # cat debug_level
> >> > 1
> >> >
> >> > Could you please tell me what is wrong??
> >> >
> >> > ib0: enabling connected mode will cause multicast packet drops
> >> > ib0: mtu > 2044 will cause multicast packet drops.
> >> > ib0: mtu > 2044 will cause multicast packet drops.
> >> > ib1: enabling connected mode will cause multicast packet drops
> >> > ib1: mtu > 2044 will cause multicast packet drops.
> >> > ib1: mtu > 2044 will cause multicast packet drops.
> >> > ib0: Event 17 on device mthca0 port 1
> >> > ib0: Not flushing - IPOIB_FLAG_INITIALIZED not set.
> >> > ib0: Event 11 on device mthca0 port 1
> >> > ib0: Not flushing - IPOIB_FLAG_INITIALIZED not set.
> >> > ib0: Event 9 on device mthca0 port 1
> >> > ib0: Not flushing - IPOIB_FLAG_INITIALIZED not set.
> >> > ib0.8001: max_srq_sge=27
> >> > ib0.8001: max_cm_mtu = 0xfff0, num_frags=16
> >> > ib0.8001: bringing up interface
> >> > ib0.8001: IPOIB_FLAG_OPER_UP not set
> >> > ib0.8001: IPOIB_FLAG_OPER_UP not set
> >> > ADDRCONF(NETDEV_UP): ib0.8001: link is not ready
> >> > ib0.8001: IPOIB_FLAG_OPER_UP not set
> >> >
> >> > regards.
> >> >
> >> > 2011/10/27 Hal Rosenstock <hal.rosenstock at gmail.com>:
> >> >>
> >> >>
> >> >> On Wed, Oct 26, 2011 at 10:03 AM, Hiroyuki Sato <hiroysato at gmail.com
> >
> >> >> wrote:
> >> >>>
> >> >>> Dear members.
> >> >>>
> >> >>> I have some question about Infiniband Partitions.
> >> >>>
> >> >>> I would like to build Linux box with IPoIB Router.
> >> >>>
> >> >>> * Questions
> >> >>>
> >> >>> My Question is the following.
> >> >>>
> >> >>> a) Can I create the following IPoIB network ??
> >> >>>
> >> >>> b) If so, how to setup /etc/opensm/partitions.conf??
> >> >>>
> >> >>> * Problem.
> >> >>>
> >> >>> Ping IB Client to Router : I got the followig error.
> >> >>>
> >> >>> Network is unreachable.
> >> >>>
> >> >>> * Environment
> >> >>>
> >> >>> OFED: OFED-1.5.3.2
> >> >>> OS: Scientific Linux 6.1
> >> >>>
> >> >>> * Linux box (Router)
> >> >>>
> >> >>> 1 ethernet interface (eth0)
> >> >>> 1 Infiniband interfae (ib0)
> >> >>> Run subnet manager.
> >> >>>
> >> >>> * Ethernet setup
> >> >>>
> >> >>> a) add VLAN ID 120 eth0.120 (192.168.120.0/24)
> >> >>> b) add VLAN ID 130 eth0.130 (192.168.130.0/24)
> >> >>>
> >> >>> * Infiniband Setup
> >> >>> a) add subinterface 0 ib0.8000 (192.168.0.0/24)
> >> >>> b) add subinterface 1 ib0.8001 (192.168.1.0/24)
> >> >>> c) Run opensm
> >> >>>
> >> >>>
> >> >>> I crated subinterface with the following command.
> >> >>>
> >> >>> echo 0 > /sys/class/net/ib0/create_child
> >> >>> echo 1 > /sys/class/net/ib0/create_child
> >> >>
> >> >>
> >> >> echo 0x8001 > /sys/class/net/ib0/create_child
> >> >> echo 0x8002 > /sys/class/net/ib0/create_child
> >> >>
> >> >>>
> >> >>> and assign ip address with ifconfig
> >> >>>
> >> >>> * Logical Diagram
> >> >>>
> >> >>> +----------- 192.168.120.0/24 ---
> >> >>> |
> >> >>> | +---- 192.168.130.0/24
> >> >>> eth0.120| | eth0.130
> >> >>> +----------+
> >> >>> | Router |
> >> >>> +----------+
> >> >>> ib0.8000| |ib0.8001
> >> >>> 192.168.0.1 | +---- 192.168.1.0/24
> >> >>> |
> >> >>> +----------- 192.168.0.0/24 --- IB Cllient.
> (ib0.8000)
> >> >>> (192.168.0.2)
> >> >>>
> >> >>> * /etc/opensm/partitions.conf
> >> >>>
> >> >>> Default=0x7fff : ALL=full
> >> >>> Net0=0x00, ipoib : ALL=full
> >> >>> Net1=0x01, ipoib : ALL=full
> >> >>
> >> >>
> >> >> Default=0x7fff: ALL=full
> >> >> Net0=0x0001, ipoib: ALL=full
> >> >> Net1=0x0002, ipoib: ALL=full
> >> >>
> >> >>
> >> >>>
> >> >>> Coule you please tell me what is wrong??
> >> >>
> >> >>
> >> >> The one thing in the above I'm not sure of is whether the default
> >> >> partition
> >> >> also needs the ipoib flag above so you might need to add that too if
> >> >> the
> >> >> above doesn't work.
> >> >>
> >> >> -- Hal
> >> >>
> >> >>>
> >> >>> Sincerely.
> >> >>>
> >> >>> --
> >> >>> Hiroyuki Sato.
> >> >>> _______________________________________________
> >> >>> ewg mailing list
> >> >>> ewg at lists.openfabrics.org
> >> >>> http://lists.openfabrics.org/cgi-bin/mailman/listinfo/ewg
> >> >>
> >> >>
> >> >
> >> >
> >> >
> >> > --
> >> > Hiroyuki Sato
> >> >
> >>
> >>
> >>
> >> --
> >> Hiroyuki Sato
> >
> >
>
>
>
> --
> Hiroyuki Sato
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.openfabrics.org/pipermail/ewg/attachments/20111027/a3a2326e/attachment.html>
More information about the ewg
mailing list