[ewg] [Q] how to setup /etc/opensm/partitions.conf??

Hal Rosenstock hal.rosenstock at gmail.com
Thu Oct 27 04:58:38 PDT 2011


On Thu, Oct 27, 2011 at 7:48 AM, Hiroyuki Sato <hiroysato at gmail.com> wrote:

> Hello Hal
>
> Thank you for your information.
>
> I read this documents.  PP 45.
>
> http://www.mellanox.com/related-docs/prod_software/Mellanox_OFED_Linux_user_manual_1_5_1.pdf
>
>  Decide on the PKey to be used in the subnet.
>  Valid values are 0-255.
>

Valid values for pkey are 1-32767 (1-0x7ffff).


>  The actual PKey used is a 16-bit number with
>  the most significant bit set.
>  For example, a value of 0 will give a
>  PKey with the value 0x8000.
>
> Is this wrong??
>

Pkey 0/0x8000 is invalid so this is wrong. What it's showing about turning
on the full membership bit in the pkey is correct. A better example would be
showing a value of 1 being transformed into 0x8001 pkey.

-- Hal


>
>
> 2011/10/27 Hal Rosenstock <hal.rosenstock at gmail.com>:
> >
> >
> > On Thu, Oct 27, 2011 at 7:05 AM, Hiroyuki Sato <hiroysato at gmail.com>
> wrote:
> >>
> >> Hello Hal and Richard.
> >>
> >> I found what is the problem.
> >>
> >> Q: Can I create subinterface with 0 (0x8000)???
> >
> >
> > That uses an invalid pkey so this is not a valid configuration.
> >
> > -- Hal
> >
> >>
> >> My Test result.
> >>
> >>  partitions.conf
> >>
> >>   Default=0x7fff,        : ALL=full  ;
> >>   Net0=0x0000,     ipoib : ALL=full ;
> >>   Net1=0x0001,     ipoib : ALL=full ;
> >>   Net2=0x0002,     ipoib : ALL=full ;
> >>
> >>  1) Server1 ib.8000 -> Server1 ib.8000
> >>    Ping NG
> >>
> >>  2) Server1 ib.8001 -> Server1 ib.8001
> >>
> >>  The differece is subinterfae number.
> >>
> >> Yesterday,  I always tested subinterface 0x8000.
> >>
> >>
> >>
> >> And about previous post,
> >>
> >> The partitions.conf which I posted few hous ago,
> >> was semi colon missing.
> >>
> >> >    * partitions.conf is the following
> >> >      Default=0x7fff,  ipoib : ALL=full
> >> >      Net0=0x0001,     ipoib : ALL=full
> >>
> >> Should be
> >>
> >>       Default=0x7fff,  ipoib : ALL=full ;
> >>       Net0=0x0001,     ipoib : ALL=full ;
> >>
> >> Thank you again.
> >>
> >>
> >> 2011/10/27 Hiroyuki Sato <hiroysato at gmail.com>:
> >> > Hello Hal
> >> >
> >> > Thank you for your information.
> >> >
> >> > I simplified my test environment
> >> > The environment is the folloing.
> >> >
> >> > I'm not sure what is wrong Test2 (Partition test)
> >> > And Could you please tell me how to check the problem??
> >> > (tool, logfile, etc.. )
> >> >
> >> > * Environment
> >> >
> >> >  OS: Scientific Linux6.1
> >> >  OFED: 1.5.3.
> >> >
> >> > * Diagram
> >> >
> >> >  +----------+ib0            ib0+----------+
> >> >  | Server1  |------------------| Server2  |
> >> >  +----------+                  +----------+
> >> >
> >> >
> >> >
> >> >
> >> > 1) Test1 (Simple IPoIB no partition Ping test)
> >> >
> >> >  Server1
> >> >
> >> >    a) /sbin/ifconfig ib0 inet 192.168.1.1/24
> >> >    b) /sbn/service opensmd start
> >> >    3) /sbin/service opensmd start
> >> >    * no /etc/opensm/partitions.conf
> >> >
> >> >  Server2
> >> >    /sbin/ifconfig ib0 inet 192.168.1.1/24
> >> >
> >> >  Test from server1(192.168.1.1) to server2(192.168.1.2)
> >> >    ping 192.168.1.2
> >> >    The result was OK.
> >> >
> >> > 2) Test2 (Add partition by hand and Ping test)
> >> >
> >> >  reboot Server 1 and Server 2
> >> >
> >> >  Server 1
> >> >    1) echo 0x8001 > /sys/class/net/ib0/create_child
> >> >    2) /sbin/ifconfig ib0.8001 inet 192.168.0.1/24
> >> >    3) /sbin/service opensmd start
> >> >
> >> >    * partitions.conf is the following
> >> >      Default=0x7fff,  ipoib : ALL=full
> >> >      Net0=0x0001,     ipoib : ALL=full
> >> >
> >> >  Server2
> >> >    1) echo 0x8001 > /sys/class/net/ib0/create_child
> >> >    2) /sbin/ifconfig ib0.8001 inet 192.168.0.2/24
> >> >
> >> >  Test from server1(ib0.8001/192.168.0.1) to
> >> > server2(ib0.8001/192.168.0.2)
> >> >
> >> >    ping 192.168.0.2
> >> >    no response.
> >> >
> >> > 3) Test2 Log
> >> >
> >> >  I enabled debug_log parameter in ib_ipoib modules.
> >> >
> >> >  cd /sys/module/ib_ipoib/parameters
> >> >  # cat debug_level
> >> >  1
> >> >
> >> >  Could you please tell me what is wrong??
> >> >
> >> >  ib0: enabling connected mode will cause multicast packet drops
> >> >  ib0: mtu > 2044 will cause multicast packet drops.
> >> >  ib0: mtu > 2044 will cause multicast packet drops.
> >> >  ib1: enabling connected mode will cause multicast packet drops
> >> >  ib1: mtu > 2044 will cause multicast packet drops.
> >> >  ib1: mtu > 2044 will cause multicast packet drops.
> >> >  ib0: Event 17 on device mthca0 port 1
> >> >  ib0: Not flushing - IPOIB_FLAG_INITIALIZED not set.
> >> >  ib0: Event 11 on device mthca0 port 1
> >> >  ib0: Not flushing - IPOIB_FLAG_INITIALIZED not set.
> >> >  ib0: Event 9 on device mthca0 port 1
> >> >  ib0: Not flushing - IPOIB_FLAG_INITIALIZED not set.
> >> >  ib0.8001: max_srq_sge=27
> >> >  ib0.8001: max_cm_mtu = 0xfff0, num_frags=16
> >> >  ib0.8001: bringing up interface
> >> >  ib0.8001: IPOIB_FLAG_OPER_UP not set
> >> >  ib0.8001: IPOIB_FLAG_OPER_UP not set
> >> >  ADDRCONF(NETDEV_UP): ib0.8001: link is not ready
> >> >  ib0.8001: IPOIB_FLAG_OPER_UP not set
> >> >
> >> > regards.
> >> >
> >> > 2011/10/27 Hal Rosenstock <hal.rosenstock at gmail.com>:
> >> >>
> >> >>
> >> >> On Wed, Oct 26, 2011 at 10:03 AM, Hiroyuki Sato <hiroysato at gmail.com
> >
> >> >> wrote:
> >> >>>
> >> >>> Dear members.
> >> >>>
> >> >>> I have some question about Infiniband Partitions.
> >> >>>
> >> >>> I would like to build Linux box with IPoIB Router.
> >> >>>
> >> >>> * Questions
> >> >>>
> >> >>>  My Question is the following.
> >> >>>
> >> >>>  a) Can I create the following IPoIB network ??
> >> >>>
> >> >>>  b) If so, how to setup /etc/opensm/partitions.conf??
> >> >>>
> >> >>> * Problem.
> >> >>>
> >> >>>  Ping IB Client to Router : I got the followig error.
> >> >>>
> >> >>>  Network is unreachable.
> >> >>>
> >> >>> * Environment
> >> >>>
> >> >>>  OFED: OFED-1.5.3.2
> >> >>>  OS:   Scientific Linux 6.1
> >> >>>
> >> >>> * Linux box (Router)
> >> >>>
> >> >>>  1 ethernet interface  (eth0)
> >> >>>  1 Infiniband interfae (ib0)
> >> >>>  Run subnet manager.
> >> >>>
> >> >>> * Ethernet setup
> >> >>>
> >> >>>  a) add VLAN ID 120 eth0.120 (192.168.120.0/24)
> >> >>>  b) add VLAN ID 130 eth0.130 (192.168.130.0/24)
> >> >>>
> >> >>> * Infiniband Setup
> >> >>>  a) add subinterface 0 ib0.8000 (192.168.0.0/24)
> >> >>>  b) add subinterface 1 ib0.8001 (192.168.1.0/24)
> >> >>>  c) Run opensm
> >> >>>
> >> >>>
> >> >>>  I crated subinterface with the following command.
> >> >>>
> >> >>>  echo 0 > /sys/class/net/ib0/create_child
> >> >>>  echo 1 > /sys/class/net/ib0/create_child
> >> >>
> >> >>
> >> >> echo 0x8001 > /sys/class/net/ib0/create_child
> >> >> echo 0x8002 > /sys/class/net/ib0/create_child
> >> >>
> >> >>>
> >> >>>  and assign ip address with ifconfig
> >> >>>
> >> >>> * Logical Diagram
> >> >>>
> >> >>>                 +----------- 192.168.120.0/24 ---
> >> >>>                 |
> >> >>>                 |        +---- 192.168.130.0/24
> >> >>>         eth0.120|        | eth0.130
> >> >>>                +----------+
> >> >>>                | Router   |
> >> >>>                +----------+
> >> >>>         ib0.8000|        |ib0.8001
> >> >>>     192.168.0.1 |        +---- 192.168.1.0/24
> >> >>>                 |
> >> >>>                 +----------- 192.168.0.0/24 --- IB Cllient.
> (ib0.8000)
> >> >>>                                                 (192.168.0.2)
> >> >>>
> >> >>> * /etc/opensm/partitions.conf
> >> >>>
> >> >>>  Default=0x7fff   : ALL=full
> >> >>>  Net0=0x00, ipoib : ALL=full
> >> >>>  Net1=0x01, ipoib : ALL=full
> >> >>
> >> >>
> >> >> Default=0x7fff: ALL=full
> >> >> Net0=0x0001, ipoib: ALL=full
> >> >> Net1=0x0002, ipoib: ALL=full
> >> >>
> >> >>
> >> >>>
> >> >>> Coule you please tell me what is wrong??
> >> >>
> >> >>
> >> >> The one thing in the above I'm not sure of is whether the default
> >> >> partition
> >> >> also needs the ipoib flag above so you might need to add that too if
> >> >> the
> >> >> above doesn't work.
> >> >>
> >> >> -- Hal
> >> >>
> >> >>>
> >> >>> Sincerely.
> >> >>>
> >> >>> --
> >> >>> Hiroyuki Sato.
> >> >>> _______________________________________________
> >> >>> ewg mailing list
> >> >>> ewg at lists.openfabrics.org
> >> >>> http://lists.openfabrics.org/cgi-bin/mailman/listinfo/ewg
> >> >>
> >> >>
> >> >
> >> >
> >> >
> >> > --
> >> > Hiroyuki Sato
> >> >
> >>
> >>
> >>
> >> --
> >> Hiroyuki Sato
> >
> >
>
>
>
> --
> Hiroyuki Sato
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.openfabrics.org/pipermail/ewg/attachments/20111027/a3a2326e/attachment.html>


More information about the ewg mailing list