[ewg] [Q] how to setup /etc/opensm/partitions.conf??
Hal Rosenstock
hal.rosenstock at gmail.com
Thu Oct 27 04:15:25 PDT 2011
On Thu, Oct 27, 2011 at 7:05 AM, Hiroyuki Sato <hiroysato at gmail.com> wrote:
> Hello Hal and Richard.
>
> I found what is the problem.
>
> Q: Can I create subinterface with 0 (0x8000)???
>
That uses an invalid pkey so this is not a valid configuration.
-- Hal
>
> My Test result.
>
> partitions.conf
>
> Default=0x7fff, : ALL=full ;
> Net0=0x0000, ipoib : ALL=full ;
> Net1=0x0001, ipoib : ALL=full ;
> Net2=0x0002, ipoib : ALL=full ;
>
> 1) Server1 ib.8000 -> Server1 ib.8000
> Ping NG
>
> 2) Server1 ib.8001 -> Server1 ib.8001
>
> The differece is subinterfae number.
>
> Yesterday, I always tested subinterface 0x8000.
>
>
>
> And about previous post,
>
> The partitions.conf which I posted few hous ago,
> was semi colon missing.
>
> > * partitions.conf is the following
> > Default=0x7fff, ipoib : ALL=full
> > Net0=0x0001, ipoib : ALL=full
>
> Should be
>
> Default=0x7fff, ipoib : ALL=full ;
> Net0=0x0001, ipoib : ALL=full ;
>
> Thank you again.
>
>
> 2011/10/27 Hiroyuki Sato <hiroysato at gmail.com>:
> > Hello Hal
> >
> > Thank you for your information.
> >
> > I simplified my test environment
> > The environment is the folloing.
> >
> > I'm not sure what is wrong Test2 (Partition test)
> > And Could you please tell me how to check the problem??
> > (tool, logfile, etc.. )
> >
> > * Environment
> >
> > OS: Scientific Linux6.1
> > OFED: 1.5.3.
> >
> > * Diagram
> >
> > +----------+ib0 ib0+----------+
> > | Server1 |------------------| Server2 |
> > +----------+ +----------+
> >
> >
> >
> >
> > 1) Test1 (Simple IPoIB no partition Ping test)
> >
> > Server1
> >
> > a) /sbin/ifconfig ib0 inet 192.168.1.1/24
> > b) /sbn/service opensmd start
> > 3) /sbin/service opensmd start
> > * no /etc/opensm/partitions.conf
> >
> > Server2
> > /sbin/ifconfig ib0 inet 192.168.1.1/24
> >
> > Test from server1(192.168.1.1) to server2(192.168.1.2)
> > ping 192.168.1.2
> > The result was OK.
> >
> > 2) Test2 (Add partition by hand and Ping test)
> >
> > reboot Server 1 and Server 2
> >
> > Server 1
> > 1) echo 0x8001 > /sys/class/net/ib0/create_child
> > 2) /sbin/ifconfig ib0.8001 inet 192.168.0.1/24
> > 3) /sbin/service opensmd start
> >
> > * partitions.conf is the following
> > Default=0x7fff, ipoib : ALL=full
> > Net0=0x0001, ipoib : ALL=full
> >
> > Server2
> > 1) echo 0x8001 > /sys/class/net/ib0/create_child
> > 2) /sbin/ifconfig ib0.8001 inet 192.168.0.2/24
> >
> > Test from server1(ib0.8001/192.168.0.1) to server2(ib0.8001/192.168.0.2
> )
> >
> > ping 192.168.0.2
> > no response.
> >
> > 3) Test2 Log
> >
> > I enabled debug_log parameter in ib_ipoib modules.
> >
> > cd /sys/module/ib_ipoib/parameters
> > # cat debug_level
> > 1
> >
> > Could you please tell me what is wrong??
> >
> > ib0: enabling connected mode will cause multicast packet drops
> > ib0: mtu > 2044 will cause multicast packet drops.
> > ib0: mtu > 2044 will cause multicast packet drops.
> > ib1: enabling connected mode will cause multicast packet drops
> > ib1: mtu > 2044 will cause multicast packet drops.
> > ib1: mtu > 2044 will cause multicast packet drops.
> > ib0: Event 17 on device mthca0 port 1
> > ib0: Not flushing - IPOIB_FLAG_INITIALIZED not set.
> > ib0: Event 11 on device mthca0 port 1
> > ib0: Not flushing - IPOIB_FLAG_INITIALIZED not set.
> > ib0: Event 9 on device mthca0 port 1
> > ib0: Not flushing - IPOIB_FLAG_INITIALIZED not set.
> > ib0.8001: max_srq_sge=27
> > ib0.8001: max_cm_mtu = 0xfff0, num_frags=16
> > ib0.8001: bringing up interface
> > ib0.8001: IPOIB_FLAG_OPER_UP not set
> > ib0.8001: IPOIB_FLAG_OPER_UP not set
> > ADDRCONF(NETDEV_UP): ib0.8001: link is not ready
> > ib0.8001: IPOIB_FLAG_OPER_UP not set
> >
> > regards.
> >
> > 2011/10/27 Hal Rosenstock <hal.rosenstock at gmail.com>:
> >>
> >>
> >> On Wed, Oct 26, 2011 at 10:03 AM, Hiroyuki Sato <hiroysato at gmail.com>
> wrote:
> >>>
> >>> Dear members.
> >>>
> >>> I have some question about Infiniband Partitions.
> >>>
> >>> I would like to build Linux box with IPoIB Router.
> >>>
> >>> * Questions
> >>>
> >>> My Question is the following.
> >>>
> >>> a) Can I create the following IPoIB network ??
> >>>
> >>> b) If so, how to setup /etc/opensm/partitions.conf??
> >>>
> >>> * Problem.
> >>>
> >>> Ping IB Client to Router : I got the followig error.
> >>>
> >>> Network is unreachable.
> >>>
> >>> * Environment
> >>>
> >>> OFED: OFED-1.5.3.2
> >>> OS: Scientific Linux 6.1
> >>>
> >>> * Linux box (Router)
> >>>
> >>> 1 ethernet interface (eth0)
> >>> 1 Infiniband interfae (ib0)
> >>> Run subnet manager.
> >>>
> >>> * Ethernet setup
> >>>
> >>> a) add VLAN ID 120 eth0.120 (192.168.120.0/24)
> >>> b) add VLAN ID 130 eth0.130 (192.168.130.0/24)
> >>>
> >>> * Infiniband Setup
> >>> a) add subinterface 0 ib0.8000 (192.168.0.0/24)
> >>> b) add subinterface 1 ib0.8001 (192.168.1.0/24)
> >>> c) Run opensm
> >>>
> >>>
> >>> I crated subinterface with the following command.
> >>>
> >>> echo 0 > /sys/class/net/ib0/create_child
> >>> echo 1 > /sys/class/net/ib0/create_child
> >>
> >>
> >> echo 0x8001 > /sys/class/net/ib0/create_child
> >> echo 0x8002 > /sys/class/net/ib0/create_child
> >>
> >>>
> >>> and assign ip address with ifconfig
> >>>
> >>> * Logical Diagram
> >>>
> >>> +----------- 192.168.120.0/24 ---
> >>> |
> >>> | +---- 192.168.130.0/24
> >>> eth0.120| | eth0.130
> >>> +----------+
> >>> | Router |
> >>> +----------+
> >>> ib0.8000| |ib0.8001
> >>> 192.168.0.1 | +---- 192.168.1.0/24
> >>> |
> >>> +----------- 192.168.0.0/24 --- IB Cllient. (ib0.8000)
> >>> (192.168.0.2)
> >>>
> >>> * /etc/opensm/partitions.conf
> >>>
> >>> Default=0x7fff : ALL=full
> >>> Net0=0x00, ipoib : ALL=full
> >>> Net1=0x01, ipoib : ALL=full
> >>
> >>
> >> Default=0x7fff: ALL=full
> >> Net0=0x0001, ipoib: ALL=full
> >> Net1=0x0002, ipoib: ALL=full
> >>
> >>
> >>>
> >>> Coule you please tell me what is wrong??
> >>
> >>
> >> The one thing in the above I'm not sure of is whether the default
> partition
> >> also needs the ipoib flag above so you might need to add that too if the
> >> above doesn't work.
> >>
> >> -- Hal
> >>
> >>>
> >>> Sincerely.
> >>>
> >>> --
> >>> Hiroyuki Sato.
> >>> _______________________________________________
> >>> ewg mailing list
> >>> ewg at lists.openfabrics.org
> >>> http://lists.openfabrics.org/cgi-bin/mailman/listinfo/ewg
> >>
> >>
> >
> >
> >
> > --
> > Hiroyuki Sato
> >
>
>
>
> --
> Hiroyuki Sato
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.openfabrics.org/pipermail/ewg/attachments/20111027/4ac625d9/attachment.html>
More information about the ewg
mailing list