<br><br><div class="gmail_quote">On Thu, Oct 27, 2011 at 7:48 AM, Hiroyuki Sato <span dir="ltr"><<a href="mailto:hiroysato@gmail.com" target="_blank">hiroysato@gmail.com</a>></span> wrote:<br><blockquote style="margin: 0px 0px 0px 0.8ex; padding-left: 1ex; border-left-color: rgb(204, 204, 204); border-left-width: 1px; border-left-style: solid;" class="gmail_quote">
<div>Hello Hal<br>
<br>
Thank you for your information.<br>
<br>
</div>I read this documents. PP 45.<br>
<a href="http://www.mellanox.com/related-docs/prod_software/Mellanox_OFED_Linux_user_manual_1_5_1.pdf" target="_blank">http://www.mellanox.com/related-docs/prod_software/Mellanox_OFED_Linux_user_manual_1_5_1.pdf</a><br>
<br>
Decide on the PKey to be used in the subnet.<br>
Valid values are 0-255.<br></blockquote><div> </div><div>Valid values for pkey are 1-32767 (1-0x7ffff).</div><div> </div><blockquote style="margin: 0px 0px 0px 0.8ex; padding-left: 1ex; border-left-color: rgb(204, 204, 204); border-left-width: 1px; border-left-style: solid;" class="gmail_quote">
The actual PKey used is a 16-bit number with<br>
the most significant bit set.<br>
For example, a value of 0 will give a<br>
PKey with the value 0x8000.<br>
<br>
Is this wrong??<br></blockquote><div> </div><div>Pkey 0/0x8000 is invalid so this is wrong. What it's showing about turning on the full membership bit in the pkey is correct. A better example would be showing a value of 1 being transformed into 0x8001 pkey.</div>
<div> </div><div>-- Hal</div><div> </div><blockquote style="margin: 0px 0px 0px 0.8ex; padding-left: 1ex; border-left-color: rgb(204, 204, 204); border-left-width: 1px; border-left-style: solid;" class="gmail_quote">
<div><div></div><div><br>
<br>
2011/10/27 Hal Rosenstock <<a href="mailto:hal.rosenstock@gmail.com" target="_blank">hal.rosenstock@gmail.com</a>>:<br>
><br>
><br>
> On Thu, Oct 27, 2011 at 7:05 AM, Hiroyuki Sato <<a href="mailto:hiroysato@gmail.com" target="_blank">hiroysato@gmail.com</a>> wrote:<br>
>><br>
>> Hello Hal and Richard.<br>
>><br>
>> I found what is the problem.<br>
>><br>
>> Q: Can I create subinterface with 0 (0x8000)???<br>
><br>
><br>
> That uses an invalid pkey so this is not a valid configuration.<br>
><br>
> -- Hal<br>
><br>
>><br>
>> My Test result.<br>
>><br>
>> partitions.conf<br>
>><br>
>> Default=0x7fff, : ALL=full ;<br>
>> Net0=0x0000, ipoib : ALL=full ;<br>
>> Net1=0x0001, ipoib : ALL=full ;<br>
>> Net2=0x0002, ipoib : ALL=full ;<br>
>><br>
>> 1) Server1 ib.8000 -> Server1 ib.8000<br>
>> Ping NG<br>
>><br>
>> 2) Server1 ib.8001 -> Server1 ib.8001<br>
>><br>
>> The differece is subinterfae number.<br>
>><br>
>> Yesterday, I always tested subinterface 0x8000.<br>
>><br>
>><br>
>><br>
>> And about previous post,<br>
>><br>
>> The partitions.conf which I posted few hous ago,<br>
>> was semi colon missing.<br>
>><br>
>> > * partitions.conf is the following<br>
>> > Default=0x7fff, ipoib : ALL=full<br>
>> > Net0=0x0001, ipoib : ALL=full<br>
>><br>
>> Should be<br>
>><br>
>> Default=0x7fff, ipoib : ALL=full ;<br>
>> Net0=0x0001, ipoib : ALL=full ;<br>
>><br>
>> Thank you again.<br>
>><br>
>><br>
>> 2011/10/27 Hiroyuki Sato <<a href="mailto:hiroysato@gmail.com" target="_blank">hiroysato@gmail.com</a>>:<br>
>> > Hello Hal<br>
>> ><br>
>> > Thank you for your information.<br>
>> ><br>
>> > I simplified my test environment<br>
>> > The environment is the folloing.<br>
>> ><br>
>> > I'm not sure what is wrong Test2 (Partition test)<br>
>> > And Could you please tell me how to check the problem??<br>
>> > (tool, logfile, etc.. )<br>
>> ><br>
>> > * Environment<br>
>> ><br>
>> > OS: Scientific Linux6.1<br>
>> > OFED: 1.5.3.<br>
>> ><br>
>> > * Diagram<br>
>> ><br>
>> > +----------+ib0 ib0+----------+<br>
>> > | Server1 |------------------| Server2 |<br>
>> > +----------+ +----------+<br>
>> ><br>
>> ><br>
>> ><br>
>> ><br>
>> > 1) Test1 (Simple IPoIB no partition Ping test)<br>
>> ><br>
>> > Server1<br>
>> ><br>
>> > a) /sbin/ifconfig ib0 inet <a href="http://192.168.1.1/24" target="_blank">192.168.1.1/24</a><br>
>> > b) /sbn/service opensmd start<br>
>> > 3) /sbin/service opensmd start<br>
>> > * no /etc/opensm/partitions.conf<br>
>> ><br>
>> > Server2<br>
>> > /sbin/ifconfig ib0 inet <a href="http://192.168.1.1/24" target="_blank">192.168.1.1/24</a><br>
>> ><br>
>> > Test from server1(192.168.1.1) to server2(192.168.1.2)<br>
>> > ping 192.168.1.2<br>
>> > The result was OK.<br>
>> ><br>
>> > 2) Test2 (Add partition by hand and Ping test)<br>
>> ><br>
>> > reboot Server 1 and Server 2<br>
>> ><br>
>> > Server 1<br>
>> > 1) echo 0x8001 > /sys/class/net/ib0/create_child<br>
>> > 2) /sbin/ifconfig ib0.8001 inet <a href="http://192.168.0.1/24" target="_blank">192.168.0.1/24</a><br>
>> > 3) /sbin/service opensmd start<br>
>> ><br>
>> > * partitions.conf is the following<br>
>> > Default=0x7fff, ipoib : ALL=full<br>
>> > Net0=0x0001, ipoib : ALL=full<br>
>> ><br>
>> > Server2<br>
>> > 1) echo 0x8001 > /sys/class/net/ib0/create_child<br>
>> > 2) /sbin/ifconfig ib0.8001 inet <a href="http://192.168.0.2/24" target="_blank">192.168.0.2/24</a><br>
>> ><br>
>> > Test from server1(ib0.8001/<a href="http://192.168.0.1" target="_blank">192.168.0.1</a>) to<br>
>> > server2(ib0.8001/<a href="http://192.168.0.2" target="_blank">192.168.0.2</a>)<br>
>> ><br>
>> > ping 192.168.0.2<br>
>> > no response.<br>
>> ><br>
>> > 3) Test2 Log<br>
>> ><br>
>> > I enabled debug_log parameter in ib_ipoib modules.<br>
>> ><br>
>> > cd /sys/module/ib_ipoib/parameters<br>
>> > # cat debug_level<br>
>> > 1<br>
>> ><br>
>> > Could you please tell me what is wrong??<br>
>> ><br>
>> > ib0: enabling connected mode will cause multicast packet drops<br>
>> > ib0: mtu > 2044 will cause multicast packet drops.<br>
>> > ib0: mtu > 2044 will cause multicast packet drops.<br>
>> > ib1: enabling connected mode will cause multicast packet drops<br>
>> > ib1: mtu > 2044 will cause multicast packet drops.<br>
>> > ib1: mtu > 2044 will cause multicast packet drops.<br>
>> > ib0: Event 17 on device mthca0 port 1<br>
>> > ib0: Not flushing - IPOIB_FLAG_INITIALIZED not set.<br>
>> > ib0: Event 11 on device mthca0 port 1<br>
>> > ib0: Not flushing - IPOIB_FLAG_INITIALIZED not set.<br>
>> > ib0: Event 9 on device mthca0 port 1<br>
>> > ib0: Not flushing - IPOIB_FLAG_INITIALIZED not set.<br>
>> > ib0.8001: max_srq_sge=27<br>
>> > ib0.8001: max_cm_mtu = 0xfff0, num_frags=16<br>
>> > ib0.8001: bringing up interface<br>
>> > ib0.8001: IPOIB_FLAG_OPER_UP not set<br>
>> > ib0.8001: IPOIB_FLAG_OPER_UP not set<br>
>> > ADDRCONF(NETDEV_UP): ib0.8001: link is not ready<br>
>> > ib0.8001: IPOIB_FLAG_OPER_UP not set<br>
>> ><br>
>> > regards.<br>
>> ><br>
>> > 2011/10/27 Hal Rosenstock <<a href="mailto:hal.rosenstock@gmail.com" target="_blank">hal.rosenstock@gmail.com</a>>:<br>
>> >><br>
>> >><br>
>> >> On Wed, Oct 26, 2011 at 10:03 AM, Hiroyuki Sato <<a href="mailto:hiroysato@gmail.com" target="_blank">hiroysato@gmail.com</a>><br>
>> >> wrote:<br>
>> >>><br>
>> >>> Dear members.<br>
>> >>><br>
>> >>> I have some question about Infiniband Partitions.<br>
>> >>><br>
>> >>> I would like to build Linux box with IPoIB Router.<br>
>> >>><br>
>> >>> * Questions<br>
>> >>><br>
>> >>> My Question is the following.<br>
>> >>><br>
>> >>> a) Can I create the following IPoIB network ??<br>
>> >>><br>
>> >>> b) If so, how to setup /etc/opensm/partitions.conf??<br>
>> >>><br>
>> >>> * Problem.<br>
>> >>><br>
>> >>> Ping IB Client to Router : I got the followig error.<br>
>> >>><br>
>> >>> Network is unreachable.<br>
>> >>><br>
>> >>> * Environment<br>
>> >>><br>
>> >>> OFED: OFED-1.5.3.2<br>
>> >>> OS: Scientific Linux 6.1<br>
>> >>><br>
>> >>> * Linux box (Router)<br>
>> >>><br>
>> >>> 1 ethernet interface (eth0)<br>
>> >>> 1 Infiniband interfae (ib0)<br>
>> >>> Run subnet manager.<br>
>> >>><br>
>> >>> * Ethernet setup<br>
>> >>><br>
>> >>> a) add VLAN ID 120 eth0.120 (<a href="http://192.168.120.0/24" target="_blank">192.168.120.0/24</a>)<br>
>> >>> b) add VLAN ID 130 eth0.130 (<a href="http://192.168.130.0/24" target="_blank">192.168.130.0/24</a>)<br>
>> >>><br>
>> >>> * Infiniband Setup<br>
>> >>> a) add subinterface 0 ib0.8000 (<a href="http://192.168.0.0/24" target="_blank">192.168.0.0/24</a>)<br>
>> >>> b) add subinterface 1 ib0.8001 (<a href="http://192.168.1.0/24" target="_blank">192.168.1.0/24</a>)<br>
>> >>> c) Run opensm<br>
>> >>><br>
>> >>><br>
>> >>> I crated subinterface with the following command.<br>
>> >>><br>
>> >>> echo 0 > /sys/class/net/ib0/create_child<br>
>> >>> echo 1 > /sys/class/net/ib0/create_child<br>
>> >><br>
>> >><br>
>> >> echo 0x8001 > /sys/class/net/ib0/create_child<br>
>> >> echo 0x8002 > /sys/class/net/ib0/create_child<br>
>> >><br>
>> >>><br>
>> >>> and assign ip address with ifconfig<br>
>> >>><br>
>> >>> * Logical Diagram<br>
>> >>><br>
>> >>> +----------- <a href="http://192.168.120.0/24" target="_blank">192.168.120.0/24</a> ---<br>
>> >>> |<br>
>> >>> | +---- <a href="http://192.168.130.0/24" target="_blank">192.168.130.0/24</a><br>
>> >>> eth0.120| | eth0.130<br>
>> >>> +----------+<br>
>> >>> | Router |<br>
>> >>> +----------+<br>
>> >>> ib0.8000| |ib0.8001<br>
>> >>> 192.168.0.1 | +---- <a href="http://192.168.1.0/24" target="_blank">192.168.1.0/24</a><br>
>> >>> |<br>
>> >>> +----------- <a href="http://192.168.0.0/24" target="_blank">192.168.0.0/24</a> --- IB Cllient. (ib0.8000)<br>
>> >>> (192.168.0.2)<br>
>> >>><br>
>> >>> * /etc/opensm/partitions.conf<br>
>> >>><br>
>> >>> Default=0x7fff : ALL=full<br>
>> >>> Net0=0x00, ipoib : ALL=full<br>
>> >>> Net1=0x01, ipoib : ALL=full<br>
>> >><br>
>> >><br>
>> >> Default=0x7fff: ALL=full<br>
>> >> Net0=0x0001, ipoib: ALL=full<br>
>> >> Net1=0x0002, ipoib: ALL=full<br>
>> >><br>
>> >><br>
>> >>><br>
>> >>> Coule you please tell me what is wrong??<br>
>> >><br>
>> >><br>
>> >> The one thing in the above I'm not sure of is whether the default<br>
>> >> partition<br>
>> >> also needs the ipoib flag above so you might need to add that too if<br>
>> >> the<br>
>> >> above doesn't work.<br>
>> >><br>
>> >> -- Hal<br>
>> >><br>
>> >>><br>
>> >>> Sincerely.<br>
>> >>><br>
>> >>> --<br>
>> >>> Hiroyuki Sato.<br>
>> >>> _______________________________________________<br>
>> >>> ewg mailing list<br>
>> >>> <a href="mailto:ewg@lists.openfabrics.org" target="_blank">ewg@lists.openfabrics.org</a><br>
>> >>> <a href="http://lists.openfabrics.org/cgi-bin/mailman/listinfo/ewg" target="_blank">http://lists.openfabrics.org/cgi-bin/mailman/listinfo/ewg</a><br>
>> >><br>
>> >><br>
>> ><br>
>> ><br>
>> ><br>
>> > --<br>
>> > Hiroyuki Sato<br>
>> ><br>
>><br>
>><br>
>><br>
>> --<br>
>> Hiroyuki Sato<br>
><br>
><br>
<br>
<br>
<br>
</div></div><font color="#888888">--<br>
Hiroyuki Sato<br>
</font></blockquote></div><br>