[ewg] [Q] IPoIB with multiple ports setup

Hiroyuki Sato hiroysato at gmail.com
Fri Nov 26 01:40:36 PST 2010


Hi Richard

Thank you for your information.

  SubnetManager
  192.168.100.231(port1)        Subnet Manager
  192.168.200.231(port2)        Client1(192.168.100.232)
  +----------+                  +----------+
  |          | 192.168.100.0/24 |          |
  |          port1------------port1        |
  |          port2--+       +-port2        |
  |          |      |       |   |          |
  +----------+      |       |   +----------+
                    |       |
                    |       |  Subnet Manager
                    |       |   Client2(192.168.200.233)
                    |       |   +----------+
                    |       |   |          |
                    +-------|--port1       |
                            +- port2       |
                                |          |
                                +----------+

It is mean that If I connect Client1 and Client2 directory, and run
OpenSM on client1 and client2
this topolgy would work??


BTW I'm reading

  InfiniBandTM Architecture Specification Volume 1 Release 1.2.1 for
study about Infiniband.

   CHAPTER 3: ARCHITECTURAL OVERVIEW

  It is a bit difficult for me.

I would appreciate If you could tell me more better tutorial
book/website/ml or so on..

--
Hiroyuki Sato



2010/11/26 Richard Croucher <richard.croucher at informatix-sol.com>:
> InfiniBand is really designed to run connected through a switch.  In which
> case your topology would work.   OpenSM only uses 1 port.  In your case it
> is using port 1 on Server1, hence,  it is ignoring Client2 connected on
> port2.   Connected via a switch, this would not be a problem, but without
> the switch, it is.
>
> I would directly connect your client 1 to client 2, and run OpenSM on both
> of them.   OpenSM will then elect one as the master and assign the other as
> the slave .   This should then work fine, since both clients will be
> discovered.
>
> Richard
>
>
> On Fri, 2010-11-26 at 11:13 +0900, Hiroyuki Sato wrote:
>
> Hello members.
>
> I'm newbie about Infiniband
>
> Does anybody knows good example configuration about openib and opensmd ?
>
> I would like to setup IPoIB with multiple ports
> It seems that I have to setup opensmd.
>
> I read the documentation opensm(8) which is bundled with CentOS
> but I couldn't setup multiple ports.
>
> I'm trying the following environment
>
>   server1 <-> client1 : ping OK
>   server1 <-> client2 : ping NG
>
> Could you tell me what is wrong??
>
>
> * Environemnt
>   client1 and client2 belongs in different subnets
>
>   SubnetManager
>   192.168.100.231(port1)
>   192.168.200.231(port2)        client1(192.168.100.232)
>   +----------+                  +----------+
>   |          | 192.168.100.0/24 |          |
>   |          port1------------port1        |
>   |          port2--+         port2        |
>   |          |      |           |          |
>   +----------+      |           +----------+
>                     |
>                     |
>                     |           client2(192.168.200.233)
>                     |           +----------+
>                     |           |          |
>                     +----------port1       |
>               192.168.200.0/24 port2       |
>                                 |          |
>                                 +----------+
>
>
> OS: CentOS 5.5
> Kernel: 2.6.18-194.26.1.el
> OpenSM: 3.3.3 bundled with CentOS
>
> Thanks in advice
>
> --
> Hiroyuki Sato
>
>
>   egrep -v '^(#|$)' /etc/ofed/openib.conf
>   IPOIB_LOAD=yes
>   SDP_LOAD=yes
>   RDS_LOAD=yes
>   SRP_LOAD=yes
>   ISER_LOAD=yes
>   FIXUP_MTRR_REGS=no
>
>
>   egrep -v '^(#|$)' /etc/ofed/opensm.conf
>
>   guid 0x0000000000000000
>   m_key 0x0000000000000000
>   m_key_lease_period 0
>   sm_key 0x0000000000000001
>   sa_key 0x0000000000000001
>   subnet_prefix 0xfe80000000000000
>   lmc 0
>   lmc_esp0 FALSE
>   packet_life_time 0x12
>   vl_stall_count 0x07
>   leaf_vl_stall_count 0x07
>   head_of_queue_lifetime 0x12
>   leaf_head_of_queue_lifetime 0x10
>   max_op_vls 5
>   force_link_speed 15
>   subnet_timeout 18
>   local_phy_errors_threshold 0x08
>   overrun_errors_threshold 0x08
>   partition_config_file /etc/ofed/partitions.conf
>   no_partition_enforcement FALSE
>   sweep_interval 10
>   reassign_lids FALSE
>   force_heavy_sweep FALSE
>   sweep_on_trap TRUE
>   port_profile_switch_nodes FALSE
>   port_prof_ignore_file (null)
>   routing_engine (null)
>   connect_roots FALSE
>   use_ucast_cache FALSE
>   lid_matrix_dump_file (null)
>   lfts_file (null)
>   root_guid_file (null)
>   cn_guid_file (null)
>   ids_guid_file (null)
>   guid_routing_order_file (null)
>   sa_db_file (null)
>   sm_priority 15
>   ignore_other_sm FALSE
>   sminfo_polling_timeout 10000
>   polling_retry_number 4
>   honor_guid2lid_file FALSE
>   max_wire_smps 32
>   transaction_timeout 200
>   max_msg_fifo_timeout 10000
>   single_thread FALSE
>   daemon TRUE
>   sm_inactive FALSE
>   babbling_port_policy FALSE
>   event_plugin_name (null)
>   node_name_map_name (null)
>   log_flags 0x03
>   force_log_flush FALSE
>   log_file /var/log/opensm.log
>   log_max_size 0
>   accum_log_file TRUE
>   dump_files_dir /var/log/
>   enable_quirks FALSE
>   no_clients_rereg FALSE
>   disable_multicast FALSE
>   exit_on_fatal TRUE
>   console off
>   console_port 10000
>   qos FALSE
>   qos_policy_file /etc/ofed/qos-policy.conf
>   qos_max_vls 0
>   qos_high_limit -1
>   qos_vlarb_high (null)
>   qos_vlarb_low (null)
>   qos_sl2vl (null)
>   qos_ca_max_vls 0
>   qos_ca_high_limit -1
>   qos_ca_vlarb_high (null)
>   qos_ca_vlarb_low (null)
>   qos_ca_sl2vl (null)
>   qos_sw0_max_vls 0
>   qos_sw0_high_limit -1
>   qos_sw0_vlarb_high (null)
>   qos_sw0_vlarb_low (null)
>   qos_sw0_sl2vl (null)
>   qos_swe_max_vls 0
>   qos_swe_high_limit -1
>   qos_swe_vlarb_high (null)
>   qos_swe_vlarb_low (null)
>   qos_swe_sl2vl (null)
>   qos_rtr_max_vls 0
>   qos_rtr_high_limit -1
>   qos_rtr_vlarb_high (null)
>   qos_rtr_vlarb_low (null)
>   qos_rtr_sl2vl (null)
>   prefix_routes_file /etc/ofed/prefix-routes.conf
>   consolidate_ipv6_snm_req FALSE
>
>   # cat /etc/sysconfig/network-scripts/ifcfg-ib0
>   DEVICE=ib0
>   BOOTPROTO=static
>   DHCPCLASS=
>   IPADDR=192.168.100.231
>   NETMASK=255.255.255.0
>   ONBOOT=yes
>
>   cat /etc/sysconfig/network-scripts/ifcfg-ib1
>   DEVICE=ib1
>   BOOTPROTO=static
>   DHCPCLASS=
>   IPADDR=192.168.200.231
>   NETMASK=255.255.255.0
>   ONBOOT=yes
>
> client1
>
>   /sbin/chkconfig oepnibd on
>
>   cat /etc/sysconfig/network-scripts/ifcfg-ib0
>   DEVICE=ib0
>   BOOTPROTO=static
>   DHCPCLASS=
>   IPADDR=192.168.100.232
>   NETMASK=255.255.255.0
>   ONBOOT=yes
>
> client2
>
>   /sbin/chkconfig oepnibd on
>
>   cat /etc/sysconfig/network-
> scripts/ifcfg-ib0
>   DEVICE=ib0
>   BOOTPROTO=static
>   DHCPCLASS=
>   IPADDR=192.168.200.232
>   NETMASK=255.255.255.0
>   ONBOOT=yes
> _______________________________________________
> ewg mailing list
> ewg at lists.openfabrics.org
> http://lists.openfabrics.org/cgi-bin/mailman/listinfo/ewg
>
>



More information about the ewg mailing list