[ewg] [Q] IPoIB with multiple ports setup
Hiroyuki Sato
hiroysato at gmail.com
Fri Nov 26 03:14:18 PST 2010
Hi Ricard
Thank you for your advice.
I succeed to ping to client1 and client2 from server1
I run subnet manager on each ports.
Now I have just 2cable and 3IB cards
so I have to evaluate IB on the following environment.
I'll test IB and I'll planning to buy IB Switch.
SubnetManager
*both port*
^^^^^^^^^^^^
192.168.100.231(port1)
192.168.200.231(port2) client1(192.168.100.232)
+----------+ +----------+
| | 192.168.100.0/24 | |
| port1------------port1 |
| port2--+ port2 |
| | | | |
+----------+ | +----------+
|
|
| client2(192.168.200.233)
| +----------+
| | |
+----------port1 |
192.168.200.0/24 port2 |
| |
+----------+
Server config
Check IB
# mstflint -d 06:00.0 q
Image type: Failsafe
FW Version: 5.3.0
Rom Info: type=GPXE version=1.0.0 devid=25218 port=1
I.S. Version: 1
Device ID: 25218
Chip Revision: A0
Description: Node Port1 Port2 Sys image
GUIDs: 0002c90200220eb8 0002c90200220eb9 0002c90200220eba
0005ad000100d050
Board ID: ??? (MT_0370110001)
VSD: ???
PSID: MT_0370110001
Execute opensm with each GUID
/usr/sbin/opensm -B -g 0002c90200220eba
/usr/sbin/opensm -B -g 0002c90200220eb9
--
Hiroyuki Sato
2010/11/26 Richard Croucher <richard.croucher at informatix-sol.com>:
> In theory but it will be 3 different subnets and I not convinced it will
> work unless you bind SM to specific ports.
>
> Initially just have Client1port2 connected to Client2 port2owith a SM on
> both, no other connections. This should work. After that you can look at
> adding extra cables
>
>
> On Fri, 2010-11-26 at 18:40 +0900, Hiroyuki Sato wrote:
>
> Hi Richard
>
> Thank you for your information.
>
> SubnetManager
> 192.168.100.231(port1) Subnet Manager
> 192.168.200.231(port2) Client1(192.168.100.232)
> +----------+ +----------+
> | | 192.168.100.0/24 | |
> | port1------------port1 |
> | port2--+ +-port2 |
> | | | | | |
> +----------+ | | +----------+
> | |
> | | Subnet Manager
> | | Client2(192.168.200.233)
> | | +----------+
> | | | |
> +-------|--port1 |
> +- port2 |
> | |
> +----------+
>
> It is mean that If I connect Client1 and Client2 directory, and run
> OpenSM on client1 and client2
> this topolgy would work??
>
>
> BTW I'm reading
>
> InfiniBandTM Architecture Specification Volume 1 Release 1.2.1 for
> study about Infiniband.
>
> CHAPTER 3: ARCHITECTURAL OVERVIEW
>
> It is a bit difficult for me.
>
> I would appreciate If you could tell me more better tutorial
> book/website/ml or so on..
>
> --
> Hiroyuki Sato
>
>
>
> 2010/11/26 Richard Croucher <richard.croucher at informatix-sol.com>:
>> InfiniBand is really designed to run connected through a switch. In which
>> case your topology would work. OpenSM only uses 1 port. In your case it
>> is using port 1 on Server1, hence, it is ignoring Client2 connected on
>> port2. Connected via a switch, this would not be a problem, but without
>> the switch, it is.
>>
>> I would directly connect your client 1 to client 2, and run OpenSM on both
>> of them. OpenSM will then elect one as the master and assign the other
>> as
>> the slave . This should then work fine, since both clients will be
>> discovered.
>>
>> Richard
>>
>>
>> On Fri, 2010-11-26 at 11:13 +0900, Hiroyuki Sato wrote:
>>
>> Hello members.
>>
>> I'm newbie about Infiniband
>>
>> Does anybody knows good example configuration about openib and opensmd ?
>>
>> I would like to setup IPoIB with multiple ports
>> It seems that I have to setup opensmd.
>>
>> I read the documentation opensm(8) which is bundled with CentOS
>> but I couldn't setup multiple ports.
>>
>> I'm trying the following environment
>>
>> server1 <-> client1 : ping OK
>> server1 <-> client2 : ping NG
>>
>> Could you tell me what is wrong??
>>
>>
>> * Environemnt
>> client1 and client2 belongs in different subnets
>>
>> SubnetManager
>> 192.168.100.231(port1)
>> 192.168.200.231(port2) client1(192.168.100.232)
>> +----------+ +----------+
>> | | 192.168.100.0/24 | |
>> | port1------------port1 |
>> | port2--+ port2 |
>> | | | | |
>> +----------+ | +----------+
>> |
>> |
>> | client2(192.168.200.233)
>> | +----------+
>> | | |
>> +----------port1 |
>> 192.168.200.0/24 port2 |
>> | |
>> +----------+
>>
>>
>> OS: CentOS 5.5
>> Kernel: 2.6.18-194.26.1.el
>> OpenSM: 3.3.3 bundled with CentOS
>>
>> Thanks in advice
>>
>> --
>> Hiroyuki Sato
>>
>>
>> egrep -v '^(#|$)' /etc/ofed/openib.conf
>> IPOIB_LOAD=yes
>> SDP_LOAD=yes
>> RDS_LOAD=yes
>> SRP_LOAD=yes
>> ISER_LOAD=yes
>> FIXUP_MTRR_REGS=no
>>
>>
>> egrep -v '^(#|$)' /etc/ofed/opensm.conf
>>
>> guid 0x0000000000000000
>> m_key 0x0000000000000000
>> m_key_lease_period 0
>> sm_key 0x0000000000000001
>> sa_key 0x0000000000000001
>> subnet_prefix 0xfe80000000000000
>> lmc 0
>> lmc_esp0 FALSE
>> packet_life_time 0x12
>> vl_stall_count 0x07
>> leaf_vl_stall_count 0x07
>> head_of_queue_lifetime 0x12
>> leaf_head_of_queue_lifetime 0x10
>> max_op_vls 5
>> force_link_speed 15
>> subnet_timeout 18
>> local_phy_errors_threshold 0x08
>> overrun_errors_threshold 0x08
>> partition_config_file /etc/ofed/partitions.conf
>> no_partition_enforcement FALSE
>> sweep_interval 10
>> reassign_lids FALSE
>> force_heavy_sweep FALSE
>> sweep_on_trap TRUE
>> port_profile_switch_nodes FALSE
>> port_prof_ignore_file (null)
>> routing_engine (null)
>> connect_roots FALSE
>> use_ucast_cache FALSE
>> lid_matrix_dump_file (null)
>> lfts_file (null)
>> root_guid_file (null)
>> cn_guid_file (null)
>> ids_guid_file (null)
>> guid_routing_order_file (null)
>> sa_db_file (null)
>> sm_priority 15
>> ignore_other_sm FALSE
>> sminfo_polling_timeout 10000
>> polling_retry_number 4
>> honor_guid2lid_file FALSE
>> max_wire_smps 32
>> transaction_timeout 200
>> max_msg_fifo_timeout 10000
>> single_thread FALSE
>> daemon TRUE
>> sm_inactive FALSE
>> babbling_port_policy FALSE
>> event_plugin_name (null)
>> node_name_map_name (null)
>> log_flags 0x03
>> force_log_flush FALSE
>> log_file /var/log/opensm.log
>> log_max_size 0
>> accum_log_file TRUE
>> dump_files_dir /var/log/
>> enable_quirks FALSE
>> no_clients_rereg FALSE
>> disable_multicast FALSE
>> exit_on_fatal TRUE
>> console off
>> console_port 10000
>> qos FALSE
>> qos_policy_file /etc/ofed/qos-policy.conf
>> qos_max_vls 0
>> qos_high_limit -1
>> qos_vlarb_high (null)
>> qos_vlarb_low (null)
>> qos_sl2vl (null)
>> qos_ca_max_vls 0
>> qos_ca_high_limit -1
>> qos_ca_vlarb_high (null)
>> qos_ca_vlarb_low (null)
>> qos_ca_sl2vl (null)
>> qos_sw0_max_vls 0
>> qos_sw0_high_limit -1
>> qos_sw0_vlarb_high (null)
>> qos_sw0_vlarb_low (null)
>> qos_sw0_sl2vl (null)
>> qos_swe_max_vls 0
>> qos_swe_high_limit -1
>> qos_swe_vlarb_high (null)
>> qos_swe_vlarb_low (null)
>> qos_swe_sl2vl (null)
>> qos_rtr_max_vls 0
>> qos_rtr_high_limit -1
>> qos_rtr_vlarb_high (null)
>> qos_rtr_vlarb_low (null)
>> qos_rtr_sl2vl (null)
>> prefix_routes_file /etc/ofed/prefix-routes.conf
>> consolidate_ipv6_snm_req FALSE
>>
>> # cat /etc/sysconfig/network-scripts/ifcfg-ib0
>> DEVICE=ib0
>> BOOTPROTO=static
>> DHCPCLASS=
>> IPADDR=192.168.100.231
>> NETMASK=255.255.255.0
>> ONBOOT=yes
>>
>> cat /etc/sysconfig/network-scripts/ifcfg-ib1
>> DEVICE=ib1
>> BOOTPROTO=static
>> DHCPCLASS=
>> IPADDR=192.168.200.231
>> NETMASK=255.255.255.0
>> ONBOOT=yes
>>
>> client1
>>
>> /sbin/chkconfig oepnibd on
>>
>> cat /etc/sysconfig/network-scripts/ifcfg-ib0
>> DEVICE=ib0
>> BOOTPROTO=static
>> DHCPCLASS=
>> IPADDR=192.168.100.232
>> NETMASK=255.255.255.0
>> ONBOOT=yes
>>
>> client2
>>
>> /sbin/chkconfig oepnibd on
>>
>> cat /etc/sysconfig/network-
>> scripts/ifcfg-ib0
>> DEVICE=ib0
>> BOOTPROTO=static
>> DHCPCLASS=
>> IPADDR=192.168.200.232
>> NETMASK=255.255.255.0
>> ONBOOT=yes
>> _______________________________________________
>> ewg mailing list
>> ewg at lists.openfabrics.org
>> http://lists.openfabrics.org/cgi-bin/mailman/listinfo/ewg
>>
>>
>
>
More information about the ewg
mailing list