<div dir="ltr">Hal,<div><br></div><div>From what I can tell, the start up script that starts opensm in Xsigo only specifies the following command line parameters:</div><div><br></div><div><font face="courier new, monospace">"-t 2000 -L 100 -y -q loopback -P /tmp/osmpart.conf -F /opt/xsigo/xsigos/current/ofed/etc/opensm.opts"</font></div>
<div><br></div><div>The opensm.opts contains:</div><div><div><font face="courier new, monospace"># SA database file name</font></div><div><font face="courier new, monospace">sa_db_file /var/log/opensm-sa.dump</font></div>
<div><font face="courier new, monospace"><br></font></div><div><font face="courier new, monospace"># If TRUE causes OpenSM to dump SA database at the end of</font></div><div><font face="courier new, monospace"># every light sweep, regardless of the verbosity level</font></div>
<div><font face="courier new, monospace">sa_db_dump TRUE</font></div><div><font face="courier new, monospace"><br></font></div><div><font face="courier new, monospace"># The directory to hold the file OpenSM dumps</font></div>
<div><font face="courier new, monospace">dump_files_dir /var/log/</font></div></div><div><br></div><div>And the osmpart.conf contains:</div><div><div><font face="courier new, monospace"> Default=0x7fff,ipoib: ALL=full ;</font></div>
</div><div><br></div><div>They are running OpenSM 3.3.5 so it seems that it is pretty vanilla. However, I know that we set the priority of the SMs in their management tool, so I'm wondering if they are passing some additional parameters through the loopback interface. I guess they could have patched the OpenSM code, but I'm not sure they have done that.</div>
<div><br></div><div>I logged into the opensm console and dumped the config. Disable multicast is set to false. It looks like MulticastFDBTop was implemented back in 2009, so this version should support it. Can I set use_mfttop using this version, if not do you know what version I can?</div>
<div><br></div><div>In my testing with ibsim, the LIDs between the real environment and simulated environment appeared to be the same as well as the routing, so I don't believe that I'd run into a problem moving to OpenSM as a primary SM. Do you see anything in the running config that would be concerning to you that should be configured with OpenSM? The differences that I see that I think may drastically change the network behavior are transaction_timeout and babbling_port_policy, but I'm not 100% sure.</div>
<div><br></div><div><div><font face="courier new, monospace">OpenSM $ dump_conf</font></div><div><font face="courier new, monospace">#</font></div><div><font face="courier new, monospace"># DEVICE ATTRIBUTES OPTIONS</font></div>
<div><font face="courier new, monospace">#</font></div><div><font face="courier new, monospace"># The port GUID on which the OpenSM is running</font></div><div><font face="courier new, monospace">guid 0x0000000000000000</font></div>
<div><font face="courier new, monospace"><br></font></div><div><font face="courier new, monospace"># M_Key value sent to all ports qualifying all Set(PortInfo)</font></div><div><font face="courier new, monospace">m_key 0x0000000000000000</font></div>
<div><font face="courier new, monospace"><br></font></div><div><font face="courier new, monospace"># The lease period used for the M_Key on this subnet in [sec]</font></div><div><font face="courier new, monospace">m_key_lease_period 0</font></div>
<div><font face="courier new, monospace"><br></font></div><div><font face="courier new, monospace"># SM_Key value of the SM used for SM authentication</font></div><div><font face="courier new, monospace">sm_key 0x0000000000000001</font></div>
<div><font face="courier new, monospace"><br></font></div><div><font face="courier new, monospace"># SM_Key value to qualify rcv SA queries as 'trusted'</font></div><div><font face="courier new, monospace">sa_key 0x0000000000000001</font></div>
<div><font face="courier new, monospace"><br></font></div><div><font face="courier new, monospace"># Note that for both values above (sm_key and sa_key)</font></div><div><font face="courier new, monospace"># OpenSM version 3.2.1 and below used the default value '1'</font></div>
<div><font face="courier new, monospace"># in a host byte order, it is fixed now but you may need to</font></div><div><font face="courier new, monospace"># change the values to interoperate with old OpenSM running</font></div>
<div><font face="courier new, monospace"># on a little endian machine.</font></div><div><font face="courier new, monospace"><br></font></div><div><font face="courier new, monospace"># Subnet prefix used on this subnet</font></div>
<div><font face="courier new, monospace">subnet_prefix 0xfe80000000000000</font></div><div><font face="courier new, monospace"><br></font></div><div><font face="courier new, monospace"># The LMC value used on this subnet</font></div>
<div><font face="courier new, monospace">lmc 0</font></div><div><font face="courier new, monospace"><br></font></div><div><font face="courier new, monospace"># lmc_esp0 determines whether LMC value used on subnet is used for</font></div>
<div><font face="courier new, monospace"># enhanced switch port 0. If TRUE, LMC value for subnet is used for</font></div><div><font face="courier new, monospace"># ESP0. Otherwise, LMC value for ESP0s is 0.</font></div><div>
<font face="courier new, monospace">lmc_esp0 FALSE</font></div><div><font face="courier new, monospace"><br></font></div><div><font face="courier new, monospace"># sm_sl determines SMSL used for SM/SA communication</font></div>
<div><font face="courier new, monospace">sm_sl 0</font></div><div><font face="courier new, monospace"><br></font></div><div><font face="courier new, monospace"># The code of maximal time a packet can live in a switch</font></div>
<div><font face="courier new, monospace"># The actual time is 4.096usec * 2^<packet_life_time></font></div><div><font face="courier new, monospace"># The value 0x14 disables this mechanism</font></div><div><font face="courier new, monospace">packet_life_time 0x12</font></div>
<div><font face="courier new, monospace"><br></font></div><div><font face="courier new, monospace"># The number of sequential packets dropped that cause the port</font></div><div><font face="courier new, monospace"># to enter the VLStalled state. The result of setting this value to</font></div>
<div><font face="courier new, monospace"># zero is undefined.</font></div><div><font face="courier new, monospace">vl_stall_count 0x07</font></div><div><font face="courier new, monospace"><br></font></div><div><font face="courier new, monospace"># The number of sequential packets dropped that cause the port</font></div>
<div><font face="courier new, monospace"># to enter the VLStalled state. This value is for switch ports</font></div><div><font face="courier new, monospace"># driving a CA or router port. The result of setting this value</font></div>
<div><font face="courier new, monospace"># to zero is undefined.</font></div><div><font face="courier new, monospace">leaf_vl_stall_count 0x07</font></div><div><font face="courier new, monospace"><br></font></div><div><font face="courier new, monospace"># The code of maximal time a packet can wait at the head of</font></div>
<div><font face="courier new, monospace"># transmission queue.</font></div><div><font face="courier new, monospace"># The actual time is 4.096usec * 2^<head_of_queue_lifetime></font></div><div><font face="courier new, monospace"># The value 0x14 disables this mechanism</font></div>
<div><font face="courier new, monospace">head_of_queue_lifetime 0x12</font></div><div><font face="courier new, monospace"><br></font></div><div><font face="courier new, monospace"># The maximal time a packet can wait at the head of queue on</font></div>
<div><font face="courier new, monospace"># switch port connected to a CA or router port</font></div><div><font face="courier new, monospace">leaf_head_of_queue_lifetime 0x10</font></div><div><font face="courier new, monospace"><br>
</font></div><div><font face="courier new, monospace"># Limit the maximal operational VLs</font></div><div><font face="courier new, monospace">max_op_vls 5</font></div><div><font face="courier new, monospace"><br></font></div>
<div><font face="courier new, monospace"># Force PortInfo:LinkSpeedEnabled on switch ports</font></div><div><font face="courier new, monospace"># If 0, don't modify PortInfo:LinkSpeedEnabled on switch port</font></div>
<div><font face="courier new, monospace"># Otherwise, use value for PortInfo:LinkSpeedEnabled on switch port</font></div><div><font face="courier new, monospace"># Values are (IB Spec 1.2.1, 14.2.5.6 Table 146 "PortInfo")</font></div>
<div><font face="courier new, monospace"># 1: 2.5 Gbps</font></div><div><font face="courier new, monospace"># 3: 2.5 or 5.0 Gbps</font></div><div><font face="courier new, monospace"># 5: 2.5 or 10.0 Gbps</font></div>
<div><font face="courier new, monospace"># 7: 2.5 or 5.0 or 10.0 Gbps</font></div><div><font face="courier new, monospace"># 2,4,6,8-14 Reserved</font></div><div><font face="courier new, monospace"># Default 15: set to PortInfo:LinkSpeedSupported</font></div>
<div><font face="courier new, monospace">force_link_speed 15</font></div><div><font face="courier new, monospace"><br></font></div><div><font face="courier new, monospace"># The subnet_timeout code that will be set for all the ports</font></div>
<div><font face="courier new, monospace"># The actual timeout is 4.096usec * 2^<subnet_timeout></font></div><div><font face="courier new, monospace">subnet_timeout 18</font></div><div><font face="courier new, monospace"><br>
</font></div><div><font face="courier new, monospace"># Threshold of local phy errors for sending Trap 129</font></div><div><font face="courier new, monospace">local_phy_errors_threshold 0x08</font></div><div><font face="courier new, monospace"><br>
</font></div><div><font face="courier new, monospace"># Threshold of credit overrun errors for sending Trap 130</font></div><div><font face="courier new, monospace">overrun_errors_threshold 0x08</font></div><div><font face="courier new, monospace"><br>
</font></div><div><font face="courier new, monospace">#</font></div><div><font face="courier new, monospace"># PARTITIONING OPTIONS</font></div><div><font face="courier new, monospace">#</font></div><div><font face="courier new, monospace"># Partition configuration file to be used</font></div>
<div><font face="courier new, monospace">partition_config_file /tmp/osmpart.conf</font></div><div><font face="courier new, monospace"><br></font></div><div><font face="courier new, monospace"># Disable partition enforcement by switches</font></div>
<div><font face="courier new, monospace">no_partition_enforcement FALSE</font></div><div><font face="courier new, monospace"><br></font></div><div><font face="courier new, monospace">#</font></div><div><font face="courier new, monospace"># SWEEP OPTIONS</font></div>
<div><font face="courier new, monospace">#</font></div><div><font face="courier new, monospace"># The number of seconds between subnet sweeps (0 disables it)</font></div><div><font face="courier new, monospace">sweep_interval 10</font></div>
<div><font face="courier new, monospace"><br></font></div><div><font face="courier new, monospace"># If TRUE cause all lids to be reassigned</font></div><div><font face="courier new, monospace">reassign_lids FALSE</font></div>
<div><font face="courier new, monospace"><br></font></div><div><font face="courier new, monospace"># If TRUE forces every sweep to be a heavy sweep</font></div><div><font face="courier new, monospace">force_heavy_sweep FALSE</font></div>
<div><font face="courier new, monospace"><br></font></div><div><font face="courier new, monospace"># If TRUE every trap will cause a heavy sweep.</font></div><div><font face="courier new, monospace"># NOTE: successive identical traps (>10) are suppressed</font></div>
<div><font face="courier new, monospace">sweep_on_trap TRUE</font></div><div><font face="courier new, monospace"><br></font></div><div><font face="courier new, monospace">#</font></div><div><font face="courier new, monospace"># ROUTING OPTIONS</font></div>
<div><font face="courier new, monospace">#</font></div><div><font face="courier new, monospace"># If TRUE count switches as link subscriptions</font></div><div><font face="courier new, monospace">port_profile_switch_nodes FALSE</font></div>
<div><font face="courier new, monospace"><br></font></div><div><font face="courier new, monospace"># Name of file with port guids to be ignored by port profiling</font></div><div><font face="courier new, monospace">port_prof_ignore_file (null)</font></div>
<div><font face="courier new, monospace"><br></font></div><div><font face="courier new, monospace"># The file holding routing weighting factors per output port</font></div><div><font face="courier new, monospace">hop_weights_file (null)</font></div>
<div><font face="courier new, monospace"><br></font></div><div><font face="courier new, monospace"># Routing engine</font></div><div><font face="courier new, monospace"># Multiple routing engines can be specified separated by</font></div>
<div><font face="courier new, monospace"># commas so that specific ordering of routing algorithms will</font></div><div><font face="courier new, monospace"># be tried if earlier routing engines fail.</font></div><div><font face="courier new, monospace"># Supported engines: minhop, updn, file, ftree, lash, dor</font></div>
<div><font face="courier new, monospace">routing_engine (null)</font></div><div><font face="courier new, monospace"><br></font></div><div><font face="courier new, monospace"># Connect roots (use FALSE if unsure)</font></div>
<div><font face="courier new, monospace">connect_roots FALSE</font></div><div><font face="courier new, monospace"><br></font></div><div><font face="courier new, monospace"># Use unicast routing cache (use FALSE if unsure)</font></div>
<div><font face="courier new, monospace">use_ucast_cache FALSE</font></div><div><font face="courier new, monospace"><br></font></div><div><font face="courier new, monospace"># Lid matrix dump file name</font></div><div><font face="courier new, monospace">lid_matrix_dump_file (null)</font></div>
<div><font face="courier new, monospace"><br></font></div><div><font face="courier new, monospace"># LFTs file name</font></div><div><font face="courier new, monospace">lfts_file (null)</font></div><div><font face="courier new, monospace"><br>
</font></div><div><font face="courier new, monospace"># The file holding the root node guids (for fat-tree or Up/Down)</font></div><div><font face="courier new, monospace"># One guid in each line</font></div><div><font face="courier new, monospace">root_guid_file (null)</font></div>
<div><font face="courier new, monospace"><br></font></div><div><font face="courier new, monospace"># The file holding the fat-tree compute node guids</font></div><div><font face="courier new, monospace"># One guid in each line</font></div>
<div><font face="courier new, monospace">cn_guid_file (null)</font></div><div><font face="courier new, monospace"><br></font></div><div><font face="courier new, monospace"># The file holding the fat-tree I/O node guids</font></div>
<div><font face="courier new, monospace"># One guid in each line</font></div><div><font face="courier new, monospace">io_guid_file (null)</font></div><div><font face="courier new, monospace"><br></font></div><div><font face="courier new, monospace"># Number of reverse hops allowed for I/O nodes </font></div>
<div><font face="courier new, monospace"># Used for connectivity between I/O nodes connected to Top Switches</font></div><div><font face="courier new, monospace">max_reverse_hops 0</font></div><div><font face="courier new, monospace"><br>
</font></div><div><font face="courier new, monospace"># The file holding the node ids which will be used by Up/Down algorithm instead</font></div><div><font face="courier new, monospace"># of GUIDs (one guid and id in each line)</font></div>
<div><font face="courier new, monospace">ids_guid_file (null)</font></div><div><font face="courier new, monospace"><br></font></div><div><font face="courier new, monospace"># The file holding guid routing order guids (for MinHop and Up/Down)</font></div>
<div><font face="courier new, monospace">guid_routing_order_file (null)</font></div><div><font face="courier new, monospace"><br></font></div><div><font face="courier new, monospace"># Do mesh topology analysis (for LASH algorithm)</font></div>
<div><font face="courier new, monospace">do_mesh_analysis FALSE</font></div><div><font face="courier new, monospace"><br></font></div><div><font face="courier new, monospace"># Starting VL for LASH algorithm</font></div><div>
<font face="courier new, monospace">lash_start_vl 0</font></div><div><font face="courier new, monospace"><br></font></div><div><font face="courier new, monospace"># SA database file name</font></div><div><font face="courier new, monospace">sa_db_file /var/log/opensm-sa.dump</font></div>
<div><font face="courier new, monospace"><br></font></div><div><font face="courier new, monospace"># If TRUE causes OpenSM to dump SA database at the end of</font></div><div><font face="courier new, monospace"># every light sweep, regardless of the verbosity level</font></div>
<div><font face="courier new, monospace">sa_db_dump TRUE</font></div><div><font face="courier new, monospace"><br></font></div><div><font face="courier new, monospace">#</font></div><div><font face="courier new, monospace"># HANDOVER - MULTIPLE SMs OPTIONS</font></div>
<div><font face="courier new, monospace">#</font></div><div><font face="courier new, monospace"># SM priority used for deciding who is the master</font></div><div><font face="courier new, monospace"># Range goes from 0 (lowest priority) to 15 (highest).</font></div>
<div><font face="courier new, monospace">sm_priority 5</font></div><div><font face="courier new, monospace"><br></font></div><div><font face="courier new, monospace"># If TRUE other SMs on the subnet should be ignored</font></div>
<div><font face="courier new, monospace">ignore_other_sm FALSE</font></div><div><font face="courier new, monospace"><br></font></div><div><font face="courier new, monospace"># Timeout in [msec] between two polls of active master SM</font></div>
<div><font face="courier new, monospace">sminfo_polling_timeout 10000</font></div><div><font face="courier new, monospace"><br></font></div><div><font face="courier new, monospace"># Number of failing polls of remote SM that declares it dead</font></div>
<div><font face="courier new, monospace">polling_retry_number 4</font></div><div><font face="courier new, monospace"><br></font></div><div><font face="courier new, monospace"># If TRUE honor the guid2lid file when coming out of standby</font></div>
<div><font face="courier new, monospace"># state, if such file exists and is valid</font></div><div><font face="courier new, monospace">honor_guid2lid_file FALSE</font></div><div><font face="courier new, monospace"><br></font></div>
<div><font face="courier new, monospace">#</font></div><div><font face="courier new, monospace"># TIMING AND THREADING OPTIONS</font></div><div><font face="courier new, monospace">#</font></div><div><font face="courier new, monospace"># Maximum number of SMPs sent in parallel</font></div>
<div><font face="courier new, monospace">max_wire_smps 4</font></div><div><font face="courier new, monospace"><br></font></div><div><font face="courier new, monospace"># The maximum time in [msec] allowed for a transaction to complete</font></div>
<div><font face="courier new, monospace">transaction_timeout 2000</font></div><div><font face="courier new, monospace"><br></font></div><div><font face="courier new, monospace"># The maximum number of retries allowed for a transaction to complete</font></div>
<div><font face="courier new, monospace">transaction_retries 3</font></div><div><font face="courier new, monospace"><br></font></div><div><font face="courier new, monospace"># Maximal time in [msec] a message can stay in the incoming message queue.</font></div>
<div><font face="courier new, monospace"># If there is more than one message in the queue and the last message</font></div><div><font face="courier new, monospace"># stayed in the queue more than this value, any SA request will be</font></div>
<div><font face="courier new, monospace"># immediately returned with a BUSY status.</font></div><div><font face="courier new, monospace">max_msg_fifo_timeout 10000</font></div><div><font face="courier new, monospace"><br>
</font></div><div><font face="courier new, monospace"># Use a single thread for handling SA queries</font></div><div><font face="courier new, monospace">single_thread FALSE</font></div><div><font face="courier new, monospace"><br>
</font></div><div><font face="courier new, monospace">#</font></div><div><font face="courier new, monospace"># MISC OPTIONS</font></div><div><font face="courier new, monospace">#</font></div><div><font face="courier new, monospace"># Daemon mode</font></div>
<div><font face="courier new, monospace">daemon FALSE</font></div><div><font face="courier new, monospace"><br></font></div><div><font face="courier new, monospace"># SM Inactive</font></div><div><font face="courier new, monospace">sm_inactive FALSE</font></div>
<div><font face="courier new, monospace"><br></font></div><div><font face="courier new, monospace"># Babbling Port Policy</font></div><div><font face="courier new, monospace">babbling_port_policy FALSE</font></div><div><font face="courier new, monospace"><br>
</font></div><div><font face="courier new, monospace">#</font></div><div><font face="courier new, monospace"># Performance Manager Options</font></div><div><font face="courier new, monospace">#</font></div><div><font face="courier new, monospace"># perfmgr enable</font></div>
<div><font face="courier new, monospace">perfmgr FALSE</font></div><div><font face="courier new, monospace"><br></font></div><div><font face="courier new, monospace"># perfmgr redirection enable</font></div><div><font face="courier new, monospace">perfmgr_redir TRUE</font></div>
<div><font face="courier new, monospace"><br></font></div><div><font face="courier new, monospace"># sweep time in seconds</font></div><div><font face="courier new, monospace">perfmgr_sweep_time_s 180</font></div><div><font face="courier new, monospace"><br>
</font></div><div><font face="courier new, monospace"># Max outstanding queries</font></div><div><font face="courier new, monospace">perfmgr_max_outstanding_queries 500</font></div><div><font face="courier new, monospace"><br>
</font></div><div><font face="courier new, monospace">#</font></div><div><font face="courier new, monospace"># Event DB Options</font></div><div><font face="courier new, monospace">#</font></div><div><font face="courier new, monospace"># Dump file to dump the events to</font></div>
<div><font face="courier new, monospace">event_db_dump_file (null)</font></div><div><font face="courier new, monospace"><br></font></div><div><font face="courier new, monospace">#</font></div><div><font face="courier new, monospace"># Event Plugin Options</font></div>
<div><font face="courier new, monospace">#</font></div><div><font face="courier new, monospace">event_plugin_name (null)</font></div><div><font face="courier new, monospace"><br></font></div><div><font face="courier new, monospace">#</font></div>
<div><font face="courier new, monospace"># Node name map for mapping node's to more descriptive node descriptions</font></div><div><font face="courier new, monospace"># (man ibnetdiscover for more information)</font></div>
<div><font face="courier new, monospace">#</font></div><div><font face="courier new, monospace">node_name_map_name (null)</font></div><div><font face="courier new, monospace"><br></font></div><div><font face="courier new, monospace">#</font></div>
<div><font face="courier new, monospace"># DEBUG FEATURES</font></div><div><font face="courier new, monospace">#</font></div><div><font face="courier new, monospace"># The log flags used</font></div><div><font face="courier new, monospace">log_flags 0x03</font></div>
<div><font face="courier new, monospace"><br></font></div><div><font face="courier new, monospace"># Force flush of the log file after each log message</font></div><div><font face="courier new, monospace">force_log_flush FALSE</font></div>
<div><font face="courier new, monospace"><br></font></div><div><font face="courier new, monospace"># Log file to be used</font></div><div><font face="courier new, monospace">log_file /var/log/opensm.log</font></div><div><font face="courier new, monospace"><br>
</font></div><div><font face="courier new, monospace"># Limit the size of the log file in MB. If overrun, log is restarted</font></div><div><font face="courier new, monospace">log_max_size 100</font></div><div><font face="courier new, monospace"><br>
</font></div><div><font face="courier new, monospace"># If TRUE will accumulate the log over multiple OpenSM sessions</font></div><div><font face="courier new, monospace">accum_log_file TRUE</font></div><div><font face="courier new, monospace"><br>
</font></div><div><font face="courier new, monospace"># The directory to hold the file OpenSM dumps</font></div><div><font face="courier new, monospace">dump_files_dir /var/log/</font></div><div><font face="courier new, monospace"><br>
</font></div><div><font face="courier new, monospace"># If TRUE enables new high risk options and hardware specific quirks</font></div><div><font face="courier new, monospace">enable_quirks FALSE</font></div><div><font face="courier new, monospace"><br>
</font></div><div><font face="courier new, monospace"># If TRUE disables client reregistration</font></div><div><font face="courier new, monospace">no_clients_rereg FALSE</font></div><div><font face="courier new, monospace"><br>
</font></div><div><font face="courier new, monospace"># If TRUE OpenSM should disable multicast support and</font></div><div><font face="courier new, monospace"># no multicast routing is performed if TRUE</font></div><div>
<font face="courier new, monospace">disable_multicast FALSE</font></div><div><font face="courier new, monospace"><br></font></div><div><font face="courier new, monospace"># If TRUE opensm will exit on fatal initialization issues</font></div>
<div><font face="courier new, monospace">exit_on_fatal FALSE</font></div><div><font face="courier new, monospace"><br></font></div><div><font face="courier new, monospace"># console [off|local|loopback|socket]</font></div>
<div><font face="courier new, monospace">console loopback</font></div><div><font face="courier new, monospace"><br></font></div><div><font face="courier new, monospace"># Telnet port for console (default 10000)</font></div>
<div><font face="courier new, monospace">console_port 10000</font></div><div><font face="courier new, monospace"><br></font></div><div><font face="courier new, monospace">#</font></div><div><font face="courier new, monospace"># QoS OPTIONS</font></div>
<div><font face="courier new, monospace">#</font></div><div><font face="courier new, monospace"># Enable QoS setup</font></div><div><font face="courier new, monospace">qos FALSE</font></div><div><font face="courier new, monospace"><br>
</font></div><div><font face="courier new, monospace"># QoS policy file to be used</font></div><div><font face="courier new, monospace">qos_policy_file /opt/xsigo/xsigos/current/ofed/etc/opensm/qos-policy.conf</font></div>
<div><font face="courier new, monospace"><br></font></div><div><font face="courier new, monospace"># QoS default options</font></div><div><font face="courier new, monospace">qos_max_vls 0</font></div><div><font face="courier new, monospace">qos_high_limit -1</font></div>
<div><font face="courier new, monospace">qos_vlarb_high (null)</font></div><div><font face="courier new, monospace">qos_vlarb_low (null)</font></div><div><font face="courier new, monospace">qos_sl2vl (null)</font></div><div>
<font face="courier new, monospace"><br></font></div><div><font face="courier new, monospace"># QoS CA options</font></div><div><font face="courier new, monospace">qos_ca_max_vls 0</font></div><div><font face="courier new, monospace">qos_ca_high_limit -1</font></div>
<div><font face="courier new, monospace">qos_ca_vlarb_high (null)</font></div><div><font face="courier new, monospace">qos_ca_vlarb_low (null)</font></div><div><font face="courier new, monospace">qos_ca_sl2vl (null)</font></div>
<div><font face="courier new, monospace"><br></font></div><div><font face="courier new, monospace"># QoS Switch Port 0 options</font></div><div><font face="courier new, monospace">qos_sw0_max_vls 0</font></div><div><font face="courier new, monospace">qos_sw0_high_limit -1</font></div>
<div><font face="courier new, monospace">qos_sw0_vlarb_high (null)</font></div><div><font face="courier new, monospace">qos_sw0_vlarb_low (null)</font></div><div><font face="courier new, monospace">qos_sw0_sl2vl (null)</font></div>
<div><font face="courier new, monospace"><br></font></div><div><font face="courier new, monospace"># QoS Switch external ports options</font></div><div><font face="courier new, monospace">qos_swe_max_vls 0</font></div><div>
<font face="courier new, monospace">qos_swe_high_limit -1</font></div><div><font face="courier new, monospace">qos_swe_vlarb_high (null)</font></div><div><font face="courier new, monospace">qos_swe_vlarb_low (null)</font></div>
<div><font face="courier new, monospace">qos_swe_sl2vl (null)</font></div><div><font face="courier new, monospace"><br></font></div><div><font face="courier new, monospace"># QoS Router ports options</font></div><div><font face="courier new, monospace">qos_rtr_max_vls 0</font></div>
<div><font face="courier new, monospace">qos_rtr_high_limit -1</font></div><div><font face="courier new, monospace">qos_rtr_vlarb_high (null)</font></div><div><font face="courier new, monospace">qos_rtr_vlarb_low (null)</font></div>
<div><font face="courier new, monospace">qos_rtr_sl2vl (null)</font></div><div><font face="courier new, monospace"><br></font></div><div><font face="courier new, monospace"># Prefix routes file name</font></div><div><font face="courier new, monospace">prefix_routes_file /opt/xsigo/xsigos/current/ofed/etc/opensm/prefix-routes.conf</font></div>
<div><font face="courier new, monospace"><br></font></div><div><font face="courier new, monospace">#</font></div><div><font face="courier new, monospace"># IPv6 Solicited Node Multicast (SNM) Options</font></div><div><font face="courier new, monospace">#</font></div>
<div><font face="courier new, monospace">consolidate_ipv6_snm_req FALSE</font></div><div><font face="courier new, monospace"><br></font></div><div><font face="courier new, monospace">OpenSM $ </font></div></div><div><br></div>
<div>Thanks again for all your help.</div></div><div class="gmail_extra"><br clear="all"><div><div><span style="font-family:arial,sans-serif;font-size:13px;background-color:rgb(255,255,255)"><br></span></div><span style="font-family:arial,sans-serif;font-size:13px;background-color:rgb(255,255,255)">Robert LeBlanc</span><br style="font-family:arial,sans-serif;font-size:13px;background-color:rgb(255,255,255)">
<span style="font-family:arial,sans-serif;font-size:13px;background-color:rgb(255,255,255)">OIT Infrastructure & Virtualization Engineer</span><br style="font-family:arial,sans-serif;font-size:13px;background-color:rgb(255,255,255)">
<span style="font-family:arial,sans-serif;font-size:13px;background-color:rgb(255,255,255)">Brigham Young University</span></div>
<br><br><div class="gmail_quote">On Wed, Nov 13, 2013 at 12:27 PM, Robert LeBlanc <span dir="ltr"><<a href="mailto:robert_leblanc@byu.edu" target="_blank">robert_leblanc@byu.edu</a>></span> wrote:<br><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex">
<div dir="ltr">They told me in the past that we could use our own external subnet manager or the one built into their box.</div><div class="gmail_extra"><div class="im"><br clear="all"><div><div><span style="font-size:13px;font-family:arial,sans-serif"><br>
</span></div><span style="font-size:13px;font-family:arial,sans-serif">Robert LeBlanc</span><br style="font-size:13px;font-family:arial,sans-serif"><span style="font-size:13px;font-family:arial,sans-serif">OIT Infrastructure & Virtualization Engineer</span><br style="font-size:13px;font-family:arial,sans-serif">
<span style="font-size:13px;font-family:arial,sans-serif">Brigham Young University</span></div>
<br><br></div><div><div class="h5"><div class="gmail_quote">On Wed, Nov 13, 2013 at 12:25 PM, Hal Rosenstock <span dir="ltr"><<a href="mailto:hal.rosenstock@gmail.com" target="_blank">hal.rosenstock@gmail.com</a>></span> wrote:<br>
<blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex">
<div dir="ltr">Yes but I'm not sure what the Xsigo SM "special sauce" is so those boxes may not function properly.</div><div><div><div class="gmail_extra"><br><br><div class="gmail_quote">
On Wed, Nov 13, 2013 at 2:13 PM, Robert LeBlanc <span dir="ltr"><<a href="mailto:robert_leblanc@byu.edu" target="_blank">robert_leblanc@byu.edu</a>></span> wrote:<br>
<blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex"><div dir="ltr">The front line Oracle tech is giving me some hog wash that it is a problem with Dell and Mellanox and that the subnet manager is not at fault (although they are passing the request to engineering). I think I'm just going to run OpenSM on this test node (after reducing the priority on the Oracle sm) and see if the problem clears up.</div>
<div class="gmail_extra"><div><br clear="all"><div><div><span style="font-family:arial,sans-serif;font-size:13px"><br></span></div><span style="font-family:arial,sans-serif;font-size:13px">Robert LeBlanc</span><br style="font-family:arial,sans-serif;font-size:13px">
<span style="font-family:arial,sans-serif;font-size:13px">OIT Infrastructure & Virtualization Engineer</span><br style="font-family:arial,sans-serif;font-size:13px">
<span style="font-family:arial,sans-serif;font-size:13px">Brigham Young University</span></div>
<br><br></div><div><div><div class="gmail_quote">On Wed, Nov 13, 2013 at 12:08 PM, Hal Rosenstock <span dir="ltr"><<a href="mailto:hal.rosenstock@gmail.com" target="_blank">hal.rosenstock@gmail.com</a>></span> wrote:<br>
<blockquote style="margin:0px 0px 0px 0.8ex;padding-left:1ex;border-left-color:rgb(204,204,204);border-left-width:1px;border-left-style:solid" class="gmail_quote">
<div dir="ltr">Yes, IPoIB uses multicast groups for the IP broadcast group and any IP multicast groups. You can see those with saquery -g. But depending on the locations of the ports running IPoIB and your topology, a multicast group may or may not be routed via a particular switch.</div>
<div><div>
<div class="gmail_extra"><br><br><div class="gmail_quote">On Wed, Nov 13, 2013 at 2:06 PM, Robert LeBlanc <span dir="ltr"><<a href="mailto:robert_leblanc@byu.edu" target="_blank">robert_leblanc@byu.edu</a>></span> wrote:<br>
<blockquote style="margin:0px 0px 0px 0.8ex;padding-left:1ex;border-left-color:rgb(204,204,204);border-left-width:1px;border-left-style:solid" class="gmail_quote"><p dir="ltr">Ipoib uses multicast, right? I'm guessing that is why I can't get ipoib to work on our blades but our rack servers can.</p>
<p dir="ltr">Robert LeBlanc<br>
Virtualization and Server Engineer<br>
Brigham Young University</p>
<p dir="ltr">Sent from a mobile device, please excuse any typos.</p><div><div>
<div class="gmail_quote">On Nov 13, 2013 12:03 PM, "Hal Rosenstock" <<a href="mailto:hal.rosenstock@gmail.com" target="_blank">hal.rosenstock@gmail.com</a>> wrote:<br type="attribution"><blockquote style="margin:0px 0px 0px 0.8ex;padding-left:1ex;border-left-color:rgb(204,204,204);border-left-width:1px;border-left-style:solid" class="gmail_quote">
<div dir="ltr"><div>That should be fine. 7.4.3000 looks like the latest.</div><div> </div><div>This looks like an SM issue missetting that parameter in the switch assuming that there are some MC groups routed through that switch.</div>
</div><div class="gmail_extra"><br><br><div class="gmail_quote">On Wed, Nov 13, 2013 at 1:55 PM, Robert LeBlanc <span dir="ltr"><<a href="mailto:robert_leblanc@byu.edu" target="_blank">robert_leblanc@byu.edu</a>></span> wrote:<br>
<blockquote style="margin:0px 0px 0px 0.8ex;padding-left:1ex;border-left-color:rgb(204,204,204);border-left-width:1px;border-left-style:solid" class="gmail_quote"><div dir="ltr"><div><font face="courier new, monospace">[root@desxi003 ~]# flint -d /dev/mst/SW_MT48438_0x2c90200448e28_lid-0x0034 q</font></div>
<div><font face="courier new, monospace">Image type: FS2</font></div><div>
<font face="courier new, monospace">FW Version: 7.4.0</font></div><div><font face="courier new, monospace">Device ID: 48438</font></div><div><font face="courier new, monospace">Description: Node Sys image</font></div>
<div><font face="courier new, monospace">GUIDs: 0002c90200448e28 0002c90200448e2b </font></div><div><font face="courier new, monospace">Board ID: n/a (DEL08D0110003)</font></div><div><font face="courier new, monospace">VSD: n/a</font></div>
<div><font face="courier new, monospace">PSID: DEL08D0110003</font></div><div><br></div></div><div class="gmail_extra"><div><br clear="all"><div><div><span style="font-family:arial,sans-serif;font-size:13px"><br>
</span></div><span style="font-family:arial,sans-serif;font-size:13px">Robert LeBlanc</span><br style="font-family:arial,sans-serif;font-size:13px"><span style="font-family:arial,sans-serif;font-size:13px">OIT Infrastructure & Virtualization Engineer</span><br style="font-family:arial,sans-serif;font-size:13px">
<span style="font-family:arial,sans-serif;font-size:13px">Brigham Young University</span></div>
<br><br></div><div><div><div class="gmail_quote">On Wed, Nov 13, 2013 at 11:52 AM, Hal Rosenstock <span dir="ltr"><<a href="mailto:hal.rosenstock@gmail.com" target="_blank">hal.rosenstock@gmail.com</a>></span> wrote:<br>
<blockquote style="margin:0px 0px 0px 0.8ex;padding-left:1ex;border-left-color:rgb(204,204,204);border-left-width:1px;border-left-style:solid" class="gmail_quote">
<div dir="ltr"><div>What's the latest firmware version ?</div><div> </div><div>Can you determine the firmware version of the switches ? vendstat -N <switch lid> might work to show this.</div><div> </div><div>This is important...</div>
<div> </div><div>Thanks.</div><span><font color="#888888"><div> </div><div>-- Hal</div></font></span></div><div><div><div class="gmail_extra"><br><br><div class="gmail_quote">On Wed, Nov 13, 2013 at 1:46 PM, Robert LeBlanc <span dir="ltr"><<a href="mailto:robert_leblanc@byu.edu" target="_blank">robert_leblanc@byu.edu</a>></span> wrote:<br>
<blockquote style="margin:0px 0px 0px 0.8ex;padding-left:1ex;border-left-color:rgb(204,204,204);border-left-width:1px;border-left-style:solid" class="gmail_quote"><div dir="ltr">Thanks for all the help so far, this is a great community! I've fed all this info back to Oracle and I'll have to see what they say.<div>
<br></div><div>Thanks,</div></div><div class="gmail_extra"><div><br clear="all">
<div><div><span style="font-family:arial,sans-serif;font-size:13px"><br></span></div><span style="font-family:arial,sans-serif;font-size:13px">Robert LeBlanc</span><br style="font-family:arial,sans-serif;font-size:13px">
<span style="font-family:arial,sans-serif;font-size:13px">OIT Infrastructure & Virtualization Engineer</span><br style="font-family:arial,sans-serif;font-size:13px">
<span style="font-family:arial,sans-serif;font-size:13px">Brigham Young University</span></div>
<br><br></div><div><div><div class="gmail_quote">On Wed, Nov 13, 2013 at 11:40 AM, Hal Rosenstock <span dir="ltr"><<a href="mailto:hal.rosenstock@gmail.com" target="_blank">hal.rosenstock@gmail.com</a>></span> wrote:<br>
<blockquote style="margin:0px 0px 0px 0.8ex;padding-left:1ex;border-left-color:rgb(204,204,204);border-left-width:1px;border-left-style:solid" class="gmail_quote">
<div dir="ltr"><div>Yes, this is the cause of the issues.</div><div> </div><div>smpdump (and smpquery) merely query (get) and don't set parameters and anyhow, the SM would overwrite it when it thought it needed to update it. It's an SM and/or firmware issue.</div>
</div><div><div><div class="gmail_extra"><br><br><div class="gmail_quote">On Wed, Nov 13, 2013 at 1:38 PM, Robert LeBlanc <span dir="ltr"><<a href="mailto:robert_leblanc@byu.edu" target="_blank">robert_leblanc@byu.edu</a>></span> wrote:<br>
<blockquote style="margin:0px 0px 0px 0.8ex;padding-left:1ex;border-left-color:rgb(204,204,204);border-left-width:1px;border-left-style:solid" class="gmail_quote"><div dir="ltr">We are on the latest version of firmware for all of our switches (as of last month). I guess I'll have to check with Oracle and see if they are setting this parameter in their subnet manager. Just to confirm, using smpdump (or similar) to change the value won't do any good because the subnet manager will just change it back?<div>
<br></div><div>I think this is the cause of the problems, now to get it fixed.<br><div><br></div><div>Thanks,</div></div></div><div class="gmail_extra"><div><br clear="all"><div><div><span style="font-family:arial,sans-serif;font-size:13px"><br>
</span></div><span style="font-family:arial,sans-serif;font-size:13px">Robert LeBlanc</span><br style="font-family:arial,sans-serif;font-size:13px"><span style="font-family:arial,sans-serif;font-size:13px">OIT Infrastructure & Virtualization Engineer</span><br style="font-family:arial,sans-serif;font-size:13px">
<span style="font-family:arial,sans-serif;font-size:13px">Brigham Young University</span></div>
<br><br></div><div><div><div class="gmail_quote">On Wed, Nov 13, 2013 at 11:34 AM, Hal Rosenstock <span dir="ltr"><<a href="mailto:hal.rosenstock@gmail.com" target="_blank">hal.rosenstock@gmail.com</a>></span> wrote:<br>
<blockquote style="margin:0px 0px 0px 0.8ex;padding-left:1ex;border-left-color:rgb(204,204,204);border-left-width:1px;border-left-style:solid" class="gmail_quote">
<div dir="ltr"><div>In general, MulticastFDBTop should be 0 or some value above 0xc000.</div><div> </div><div><font size="1" face="ArialMT"><font size="1" face="ArialMT"><p align="LEFT">Indicates the upper bound of the range of the multicast</p>
<p align="LEFT">forwarding table. Packets received with MLIDs greater</p>
<p align="LEFT">than MulticastFDBTop are considered to be outside the</p>
<p align="LEFT">range of the Multicast Forwarding Table (see </p></font></font><p align="LEFT"><font color="#0000ff" size="1" face="ArialMT"><font color="#0000ff" size="1" face="ArialMT"><font color="#0000ff" size="1" face="ArialMT">18.2.4.3.3</font></font></font></p>
<font color="#0000ff" size="1" face="ArialMT"><font color="#0000ff" size="1" face="ArialMT"><font color="#0000ff" size="1" face="ArialMT">
<p align="LEFT">Required Multicast Relay on page 1072</p></font></font></font><p align="LEFT"><font size="1" face="ArialMT"><font size="1" face="ArialMT">). A valid MulticastFDBTop</font></font></p><font size="1" face="ArialMT"><font size="1" face="ArialMT">
<p align="LEFT">is less than MulticastFDBCap + 0xC000.</p>
<p align="LEFT">This component applies only to switches that implement</p>
<p align="LEFT">the optional multicast forwarding service. A switch</p>
<p align="LEFT">shall ignore the MulticastFDBTop component if it has</p>
<p align="LEFT">the value zero. The initial value for MulticastFDBTop</p>
<p align="LEFT">shall be set to zero. A value of 0xBFFF means there are</p>
<p>no MulticastForwardingTable entries.</p></font></font></div><div>It is set by OpenSM. There is a parameter to disable it's use (use_mfttop) which can be set to FALSE. This may depend on which OpenSM version you are running. In order to get out of this state, you may need to reset any switches which have this parameter set like this.</div>
<div> </div><div>Any idea on the firmware versions in your various switches ?</div><span><font color="#888888"><div> </div><div>-- Hal</div></font></span></div><div><div><div class="gmail_extra">
<br><br><div class="gmail_quote">On Wed, Nov 13, 2013 at 1:16 PM, Robert LeBlanc <span dir="ltr"><<a href="mailto:robert_leblanc@byu.edu" target="_blank">robert_leblanc@byu.edu</a>></span> wrote:<br>
<blockquote style="margin:0px 0px 0px 0.8ex;padding-left:1ex;border-left-color:rgb(204,204,204);border-left-width:1px;border-left-style:solid" class="gmail_quote"><div dir="ltr">Sorry to take so long, I've been busy with other things. Here is the output:<div>
<br></div><div><div>
<font face="courier new, monospace">[root@desxi003 ~]# smpquery si 52</font></div><div><font face="courier new, monospace"># Switch info: Lid 52</font></div>
<div><font face="courier new, monospace">LinearFdbCap:....................49152</font></div><div><font face="courier new, monospace">RandomFdbCap:....................0</font></div><div><font face="courier new, monospace">McastFdbCap:.....................4096</font></div>
<div><font face="courier new, monospace">LinearFdbTop:....................189</font></div><div><font face="courier new, monospace">DefPort:.........................0</font></div><div><font face="courier new, monospace">DefMcastPrimPort:................255</font></div>
<div><font face="courier new, monospace">DefMcastNotPrimPort:.............255</font></div><div><font face="courier new, monospace">LifeTime:........................18</font></div><div><font face="courier new, monospace">StateChange:.....................0</font></div>
<div><font face="courier new, monospace">OptSLtoVLMapping:................1</font></div><div><font face="courier new, monospace">LidsPerPort:.....................0</font></div><div><font face="courier new, monospace">PartEnforceCap:..................32</font></div>
<div><font face="courier new, monospace">InboundPartEnf:..................1</font></div><div><font face="courier new, monospace">OutboundPartEnf:.................1</font></div><div><font face="courier new, monospace">FilterRawInbound:................1</font></div>
<div><font face="courier new, monospace">FilterRawOutbound:...............1</font></div><div><font face="courier new, monospace">EnhancedPort0:...................0</font></div><div><font face="courier new, monospace">MulticastFDBTop:.................0xbfff</font></div>
<div><font face="courier new, monospace">[root@desxi003 ~]# smpquery pi 52 0</font></div><div><font face="courier new, monospace"># Port info: Lid 52 port 0</font></div><div><font face="courier new, monospace">Mkey:............................0x0000000000000000</font></div>
<div><font face="courier new, monospace">GidPrefix:.......................0xfe80000000000000</font></div><div><font face="courier new, monospace">Lid:.............................52</font></div><div><font face="courier new, monospace">SMLid:...........................49</font></div>
<div><font face="courier new, monospace">CapMask:.........................0x42500848</font></div><div><font face="courier new, monospace"> IsTrapSupported</font></div><div><font face="courier new, monospace"> IsSLMappingSupported</font></div>
<div><font face="courier new, monospace"> IsSystemImageGUIDsupported</font></div><div><font face="courier new, monospace"> IsVendorClassSupported</font></div>
<div>
<font face="courier new, monospace"> IsCapabilityMaskNoticeSupported</font></div><div><font face="courier new, monospace"> IsClientRegistrationSupported</font></div>
<div><font face="courier new, monospace"> IsMulticastFDBTopSupported</font></div><div><font face="courier new, monospace">DiagCode:........................0x0000</font></div><div><font face="courier new, monospace">MkeyLeasePeriod:.................0</font></div>
<div><font face="courier new, monospace">LocalPort:.......................1</font></div><div><font face="courier new, monospace">LinkWidthEnabled:................1X or 4X</font></div><div><font face="courier new, monospace">LinkWidthSupported:..............1X or 4X</font></div>
<div><font face="courier new, monospace">LinkWidthActive:.................4X</font></div><div><font face="courier new, monospace">LinkSpeedSupported:..............2.5 Gbps or 5.0 Gbps or 10.0 Gbps</font></div><div><font face="courier new, monospace">LinkState:.......................Active</font></div>
<div><font face="courier new, monospace">PhysLinkState:...................LinkUp</font></div><div><font face="courier new, monospace">LinkDownDefState:................Polling</font></div><div><font face="courier new, monospace">ProtectBits:.....................0</font></div>
<div><font face="courier new, monospace">LMC:.............................0</font></div><div><font face="courier new, monospace">LinkSpeedActive:.................10.0 Gbps</font></div><div><font face="courier new, monospace">LinkSpeedEnabled:................2.5 Gbps or 5.0 Gbps or 10.0 Gbps</font></div>
<div><font face="courier new, monospace">NeighborMTU:.....................4096</font></div><div><font face="courier new, monospace">SMSL:............................0</font></div><div><font face="courier new, monospace">VLCap:...........................VL0</font></div>
<div><font face="courier new, monospace">InitType:........................0x00</font></div><div><font face="courier new, monospace">VLHighLimit:.....................0</font></div><div><font face="courier new, monospace">VLArbHighCap:....................0</font></div>
<div><font face="courier new, monospace">VLArbLowCap:.....................0</font></div><div><font face="courier new, monospace">InitReply:.......................0x00</font></div><div><font face="courier new, monospace">MtuCap:..........................4096</font></div>
<div><font face="courier new, monospace">VLStallCount:....................0</font></div><div><font face="courier new, monospace">HoqLife:.........................0</font></div><div><font face="courier new, monospace">OperVLs:.........................VL0</font></div>
<div><font face="courier new, monospace">PartEnforceInb:..................0</font></div><div><font face="courier new, monospace">PartEnforceOutb:.................0</font></div><div><font face="courier new, monospace">FilterRawInb:....................0</font></div>
<div><font face="courier new, monospace">FilterRawOutb:...................0</font></div><div><font face="courier new, monospace">MkeyViolations:..................0</font></div><div><font face="courier new, monospace">PkeyViolations:..................0</font></div>
<div><font face="courier new, monospace">QkeyViolations:..................0</font></div><div><font face="courier new, monospace">GuidCap:.........................1</font></div><div><font face="courier new, monospace">ClientReregister:................0</font></div>
<div><font face="courier new, monospace">McastPkeyTrapSuppressionEnabled:.0</font></div><div><font face="courier new, monospace">SubnetTimeout:...................18</font></div><div><font face="courier new, monospace">RespTimeVal:.....................20</font></div>
<div><font face="courier new, monospace">LocalPhysErr:....................0</font></div><div><font face="courier new, monospace">OverrunErr:......................0</font></div><div><font face="courier new, monospace">MaxCreditHint:...................0</font></div>
<div><font face="courier new, monospace">RoundTrip:.......................0</font></div></div><div><br></div><div>From what I've read in the Mellanox Release Notes MultiCastFDBTop=0xBFFF is supposed to discard MC traffic. The question is, how do I set this value to something else and what should it be set to?</div>
<div><br></div><div>Thanks,</div></div><div class="gmail_extra"><div><br clear="all"><div><div><span style="font-family:arial,sans-serif;font-size:13px"><br></span></div><span style="font-family:arial,sans-serif;font-size:13px">Robert LeBlanc</span><br style="font-family:arial,sans-serif;font-size:13px">
<span style="font-family:arial,sans-serif;font-size:13px">OIT Infrastructure & Virtualization Engineer</span><br style="font-family:arial,sans-serif;font-size:13px">
<span style="font-family:arial,sans-serif;font-size:13px">Brigham Young University</span></div>
<br><br></div><div><div><div class="gmail_quote">On Wed, Oct 30, 2013 at 12:28 PM, Hal Rosenstock <span dir="ltr"><<a href="mailto:hal.rosenstock@gmail.com" target="_blank">hal.rosenstock@gmail.com</a>></span> wrote:<br>
<blockquote style="margin:0px 0px 0px 0.8ex;padding-left:1ex;border-left-color:rgb(204,204,204);border-left-width:1px;border-left-style:solid" class="gmail_quote">
<div dir="ltr"><div> Determine LID of switch (in the below say switch is lid x)</div><div><div> </div><div>Then:</div><div> </div><div>smpquery si x</div><div>(of interest are McastFdbCap and MulticastFDBTop)</div><div> </div>
<div>smpquery pi x 0</div><div>(of interest is CapMask)</div><div> </div><div>ibroute -M x</div></div><div> </div></div><div><div><div class="gmail_extra"><br><br><div class="gmail_quote">On Tue, Oct 29, 2013 at 3:56 PM, Robert LeBlanc <span dir="ltr"><<a href="mailto:robert_leblanc@byu.edu" target="_blank">robert_leblanc@byu.edu</a>></span> wrote:<br>
<blockquote style="margin:0px 0px 0px 0.8ex;padding-left:1ex;border-left-color:rgb(204,204,204);border-left-width:1px;border-left-style:solid" class="gmail_quote"><div dir="ltr">Both ports show up in the "saquery MCMR" results with a JoinState of 0x1.<div>
<br></div><div>
How can I dump the parameters of a non-managed switch so that I can confirm that multicast is not turned off on the Dell chassis IB switches?</div>
</div><div class="gmail_extra"><div><br clear="all"><div><div><span style="font-family:arial,sans-serif;font-size:13px"><br></span></div><span style="font-family:arial,sans-serif;font-size:13px">Robert LeBlanc</span><br style="font-family:arial,sans-serif;font-size:13px">
<span style="font-family:arial,sans-serif;font-size:13px">OIT Infrastructure & Virtualization Engineer</span><br style="font-family:arial,sans-serif;font-size:13px">
<span style="font-family:arial,sans-serif;font-size:13px">Brigham Young University</span></div>
<br><br></div><div><div><div class="gmail_quote">On Mon, Oct 28, 2013 at 5:04 PM, Coulter, Susan K <span dir="ltr"><<a href="mailto:skc@lanl.gov" target="_blank">skc@lanl.gov</a>></span> wrote:<br><blockquote style="margin:0px 0px 0px 0.8ex;padding-left:1ex;border-left-color:rgb(204,204,204);border-left-width:1px;border-left-style:solid" class="gmail_quote">
<div style="word-wrap:break-word">
<div><br>
</div>
<div>/sys/class/net should give you the details on your devices, like this:</div>
<div><br>
</div>
<div>
<div>-bash-4.1# cd /sys/class/net</div>
<div>-bash-4.1# ls -l</div>
<div>total 0</div>
<div>lrwxrwxrwx 1 root root 0 Oct 23 12:59 eth0 -> ../../devices/pci0000:00/0000:00:02.0/0000:04:00.0/net/eth0</div>
<div>lrwxrwxrwx 1 root root 0 Oct 23 12:59 eth1 -> ../../devices/pci0000:00/0000:00:02.0/0000:04:00.1/net/eth1</div>
<div>lrwxrwxrwx 1 root root 0 Oct 23 15:42 ib0 -> ../../devices/pci0000:40/0000:40:0c.0/0000:47:00.0/net/ib0</div>
<div>lrwxrwxrwx 1 root root 0 Oct 23 15:42 ib1 -> ../../devices/pci0000:40/0000:40:0c.0/0000:47:00.0/net/ib1</div>
<div>lrwxrwxrwx 1 root root 0 Oct 23 15:42 ib2 -> ../../devices/pci0000:c0/0000:c0:0c.0/0000:c7:00.0/net/ib2</div>
<div>lrwxrwxrwx 1 root root 0 Oct 23 15:42 ib3 -> ../../devices/pci0000:c0/0000:c0:0c.0/0000:c7:00.0/net/ib3</div>
<div><br>
</div>
</div>
<div>Then use "lspci | grep Mell" to get the pci device numbers.</div>
<div><br>
</div>
<div>
<div>47:00.0 Network controller: Mellanox Technologies MT26428 [ConnectX VPI PCIe 2.0 5GT/s - IB QDR / 10GigE] (rev b0)</div>
<div>c7:00.0 Network controller: Mellanox Technologies MT26428 [ConnectX VPI PCIe 2.0 5GT/s - IB QDR / 10GigE] (rev b0)</div>
</div>
<div><br>
</div>
<div>In this example, ib0 and 1 are referencing the device at 47:00.0</div>
<div>And ib2 and ib3 are referencing the device at c7:00.0</div>
<div><br>
</div>
<div>That said, if you only have one card - this is probably not the problem.</div>
<div>Additionally, since the arp requests are being seen going out ib0, your emulation appears to be working.</div>
<div><br>
</div>
<div>If those arp requests are not being seen on the other end, it seems like a problem with the mgids.</div>
<div>Like maybe the port you are trying to reach is not in the IPoIB multicast group?</div>
<div><br>
</div>
<div>You can look at all the multicast member records with "saquery MCMR".</div>
<div>Or - you can grep for mcmr_rcv_join_mgrp references in your SM logs …</div>
<div><br>
</div>
<div>HTH</div><div><div>
<div><br>
</div>
<div><br>
</div>
<div><br>
</div>
<div>
<div>On Oct 28, 2013, at 1:08 PM, Robert LeBlanc <<a href="mailto:robert_leblanc@byu.edu" target="_blank">robert_leblanc@byu.edu</a>> wrote:</div>
<br>
<blockquote type="cite">
<div dir="ltr">I can ibping between both hosts just fine.
<div><br>
</div>
<div>
<div><font face="courier new, monospace">[root@desxi003 ~]# ibping 0x37</font></div>
<div><font face="courier new, monospace">Pong from desxi004.(none) (Lid 55): time 0.111 ms</font></div>
<div><font face="courier new, monospace">Pong from desxi004.(none) (Lid 55): time 0.189 ms</font></div>
<div><font face="courier new, monospace">Pong from desxi004.(none) (Lid 55): time 0.189 ms</font></div>
<div><font face="courier new, monospace">Pong from desxi004.(none) (Lid 55): time 0.179 ms</font></div>
<div><font face="courier new, monospace">^C</font></div>
<div><font face="courier new, monospace">--- desxi004.(none) (Lid 55) ibping statistics ---</font></div>
<div><font face="courier new, monospace">4 packets transmitted, 4 received, 0% packet loss, time 3086 ms</font></div>
<div><font face="courier new, monospace">rtt min/avg/max = 0.111/0.167/0.189 ms</font></div>
</div>
<div><font face="courier new, monospace"><br>
</font></div>
<div>
<div><font face="courier new, monospace">[root@desxi004 ~]# ibping 0x2d</font></div>
<div><font face="courier new, monospace">Pong from desxi003.(none) (Lid 45): time 0.156 ms</font></div>
<div><font face="courier new, monospace">Pong from desxi003.(none) (Lid 45): time 0.175 ms</font></div>
<div><font face="courier new, monospace">Pong from desxi003.(none) (Lid 45): time 0.176 ms</font></div>
<div><font face="courier new, monospace">^C</font></div>
<div><font face="courier new, monospace">--- desxi003.(none) (Lid 45) ibping statistics ---</font></div>
<div><font face="courier new, monospace">3 packets transmitted, 3 received, 0% packet loss, time 2302 ms</font></div>
<div><font face="courier new, monospace">rtt min/avg/max = 0.156/0.169/0.176 ms</font></div>
</div>
<div><br>
</div>
<div>When I do an Ethernet ping to the IPoIB address, tcpdump only shows the outgoing ARP request.</div>
<div><br>
</div>
<div>
<div><font face="courier new, monospace">[root@desxi003 ~]# tcpdump -i ib0</font></div>
<div><font face="courier new, monospace">tcpdump: verbose output suppressed, use -v or -vv for full protocol decode</font></div>
<div><font face="courier new, monospace">listening on ib0, link-type LINUX_SLL (Linux cooked), capture size 65535 bytes</font></div>
<div><font face="courier new, monospace">19:00:08.950320 ARP, Request who-has 192.168.9.4 tell 192.168.9.3, length 56</font></div>
<div><font face="courier new, monospace">19:00:09.950320 ARP, Request who-has 192.168.9.4 tell 192.168.9.3, length 56</font></div>
<div><font face="courier new, monospace">19:00:10.950307 ARP, Request who-has 192.168.9.4 tell 192.168.9.3, length 56</font></div>
</div>
<div><br>
</div>
<div>Running tcpdump on the rack servers I don't see the ARP request there which I should.</div>
<div><br>
</div>
<div>From what I've read, ib0 should be mapped to the first port and ib1 should be mapped to the second port. We have one IB card with two ports. The modprobe is the default installed with the Mellanox drivers.</div>
<div><br>
</div>
<div>
<div><font face="courier new, monospace">[root@desxi003 etc]# cat modprobe.d/ib_ipoib.conf</font></div>
<div><font face="courier new, monospace"># install ib_ipoib modprobe --ignore-install ib_ipoib && /sbin/ib_ipoib_sysctl load</font></div>
<div><font face="courier new, monospace"># remove ib_ipoib /sbin/ib_ipoib_sysctl unload ; modprobe -r --ignore-remove ib_ipoib</font></div>
<div><font face="courier new, monospace">alias ib0 ib_ipoib</font></div>
<div><font face="courier new, monospace">alias ib1 ib_ipoib</font></div>
</div>
<div><br>
</div>
<div>Can you give me some pointers on digging into the device layer to make sure IPoIB is connected correctly? Would I look in /sys or /proc for that?</div>
<div><br>
</div>
<div>Dell has not been able to replicate the problem in their environment and they only support Red Hat and won't work with my CentOS live CD. These blades don't have internal hard drives so it makes it hard to install any OS. I don't know if I can engage Mellanox
since they build the switch hardware and driver stack we are using.</div>
<div><br>
</div>
<div>I really appreciate all the help you guys have given thus far, I'm learning a lot as this progresses. I'm reading through <a href="https://tools.ietf.org/html/rfc4391" target="_blank">https://tools.ietf.org/html/rfc4391</a> trying to understand IPoIB from top to bottom.</div>
<div><br>
</div>
<div>Thanks,</div>
</div>
<div class="gmail_extra"><br clear="all">
<div>
<div><span style="font-family:arial,sans-serif;font-size:13px"><br>
</span></div>
<span style="font-family:arial,sans-serif;font-size:13px">Robert LeBlanc</span><br style="font-family:arial,sans-serif;font-size:13px">
<span style="font-family:arial,sans-serif;font-size:13px">OIT Infrastructure & Virtualization Engineer</span><br style="font-family:arial,sans-serif;font-size:13px">
<span style="font-family:arial,sans-serif;font-size:13px">Brigham Young University</span></div>
<br>
<br>
<div class="gmail_quote">On Mon, Oct 28, 2013 at 12:53 PM, Coulter, Susan K <span dir="ltr">
<<a href="mailto:skc@lanl.gov" target="_blank">skc@lanl.gov</a>></span> wrote:<br>
<blockquote style="margin:0px 0px 0px 0.8ex;padding-left:1ex;border-left-color:rgb(204,204,204);border-left-width:1px;border-left-style:solid" class="gmail_quote">
<div style="word-wrap:break-word">
<div><br>
</div>
If you are not seeing any packets leave the ib0 interface, it sounds like the emulation layer is not connected to the right device.
<div><br>
</div>
<div>If ib_ipoib kernel module is loaded, and a simple native IB test works between those blades - (like ib_read_bw) you need to dig into the device layer and insure ipoib is "connected" to the right device. </div>
<div><br>
</div>
<div>Do you have more than 1 IB card? </div>
<div>What does your modprobe config look like for ipoib?</div>
<div><br>
</div>
<div><br>
</div>
<div>
<div>
<div>On Oct 28, 2013, at 12:38 PM, Robert LeBlanc <<a href="mailto:robert_leblanc@byu.edu" target="_blank">robert_leblanc@byu.edu</a>></div>
<div>
<div>
<div> wrote:</div>
<br>
<blockquote type="cite">
<div dir="ltr">These ESX hosts (2 blade server and 2 rack servers) are booted into a CentOS 6.2 Live CD that I built. Right now everything I'm trying to get working is CentOS 6.2. All of our other hosts are running ESXi and have IPoIB interfaces, but none of
them are configured and I'm not trying to get those working right now.
<div><br>
</div>
<div>Ideally, we would like our ESX hosts to communicate with each other for vMotion and protected VM traffic as well as with our Commvault backup servers (Windows) over IPoIB (or Oracle's PVI which is very similar).</div>
</div>
<div class="gmail_extra"><br clear="all">
<div>
<div><span style="font-family:arial,sans-serif;font-size:13px"><br>
</span></div>
<span style="font-family:arial,sans-serif;font-size:13px">Robert LeBlanc</span><br style="font-family:arial,sans-serif;font-size:13px">
<span style="font-family:arial,sans-serif;font-size:13px">OIT Infrastructure & Virtualization Engineer</span><br style="font-family:arial,sans-serif;font-size:13px">
<span style="font-family:arial,sans-serif;font-size:13px">Brigham Young University</span></div>
<br>
<br>
<div class="gmail_quote">On Mon, Oct 28, 2013 at 12:33 PM, Hal Rosenstock <span dir="ltr">
<<a href="mailto:hal.rosenstock@gmail.com" target="_blank">hal.rosenstock@gmail.com</a>></span> wrote:<br>
<blockquote style="margin:0px 0px 0px 0.8ex;padding-left:1ex;border-left-color:rgb(204,204,204);border-left-width:1px;border-left-style:solid" class="gmail_quote">
<div dir="ltr">Are those ESXi IPoIB interfaces ? Do some of these work and others not ? Are there normal Linux IPoIB interfaces ? Do they work ?</div>
<div>
<div>
<div class="gmail_extra"><br>
<br>
<div class="gmail_quote">On Mon, Oct 28, 2013 at 2:24 PM, Robert LeBlanc <span dir="ltr">
<<a href="mailto:robert_leblanc@byu.edu" target="_blank">robert_leblanc@byu.edu</a>></span> wrote:<br>
<blockquote style="margin:0px 0px 0px 0.8ex;padding-left:1ex;border-left-color:rgb(204,204,204);border-left-width:1px;border-left-style:solid" class="gmail_quote">
<div dir="ltr">Yes, I can not ping them over the IPoIB interface. It is a very simple network set-up.
<div><br>
</div>
<div>desxi003</div>
<div>
<div><font face="courier new, monospace">8: ib0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 65520 qdisc pfifo_fast state UP qlen 256</font></div>
<div><font face="courier new, monospace"> link/infiniband 80:20:00:54:fe:80:00:00:00:00:00:00:f0:4d:a2:90:97:78:e7:d1 brd 00:ff:ff:ff:ff:12:40:1b:ff:ff:00:00:00:00:00:00:ff:ff:ff:ff</font></div>
<div><font face="courier new, monospace"> inet <a href="http://192.168.9.3/24" target="_blank">
192.168.9.3/24</a> brd 192.168.9.255 scope global ib0</font></div>
<div><font face="courier new, monospace"> inet6 fe80::f24d:a290:9778:e7d1/64 scope link</font></div>
<div><font face="courier new, monospace"> valid_lft forever preferred_lft forever</font></div>
</div>
<div><br>
</div>
<div>desxi004</div>
<div>
<div><font face="courier new, monospace">8: ib0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 65520 qdisc pfifo_fast state UP qlen 256</font></div>
<div><font face="courier new, monospace"> link/infiniband 80:20:00:54:fe:80:00:00:00:00:00:00:f0:4d:a2:90:97:78:e7:15 brd 00:ff:ff:ff:ff:12:40:1b:ff:ff:00:00:00:00:00:00:ff:ff:ff:ff</font></div>
<div><font face="courier new, monospace"> inet <a href="http://192.168.9.4/24" target="_blank">
192.168.9.4/24</a> brd 192.168.9.255 scope global ib0</font></div>
<div><font face="courier new, monospace"> inet6 fe80::f24d:a290:9778:e715/64 scope link</font></div>
<div><font face="courier new, monospace"> valid_lft forever preferred_lft forever</font></div>
</div>
<div><br>
</div>
</div>
<div class="gmail_extra">
<div><br clear="all">
<div>
<div><span style="font-family:arial,sans-serif;font-size:13px"><br>
</span></div>
<span style="font-family:arial,sans-serif;font-size:13px">Robert LeBlanc</span><br style="font-family:arial,sans-serif;font-size:13px">
<span style="font-family:arial,sans-serif;font-size:13px">OIT Infrastructure & Virtualization Engineer</span><br style="font-family:arial,sans-serif;font-size:13px">
<span style="font-family:arial,sans-serif;font-size:13px">Brigham Young University</span></div>
<br>
<br>
</div>
<div>
<div class="gmail_quote">On Mon, Oct 28, 2013 at 12:22 PM, Hal Rosenstock <span dir="ltr">
<<a href="mailto:hal.rosenstock@gmail.com" target="_blank">hal.rosenstock@gmail.com</a>></span> wrote:<br>
<blockquote style="margin:0px 0px 0px 0.8ex;padding-left:1ex;border-left-color:rgb(204,204,204);border-left-width:1px;border-left-style:solid" class="gmail_quote">
<div dir="ltr">So these 2 hosts have trouble talking IPoIB to each other ?</div>
<div>
<div class="gmail_extra"><br>
<br>
<div class="gmail_quote">On Mon, Oct 28, 2013 at 2:16 PM, Robert LeBlanc <span dir="ltr">
<<a href="mailto:robert_leblanc@byu.edu" target="_blank">robert_leblanc@byu.edu</a>></span> wrote:<br>
<blockquote style="margin:0px 0px 0px 0.8ex;padding-left:1ex;border-left-color:rgb(204,204,204);border-left-width:1px;border-left-style:solid" class="gmail_quote">
<div dir="ltr">I was just wondering about that. It seems reasonable that the broadcast traffic would go over multicast, but effectively channels would be created for node to node communication, otherwise the entire multicast group would be limited to 10 Gbps
(in this instance) for the whole group. That doesn't scale very well.
<div><br>
</div>
<div>The things I've read about IPoIB performance tuning seems pretty vague, and the changes most people recommend seem to be already in place on the systems I'm using. Some people said, try using a newer version of Ubuntu, but ultimately, I have very little
control over VMware. Once I can get the Linux machines to communicate IPoIB between the racks and blades, then I'm going to turn my attention over to performance optimization. It doesn't seem to make much sense to spend time there when it is not working at
all for most machines.</div>
<div><br>
</div>
<div>I've done ibtracert between the two nodes, is that what you mean by walking the route?</div>
<div><br>
</div>
<div>
<div><font face="courier new, monospace">[root@desxi003 ~]# ibtracert -m 0xc000 0x2d 0x37</font></div>
<div><font face="courier new, monospace">From ca 0xf04da2909778e7d0 port 1 lid 45-45 "localhost HCA-1"</font></div>
<div><font face="courier new, monospace">[1] -> switch 0x2c90200448ec8[17] lid 51 "Infiniscale-IV Mellanox Technologies"</font></div>
<div><font face="courier new, monospace">[18] -> ca 0xf04da2909778e714[1] lid 55 "localhost HCA-1"</font></div>
<div><font face="courier new, monospace">To ca 0xf04da2909778e714 port 1 lid 55-55 "localhost HCA-1"</font></div>
</div>
<div><font face="courier new, monospace"><br>
</font></div>
<div>
<div><font face="courier new, monospace">[root@desxi004 ~]# ibtracert -m 0xc000 0x37 0x2d</font></div>
<div><font face="courier new, monospace">From ca 0xf04da2909778e714 port 1 lid 55-55 "localhost HCA-1"</font></div>
<div><font face="courier new, monospace">[1] -> switch 0x2c90200448ec8[18] lid 51 "Infiniscale-IV Mellanox Technologies"</font></div>
<div><font face="courier new, monospace">[17] -> ca 0xf04da2909778e7d0[1] lid 45 "localhost HCA-1"</font></div>
<div><font face="courier new, monospace">To ca 0xf04da2909778e7d0 port 1 lid 45-45 "localhost HCA-1"</font></div>
</div>
<div><br>
</div>
<div>As you can see, the route is on the same switch, the blades are right next to each other.</div>
</div>
<div class="gmail_extra">
<div><br clear="all">
<div>
<div><span style="font-family:arial,sans-serif;font-size:13px"><br>
</span></div>
<span style="font-family:arial,sans-serif;font-size:13px">Robert LeBlanc</span><br style="font-family:arial,sans-serif;font-size:13px">
<span style="font-family:arial,sans-serif;font-size:13px">OIT Infrastructure & Virtualization Engineer</span><br style="font-family:arial,sans-serif;font-size:13px">
<span style="font-family:arial,sans-serif;font-size:13px">Brigham Young University</span></div>
<br>
<br>
</div>
<div>
<div class="gmail_quote">On Mon, Oct 28, 2013 at 12:05 PM, Hal Rosenstock <span dir="ltr">
<<a href="mailto:hal.rosenstock@gmail.com" target="_blank">hal.rosenstock@gmail.com</a>></span> wrote:<br>
<blockquote style="margin:0px 0px 0px 0.8ex;padding-left:1ex;border-left-color:rgb(204,204,204);border-left-width:1px;border-left-style:solid" class="gmail_quote">
<div dir="ltr">
<div>Which mystery is explained ? The 10 Gbps is a multicast only limit and does not apply to unicast. The BW limitation you're seeing is due to other factors. There's been much written about IPoIB performance.</div>
<div> </div>
<div>If all the MC members are joined and routed, then the IPoIB connectivity issue is some other issue. Are you sure this is the case ? Did you walk the route between 2 nodes where you have a connectivity issue ?</div>
</div>
<div>
<div class="gmail_extra"><br>
<br>
<div class="gmail_quote">On Mon, Oct 28, 2013 at 1:58 PM, Robert LeBlanc <span dir="ltr">
<<a href="mailto:robert_leblanc@byu.edu" target="_blank">robert_leblanc@byu.edu</a>></span> wrote:<br>
<blockquote style="margin:0px 0px 0px 0.8ex;padding-left:1ex;border-left-color:rgb(204,204,204);border-left-width:1px;border-left-style:solid" class="gmail_quote">
<div dir="ltr">Well, that explains one mystery, now I need to figure out why it seems the Dell blades are not passing the traffic.</div>
<div class="gmail_extra">
<div><br clear="all">
<div>
<div><span style="font-family:arial,sans-serif;font-size:13px"><br>
</span></div>
<span style="font-family:arial,sans-serif;font-size:13px">Robert LeBlanc</span><br style="font-family:arial,sans-serif;font-size:13px">
<span style="font-family:arial,sans-serif;font-size:13px">OIT Infrastructure & Virtualization Engineer</span><br style="font-family:arial,sans-serif;font-size:13px">
<span style="font-family:arial,sans-serif;font-size:13px">Brigham Young University</span></div>
<br>
<br>
</div>
<div>
<div class="gmail_quote">On Mon, Oct 28, 2013 at 11:51 AM, Hal Rosenstock <span dir="ltr">
<<a href="mailto:hal.rosenstock@gmail.com" target="_blank">hal.rosenstock@gmail.com</a>></span> wrote:<br>
<blockquote style="margin:0px 0px 0px 0.8ex;padding-left:1ex;border-left-color:rgb(204,204,204);border-left-width:1px;border-left-style:solid" class="gmail_quote">
<div dir="ltr">
<div>Yes, that's the IPoIB IPv4 broadcast group for the default (0xffff) partition. 0x80 part of mtu and rate just means "is equal to". mtu 0x04 is 2K (2048) and rate 0x3 is 10 Gb/sec. These are indeed the defaults.</div>
</div>
<div>
<div class="gmail_extra"><br>
<br>
<div class="gmail_quote">On Mon, Oct 28, 2013 at 1:45 PM, Robert LeBlanc <span dir="ltr">
<<a href="mailto:robert_leblanc@byu.edu" target="_blank">robert_leblanc@byu.edu</a>></span> wrote:<br>
<blockquote style="margin:0px 0px 0px 0.8ex;padding-left:1ex;border-left-color:rgb(204,204,204);border-left-width:1px;border-left-style:solid" class="gmail_quote">
<div dir="ltr">The info for that MGID is:
<div>
<div><font face="courier new, monospace">MCMemberRecord group dump:</font></div>
<div><font face="courier new, monospace"> MGID....................ff12:401b:ffff::ffff:ffff</font></div>
<div><font face="courier new, monospace"> Mlid....................0xC000</font></div>
<div><font face="courier new, monospace"> Mtu.....................0x84</font></div>
<div><font face="courier new, monospace"> pkey....................0xFFFF</font></div>
<div><font face="courier new, monospace"> Rate....................0x83</font></div>
<div><font face="courier new, monospace"> SL......................0x0</font></div>
</div>
<div><br>
</div>
<div>I don't understand the MTU and Rate (130 and 131 dec). When I run iperf between the two hosts over IPoIB in connected mode and MTU 65520. I've tried multiple threads, but the sum is still 10 Gbps.</div>
</div>
<div class="gmail_extra">
<div><br clear="all">
<div>
<div><span style="font-family:arial,sans-serif;font-size:13px"><br>
</span></div>
<span style="font-family:arial,sans-serif;font-size:13px">Robert LeBlanc</span><br style="font-family:arial,sans-serif;font-size:13px">
<span style="font-family:arial,sans-serif;font-size:13px">OIT Infrastructure & Virtualization Engineer</span><br style="font-family:arial,sans-serif;font-size:13px">
<span style="font-family:arial,sans-serif;font-size:13px">Brigham Young University</span></div>
<br>
<br>
</div>
<div>
<div class="gmail_quote">On Mon, Oct 28, 2013 at 11:40 AM, Hal Rosenstock <span dir="ltr">
<<a href="mailto:hal.rosenstock@gmail.com" target="_blank">hal.rosenstock@gmail.com</a>></span> wrote:<br>
<blockquote style="margin:0px 0px 0px 0.8ex;padding-left:1ex;border-left-color:rgb(204,204,204);border-left-width:1px;border-left-style:solid" class="gmail_quote">
<div dir="ltr">
<div>
<div>saquery -g should show what MGID is mapped to MLID 0xc000 and the group parameters.</div>
<div> </div>
</div>
<div>When you say 10 Gbps max, is that multicast or unicast ? That limit is only on the multicast.</div>
</div>
<div>
<div class="gmail_extra"><br>
<br>
<div class="gmail_quote">On Mon, Oct 28, 2013 at 1:28 PM, Robert LeBlanc <span dir="ltr">
<<a href="mailto:robert_leblanc@byu.edu" target="_blank">robert_leblanc@byu.edu</a>></span> wrote:<br>
<blockquote style="margin:0px 0px 0px 0.8ex;padding-left:1ex;border-left-color:rgb(204,204,204);border-left-width:1px;border-left-style:solid" class="gmail_quote">
<div dir="ltr">Well, that can explain why I'm only able to get 10 Gbps max from the two hosts that are working.
<div><br>
</div>
<div>I have tried updn and dnup and they didn't help either. I think the only thing that will help is Automatic Path Migration is it tries very hard to route the alternative LIDs through different systemguids. I suspect it would require re-LIDing everything
which would mean an outage. I'm still trying to get answers from Oracle if that is even a possibility. I've tried seeding some of the algorithms with information like root nodes, etc, but none of them worked better.</div>
<div><br>
</div>
<div>The MLID 0xc000 exists and I can see all the nodes joined to the group using saquery. I've checked the route using ibtracert specifying the MLID. The only thing I'm not sure how to check is the group parameters. What tool would I use for that?</div>
<div><br>
</div>
<div>Thanks,</div>
</div>
<div class="gmail_extra">
<div><br clear="all">
<div>
<div><span style="font-family:arial,sans-serif;font-size:13px"><br>
</span></div>
<span style="font-family:arial,sans-serif;font-size:13px">Robert LeBlanc</span><br style="font-family:arial,sans-serif;font-size:13px">
<span style="font-family:arial,sans-serif;font-size:13px">OIT Infrastructure & Virtualization Engineer</span><br style="font-family:arial,sans-serif;font-size:13px">
<span style="font-family:arial,sans-serif;font-size:13px">Brigham Young University</span></div>
<br>
<br>
</div>
<div>
<div class="gmail_quote">On Mon, Oct 28, 2013 at 11:16 AM, Hal Rosenstock <span dir="ltr">
<<a href="mailto:hal.rosenstock@gmail.com" target="_blank">hal.rosenstock@gmail.com</a>></span> wrote:<br>
<blockquote style="margin:0px 0px 0px 0.8ex;padding-left:1ex;border-left-color:rgb(204,204,204);border-left-width:1px;border-left-style:solid" class="gmail_quote">
<div dir="ltr">
<div>Xsigo's SM is not "straight" OpenSM. They have some proprietary enhancements and it may be based on old vintage of OpenSM. You will likely need to work with them/Oracle now on issues.</div>
<div> </div>
<div>Lack of a partitions file does mean default partition and default rate (10 Gbps) so from what I saw all ports had sufficient rate to join MC group.</div>
<div> </div>
<div>There are certain topology requirements for running various routing algorithms. Did you try updn or dnup ?</div>
<div> </div>
<div>The key is determining whether the IPoIB broadcast group is setup correctly. What MLID is the group built on (usually 0xc000) ? What are the group parameters (rate, MTU) ? Are all members that are running IPoIB joined ? Is the group routed to all such
members ? There are infiniband-diags for all of this.</div>
</div>
<div>
<div class="gmail_extra"><br>
<br>
<div class="gmail_quote">On Mon, Oct 28, 2013 at 12:19 PM, Robert LeBlanc <span dir="ltr">
<<a href="mailto:robert_leblanc@byu.edu" target="_blank">robert_leblanc@byu.edu</a>></span> wrote:<br>
<blockquote style="margin:0px 0px 0px 0.8ex;padding-left:1ex;border-left-color:rgb(204,204,204);border-left-width:1px;border-left-style:solid" class="gmail_quote">
<div dir="ltr">OpenSM (the SM runs on Xsigo so they manage it) is using minhop. I've loaded the ibnetdiscover output into ibsim and run all the different routing algorithms against it with and without scatter ports. Minhop had 50% of our hosts running all paths
through a single IS5030 switch (at least the LIDs we need which represent Ethernet and Fibre Channel cards the hosts should communicate with). Ftree, dor, and dfsssp failed back to minhop, the others routed more paths through the same IS5030 in some cases
increasing our host count with single point of failure to 75%.
<div><br>
</div>
<div>As far as I can tell there is no partitions.conf file so I assume we are using the default partition. There is an opensm.opts file, but it only specifies logging information.</div>
<div>
<div><font face="courier new, monospace"># SA database file name</font></div>
<div><font face="courier new, monospace">sa_db_file /var/log/opensm-sa.dump</font></div>
<div><font face="courier new, monospace"><br>
</font></div>
<div><font face="courier new, monospace"># If TRUE causes OpenSM to dump SA database at the end of</font></div>
<div><font face="courier new, monospace"># every light sweep, regardless of the verbosity level</font></div>
<div><font face="courier new, monospace">sa_db_dump TRUE</font></div>
<div><font face="courier new, monospace"><br>
</font></div>
<div><font face="courier new, monospace"># The directory to hold the file OpenSM dumps</font></div>
<div><font face="courier new, monospace">dump_files_dir /var/log/</font></div>
</div>
<div><br>
</div>
<div>The SM node is:</div>
<div>
<div><font face="courier new, monospace">xsigoa:/opt/xsigo/xsigos/current/ofed/etc# ibaddr</font></div>
<div><font face="courier new, monospace">GID fe80::13:9702:100:979 LID start 0x1 end 0x1</font></div>
</div>
<div><br>
</div>
<div>We do have Switch-X in two of the Dell m1000e chassis but the cards, ports 17-32, are FDR10 (the switch may be straight FDR, but I'm not 100% sure). The IS5030 are QDR which the Switch-X are connected to, the switches in the Xsigo directors are QDR, but
the Ethernet and Fibre Channel cards are DDR. The DDR cards will not be running IPoIB (at least to my knowledge they don't have the ability), only the hosts should be leveraging IPoIB. I hope that clears up some of your questions. If you have more, I will
try to answer them.<br>
<div><br>
</div>
<div><br>
</div>
</div>
</div>
<div class="gmail_extra">
<div><br clear="all">
<div>
<div><span style="font-family:arial,sans-serif;font-size:13px"><br>
</span></div>
<span style="font-family:arial,sans-serif;font-size:13px">Robert LeBlanc</span><br style="font-family:arial,sans-serif;font-size:13px">
<span style="font-family:arial,sans-serif;font-size:13px">OIT Infrastructure & Virtualization Engineer</span><br style="font-family:arial,sans-serif;font-size:13px">
<span style="font-family:arial,sans-serif;font-size:13px">Brigham Young University</span></div>
<br>
<br>
</div>
<div>
<div class="gmail_quote">On Mon, Oct 28, 2013 at 9:57 AM, Hal Rosenstock <span dir="ltr">
<<a href="mailto:hal.rosenstock@gmail.com" target="_blank">hal.rosenstock@gmail.com</a>></span> wrote:<br>
<blockquote style="margin:0px 0px 0px 0.8ex;padding-left:1ex;border-left-color:rgb(204,204,204);border-left-width:1px;border-left-style:solid" class="gmail_quote">
<div dir="ltr">
<div>What routing algorithm is configured in OpenSM ? What does your partitions.conf file look like ? Which node is your OpenSM ?</div>
<div> </div>
<div>Also, I only see QDR and DDR links although you have Switch-X so I assume all FDR ports are connected to slower (QDR) devices. I don't see any FDR-10 ports but maybe they're also connected to QDR ports so show up as QDR in the topology.</div>
<div> </div>
<div>There are DDR CAs in Xsigo box but not sure whether or not they run IPoIB.</div>
<span><font color="#888888">
<div> </div>
<div>-- Hal</div>
</font></span></div>
<div class="gmail_extra"><br>
<br>
<div class="gmail_quote">
<div>On Sun, Oct 27, 2013 at 9:46 PM, Robert LeBlanc <span dir="ltr"><<a href="mailto:robert_leblanc@byu.edu" target="_blank">robert_leblanc@byu.edu</a>></span> wrote:<br>
</div>
<blockquote style="margin:0px 0px 0px 0.8ex;padding-left:1ex;border-left-color:rgb(204,204,204);border-left-width:1px;border-left-style:solid" class="gmail_quote">
<div>
<div dir="ltr">Since you guys are amazingly helpful, I thought I would pick your brains in a new problem.
<div><br>
</div>
<div>We have two Xsigo directors cross connected to four Mellanox IS5030 switches. Connected to those we have four Dell m1000e chassis each with two IB switches (two chassis have QDR and two have FDR10). We have 9 dual-port rack servers connected to the IS5030
switches. For testing purposes we have an additional Dell m1000e QDR chassis connected to one Xsigo director and two dual-port FDR10 rack servers connected to the other Xsigo director.</div>
<div><br>
</div>
<div>I can get IPoIB to work between the two test rack servers connected to the one Xsigo director. But I can not get IPoIB to work between any blades either right next to each other or to the working rack servers. I'm using the same exact live CentOS ISO on
all four servers. I've checked opensm and the blades have joined the multicast group 0xc000 properly. tcpdump basically says that traffic is not leaving the blades. tcpdump also shows no traffic entering the blades from the rack servers. An ibtracert using
0xc000 mlid shows that routing exists between hosts.</div>
<div><br>
</div>
<div>I've read about MulticastFDBTop=0xBFFF but I don't know how to set it and I doubt it would have been set by default.</div>
<div><br>
</div>
<div>Anyone have some ideas on troubleshooting steps to try? I think Google is tired of me asking questions about it.</div>
<div><br>
</div>
<div>Thanks,<br clear="all">
<div>
<div><span style="font-family:arial,sans-serif;font-size:13px"><br>
</span></div>
<span style="font-family:arial,sans-serif;font-size:13px">Robert LeBlanc</span><br style="font-family:arial,sans-serif;font-size:13px">
<span style="font-family:arial,sans-serif;font-size:13px">OIT Infrastructure & Virtualization Engineer</span><br style="font-family:arial,sans-serif;font-size:13px">
<span style="font-family:arial,sans-serif;font-size:13px">Brigham Young University</span></div>
</div>
</div>
<br>
</div>
<div>_______________________________________________<br>
Users mailing list<br>
<a href="mailto:Users@lists.openfabrics.org" target="_blank">Users@lists.openfabrics.org</a><br>
<a href="http://lists.openfabrics.org/cgi-bin/mailman/listinfo/users" target="_blank">http://lists.openfabrics.org/cgi-bin/mailman/listinfo/users</a><br>
<br>
</div>
</blockquote>
</div>
<br>
</div>
</blockquote>
</div>
<br>
</div>
</div>
</blockquote>
</div>
<br>
</div>
</div>
</blockquote>
</div>
<br>
</div>
</div>
</blockquote>
</div>
<br>
</div>
</div>
</blockquote>
</div>
<br>
</div>
</div>
</blockquote>
</div>
<br>
</div>
</div>
</blockquote>
</div>
<br>
</div>
</div>
</blockquote>
</div>
<br>
</div>
</div>
</blockquote>
</div>
<br>
</div>
</div>
</blockquote>
</div>
<br>
</div>
</div>
</blockquote>
</div>
<br>
</div>
</div>
</blockquote>
</div>
<br>
</div>
</div>
</div>
</blockquote>
</div>
<br>
</div>
_______________________________________________<br>
Users mailing list<br>
<a href="mailto:Users@lists.openfabrics.org" target="_blank">Users@lists.openfabrics.org</a><br>
<a href="http://lists.openfabrics.org/cgi-bin/mailman/listinfo/users" target="_blank">http://lists.openfabrics.org/cgi-bin/mailman/listinfo/users</a><br>
</blockquote>
</div>
</div>
</div>
<br>
<div>
<div>
<div>====================================</div>
<div><br>
</div>
<div>Susan Coulter<br>
HPC-3 Network/Infrastructure<br>
<a href="tel:505-667-8425" value="+15056678425" target="_blank">505-667-8425</a><br>
Increase the Peace...<br>
An eye for an eye leaves the whole world blind<br>
====================================</div>
</div>
</div>
<br>
</div>
</div>
</blockquote>
</div>
<br>
</div>
</blockquote>
</div>
<br>
<div>
<div>
<div>====================================</div>
<div><br>
</div>
<div>Susan Coulter<br>
HPC-3 Network/Infrastructure<br>
<a href="tel:505-667-8425" value="+15056678425" target="_blank">505-667-8425</a><br>
Increase the Peace...<br>
An eye for an eye leaves the whole world blind<br>
====================================</div>
</div>
</div>
<br>
</div></div></div>
</blockquote></div><br></div></div></div>
</blockquote></div><br></div>
</div></div></blockquote></div><br></div></div></div>
</blockquote></div><br></div>
</div></div></blockquote></div><br></div></div></div>
</blockquote></div><br></div>
</div></div></blockquote></div><br></div></div></div>
</blockquote></div><br></div>
</div></div></blockquote></div><br></div></div></div>
</blockquote></div><br></div>
</blockquote></div>
</div></div></blockquote></div><br></div>
</div></div></blockquote></div><br></div></div></div>
</blockquote></div><br></div>
</div></div></blockquote></div><br></div></div></div>
</blockquote></div><br></div>