<br><br>
<div class="gmail_quote">On Tue, Aug 4, 2009 at 11:27 AM, Sasha Khapyorsky <span dir="ltr"><<a href="mailto:sashak@voltaire.com" target="_blank">sashak@voltaire.com</a>></span> wrote:<br>
<blockquote class="gmail_quote" style="PADDING-LEFT: 1ex; MARGIN: 0px 0px 0px 0.8ex; BORDER-LEFT: #ccc 1px solid">Hi,<br>
<div><br>On 19:28 Thu 30 Jul , Hal Rosenstock wrote:<br>><br>> Currently, MADs are pipelined to a single switch at a time which<br>> effectively serializes these requests due to processing at the SMA.<br>> This patch pipelines (stripes) them across the switches first before<br>
> proceeding with successive blocks. As a result of this striping,<br>> multiple switches can process the set and respond concurrently<br>> which results in an improvement to the subnet initialization time.<br><br>
</div>The idea is nice. However I have some initial comments about an<br>implementation.<br><br>BTW should there be a reason for an option to preserve the current<br>behavior? (I don't know, just asking)</blockquote>
<div> </div>
<div>I asked this in an email on the thread on this. It's up to you. I don't see a need but if we want to be conservative, it can be added.</div>
<div> </div>
<blockquote class="gmail_quote" style="PADDING-LEFT: 1ex; MARGIN: 0px 0px 0px 0.8ex; BORDER-LEFT: #ccc 1px solid"><span></span><br>
<div><br>> This patch also introduces a new config option (max_smps_per_node)<br>> which indicates how deep the per node pipeline is (current default is 4).<br>> This also has the effect of limiting the number of times that the switch<br>
> list is traversed. Maybe this embellishment is unnecessary.<br><br></div>Then why is it needed?</blockquote>
<div> </div>
<div>Also, as was discussed in the thread on this, it gives a way to control possible VL15 overflow.</div>
<div> </div>
<blockquote class="gmail_quote" style="PADDING-LEFT: 1ex; MARGIN: 0px 0px 0px 0.8ex; BORDER-LEFT: #ccc 1px solid"><span></span><br>
<div><br>> All unicast routing protocols are updated for this with the exception<br>> of file.<br>><br>> A similar subsequent change will do this for MFTs.<br>><br>> Yevgeny Kliteynik <<a href="mailto:kliteyn@dev.mellanox.co.il" target="_blank">kliteyn@dev.mellanox.co.il</a>> wrote:<br>
><br>> With a small cluster of 17 IS4 switches and 11 HCAs and<br>> to artificially increase the cluster, LMC of 7 was used<br>> including EnhancedSwitchPort 0 LMC.<br>><br>> With the new code, LFT configuration is more than twice as<br>
> fast as with the old code :)<br>> Current ucast manager ran on avarage for ~250msec, with the<br>> new code - 110-120msec.<br>><br>> Routing calculation phase of the ucast manager took ~1200 usec,<br>> the rest was sending the blocks and waiting for no more pending<br>
> transactions.<br>><br>> No noticeable difference between various max_smps_per_node values<br>> was observed.<br><br></div>What is the reason?</blockquote>
<div> </div>
<div>I think the reason was max_wire_smps may have kicked in but Yevgeny is best to elaborate on this.</div>
<div> </div>
<blockquote class="gmail_quote" style="PADDING-LEFT: 1ex; MARGIN: 0px 0px 0px 0.8ex; BORDER-LEFT: #ccc 1px solid"><span></span>And what was value of 'max_wire_smps'?<br>
<div>
<div></div>
<div></div></div></blockquote>
<div><br>> Here are some detailed results of different executions (the<br>> number on the left is timer value in usec):<br>><br>> Current ucast manager (w/o the optimization):<br>><br>> 000000 [LFT]: osm_ucast_mgr_process() - START<br>
> 001131 [LFT]: ucast_mgr_process_tbl() - START<br>> 032251 [LFT]: ucast_mgr_process_tbl() - END<br>> 032263 [LFT]: osm_ucast_mgr_process() - END<br>> 253416 [LFT]: Done wait_for_pending_transactions()<br>><br>
> New code, max_smps_per_node=0:<br>><br>> 001417 [LFT]: osm_ucast_mgr_process() - START (0 max_smps_per_node)<br>> 002690 [LFT]: ucast_mgr_process_tbl() - START<br>> 032946 [LFT]: ucast_mgr_process_tbl() - END<br>
> 032948 [LFT]: osm_ucast_pipeline_tbl() - START<br>> 033846 [LFT]: osm_ucast_pipeline_tbl() - END<br>> 033858 [LFT]: osm_ucast_mgr_process() - END<br>> 108203 [LFT]: Done wait_for_pending_transactions()<br>><br>
> New code, max_smps_per_node=1:<br>><br>> 007474 [LFT]: osm_ucast_mgr_process() - START (1 max_smps_per_node)<br>> 008735 [LFT]: ucast_mgr_process_tbl() - START<br>> 040071 [LFT]: ucast_mgr_process_tbl() - END<br>
> 040074 [LFT]: osm_ucast_pipeline_tbl() - START<br>> 040103 [LFT]: osm_ucast_pipeline_tbl() - END<br>> 040114 [LFT]: osm_ucast_mgr_process() - END<br>> 120097 [LFT]: Done wait_for_pending_transactions()<br>><br>
> New code, max_smps_per_node=4:<br>><br>> 004137 [LFT]: osm_ucast_mgr_process() - START (4 max_smps_per_node)<br>> 005380 [LFT]: ucast_mgr_process_tbl() - START<br>> 037436 [LFT]: ucast_mgr_process_tbl() - END<br>
> 037439 [LFT]: osm_ucast_pipeline_tbl() - START<br>> 037495 [LFT]: osm_ucast_pipeline_tbl() - END<br>> 037506 [LFT]: osm_ucast_mgr_process() - END<br>> 114983 [LFT]: Done wait_for_pending_transactions()<br>><br>
><br>> With IS3 based Qlogic switches, which do not handle DR packets forwarding<br>> in HW, with a fabric of ~1100 HCAs, ~280 switches:<br>><br>> Current OSM configures LFTs in ~2 seconds.<br>> New algorithm does the same job in 1.4-1.6 seconds (30%-20% speed up),<br>
> depending on the max_smps_per_node value.<br>><br>> As in case of IS4 switches, the shortest config time was obtained with<br>> max_smps_per_node=0, which is unlimited pipeline.<br>><br>><br>> Signed-off-by: Hal Rosenstock <<a href="mailto:hal.rosenstock@gmail.com" target="_blank">hal.rosenstock@gmail.com</a>><br>
> ---<br>> Changes since v1:<br>> Added Yevgeny's performance data to patch description above<br>> No change to actual patch<br>><br>> diff --git a/opensm/include/opensm/osm_base.h b/opensm/include/opensm/osm_base.h<br>
> index 0537002..617e8a9 100644<br>> --- a/opensm/include/opensm/osm_base.h<br>> +++ b/opensm/include/opensm/osm_base.h<br>> @@ -1,6 +1,6 @@<br>> /*<br>> * Copyright (c) 2004-2008 Voltaire, Inc. All rights reserved.<br>
> - * Copyright (c) 2002-2006 Mellanox Technologies LTD. All rights reserved.<br>> + * Copyright (c) 2002-2009 Mellanox Technologies LTD. All rights reserved.<br>> * Copyright (c) 1996-2003 Intel Corporation. All rights reserved.<br>
> * Copyright (c) 2009 Sun Microsystems, Inc. All rights reserved.<br>> *<br>> @@ -449,6 +449,18 @@ BEGIN_C_DECLS<br>> */<br>> #define OSM_DEFAULT_SMP_MAX_ON_WIRE 4<br>> /***********/<br>> +/****d* OpenSM: Base/OSM_DEFAULT_SMP_MAX_PER_NODE<br>
> +* NAME<br>> +* OSM_DEFAULT_SMP_MAX_PER_NODE<br>> +*<br>> +* DESCRIPTION<br>> +* Specifies the default number of VL15 SMP MADs allowed<br>> +* per node for certain attributes.<br>> +*<br>> +* SYNOPSIS<br>
> +*/<br>> +#define OSM_DEFAULT_SMP_MAX_PER_NODE 4<br>> +/***********/<br>> /****d* OpenSM: Base/OSM_SM_DEFAULT_QP0_RCV_SIZE<br>> * NAME<br>> * OSM_SM_DEFAULT_QP0_RCV_SIZE<br>> diff --git a/opensm/include/opensm/osm_sm.h b/opensm/include/opensm/osm_sm.h<br>
> index cc8321d..1776380 100644<br>> --- a/opensm/include/opensm/osm_sm.h<br>> +++ b/opensm/include/opensm/osm_sm.h<br>> @@ -1,6 +1,6 @@<br>> /*<br>> * Copyright (c) 2004-2008 Voltaire, Inc. All rights reserved.<br>
> - * Copyright (c) 2002-2005 Mellanox Technologies LTD. All rights reserved.<br>> + * Copyright (c) 2002-2009 Mellanox Technologies LTD. All rights reserved.<br>> * Copyright (c) 1996-2003 Intel Corporation. All rights reserved.<br>
> *<br>> * This software is available to you under a choice of one of two<br>> @@ -130,6 +130,7 @@ typedef struct osm_sm {<br>> osm_sm_mad_ctrl_t mad_ctrl;<br>> osm_lid_mgr_t lid_mgr;<br>> osm_ucast_mgr_t ucast_mgr;<br>
> + boolean_t lfts_updated;<br><br></div>
<blockquote class="gmail_quote" style="PADDING-LEFT: 1ex; MARGIN: 0px 0px 0px 0.8ex; BORDER-LEFT: #ccc 1px solid">The name is unclear - actually this means "update in progress".</blockquote>
<div> </div>
<div>OK.</div>
<div> </div>
<blockquote class="gmail_quote" style="PADDING-LEFT: 1ex; MARGIN: 0px 0px 0px 0.8ex; BORDER-LEFT: #ccc 1px solid"><span></span><br>
<div><br>> cl_disp_reg_handle_t sweep_fail_disp_h;<br>> cl_disp_reg_handle_t ni_disp_h;<br>> cl_disp_reg_handle_t pi_disp_h;<br>> @@ -524,6 +525,45 @@ osm_resp_send(IN osm_sm_t * sm,<br>> *<br>
> *********/<br>><br>> +/****f* OpenSM: SM/osm_sm_set_next_lft_block<br>> +* NAME<br>> +* osm_sm_set_next_lft_block<br>> +*<br>> +* DESCRIPTION<br>> +* Set the next LFT (LinearForwardingTable) block in the indicated switch.<br>
> +*<br>> +* SYNOPSIS<br>> +*/<br>> +void<br>> +osm_sm_set_next_lft_block(IN osm_sm_t *p_sm, IN osm_switch_t *p_sw,<br>> + IN uint8_t *p_block, IN osm_dr_path_t *p_path,<br>> + IN osm_madw_context_t *p_context);<br>
<br></div>Why should it be in osm_sm.[ch]? osm_ucast_mgr.c or osm_switch.c seem<br>much more appropriate place for this.</blockquote>
<div> </div>
<div>OK.</div>
<div> </div>
<blockquote class="gmail_quote" style="PADDING-LEFT: 1ex; MARGIN: 0px 0px 0px 0.8ex; BORDER-LEFT: #ccc 1px solid"><span></span><br>
<div>
<div></div>
<div><br>> +/*<br>> +* PARAMETERS<br>> +* p_sm<br>> +* [in] Pointer to an osm_sm_t object.<br>> +*<br>> +* p_switch<br>> +* [in] Pointer to the switch object.<br>> +*<br>
> +* p_block<br>> +* [in] Pointer to the forwarding table block.<br>> +*<br>> +* p_path<br>> +* [in] Pointer to a directed route path object.<br>> +*<br>> +* p_context<br>
> +* [in] Mad wrapper context structure to be copied into the wrapper<br>> +* context, and thus visible to the recipient of the response.<br>> +*<br>> +* RETURN VALUES<br>> +* None<br>
> +*<br>> +* NOTES<br>> +*<br>> +* SEE ALSO<br>> +*********/<br>> +<br>> /****f* OpenSM: SM/osm_sm_mcgrp_join<br>> * NAME<br>> * osm_sm_mcgrp_join<br>> diff --git a/opensm/include/opensm/osm_subnet.h b/opensm/include/opensm/osm_subnet.h<br>
> index 59a32ad..f12afae 100644<br>> --- a/opensm/include/opensm/osm_subnet.h<br>> +++ b/opensm/include/opensm/osm_subnet.h<br>> @@ -1,6 +1,6 @@<br>> /*<br>> * Copyright (c) 2004-2008 Voltaire, Inc. All rights reserved.<br>
> - * Copyright (c) 2002-2008 Mellanox Technologies LTD. All rights reserved.<br>> + * Copyright (c) 2002-2009 Mellanox Technologies LTD. All rights reserved.<br>> * Copyright (c) 1996-2003 Intel Corporation. All rights reserved.<br>
> * Copyright (c) 2008 Xsigo Systems Inc. All rights reserved.<br>> *<br>> @@ -147,6 +147,7 @@ typedef struct osm_subn_opt {<br>> uint32_t sweep_interval;<br>> uint32_t max_wire_smps;<br>> uint32_t transaction_timeout;<br>
> + uint32_t max_smps_per_node;<br>> uint8_t sm_priority;<br>> uint8_t lmc;<br>> boolean_t lmc_esp0;<br>> diff --git a/opensm/include/opensm/osm_switch.h b/opensm/include/opensm/osm_switch.h<br>
> index 7ce28c5..e12113f 100644<br>> --- a/opensm/include/opensm/osm_switch.h<br>> +++ b/opensm/include/opensm/osm_switch.h<br>> @@ -1,6 +1,6 @@<br>> /*<br>> * Copyright (c) 2004-2008 Voltaire, Inc. All rights reserved.<br>
> - * Copyright (c) 2002-2008 Mellanox Technologies LTD. All rights reserved.<br>> + * Copyright (c) 2002-2009 Mellanox Technologies LTD. All rights reserved.<br>> * Copyright (c) 1996-2003 Intel Corporation. All rights reserved.<br>
> *<br>> * This software is available to you under a choice of one of two<br>> @@ -102,6 +102,7 @@ typedef struct osm_switch {<br>> osm_port_profile_t *p_prof;<br>> uint8_t *lft;<br>> uint8_t *new_lft;<br>
> + uint16_t lft_block_id_ho;<br>> osm_mcast_tbl_t mcast_tbl;<br>> unsigned endport_links;<br>> unsigned need_update;<br>> diff --git a/opensm/include/opensm/osm_ucast_mgr.h b/opensm/include/opensm/osm_ucast_mgr.h<br>
> index a040476..fdea49a 100644<br>> --- a/opensm/include/opensm/osm_ucast_mgr.h<br>> +++ b/opensm/include/opensm/osm_ucast_mgr.h<br>> @@ -1,6 +1,6 @@<br>> /*<br>> * Copyright (c) 2004-2008 Voltaire, Inc. All rights reserved.<br>
> - * Copyright (c) 2002-2008 Mellanox Technologies LTD. All rights reserved.<br>> + * Copyright (c) 2002-2009 Mellanox Technologies LTD. All rights reserved.<br>> * Copyright (c) 1996-2003 Intel Corporation. All rights reserved.<br>
> *<br>> * This software is available to you under a choice of one of two<br>> @@ -233,17 +233,42 @@ osm_ucast_mgr_init(IN osm_ucast_mgr_t * const p_mgr, IN struct osm_sm * sm);<br>> * osm_ucast_mgr_destroy<br>
> *********/<br>><br>> -/****f* OpenSM: Unicast Manager/osm_ucast_mgr_set_fwd_table<br>> +/****f* OpenSM: Unicast Manager/osm_ucast_pipeline_tbl<br>> * NAME<br>> -* osm_ucast_mgr_set_fwd_table<br>> +* osm_ucast_pipeline_tbl<br>
> *<br>> * DESCRIPTION<br>> -* Setup forwarding table for the switch (from prepared new_lft).<br>> +* The osm_ucast_pipeline_tbl function pipelines the LFT<br>> +* (LinearForwardingTable) sets across the switches<br>
> +* (from prepared new_lft).<br>> *<br>> * SYNOPSIS<br>> */<br>> -int osm_ucast_mgr_set_fwd_table(IN osm_ucast_mgr_t * const p_mgr,<br>> - IN osm_switch_t * const p_sw);<br>
> +void osm_ucast_pipeline_tbl(IN osm_ucast_mgr_t * p_mgr);<br>> +/*<br>> +* PARAMETERS<br>> +* p_mgr<br>> +* [in] Pointer to an osm_ucast_mgr_t object.<br>> +*<br>> +* RETURN VALUES<br>
> +* None.<br>> +*<br>> +* NOTES<br>> +*<br>> +* SEE ALSO<br>> +*********/<br>> +<br>> +/****f* OpenSM: Unicast Manager/osm_ucast_mgr_set_fwd_tbl_top<br>> +* NAME<br>> +* osm_ucast_mgr_set_fwd_tbl_top<br>
> +*<br>> +* DESCRIPTION<br>> +* Setup LinearFDBTop for the switch.<br>> +*<br>> +* SYNOPSIS<br>> +*/<br>> +int osm_ucast_mgr_set_fwd_tbl_top(IN osm_ucast_mgr_t * const p_mgr,<br>> + IN osm_switch_t * const p_sw);<br>
<br></div></div>I don't really like such separation (osm_ucast_mgr_set_fwd_tbl_top and<br>osm_ucast_pipeline_tbl).</blockquote>
<div> </div>
<div>Why not ? What's the matter with doing this ?</div>
<div> </div>
<blockquote class="gmail_quote" style="PADDING-LEFT: 1ex; MARGIN: 0px 0px 0px 0.8ex; BORDER-LEFT: #ccc 1px solid"><span></span>Why to not use a single function and update all<br>routing engines appropriately (you need to do it anyway), so that this<br>
will only fill up new_lfts table?</blockquote>
<div> </div>
<div>I'm not following what you're describing. set_fwd_tbl_top sets LinearFDBTop whereas pipeline_tbl starts the cascade of LFT sets based on max_smps_per_node.</div>
<div> </div>
<blockquote class="gmail_quote" style="PADDING-LEFT: 1ex; MARGIN: 0px 0px 0px 0.8ex; BORDER-LEFT: #ccc 1px solid"><span></span><br>
<div>
<div></div>
<div><br>> /*<br>> * PARAMETERS<br>> * p_mgr<br>> diff --git a/opensm/opensm/osm_lin_fwd_rcv.c b/opensm/opensm/osm_lin_fwd_rcv.c<br>> index 2edb8d3..cb131b4 100644<br>> --- a/opensm/opensm/osm_lin_fwd_rcv.c<br>
> +++ b/opensm/opensm/osm_lin_fwd_rcv.c<br>> @@ -1,6 +1,6 @@<br>> /*<br>> * Copyright (c) 2004-2008 Voltaire, Inc. All rights reserved.<br>> - * Copyright (c) 2002-2005 Mellanox Technologies LTD. All rights reserved.<br>
> + * Copyright (c) 2002-2009 Mellanox Technologies LTD. All rights reserved.<br>> * Copyright (c) 1996-2003 Intel Corporation. All rights reserved.<br>> *<br>> * This software is available to you under a choice of one of two<br>
> @@ -36,7 +36,7 @@<br>> /*<br>> * Abstract:<br>> * Implementation of osm_lft_rcv_t.<br>> - * This object represents the NodeDescription Receiver object.<br>> + * This object represents the Linear Forwarding Table Receiver object.<br>
> * This object is part of the opensm family of objects.<br>> */<br>><br>> @@ -55,6 +55,7 @@ void osm_lft_rcv_process(IN void *context, IN void *data)<br>> {<br>> osm_sm_t *sm = context;<br>> osm_madw_t *p_madw = data;<br>
> + osm_dr_path_t *p_path;<br>> ib_smp_t *p_smp;<br>> uint32_t block_num;<br>> osm_switch_t *p_sw;<br>> @@ -62,6 +63,8 @@ void osm_lft_rcv_process(IN void *context, IN void *data)<br>> uint8_t *p_block;<br>
> ib_net64_t node_guid;<br>> ib_api_status_t status;<br>> + uint8_t block[IB_SMP_DATA_SIZE];<br>> + osm_madw_context_t mad_context;<br>><br>> CL_ASSERT(sm);<br>><br>> @@ -94,6 +97,16 @@ void osm_lft_rcv_process(IN void *context, IN void *data)<br>
> "\n\t\t\t\tSwitch 0x%" PRIx64 "\n",<br>> ib_get_err_str(status), cl_ntoh64(node_guid));<br>> }<br>> +<br>> + p_path = osm_physp_get_dr_path_ptr(osm_node_get_physp_ptr(p_sw->p_node, 0));<br>
> +<br>> + mad_context.lft_context.node_guid = node_guid;<br>> + mad_context.lft_context.set_method = TRUE;<br>> +<br>> + osm_sm_set_next_lft_block(sm, p_sw, &block[0], p_path,<br>
> + &mad_context);<br>> +<br>> + p_sw->lft_block_id_ho++;<br><br></div></div>Wouldn't it be simpler to encode block_id in a mad context?</blockquote>
<div> </div>
<div>Why simpler ? I think it complicates the receiver code to do that (assuming max_smps_per_node remains).</div>
<div> </div>
<blockquote class="gmail_quote" style="PADDING-LEFT: 1ex; MARGIN: 0px 0px 0px 0.8ex; BORDER-LEFT: #ccc 1px solid"><span id=""></span><br>
<div>
<div></div>
<div><br>> }<br>><br>> CL_PLOCK_RELEASE(sm->p_lock);<br>> diff --git a/opensm/opensm/osm_sm.c b/opensm/opensm/osm_sm.c<br>> index daa60ff..4e0fd2a 100644<br>> --- a/opensm/opensm/osm_sm.c<br>
> +++ b/opensm/opensm/osm_sm.c<br>> @@ -1,6 +1,6 @@<br>> /*<br>> * Copyright (c) 2004-2008 Voltaire, Inc. All rights reserved.<br>> - * Copyright (c) 2002-2005 Mellanox Technologies LTD. All rights reserved.<br>
> + * Copyright (c) 2002-2009 Mellanox Technologies LTD. All rights reserved.<br>> * Copyright (c) 1996-2003 Intel Corporation. All rights reserved.<br>> * Copyright (c) 2008 Xsigo Systems Inc. All rights reserved.<br>
> *<br>> @@ -441,6 +441,45 @@ Exit:<br>><br>> /**********************************************************************<br>> **********************************************************************/<br>> +void osm_sm_set_next_lft_block(IN osm_sm_t *p_sm, IN osm_switch_t *p_sw,<br>
> + IN uint8_t *p_block, IN osm_dr_path_t *p_path,<br>> + IN osm_madw_context_t *context)<br>> +{<br>> + ib_api_status_t status;<br>> +<br>> + for (;<br>
> + osm_switch_get_lft_block(p_sw, p_sw->lft_block_id_ho, p_block);<br>> + p_sw->lft_block_id_ho++) {<br>> + if (!p_sw->need_update && !p_sm->p_subn->need_update &&<br>
> + !memcmp(p_block,<br>> + p_sw->new_lft + p_sw->lft_block_id_ho * IB_SMP_DATA_SIZE,<br>> + IB_SMP_DATA_SIZE))<br>> + continue;<br>
> +<br>> + p_sm->lfts_updated = 1;<br>> +<br>> + OSM_LOG(p_sm->p_log, OSM_LOG_DEBUG,<br>> + "Writing FT block %u to switch 0x%" PRIx64 "\n",<br>
> + p_sw->lft_block_id_ho,<br>> + cl_ntoh64(context->lft_context.node_guid));<br>> +<br>> + status = osm_req_set(p_sm, p_path,<br>> + p_sw->new_lft +<br>
> + p_sw->lft_block_id_ho * IB_SMP_DATA_SIZE,<br>> + IB_SMP_DATA_SIZE, IB_MAD_ATTR_LIN_FWD_TBL,<br>> + cl_hton32(p_sw->lft_block_id_ho),<br>
> + CL_DISP_MSGID_NONE, context);<br>> +<br>> + if (status != IB_SUCCESS)<br>> + OSM_LOG(p_sm->p_log, OSM_LOG_ERROR, "ERR 2E11: "<br>
> + "Sending linear fwd. tbl. block failed (%s)\n",<br>> + ib_get_err_str(status));<br>> + break;<br>> + }<br>> +}<br>> +<br>
> +/**********************************************************************<br>> + **********************************************************************/<br>> static ib_api_status_t sm_mgrp_process(IN osm_sm_t * p_sm,<br>
> IN osm_mgrp_t * p_mgrp)<br>> {<br>> diff --git a/opensm/opensm/osm_subnet.c b/opensm/opensm/osm_subnet.c<br>> index ec15f8a..1964b7f 100644<br>> --- a/opensm/opensm/osm_subnet.c<br>
> +++ b/opensm/opensm/osm_subnet.c<br>> @@ -1,6 +1,6 @@<br>> /*<br>> * Copyright (c) 2004-2008 Voltaire, Inc. All rights reserved.<br>> - * Copyright (c) 2002-2008 Mellanox Technologies LTD. All rights reserved.<br>
> + * Copyright (c) 2002-2009 Mellanox Technologies LTD. All rights reserved.<br>> * Copyright (c) 1996-2003 Intel Corporation. All rights reserved.<br>> * Copyright (c) 2008 Xsigo Systems Inc. All rights reserved.<br>
> *<br>> @@ -295,6 +295,7 @@ static const opt_rec_t opt_tbl[] = {<br>> { "m_key_lease_period", OPT_OFFSET(m_key_lease_period), opts_parse_net16, NULL, 1 },<br>> { "sweep_interval", OPT_OFFSET(sweep_interval), opts_parse_uint32, NULL, 1 },<br>
> { "max_wire_smps", OPT_OFFSET(max_wire_smps), opts_parse_uint32, NULL, 1 },<br>> + { "max_smps_per_node", OPT_OFFSET(max_smps_per_node), opts_parse_uint32, NULL, 1 },<br>> { "console", OPT_OFFSET(console), opts_parse_charp, NULL, 0 },<br>
> { "console_port", OPT_OFFSET(console_port), opts_parse_uint16, NULL, 0 },<br>> { "transaction_timeout", OPT_OFFSET(transaction_timeout), opts_parse_uint32, NULL, 1 },<br>> @@ -671,6 +672,7 @@ void osm_subn_set_default_opt(IN osm_subn_opt_t * const p_opt)<br>
> p_opt->m_key_lease_period = 0;<br>> p_opt->sweep_interval = OSM_DEFAULT_SWEEP_INTERVAL_SECS;<br>> p_opt->max_wire_smps = OSM_DEFAULT_SMP_MAX_ON_WIRE;<br>> + p_opt->max_smps_per_node = OSM_DEFAULT_SMP_MAX_PER_NODE;<br>
> p_opt->console = strdup(OSM_DEFAULT_CONSOLE);<br>> p_opt->console_port = OSM_DEFAULT_CONSOLE_PORT;<br>> p_opt->transaction_timeout = OSM_DEFAULT_TRANS_TIMEOUT_MILLISEC;<br>> @@ -1461,6 +1463,10 @@ int osm_subn_output_conf(FILE *out, IN osm_subn_opt_t *const p_opts)<br>
> "max_wire_smps %u\n\n"<br>> "# The maximum time in [msec] allowed for a transaction to complete\n"<br>> "transaction_timeout %u\n\n"<br>> + "# Maximum number of SMPs per node sent in parallel\n"<br>
> + "# (0 means unlimited)\n"<br>> + "# Only applies to certain attributes\n"<br>> + "max_smps_per_node %u\n\n"<br>> "# Maximal time in [msec] a message can stay in the incoming message queue.\n"<br>
> "# If there is more than one message in the queue and the last message\n"<br>> "# stayed in the queue more than this value, any SA request will be\n"<br>> @@ -1470,6 +1476,7 @@ int osm_subn_output_conf(FILE *out, IN osm_subn_opt_t *const p_opts)<br>
> "single_thread %s\n\n",<br>> p_opts->max_wire_smps,<br>> p_opts->transaction_timeout,<br>> + p_opts->max_smps_per_node,<br>> p_opts->max_msg_fifo_timeout,<br>
> p_opts->single_thread ? "TRUE" : "FALSE");<br>><br>> diff --git a/opensm/opensm/osm_ucast_cache.c b/opensm/opensm/osm_ucast_cache.c<br>> index 216b496..31c930b 100644<br>
> --- a/opensm/opensm/osm_ucast_cache.c<br>> +++ b/opensm/opensm/osm_ucast_cache.c<br>> @@ -1,5 +1,5 @@<br>> /*<br>> - * Copyright (c) 2008 Mellanox Technologies LTD. All rights reserved.<br>> + * Copyright (c) 2008,2009 Mellanox Technologies LTD. All rights reserved.<br>
> *<br>> * This software is available to you under a choice of one of two<br>> * licenses. You may choose to be licensed under the terms of the GNU<br>> @@ -1085,9 +1085,11 @@ int osm_ucast_cache_process(osm_ucast_mgr_t * p_mgr)<br>
> memset(p_sw->lft, OSM_NO_PATH, IB_LID_UCAST_END_HO + 1);<br>> }<br>><br>> - osm_ucast_mgr_set_fwd_table(p_mgr, p_sw);<br>> + osm_ucast_mgr_set_fwd_tbl_top(p_mgr, p_sw);<br>
> }<br>><br>> + osm_ucast_pipeline_tbl(p_mgr);<br>> +<br>> return 0;<br>> }<br>><br>> diff --git a/opensm/opensm/osm_ucast_file.c b/opensm/opensm/osm_ucast_file.c<br>> index 2505c46..099e8ba 100644<br>
> --- a/opensm/opensm/osm_ucast_file.c<br>> +++ b/opensm/opensm/osm_ucast_file.c<br>> @@ -168,8 +168,8 @@ static int do_ucast_file_load(void *context)<br>> "routing algorithm\n");<br>
> } else if (!strncmp(p, "Unicast lids", 12)) {<br>> if (p_sw)<br>> - osm_ucast_mgr_set_fwd_table(&p_osm->sm.<br>> - ucast_mgr, p_sw);<br>
> + osm_ucast_mgr_set_fwd_tbl_top(&p_osm->sm.<br>> + ucast_mgr, p_sw);<br>> q = strstr(p, " guid 0x");<br>
> if (!q) {<br>> OSM_LOG(&p_osm->log, OSM_LOG_ERROR,<br>> @@ -247,7 +247,7 @@ static int do_ucast_file_load(void *context)<br>> }<br>><br>> if (p_sw)<br>
> - osm_ucast_mgr_set_fwd_table(&p_osm->sm.ucast_mgr, p_sw);<br>> + osm_ucast_mgr_set_fwd_tbl_top(&p_osm->sm.ucast_mgr, p_sw);<br>><br>> fclose(file);<br>> return 0;<br>
<br></div></div>I suppose that this breaks 'file' routing engine (did you test it?) -<br>instead of switch LFTs setup this will only update its TOPs.</blockquote>
<div> </div>
<div>At this point, I don't recall.</div>
<div> </div>
<blockquote class="gmail_quote" style="PADDING-LEFT: 1ex; MARGIN: 0px 0px 0px 0.8ex; BORDER-LEFT: #ccc 1px solid"><span id=""></span><br>
<div>
<div></div>
<div><br>> diff --git a/opensm/opensm/osm_ucast_ftree.c b/opensm/opensm/osm_ucast_ftree.c<br>> index bde6dbd..d65c685 100644<br>> --- a/opensm/opensm/osm_ucast_ftree.c<br>> +++ b/opensm/opensm/osm_ucast_ftree.c<br>
> @@ -2,7 +2,7 @@<br>> * Copyright (c) 2009 Simula Research Laboratory. All rights reserved.<br>> * Copyright (c) 2009 Sun Microsystems, Inc. All rights reserved.<br>> * Copyright (c) 2004-2008 Voltaire, Inc. All rights reserved.<br>
> - * Copyright (c) 2002-2007 Mellanox Technologies LTD. All rights reserved.<br>> + * Copyright (c) 2002-2009 Mellanox Technologies LTD. All rights reserved.<br>> * Copyright (c) 1996-2003 Intel Corporation. All rights reserved.<br>
> *<br>> * This software is available to you under a choice of one of two<br>> @@ -1905,8 +1905,8 @@ static void set_sw_fwd_table(IN cl_map_item_t * const p_map_item,<br>> ftree_fabric_t *p_ftree = (ftree_fabric_t *) context;<br>
><br>> p_sw->p_osm_sw->max_lid_ho = p_ftree->lft_max_lid;<br>> - osm_ucast_mgr_set_fwd_table(&p_ftree->p_osm->sm.ucast_mgr,<br>> - p_sw->p_osm_sw);<br>
> + osm_ucast_mgr_set_fwd_tbl_top(&p_ftree->p_osm->sm.ucast_mgr,<br>> + p_sw->p_osm_sw);<br>> }<br>><br>> /***************************************************<br>
> @@ -4005,6 +4005,8 @@ static int do_routing(IN void *context)<br>> /* for each switch, set its fwd table */<br>> cl_qmap_apply_func(&p_ftree->sw_tbl, set_sw_fwd_table, (void *)p_ftree);<br>><br>
> + osm_ucast_pipeline_tbl(&p_ftree->p_osm->sm.ucast_mgr);<br>> +<br>> /* write out hca ordering file */<br>> fabric_dump_hca_ordering(p_ftree);<br>><br>> diff --git a/opensm/opensm/osm_ucast_lash.c b/opensm/opensm/osm_ucast_lash.c<br>
> index 12b5e34..adf5f6c 100644<br>> --- a/opensm/opensm/osm_ucast_lash.c<br>> +++ b/opensm/opensm/osm_ucast_lash.c<br>> @@ -1,6 +1,6 @@<br>> /*<br>> * Copyright (c) 2004-2008 Voltaire, Inc. All rights reserved.<br>
> - * Copyright (c) 2002-2008 Mellanox Technologies LTD. All rights reserved.<br>> + * Copyright (c) 2002-2009 Mellanox Technologies LTD. All rights reserved.<br>> * Copyright (c) 1996-2003 Intel Corporation. All rights reserved.<br>
> * Copyright (c) 2007 Simula Research Laboratory. All rights reserved.<br>> * Copyright (c) 2007 Silicon Graphics Inc. All rights reserved.<br>> @@ -1045,8 +1045,11 @@ static void populate_fwd_tbls(lash_t * p_lash)<br>
> physical_egress_port);<br>> }<br>> } /* for */<br>> - osm_ucast_mgr_set_fwd_table(&p_osm->sm.ucast_mgr, p_sw);<br>
> + osm_ucast_mgr_set_fwd_tbl_top(&p_osm->sm.ucast_mgr, p_sw);<br>> }<br>> +<br>> + osm_ucast_pipeline_tbl(&p_osm->sm.ucast_mgr);<br>> +<br>> OSM_LOG_EXIT(p_log);<br>
> }<br>><br>> diff --git a/opensm/opensm/osm_ucast_mgr.c b/opensm/opensm/osm_ucast_mgr.c<br>> index 78a7031..86d1c98 100644<br>> --- a/opensm/opensm/osm_ucast_mgr.c<br>> +++ b/opensm/opensm/osm_ucast_mgr.c<br>
> @@ -1,6 +1,6 @@<br>> /*<br>> * Copyright (c) 2004-2008 Voltaire, Inc. All rights reserved.<br>> - * Copyright (c) 2002-2008 Mellanox Technologies LTD. All rights reserved.<br>> + * Copyright (c) 2002-2009 Mellanox Technologies LTD. All rights reserved.<br>
> * Copyright (c) 1996-2003 Intel Corporation. All rights reserved.<br>> *<br>> * This software is available to you under a choice of one of two<br>> @@ -315,16 +315,14 @@ Exit:<br>><br>> /**********************************************************************<br>
> **********************************************************************/<br>> -int osm_ucast_mgr_set_fwd_table(IN osm_ucast_mgr_t * p_mgr,<br>> - IN osm_switch_t * p_sw)<br>> +int osm_ucast_mgr_set_fwd_tbl_top(IN osm_ucast_mgr_t * p_mgr,<br>
> + IN osm_switch_t * p_sw)<br>> {<br>> osm_node_t *p_node;<br>> osm_dr_path_t *p_path;<br>> osm_madw_context_t context;<br>> ib_api_status_t status;<br>
> ib_switch_info_t si;<br>> - uint16_t block_id_ho = 0;<br>> - uint8_t block[IB_SMP_DATA_SIZE];<br>> boolean_t set_swinfo_require = FALSE;<br>> uint16_t lin_top;<br>> uint8_t life_state;<br>
> @@ -382,48 +380,8 @@ int osm_ucast_mgr_set_fwd_table(IN osm_ucast_mgr_t * p_mgr,<br>> ib_get_err_str(status));<br>> }<br>><br>> - /*<br>> - Send linear forwarding table blocks to the switch<br>
> - as long as the switch indicates it has blocks needing<br>> - configuration.<br>> - */<br>> -<br>> - context.lft_context.node_guid = osm_node_get_node_guid(p_node);<br>> - context.lft_context.set_method = TRUE;<br>
> -<br>> - if (!p_sw->new_lft) {<br>> - /* any routing should provide the new_lft */<br>> - CL_ASSERT(p_mgr->p_subn->opt.use_ucast_cache &&<br>> - p_mgr->cache_valid && !p_sw->need_update);<br>
> - goto Exit;<br>> - }<br>> -<br>> - for (block_id_ho = 0;<br>> - osm_switch_get_lft_block(p_sw, block_id_ho, block);<br>> - block_id_ho++) {<br>> - if (!p_sw->need_update && !p_mgr->p_subn->need_update &&<br>
> - !memcmp(block,<br>> - p_sw->new_lft + block_id_ho * IB_SMP_DATA_SIZE,<br>> - IB_SMP_DATA_SIZE))<br>> - continue;<br>> -<br>
> - OSM_LOG(p_mgr->p_log, OSM_LOG_DEBUG,<br>> - "Writing FT block %u\n", block_id_ho);<br>> -<br>> - status = osm_req_set(p_mgr->sm, p_path,<br>> - p_sw->new_lft +<br>
> - block_id_ho * IB_SMP_DATA_SIZE,<br>> - sizeof(block), IB_MAD_ATTR_LIN_FWD_TBL,<br>> - cl_hton32(block_id_ho), CL_DISP_MSGID_NONE,<br>
> - &context);<br>> + p_sw->lft_block_id_ho = 0;<br>><br>> - if (status != IB_SUCCESS)<br>> - OSM_LOG(p_mgr->p_log, OSM_LOG_ERROR, "ERR 3A05: "<br>
> - "Sending linear fwd. tbl. block failed (%s)\n",<br>> - ib_get_err_str(status));<br>> - }<br>> -<br>> -Exit:<br>> OSM_LOG_EXIT(p_mgr->p_log);<br>
> return 0;<br>> }<br>> @@ -508,7 +466,7 @@ static void ucast_mgr_process_tbl(IN cl_map_item_t * p_map_item,<br>> }<br>> }<br>><br>> - osm_ucast_mgr_set_fwd_table(p_mgr, p_sw);<br>
> + osm_ucast_mgr_set_fwd_tbl_top(p_mgr, p_sw);<br>><br>> if (p_mgr->p_subn->opt.lmc)<br>> free_ports_priv(p_mgr);<br>> @@ -516,6 +474,47 @@ static void ucast_mgr_process_tbl(IN cl_map_item_t * p_map_item,<br>
> OSM_LOG_EXIT(p_mgr->p_log);<br>> }<br>><br>> +static void ucast_mgr_pipeline_tbl(IN osm_switch_t *p_sw,<br>> + IN osm_ucast_mgr_t *p_mgr)<br>> +{<br>> + osm_dr_path_t *p_path;<br>
> + osm_madw_context_t mad_context;<br>> + uint8_t block[IB_SMP_DATA_SIZE];<br>> +<br>> + OSM_LOG_ENTER(p_mgr->p_log);<br>> +<br>> + CL_ASSERT(p_sw && p_sw->p_node);<br>> +<br>
> + OSM_LOG(p_mgr->p_log, OSM_LOG_DEBUG,<br>> + "Processing switch 0x%" PRIx64 "\n",<br>> + cl_ntoh64(osm_node_get_node_guid(p_sw->p_node)));<br>> +<br>> + /*<br>
> + Send linear forwarding table blocks to the switch<br>> + as long as the switch indicates it has blocks needing<br>> + configuration.<br>> + */<br>> + if (!p_sw->new_lft) {<br>
> + /* any routing should provide the new_lft */<br>> + CL_ASSERT(p_mgr->p_subn->opt.use_ucast_cache &&<br>> + p_mgr->cache_valid && !p_sw->need_update);<br>
> + goto Exit;<br>> + }<br>> +<br>> + p_path = osm_physp_get_dr_path_ptr(osm_node_get_physp_ptr(p_sw->p_node, 0));<br>> +<br>> + mad_context.lft_context.node_guid = osm_node_get_node_guid(p_sw->p_node);<br>
> + mad_context.lft_context.set_method = TRUE;<br>> +<br>> + osm_sm_set_next_lft_block(p_mgr->sm, p_sw, &block[0], p_path,<br>> + &mad_context);<br>> +<br>> + p_sw->lft_block_id_ho++;<br>
> +<br>> +Exit:<br>> + OSM_LOG_EXIT(p_mgr->p_log);<br>> +}<br>> +<br>> /**********************************************************************<br>> **********************************************************************/<br>
> static void ucast_mgr_process_neighbors(IN cl_map_item_t * p_map_item,<br>> @@ -870,6 +869,28 @@ static void sort_ports_by_switch_load(osm_ucast_mgr_t * m)<br>> add_sw_endports_to_order_list(s[i], m);<br>
> }<br>><br>> +void osm_ucast_pipeline_tbl(osm_ucast_mgr_t * p_mgr)<br>> +{<br>> + cl_qmap_t *p_sw_tbl;<br>> + osm_switch_t *p_sw;<br>> + int i;<br>> +<br>> + for (i = 0;<br>> + !p_mgr->p_subn->opt.max_smps_per_node ||<br>
> + i < p_mgr->p_subn->opt.max_smps_per_node;<br>> + i++) {<br>> + p_mgr->sm->lfts_updated = 0;<br>> + p_sw_tbl = &p_mgr->p_subn->sw_guid_tbl;<br>
> + p_sw = (osm_switch_t *) cl_qmap_head(p_sw_tbl);<br>> + while (p_sw != (osm_switch_t *) cl_qmap_end(p_sw_tbl)) {<br>> + ucast_mgr_pipeline_tbl(p_sw, p_mgr);<br>> + p_sw = (osm_switch_t *) cl_qmap_next(&p_sw->map_item);<br>
> + }<br>> + if (!p_mgr->sm->lfts_updated)<br>> + break;<br>> + }<br>> +}<br><br></div></div>Is it possible (for example in case of send errors) that "partial" LFT<br>
blocks sending will trigger wait_for_pending_transaction() completion?</blockquote>
<div> </div>
<div>I don't know. Is this different from the original algorithm in the case of send errors ?</div>
<div> </div>
<div>-- Hal</div>
<div> </div>
<blockquote class="gmail_quote" style="PADDING-LEFT: 1ex; MARGIN: 0px 0px 0px 0.8ex; BORDER-LEFT: #ccc 1px solid"><span id=""></span><br><font color="#888888"><br>Sasha<br></font>
<div>
<div></div>
<div><br>> +<br>> static int ucast_mgr_build_lfts(osm_ucast_mgr_t * p_mgr)<br>> {<br>> cl_qlist_init(&p_mgr->port_order_list);<br>> @@ -904,6 +925,8 @@ static int ucast_mgr_build_lfts(osm_ucast_mgr_t * p_mgr)<br>
> cl_qmap_apply_func(&p_mgr->p_subn->sw_guid_tbl, ucast_mgr_process_tbl,<br>> p_mgr);<br>><br>> + osm_ucast_pipeline_tbl(p_mgr);<br>> +<br>> cl_qlist_remove_all(&p_mgr->port_order_list);<br>
><br>> return 0;<br>><br>_______________________________________________<br>general mailing list<br><a href="mailto:general@lists.openfabrics.org" target="_blank">general@lists.openfabrics.org</a><br><a href="http://lists.openfabrics.org/cgi-bin/mailman/listinfo/general" target="_blank">http://lists.openfabrics.org/cgi-bin/mailman/listinfo/general</a><br>
<br>To unsubscribe, please visit <a href="http://openib.org/mailman/listinfo/openib-general" target="_blank">http://openib.org/mailman/listinfo/openib-general</a><br></div></div></blockquote></div><br>