[ofa-general] Re: opensm: a bug in heavy sweep? - no LFT re-configuration

[Sasha Khapyorsky]

On 14:59 Sun 22 Jul     , Eitan Zahavi wrote:
> Hi Sasha
> 
> Let's assume someone has reset a switch on the fabric.
> What would cause the SM to re-assign the LFT of that switch?

OpenSM will sweep and drop this switch and when switch will back it will
be initialized again. But if the reset was too fast (relative to
discovery), we can be in trouble (and maybe not only with LFTs).

> I assumed that there is a mechanism to do that.

Not for "fast" switch reboot.

Hmm, I think we could try to detect this case by comparing
SwitchInfo:LinerFDBTop with current p_sw->max_lid_ho or even by seeing
that PortInfo:LID is not set. Something like below:


diff --git a/opensm/include/opensm/osm_switch.h b/opensm/include/opensm/osm_switch.h
index 5b2b19e..62c072f 100644
--- a/opensm/include/opensm/osm_switch.h
+++ b/opensm/include/opensm/osm_switch.h
@@ -112,6 +112,7 @@ typedef struct _osm_switch
 	osm_fwd_tbl_t				fwd_tbl;
 	osm_mcast_tbl_t				mcast_tbl;
 	uint32_t				discovery_count;
+	unsigned				update_ft;
 	void					*priv;
 } osm_switch_t;
 /*
@@ -152,6 +153,10 @@ typedef struct _osm_switch
 *		during the current fabric sweep.  This number is reset
 *		to zero at the start of a sweep.
 *
+*	update_ft
+*		When set fwd tables will be updated regardless to entry
+*		values locally stored in fwd tables images
+*
 * SEE ALSO
 *	Switch object
 *********/
diff --git a/opensm/opensm/osm_port_info_rcv.c b/opensm/opensm/osm_port_info_rcv.c
index adece65..8bbbcac 100644
--- a/opensm/opensm/osm_port_info_rcv.c
+++ b/opensm/opensm/osm_port_info_rcv.c
@@ -336,6 +336,9 @@ __osm_pi_rcv_process_switch_port(
       break;
     }
   }
+  else if (port_num == 0 && p_node->sw &&
+           (!p_pi->base_lid || !p_pi->master_sm_base_lid))
+    p_node->sw->update_ft = 1;
 
   /*
     Update the PortInfo attribute.
diff --git a/opensm/opensm/osm_ucast_mgr.c b/opensm/opensm/osm_ucast_mgr.c
index b44a3ba..03516ae 100644
--- a/opensm/opensm/osm_ucast_mgr.c
+++ b/opensm/opensm/osm_ucast_mgr.c
@@ -811,7 +811,8 @@ osm_ucast_mgr_set_fwd_table(
        osm_switch_get_fwd_tbl_block( p_sw, block_id_ho, block ) ;
        block_id_ho++ )
   {
-    if (!memcmp(block, p_mgr->lft_buf + block_id_ho * 64, 64))
+    if (!p_sw->update_ft &&
+        !memcmp(block, p_mgr->lft_buf + block_id_ho * 64, 64))
       continue;
 
     if( osm_log_is_active( p_mgr->p_log, OSM_LOG_DEBUG ) )
@@ -850,6 +851,7 @@ osm_ucast_mgr_set_fwd_table(
     }
   }
 
+  p_sw->update_ft = 0;
   OSM_LOG_EXIT( p_mgr->p_log );
 }
 


BTW what do you think is the best way to detect switch power up? I
didn't really find a strong requirement for at powerup initialization of
any suitable component.

> Anyway, kill -HUP should flush out the state and restart from scratch.

Thinking more about it I'm not sure. Similar flush will be required for
another "stored" components like pkey, sl2vl tables etc.. So it is more
than just "regular" heavy sweep, another signal or option could be used
for this, but OTOH it becomes very close to OpenSM restarting..

Sasha

> 
> 
> Eitan
> 
> > -----Original Message-----
> > From: Sasha Khapyorsky [mailto:sashak at voltaire.com] 
> > Sent: Sunday, July 22, 2007 1:22 PM
> > To: Eitan Zahavi
> > Cc: OPENIB; hal.rosenstock at gmail.com; Yevgeny Kliteynik
> > Subject: Re: opensm: a bug in heavy sweep? - no LFT re-configuration
> > 
> > Hi Eitan,
> > 
> > On 09:36 Sun 22 Jul     , Eitan Zahavi wrote:
> > > Hi Sasha
> > > 
> > > I am running some tests manually and apparently it looks 
> > like I found 
> > > a bug. Here is the sequence of things:
> > > 1. SM sweeps the fabric assign LFTs
> > > 2. I manually modify some LFTs (single entry now marked 
> > UNREACHABLE 3. 
> > > I force some switch change bit to 1 or issue kill -HUP 4. The SM 
> > > reports SUBNET UP 5. The modified LFT entry is still 
> > UNREACHABLE and 
> > > the path is broken
> > 
> > Right, in most cases (unless OpenSM has its own changes in 
> > the same LFT
> > block) OpenSM will refer its own LFT image for  "need to update"
> > decision, so _manual_ changes will not trigger new update. 
> > Rerunning OpenSM should help however.
> > 
> > > It looks to me some optimization of routing does not fully reroute 
> > > unless some condition is met - but that condition does not 
> > include the 
> > > above triggers listed in step 3.
> > 
> > Rereading all fabrics LFTs by default seems to be too 
> > expensive operations. At least by default, if it is real 
> > requirement this could be enforced manually, for example when 
> > kill -HUP is used. Thoughts?
> > 
> > Sasha
> >