[ofa-general] Re: opensm: a bug in heavy sweep? - no LFT re-configuration

[Hal Rosenstock]

On 7/23/07, Hal Rosenstock <hal.rosenstock at gmail.com> wrote:
>
> Hi Sasha,
>
> On 7/22/07, Sasha Khapyorsky <sashak at voltaire.com> wrote:
>
> > On 14:59 Sun 22 Jul     , Eitan Zahavi wrote:
> > > Hi Sasha
> > >
> > > Let's assume someone has reset a switch on the fabric.
> > > What would cause the SM to re-assign the LFT of that switch?
> >
> > OpenSM will sweep and drop this switch and when switch will back it will
> > be initialized again. But if the reset was too fast (relative to
> > discovery), we can be in trouble (and maybe not only with LFTs).
> >
> > > I assumed that there is a mechanism to do that.
> >
> > Not for "fast" switch reboot.
> >
> > Hmm, I think we could try to detect this by comparing
> > SwitchInfo:LinerFDBTop with current p_sw->max_lid_ho or even by seeing
> > that PortInfo:LID is not set.
>
>
> Not sure about checking PortInfo:LID. Wouldn't that approach need to be
> qualified by PortState (armed or active) ? LFTTop seems better to me or
> perhaps a combination of the two but I may be missing something.
>

Another thought on this :-(

Not sure that resetting either LID or LFTTop is required by the spec either
so this is relying on "beyond the spec" behavior and may not be true for all
switch implementations.

-- Hal


>
> > Something like below:
> >
> >
> > diff --git a/opensm/include/opensm/osm_switch.h
> > b/opensm/include/opensm/osm_switch.h
> > index 5b2b19e..62c072f 100644
> > --- a/opensm/include/opensm/osm_switch.h
> > +++ b/opensm/include/opensm/osm_switch.h
> > @@ -112,6 +112,7 @@ typedef struct _osm_switch
> >        osm_fwd_tbl_t                           fwd_tbl;
> >        osm_mcast_tbl_t                         mcast_tbl;
> >        uint32_t                                discovery_count;
> > +       unsigned                                update_ft;
> >        void                                    *priv;
> > } osm_switch_t;
> > /*
> > @@ -152,6 +153,10 @@ typedef struct _osm_switch
> > *              during the current fabric sweep.  This number is reset
> > *              to zero at the start of a sweep.
> > *
> > +*      update_ft
> > +*              When set fwd tables will be updated regardless to entry
> > +*              values locally stored in fwd tables images
> > +*
> > * SEE ALSO
> > *      Switch object
> > *********/
> > diff --git a/opensm/opensm/osm_port_info_rcv.c
> > b/opensm/opensm/osm_port_info_rcv.c
> > index adece65..8bbbcac 100644
> > --- a/opensm/opensm/osm_port_info_rcv.c
> > +++ b/opensm/opensm/osm_port_info_rcv.c
> > @@ -336,6 +336,9 @@ __osm_pi_rcv_process_switch_port(
> >       break;
> >     }
> >   }
> > +  else if (port_num == 0 && p_node->sw &&
> > +           (!p_pi->base_lid || !p_pi->master_sm_base_lid))
> > +    p_node->sw->update_ft = 1;
> >
> >   /*
> >     Update the PortInfo attribute.
> > diff --git a/opensm/opensm/osm_ucast_mgr.c
> > b/opensm/opensm/osm_ucast_mgr.c
> > index b44a3ba..03516ae 100644
> > --- a/opensm/opensm/osm_ucast_mgr.c
> > +++ b/opensm/opensm/osm_ucast_mgr.c
> > @@ -811,7 +811,8 @@ osm_ucast_mgr_set_fwd_table(
> >        osm_switch_get_fwd_tbl_block( p_sw, block_id_ho, block ) ;
> >        block_id_ho++ )
> >   {
> > -    if (!memcmp(block, p_mgr->lft_buf + block_id_ho * 64, 64))
> > +    if (!p_sw->update_ft &&
> > +        !memcmp(block, p_mgr->lft_buf + block_id_ho * 64, 64))
> >       continue;
> >
> >     if( osm_log_is_active( p_mgr->p_log, OSM_LOG_DEBUG ) )
> > @@ -850,6 +851,7 @@ osm_ucast_mgr_set_fwd_table(
> >     }
> >   }
> >
> > +  p_sw->update_ft = 0;
> >   OSM_LOG_EXIT( p_mgr->p_log );
> > }
> >
> >
> >
> > BTW what do you think is the best way to detect switch power up? I
> > didn't really find a strong requirement for at powerup initialization of
> > any suitable component.
>
>
> Peer switch link state change is insufficient to differentiate switch
> reboot from "normal" link up/down. There is no IB standard indication of
> this.
>
>
>
> > > Anyway, kill -HUP should flush out the state and restart from scratch.
> >
> > Thinking more about it I'm not sure. Similar flush will be required for
> > another "stored" components like pkey, sl2vl tables etc.. So it is more
> > than just "regular" heavy sweep, another signal or option could be used
> > for this, but OTOH it becomes very close to OpenSM restarting..
>
>
> Shouldn't this be automatic rather than requiring the admin to issue a
> signal somehow ?
>
> -- Hal
>
>
> Sasha
> >
> > >
> > >
> > > Eitan
> > >
> > > > -----Original Message-----
> > > > From: Sasha Khapyorsky [mailto: sashak at voltaire.com]
> > > > Sent: Sunday, July 22, 2007 1:22 PM
> > > > To: Eitan Zahavi
> > > > Cc: OPENIB; hal.rosenstock at gmail.com ; Yevgeny Kliteynik
> > > > Subject: Re: opensm: a bug in heavy sweep? - no LFT re-configuration
> > > >
> > > > Hi Eitan,
> > > >
> > > > On 09:36 Sun 22 Jul     , Eitan Zahavi wrote:
> > > > > Hi Sasha
> > > > >
> > > > > I am running some tests manually and apparently it looks
> > > > like I found
> > > > > a bug. Here is the sequence of things:
> > > > > 1. SM sweeps the fabric assign LFTs
> > > > > 2. I manually modify some LFTs (single entry now marked
> > > > UNREACHABLE 3.
> > > > > I force some switch change bit to 1 or issue kill -HUP 4. The SM
> > > > > reports SUBNET UP 5. The modified LFT entry is still
> > > > UNREACHABLE and
> > > > > the path is broken
> > > >
> > > > Right, in most cases (unless OpenSM has its own changes in
> > > > the same LFT
> > > > block) OpenSM will refer its own LFT image for  "need to update"
> > > > decision, so _manual_ changes will not trigger new update.
> > > > Rerunning OpenSM should help however.
> > > >
> > > > > It looks to me some optimization of routing does not fully reroute
> >
> > > > > unless some condition is met - but that condition does not
> > > > include the
> > > > > above triggers listed in step 3.
> > > >
> > > > Rereading all fabrics LFTs by default seems to be too
> > > > expensive operations. At least by default, if it is real
> > > > requirement this could be enforced manually, for example when
> > > > kill -HUP is used. Thoughts?
> > > >
> > > > Sasha
> > > >
> >
>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.openfabrics.org/pipermail/general/attachments/20070723/92545ea4/attachment.html>