[ofa-general] RE: [PATCH] OpenSM handling of "Babbling" Ports

Hal Rosenstock halr at voltaire.com
Mon Jul 9 09:57:47 PDT 2007


Hi Amit,

On Mon, 2007-07-09 at 12:40, Amit Krig wrote:
> Hi Hal
> 
> I was only talking on logical link == Active state.
> In this state the watchdog can bring the physical link to recovery state
> while the logical link will bounce between Active and ActiveDefer.

OK; I follow this but I'm not sure what you are saying about "applying"
it to the patch in question.

> Regarding the responsive issue, OpenSM in this scenario should move the
> logical link in the responsive side to Init state

rather than disabling it on some threshold. What about the other similar
traps 129 and 130 ? How should they be handled ?

> that way the watchdog will stop bringing down the link and then do the checks

I think the checks will still fail but this seems like it would stop the
traps from being generated (so fast).

-- Hal

> Amit
> 
> -----Original Message-----
> From: Hal Rosenstock [mailto:halr at voltaire.com] 
> Sent: Monday, July 09, 2007 4:43 PM
> To: Amit Krig
> Cc: general at lists.openfabrics.org; Eitan Zahavi; Yevgeny Kliteynik
> Subject: RE: [PATCH] OpenSM handling of "Babbling" Ports
> 
> Hi Amit,
> 
> On Mon, 2007-07-09 at 09:27, Amit Krig wrote:
> > Hi Hal,
> > 
> > In such case OpenSM should first check that the OPVL fields of the 
> > ports (the one that sends the traps and its peer) are identical, If 
> > you have a mismatch in the OPVL field, the link watchdog mechanism 
> > will retrain the logical link in high rate
> 
> OpVLs only takes "effect" if set after link active only if the link is
> bounced (not if it stays active).
> 
> Also and more significantly, in terms of the specific issue, the peer
> SMA is often non responsive or shortly becomes non responsive so the
> peer OpVLs cannot readily be verified post this being detected.
> 
> -- Hal
> 
> > Amit
> > 
> > 
> > -----Original Message-----
> > From: Hal Rosenstock [mailto:halr at voltaire.com]
> > Sent: Thursday, July 05, 2007 3:58 PM
> > To: general at lists.openfabrics.org
> > Cc: Eitan Zahavi; Yevgeny Kliteynik
> > Subject: [PATCH] OpenSM handling of "Babbling" Ports
> > 
> > A "babbling" port is a port which causes traps to be generated 
> > frequently.
> > It may directly be "this" port which generates the traps or the peer 
> > port detecting the issue and that the SMA on switch port 0 generates 
> > the traps.
> > This has only currently been observed for trap 131 but will also apply
> 
> > for traps 129 and 130 as well which are other urgent and similar
> traps.
> > 
> > Note that there appears to be a bug in Mellanox firmware for both
> > Anafa-2 and Tavor at a minimum which causes the max trap rate not to 
> > be adhered to and relief for this does not appear to be in short term 
> > sight.
> > 
> > Policy
> > When a bablbing port is detected, OpenSM will disable the port or its 
> > peer switch port (depending on which trap) which should terminate the 
> > trap storm.
> > 
> > Detection
> > 250 consecutive traps of this type will be used as the (initial) 
> > threshold. The reason for this is so as to not prematurely detect this
> 
> > and disable a port.
> > 
> > Recovery
> > Admin would reenable port when OK again. (This usually involves 
> > rebooting the node causing the trap to be indicated.)
> > 
> > Signed-off-by: Hal Rosenstock <halr at voltaire.com>
> > 
> > diff --git a/opensm/include/opensm/osm_subnet.h
> > b/opensm/include/opensm/osm_subnet.h
> > index bedd63f..1150703 100644
> > --- a/opensm/include/opensm/osm_subnet.h
> > +++ b/opensm/include/opensm/osm_subnet.h
> > @@ -286,6 +286,7 @@ typedef struct _osm_subn_opt
> >    boolean_t                honor_guid2lid_file;
> >    boolean_t                daemon;
> >    boolean_t                sm_inactive;
> > +  boolean_t                babbling_port_policy;
> >    osm_qos_options_t        qos_options;
> >    osm_qos_options_t        qos_ca_options;
> >    osm_qos_options_t        qos_sw0_options;
> > @@ -487,6 +488,9 @@ typedef struct _osm_subn_opt
> >  *
> >  *	sm_inactive
> >  *		OpenSM will start with SM in not active state.
> > +*
> > +*	babbling_port_policy
> > +*		OpenSM will enforce its "babbling" port policy.
> >  *	
> >  *	perfmgr
> >  *		Enable or disable the performance manager
> > diff --git a/opensm/opensm/osm_subnet.c b/opensm/opensm/osm_subnet.c 
> > index 726b665..87b71e5 100644
> > --- a/opensm/opensm/osm_subnet.c
> > +++ b/opensm/opensm/osm_subnet.c
> > @@ -472,6 +472,7 @@ osm_subn_set_default_opt(
> >    p_opt->honor_guid2lid_file = FALSE;
> >    p_opt->daemon = FALSE;
> >    p_opt->sm_inactive = FALSE;
> > +  p_opt->babbling_port_policy = FALSE;
> >  #ifdef ENABLE_OSM_PERF_MGR
> >    p_opt->perfmgr = FALSE;
> >    p_opt->perfmgr_sweep_time_s = OSM_PERFMGR_DEFAULT_SWEEP_TIME_S; @@
> > -1358,6 +1359,10 @@ osm_subn_parse_conf_file(
> >          "sm_inactive",
> >          p_key, p_val, &p_opts->sm_inactive);
> >  
> > +      __osm_subn_opts_unpack_boolean(
> > +        "babbling_port_policy",
> > +        p_key, p_val, &p_opts->babbling_port_policy);
> > +
> >  #ifdef ENABLE_OSM_PERF_MGR
> >        __osm_subn_opts_unpack_boolean(
> >          "perfmgr",
> > @@ -1631,9 +1636,12 @@ osm_subn_write_conf_file(
> >      "# Daemon mode\n"
> >      "daemon %s\n\n"
> >      "# SM Inactive\n"
> > -    "sm_inactive %s\n\n",
> > +    "sm_inactive %s\n\n"
> > +    "# Babbling Port Policy\n"
> > +    "babbling_port_policy %s\n\n",
> >      p_opts->daemon ? "TRUE" : "FALSE",
> > -    p_opts->sm_inactive ? "TRUE" : "FALSE"
> > +    p_opts->sm_inactive ? "TRUE" : "FALSE",
> > +    p_opts->babbling_port_policy ? "TRUE" : "FALSE"
> >      );
> >  
> >  #ifdef ENABLE_OSM_PERF_MGR
> > diff --git a/opensm/opensm/osm_trap_rcv.c 
> > b/opensm/opensm/osm_trap_rcv.c index 5900c51..fbb6dac 100644
> > --- a/opensm/opensm/osm_trap_rcv.c
> > +++ b/opensm/opensm/osm_trap_rcv.c
> > @@ -1,5 +1,5 @@
> >  /*
> > - * Copyright (c) 2004-2006 Voltaire, Inc. All rights reserved.
> > + * Copyright (c) 2004-2007 Voltaire, Inc. All rights reserved.
> >   * Copyright (c) 2002-2006 Mellanox Technologies LTD. All rights 
> > reserved.
> >   * Copyright (c) 1996-2003 Intel Corporation. All rights reserved.
> >   *
> > @@ -548,6 +548,61 @@ __osm_trap_rcv_process_request(
> >          }
> >          else
> >          {
> > +          /* When babbling port policy option is enabled and
> > +             Threshold for disabling a "babbling" port is exceeded */
> > +          if ( p_rcv->p_subn->opt.babbling_port_policy &&
> > +               num_received >= 250 )
> > +          {
> > +            uint8_t               payload[IB_SMP_DATA_SIZE];
> > +            ib_port_info_t*       p_pi = (ib_port_info_t*)payload;
> > +            const ib_port_info_t* p_old_pi;
> > +            osm_madw_context_t    context;
> > +
> > +            /* If trap 131, might want to disable peer port if
> > available */
> > +            /* but peer port has been observed not to respond to SM 
> > + requests */
> > +
> > +            osm_log( p_rcv->p_log, OSM_LOG_ERROR,
> > +                     "__osm_trap_rcv_process_request: ERR 3810: "
> > +                     " Disabling physical port lid:0x%02X num:%u\n",
> > +                     cl_ntoh16(p_ntci->data_details.ntc_129_131.lid),
> > +                     p_ntci->data_details.ntc_129_131.port_num
> > +                     );
> > +
> > +            p_old_pi = &p_physp->port_info;
> > +            memcpy( payload, p_old_pi, sizeof(ib_port_info_t) );
> > +
> > +            /* Set port to disabled/down */
> > +            ib_port_info_set_port_state( p_pi, IB_LINK_DOWN );
> > +            ib_port_info_set_port_phys_state( 
> > + IB_PORT_PHYS_STATE_DISABLED, p_pi );
> > +
> > +            context.pi_context.node_guid = osm_node_get_node_guid(
> > osm_physp_get_node_ptr( p_physp ) );
> > +            context.pi_context.port_guid = osm_physp_get_port_guid(
> > p_physp );
> > +            context.pi_context.set_method = TRUE;
> > +            context.pi_context.update_master_sm_base_lid = FALSE;
> > +            context.pi_context.light_sweep = FALSE;
> > +            context.pi_context.active_transition = FALSE;
> > +
> > +            status = osm_req_set( &p_rcv->p_subn->p_osm->sm.req,
> > +                                   osm_physp_get_dr_path_ptr( p_physp
> > ),
> > +                                   payload,
> > +                                   sizeof(payload),
> > +                                   IB_MAD_ATTR_PORT_INFO,
> > +                                   cl_hton32(osm_physp_get_port_num(
> > p_physp )),
> > +                                   CL_DISP_MSGID_NONE,
> > +                                  &context );
> > +
> > +            if( status == IB_SUCCESS )
> > +            {
> > +               goto Exit;
> > +            }
> > +            else
> > +            {
> > +               osm_log( p_rcv->p_log, OSM_LOG_ERROR,
> > +                        "__osm_trap_rcv_process_request: ERR 3811: "
> > +                        "Request to set PortInfo failed\n" );
> > +            }
> > +          }
> > +
> >            osm_log( p_rcv->p_log, OSM_LOG_VERBOSE,
> >                     "__osm_trap_rcv_process_request: "
> >                     "Marking unhealthy physical port by lid:0x%02X 
> > num:%u\n",
> > 
> > 
> > 
> > 
> 




More information about the general mailing list