[ofa-general] Lost in-service traps during Open SM migration

Sasha Khapyorsky sashak at voltaire.com
Thu Jul 26 19:59:52 PDT 2007


On 12:37 Thu 26 Jul     , lbt wrote:
>  Thanks for the suggestion Sasha!
> 
>  Our host stack does receive a rereregistration notice and does resubscribe
>  all handlers at
>  that point in time. At the time of the SM migration, our stack prints out
>  some informational messages to
>  confirm this:
>  Jul 18 14:31:09 localhost kernel: Event IB_EVENT_CLIENT_REREGISTER occurred
>  on port 1
>  Jul 18 14:31:09 localhost kernel: OpemSM migrated, old SM LID=1 new SM LID=8
> 
>  And also confirmed in the SM logs that after the migration, the higher
>  priority SM is getting a subscription request for in-service trap:
>  Jul 18 14:32:13 103550 [41E02960] -> osm_infr_rcv_process_set_method:
>  Subscribe Request with QPN: 0x000001
>  Jul 18 14:32:13 103554 [41E02960] -> osm_infr_get_by_rec: [
>  Jul 18 14:32:13 103558 [41E02960] -> __dump_all_informs: [
>  Jul 18 14:32:13 103562 [41E02960] -> InformInfo dump:
>                                 gid.....................0x0000000000000000 :
>  0x0000000000000000
>                                 lid_range_begin.........0xFFFF
>                                 lid_range_end...........0x0
>                                 is_generic..............0x1
>                                 subscribe...............0x0
>                                 trap_type...............0x3
>                                 trap_num................64
>                                 qpn.....................0x000001
>                                 resp_time_val...........0x0
>                                 node_type...............0x000004
>  Jul 18 14:32:13 103569 [41E02960] -> __dump_all_informs: ]
> 
>  It maybe a problem if the resubscription of the in-service handler occurs
>  after the in-service notice was forwarded, but I think the problem is that
>  there is never a notice that is forwared for the higher priority SM port
>  that is restored.

And after OpenSM migration, did you receive in-service notices for
another ports? Does the problem happen only in migration time?

>  Perhaps, neither SM (the lower priority and higher
>  priority one), generates an in-service trap because of the timing  gap
>  between when the restored port is detected and "marked" (i.e. added to
>  new_ports_list) and when in-service traps are generated for new ports.
>  During SM migration, the lower priority SM detects the new port, but the
>  higher priority SM does the trap generation (but it doesn't realize that
>  it's own port is a new port and thus doesn't generate a trap for it).
> 
>  Our host stack executes some functions when a port is restored  (in our
>  in-service subscription handler).
>  Am I not supposed to receive an in-service trap for a restored port that
>  happens to be the Master SM,

Yes, I guess you are.

>  and instead  execute these actions with a
>  client reregistration event?

Client reregistration request is not suitable here - SM can ask for
client reregistration at any time (in practice OpenSM now does it only
when enters MASTER state, but it is also optional).

Sasha

> 
>  Thanks again for your help!
>  Lan
> 
> 
> 
>  On 7/25/07, Sasha Khapyorsky <sashak at voltaire.com> wrote:
> >
> > Hi Lan,
> >
> > On 09:57 Wed 25 Jul     , lbt wrote:
> > >  Hello,
> > >
> > >  I have been seeing a problem where a subscriber for in-service traps is
> > not
> > >  getting informed when the port of master openSM is restored (i.e.
> > causing an
> > >  SM migration).
> > >
> > >  I have an IB subnet with 2 nodes running OpenSM , different priorities
> > of
> > >  course (OpenSM Rev:openib-2.0.5). I also have another node on the
> > subnet
> > >  that has subscribed for the forwarding of any
> > IB_SA_GENERIC_TRAP_NUM_IN_SVC
> > >  trap events. I've been doing cable pull tests on the IB ports, to check
> > if
> > >  the in-service handler I have subscribed gets invoked when I restore
> > the
> > >  cable. I've noticed that everything works as expected ( i.e. my
> > in-service
> > >  handler is invoked) whenever I restore the cable on the lower priority
> > SM IB
> > >  port without ever touching the master SM port. But if I cause an SM
> > >  migration, by restoring the port of the higher priority SM, the
> > in-service
> > >  trap does not get generated as expected on a cable restore.
> > >
> > >  Steps to Reproduce:
> > >  1) Start with port to higher priority SM disconnected.
> > >  2) restore port cable on the higher priority SM
> > >  --> This causes an SM Migration as expected, SM's migration happens
> > okay
> > >  --> I expected the restoration of the higher priority SM to tit to also
> > >  trigger an in-service trap as well and notify subscribers, but it
> > doesn't
> > >  occur
> > >
> > >  I have collected debug messages log for both open SM's, and it appears
> > that
> > >  the reason is because:
> > >  1) in-service traps are generated based on what ports are added on the
> > >  Master SM's new_ports_list, but these traps are generated only after
> > LID
> > >  assignment
> > >  2) when the higher priority SM port is restored, the restored port gets
> > >  added to the lower priority SM's new_ports_list (since it's still the
> > Master
> > >  SM at that point in time)
> > >  3) the handover of Master  SM  from lower priority to higher priority
> > SM
> > >  occurs (before LID assignment and thus a chance for traps get generated
> > for
> > >  those ports on new_ports_list)
> > >  4) the higher priority SM is now Master SM, but it has an empty
> > >  new_ports_list, so no trap generated either
> > >
> > >  Does this look like a legitimate Open SM bug? Any feedback would be
> > much
> > >  appreciated, and if I can help further in any way please let me know .
> >
> > As far as I know when OpenSM (even old like 2.0.5) becomes master it
> > requests client to reregister SA related stuff (by setting this bit in
> > PortInfo).
> >
> > Probably your port doesn't not support this (you could verify by seeing
> > PortInfo:CapabilityMask - use 'smpquery portinfo <client-port-lid>') or
> > maybe your host stack doesn't do reregistration?
> >
> > Anyway you could track this in the OpenSM code in osm_lid_mgr.c
> > __osm_lid_mgr_set_physp_pi() whenever client reregistration bit is set
> > (with ib_port_info_set_client_rereg()) or not. Then we will know more
> > about this problem.
> >
> > Sasha
> >
> > >
> > >
> > >  Subset of logs from lower priority SM during the cable restore of
> > higher
> > >  priority SM port:
> > >  ### Jul 18 14:31:56 614522 [41401960] ->
> > __osm_trap_rcv_process_request:
> > >  Received Generic Notice type:0x03 num:128 Producer:2 from LID:0x000A
> > >  TID:0x00000016000012e1
> > >  ### Jul 18 14:31:56 614823 [41401960] -> osm_state_mgr_process:
> > Received
> > >  signal OSM_SIGNAL_SWEEP in state OSM_SM_STATE_IDLE
> > >  ### 14:31:56 ******************** INITIATING HEAVY SWEEP
> > >  **********************
> > >  ### Jul 18 14:31:56 616887 [42803960] -> osm_state_mgr_process:
> > Received
> > >  signal OSM_SIGNAL_NO_PENDING_TRANSACTIONS in state
> > >  OSM_SM_STATE_SWEEP_HEAVY_SELF
> > >  Jul 18 14:31:56 626078 [42803960] -> __osm_ni_rcv_process_new: Adding
> > port
> > >  GUID:0x00504501483e0000 to new_ports_list
> > >  Jul 18 14:31:56 626524 [42803960] -> osm_state_mgr_process: Received
> > signal
> > >  OSM_SIGNAL_CHANGE_DETECTED in state OSM_SM_STATE_SWEEP_HEAVY_SUBNET
> > >  Jul 18 14:31:56 632630 [41E02960] -> osm_state_mgr_process: Received
> > signal
> > >  OSM_SIGNAL_NO_PENDING_TRANSACTIONS in state
> > OSM_SM_STATE_SWEEP_HEAVY_SUBNET
> > >  14:31:56 ********************* HEAVY SWEEP COMPLETE
> > ***********************
> > >  Jul 18 14:31:56 632773 [41E02960] -> osm_sm_state_mgr_process: Received
> > >  signal OSM_SM_SIGNAL_HANDOVER_SENT in state IB_SMINFO_STATE_MASTER###
> > >  14:31:56 ******************** ENTERING SM STANDBY STATE
> > *******************
> > >
> > >  Subset of logs from higher priority SM during the cable restore of
> > higher
> > >  priority SM port:
> > >
> > >  Jul 18 14:32:02 995600 [41401960] -> osm_sm_state_mgr_process: [
> > >  Jul 18 14:32:02 995605 [41401960] -> osm_sm_state_mgr_process: Received
> > >  signal OSM_SM_SIGNAL_DISCOVERY_COMPLETED in state
> > >  IB_SMINFO_STATE_DISCOVERING
> > >  Jul 18 14:32:02 995609 [41401960] -> Entering MASTER state
> > >  Jul 18 14:32:02 995888 [41401960] -> __osm_sm_state_mgr_master_msg:
> > >  ******************** ENTERING SM MASTER STATE ********************
> > >  Jul 18 14:32:03 009014 [41401960] ->
> > __osm_state_mgr_set_sm_lid_done_msg:
> > >  **** SM LID ASSIGNMENT COMPLETE - STARTING SUBNET LID CONFIG *****
> > >  Jul 18 14:32:03 024047 [41E02960] -> __osm_state_mgr_lid_assign_msg
> > >  ***** LID ASSIGNMENT COMPLETE - STARTING SWITCH TABLE CONFIG *****
> > >  Jul 18 14:32:03 024052 [41E02960] -> __osm_state_mgr_report_new_ports:
> > [
> > >  ----> no in-service traps are generated and notices forwarded because
> > there
> > >  are no ports on this list
> > >  Jul 18 14:32:03 024057 [41E02960] -> __osm_state_mgr_report_new_ports:
> > ]
> > >
> > >
> > >  Thanks!
> > >  Lan
> >
> > > _______________________________________________
> > > general mailing list
> > > general at lists.openfabrics.org
> > > http://lists.openfabrics.org/cgi-bin/mailman/listinfo/general
> > >
> > > To unsubscribe, please visit
> > http://openib.org/mailman/listinfo/openib-general
> >



More information about the general mailing list