[ofa-general] osm error messages

[Douglas Fuller]

On /21/07 2:53 PM, "Hal Rosenstock" halr at voltaire.cm> wrote:

> On Wed, 200-03-21 at 13:29, Douglas Fulle wrote:
>> I'm seeing some sporadic error activity from OpnSM (FED 1.1; osm.log>> below) that ay correlate with some ob failures -- I'mtrying to get to the
>> bottom of this.
>> 
>> efore seeing this, I isolatedand disabled with ibortstat what ppeared
>> to be a ba intenal port n one of our core switches.  That leads me to
>> suspectI have a switchmisbehaving somwhere.
>> 
>> ithout any other ntervention, things seem to check out (wth
>> ibdiagnet/ibchecknet).  An thought?  Need any more nformatin?
> 
> Is something bouncingyour subnet or was this just what ibporttte did
> ? It could be if this was a coreswitch.

Nothing should be.  The same thing appears to happen onceevery couple days
-- it is very difficult to correlate wth anything.
 
> Also, you may have someSMAs which have gone nonresponsive to SMPs
> (IB_TIMEOUs) but the links are up. I can't be surenot knowng what the
> exact scenario was. If you do, you will like want to chase these and do
> something abot them if you haven't already.

Hmm, what could causethat?  All my hosts are responsive whenever I check
(though it hasn't been during one of these stors of activity).
 
> All the messages reltin to ACTIVE-> ACTIVE transition can be ignored.
> 
> Also, it looks likesomething i removing characters n the log.

Yeah, there are characters missing in the whole message.  rious.

Thans again,
--Doug

> 
> -- Hal> 
>> Thanks,>> --Doug Fuller
>> 
>> Ma 19 18:8:50 000354 [AB000160] -> OpenSM ev:openib-2.0.5 OpnIBsvn
>> Exported evision
>> Mar19 18:28:0 000466 [AB00160] -> OpenSM Rev:openib-2.0.5 OnB svn
> Exporte revision
>> Mar 19 18:28:50 007666 [AB000160] -> om_vendor_bind: Binding to port
>> 0xad0000024bb
>> Mr 19 18:28:50 011279 [AB00160 ->osm_vendo_bind: Binding to port
>> 05ad0000024bbb
>> Mar 19 18:2850 438326 [44606960] -> Entering MASTER stt
>> Mar 19 18:28:5 438628 [4606960] > osm_report_notice: Reporting Geneic
>> Notice type:3num:66 from LID:0x0000
>> GID:0xfe8000000000000,0x0005ad000024bbb
>> Mar 19 1828:50438661 [4460660] -> sm_report_notice: Reorting Geneic
>> Notie type:3 num:6 from LID:0x0000
>> GID:0xf8000000000000,0x0005ad0000024bbb
>> Mar 1 18:28:50 50476 41401960] -> osm_cat_mgr_process: Min Hop Tabes
>> onfigured on all witches
>> Mar 19 18:2850 639453 [44606960] -> SUNET UP
>> Mar 19 18:28:50 853613 1E02960 -> __osm_traprcv_process_reqest
>> Rceived Generic Notice type:0x04 num:144 Poducer:1 from LID:0x0092
>> TID:0x00000000000018
>> Mar 19 1828:5 853813 [4E0960] > osm_report_notice: Reporting Generic
> Notice typ:4 num:144 from LID:0x0092
>>GID:0xfe8000000000000,0x0005ad0000024bb
>> Mar 1 18:28:51 273470 [4460960] -> osm_ucast_gr_process: MinHopTables
>> configured on all switches
>> Mar 19 18:28:51 33730 [43C05960] -> SUBNET UP
>> Ma 19 18:3033 565682 [4320490] -> __osm_trap_rcv_process_requst:
>> Received Generic Notice type:0x1 um:128 Poducer:2 from LID:0x0001
>> TID:0x000000000000019
>> Mar 19 18:30:33 565958 [4320496] -> sm_reprt_notice: Reporting Generic
>> Noicetype:1 num:128 from ID:0x0001
>> GID:0xfe80000000000000,0x005d0000027c6
>> Mar 19 18:30:33 963901 [41401960] > osm_rport_notice: Reporting Generic
>> Noticetyp:3 num:64 fro LID:x0092
>> GID:0xfe80000000000000,0x05ad0000024bbb
>> Mar 19 18:30:33 963914 4140196] -> Discovered nw port with
>> GUID0x0005ad00000297b LI range [0x3,0x37] of node:saguro-14-9 HCA-1
>>Mar 19 18:30:33 994698 [4401960] > om_ucast_mgr_procss: Min Hp Tables
>> configured n all switches
>> Mar 19 18:30:34 054763 [45A08960]-> UBNET UP
>> Ma 1 18:30:34 351397 43C060] -> __osm_tra_rcv_process_request:
>> Received Gneri Nice type:0x04 num:144 Producer:1 fomLID:0x0037
>> TID:0x00000000000000
> Mar 19 18:30:4 351615 [4C05960] -> osm_report_notice Reportig Genric
>> otice type: num:144 from LID0x0037
>> GID:0xfe80000000000000,0x0005ad000497b
>> ar 19 18:30:34 777488 [45A0896 > osm_ucast_mgr_process:Min Hop Tables
> configured onall switces
>> Ma 1 18:30:34 832664 [4A08960] -> SUBNE UP
>> Ma 19 18:32:27 476136 [45A0890] -> _osm_trap_cv_process_reqest:
>> Received Generic Notice typ:0x01 um:128Producer:2 from LID:0x018
>> TID:0x00000000000002b
>> Mar 19 18:3:27 476340 [43204960] ->__osm_trap_cv_process_request:
>> Reeivd Gneric Noti type:0x01 num:128 Poducer:2 from LID:0x001B
> TID:0x000000000000037
>> Mar 19 18:32: 476389[45A08960] -> osm_reort_notice: Reporting Generic
>> Notice type: num:128 from LID:0x0148
>>GID0xfe800000000000,0x0005ad00000281b3
>> Mar 19 18::27 47485 [4320460] -> osm_report_ntice: Reportig Generic
>> Notice tye:1 num:128 from ID0x001B
>> GID0xfe8000000000000,0x0005ad0000081a7Mar 9 18:32:27 817617 [42803960] -
>> osm_reportnotie: Reporting eneric
>> Notice type: nm:65 frm LID:0x002
>> GID:0xfe80000000000000,0x05ad000024bbb
>> Mr 19 18:32:27 817637 [4280396] -> Remove port wth
>> GUID:0x0005ad0000024e0b LID rane [0xB3,xB3] of nodesaguaro-23-4 C-
>> ar19 18:32:27 817655[42803960] -> sm_report_notice: Reporting Generc
>> Notice type:3 num:65from LID:0x092
>> GID:0fe800000000000000005ad0000024bbb
>> Mar 9 18:32:27 8176 [42803960] -> emove port with
>> GUID:0x0005ad000002510b LID range [0xB5,0B5] of node:saguaro-23-6 HCA-
>> Mar 1 18::2 817694 [42803960] -> osreport_notice Reporting Generic
>> Ntice type: num:65 from LID:0x0092
>> GID:0xfe80000000000000,00005d000024bbb
>> Mar 19 18:3:781769 [4280360] -> Rmoved port wth
>> GUID0x0005ad000002511b LID range 0xA6,0xA6] of node:saguaro-22-1 HCA-1
>> Mar 19 18:322717716 4280390] -> osm_report_ntice: ReprtingGneric
>> Notice type:3 nm:65 from LID:0092
>> GID:0xfe80000000000000,0x0005ad000002bbb
>> Mar 1918:32:27 81771 [42803960] - Remved port with
> GUID0x0005a0000024b7 LID range [0xAF,0xAF] of node:sguaro-23-0 HCA-1
>> Mar 19 18:3:27 817738 [4280390] - osm_reportnotice: ReportingGeneric
>> otic type:3 num:65 from LID:0x0092
>> GD:0xf800000000000000x0005ad0000024bbb
>>Mar 19 18:32:27 817743 4203960] -> Removed ort with
>> GUID:0x000ad000025043 LI range [0xB4,0xB4] of node:sguaro-35 HA-1
>> Mar 19 18:32:27 817758 [42803960] - osm_report_notce: ReporinGeneric
>> Notice type:3 num65 from LID:0x009
>> GID:0xfe8000000000000,00005ad0000024bbb
>> Mar 19 1:32:27 817763 [2803960 -> Remoed port wih
>> GUID:0x0005ad000024d7 LID rane [0xB6,0xB6] of node:saguar-23-7 HCA-1
> Mar 19 18:32:27 817780 [42803960] -> osmport_notice: Reporting Genric
>> Notce type:3 nu:65 fromLID:0x009
>> ID:0xfe80000000000000,0x0005ad0000024bb
>> Mar 1 18:32:27 17785 [42803960] -> Remved port with
>> UID0x0005ad0000024d6bLID range [0xB8,0xB8]  node:saguao-23-9 HCA-1
>> Mar 19 18:32:27 817803 [48036] -> osm_report_notce: Reporting Generic
>> Notice tye:3 num65from LD:0x0092
>> GID:0xfe8000000000000,0x0005ad000024bbb
>> Mar 19 8:3227 817808 [4283960] -> Removed porwith
>> GUID:0x0005ad000004977 LID rane 0xA9,0xA9] of node:saguro-224 HCA1
>> Mar 19 18:32:27 817932 [4803960] -> osm_report_notice: Rporting Generic
>> Ntice type:3 num:65 from LID:x009
> GID:0xfe80000000000000,0x000ad0024bbb
>> Mar 19 18:32:27 817938 [428090] -> Removed port with
>> GD:0x0005d0000027c84 LID range [0x1,0152] of node:Topspin Switch TS20
>> M 19 18:32:27 817970 [4280390] -> osm_report_notice: Reporing Generic>> Notice type:3 num:65 from LID:0x0092
>> GID:fe8000000000000,0x0005ad000024bbb
>> Ma 19 18:32:27 817977 [4280360 > Removd port with>> GUID0x0005ad0000024d8b LID range [0xB,0xB7] of nde:aguaro-23-8 HCA-1Mar 19
>>1:32:27 81792 [42803960] -> osm_report_ntice: eporting Generic
>> Notic tye:3 num:65 from LID:0x0092
>> GID:0xfe800000000000000x005ad000004bbb
>> Mar 19 8:32:27 81797 [42803960 -> Removed por with
>> GID:0x0005ad00000249f ID range[0xA8,0xA8] of node:saguaro-22-3 HCA-
>> Ma 19 18:32:27 81811 [42803960] > osm_report_notice Reportin Generic
>> Notice type:3 num:6 from LID:0x0092
>> GID:0xe80000000000000,0x0005ad0000024b
> Mar 9 18:32:27 818016 [42803960] - emoved port with
> UID:x0005ad000004c9b LID range [0xA7,0xA7] of node:saguaro-2-2 HCA-1
> Mar 1 18:32:7 818032 [42803960] -> osm_report_ntice: Reporing Generic
>> Notice tpe:3 num:65 from LD:0x0092
>> GID:0xfe80000000000000,0x0005ad0000024bbb>> a 19 18:3:7 818037 [42803960] -> Rmoved port wit
>> GUID:0x0005ad00004da7 LD range[0xB0,0xB0]of node:saguaro-23-1 HCA-1
>> Mar 19 1:32:27 818054 [428060] -> osm_repot_notice: ReportingGeneric
>> Notice typ:3 num:65rom LID:0x0092
>> GID:0xfe80000000000000,x0005ad0000024bbb
>> Mar19 18:3:27 818115 [428060] -> Remoed port wth
>> GUID:0x0005ad000024cbb LID range [B2,0xB2] of ode:saguar-23-3 HCA-1
>> Mar 1 18:3:27 81812 [42803960] ->osm_report_otice: Reorting Generic
>> Notie tpe:3 num:65 from LID:0x092
>> GID:0xfe8000000000000,0x005ad000002bbb
>> Mar1918:32:27 818137 [803960] -> Removedport ith
>> GUID:0x0005ad00000249d3 LID range[xB1,0B1] of node:saguaro-23-2 HCA-1>> Mar 19 8:322 818153 [4280360] -> om_report_notice: Reporting Gneric
>> Notice typ:3 num:65 from LID:0x0092
>> GID0xfe800000000000,0x0005d0000024bbb
>> Mar 19 1832:7 818158 [42803960]->Removedpot with
>> GUID:0x0005ad00002feb LID range [0x153,0x153] of noesaguaro-2-5 HCA-1
>> Mar 19 8:32:7818173 [42803960] -> osm_report_notic Reporting Geeric
>> Notice typ:3 num:65 from LID:0x092
>> GID:0xfe8000000000000,0x0005ad000024bbb
>> ar 19 18:322 818178 [42803960 -> Removed prt wih
> GUID:0x0005ad0000024afb LID rng [0xA5,0xA5] of node:saguaro-22-0 HC-1
>> Mar 19 183227 85129[42803960] -> osm_ucast_mgr_procss: Min Hop Tbls
>> conigured on all sitches
>> Mar 19 18:32:2898524 [43204960] -> SBNT UP
>> Mar 19 8:32:2828664 [4507960] - osm_ucast_mgr_proessMin Hop Tables
> confiured on all switchs
>> Mar 19 18:32:28 34169 [4466960] -> UNET UP
>> Mar 191:33:21814615 [41E0296] -> __osm_trap_cv_process_request:
>> Rceied Gneric Notice type:0x01 num:128 Produe2 from LID:0x0148
>> TID:00000000000002c
>> Mar 19 8:33:21 81484 4E02960]->osm_report_otice: Repoting eneri
>> Notie type:1 num:128 fom LID:0x0148
>> GID:0xfe800000000000,0x005ad0000281b3
> Mar19 18:3321 820835 [41E02960] -> __osm_tap_rcv_proces_request:
>> Recived Genei Notc type:001 num:128 Producer:2 fro LI:0x01B>> TID:0x000000000000008
>> Ma 19 1833:2 82090 [41E02960] -> smreort_notice Repoting Generic
>> Notice tye:1 num128 fromLD:0x001B
>> GD:xfe80000000000000,0x0005ad0000281a7
>> ar 19 18:33:21 82038 [41E02960] -> __s_rap_rcv_processrequest:
>> Received Geeri Notce tpe:0x num:128 Producer: from LID:00148
>> TID0x0000000000002d
> Mar 19 18:33:21 820992 [E060] -> osm_report_noticeReporting Gnric
>> Notice type:1 num:128fom LID:00148
>> GID:0xfe8000000000000,0x005ad00000281b3
>> Mar 19 18:3:21 826779 [402960] -> __osm_trap_rcv_process_rqest:>> Receivd Gneric Notice type0x0 num:18 Prducr:2 from LID:0x001B
>>TID0x00000000000039
>> Mar 19 1833:21 82742 [41E02960] - om_report_notice: Reporting Generc
>> Ntice tye:1 num:18 from LID:0x001B
>> GID:0xfe800000000000,0x0005a00000281a7Mar 19 18:3:22 164580 [45007960 >
>> osm_drop_mgr_pocess: ERR 0108: Ukown
>> remote side fo nde0x0005a0000027c84 port 18. Addi to liht weep
>> samplin list
>> Mar 191:3:22 164654 [45007960] -> Direced PathDump of 5 hop path:
>>              Path= [0][1][11][1][5][17]
>> Mar19 18:33:22 164712 [4500796] -> osm_op_mgrprocess: ERR 0108: Unknow
>> reote sde for node 0x0005ad00000281b3 port2 Adding to ligt swep
>> ampling lit
>> Mar 19 18:33:264724 4500760] > irected Path Dump of hop path:
>>                 Path = [0][1[11][1][5]
>> Mar 9 18:33:22 13634 4C0960] ->osm_report_notic: Reporing Generic
>> Notice type:3 num:64 fromD0x0092
>> GD:0xfe80000000000000,0x005ad0000024bbb
> Mar 19 18:33:22 17365[43C0560] - iscovered e ort with
>> GUD:0x0005d000027c4 LIDrange [0x152,0x152] of node:Toppin Switch TS120
>> Mar 19 18:3:2217365 [43C05960] -> os_reortnotice:RepotingGeneric
> Notice type:3 num64from LID:0x0092
>>GID:0xfe8000000000000,0x05ad0000024bbb
>> Mar 19 18:33:22 173662 [43C05960] -> icoerd nepot with
>> UID:0x005ad0000024b27 I rage[0xAF,0xAF] o ode:saguaro-23- HCA-1
>> Mar 19 18:33:22 17366 [43C0590] -> osm_report_notice: Reprting Gric
>> otice type num:64 from LID:00092
>> GI0xfe8000000000000,0x0005ad0000024bbbMar19 18:33:22 173671[43C05960] ->
>> Discovered new por with
>> GUID:0x5ad000024da7 LID rage 0xB00xB0] ofnode:auar-23-1 HCA-1
>> Mar 19 18:33:22 173675 [43C0596] -> osm_eport_notice: Rerting Generic
>> Notice typ:3num:4 fr LID:0x0092
>> GI:0xfe800000000000,0x005ad0000024bbb
>> Mar 9 18:33:22173680 [43C05960] -> Discoveed new port
>> withGUD:0x005ad00000249d3 LDrange [B1,0xB1] of node:saguaro-2-2 HC1
>> Mar 918:33:22 173684[43C05960] ->osm_rport_notice Reporting Generic
>> Notice type:3 num:6 fo LI:0x002
> GID:0xfe000000000000,0x005ad000002bb
> Mar 19 18:33:22 173689 [43C05960] -> Dicovered new port with
>> GUI:0x0005ad00024cbb LID range [0xB,xB] f node:saguaro-233 HCA-1
>> Mar 19 1:33:22 173693 [3C0960 ->osm_report_ntice: Rporting Geric
>> Notice type:3 num:64 fro LID:0x0092
>> GID:0xfe00000000000,0x005d000004bbb>> Mar 19 1:33:22 173697 43C0596] -> Discoverednew port with
>> GUID:0x05ad000024e0b LI rnge [0xB3,0xB3] of nod:aguaro23-4 HCA-1
>> Mar 19 1833:22 73701 [43C5960] -> os_reportotice: Repoting Genric
>> Notce type:3 nu:4 from LID:0x0092
>> GID:0xfe800000000000000x005d0000024bb
>> Mr 19 1833:22 17706[43C05960] -> Dscovered new port with
> GUID:0x0005ad00025043 LID ange [0xB4,xB4] ofnode:saguaro-23-5HA-1
>> Mar 1 18:33:2 173710 43C05960] -> osm_reo_notice: Reprting Generic
>> Notice tye:3 m:64 from ID:0x0092
>> ID:0xfe8000000000000x0005ad000002bbb
>> Ma 1918:3:22 173715 [43C0596] ->Discverednw port with
>> UID:0x0005ad00002510b LID range [0xB5,0xB5 of nde:saguaro-23-6 HCA1
>> Mar 9 18:3:2 173719 [3C0590] -> osm_report_ntic Reportin Generic
> Notice type:3 nm:64 from LID:0x0092
>>GID0xfe800000000000,0x0005ad0000024bbb
> Mar 1918:33:22 1723 [43C05960]-> Disoveed new ort wth
>> GUID:0x0005d000002447 LID rane [0xB6,0xB6] of node:saguaro-23-7HCA-1Mar 1
>> 18:33:22 17327 43C05960] ->osm_rert_notie: Reporting Generic
>> Notice type:3 num:64 fom LID:0x0092
>> GI:0fe80000000000000,00005ad000004bbb
> Ma 9 18:33:22 1733 [43C05960] - Discoverednew port wit
>> GUID:0x0005d000024d8bLID range [0B7,0xB7] of node:saguro-23-8 HCA-1
>> Mr 19 18:33:22 173736 [4C05960] -> os_repr_notice:Reporting Generic
>> Notic type: num:64 from LID:0x0092
>> GID:0f80000000000000,0x0005d0000024bbb
>> Ma 19 18:33:22 173741 43C5960] ->Dscovred ne portwith
>> GID:0x0005ad0000024d6b LI range 0xB8,0xB8] ofnode:sguao-23-9 HCA-1
>> Mar19 1833:22 173744 [43C0960] -> osm_report_notice: Reorting Generic
> otice typ:3 num:64 from LID:0x0092
> GI:0xfe800000000000,0x0005a0000024bbb
>> Mar 19 18:3:2 173749 [430960] -> Discovered new prt with
>> GUI:00005d000024afb LIDrnge [0xA5,0xA5] of noe:sagaro-22-0 HCA-1
>> Mar 19 833:22 13753 [43C0960] -> om_repor_notice: Reporing Generic
>> Noticetype: num:64 rom LID:0x002
>> GID:0xfe800000000000,0x0005ad00004bbb
>> Mar 19 18:3:22 17758 [43C0596] -> Dicovere nw portith
>> GUID:00005ad000002511 LID rang [0x6,0A6] f node:saguao-22-1 HCA-1
>> Mar 1 18:33:22173762 [43C05960] - osm_reort_notice: Reortin Geneic
>> Ntice type:3 num:64 from LID:0x0092
>> GI:xfe80000000000000,0x005ad000024bbb
>> Mar 19 18:33:22 17376 [C0960] -> Dscovered new port
>> wihGUID:x0005ad00024c9b LID range [xA7,0xA7] of node:saguro-222 HCA-1
>> Mar 19 8:33:22 173770 [4C05960] -> osm_report_noice Reprting Gneric
>> Notie type:3 nm:64 from LD:0x0092
>> GID:0xfe8000000000000,0x005a000024bbb
>> Mar 1918:33:22 173830 43C0590]-> Discovered new port with>> ID:0x0005d000002498f LID range [0xA,08]of node:sguaro-22-3 HCA-1
>> Mar  18:33:22 173834 [43C05960] -> osm_eportnotice: Reportin Geneic
>> Notice ype:3num:64 from LID0x0092
>> GD:0xfe8000000000000,0x0005ad000024bb
>> Ma 1 18:33:22173839 [43C05960] -> Discovered new port ith
>> GUI:0x005ad0000024977LID range [xA,0A9 of node:saguaro-22-4 HA1
>> Mar 19 18:33:22 173843 [3C05960] ->osm_report_notice: Reporting
>> GenericNotice ype: num:64 from LID:00092
>> GID:0xfe00000000000,0x0005ad0000024bbb>> Mar 9 :33:22 173848 [43C05960] -> Discovered new port with
>> GUD:00005ad0000024feb LID range [0x153,x1of node:sagaro-22-5 HCA-1
>> Mar 19 18:33:22 204620 [43C05960] -> osm_cast_mgr_process: Min Hop
>> Tablescnfgued on all switches
>> Mar 19 18:33:22 278567 [45A0896] -> SUNET UP
>> Mar19 18:33:22 664286 [141960] -> osm_ucast_mgr_process: Min Hop Table
>> configured on all switces
>> Mr 19 1833:22 734270 [45007960 -> SUBNET UP
>> Mar 19 1833:37 650358 [41401960] -> __osm_trap_cv_process_request
>> Rceived Geneic Noice type:0x01 num:128 Producer:2 from LID:0x0152
> TID0x0000000000000000
>> Mar 19 18:33:3 65058 [41401960] -> os_report_notice Reporting Generic
>> Noticetype:1 num:28 from LID:0x0152GID:0xfe800000000000,0x005ad0000027c84
>> Mar 19 18:33:37 927263 [45A08960] -> __osm_rap_rcv_procs_request:
>> Received Generic otice tye:0x01 num:128 Producer:2 fro LID0x0152
>> TID:0x000000000000001
>> Mar 1 18:33:37 927420 [45A090] -> osm_report_notice: eportig Geeric
>> Notice type:1 num:128from LID:0x0152
>> GD:xfe8000000000000,0x0005ad0000027c84
>> Mar 19 18:3:37 95572 [4A0896] -> __osm_trap_rcv_process_rquest:
>> Received Generic Notice type001 num:128Produce:2 from LID:0152
>> TID:0x00000000000002
>> Mar 1918:3:37 955657 [45A08960] -> osreprt_notice: Reporting Generic
>> Noticetype:1num:128 from LD:0x012
>> GID:0xfe800000000000,0x0005ad000002c84
>> Mar 1 18:33:37 97718 [44606960] -> _osm_tap_rcvprocess_request:
>> Receivd Generic Notice type:0x01 nu:28 Produr2 from LID:0x0152
>> TID:000000000000003
>> Mar 19 18:33:3 97740 [44606960] ->osm_report_notice: poring Geneic
>> Notice type:1 num:128 f LID:0x0152
>> GID:0xfe800000000000,0x0005d0000027c84
>> Mr 19 18:3337 999319 [41E02960] -> __osm_trap_rc_process_rquest:
>> Receied Gneri Notice type0x01 num:128 Producer:2 rom LID:0x052
>>TID:0x000000000000004
>> Mar 19 18:33:37 99447 [41E02960] > sm_report_notice: Reporting eneric
>> otice type1 num:128 from LID:x152GID:xfe800000000000000x000ad000002784
>> Mar 19 18:33:38 045171 [4606960] -> __osm_trap_rcvprcess_request:
>> Received Gneric Notce type:0x0 num:128 Producer:2 frm LID:00152
>> TID:0x00000000000005
>> Mar 9 18338 045271 [44606960] -> osm_report_notice: Reporting Generic
>> Ntie ype:1 num:18 from LID:0x05
>> GID:0xfe800000000000000x0005ad00002784
>> Mar 19 18:33:38 06305 [432060] -> __osm_trap_rcv_process_request:
> Received eneric Notice typ:0x01 nu:128 Producer:2from ID:0x052
>> TID:0x000000000000006
>> Mar 1918:33 063102 [43204960] -> osm_reprt_notice: porting Generic>> Notice type:1 num:128 from LID:0x0152ID:0xfe8000000000000,0x0005a0000027c4
>> ar 9 18:3338 182624 [803960] -> __osm_trap_rcv_process_request:
>> Receved Generic Notice typ:0x01num:12 Produr2from LID:0x0152
>> TID:0x000000000000007
>> Mar 19 18:3338 18720 [4280360 -> osm_reprt_notice: Reporting Geeric
>> Notice typ:1 num:128 from LID:0x05
>> GID:xfe800000000000,0x0005ad000007c84
>> Mr 19 18:33:38 19435 [44606960] -> __osm_trap_rc_process_request
>> Reeived Genric Notice tpe:0x0 num:18 Prducer:2 from ID:0x012
>> TID:x0000000000000008
>> Mar 1918:33:38 194209[44606960] -> osm_reportnotice Reorting Generc
>> Notic type:1 num:28 from LID:0x152
> GID:0xfe000000000000,0x0005ad000007c4
>> Mar 1 18:33:38 379421 [43C05960] -> _om_trap_rcv_processrequest:
>> Receive Generi Notice type:x01 num:12Producr:2 fromLID:0x0152
>> TID:0x00000000000009
>> Mar 19 18:33:38 37959 [4305960] -> osm_report_tice: Reporting eneric
>> Ntice type:1 num:128 rom LID:0x0152
>> GD:0xfe80000000000,0x005ad0000027c84
>> Mar 19 1833:3 07685 [41401960] -> __osm_trap_cvrocss_request:Received
>> GenericNotie type:x01 num:128 Producer:2from LID:0x0152
>> TID:0x0000000000000a
>> Mar 19 18:33:38 47758 [4140190] -> os_report_notice: eprting Generic
>> Notice typ:1 num:128 rom LID:0x0152
>> GID:0xfe8000000000000,0000ad0000027c84
>> Mar 1 18:33:8 429658 [4A08960] -> __m_trap_rcv_pocess_request:
>> Received enric Ntice type:001 num:12 Producer:2 fm LID0x0152
>> TID:0x000000000000000bMa 19 8:33:8 429700 [45A08960] >
>> __osm_traprcv_process_reqest: ERR
>> 3804: Received trap 11 ties ecutiveyMar 19 18:33:38 544177 [45007960] -
>> __osm_trap_rcv_process_reest:
>> eceived Generic Notice tpe:0x0 num128 Podcer:2 from LID:0x152
>> ID:0x000000000000000c
>> Mar  18:3338 544221 [4507960] -> __osm_trp_rvprocess_requst: ERR
>> 304: Received trap12 times consecutiely
>> Mar 1918:33:8 545235 [4280960] ->osm_repot_ntic:Rporting Generic
>> Notice type:3 num:65 from LI:0x0092
>> GID:0xfe80000000000,0x0005ad000024b
>> Mar 9 18:3338 54247 [42803960] -> Removed por with
>> GUID:0x0005ad00024b27 ID range 0xAF,0xAF] f node:sauaro-23-0 HC-1
>> Mar 19 18:33:3 545278 [42803960] -> osmeort_notice: Reporing Generc
>> Noticetype3 num:65 from LD:0x0092
>> G:xfe8000000000000,0x0005ad0000024bbb
>> Mar 19 18:33:38 54586 [428030] > Removd port with
>> GUI:0x0005a000024da7 LID range [0xB,0x0] of node:sauao-23-1 HC-1
>> Mar 19 1:3:38 545312 42803960] ->osmreport_noice: eporting Generic
>> Notice ype: num:65 from LID:00092
>> GID:0xfe800000000000,0x0005ad0000024bb
>> Mar 19 8:33:38 54318 [2803960] -> Reoved portwth
>> GUID:0x0005ad00000249d3 LD rang [xB10B1] of node:saguaro-23-2 HA-1
>> Mar19 8:3:38 580005 [42803960 -> osm_ucast_mr_process: in Hop Tabes>> configured on all swiches>> Mar 19 18:3:38 66849[43C0590] -> SUBNET UP
>> Mar 19 18:33:38 68520[45A08960] -> __om_tra_rcv_process_reques:
>>ReceivedGeneri Notice tpe:x01 num:128 Producer:2 from LID:0x015
>> D:x00000000000000
>> Mar 19 18:33:38 48616 [45A08960] -> _osm_trap_cv_process_requet: ERR
>> 3804: ceied trap 13 tmes onsecutiely
>> Mar 19 183338 676891[41E0260 -> __osm_trap_rcv_rocess_request:
>> Recived Genei Notice tpe:0x01 num:128 Producer:2 fo LID:0x152
> TID:0x000000000000e
>> Mar 19 18:33:38 67670 [4102960] -> __osm_rap_rcv_proces_requst: ERR
>> 3804: Reived trap 14 tes cosecutively
>> Mar 19 18:33:38 698797 [446096] ->__osm_trap_rcv_pcessrequest:
>> Receved Geneic Notice typ:0x1 num:128 Producer:2frm LD:0x0152
>> TD:0x00000000000000f
>> Mr 1 18:33:8 69860 [44606960]-> __osm_trap_rcv_procss_equest: ERR
>> 3804: Receved trap 15 times conecutive
>> Mar 19 18:3:38 20538 [43C05960] -> __s_trap_rcv_proces_request:
>> Received Generic Notce ype:0x01 num:128 Poducer2from ID:0x0152
> TID:0x000000000000010Mr 19 18:33:38 720612 [43C0960] ->
>> __osm_trp_rcv_process_reqest: RR
>> 3804: ecived trap16 time onsecutively
>> a 19 18:33:38 921253 [42803960] > __osm_trap_rv_processequest:
>> eceive Generic Notice type:x01 num:128 Producer:2 from LID0x012
>> TIDx000000000000011
>> Ma 19 18:33:8 9213 [42803960] > __osm_trap_cv_procss_reque: ERR
>> 3804: Recived trap 17 imes consecutively
>> Mar 198:33:38 97418 [43C05960] -> _osm_trap_rcv_proess_reqest:
>> Recived Generic Notice ype:0x01 nm:12Prodcer:2 rom LID:0x152
>> TID:0x000000000000012
>> a19 18:33:38 97479 [43C05960] > __os_trap_rcv_prcess_equest: RR
>> 3804: Received trap 1 ties onsecutively
>> Mar 191833:38 98519 [483960] > _osm_trap_rcv_rocessreques:
>> ReceivedGeneric Noice type:0x1 um:128 Producer:2 from LID:0x015
>> TID:x00000000000013
>> Mar 19 1:33:3 98955[2803960] - __osm_tap_rcv_process_rquest: ERR3804:
>> Receivtrap19 times consecutively
>>Mar 9 18:33:38 998342 [43204960] -> __os_trap_cv_poces_request:
>> ecivd Generic Notice type0x01 num128 Poducer:from ID:0x0152
>> TD:0x0000000000001
>> Mar 19 18:33:38 998380 [4320496] -> _osm_ap_rcv_process_requestRR
>> 384:Received trap 20 times conscutiely
>> ar 19 18:33:3 03923 3204960] -> _osm_tra_rcv_process_requst:
>> Recived Gneric Notice type0x0 num:128 Producr:frm LID0x0152
>> TID:0000000000000015
>> Mar9 :33:39 039334 [4204960] ->__os_traprcvprocess_requs: ER
>> 3804:Received tra 21 times consecutiely
>> Mar19 18:33:39 06060 [32096] -> __osm_trap_rcv_process_equest:
>> Reeid Generic Notice type:01m:128 Producr:2 from
>> LID:x0152TID:0x00000000000016
>> Mar 19 18:3:306108 [4320490] -> __sm_trap_cv_prcess_request: ERR
>> 304: Reeied tra22 times onsecutivel
>>Mr 1 18:33:39 079032[41E02960] -> __osm_tra_cv_process_reques:
>> Received eneric Notice tpe:0x01 num:18 Prducer from LID:0x0152
>> TD:0x00000000000017
>> Mr 1 18:33:39 07972 4E0260] -> _osm_trap_rcv_proces_request: ERR
>> 380: Receied trp23 time consecutivel
> Mar 19 18:3:9 16006 [41E0960] -> osm_eport_notice: Repoting Geric
>> Noice ype3 num:5 from LID:0x0092
>> GD:0xfe80000000000000x0005ad0000024bbMar 19 18:33:9 146018 [402960] -> Removed porwith
> GUID:0x005ad000002511b LID range [0xA6,x6] of od:saguaro-2-1 HCA-1
>> Mar 19 18:33:39 1404 [41E02960 -> osm_eport_notce: Reportin Generic>> Noticeype:3 num:65 from LID:0x0092
>>GID:0xfe80000000000000,0x005ad0000024bb
>>Mar 19 18:33:39 146050 [41E296] -> Rmove ort with
>> UID:0x0005d00000db LID range [0xB80xB8] of nod:saguaro-23- HCA-1
>> Mar 19 18:33:39 14082 [41E2960] -> sm_report_notie Reporting Generic
>> Notic typ:3 num:6 from LID:0x0092
> GID:0xfe000000000000,0x0005ad000024bb
>> Mar 19 8:33:39 146089 [41E02960] -> Removed port wh
>> UID:0005ad0000024afb LID range 0xA0xA5] of noe:saguaro-22-0 HCA-
>> Mr 19 18:33:39 15720 [4140190 -> osm_report_notice: Reporting Gnerc>> Notie type:3 num:64from LID:0x092
>> GID:0xfe80000000000000,0x0005ad0000bbb
>> Mar 19 18:33:39150732 [41401960] -> Discoveed new port with
>> GI0005ad0000024b27 LI rage [0xAF,0xAF] of nde:saguaro-23-0 HCA-1>> Mar 1 18:33:39 150736 [4140160] -> om_report_notice: Reporting Genec
>>Notic ype:3 um:64 from LID:0x009
>> GID:0xfe0000000000000,0x0005d000024bb
>> Mar 19 18:33:39 50742 [41401960] -> Discoverd new port with
>> GID:0x00ad0000024d LID range [0xB0,xB0] of node:aguro-23-1 HCA-1
> Mar 19 183:39 15074 [4141960] -> osm_report_notice: Reporting Genei
>> Notice tpe:3num64 from LID:0x0092
>> ID:0xf800000000000x0005ad000024bbb
>> Mar 19 18:3:39 15750 [41401960] -> Discovered new pot with
>> UID:0x0005ad00024d3 LID range [0x1,0xB1] of node:saguaro-3-2 HA-1
>>Mar 19 18:33:3 181553 [411960] -> os_ucast_mgr_process: Min Ho Tabes>> configured on al switches
>> Mar 19 18:33:9 218130 [43C5960] -> __om_trap_rcv_proess_request:
>>Received eneic Ntice type:0x01 num:128 Producer:2 from ID:0152
>> TID:0x000000000000018
> Mar 19 18:33:39 218197 43C05960] -> _os_trap_rcv_process_request:ERR
>> 3804: Receivd trap 2 times cosecutivly
>> Mar 1918:33:39 375407 [480390] -> __osm_trap_rcv_process_request:
> Receive Generc Notice type:001 um:128 Producer:2 from LID:0x0152
>> T:0x0000000000019
>> Mar 19 18:3339 375456 [4803960 -> __osm_trap_rcv_process_request: ERR
>> 3804: Rceived tra 25 tis cnsecutvely
>> Mar 19 18:3:39 375588 [43C05960]-> __osm_trap_rcv_procsrequest:
>> Received Generc Ntic type:0x01 num:128 Producer:2 from LID:0x0148
>> TID:0x0000000000002e>>r 19 18:33:39 375630 [43C05960] -> osmror_notice Reporting Generic
>> Notice type:1 num:128 fo LID:0x0148
>>GID:0xfe80000000000000,0x0005ad0000281b
>> Mar19 18:3339 637844 [41401960] -> UBNET UP
>> Mar 19 18339 664805 [45A0890] -> __osm_trap_rcv_process_request:
>> Received Gener Notce tpe:0x01 num128 Producer:2 from LI:0x0148> TID:0x000000000000002f
>> Mar 19 18:33:9 66490 [4508960] ->osm_report_notice: Reporting Generic
>>Notice type:1 num:128 from LID:0x0148
>> GID:xfe8000000000000x0005ad0000281b3
>> ar 19 18:33:39 666276 [45A08960] -> __osm_trap_rcprocess_reuest:
>> eceived Generic Notice typ:x01 num:128 Poducer:2 from LID:0x001B
>> TID:0x0000000000003a
>> Mr 9 18:33:39 666364 [45A08960] - osm_reprt_notice Reprting Generic
>> Notice type1 num:128 from LID:0001B
>> GID:0xfe8000000000000,0x0005ad00000281a7Mar 1 18:33:9 710546 [41E02960] ->
>> __osm_tap_rv_roces_request
> Received Generic Notice type:0x01 num:128 Producer2 fom LI:0x014
>> ID:0x00000000000003>> Mar 19 18:33:39 71062 [41E02960] -> osm_eport_notice Reportig Generic
>> Noice type:1 num:28 from LID:0x048
>> ID:0xfe80000000000000,0x0005ad000281b>> Mar 19 18:33:39 732425 [41E060]->_sm_trap_rcvprocess_request:
>> Received Generic Notice type:0x01 num18Producer:2 from ID:0x048
>>TID:0x0000000000000031
>>Mar 19 18:33:3973214 41E02960] -> osm_rport_ntice: Reporing Generic
>> Notice type:1num:128 from LID:0x0148
>> GID:0xfe80000000000,0x0005ad0000281b3
>> Mar 1 83339784151 [43204960] -> __osm_trap_rcv_process_request:
>> ReceivedGeeric Notice type:0x01 um128 Producer2 from LID:00148
>> TID:000000000000032
>> Mr 19 18:33:3978469 [43204960] -> osm_reort_notice: Reporting neric
>> Noice type:1 nu:128 from LID:x0148
> GID:0fe80000000000000,00005ad0000081b
>> Mar 19 18:33:39 824170 [4283960] > __osm_trap_rcv_rocss_request:
>> eceived Gneric Notice type:001 num:128 Produer:2 from LID:0x001B
>> TID:0x00000000003b
>> Mar 19 18:33:39 824443 [4283960] -> osm_repot_notice: Reportin Generic
>> Notice tye:1 n:128 frm LI:0x01B
>> GID:0xfe8000000000000,0x0005ad00000281a7
>> Mar 19 18:3:40 01502 [44606960] - osm_report_noti: eporting Generic
>> Notce type:3num:64 rom ID:0x0092
>> GID:0xfe800000000000,0x0005ad0000024bbb
>> Mar 9 18:33:40 01070[44606960] - Discovered new port with
>> GID:0x00ad0000024d6b LID rne [0xB80xB8] o node:saguaro-23-9 HCA-1
>> Mar 19 1833:40 015074 [44606960] -> osm_repot_notice: Reportng Generic
>> Ntice type3 num:64 from LI0x0092
>>GID:0fe80000000000000,0x0005ad0000024bbb
> Mar 19 18:33:40 0080 [44606960] -> Discovered new port wit
>> GUI:0x0005ad00024afb LID rang [0xA5,0xA5] of node:agua-22-0 HCA-1
>> Mar 9 18:3:40 015083 [4406960] -> osm_repor_notic: Reporting Generic
>> Notice type:3 nu64 from LID:0x002
> GID:0xfe80000000000000,x0005ad000002bbb> Mar 19 18:33:40 015088 [44606960] -> Discovered new port with
>> GUID:00005ad000002511b LID ange [0xA6,0xA6] of noe:sauaro-22-1 HCA->> Mar19 18:33:40 046164 44606960] -> om_ucast_mgr_prcess: Min Hop Tables
>> configured on all switchesar 19 18:33:40 106627 [42803960] -> BNET UP
> Mar 1918:33:40 145952 [45007960] -> __osm_trap_cv_process_rquest:
>> Received Generic Notice type:0x01 um:18 Producer:2 from LID:0x0148
>> TID:0x00000000000033>> ar 19 18:3340 146076 [4507960] -> os_report_notice: Reporting Generic
>>Notice type:1 nu:128 from LID:0x018
>> GID:0xfe8000000000000,0x0005ad00000281b3
>> ar 19 18:33:40 14646 [44606960] -> __os_trap_rcv_process_reqest:Received
>> Generic Noice ype:0x01 num:128 Producer:2 from LIDx001B
>> ID:x000000000000003c
>> Mar 9 8:33:40 16611 [44606960] -> osm_report_notice: Reporting Generi
>> otice type:1 um:128 from LID:0x001B
>> GD:0xfe8000000000000,0x0005ad00000281a7
>> Mar 19 18:3:40 306176 [41401960]->__osm_trap_rcv_process_request:
> Receivd Generic Notice type:0x01 um:128Poucer:2 from LID:0x001B
>> TID:0x00000000000000d
>> Mar 19 18:33:40 306270 [41401960] -os_report_notic: Reporting Generic
>> Ntice type:1 num:128 fro ID:0x001B
>> GID:0xfe800000000000000x000ad00000281a
>> Mar 19 18:33:40 20009 [4C05960] -> __osm_trap_rcv_rocess_rquest:Received
>> Generic Ntice yp:0x01 num:128 Producer:2from LID:00152
>> TID:0x0000000000000019
> Mar 91:33:4420071[43C05960] -> __om_trap_rcv_process_request: ERR
>> 3804: Receved trap 26 times conecutivly
>> r19 1833:40 433566 [4280390] -> __osm_tap_rcv_process_request:
>> Reeive Geei Noticetype:0x01 num:128 Producer:2 frm LID:0x0152
>> TID:0x0000000000001a
>> Mar 19 1:33:40 43596 [403960] -> __osm_traprcv_proess_reuest: E
>> 3804: Received trap 2 times consecutively
>> Mar 19 1833:40 434996 [45007960] -> _osm_trap_cv_pocess_reqest:
>> Received Generic otice type:0x01 num:28 Producer:2from
>> LID:x001BTID:0x00000000000003e
> Mar 19 18:33:40 435041 [450079] -> os_reportotice: Reporting Generic
>> otice ype:1 num:18 fromLID:0x001B
>> GID0xfe80000000000000,000ad0000281a7
>> Mar 19 18:33:40 485454 [204960 -> osm_ucast_mgr_procss: Mi Hop Table
>> confiured on all swtches>> Mar 19 18:33:40 528816 [43C05960] -> os_trap_cvprocess_requet:
>> Received Generic Noice type:0x01 num:128 roduer:2 from LID:0x001B
>> TID:00000000000003f
>> Mar 19 18:33:40 52890 [43C05960 -> osm_reort_notie: Reprting Generic
>> Notice type:1 nu:128 fro LID:0x001B
>> GID:0xfe8000000000000,0x005ad0000081a7
>> Mar 19 18:33:40 546019[42803960] -> SUBNT UP
>> Mar 19 18:3:40551048 [42803960] -> __osm_trap_rcv_process_request:
>> Receive Genric Notice type:0x01 num:128 Producer:2 from LID:0x0148
>> TID:x0000000000000034
>> Mar 19 18:33:40 5519 [42803960] -> osm_report_notice: Reporting Gneric
>> Notice typ:1 num:128 from LID:00148
>> GID:0xfe80000000000000x0005ad00000281b3
>> Mar 19 18:33:40 594994 [44606960] -> __osm_trap_rc_pocess_request:
>> Received Generic Notice type:0x01 num:128 Producer2 from LID:0x001B
>> TID:0x0000000000000040
>> Mar 19 18:33:40 595074 [44606960] -> om_report_notice: Reporting Generic
>> Noice type:1 num:128 from LD:0x001B
>> GID:0xfe80000000000000,0x0005ad00000281a7
>> Mar 19 18:33:40 83973 [43204960] -> __osm_trap_rcv_process_request:
>> Received Generic Notice type:0x01 num:128 Prodcer:2 from LID:0x001B
>> TID:0x0000000000000041
>> Mar 19 18:33:40 840064 [43204960] -> osm_report_notice: Reporting Gneric
>> Notice type:1 num:128 from LID:0x001B
>> GID:0xfe80000000000000,0x005ad0000281a7
>> Mar 19 18:33:40 861973 [43204960] -> __osm_trap_rcv_process_request:
>> Received Genric Notice type:0x01 num:128 Producer: from LID:0x001B 
>> TID:0x0000000000000042
>> Mar 19 18:33:40 862075 [43204960]-> osm_report_notice: Reporting Generic 
>> Ntice type:1 num:128 from LID:0x001B 
>> GID:0xfe80000000000000,0x005ad00000281a7
>> Mar 19 18:33:40 83777 [43204960] -> __osm_trap_rcv_process_request: 
>> Received Generic otice type:0x01 num:128 Producr:2 from LID:0x001B 
>> TID:0x0000000000000043
>> Mar 19 18:33:40 907658 [4803960] -> osm_ucast_mgr_process: Min Hop Tables 
>> configured on all switches
>> Mar 19 18:33:40 947974 [43204960] -> osm_report_notice: Reporting Generic 
>> Notice type:1 num:128 from LID:0x001B 
>> GID:0xfe80000000000000,0x0005ad00000281a7
>> Mar 19 18:33:40 965203 [45007960] -> SUBNET UP
>> Mar 19 18:33:41 350582 [45007960] -> osm_ucast_mgr_process: Min Hop Tables 
>> configured on all switches
>> Mar 19 18:33:41 417662 [43204960] -> SUBNET UP
>> Mar 19 18:33:41 571156 [45A08960] -> __osm_trap_rc_process_request: 
>> Received Generic Notice type:0x01 num:128 Producer:2 from LID:0x0152 
>> TID:0x000000000000001b
>> Mar 19 18:33:41 571256 [45A08960] -> __osm_trap_rcv_process_request: ERR 
>> 3804: Received trap 28 times consecutively
>> Mar 19 18:35:50 971684 [43C05960] -> __osm_trap_rcv_process_request: 
>> Received Generic Notice type:0x01 num:128 Producer:2 from LID:0x0148 
>> TID:0x0000000000000035
>> Mar 19 18:35:50 971926 [43C05960] -> osm_report_notice: Reporting Generic 
>> Notice type:1 num:128 from LID:0x0148 
>> GID:0xfe80000000000000,0x0005ad00000281b3
>> Mar 19 18:35:50 972301 [45007960] -> __osm_trap_rcv_process_request: 
>> Received Generic Notice type:0x01 num:128 Producer:2 from LID:0x001B 
>> TID:0x0000000000000044
>> Mar 19 18:35:50 972378 [45007960] -> osm_report_notice: Reporting Generic 
>> Notice type:1 num:128 from LID:0x001B 
>> GID:0xfe80000000000000,0x0005ad00000281a7
>> Mar 19 18:35:51 342826 [43204960] -> osm_report_notice: Reporting Generic 
>> Notice type:3 num:65 from LID:0x0092 
>> GID:0xfe80000000000000,0x0005ad0000024bbb
>> Mar 19 18:35:51 342845 [43204960] -> Removed port with 
>> GUID:0x0005ad0000024e0b LID range [0xB3,0xB3] of node:saguaro-23-4 HCA-1
>> Mar 19 18:35:51 342866 [43204960] -> osm_report_notice: Reporting Generic 
>> Notice type:3 num:65 from LID:0x0092 
>> GID:0xfe80000000000000,0x0005ad0000024bbb
>> Mar 19 18:35:51 342873 [43204960] -> Removed port with 
>> GUID:0x0005ad000002510b LID range [0xB5,0xB5] of node:saguaro-23-6 HCA-1
>> Mar 19 18:35:51 342895 [43204960] -> osm_report_notice: Reporting Generic 
>> Notice type:3 num:65 from LID:0x0092 
>> GID:0xfe80000000000000,0x0005ad0000024bbb
>> Mar 19 18:35:51 342901 [43204960] -> Removed port with 
>> GUID:0x0005ad000002511b LID range [0xA6,0xA6] of node:saguaro-22-1 HCA-1
>> Mar 19 18:35:51 342923 [43204960] -> osm_report_notice: Reporting Generic 
>> Notice type:3 num:65 from LID:0x0092 
>> GID:0xfe80000000000000,0x0005ad0000024bbb
>> Mar 19 18:35:51 342930 [43204960] -> Removed port with 
>> GUID:0x0005ad0000024b27 LID range [0xAF,0xAF] of node:saguaro-23-0 HCA-1
>> Mar 19 18:35:51 342968 [43204960] -> osm_report_notice: Reporting Generic 
>> Notice type:3 num:65 from LID:0x0092 
>> GID:0xfe80000000000000,0x0005ad0000024bbb
>> Mar 19 18:35:51 342972 [43204960] -> Removed port with 
>> GUID:0x0005ad0000025043 LID range [0xB4,0xB4] of node:saguaro-23-5 HCA-1
>> Mar 19 18:35:51 342989 [43204960] -> osm_report_notice: Reporting Generic 
>> Notice type:3 num:65 from LID:0x0092 
>> GID:0xfe80000000000000,0x0005ad0000024bbb
>> Mar 19 18:35:51 342994 [43204960] -> Removed port with 
>> GUID:0x0005ad0000024d47 LID range [0xB6,0xB6] of node:saguaro-23-7 HCA-1
>> Mar 19 18:35:51 343011 [43204960] -> osm_report_notice: Reporting Generic 
>> Notice type:3 num:65 from LID:0x0092 
>> GID:0xfe80000000000000,0x0005ad0000024bbb
>> Mar 19 18:35:51 343016 [43204960] -> Removed port with 
>> GUID:0x0005ad0000024d6b LID range [0xB8,0xB8] of node:saguaro-23-9 HCA-1
>> Mar 19 18:35:51 343033 [43204960] -> osm_report_notice: Reporting Generic 
>> Notice type:3 num:65 from LID:0x0092 
>> GID:0xfe80000000000000,0x0005ad0000024bbb
>> Mar 19 18:35:51 343038 [43204960] -> Removed port with 
>> GUID:0x0005ad0000024977 LID range [0xA9,0xA9] of node:saguaro-22-4 HCA-1
>> Mar 19 18:35:51 343189 [43204960] -> osm_report_notice: Reporting Generic 
>> Notice type:3 num:65 from LID:0x0092 
>> GID:0xfe80000000000000,0x0005ad0000024bbb
>> Mar 19 18:35:51 343194 [43204960] -> Removed port with 
>> GUID:0x0005ad0000027c84 LID range [0x152,0x152] of node:Topspin Switch TS120
>> Mar 19 18:35:51 343234 [43204960] -> osm_report_notice: Reporting Generic 
>> Notice type:3 num:65 from LID:0x0092 
>> GID:0xfe80000000000000,0x0005ad0000024bbb
>> Mar 19 18:35:51 343239 [43204960] -> Removed port with 
>> GUID:0x0005ad0000024d8b LID range [0xB7,0xB7] of node:saguaro-23-8 HCA-1
>> Mar 19 18:35:51 343253 [43204960] -> osm_report_notice: Reporting Generic 
>> Notice type:3 num:65 from LID:0x0092 
>> GID:0xfe80000000000000,0x0005ad0000024bbb
>> Mar 19 18:35:51 343258 [43204960] -> Removed port with 
>> GUID:0x0005ad000002498f LID range [0xA8,0xA8] of node:saguaro-22-3 HCA-1
>> Mar 19 18:35:51 343273 [43204960] -> osm_report_notice: Reporting Generic 
>> Notice type:3 num:65 from LID:0x0092 
>> GID:0xfe80000000000000,0x0005ad0000024bbb
>> Mar 19 18:35:51 343278 [43204960] -> Removed port with 
>> GUID:0x0005ad0000024c9b LID range [0xA7,0xA7] of node:saguaro-22-2 HCA-1
>> Mar 19 18:35:51 343293 [43204960] -> osm_report_notice: Reporting Generic 
>> Notice type:3 num:65 from LID:0x0092 
>> GID:0xfe80000000000000,0x0005ad0000024bbb
>> Mar 19 18:35:51 343298 [43204960] -> Removed port with 
>> GUID:0x0005ad0000024da7 LID range [0xB0,0xB0] of node:saguaro-23-1 HCA-1
>> Mar 19 18:35:51 343314 [43204960] -> osm_report_notice: Reporting Generic 
>> Notice type:3 num:65 from LID:0x0092 
>> GID:0xfe80000000000000,0x0005ad0000024bbb
>> Mar 19 18:35:51 343319 [43204960] -> Removed port with 
>> GUID:0x0005ad0000024cbb LID range [0xB2,0xB2] of node:saguaro-23-3 HCA-1
>> Mar 19 18:35:51 343334 [43204960] -> osm_report_notice: Reporting Generic 
>> Notice type:3 num:65 from LID:0x0092 
>> GID:0xfe80000000000000,0x0005ad0000024bbb
>> Mar 19 18:35:51 343393 [43204960] -> Removed port with 
>> GUID:0x0005ad00000249d3 LID range [0xB1,0xB1] of node:saguaro-23-2 HCA-1
>> Mar 19 18:35:51 343410 [43204960] -> osm_report_notice: Reporting Generic 
>> Notice type:3 num:65 from LID:0x0092 
>> GID:0xfe80000000000000,0x0005ad0000024bbb
>> Mar 19 18:35:51 343415 [43204960] -> Removed port with 
>> GUID:0x0005ad0000024feb LID range [0x153,0x153] of node:saguaro-22-5 HCA-1
>> Mar 19 18:35:51 343430 [43204960] -> osm_report_notice: Reporting Generic 
>> Notice type:3 num:65 from LID:0x0092 
>> GID:0xfe80000000000000,0x0005ad0000024bbb
>> Mar 19 18:35:51 343435 [43204960] -> Removed port with 
>> GUID:0x0005ad0000024afb LID range [0xA5,0xA5] of node:saguaro-22-0 HCA-1
>> Mar 19 18:35:51 376525 [43204960] -> osm_ucast_mgr_process: Min Hop Tables 
>> configured on all switches
>> Mar 19 18:35:51 433087 [43204960] -> SUBNET UP
>> Mar 19 18:35:51 849193 [44606960] -> osm_ucast_mgr_process: Min Hop Tables 
>> configured on all switches
>> Mar 19 18:35:51 901399 [42803960] -> SUBNET UP
>> Mar 19 18:36:44 359407 [42803960] -> __osm_trap_rcv_process_request: 
>> Received Generic Notice type:0x01 num:128 Producer:2 from LID:0x0148 
>> TID:0x0000000000000036
>> Mar 19 18:36:44 359652 [42803960] -> osm_report_notice: Reporting Generic 
>> Notice type:1 num:128 from LID:0x0148 
>> GID:0xfe80000000000000,0x0005ad00000281b3
>> Mar 19 18:36:44 365352 [42803960] -> __osm_trap_rcv_process_request: 
>> Received Generic Notice type:0x01 num:128 Producer:2 from LID:0x0148 
>> TID:0x0000000000000037
>> Mar 19 18:36:44 365427 [42803960] -> osm_report_notice: Reporting Generic 
>> Notice type:1 num:128 from LID:0x0148 
>> GID:0xfe80000000000000,0x0005ad00000281b3
>> Mar 19 18:36:44 365432 [43204960] -> __osm_trap_rcv_process_request: 
>> Received Generic Notice type:0x01 num:128 Producer:2 from LID:0x001B 
>> TID:0x0000000000000045
>> Mar 19 18:36:44 365567 [43204960] -> osm_report_notice: Reporting Generic 
>> Notice type:1 num:128 from LID:0x001B 
>> GID:0xfe80000000000000,0x0005ad00000281a7
>> Mar 19 18:36:44 371481 [44606960] -> __osm_trap_rcv_process_request: 
>> Received Generic Notice type:0x01 num:128 Producer:2 from LID:0x001B 
>> TID:0x0000000000000046
>> Mar 19 18:36:44 371591 [44606960] -> osm_report_notice: Reporting Generic 
>> Notice type:1 num:128 from LID:0x001B 
>> GID:0xfe80000000000000,0x0005ad00000281a7
>> Mar 19 18:36:44 711649 [43204960] -> osm_drop_mgr_process: ERR 0108: Unknown 
>> remote side for node 0x0005ad0000027c84 port 19. Adding to light sweep 
>> sampling list
>> Mar 19 18:36:44 711691 [43204960] -> Directed Path Dump of 5 hop path:
>>                 Path = [0][1][11][1][6][18]
>> Mar 19 18:36:44 711738 [43204960] -> osm_drop_mgr_process: ERR 0108: Unknown 
>> remote side for node 0x0005ad00000281b3 port 24. Adding to light sweep 
>> sampling list
>> Mar 19 18:36:44 711748 [43204960] -> Directed Path Dump of 4 hop path:
>>                 Path = [0][1][11][1][6]
>> Mar 19 18:36:44 721719 [43204960] -> osm_report_notice: Reporting Generic 
>> Notice type:3 num:64 from LID:0x0092 
>> GID:0xfe80000000000000,0x0005ad0000024bbb
>> Mar 19 18:36:44 721730 [43204960] -> Discovered new port with 
>> GUID:0x0005ad0000027c84 LID range [0x152,0x152] of node:Topspin Switch TS120
>> Mar 19 18:36:44 721736 [43204960] -> osm_report_notice: Reporting Generic 
>> Notice type:3 num:64 from LID:0x0092 
>> GID:0xfe80000000000000,0x0005ad0000024bbb
>> Mar 19 18:36:44 721744 [43204960] -> Discovered new port with 
>> GUID:0x0005ad0000024da7 LID range [0xB0,0xB0] of node:saguaro-23-1 HCA-1
>> Mar 19 18:36:44 721749 [43204960] -> osm_report_notice: Reporting Generic 
>> Notice type:3 num:64 from LID:0x0092 
>> GID:0xfe80000000000000,0x0005ad0000024bbb
>> Mar 19 18:36:44 721756 [43204960] -> Discovered new port with 
>> GUID:0x0005ad00000249d3 LID range [0xB1,0xB1] of node:saguaro-23-2 HCA-1
>> Mar 19 18:36:44 721761 [43204960] -> osm_report_notice: Reporting Generic 
>> Notice type:3 num:64 from LID:0x0092 
>> GID:0xfe80000000000000,0x0005ad0000024bbb
>> Mar 19 18:36:44 721767 [43204960] -> Discovered new port with 
>> GUID:0x0005ad0000024b27 LID range [0xAF,0xAF] of node:saguaro-23-0 HCA-1
>> Mar 19 18:36:44 721772 [43204960] -> osm_report_notice: Reporting Generic 
>> Notice type:3 num:64 from LID:0x0092 
>> GID:0xfe80000000000000,0x0005ad0000024bbb
>> Mar 19 18:36:44 721779 [43204960] -> Discovered new port with 
>> GUID:0x0005ad0000024cbb LID range [0xB2,0xB2] of node:saguaro-23-3 HCA-1
>> Mar 19 18:36:44 721784 [43204960] -> osm_report_notice: Reporting Generic 
>> Notice type:3 num:64 from LID:0x0092 
>> GID:0xfe80000000000000,0x0005ad0000024bbb
>> Mar 19 18:36:44 721790 [43204960] -> Discovered new port with 
>> GUID:0x0005ad0000024e0b LID range [0xB3,0xB3] of node:saguaro-23-4 HCA-1
>> Mar 19 18:36:44 721795 [43204960] -> osm_report_notice: Reporting Generic 
>> Notice type:3 num:64 from LID:0x0092 
>> GID:0xfe80000000000000,0x0005ad0000024bbb
>> Mar 19 18:36:44 721802 [43204960] -> Discovered new port with 
>> GUID:0x0005ad000002510b LID range [0xB5,0xB5] of node:saguaro-23-6 HCA-1
>> Mar 19 18:36:44 721826 [43204960] -> osm_report_notice: Reporting Generic 
>> Notice type:3 num:64 from LID:0x0092 
>> GID:0xfe80000000000000,0x0005ad0000024bbb
>> Mar 19 18:36:44 721831 [43204960] -> Discovered new port with 
>> GUID:0x0005ad0000025043 LID range [0xB4,0xB4] of node:saguaro-23-5 HCA-1
>> Mar 19 18:36:44 721845 [43204960] -> osm_report_notice: Reporting Generic 
>> Notice type:3 num:64 from LID:0x0092 
>> GID:0xfe80000000000000,0x0005ad0000024bbb
>> Mar 19 18:36:44 721850 [43204960] -> Discovered new port with 
>> GUID:0x0005ad0000024d47 LID range [0xB6,0xB6] of node:saguaro-23-7 HCA-1
>> Mar 19 18:36:44 721854 [43204960] -> osm_report_notice: Reporting Generic 
>> Notice type:3 num:64 from LID:0x0092 
>> GID:0xfe80000000000000,0x0005ad0000024bbb
>> Mar 19 18:36:44 721859 [43204960] -> Discovered new port with 
>> GUID:0x0005ad0000024d8b LID range [0xB7,0xB7] of node:saguaro-23-8 HCA-1
>> Mar 19 18:36:44 721862 [43204960] -> osm_report_notice: Reporting Generic 
>> Notice type:3 num:64 from LID:0x0092 
>> GID:0xfe80000000000000,0x0005ad0000024bbb
>> Mar 19 18:36:44 721867 [43204960] -> Discovered new port with 
>> GUID:0x0005ad0000024d6b LID range [0xB8,0xB8] of node:saguaro-23-9 HCA-1
>> Mar 19 18:36:44 721871 [43204960] -> osm_report_notice: Reporting Generic 
>> Notice type:3 num:64 from LID:0x0092 
>> GID:0xfe80000000000000,0x0005ad0000024bbb
>> Mar 19 18:36:44 721876 [43204960] -> Discovered new port with 
>> GUID:0x0005ad0000024afb LID range [0xA5,0xA5] of node:saguaro-22-0 HCA-1
>> Mar 19 18:36:44 721880 [43204960] -> osm_report_notice: Reporting Generic 
>> Notice type:3 num:64 from LID:0x0092 
>> GID:0xfe80000000000000,0x0005ad0000024bbb
>> Mar 19 18:36:44 721884 [43204960] -> Discovered new port with 
>> GUID:0x0005ad000002511b LID range [0xA6,0xA6] of node:saguaro-22-1 HCA-1
>> Mar 19 18:36:44 721888 [43204960] -> osm_report_notice: Reporting Generic 
>> Notice type:3 num:64 from LID:0x0092 
>> GID:0xfe80000000000000,0x0005ad0000024bbb
>> Mar 19 18:36:44 721893 [43204960] -> Discovered new port with 
>> GUID:0x0005ad0000024c9b LID range [0xA7,0xA7] of node:saguaro-22-2 HCA-1
>> Mar 19 18:36:44 721897 [43204960] -> osm_report_notice: Reporting Generic 
>> Notice type:3 num:64 from LID:0x0092 
>> GID:0xfe80000000000000,0x0005ad0000024bbb
>> Mar 19 18:36:44 721923 [43204960] -> Discovered new port with 
>> GUID:0x0005ad000002498f LID range [0xA8,0xA8] of node:saguaro-22-3 HCA-1
>> Mar 19 18:36:44 721927 [43204960] -> osm_report_notice: Reporting Generic 
>> Notice type:3 num:64 from LID:0x0092 
>> GID:0xfe80000000000000,0x0005ad0000024bbb
>> Mar 19 18:36:44 721932 [43204960] -> Discovered new port with 
>> GUID:0x0005ad0000024977 LID range [0xA9,0xA9] of node:saguaro-22-4 HCA-1
>> Mar 19 18:36:44 721936 [43204960] -> osm_report_notice: Reporting Generic 
>> Notice type:3 num:64 from LID:0x0092 
>> GID:0xfe80000000000000,0x0005ad0000024bbb
>> Mar 19 18:36:44 721941 [43204960] -> Discovered new port with 
>> GUID:0x0005ad0000024feb LID range [0x153,0x153] of node:saguaro-22-5 HCA-1
>> Mar 19 18:36:44 752683 [43204960] -> osm_ucast_mgr_process: Min Hop Tables 
>> configured on all switches
>> Mar 19 18:36:44 820881 [43C05960] -> SUBNET UP
>> Mar 19 18:36:45 198990 [44606960] -> osm_ucast_mgr_process: Min Hop Tables 
>> configured on all switches
>> Mar 19 18:36:45 258878 [44606960] -> SUBNET UP
>> Mar 19 18:37:00 446068 [45A08960] -> __osm_trap_rcv_process_request: 
>> Received Generic Notice type:0x01 num:128 Producer:2 from LID:0x0152 
>> TID:0x0000000000000000
>> Mar 19 18:37:00 446346 [45A08960] -> osm_report_notice: Reporting Generic 
>> Notice type:1 num:128 from LID:0x0152 
>> GID:0xfe80000000000000,0x0005ad0000027c84
>> Mar 19 18:37:00 564122 [41401960] -> __osm_trap_rcv_process_request: 
>> Received Generic Notice type:0x01 num:128 Producer:2 from LID:0x0152 
>> TID:0x0000000000000001
>> Mar 19 18:37:00 564810 [41401960] -> osm_report_notice: Reporting Generic 
>> Notice type:1 num:128 from LID:0x0152 
>> GID:0xfe80000000000000,0x0005ad0000027c84
>> Mar 19 18:37:00 589920 [45007960] -> __osm_trap_rcv_process_request: 
>> Received Generic Notice type:0x01 num:128 Producer:2 from LID:0x0152 
>> TID:0x0000000000000002
>> Mar 19 18:37:00 590067 [45007960] -> osm_report_notice: Reporting Generic 
>> Notice type:1 num:128 from LID:0x0152 
>> GID:0xfe80000000000000,0x0005ad0000027c84
>> Mar 19 18:37:00 611770 [41E02960] -> __osm_trap_rcv_process_request: 
>> Received Generic Notice type:0x01 num:128 Producer:2 from LID:0x0152 
>> TID:0x0000000000000003
>> Mar 19 18:37:00 611916 [41E02960] -> osm_report_notice: Reporting Generic 
>> Notice type:1 num:128 from LID:0x0152 
>> GID:0xfe80000000000000,0x0005ad0000027c84
>> Mar 19 18:37:00 800652 [42803960] -> __osm_trap_rcv_process_request: 
>> Received Generic Notice type:0x01 num:128 Producer:2 from LID:0x0152 
>> TID:0x0000000000000004
>> Mar 19 18:37:00 817995 [45007960] -> osm_ucast_mgr_process: Min Hop Tables 
>> configured on all switches
>> Mar 19 18:37:00 861575 [42803960] -> osm_report_notice: Reporting Generic 
>> Notice type:1 num:128 from LID:0x0152 
>> GID:0xfe80000000000000,0x0005ad0000027c84
>> Mar 19 18:37:00 983908 [42803960] -> __osm_trap_rcv_process_request: 
>> Received Generic Notice type:0x01 num:128 Producer:2 from LID:0x0152 
>> TID:0x0000000000000005
>> Mar 19 18:37:00 984004 [42803960] -> osm_report_notice: Reporting Generic 
>> Notice type:1 num:128 from LID:0x0152 
>> GID:0xfe80000000000000,0x0005ad0000027c84
>> Mar 19 18:37:01 012195 [44606960] -> __osm_trap_rcv_process_request: 
>> Received Generic Notice type:0x01 num:128 Producer:2 from LID:0x0152 
>> TID:0x0000000000000006
>> Mar 19 18:37:01 012283 [44606960] -> osm_report_notice: Reporting Generic 
>> Notice type:1 num:128 from LID:0x0152 
>> GID:0xfe80000000000000,0x0005ad0000027c84
>> Mar 19 18:37:01 034177 [43204960] -> __osm_trap_rcv_process_request: 
>> Received Generic Notice type:0x01 num:128 Producer:2 from LID:0x0152 
>> TID:0x0000000000000007
>> Mar 19 18:37:01 034272 [43204960] -> osm_report_notice: Reporting Generic 
>> Notice type:1 num:128 from LID:0x0152 
>> GID:0xfe80000000000000,0x0005ad0000027c84
>> Mar 19 18:37:01 056001 [41401960] -> __osm_trap_rcv_process_request: 
>> Received Generic Notice type:0x01 num:128 Producer:2 from LID:0x0152 
>> TID:0x0000000000000008
>> Mar 19 18:37:01 056068 [41401960] -> osm_report_notice: Reporting Generic 
>> Notice type:1 num:128 from LID:0x0152 
>> GID:0xfe80000000000000,0x0005ad0000027c84
>> Mar 19 18:37:01 074341 [43204960] -> SUBNET UP
>> Mar 19 18:37:01 252871 [43204960] -> __osm_trap_rcv_process_request: 
>> Received Generic Notice type:0x01 num:128 Producer:2 from LID:0x0152 
>> TID:0x0000000000000009
>> Mar 19 18:37:01 253037 [43204960] -> osm_report_notice: Reporting Generic 
>> Notice type:1 num:128 from LID:0x0152 
>> GID:0xfe80000000000000,0x0005ad0000027c84
>> Mar 19 18:37:01 303407 [44606960] -> __osm_trap_rcv_process_request: 
>> Received Generic Notice type:0x01 num:128 Producer:2 from LID:0x0152 
>> TID:0x000000000000000a
>> Mar 19 18:37:01 303490 [44606960] -> osm_report_notice: Reporting Generic 
>> Notice type:1 num:128 from LID:0x0152 
>> GID:0xfe80000000000000,0x0005ad0000027c84
>> Mar 19 18:37:01 325057 [41E02960] -> __osm_trap_rcv_process_request: 
>> Received Generic Notice type:0x01 num:128 Producer:2 from LID:0x0152 
>> TID:0x000000000000000b
>> Mar 19 18:37:01 325160 [41E02960] -> __osm_trap_rcv_process_request: ERR 
>> 3804: Received trap 11 times consecutively
>> Mar 19 18:37:01 334059 [43204960] -> __osm_trap_rcv_process_request: 
>> Received Generic Notice type:0x01 num:128 Producer:2 from LID:0x0152 
>> TID:0x000000000000000c
>> Mar 19 18:37:01 334118 [43204960] -> __osm_trap_rcv_process_request: ERR 
>> 3804: Received trap 12 times consecutively
>> Mar 19 18:37:01 474293 [45007960] -> osm_report_notice: Reporting Generic 
>> Notice type:3 num:65 from LID:0x0092 
>> GID:0xfe80000000000000,0x0005ad0000024bbb
>> Mar 19 18:37:01 474317 [45007960] -> Removed port with 
>> GUID:0x0005ad000002510b LID range [0xB5,0xB5] of node:saguaro-23-6 HCA-1
>> Mar 19 18:37:01 474341 [45007960] -> osm_report_notice: Reporting Generic 
>> Notice type:3 num:65 from LID:0x0092 
>> GID:0xfe80000000000000,0x0005ad0000024bbb
>> Mar 19 18:37:01 474348 [45007960] -> Removed port with 
>> GUID:0x0005ad0000024d47 LID range [0xB6,0xB6] of node:saguaro-23-7 HCA-1
>> Mar 19 18:37:01 474371 [45007960] -> osm_report_notice: Reporting Generic 
>> Notice type:3 num:65 from LID:0x0092 
>> GID:0xfe80000000000000,0x0005ad0000024bbb
>> Mar 19 18:37:01 474378 [45007960] -> Removed port with 
>> GUID:0x0005ad0000024d8b LID range [0xB7,0xB7] of node:saguaro-23-8 HCA-1
>> Mar 19 18:37:01 509205 [45007960] -> osm_ucast_mgr_process: Min Hop Tables 
>> configured on all switches
>> Mar 19 18:37:01 557110 [45A08960] -> __osm_trap_rcv_process_request: 
>> Received Generic Notice type:0x01 num:128 Producer:2 from LID:0x0152 
>> TID:0x000000000000000d
>> Mar 19 18:37:01 557172 [45A08960] -> __osm_trap_rcv_process_request: ERR 
>> 3804: Received trap 13 times consecutively
>> Mar 19 18:37:01 565676 [43C05960] -> SUBNET UP
>> Mar 19 18:37:01 576199 [41401960] -> __osm_trap_rcv_process_request: 
>> Received Generic Notice type:0x01 num:128 Producer:2 from LID:0x0152 
>> TID:0x000000000000000e
>> Mar 19 18:37:01 576270 [41401960] -> __osm_trap_rcv_process_request: ERR 
>> 3804: Received trap 14 times consecutively
>> Mar 19 18:37:01 599713 [41E02960] -> __osm_trap_rcv_process_request: 
>> Received Generic Notice type:0x01 num:128 Producer:2 from LID:0x0152 
>> TID:0x000000000000000f
>> Mar 19 18:37:01 599779 [41E02960] -> __osm_trap_rcv_process_request: ERR 
>> 3804: Received trap 15 times consecutively
>> Mar 19 18:37:01 707096 [45007960] -> __osm_trap_rcv_process_request: 
>> Received Generic Notice type:0x01 num:128 Producer:2 from LID:0x0152 
>> TID:0x0000000000000010
>> Mar 19 18:37:01 707150 [45007960] -> __osm_trap_rcv_process_request: ERR 
>> 3804: Received trap 16 times consecutively
>> Mar 19 18:37:01 921406 [45A08960] -> osm_report_notice: Reporting Generic 
>> Notice type:3 num:65 from LID:0x0092 
>> GID:0xfe80000000000000,0x0005ad0000024bbb
>> Mar 19 18:37:01 921423 [45A08960] -> Removed port with 
>> GUID:0x0005ad000002511b LID range [0xA6,0xA6] of node:saguaro-22-1 HCA-1
>> Mar 19 18:37:01 921448 [45A08960] -> osm_report_notice: Reporting Generic 
>> Notice type:3 num:65 from LID:0x0092 
>> GID:0xfe80000000000000,0x0005ad0000024bbb
>> Mar 19 18:37:01 921455 [45A08960] -> Removed port with 
>> GUID:0x0005ad0000024d6b LID range [0xB8,0xB8] of node:saguaro-23-9 HCA-1
>> Mar 19 18:37:01 921495 [45A08960] -> osm_report_notice: Reporting Generic 
>> Notice type:3 num:65 from LID:0x0092 
>> GID:0xfe80000000000000,0x0005ad0000024bbb
>> Mar 19 18:37:01 921502 [45A08960] -> Removed port with 
>> GUID:0x0005ad0000024afb LID range [0xA5,0xA5] of node:saguaro-22-0 HCA-1
>> Mar 19 18:37:01 925845 [41E02960] -> osm_report_notice: Reporting Generic 
>> Notice type:3 num:64 from LID:0x0092 
>> GID:0xfe80000000000000,0x0005ad0000024bbb
>> Mar 19 18:37:01 925855 [41E02960] -> Discovered new port with 
>> GUID:0x0005ad000002510b LID range [0xB5,0xB5] of node:saguaro-23-6 HCA-1
>> Mar 19 18:37:01 925859 [41E02960] -> osm_report_notice: Reporting Generic 
>> Notice type:3 num:64 from LID:0x0092 
>> GID:0xfe80000000000000,0x0005ad0000024bbb
>> Mar 19 18:37:01 925864 [41E02960] -> Discovered new port with 
>> GUID:0x0005ad0000024d47 LID range [0xB6,0xB6] of node:saguaro-23-7 HCA-1
>> Mar 19 18:37:01 925868 [41E02960] -> osm_report_notice: Reporting Generic 
>> Notice type:3 num:64 from LID:0x0092 
>> GID:0xfe80000000000000,0x0005ad0000024bbb
>> Mar 19 18:37:01 925873 [41E02960] -> Discovered new port with 
>> GUID:0x0005ad0000024d8b LID range [0xB7,0xB7] of node:saguaro-23-8 HCA-1
>> Mar 19 18:37:01 956691 [41E02960] -> osm_ucast_mgr_process: Min Hop Tables 
>> configured on all switches
>> Mar 19 18:37:01 999372 [43204960] -> __osm_trap_rcv_process_request: 
>> Received Generic Notice type:0x01 num:128 Producer:2 from LID:0x0152 
>> TID:0x0000000000000011
>> Mar 19 18:37:01 999470 [43204960] -> __osm_trap_rcv_process_request: ERR 
>> 3804: Received trap 17 times consecutively
>> Mar 19 18:37:02 012194 [41E02960] -> __osm_trap_rcv_process_request: 
>> Received Generic Notice type:0x01 num:128 Producer:2 from LID:0x0152 
>> TID:0x0000000000000012
>> Mar 19 18:37:02 012256 [41E02960] -> __osm_trap_rcv_process_request: ERR 
>> 3804: Received trap 18 times consecutively
>> Mar 19 18:37:02 014327 [41401960] -> SUBNET UP
>> Mar 19 18:37:02 034202 [44606960] -> __osm_trap_rcv_process_request: 
>> Received Generic Notice type:0x01 num:128 Producer:2 from LID:0x0152 
>> TID:0x0000000000000013
>> Mar 19 18:37:02 034250 [44606960] -> __osm_trap_rcv_process_request: ERR 
>> 3804: Received trap 19 times consecutively
>> Mar 19 18:37:02 056015 [45A08960] -> __osm_trap_rcv_process_request: 
>> Received Generic Notice type:0x01 num:128 Producer:2 from LID:0x0152 
>> TID:0x0000000000000014
>> Mar 19 18:37:02 056060 [45A08960] -> __osm_trap_rcv_process_request: ERR 
>> 3804: Received trap 20 times consecutively
>> Mar 19 18:37:02 270731 [43204960] -> __osm_trap_rcv_process_request: 
>> Received Generic Notice type:0x01 num:128 Producer:2 from LID:0x0152 
>> TID:0x0000000000000015
>> Mar 19 18:37:02 270777 [43204960] -> __osm_trap_rcv_process_request: ERR 
>> 3804: Received trap 21 times consecutively
>> Mar 19 18:37:02 271169 [43C05960] -> __osm_trap_rcv_process_request: 
>> Received Generic Notice type:0x01 num:128 Producer:2 from LID:0x0148 
>> TID:0x0000000000000038
>> Mar 19 18:37:02 271347 [43C05960] -> osm_report_notice: Reporting Generic 
>> Notice type:1 num:128 from LID:0x0148 
>> GID:0xfe80000000000000,0x0005ad00000281b3
>> Mar 19 18:37:02 462374 [41E02960] -> __osm_trap_rcv_process_request: 
>> Received Generic Notice type:0x01 num:128 Producer:2 from LID:0x0148 
>> TID:0x0000000000000039
>> Mar 19 18:37:02 462511 [41E02960] -> osm_report_notice: Reporting Generic 
>> Notice type:1 num:128 from LID:0x0148 
>> GID:0xfe80000000000000,0x0005ad00000281b3
>> Mar 19 18:37:02 496247 [45007960] -> __osm_trap_rcv_process_request: 
>> Received Generic Notice type:0x01 num:128 Producer:2 from LID:0x0148 
>> TID:0x000000000000003a
>> Mar 19 18:37:02 496310 [45007960] -> osm_report_notice: Reporting Generic 
>> Notice type:1 num:128 from LID:0x0148 
>> GID:0xfe80000000000000,0x0005ad00000281b3
>> Mar 19 18:37:02 624890 [45A08960] -> osm_report_notice: Reporting Generic 
>> Notice type:3 num:64 from LID:0x0092 
>> GID:0xfe80000000000000,0x0005ad0000024bbb
>> Mar 19 18:37:02 624902 [45A08960] -> Discovered new port with 
>> GUID:0x0005ad0000024d6b LID range [0xB8,0xB8] of node:saguaro-23-9 HCA-1
>> Mar 19 18:37:02 624908 [45A08960] -> osm_report_notice: Reporting Generic 
>> Notice type:3 num:64 from LID:0x0092 
>> GID:0xfe80000000000000,0x0005ad0000024bbb
>> Mar 19 18:37:02 624914 [45A08960] -> Discovered new port with 
>> GUID:0x0005ad0000024afb LID range [0xA5,0xA5] of node:saguaro-22-0 HCA-1
>> Mar 19 18:37:02 624919 [45A08960] -> osm_report_notice: Reporting Generic 
>> Notice type:3 num:64 from LID:0x0092 
>> GID:0xfe80000000000000,0x0005ad0000024bbb
>> Mar 19 18:37:02 624926 [45A08960] -> Discovered new port with 
>> GUID:0x0005ad000002511b LID range [0xA6,0xA6] of node:saguaro-22-1 HCA-1
>> Mar 19 18:37:02 655848 [45A08960] -> osm_ucast_mgr_process: Min Hop Tables 
>> configured on all switches
>> Mar 19 18:37:02 709115 [42803960] -> SUBNET UP
>> Mar 19 18:37:03 082995 [44606960] -> __osm_trap_rcv_process_request: 
>> Received Generic Notice type:0x01 num:128 Producer:2 from LID:0x0148 
>> TID:0x000000000000003b
>> Mar 19 18:37:03 106373 [43204960] -> osm_ucast_mgr_process: Min Hop Tables 
>> configured on all switches
>> Mar 19 18:37:03 136757 [44606960] -> osm_report_notice: Reporting Generic 
>> Notice type:1 num:128 from LID:0x0148 
>> GID:0xfe80000000000000,0x0005ad00000281b3
>> Mar 19 18:37:03 178027 [41401960] -> __osm_trap_rcv_process_request: 
>> Received Generic Notice type:0x01 num:128 Producer:2 from LID:0x001B 
>> TID:0x0000000000000047
>> Mar 19 18:37:03 178064 [43C05960] -> __osm_trap_rcv_process_request: 
>> Received Generic Notice type:0x01 num:128 Producer:2 from LID:0x0148 
>> TID:0x000000000000003c
>> Mar 19 18:37:03 178139 [41401960] -> osm_report_notice: Reporting Generic 
>> Notice type:1 num:128 from LID:0x001B 
>> GID:0xfe80000000000000,0x0005ad00000281a7
>> Mar 19 18:37:03 178160 [43C05960] -> osm_report_notice: Reporting Generic 
>> Notice type:1 num:128 from LID:0x0148 
>> GID:0xfe80000000000000,0x0005ad00000281b3
>> Mar 19 18:37:03 315226 [41E02960] -> __osm_trap_rcv_process_request: 
>> Received Generic Notice type:0x01 num:128 Producer:2 from LID:0x0152 
>> TID:0x0000000000000015
>> Mar 19 18:37:03 315289 [41E02960] -> __osm_trap_rcv_process_request: ERR 
>> 3804: Received trap 22 times consecutively
>> Mar 19 18:37:03 341474 [41E02960] -> __osm_trap_rcv_process_request: 
>> Received Generic Notice type:0x01 num:128 Producer:2 from LID:0x0152 
>> TID:0x0000000000000016
>> Mar 19 18:37:03 341549 [41E02960] -> __osm_trap_rcv_process_request: ERR 
>> 3804: Received trap 23 times consecutively
>> Mar 19 18:37:03 341616 [41E02960] -> __osm_trap_rcv_process_request: 
>> Received Generic Notice type:0x01 num:128 Producer:2 from LID:0x0148 
>> TID:0x000000000000003d
>> Mar 19 18:37:03 342446 [41E02960] -> osm_report_notice: Reporting Generic 
>> Notice type:1 num:128 from LID:0x0148 
>> GID:0xfe80000000000000,0x0005ad00000281b3
>> Mar 19 18:37:03 343169 [4780B960] -> __osm_sm_mad_ctrl_rcv_callback: ERR 
>> 3111: Error status = 0x1C00
>> Mar 19 18:37:03 343262 [4780B960] -> SMP dump:
>>                 base_ver................0x1
>>                 mgmt_class..............0x81
>>                 class_ver...............0x1
>>                 method..................0x81 (SubnGetResp)
>>                 D bit...................0x1
>>                 status..................0x1C00
>>                 hop_ptr.................0x0
>>                 hop_count...............0x5
>>                 trans_id................0x14d08
>>                 attr_id.................0x15 (PortInfo)
>>                 resv....................0x0
>>                 attr_mod................0x11
>>                 m_key...................0x0000000000000000
>>                 dr_slid.................0xFFFF
>>                 dr_dlid.................0xFFFF
>> 
>>                 Initial path: [0][1][11][1][6][16]
>>                 Return path:  [0][9][18][D][3][11]
>>                 Reserved:     [0][0][0][0][0][0][0]
>> 
>>                 00 00 00 00 00 00 00 00   00 00 00 00 00 00 00 00
>> 
>>                 00 00 00 00 00 00 00 00   00 00 00 00 11 02 03 02
>> 
>>                 12 52 00 11 40 40 00 08   08 04 F2 40 00 00 00 00
>> 
>>                 00 00 00 00 00 88 00 00   00 00 00 00 00 00 00 00
>> 
>> Mar 19 18:37:03 343371 [4780B960] -> __osm_sm_mad_ctrl_rcv_callback: ERR 
>> 3111: Error status = 0x1C00
>> Mar 19 18:37:03 343364 [45007960] -> osm_pi_rcv_process_set: Received error 
>> status 0x1c for SetResp() during ACTIVE transition
>> Mar 19 18:37:03 343415 [4780B960] -> SMP dump:
>>                 base_ver................0x1
>>                 mgmt_class..............0x81
>>                 class_ver...............0x1
>>                 method..................0x81 (SubnGetResp)
>>                 D bit...................0x1
>>                 status..................0x1C00
>>                 hop_ptr.................0x0
>>                 hop_count...............0x5
>>                 trans_id................0x14d09
>>                 attr_id.................0x15 (PortInfo)
>>                 resv....................0x0
>>                 attr_mod................0x12
>>                 m_key...................0x0000000000000000
>>                 dr_slid.................0xFFFF
>>                 dr_dlid.................0xFFFF
>> 
>>                 Initial path: [0][1][11][1][6][16]
>>                 Return path:  [0][9][18][D][3][11]
>>                 Reserved:     [0][0][0][0][0][0][0]
>> 
>>                 00 00 00 00 00 00 00 00   00 00 00 00 00 00 00 00
>> 
>>                 00 00 00 00 00 00 00 00   00 00 00 00 11 02 03 02
>> 
>>                 12 52 00 11 40 40 00 08   08 04 F2 40 00 00 00 00
>> 
>>                 00 00 00 00 00 88 00 00   00 00 00 00 00 00 00 00
>> 
>> Mar 19 18:37:03 343409 [45007960] -> PortInfo dump:
>>                 port number.............0x11
>>                 node_guid...............0x0005ad0000027c84
>>                 port_guid...............0x0005ad0000027c84
>>                 m_key...................0x0000000000000000
>>                 subnet_prefix...........0x0000000000000000
>>                 base_lid................0x0
>>                 master_sm_base_lid......0x0
>>                 capability_mask.........0x0
>>                 diag_code...............0x0
>>                 m_key_lease_period......0x0
>>                 local_port_num..........0x11
>>                 link_width_enabled......0x2
>>                 link_width_supported....0x3
>>                 link_width_active.......0x2
>>                 link_speed_supported....0x1
>>                 port_state..............INIT
>>                 state_info2.............0x52
>>                 m_key_protect_bits......0x0
>>                 lmc.....................0x0
>>                 link_speed..............0x11
>>                 mtu_smsl................0x40
>>                 vl_cap_init_type........0x40
>>                 vl_high_limit...........0x0
>>                 vl_arb_high_cap.........0x8
>>                 vl_arb_low_cap..........0x8
>>                 init_rep_mtu_cap........0x4
>>                 vl_stall_life...........0xF2
>>                 vl_enforce..............0x40
>>                 m_key_violations........0x0
>>                 p_key_violations........0x0
>>                 q_key_violations........0x0
>>                 guid_cap................0x0
>>                 client_reregister.......0x0
>>                 subnet_timeout..........0x0
>>                 resp_time_value.........0x0
>>                 error_threshold.........0x88
>> Mar 19 18:37:03 343481 [45007960] -> Capabilities Mask:
>> Mar 19 18:37:03 343532 [45007960] -> osm_pi_rcv_process_set: Received error 
>> status 0x1c for SetResp() during ACTIVE transition
>> Mar 19 18:37:03 343537 [45007960] -> PortInfo dump:
>>                 port number.............0x12
>>                 node_guid...............0x0005ad0000027c84
>>                 port_guid...............0x0005ad0000027c84
>>                 m_key...................0x0000000000000000
>>                 subnet_prefix...........0x0000000000000000
>>                 base_lid................0x0
>>                 master_sm_base_lid......0x0
>>                 capability_mask.........0x0
>>                 diag_code...............0x0
>>                 m_key_lease_period......0x0
>>                 local_port_num..........0x11
>>                 link_width_enabled......0x2
>>                 link_width_supported....0x3
>>                 link_width_active.......0x2
>>                 link_speed_supported....0x1
>>                 port_state..............INIT
>>                 state_info2.............0x52
>>                 m_key_protect_bits......0x0
>>                 lmc.....................0x0
>>                 link_speed..............0x11
>>                 mtu_smsl................0x40
>>                 vl_cap_init_type........0x40
>>                 vl_high_limit...........0x0
>>                 vl_arb_high_cap.........0x8
>>                 vl_arb_low_cap..........0x8
>>                 init_rep_mtu_cap........0x4
>>                 vl_stall_life...........0xF2
>>                 vl_enforce..............0x40
>>                 m_key_violations........0x0
>>                 p_key_violations........0x0
>>                 q_key_violations........0x0
>>                 guid_cap................0x0
>>                 client_reregister.......0x0
>>                 subnet_timeout..........0x0
>>                 resp_time_value.........0x0
>>                 error_threshold.........0x88
>> Mar 19 18:37:03 343555 [45007960] -> Capabilities Mask:
>> Mar 19 18:37:03 348684 [45007960] -> SUBNET UP
>> Mar 19 18:37:03 461748 [44606960] -> __osm_trap_rcv_process_request: 
>> Received Generic Notice type:0x01 num:128 Producer:2 from LID:0x001B 
>> TID:0x0000000000000048
>> Mar 19 18:37:03 461958 [44606960] -> osm_report_notice: Reporting Generic 
>> Notice type:1 num:128 from LID:0x001B 
>> GID:0xfe80000000000000,0x0005ad00000281a7
>> Mar 19 18:37:03 484827 [43C05960] -> __osm_trap_rcv_process_request: 
>> Received Generic Notice type:0x01 num:128 Producer:2 from LID:0x0148 
>> TID:0x000000000000003e
>> Mar 19 18:37:03 486448 [43C05960] -> osm_report_notice: Reporting Generic 
>> Notice type:1 num:128 from LID:0x0148 
>> GID:0xfe80000000000000,0x0005ad00000281b3
>> Mar 19 18:37:03 528040 [43204960] -> __osm_trap_rcv_process_request: 
>> Received Generic Notice type:0x01 num:128 Producer:2 from LID:0x001B 
>> TID:0x0000000000000049
>> Mar 19 18:37:03 528154 [43204960] -> osm_report_notice: Reporting Generic 
>> Notice type:1 num:128 from LID:0x001B 
>> GID:0xfe80000000000000,0x0005ad00000281a7
>> Mar 19 18:37:03 580196 [43C05960] -> __osm_trap_rcv_process_request: 
>> Received Generic Notice type:0x01 num:128 Producer:2 from LID:0x001B 
>> TID:0x000000000000004a
>> Mar 19 18:37:03 580534 [43C05960] -> osm_report_notice: Reporting Generic 
>> Notice type:1 num:128 from LID:0x001B 
>> GID:0xfe80000000000000,0x0005ad00000281a7
>> Mar 19 18:37:03 599784 [44606960] -> __osm_trap_rcv_process_request: 
>> Received Generic Notice type:0x01 num:128 Producer:2 from LID:0x001B 
>> TID:0x000000000000004b
>> Mar 19 18:37:03 599879 [44606960] -> osm_report_notice: Reporting Generic 
>> Notice type:1 num:128 from LID:0x001B 
>> GID:0xfe80000000000000,0x0005ad00000281a7
>> Mar 19 18:37:03 621883 [45A08960] -> __osm_trap_rcv_process_request: 
>> Received Generic Notice type:0x01 num:128 Producer:2 from LID:0x001B 
>> TID:0x000000000000004c
>> Mar 19 18:37:03 621940 [45A08960] -> osm_report_notice: Reporting Generic 
>> Notice type:1 num:128 from LID:0x001B 
>> GID:0xfe80000000000000,0x0005ad00000281a7
>> Mar 19 18:37:03 707894 [43C05960] -> osm_ucast_mgr_process: Min Hop Tables 
>> configured on all switches
>> Mar 19 18:37:03 764678 [43204960] -> SUBNET UP
>> Mar 19 18:37:03 783783 [41401960] -> __osm_trap_rcv_process_request: 
>> Received Generic Notice type:0x01 num:128 Producer:2 from LID:0x001B 
>> TID:0x000000000000004d
>> Mar 19 18:37:03 783844 [41401960] -> osm_report_notice: Reporting Generic 
>> Notice type:1 num:128 from LID:0x001B 
>> GID:0xfe80000000000000,0x0005ad00000281a7
>> Mar 19 18:37:04 000228 [43204960] -> __osm_trap_rcv_process_request: 
>> Received Generic Notice type:0x01 num:128 Producer:2 from LID:0x001B 
>> TID:0x000000000000004e
>> Mar 19 18:37:04 000628 [43204960] -> osm_report_notice: Reporting Generic 
>> Notice type:1 num:128 from LID:0x001B 
>> GID:0xfe80000000000000,0x0005ad00000281a7
>> Mar 19 18:37:04 022198 [43C05960] -> __osm_trap_rcv_process_request: 
>> Received Generic Notice type:0x01 num:128 Producer:2 from LID:0x001B 
>> TID:0x000000000000004f
>> Mar 19 18:37:04 022299 [43C05960] -> osm_report_notice: Reporting Generic 
>> Notice type:1 num:128 from LID:0x001B 
>> GID:0xfe80000000000000,0x0005ad00000281a7
>> Mar 19 18:37:04 043985 [42803960] -> __osm_trap_rcv_process_request: 
>> Received Generic Notice type:0x01 num:128 Producer:2 from LID:0x001B 
>> TID:0x0000000000000050
>> Mar 19 18:37:04 044052 [42803960] -> osm_report_notice: Reporting Generic 
>> Notice type:1 num:128 from LID:0x001B 
>> GID:0xfe80000000000000,0x0005ad00000281a7
>> Mar 19 18:37:04 155809 [45A08960] -> osm_ucast_mgr_process: Min Hop Tables 
>> configured on all switches
>> Mar 19 18:37:04 210448 [41401960] -> SUBNET UP
>> Mar 19 18:37:04 504490 [43204960] -> __osm_trap_rcv_process_request: 
>> Received Generic Notice type:0x01 num:128 Producer:2 from LID:0x0152 
>> TID:0x0000000000000017
>> Mar 19 18:37:04 504569 [43204960] -> __osm_trap_rcv_process_request: ERR 
>> 3804: Received trap 24 times consecutively
>> Mar 19 18:37:04 570084 [42803960] -> osm_ucast_mgr_process: Min Hop Tables 
>> configured on all switches
>> Mar 19 18:37:04 626298 [43C05960] -> SUBNET UP
>> Mar 19 18:37:54 424084 [41E02960] -> __osm_trap_rcv_process_request: 
>> Received Generic Notice type:0x01 num:128 Producer:2 from LID:0x001B 
>> TID:0x0000000000000051
>> Mar 19 18:37:54 424430 [41E02960] -> osm_report_notice: Reporting Generic 
>> Notice type:1 num:128 from LID:0x001B 
>> GID:0xfe80000000000000,0x0005ad00000281a7
>> Mar 19 18:37:54 424457 [41E02960] -> __osm_trap_rcv_process_request: 
>> Received Generic Notice type:0x01 num:128 Producer:2 from LID:0x0148 
>> TID:0x000000000000003f
>> Mar 19 18:37:54 424522 [41E02960] -> osm_report_notice: Reporting Generic 
>> Notice type:1 num:128 from LID:0x0148 
>> GID:0xfe80000000000000,0x0005ad00000281b3
>> Mar 19 18:37:54 722515 [44606960] -> osm_report_notice: Reporting Generic 
>> Notice type:3 num:65 from LID:0x0092 
>> GID:0xfe80000000000000,0x0005ad0000024bbb
>> Mar 19 18:37:54 722536 [44606960] -> Removed port with 
>> GUID:0x0005ad0000024e0b LID range [0xB3,0xB3] of node:saguaro-23-4 HCA-1
>> Mar 19 18:37:54 722558 [44606960] -> osm_report_notice: Reporting Generic 
>> Notice type:3 num:65 from LID:0x0092 
>> GID:0xfe80000000000000,0x0005ad0000024bbb
>> Mar 19 18:37:54 722565 [44606960] -> Removed port with 
>> GUID:0x0005ad000002510b LID range [0xB5,0xB5] of node:saguaro-23-6 HCA-1
>> Mar 19 18:37:54 722587 [44606960] -> osm_report_notice: Reporting Generic 
>> Notice type:3 num:65 from LID:0x0092 
>> GID:0xfe80000000000000,0x0005ad0000024bbb
>> Mar 19 18:37:54 722594 [44606960] -> Removed port with 
>> GUID:0x0005ad000002511b LID range [0xA6,0xA6] of node:saguaro-22-1 HCA-1
>> Mar 19 18:37:54 722636 [44606960] -> osm_report_notice: Reporting Generic 
>> Notice type:3 num:65 from LID:0x0092 
>> GID:0xfe80000000000000,0x0005ad0000024bbb
>> Mar 19 18:37:54 722641 [44606960] -> Removed port with 
>> GUID:0x0005ad0000024b27 LID range [0xAF,0xAF] of node:saguaro-23-0 HCA-1
>> Mar 19 18:37:54 722658 [44606960] -> osm_report_notice: Reporting Generic 
>> Notice type:3 num:65 from LID:0x0092 
>> GID:0xfe80000000000000,0x0005ad0000024bbb
>> Mar 19 18:37:54 722663 [44606960] -> Removed port with 
>> GUID:0x0005ad0000025043 LID range [0xB4,0xB4] of node:saguaro-23-5 HCA-1
>> Mar 19 18:37:54 722679 [44606960] -> osm_report_notice: Reporting Generic 
>> Notice type:3 num:65 from LID:0x0092 
>> GID:0xfe80000000000000,0x0005ad0000024bbb
>> Mar 19 18:37:54 722684 [44606960] -> Removed port with 
>> GUID:0x0005ad0000024d47 LID range [0xB6,0xB6] of node:saguaro-23-7 HCA-1
>> Mar 19 18:37:54 722701 [44606960] -> osm_report_notice: Reporting Generic 
>> Notice type:3 num:65 from LID:0x0092 
>> GID:0xfe80000000000000,0x0005ad0000024bbb
>> Mar 19 18:37:54 722706 [44606960] -> Removed port with 
>> GUID:0x0005ad0000024d6b LID range [0xB8,0xB8] of node:saguaro-23-9 HCA-1
>> Mar 19 18:37:54 722723 [44606960] -> osm_report_notice: Reporting Generic 
>> Notice type:3 num:65 from LID:0x0092 
>> GID:0xfe80000000000000,0x0005ad0000024bbb
>> Mar 19 18:37:54 722728 [44606960] -> Removed port with 
>> GUID:0x0005ad0000024977 LID range [0xA9,0xA9] of node:saguaro-22-4 HCA-1
>> Mar 19 18:37:54 722875 [44606960] -> osm_report_notice: Reporting Generic 
>> Notice type:3 num:65 from LID:0x0092 
>> GID:0xfe80000000000000,0x0005ad0000024bbb
>> Mar 19 18:37:54 722880 [44606960] -> Removed port with 
>> GUID:0x0005ad0000027c84 LID range [0x152,0x152] of node:Topspin Switch TS120
>> Mar 19 18:37:54 722909 [44606960] -> osm_report_notice: Reporting Generic 
>> Notice type:3 num:65 from LID:0x0092 
>> GID:0xfe80000000000000,0x0005ad0000024bbb
>> Mar 19 18:37:54 722915 [44606960] -> Removed port with 
>> GUID:0x0005ad0000024d8b LID range [0xB7,0xB7] of node:saguaro-23-8 HCA-1
>> Mar 19 18:37:54 722929 [44606960] -> osm_report_notice: Reporting Generic 
>> Notice type:3 num:65 from LID:0x0092 
>> GID:0xfe80000000000000,0x0005ad0000024bbb
>> Mar 19 18:37:54 722934 [44606960] -> Removed port with 
>> GUID:0x0005ad000002498f LID range [0xA8,0xA8] of node:saguaro-22-3 HCA-1
>> Mar 19 18:37:54 722949 [44606960] -> osm_report_notice: Reporting Generic 
>> Notice type:3 num:65 from LID:0x0092 
>> GID:0xfe80000000000000,0x0005ad0000024bbb
>> Mar 19 18:37:54 722955 [44606960] -> Removed port with 
>> GUID:0x0005ad0000024c9b LID range [0xA7,0xA7] of node:saguaro-22-2 HCA-1
>> Mar 19 18:37:54 722970 [44606960] -> osm_report_notice: Reporting Generic 
>> Notice type:3 num:65 from LID:0x0092 
>> GID:0xfe80000000000000,0x0005ad0000024bbb
>> Mar 19 18:37:54 722975 [44606960] -> Removed port with 
>> GUID:0x0005ad0000024da7 LID range [0xB0,0xB0] of node:saguaro-23-1 HCA-1
>> Mar 19 18:37:54 722992 [44606960] -> osm_report_notice: Reporting Generic 
>> Notice type:3 num:65 from LID:0x0092 
>> GID:0xfe80000000000000,0x0005ad0000024bbb
>> Mar 19 18:37:54 722997 [44606960] -> Removed port with 
>> GUID:0x0005ad0000024cbb LID range [0xB2,0xB2] of node:saguaro-23-3 HCA-1
>> Mar 19 18:37:54 723012 [44606960] -> osm_report_notice: Reporting Generic 
>> Notice type:3 num:65 from LID:0x0092 
>> GID:0xfe80000000000000,0x0005ad0000024bbb
>> Mar 19 18:37:54 723073 [44606960] -> Removed port with 
>> GUID:0x0005ad00000249d3 LID range [0xB1,0xB1] of node:saguaro-23-2 HCA-1
>> Mar 19 18:37:54 723090 [44606960] -> osm_report_notice: Reporting Generic 
>> Notice type:3 num:65 from LID:0x0092 
>> GID:0xfe80000000000000,0x0005ad0000024bbb
>> Mar 19 18:37:54 723095 [44606960] -> Removed port with 
>> GUID:0x0005ad0000024feb LID range [0x153,0x153] of node:saguaro-22-5 HCA-1
>> Mar 19 18:37:54 723111 [44606960] -> osm_report_notice: Reporting Generic 
>> Notice type:3 num:65 from LID:0x0092 
>> GID:0xfe80000000000000,0x0005ad0000024bbb
>> Mar 19 18:37:54 723116 [44606960] -> Removed port with 
>> GUID:0x0005ad0000024afb LID range [0xA5,0xA5] of node:saguaro-22-0 HCA-1
>> Mar 19 18:37:54 756302 [44606960] -> osm_ucast_mgr_process: Min Hop Tables 
>> configured on all switches
>> Mar 19 18:37:54 806787 [45A08960] -> SUBNET UP
>> Mar 19 18:37:55 149566 [41E02960] -> osm_ucast_mgr_process: Min Hop Tables 
>> configured on all switches
>> Mar 19 18:37:55 198855 [41401960] -> SUBNET UP
>> Mar 19 18:38:48 131054 [41E02960] -> __osm_trap_rcv_process_request: 
>> Received Generic Notice type:0x01 num:128 Producer:2 from LID:0x0148 
>> TID:0x0000000000000040
>> Mar 19 18:38:48 131349 [41E02960] -> osm_report_notice: Reporting Generic 
>> Notice type:1 num:128 from LID:0x0148 
>> GID:0xfe80000000000000,0x0005ad00000281b3
>> Mar 19 18:38:48 137230 [44606960] -> __osm_trap_rcv_process_request: 
>> Received Generic Notice type:0x01 num:128 Producer:2 from LID:0x001B 
>> TID:0x0000000000000052
>> Mar 19 18:38:48 137268 [45007960] -> __osm_trap_rcv_process_request: 
>> Received Generic Notice type:0x01 num:128 Producer:2 from LID:0x0148 
>> TID:0x0000000000000041
>> Mar 19 18:38:48 137395 [44606960] -> osm_report_notice: Reporting Generic 
>> Notice type:1 num:128 from LID:0x001B 
>> GID:0xfe80000000000000,0x0005ad00000281a7
>> Mar 19 18:38:48 137432 [45007960] -> osm_report_notice: Reporting Generic 
>> Notice type:1 num:128 from LID:0x0148 
>> GID:0xfe80000000000000,0x0005ad00000281b3
>> Mar 19 18:38:48 143370 [45A08960] -> __osm_trap_rcv_process_request: 
>> Received Generic Notice type:0x01 num:128 Producer:2 from LID:0x001B 
>> TID:0x0000000000000053
>> Mar 19 18:38:48 144327 [45A08960] -> osm_report_notice: Reporting Generic 
>> Notice type:1 num:128 from LID:0x001B 
>> GID:0xfe80000000000000,0x0005ad00000281a7
>> Mar 19 18:38:48 529052 [41E02960] -> osm_report_notice: Reporting Generic 
>> Notice type:3 num:64 from LID:0x0092 
>> GID:0xfe80000000000000,0x0005ad0000024bbb
>> Mar 19 18:38:48 529065 [41E02960] -> Discovered new port with 
>> GUID:0x0005ad0000027c84 LID range [0x152,0x152] of node:Topspin Switch TS120
>> Mar 19 18:38:48 529071 [41E02960] -> osm_report_notice: Reporting Generic 
>> Notice type:3 num:64 from LID:0x0092 
>> GID:0xfe80000000000000,0x0005ad0000024bbb
>> Mar 19 18:38:48 529078 [41E02960] -> Discovered new port with 
>> GUID:0x0005ad0000024b27 LID range [0xAF,0xAF] of node:saguaro-23-0 HCA-1
>> Mar 19 18:38:48 529083 [41E02960] -> osm_report_notice: Reporting Generic 
>> Notice type:3 num:64 from LID:0x0092 
>> GID:0xfe80000000000000,0x0005ad0000024bbb
>> Mar 19 18:38:48 529090 [41E02960] -> Discovered new port with 
>> GUID:0x0005ad00000249d3 LID range [0xB1,0xB1] of node:saguaro-23-2 HCA-1
>> Mar 19 18:38:48 529095 [41E02960] -> osm_report_notice: Reporting Generic 
>> Notice type:3 num:64 from LID:0x0092 
>> GID:0xfe80000000000000,0x0005ad0000024bbb
>> Mar 19 18:38:48 529101 [41E02960] -> Discovered new port with 
>> GUID:0x0005ad0000024da7 LID range [0xB0,0xB0] of node:saguaro-23-1 HCA-1
>> Mar 19 18:38:48 529106 [41E02960] -> osm_report_notice: Reporting Generic 
>> Notice type:3 num:64 from LID:0x0092 
>> GID:0xfe80000000000000,0x0005ad0000024bbb
>> Mar 19 18:38:48 529113 [41E02960] -> Discovered new port with 
>> GUID:0x0005ad0000024cbb LID range [0xB2,0xB2] of node:saguaro-23-3 HCA-1
>> Mar 19 18:38:48 529118 [41E02960] -> osm_report_notice: Reporting Generic 
>> Notice type:3 num:64 from LID:0x0092 
>> GID:0xfe80000000000000,0x0005ad0000024bbb
>> Mar 19 18:38:48 529124 [41E02960] -> Discovered new port with 
>> GUID:0x0005ad0000024e0b LID range [0xB3,0xB3] of node:saguaro-23-4 HCA-1
>> Mar 19 18:38:48 529129 [41E02960] -> osm_report_notice: Reporting Generic 
>> Notice type:3 num:64 from LID:0x0092 
>> GID:0xfe80000000000000,0x0005ad0000024bbb
>> Mar 19 18:38:48 529136 [41E02960] -> Discovered new port with 
>> GUID:0x0005ad0000025043 LID range [0xB4,0xB4] of node:saguaro-23-5 HCA-1
>> Mar 19 18:38:48 529141 [41E02960] -> osm_report_notice: Reporting Generic 
>> Notice type:3 num:64 from LID:0x0092 
>> GID:0xfe80000000000000,0x0005ad0000024bbb
>> Mar 19 18:38:48 529147 [41E02960] -> Discovered new port with 
>> GUID:0x0005ad000002510b LID range [0xB5,0xB5] of node:saguaro-23-6 HCA-1
>> Mar 19 18:38:48 529152 [41E02960] -> osm_report_notice: Reporting Generic 
>> Notice type:3 num:64 from LID:0x0092 
>> GID:0xfe80000000000000,0x0005ad0000024bbb
>> Mar 19 18:38:48 529159 [41E02960] -> Discovered new port with 
>> GUID:0x0005ad0000024d47 LID range [0xB6,0xB6] of node:saguaro-23-7 HCA-1
>> Mar 19 18:38:48 529164 [41E02960] -> osm_report_notice: Reporting Generic 
>> Notice type:3 num:64 from LID:0x0092 
>> GID:0xfe80000000000000,0x0005ad0000024bbb
>> Mar 19 18:38:48 529170 [41E02960] -> Discovered new port with 
>> GUID:0x0005ad0000024d8b LID range [0xB7,0xB7] of node:saguaro-23-8 HCA-1
>> Mar 19 18:38:48 529175 [41E02960] -> osm_report_notice: Reporting Generic 
>> Notice type:3 num:64 from LID:0x0092 
>> GID:0xfe80000000000000,0x0005ad0000024bbb
>> Mar 19 18:38:48 529182 [41E02960] -> Discovered new port with 
>> GUID:0x0005ad0000024d6b LID range [0xB8,0xB8] of node:saguaro-23-9 HCA-1
>> Mar 19 18:38:48 529186 [41E02960] -> osm_report_notice: Reporting Generic 
>> Notice type:3 num:64 from LID:0x0092 
>> GID:0xfe80000000000000,0x0005ad0000024bbb
>> Mar 19 18:38:48 529193 [41E02960] -> Discovered new port with 
>> GUID:0x0005ad0000024afb LID range [0xA5,0xA5] of node:saguaro-22-0 HCA-1
>> Mar 19 18:38:48 529198 [41E02960] -> osm_report_notice: Reporting Generic 
>> Notice type:3 num:64 from LID:0x0092 
>> GID:0xfe80000000000000,0x0005ad0000024bbb
>> Mar 19 18:38:48 529204 [41E02960] -> Discovered new port with 
>> GUID:0x0005ad000002511b LID range [0xA6,0xA6] of node:saguaro-22-1 HCA-1
>> Mar 19 18:38:48 529209 [41E02960] -> osm_report_notice: Reporting Generic 
>> Notice type:3 num:64 from LID:0x0092 
>> GID:0xfe80000000000000,0x0005ad0000024bbb
>> Mar 19 18:38:48 529216 [41E02960] -> Discovered new port with 
>> GUID:0x0005ad0000024c9b LID range [0xA7,0xA7] of node:saguaro-22-2 HCA-1
>> Mar 19 18:38:48 529271 [41E02960] -> osm_report_notice: Reporting Generic 
>> Notice type:3 num:64 from LID:0x0092 
>> GID:0xfe80000000000000,0x0005ad0000024bbb
>> Mar 19 18:38:48 529277 [41E02960] -> Discovered new port with 
>> GUID:0x0005ad000002498f LID range [0xA8,0xA8] of node:saguaro-22-3 HCA-1
>> Mar 19 18:38:48 529281 [41E02960] -> osm_report_notice: Reporting Generic 
>> Notice type:3 num:64 from LID:0x0092 
>> GID:0xfe80000000000000,0x0005ad0000024bbb
>> Mar 19 18:38:48 529286 [41E02960] -> Discovered new port with 
>> GUID:0x0005ad0000024977 LID range [0xA9,0xA9] of node:saguaro-22-4 HCA-1
>> Mar 19 18:38:48 529290 [41E02960] -> osm_report_notice: Reporting Generic 
>> Notice type:3 num:64 from LID:0x0092 
>> GID:0xfe80000000000000,0x0005ad0000024bbb
>> Mar 19 18:38:48 529294 [41E02960] -> Discovered new port with 
>> GUID:0x0005ad0000024feb LID range [0x153,0x153] of node:saguaro-22-5 HCA-1
>> Mar 19 18:38:48 560082 [41E02960] -> osm_ucast_mgr_process: Min Hop Tables 
>> configured on all switches
>> Mar 19 18:38:48 630464 [43204960] -> SUBNET UP
>> Mar 19 18:38:49 018498 [44606960] -> osm_ucast_mgr_process: Min Hop Tables 
>> configured on all switches
>> Mar 19 18:38:49 073355 [45007960] -> SUBNET UP
>> Mar 19 18:39:04 189829 [45007960] -> __osm_trap_rcv_process_request: 
>> Received Generic Notice type:0x01 num:128 Producer:2 from LID:0x0152 
>> TID:0x0000000000000000
>> Mar 19 18:39:04 190072 [45007960] -> osm_report_notice: Reporting Generic 
>> Notice type:1 num:128 from LID:0x0152 
>> GID:0xfe80000000000000,0x0005ad0000027c84
>> Mar 19 18:39:04 307827 [44606960] -> __osm_trap_rcv_process_request: 
>> Received Generic Notice type:0x01 num:128 Producer:2 from LID:0x0152 
>> TID:0x0000000000000001
>> Mar 19 18:39:04 307940 [44606960] -> osm_report_notice: Reporting Generic 
>> Notice type:1 num:128 from LID:0x0152 
>> GID:0xfe80000000000000,0x0005ad0000027c84
>> Mar 19 18:39:04 330104 [44606960] -> __osm_trap_rcv_process_request: 
>> Received Generic Notice type:0x01 num:128 Producer:2 from LID:0x0152 
>> TID:0x0000000000000002
>> Mar 19 18:39:04 330210 [44606960] -> osm_report_notice: Reporting Generic 
>> Notice type:1 num:128 from LID:0x0152 
>> GID:0xfe80000000000000,0x0005ad0000027c84
>> Mar 19 18:39:04 468676 [41401960] -> __osm_trap_rcv_process_request: 
>> Received Generic Notice type:0x01 num:128 Producer:2 from LID:0x0152 
>> TID:0x0000000000000003
>> Mar 19 18:39:04 468758 [41401960] -> osm_report_notice: Reporting Generic 
>> Notice type:1 num:128 from LID:0x0152 
>> GID:0xfe80000000000000,0x0005ad0000027c84
>> Mar 19 18:39:04 680305 [42803960] -> __osm_trap_rcv_process_request: 
>> Received Generic Notice type:0x01 num:128 Producer:2 from LID:0x0152 
>> TID:0x0000000000000004
>> Mar 19 18:39:04 680400 [42803960] -> osm_report_notice: Reporting Generic 
>> Notice type:1 num:128 from LID:0x0152 
>> GID:0xfe80000000000000,0x0005ad0000027c84
>> Mar 19 18:39:04 702144 [41E02960] -> __osm_trap_rcv_process_request: 
>> Received Generic Notice type:0x01 num:128 Producer:2 from LID:0x0152 
>> TID:0x0000000000000005
>> Mar 19 18:39:04 702286 [41E02960] -> osm_report_notice: Reporting Generic 
>> Notice type:1 num:128 from LID:0x0152 
>> GID:0xfe80000000000000,0x0005ad0000027c84
>> Mar 19 18:39:04 704346 [43204960] -> osm_report_notice: Reporting Generic 
>> Notice type:3 num:65 from LID:0x0092 
>> GID:0xfe80000000000000,0x0005ad0000024bbb
>> Mar 19 18:39:04 704354 [43204960] -> Removed port with 
>> GUID:0x0005ad00000249d3 LID range [0xB1,0xB1] of node:saguaro-23-2 HCA-1
>> Mar 19 18:39:04 739059 [43204960] -> osm_ucast_mgr_process: Min Hop Tables 
>> configured on all switches
>> Mar 19 18:39:04 739896 [41E02960] -> __osm_trap_rcv_process_request: 
>> Received Generic Notice type:0x01 num:128 Producer:2 from LID:0x0152 
>> TID:0x0000000000000006
>> Mar 19 18:39:04 783807 [41E02960] -> osm_report_notice: Reporting Generic 
>> Notice type:1 num:128 from LID:0x0152 
>> GID:0xfe80000000000000,0x0005ad0000027c84
>> Mar 19 18:39:04 797411 [44606960] -> SUBNET UP
>> Mar 19 18:39:04 849970 [42803960] -> __osm_trap_rcv_process_request: 
>> Received Generic Notice type:0x01 num:128 Producer:2 from LID:0x0152 
>> TID:0x0000000000000007
>> Mar 19 18:39:04 850195 [42803960] -> osm_report_notice: Reporting Generic 
>> Notice type:1 num:128 from LID:0x0152 
>> GID:0xfe80000000000000,0x0005ad0000027c84
>> Mar 19 18:39:04 853735 [43C05960] -> __osm_trap_rcv_process_request: 
>> Received Generic Notice type:0x01 num:128 Producer:2 from LID:0x0152 
>> TID:0x0000000000000008
>> Mar 19 18:39:04 853809 [43C05960] -> osm_report_notice: Reporting Generic 
>> Notice type:1 num:128 from LID:0x0152 
>> GID:0xfe80000000000000,0x0005ad0000027c84
>> Mar 19 18:39:04 897727 [43C05960] -> __osm_trap_rcv_process_request: 
>> Received Generic Notice type:0x01 num:128 Producer:2 from LID:0x0152 
>> TID:0x0000000000000009
>> Mar 19 18:39:04 897860 [43C05960] -> osm_report_notice: Reporting Generic 
>> Notice type:1 num:128 from LID:0x0152 
>> GID:0xfe80000000000000,0x0005ad0000027c84
>> Mar 19 18:39:04 901577 [41401960] -> __osm_trap_rcv_process_request: 
>> Received Generic Notice type:0x01 num:128 Producer:2 from LID:0x0152 
>> TID:0x000000000000000a
>> Mar 19 18:39:04 901719 [41401960] -> osm_report_notice: Reporting Generic 
>> Notice type:1 num:128 from LID:0x0152 
>> GID:0xfe80000000000000,0x0005ad0000027c84
>> Mar 19 18:39:04 923271 [45007960] -> __osm_trap_rcv_process_request: 
>> Received Generic Notice type:0x01 num:128 Producer:2 from LID:0x0152 
>> TID:0x000000000000000b
>> Mar 19 18:39:04 923377 [45007960] -> __osm_trap_rcv_process_request: ERR 
>> 3804: Received trap 11 times consecutively
>> Mar 19 18:39:05 106246 [45007960] -> __osm_trap_rcv_process_request: 
>> Received Generic Notice type:0x01 num:128 Producer:2 from LID:0x0152 
>> TID:0x000000000000000c
>> Mar 19 18:39:05 106314 [45007960] -> __osm_trap_rcv_process_request: ERR 
>> 3804: Received trap 12 times consecutively
>> Mar 19 18:39:05 178215 [44606960] -> __osm_trap_rcv_process_request: 
>> Received Generic Notice type:0x01 num:128 Producer:2 from LID:0x0152 
>> TID:0x000000000000000d
>> Mar 19 18:39:05 178258 [44606960] -> __osm_trap_rcv_process_request: ERR 
>> 3804: Received trap 13 times consecutively
>> Mar 19 18:39:05 272913 [42803960] -> __osm_trap_rcv_process_request: 
>> Received Generic Notice type:0x01 num:128 Producer:2 from LID:0x0152 
>> TID:0x000000000000000e
>> Mar 19 18:39:05 272983 [42803960] -> __osm_trap_rcv_process_request: ERR 
>> 3804: Received trap 14 times consecutively
>> Mar 19 18:39:05 339633 [43204960] -> __osm_trap_rcv_process_request: 
>> Received Generic Notice type:0x01 num:128 Producer:2 from LID:0x0152 
>> TID:0x000000000000000f
>> Mar 19 18:39:05 339679 [43204960] -> __osm_trap_rcv_process_request: ERR 
>> 3804: Received trap 15 times consecutively
>> Mar 19 18:39:05 469093 [41401960] -> __osm_trap_rcv_process_request: 
>> Received Generic Notice type:0x01 num:128 Producer:2 from LID:0x0152 
>> TID:0x0000000000000010
>> Mar 19 18:39:05 469145 [41401960] -> __osm_trap_rcv_process_request: ERR 
>> 3804: Received trap 16 times consecutively
>> Mar 19 18:39:05 484587 [44606960] -> __osm_trap_rcv_process_request: 
>> Received Generic Notice type:0x01 num:128 Producer:2 from LID:0x0152 
>> TID:0x0000000000000011
>> Mar 19 18:39:05 484633 [44606960] -> __osm_trap_rcv_process_request: ERR 
>> 3804: Received trap 17 times consecutively
>> Mar 19 18:39:05 574251 [43C05960] -> __osm_trap_rcv_process_request: 
>> Received Generic Notice type:0x01 num:128 Producer:2 from LID:0x0152 
>> TID:0x0000000000000012
>> Mar 19 18:39:05 574301 [43C05960] -> __osm_trap_rcv_process_request: ERR 
>> 3804: Received trap 18 times consecutively
>> Mar 19 18:39:05 602665 [41E02960] -> __osm_trap_rcv_process_request: 
>> Received Generic Notice type:0x01 num:128 Producer:2 from LID:0x0152 
>> TID:0x0000000000000013
>> Mar 19 18:39:05 602700 [41E02960] -> __osm_trap_rcv_process_request: ERR 
>> 3804: Received trap 19 times consecutively
>> Mar 19 18:39:05 646331 [45007960] -> __osm_trap_rcv_process_request: 
>> Received Generic Notice type:0x01 num:128 Producer:2 from LID:0x0152 
>> TID:0x0000000000000014
>> Mar 19 18:39:05 646369 [45007960] -> __osm_trap_rcv_process_request: ERR 
>> 3804: Received trap 20 times consecutively
>> Mar 19 18:39:05 834613 [41E02960] -> __osm_trap_rcv_process_request: 
>> Received Generic Notice type:0x01 num:128 Producer:2 from LID:0x0152 
>> TID:0x0000000000000015
>> Mar 19 18:39:05 834685 [41E02960] -> __osm_trap_rcv_process_request: ERR 
>> 3804: Received trap 21 times consecutively
>> Mar 19 18:39:05 851128 [45007960] -> __osm_trap_rcv_process_request: 
>> Received Generic Notice type:0x01 num:128 Producer:2 from LID:0x0152 
>> TID:0x0000000000000016
>> Mar 19 18:39:05 851166 [45007960] -> __osm_trap_rcv_process_request: ERR 
>> 3804: Received trap 22 times consecutively
>> Mar 19 18:39:05 875540 [45A08960] -> __osm_trap_rcv_process_request: 
>> Received Generic Notice type:0x01 num:128 Producer:2 from LID:0x0152 
>> TID:0x0000000000000017
>> Mar 19 18:39:05 875592 [45A08960] -> __osm_trap_rcv_process_request: ERR 
>> 3804: Received trap 23 times consecutively
>> Mar 19 18:39:05 897378 [42803960] -> __osm_trap_rcv_process_request: 
>> Received Generic Notice type:0x01 num:128 Producer:2 from LID:0x0152 
>> TID:0x0000000000000018
>> Mar 19 18:39:05 897424 [42803960] -> __osm_trap_rcv_process_request: ERR 
>> 3804: Received trap 24 times consecutively
>> Mar 19 18:39:05 907232 [4780B960] -> umad_receiver: ERR 5409: send completed 
>> with error (method=0x1 attr=0x15 trans_id=0x124ef0001c2fe) -- dropping
>> Mar 19 18:39:05 907249 [4780B960] -> umad_receiver: ERR 5411: DR SMP
>> Mar 19 18:39:05 907259 [4780B960] -> __osm_sm_mad_ctrl_send_err_cb: ERR 
>> 3113: MAD completed in error (IB_TIMEOUT)
>> Mar 19 18:39:05 907295 [4780B960] -> SMP dump:
>>                 base_ver................0x1
>>                 mgmt_class..............0x81
>>                 class_ver...............0x1
>>                 method..................0x1 (SubnGet)
>>                 D bit...................0x0
>>                 status..................0x0
>>                 hop_ptr.................0x0
>>                 hop_count...............0x6
>>                 trans_id................0x1c2fe
>>                 attr_id.................0x15 (PortInfo)
>>                 resv....................0x0
>>                 attr_mod................0x1
>>                 m_key...................0x0000000000000000
>>                 dr_slid.................0xFFFF
>>                 dr_dlid.................0xFFFF
>> 
>>                 Initial path: [0][1][11][1][6][16][8]
>>                 Return path:  [0][0][0][0][0][0][0]
>>                 Reserved:     [0][0][0][0][0][0][0]
>> 
>>                 00 00 00 00 00 00 00 00   00 00 00 00 00 00 00 00
>> 
>>                 00 00 00 00 00 00 00 00   00 00 00 00 00 00 00 00
>> 
>>                 00 00 00 00 00 00 00 00   00 00 00 00 00 00 00 00
>> 
>>                 00 00 00 00 00 00 00 00   00 00 00 00 00 00 00 00
>> 
>> Mar 19 18:39:05 907372 [41401960] -> osm_report_notice: Reporting Generic 
>> Notice type:3 num:65 from LID:0x0092 
>> GID:0xfe80000000000000,0x0005ad0000024bbb
>> Mar 19 18:39:05 907384 [41401960] -> Removed port with 
>> GUID:0x0005ad0000024e0b LID range [0xB3,0xB3] of node:saguaro-23-4 HCA-1
>> Mar 19 18:39:05 907407 [41401960] -> osm_report_notice: Reporting Generic 
>> Notice type:3 num:65 from LID:0x0092 
>> GID:0xfe80000000000000,0x0005ad0000024bbb
>> Mar 19 18:39:05 907414 [41401960] -> Removed port with 
>> GUID:0x0005ad0000025043 LID range [0xB4,0xB4] of node:saguaro-23-5 HCA-1
>> Mar 19 18:39:05 907480 [41401960] -> osm_report_notice: Reporting Generic 
>> Notice type:3 num:65 from LID:0x0092 
>> GID:0xfe80000000000000,0x0005ad0000024bbb
>> Mar 19 18:39:05 907485 [41401960] -> Removed port with 
>> GUID:0x0005ad0000024cbb LID range [0xB2,0xB2] of node:saguaro-23-3 HCA-1
>> Mar 19 18:39:05 907577 [41401960] -> osm_report_notice: Reporting Generic 
>> Notice type:3 num:65 from LID:0x0092 
>> GID:0xfe80000000000000,0x0005ad0000024bbb
>> Mar 19 18:39:05 907582 [41401960] -> Removed port with 
>> GUID:0x0005ad0000024d47 LID range [0xB6,0xB6] of node:saguaro-23-7 HCA-1
>> Mar 19 18:39:05 907618 [41401960] -> osm_drop_mgr_process: ERR 0108: Unknown 
>> remote side for node 0x0005ad0000027c84 port 8. Adding to light sweep 
>> sampling list
>> Mar 19 18:39:05 907657 [41401960] -> Directed Path Dump of 5 hop path:
>>                 Path = [0][1][11][1][6][16]
>> Mar 19 18:39:05 911559 [43204960] -> osm_report_notice: Reporting Generic 
>> Notice type:3 num:64 from LID:0x0092 
>> GID:0xfe80000000000000,0x0005ad0000024bbb
>> Mar 19 18:39:05 911572 [43204960] -> Discovered new port with 
>> GUID:0x0005ad00000249d3 LID range [0xB1,0xB1] of node:saguaro-23-2 HCA-1
>> Mar 19 18:39:05 927229 [43C05960] -> __osm_trap_rcv_process_request: 
>> Received Generic Notice type:0x01 num:128 Producer:2 from LID:0x0152 
>> TID:0x0000000000000019
>> Mar 19 18:39:05 927285 [43C05960] -> __osm_trap_rcv_process_request: ERR 
>> 3804: Received trap 25 times consecutively
>> Mar 19 18:39:05 942538 [43204960] -> osm_ucast_mgr_process: Min Hop Tables 
>> configured on all switches
>> Mar 19 18:39:06 000027 [41E02960] -> SUBNET UP
>> Mar 19 18:39:06 130255 [43204960] -> __osm_trap_rcv_process_request: 
>> Received Generic Notice type:0x01 num:128 Producer:2 from LID:0x0152 
>> TID:0x000000000000001a
>> Mar 19 18:39:06 130308 [43204960] -> __osm_trap_rcv_process_request: ERR 
>> 3804: Received trap 26 times consecutively
>> Mar 19 18:39:06 131922 [42803960] -> __osm_trap_rcv_process_request: 
>> Received Generic Notice type:0x01 num:128 Producer:2 from LID:0x0148 
>> TID:0x0000000000000042
>> Mar 19 18:39:06 132063 [42803960] -> osm_report_notice: Reporting Generic 
>> Notice type:1 num:128 from LID:0x0148 
>> GID:0xfe80000000000000,0x0005ad00000281b3
>> Mar 19 18:39:06 154579 [43C05960] -> __osm_trap_rcv_process_request: 
>> Received Generic Notice type:0x01 num:128 Producer:2 from LID:0x0152 
>> TID:0x000000000000001b
>> Mar 19 18:39:06 154681 [43C05960] -> __osm_trap_rcv_process_request: ERR 
>> 3804: Received trap 27 times consecutively
>> Mar 19 18:39:06 176248 [44606960] -> __osm_trap_rcv_process_request: 
>> Received Generic Notice type:0x01 num:128 Producer:2 from LID:0x0152 
>> TID:0x000000000000001c
>> Mar 19 18:39:06 176304 [44606960] -> __osm_trap_rcv_process_request: ERR 
>> 3804: Received trap 28 times consecutively
>> Mar 19 18:39:06 198132 [44606960] -> __osm_trap_rcv_process_request: 
>> Received Generic Notice type:0x01 num:128 Producer:2 from LID:0x0152 
>> TID:0x000000000000001d
>> Mar 19 18:39:06 198195 [44606960] -> __osm_trap_rcv_process_request: ERR 
>> 3804: Received trap 29 times consecutively
>> Mar 19 18:39:06 230022 [43C05960] -> __osm_trap_rcv_process_request: 
>> Received Generic Notice type:0x01 num:128 Producer:2 from LID:0x0152 
>> TID:0x000000000000001e
>> Mar 19 18:39:06 230108 [43C05960] -> __osm_trap_rcv_process_request: ERR 
>> 3804: Received trap 30 times consecutively
>> Mar 19 18:39:06 230229 [43204960] -> __osm_trap_rcv_process_request: 
>> Received Generic Notice type:0x01 num:128 Producer:2 from LID:0x0148 
>> TID:0x0000000000000043
>> Mar 19 18:39:06 230311 [43204960] -> osm_report_notice: Reporting Generic 
>> Notice type:1 num:128 from LID:0x0148 
>> GID:0xfe80000000000000,0x0005ad00000281b3
>> Mar 19 18:39:06 399543 [43C05960] -> osm_report_notice: Reporting Generic 
>> Notice type:3 num:64 from LID:0x0092 
>> GID:0xfe80000000000000,0x0005ad0000024bbb
>> Mar 19 18:39:06 399556 [43C05960] -> Discovered new port with 
>> GUID:0x0005ad0000024cbb LID range [0xB2,0xB2] of node:saguaro-23-3 HCA-1
>> Mar 19 18:39:06 399562 [43C05960] -> osm_report_notice: Reporting Generic 
>> Notice type:3 num:64 from LID:0x0092 
>> GID:0xfe80000000000000,0x0005ad0000024bbb
>> Mar 19 18:39:06 399569 [43C05960] -> Discovered new port with 
>> GUID:0x0005ad0000024e0b LID range [0xB3,0xB3] of node:saguaro-23-4 HCA-1
>> Mar 19 18:39:06 399574 [43C05960] -> osm_report_notice: Reporting Generic 
>> Notice type:3 num:64 from LID:0x0092 
>> GID:0xfe80000000000000,0x0005ad0000024bbb
>> Mar 19 18:39:06 399580 [43C05960] -> Discovered new port with 
>> GUID:0x0005ad0000025043 LID range [0xB4,0xB4] of node:saguaro-23-5 HCA-1
>> Mar 19 18:39:06 399585 [43C05960] -> osm_report_notice: Reporting Generic 
>> Notice type:3 num:64 from LID:0x0092 
>> GID:0xfe80000000000000,0x0005ad0000024bbb
>> Mar 19 18:39:06 399592 [43C05960] -> Discovered new port with 
>> GUID:0x0005ad0000024d47 LID range [0xB6,0xB6] of node:saguaro-23-7 HCA-1
>> Mar 19 18:39:06 430598 [43C05960] -> osm_ucast_mgr_process: Min Hop Tables 
>> configured on all switches
>> Mar 19 18:39:06 494689 [44606960] -> SUBNET UP
>> Mar 19 18:39:06 837303 [43204960] -> __osm_trap_rcv_process_request: 
>> Received Generic Notice type:0x01 num:128 Producer:2 from LID:0x0152 
>> TID:0x000000000000001f
>> Mar 19 18:39:06 837446 [43204960] -> __osm_trap_rcv_process_request: ERR 
>> 3804: Received trap 31 times consecutively
>> Mar 19 18:39:06 838528 [43C05960] -> __osm_trap_rcv_process_request: 
>> Received Generic Notice type:0x01 num:128 Producer:2 from LID:0x0148 
>> TID:0x0000000000000044
>> Mar 19 18:39:06 838636 [43C05960] -> osm_report_notice: Reporting Generic 
>> Notice type:1 num:128 from LID:0x0148 
>> GID:0xfe80000000000000,0x0005ad00000281b3
>> Mar 19 18:39:06 876308 [43C05960] -> osm_ucast_mgr_process: Min Hop Tables 
>> configured on all switches
>> Mar 19 18:39:07 028376 [45A08960] -> __osm_trap_rcv_process_request: 
>> Received Generic Notice type:0x01 num:128 Producer:2 from LID:0x0152 
>> TID:0x0000000000000020
>> Mar 19 18:39:07 028459 [45A08960] -> __osm_trap_rcv_process_request: ERR 
>> 3804: Received trap 32 times consecutively
>> Mar 19 18:39:07 028545 [43204960] -> __osm_trap_rcv_process_request: 
>> Received Generic Notice type:0x01 num:128 Producer:2 from LID:0x0148 
>> TID:0x0000000000000045
>> Mar 19 18:39:07 028652 [43204960] -> osm_report_notice: Reporting Generic 
>> Notice type:1 num:128 from LID:0x0148 
>> GID:0xfe80000000000000,0x0005ad00000281b3
>> Mar 19 18:39:07 030190 [45007960] -> __osm_trap_rcv_process_request: 
>> Received Generic Notice type:0x01 num:128 Producer:2 from LID:0x001B 
>> TID:0x0000000000000054
>> Mar 19 18:39:07 030277 [45007960] -> osm_report_notice: Reporting Generic 
>> Notice type:1 num:128 from LID:0x001B 
>> GID:0xfe80000000000000,0x0005ad00000281a7
>> Mar 19 18:39:07 096812 [41401960] -> __osm_trap_rcv_process_request: 
>> Received Generic Notice type:0x01 num:128 Producer:2 from LID:0x0148 
>> TID:0x0000000000000046
>> Mar 19 18:39:07 096959 [41401960] -> osm_report_notice: Reporting Generic 
>> Notice type:1 num:128 from LID:0x0148 
>> GID:0xfe80000000000000,0x0005ad00000281b3
>> Mar 19 18:39:07 111719 [4780B960] -> __osm_sm_mad_ctrl_rcv_callback: ERR 
>> 3111: Error status = 0x1C00
>> Mar 19 18:39:07 111759 [4780B960] -> SMP dump:
>>                 base_ver................0x1
>>                 mgmt_class..............0x81
>>                 class_ver...............0x1
>>                 method..................0x81 (SubnGetResp)
>>                 D bit...................0x1
>>                 status..................0x1C00
>>                 hop_ptr.................0x0
>>                 hop_count...............0x5
>>                 trans_id................0x1dfac
>>                 attr_id.................0x15 (PortInfo)
>>                 resv....................0x0
>>                 attr_mod................0x11
>>                 m_key...................0x0000000000000000
>>                 dr_slid.................0xFFFF
>>                 dr_dlid.................0xFFFF
>> 
>>                 Initial path: [0][1][11][1][4][16]
>>                 Return path:  [0][9][18][D][1][11]
>>                 Reserved:     [0][0][0][0][0][0][0]
>> 
>>                 00 00 00 00 00 00 00 00   00 00 00 00 00 00 00 00
>> 
>>                 00 00 00 00 00 00 00 00   00 00 00 00 11 02 03 02
>> 
>>                 12 52 00 11 40 40 00 08   08 04 F2 40 00 00 00 00
>> 
>>                 00 00 00 00 00 88 00 00   00 00 00 00 00 00 00 00
>> 
>> Mar 19 18:39:07 111810 [41E02960] -> osm_pi_rcv_process_set: Received error 
>> status 0x1c for SetResp() during ACTIVE transition
>> Mar 19 18:39:07 111814 [4780B960] -> __osm_sm_mad_ctrl_rcv_callback: ERR 
>> 3111: Error status = 0x1C00
>> Mar 19 18:39:07 111831 [41E02960] -> PortInfo dump:
>>                 port number.............0x11
>>                 node_guid...............0x0005ad0000027c84
>>                 port_guid...............0x0005ad0000027c84
>>                 m_key...................0x0000000000000000
>>                 subnet_prefix...........0x0000000000000000
>>                 base_lid................0x0
>>                 master_sm_base_lid......0x0
>>                 capability_mask.........0x0
>>                 diag_code...............0x0
>>                 m_key_lease_period......0x0
>>                 local_port_num..........0x11
>>                 link_width_enabled......0x2
>>                 link_width_supported....0x3
>>                 link_width_active.......0x2
>>                 link_speed_supported....0x1
>>                 port_state..............INIT
>>                 state_info2.............0x52
>>                 m_key_protect_bits......0x0
>>                 lmc.....................0x0
>>                 link_speed..............0x11
>>                 mtu_smsl................0x40
>>                 vl_cap_init_type........0x40
>>                 vl_high_limit...........0x0
>>                 vl_arb_high_cap.........0x8
>>                 vl_arb_low_cap..........0x8
>>                 init_rep_mtu_cap........0x4
>>                 vl_stall_life...........0xF2
>>                 vl_enforce..............0x40
>>                 m_key_violations........0x0
>>                 p_key_violations........0x0
>>                 q_key_violations........0x0
>>                 guid_cap................0x0
>>                 client_reregister.......0x0
>>                 subnet_timeout..........0x0
>>                 resp_time_value.........0x0
>>                 error_threshold.........0x88
>> Mar 19 18:39:07 111868 [41E02960] -> Capabilities Mask:
>> Mar 19 18:39:07 111844 [4780B960] -> SMP dump:
>>                 base_ver................0x1
>>                 mgmt_class..............0x81
>>                 class_ver...............0x1
>>                 method..................0x81 (SubnGetResp)
>>                 D bit...................0x1
>>                 status..................0x1C00
>>                 hop_ptr.................0x0
>>                 hop_count...............0x5
>>                 trans_id................0x1dfad
>>                 attr_id.................0x15 (PortInfo)
>>                 resv....................0x0
>>                 attr_mod................0x12
>>                 m_key...................0x0000000000000000
>>                 dr_slid.................0xFFFF
>>                 dr_dlid.................0xFFFF
>> 
>>                 Initial path: [0][1][11][1][4][16]
>>                 Return path:  [0][9][18][D][1][11]
>>                 Reserved:     [0][0][0][0][0][0][0]
>> 
>>                 00 00 00 00 00 00 00 00   00 00 00 00 00 00 00 00
>> 
>>                 00 00 00 00 00 00 00 00   00 00 00 00 11 02 03 02
>> 
>>                 12 52 00 11 40 40 00 08   08 04 F2 40 00 00 00 00
>> 
>>                 00 00 00 00 00 88 00 00   00 00 00 00 00 00 00 00
>> 
>> Mar 19 18:39:07 112011 [41401960] -> osm_pi_rcv_process_set: Received error 
>> status 0x1c for SetResp() during ACTIVE transition
>> Mar 19 18:39:07 112018 [41401960] -> PortInfo dump:
>>                 port number.............0x12
>>                 node_guid...............0x0005ad0000027c84
>>                 port_guid...............0x0005ad0000027c84
>>                 m_key...................0x0000000000000000
>>                 subnet_prefix...........0x0000000000000000
>>                 base_lid................0x0
>>                 master_sm_base_lid......0x0
>>                 capability_mask.........0x0
>>                 diag_code...............0x0
>>                 m_key_lease_period......0x0
>>                 local_port_num..........0x11
>>                 link_width_enabled......0x2
>>                 link_width_supported....0x3
>>                 link_width_active.......0x2
>>                 link_speed_supported....0x1
>>                 port_state..............INIT
>>                 state_info2.............0x52
>>                 m_key_protect_bits......0x0
>>                 lmc.....................0x0
>>                 link_speed..............0x11
>>                 mtu_smsl................0x40
>>                 vl_cap_init_type........0x40
>>                 vl_high_limit...........0x0
>>                 vl_arb_high_cap.........0x8
>>                 vl_arb_low_cap..........0x8
>>                 init_rep_mtu_cap........0x4
>>                 vl_stall_life...........0xF2
>>                 vl_enforce..............0x40
>>                 m_key_violations........0x0
>>                 p_key_violations........0x0
>>                 q_key_violations........0x0
>>                 guid_cap................0x0
>>                 client_reregister.......0x0
>>                 subnet_timeout..........0x0
>>                 resp_time_value.........0x0
>>                 error_threshold.........0x88
>> Mar 19 18:39:07 112034 [41401960] -> Capabilities Mask:
>> Mar 19 18:39:07 117211 [45A08960] -> SUBNET UP
>> Mar 19 18:39:07 354540 [41E02960] -> __osm_trap_rcv_process_request: 
>> Received Generic Notice type:0x01 num:128 Producer:2 from LID:0x0148 
>> TID:0x0000000000000047
>> Mar 19 18:39:07 354686 [41E02960] -> osm_report_notice: Reporting Generic 
>> Notice type:1 num:128 from LID:0x0148 
>> GID:0xfe80000000000000,0x0005ad00000281b3
>> Mar 19 18:39:07 383453 [42803960] -> __osm_trap_rcv_process_request: 
>> Received Generic Notice type:0x01 num:128 Producer:2 from LID:0x001B 
>> TID:0x0000000000000055
>> Mar 19 18:39:07 383530 [42803960] -> osm_report_notice: Reporting Generic 
>> Notice type:1 num:128 from LID:0x001B 
>> GID:0xfe80000000000000,0x0005ad00000281a7
>> Mar 19 18:39:07 497601 [42803960] -> osm_ucast_mgr_process: Min Hop Tables 
>> configured on all switches
>> Mar 19 18:39:07 548184 [43204960] -> __osm_trap_rcv_process_request: 
>> Received Generic Notice type:0x01 num:128 Producer:2 from LID:0x001B 
>> TID:0x0000000000000056
>> Mar 19 18:39:07 548217 [43C05960] -> SUBNET UP
>> Mar 19 18:39:07 548427 [43204960] -> osm_report_notice: Reporting Generic 
>> Notice type:1 num:128 from LID:0x001B 
>> GID:0xfe80000000000000,0x0005ad00000281a7
>> Mar 19 18:39:07 878403 [45007960] -> __osm_trap_rcv_process_request: 
>> Received Generic Notice type:0x01 num:128 Producer:2 from LID:0x001B 
>> TID:0x0000000000000057
>> Mar 19 18:39:07 887312 [45A08960] -> __osm_trap_rcv_process_request: 
>> Received Generic Notice type:0x01 num:128 Producer:2 from LID:0x001B 
>> TID:0x0000000000000058
>> Mar 19 18:39:07 888156 [41E02960] -> osm_ucast_mgr_process: Min Hop Tables 
>> configured on all switches
>> Mar 19 18:39:07 929819 [45007960] -> osm_report_notice: Reporting Generic 
>> Notice type:1 num:128 from LID:0x001B 
>> GID:0xfe80000000000000,0x0005ad00000281a7
>> Mar 19 18:39:07 929834 [45A08960] -> osm_report_notice: Reporting Generic 
>> Notice type:1 num:128 from LID:0x001B 
>> GID:0xfe80000000000000,0x0005ad00000281a7
>> Mar 19 18:39:07 931166 [45007960] -> __osm_trap_rcv_process_request: 
>> Received Generic Notice type:0x01 num:128 Producer:2 from LID:0x001B 
>> TID:0x0000000000000059
>> Mar 19 18:39:07 931288 [45007960] -> osm_report_notice: Reporting Generic 
>> Notice type:1 num:128 from LID:0x001B 
>> GID:0xfe80000000000000,0x0005ad00000281a7
>> Mar 19 18:39:07 946406 [42803960] -> SUBNET UP
>> Mar 19 18:39:08 073735 [41E02960] -> __osm_trap_rcv_process_request: 
>> Received Generic Notice type:0x01 num:128 Producer:2 from LID:0x0152 
>> TID:0x0000000000000020
>> Mar 19 18:39:08 073811 [41E02960] -> __osm_trap_rcv_process_request: ERR 
>> 3804: Received trap 33 times consecutively
>> Mar 19 18:39:08 400790 [43204960] -> osm_ucast_mgr_process: Min Hop Tables 
>> configured on all switches
>> Mar 19 18:39:08 467925 [45A08960] -> SUBNET UP
>> Mar 19 20:24:07 009911 [42803960] -> __osm_trap_rcv_process_request: 
>> Received Generic Notice type:0x01 num:128 Producer:2 from LID:0x0020 
>> TID:0x0000000000000020
>> Mar 19 20:24:07 010153 [42803960] -> osm_report_notice: Reporting Generic 
>> Notice type:1 num:128 from LID:0x0020 
>> GID:0xfe80000000000000,0x0005ad00000281ad
>> Mar 19 20:24:07 010966 [41401960] -> __osm_trap_rcv_process_request: 
>> Received Generic Notice type:0x01 num:128 Producer:2 from LID:0x0001 
>> TID:0x000000000000001a
>> Mar 19 20:24:07 011064 [41401960] -> osm_report_notice: Reporting Generic 
>> Notice type:1 num:128 from LID:0x0001 
>> GID:0xfe80000000000000,0x0005ad0000027c6a
>> Mar 19 20:24:07 390927 [43204960] -> osm_ucast_mgr_process: Min Hop Tables 
>> configured on all switches
>> Mar 19 20:24:07 453747 [43204960] -> SUBNET UP
>> Mar 19 20:24:07 839927 [45007960] -> osm_ucast_mgr_process: Min Hop Tables 
>> configured on all switches
>> Mar 19 20:24:07 895694 [45A08960] -> SUBNET UP
>> Mar 19 20:24:08 049066 [41E02960] -> __osm_trap_rcv_process_request: 
>> Received Generic Notice type:0x01 num:128 Producer:2 from LID:0x0001 
>> TID:0x000000000000001a
>> Mar 19 20:24:08 049322 [41E02960] -> osm_report_notice: Reporting Generic 
>> Notice type:1 num:128 from LID:0x0001 
>> GID:0xfe80000000000000,0x0005ad0000027c6a
>> Mar 19 20:24:08 433979 [42803960] -> osm_ucast_mgr_process: Min Hop Tables 
>> configured on all switches
>> Mar 19 20:24:08 487950 [43204960] -> SUBNET UP
>> Mar 19 20:26:28 608381 [42803960] -> __osm_trap_rcv_process_request: 
>> Received Generic Notice type:0x01 num:128 Producer:2 from LID:0x0020 
>> TID:0x0000000000000021
>> Mar 19 20:26:28 608406 [44606960] -> __osm_trap_rcv_process_request: 
>> Received Generic Notice type:0x01 num:128 Producer:2 from LID:0x0001 
>> TID:0x000000000000001b
>> Mar 19 20:26:28 608685 [42803960] -> osm_report_notice: Reporting Generic 
>> Notice type:1 num:128 from LID:0x0020 
>> GID:0xfe80000000000000,0x0005ad00000281ad
>> Mar 19 20:26:28 608693 [44606960] -> osm_report_notice: Reporting Generic 
>> Notice type:1 num:128 from LID:0x0001 
>> GID:0xfe80000000000000,0x0005ad0000027c6a
>> Mar 19 20:26:28 972140 [44606960] -> osm_ucast_mgr_process: Min Hop Tables 
>> configured on all switches
>> Mar 19 20:26:29 028682 [43C05960] -> SUBNET UP
>> Mar 19 20:26:29 399649 [43204960] -> osm_ucast_mgr_process: Min Hop Tables 
>> configured on all switches
>> Mar 19 20:26:29 465737 [45007960] -> SUBNET UP
>> Mar 19 21:30:38 775260 [45007960] -> __osm_trap_rcv_process_request: 
>> Received Generic Notice type:0x01 num:128 Producer:2 from LID:0x0146 
>> TID:0x000000000000002f
>> Mar 19 21:30:38 775533 [45007960] -> osm_report_notice: Reporting Generic 
>> Notice type:1 num:128 from LID:0x0146 
>> GID:0xfe80000000000000,0x0005ad00000281b6
>> Mar 19 21:30:38 777083 [45007960] -> __osm_trap_rcv_process_request: 
>> Received Generic Notice type:0x01 num:128 Producer:2 from LID:0x0143 
>> TID:0x0000000000000037
>> Mar 19 21:30:38 777242 [45007960] -> osm_report_notice: Reporting Generic 
>> Notice type:1 num:128 from LID:0x0143 
>> GID:0xfe80000000000000,0x0005ad00000281b9
>> Mar 19 21:30:39 144779 [43C05960] -> osm_ucast_mgr_process: Min Hop Tables 
>> configured on all switches
>> Mar 19 21:30:39 200635 [43204960] -> SUBNET UP
>> Mar 19 21:30:39 536003 [43C05960] -> osm_ucast_mgr_process: Min Hop Tables 
>> configured on all switches
>> Mar 19 21:30:39 591216 [42803960] -> SUBNET UP
>> Mar 20 14:06:48 971082 [41401960] -> __osm_trap_rcv_process_request: 
>> Received Generic Notice type:0x01 num:128 Producer:2 from LID:0x0152 
>> TID:0x0000000000000021
>> Mar 20 14:06:48 971376 [41401960] -> osm_report_notice: Reporting Generic 
>> Notice type:1 num:128 from LID:0x0152 
>> GID:0xfe80000000000000,0x0005ad0000027c84
>> Mar 20 14:06:49 346734 [42803960] -> osm_report_notice: Reporting Generic 
>> Notice type:3 num:65 from LID:0x0092 
>> GID:0xfe80000000000000,0x0005ad0000024bbb
>> Mar 20 14:06:49 346761 [42803960] -> Removed port with 
>> GUID:0x0005ad0000024b27 LID range [0xAF,0xAF] of node:saguaro-23-0 HCA-1
>> Mar 20 14:06:49 381394 [42803960] -> osm_ucast_mgr_process: Min Hop Tables 
>> configured on all switches
>> Mar 20 14:06:49 440803 [43204960] -> SUBNET UP
>> Mar 20 14:07:09 098449 [44606960] -> __osm_trap_rcv_process_request: 
>> Received Generic Notice type:0x01 num:128 Producer:2 from LID:0x0148 
>> TID:0x0000000000000048
>> Mar 20 14:07:09 098708 [44606960] -> osm_report_notice: Reporting Generic 
>> Notice type:1 num:128 from LID:0x0148 
>> GID:0xfe80000000000000,0x0005ad00000281b3
>> Mar 20 14:07:09 098733 [41E02960] -> __osm_trap_rcv_process_request: 
>> Received Generic Notice type:0x01 num:128 Producer:2 from LID:0x001B 
>> TID:0x000000000000005a
>> Mar 20 14:07:09 098777 [41E02960] -> osm_report_notice: Reporting Generic 
>> Notice type:1 num:128 from LID:0x001B 
>> GID:0xfe80000000000000,0x0005ad00000281a7
>> Mar 20 14:07:09 417844 [42803960] -> osm_report_notice: Reporting Generic 
>> Notice type:3 num:65 from LID:0x0092 
>> GID:0xfe80000000000000,0x0005ad0000024bbb
>> Mar 20 14:07:09 417862 [42803960] -> Removed port with 
>> GUID:0x0005ad0000024e0b LID range [0xB3,0xB3] of node:saguaro-23-4 HCA-1
>> Mar 20 14:07:09 417879 [42803960] -> osm_report_notice: Reporting Generic 
>> Notice type:3 num:65 from LID:0x0092 
>> GID:0xfe80000000000000,0x0005ad0000024bbb
>> Mar 20 14:07:09 417885 [42803960] -> Removed port with 
>> GUID:0x0005ad000002510b LID range [0xB5,0xB5] of node:saguaro-23-6 HCA-1
>> Mar 20 14:07:09 417902 [42803960] -> osm_report_notice: Reporting Generic 
>> Notice type:3 num:65 from LID:0x0092 
>> GID:0xfe80000000000000,0x0005ad0000024bbb
>> Mar 20 14:07:09 417907 [42803960] -> Removed port with 
>> GUID:0x0005ad000002511b LID range [0xA6,0xA6] of node:saguaro-22-1 HCA-1
>> Mar 20 14:07:09 417924 [42803960] -> osm_report_notice: Reporting Generic 
>> Notice type:3 num:65 from LID:0x0092 
>> GID:0xfe80000000000000,0x0005ad0000024bbb
>> Mar 20 14:07:09 417929 [42803960] -> Removed port with 
>> GUID:0x0005ad0000025043 LID range [0xB4,0xB4] of node:saguaro-23-5 HCA-1
>> Mar 20 14:07:09 417945 [42803960] -> osm_report_notice: Reporting Generic 
>> Notice type:3 num:65 from LID:0x0092 
>> GID:0xfe80000000000000,0x0005ad0000024bbb
>> Mar 20 14:07:09 417951 [42803960] -> Removed port with 
>> GUID:0x0005ad0000024d47 LID range [0xB6,0xB6] of node:saguaro-23-7 HCA-1
>> Mar 20 14:07:09 417967 [42803960] -> osm_report_notice: Reporting Generic 
>> Notice type:3 num:65 from LID:0x0092 
>> GID:0xfe80000000000000,0x0005ad0000024bbb
>> Mar 20 14:07:09 417973 [42803960] -> Removed port with 
>> GUID:0x0005ad0000024d6b LID range [0xB8,0xB8] of node:saguaro-23-9 HCA-1
>> Mar 20 14:07:09 417989 [42803960] -> osm_report_notice: Reporting Generic 
>> Notice type:3 num:65 from LID:0x0092 
>> GID:0xfe80000000000000,0x0005ad0000024bbb
>> Mar 20 14:07:09 417994 [42803960] -> Removed port with 
>> GUID:0x0005ad0000024977 LID range [0xA9,0xA9] of node:saguaro-22-4 HCA-1
>> Mar 20 14:07:09 418131 [42803960] -> osm_report_notice: Reporting Generic 
>> Notice type:3 num:65 from LID:0x0092 
>> GID:0xfe80000000000000,0x0005ad0000024bbb
>> Mar 20 14:07:09 418137 [42803960] -> Removed port with 
>> GUID:0x0005ad0000027c84 LID range [0x152,0x152] of node:Topspin Switch TS120
>> Mar 20 14:07:09 418168 [42803960] -> osm_report_notice: Reporting Generic 
>> Notice type:3 num:65 from LID:0x0092 
>> GID:0xfe80000000000000,0x0005ad0000024bbb
>> Mar 20 14:07:09 418173 [42803960] -> Removed port with 
>> GUID:0x0005ad0000024d8b LID range [0xB7,0xB7] of node:saguaro-23-8 HCA-1
>> Mar 20 14:07:09 418188 [42803960] -> osm_report_notice: Reporting Generic 
>> Notice type:3 num:65 from LID:0x0092 
>> GID:0xfe80000000000000,0x0005ad0000024bbb
>> Mar 20 14:07:09 418193 [42803960] -> Removed port with 
>> GUID:0x0005ad000002498f LID range [0xA8,0xA8] of node:saguaro-22-3 HCA-1
>> Mar 20 14:07:09 418207 [42803960] -> osm_report_notice: Reporting Generic 
>> Notice type:3 num:65 from LID:0x0092 
>> GID:0xfe80000000000000,0x0005ad0000024bbb
>> Mar 20 14:07:09 418212 [42803960] -> Removed port with 
>> GUID:0x0005ad0000024c9b LID range [0xA7,0xA7] of node:saguaro-22-2 HCA-1
>> Mar 20 14:07:09 418227 [42803960] -> osm_report_notice: Reporting Generic 
>> Notice type:3 num:65 from LID:0x0092 
>> GID:0xfe80000000000000,0x0005ad0000024bbb
>> Mar 20 14:07:09 418232 [42803960] -> Removed port with 
>> GUID:0x0005ad0000024da7 LID range [0xB0,0xB0] of node:saguaro-23-1 HCA-1
>> Mar 20 14:07:09 418248 [42803960] -> osm_report_notice: Reporting Generic 
>> Notice type:3 num:65 from LID:0x0092 
>> GID:0xfe80000000000000,0x0005ad0000024bbb
>> Mar 20 14:07:09 418253 [42803960] -> Removed port with 
>> GUID:0x0005ad0000024cbb LID range [0xB2,0xB2] of node:saguaro-23-3 HCA-1
>> Mar 20 14:07:09 418285 [42803960] -> osm_report_notice: Reporting Generic 
>> Notice type:3 num:65 from LID:0x0092 
>> GID:0xfe80000000000000,0x0005ad0000024bbb
>> Mar 20 14:07:09 418290 [42803960] -> Removed port with 
>> GUID:0x0005ad00000249d3 LID range [0xB1,0xB1] of node:saguaro-23-2 HCA-1
>> Mar 20 14:07:09 418306 [42803960] -> osm_report_notice: Reporting Generic 
>> Notice type:3 num:65 from LID:0x0092 
>> GID:0xfe80000000000000,0x0005ad0000024bbb
>> Mar 20 14:07:09 418362 [42803960] -> Removed port with 
>> GUID:0x0005ad0000024feb LID range [0x153,0x153] of node:saguaro-22-5 HCA-1
>> Mar 20 14:07:09 418378 [42803960] -> osm_report_notice: Reporting Generic 
>> Notice type:3 num:65 from LID:0x0092 
>> GID:0xfe80000000000000,0x0005ad0000024bbb
>> Mar 20 14:07:09 418383 [42803960] -> Removed port with 
>> GUID:0x0005ad0000024afb LID range [0xA5,0xA5] of node:saguaro-22-0 HCA-1
>> Mar 20 14:07:09 451317 [42803960] -> osm_ucast_mgr_process: Min Hop Tables 
>> configured on all switches
>> Mar 20 14:07:09 502755 [41401960] -> SUBNET UP
>> Mar 20 14:07:09 902534 [41E02960] -> osm_ucast_mgr_process: Min Hop Tables 
>> configured on all switches
>> Mar 20 14:07:09 955229 [45A08960] -> SUBNET UP
>> Mar 20 14:08:03 850926 [45A08960] -> __osm_trap_rcv_process_request: 
>> Received Generic Notice type:0x01 num:128 Producer:2 from LID:0x0148 
>> TID:0x0000000000000049
>> Mar 20 14:08:03 851134 [45A08960] -> osm_report_notice: Reporting Generic 
>> Notice type:1 num:128 from LID:0x0148 
>> GID:0xfe80000000000000,0x0005ad00000281b3
>> Mar 20 14:08:03 856880 [43204960] -> __osm_trap_rcv_process_request: 
>> Received Generic Notice type:0x01 num:128 Producer:2 from LID:0x0148 
>> TID:0x000000000000004a
>> Mar 20 14:08:03 856955 [43204960] -> osm_report_notice: Reporting Generic 
>> Notice type:1 num:128 from LID:0x0148 
>> GID:0xfe80000000000000,0x0005ad00000281b3
>> Mar 20 14:08:03 866819 [42803960] -> __osm_trap_rcv_process_request: 
>> Received Generic Notice type:0x01 num:128 Producer:2 from LID:0x001B 
>> TID:0x000000000000005b
>> Mar 20 14:08:03 866977 [42803960] -> osm_report_notice: Reporting Generic 
>> Notice type:1 num:128 from LID:0x001B 
>> GID:0xfe80000000000000,0x0005ad00000281a7
>> Mar 20 14:08:03 963024 [45A08960] -> __osm_trap_rcv_process_request: 
>> Received Generic Notice type:0x01 num:128 Producer:2 from LID:0x001B 
>> TID:0x000000000000005c
>> Mar 20 14:08:03 963130 [45A08960] -> osm_report_notice: Reporting Generic 
>> Notice type:1 num:128 from LID:0x001B 
>> GID:0xfe80000000000000,0x0005ad00000281a7
>> Mar 20 14:08:04 106856 [43C05960] -> __osm_trap_rcv_process_request: 
>> Received Generic Notice type:0x01 num:128 Producer:2 from LID:0x001B 
>> TID:0x000000000000005d
>> Mar 20 14:08:04 106995 [43C05960] -> osm_report_notice: Reporting Generic 
>> Notice type:1 num:128 from LID:0x001B 
>> GID:0xfe80000000000000,0x0005ad00000281a7
>> Mar 20 14:08:04 193747 [44606960] -> osm_report_notice: Reporting Generic 
>> Notice type:3 num:64 from LID:0x0092 
>> GID:0xfe80000000000000,0x0005ad0000024bbb
>> Mar 20 14:08:04 193766 [44606960] -> Discovered new port with 
>> GUID:0x0005ad0000027c84 LID range [0x152,0x152] of node:Topspin Switch TS120
>> Mar 20 14:08:04 193771 [44606960] -> osm_report_notice: Reporting Generic 
>> Notice type:3 num:64 from LID:0x0092 
>> GID:0xfe80000000000000,0x0005ad0000024bbb
>> Mar 20 14:08:04 193777 [44606960] -> Discovered new port with 
>> GUID:0x0005ad0000024da7 LID range [0xB0,0xB0] of node:saguaro-23-1 HCA-1
>> Mar 20 14:08:04 193781 [44606960] -> osm_report_notice: Reporting Generic 
>> Notice type:3 num:64 from LID:0x0092 
>> GID:0xfe80000000000000,0x0005ad0000024bbb
>> Mar 20 14:08:04 193786 [44606960] -> Discovered new port with 
>> GUID:0x0005ad00000249d3 LID range [0xB1,0xB1] of node:saguaro-23-2 HCA-1
>> Mar 20 14:08:04 193790 [44606960] -> osm_report_notice: Reporting Generic 
>> Notice type:3 num:64 from LID:0x0092 
>> GID:0xfe80000000000000,0x0005ad0000024bbb
>> Mar 20 14:08:04 193795 [44606960] -> Discovered new port with 
>> GUID:0x0005ad0000024cbb LID range [0xB2,0xB2] of node:saguaro-23-3 HCA-1
>> Mar 20 14:08:04 193799 [44606960] -> osm_report_notice: Reporting Generic 
>> Notice type:3 num:64 from LID:0x0092 
>> GID:0xfe80000000000000,0x0005ad0000024bbb
>> Mar 20 14:08:04 193804 [44606960] -> Discovered new port with 
>> GUID:0x0005ad0000024e0b LID range [0xB3,0xB3] of node:saguaro-23-4 HCA-1
>> Mar 20 14:08:04 193808 [44606960] -> osm_report_notice: Reporting Generic 
>> Notice type:3 num:64 from LID:0x0092 
>> GID:0xfe80000000000000,0x0005ad0000024bbb
>> Mar 20 14:08:04 193813 [44606960] -> Discovered new port with 
>> GUID:0x0005ad0000025043 LID range [0xB4,0xB4] of node:saguaro-23-5 HCA-1
>> Mar 20 14:08:04 193817 [44606960] -> osm_report_notice: Reporting Generic 
>> Notice type:3 num:64 from LID:0x0092 
>> GID:0xfe80000000000000,0x0005ad0000024bbb
>> Mar 20 14:08:04 193822 [44606960] -> Discovered new port with 
>> GUID:0x0005ad000002510b LID range [0xB5,0xB5] of node:saguaro-23-6 HCA-1
>> Mar 20 14:08:04 193826 [44606960] -> osm_report_notice: Reporting Generic 
>> Notice type:3 num:64 from LID:0x0092 
>> GID:0xfe80000000000000,0x0005ad0000024bbb
>> Mar 20 14:08:04 193830 [44606960] -> Discovered new port with 
>> GUID:0x0005ad0000024d8b LID range [0xB7,0xB7] of node:saguaro-23-8 HCA-1
>> Mar 20 14:08:04 193834 [44606960] -> osm_report_notice: Reporting Generic 
>> Notice type:3 num:64 from LID:0x0092 
>> GID:0xfe80000000000000,0x0005ad0000024bbb
>> Mar 20 14:08:04 193839 [44606960] -> Discovered new port with 
>> GUID:0x0005ad0000024d47 LID range [0xB6,0xB6] of node:saguaro-23-7 HCA-1
>> Mar 20 14:08:04 193843 [44606960] -> osm_report_notice: Reporting Generic 
>> Notice type:3 num:64 from LID:0x0092 
>> GID:0xfe80000000000000,0x0005ad0000024bbb
>> Mar 20 14:08:04 193848 [44606960] -> Discovered new port with 
>> GUID:0x0005ad0000024d6b LID range [0xB8,0xB8] of node:saguaro-23-9 HCA-1
>> Mar 20 14:08:04 193852 [44606960] -> osm_report_notice: Reporting Generic 
>> Notice type:3 num:64 from LID:0x0092 
>> GID:0xfe80000000000000,0x0005ad0000024bbb
>> Mar 20 14:08:04 193857 [44606960] -> Discovered new port with 
>> GUID:0x0005ad0000024afb LID range [0xA5,0xA5] of node:saguaro-22-0 HCA-1
>> Mar 20 14:08:04 193861 [44606960] -> osm_report_notice: Reporting Generic 
>> Notice type:3 num:64 from LID:0x0092 
>> GID:0xfe80000000000000,0x0005ad0000024bbb
>> Mar 20 14:08:04 193866 [44606960] -> Discovered new port with 
>> GUID:0x0005ad000002511b LID range [0xA6,0xA6] of node:saguaro-22-1 HCA-1
>> Mar 20 14:08:04 193870 [44606960] -> osm_report_notice: Reporting Generic 
>> Notice type:3 num:64 from LID:0x0092 
>> GID:0xfe80000000000000,0x0005ad0000024bbb
>> Mar 20 14:08:04 193874 [44606960] -> Discovered new port with 
>> GUID:0x0005ad0000024c9b LID range [0xA7,0xA7] of node:saguaro-22-2 HCA-1
>> Mar 20 14:08:04 193878 [44606960] -> osm_report_notice: Reporting Generic 
>> Notice type:3 num:64 from LID:0x0092 
>> GID:0xfe80000000000000,0x0005ad0000024bbb
>> Mar 20 14:08:04 193883 [44606960] -> Discovered new port with 
>> GUID:0x0005ad000002498f LID range [0xA8,0xA8] of node:saguaro-22-3 HCA-1
>> Mar 20 14:08:04 193938 [44606960] -> osm_report_notice: Reporting Generic 
>> Notice type:3 num:64 from LID:0x0092 
>> GID:0xfe80000000000000,0x0005ad0000024bbb
>> Mar 20 14:08:04 193944 [44606960] -> Discovered new port with 
>> GUID:0x0005ad0000024977 LID range [0xA9,0xA9] of node:saguaro-22-4 HCA-1
>> Mar 20 14:08:04 193948 [44606960] -> osm_report_notice: Reporting Generic 
>> Notice type:3 num:64 from LID:0x0092 
>> GID:0xfe80000000000000,0x0005ad0000024bbb
>> Mar 20 14:08:04 193953 [44606960] -> Discovered new port with 
>> GUID:0x0005ad0000024feb LID range [0x153,0x153] of node:saguaro-22-5 HCA-1
>> Mar 20 14:08:04 224695 [44606960] -> osm_ucast_mgr_process: Min Hop Tables 
>> configured on all switches
>> Mar 20 14:08:04 281046 [4780B960] -> __osm_sm_mad_ctrl_rcv_callback: ERR 
>> 3111: Error status = 0x1C00
>> Mar 20 14:08:04 281106 [4780B960] -> SMP dump:
>>                 base_ver................0x1
>>                 mgmt_class..............0x81
>>                 class_ver...............0x1
>>                 method..................0x81 (SubnGetResp)
>>                 D bit...................0x1
>>                 status..................0x1C00
>>                 hop_ptr.................0x0
>>                 hop_count...............0x4
>>                 trans_id................0x61eec
>>                 attr_id.................0x15 (PortInfo)
>>                 resv....................0x0
>>                 attr_mod................0x13
>>                 m_key...................0x0000000000000000
>>                 dr_slid.................0xFFFF
>>                 dr_dlid.................0xFFFF
>> 
>>                 Initial path: [0][1][17][2][4]
>>                 Return path:  [0][9][14][E][1]
>>                 Reserved:     [0][0][0][0][0][0][0]
>> 
>>                 00 00 00 00 00 00 00 00   00 00 00 00 00 00 00 00
>> 
>>                 00 00 00 00 00 00 00 00   00 00 00 00 01 03 03 02
>> 
>>                 14 52 00 11 40 40 00 08   08 04 F2 40 00 00 00 00
>> 
>>                 00 00 00 00 00 88 00 00   00 00 00 00 00 00 00 00
>> 
>> Mar 20 14:08:04 281154 [41401960] -> osm_pi_rcv_process_set: Received error 
>> status 0x1c for SetResp() during ACTIVE transition
>> Mar 20 14:08:04 281159 [41401960] -> PortInfo dump:
>>                 port number.............0x13
>>                 node_guid...............0x0005ad00000281a7
>>                 port_guid...............0x0005ad00000281a7
>>                 m_key...................0x0000000000000000
>>                 subnet_prefix...........0x0000000000000000
>>                 base_lid................0x0
>>                 master_sm_base_lid......0x0
>>                 capability_mask.........0x0
>>                 diag_code...............0x0
>>                 m_key_lease_period......0x0
>>                 local_port_num..........0x1
>>                 link_width_enabled......0x3
>>                 link_width_supported....0x3
>>                 link_width_active.......0x2
>>                 link_speed_supported....0x1
>>                 port_state..............ACTIVE
>>                 state_info2.............0x52
>>                 m_key_protect_bits......0x0
>>                 lmc.....................0x0
>>                 link_speed..............0x11
>>                 mtu_smsl................0x40
>>                 vl_cap_init_type........0x40
>>                 vl_high_limit...........0x0
>>                 vl_arb_high_cap.........0x8
>>                 vl_arb_low_cap..........0x8
>>                 init_rep_mtu_cap........0x4
>>                 vl_stall_life...........0xF2
>>                 vl_enforce..............0x40
>>                 m_key_violations........0x0
>>                 p_key_violations........0x0
>>                 q_key_violations........0x0
>>                 guid_cap................0x0
>>                 client_reregister.......0x0
>>                 subnet_timeout..........0x0
>>                 resp_time_value.........0x0
>>                 error_threshold.........0x88
>> Mar 20 14:08:04 281172 [41401960] -> Capabilities Mask:
>> Mar 20 14:08:04 281187 [4780B960] -> __osm_sm_mad_ctrl_rcv_callback: ERR 
>> 3111: Error status = 0x1C00
>> Mar 20 14:08:04 281213 [4780B960] -> SMP dump:
>>                 base_ver................0x1
>>                 mgmt_class..............0x81
>>                 class_ver...............0x1
>>                 method..................0x81 (SubnGetResp)
>>                 D bit...................0x1
>>                 status..................0x1C00
>>                 hop_ptr.................0x0
>>                 hop_count...............0x4
>>                 trans_id................0x61eed
>>                 attr_id.................0x15 (PortInfo)
>>                 resv....................0x0
>>                 attr_mod................0x17
>>                 m_key...................0x0000000000000000
>>                 dr_slid.................0xFFFF
>>                 dr_dlid.................0xFFFF
>> 
>>                 Initial path: [0][1][17][2][4]
>>                 Return path:  [0][9][14][E][1]
>>                 Reserved:     [0][0][0][0][0][0][0]
>> 
>>                 00 00 00 00 00 00 00 00   00 00 00 00 00 00 00 00
>> 
>>                 00 00 00 00 00 00 00 00   00 00 00 00 01 03 03 02
>> 
>>                 14 52 00 11 40 40 00 08   08 04 F2 40 00 00 00 00
>> 
>>                 00 00 00 00 00 88 00 00   00 00 00 00 00 00 00 00
>> 
>> Mar 20 14:08:04 281279 [4780B960] -> __osm_sm_mad_ctrl_rcv_callback: ERR 
>> 3111: Error status = 0x1C00
>> Mar 20 14:08:04 281316 [4780B960] -> SMP dump:
>>                 base_ver................0x1
>>                 mgmt_class..............0x81
>>                 class_ver...............0x1
>>                 method..................0x81 (SubnGetResp)
>>                 D bit...................0x1
>>                 status..................0x1C00
>>                 hop_ptr.................0x0
>>                 hop_count...............0x4
>>                 trans_id................0x61eee
>>                 attr_id.................0x15 (PortInfo)
>>                 resv....................0x0
>>                 attr_mod................0x18
>>                 m_key...................0x0000000000000000
>>                 dr_slid.................0xFFFF
>>                 dr_dlid.................0xFFFF
>> 
>>                 Initial path: [0][1][17][2][4]
>>                 Return path:  [0][9][14][E][1]
>>                 Reserved:     [0][0][0][0][0][0][0]
>> 
>>                 00 00 00 00 00 00 00 00   00 00 00 00 00 00 00 00
>> 
>>                 00 00 00 00 00 00 00 00   00 00 00 00 01 03 03 02
>> 
>>                 14 52 00 11 40 40 00 08   08 04 F2 40 00 00 00 00
>> 
>>                 00 00 00 00 00 88 00 00   00 00 00 00 00 00 00 00
>> 
>> Mar 20 14:08:04 281392 [4780B960] -> __osm_sm_mad_ctrl_rcv_callback: ERR 
>> 3111: Error status = 0x1C00
>> Mar 20 14:08:04 281416 [4780B960] -> SMP dump:
>>                 base_ver................0x1
>>                 mgmt_class..............0x81
>>                 class_ver...............0x1
>>                 method..................0x81 (SubnGetResp)
>>                 D bit...................0x1
>>                 status..................0x1C00
>>                 hop_ptr.................0x0
>>                 hop_count...............0x4
>>                 trans_id................0x61eef
>>                 attr_id.................0x15 (PortInfo)
>>                 resv....................0x0
>>                 attr_mod................0x16
>>                 m_key...................0x0000000000000000
>>                 dr_slid.................0xFFFF
>>                 dr_dlid.................0xFFFF
>> 
>>                 Initial path: [0][1][11][1][6]
>>                 Return path:  [0][9][18][D][3]
>>                 Reserved:     [0][0][0][0][0][0][0]
>> 
>>                 00 00 00 00 00 00 00 00   00 00 00 00 00 00 00 00
>> 
>>                 00 00 00 00 00 00 00 00   00 00 00 00 03 03 03 02
>> 
>>                 14 52 00 11 40 40 00 08   08 04 F2 40 00 00 00 00
>> 
>>                 00 00 00 00 00 88 00 00   00 00 00 00 00 00 00 00
>> 
>> Mar 20 14:08:04 281515 [44606960] -> osm_pi_rcv_process_set: Received error 
>> status 0x1c for SetResp() during ACTIVE transition
>> Mar 20 14:08:04 281522 [4780B960] -> __osm_sm_mad_ctrl_rcv_callback: ERR 
>> 3111: Error status = 0x1C00
>> Mar 20 14:08:04 281542 [44606960] -> PortInfo dump:
>>                 port number.............0x17
>>                 node_guid...............0x0005ad00000281a7
>>                 port_guid...............0x0005ad00000281a7
>>                 m_key...................0x0000000000000000
>>                 subnet_prefix...........0x0000000000000000
>>                 base_lid................0x0
>>                 master_sm_base_lid......0x0
>>                 capability_mask.........0x0
>>                 diag_code...............0x0
>>                 m_key_lease_period......0x0
>>                 local_port_num..........0x1
>>                 link_width_enabled......0x3
>>                 link_width_supported....0x3
>>                 link_width_active.......0x2
>>                 link_speed_supported....0x1
>>                 port_state..............ACTIVE
>>                 state_info2.............0x52
>>                 m_key_protect_bits......0x0
>>                 lmc.....................0x0
>>                 link_speed..............0x11
>>                 mtu_smsl................0x40
>>                 vl_cap_init_type........0x40
>>                 vl_high_limit...........0x0
>>                 vl_arb_high_cap.........0x8
>>                 vl_arb_low_cap..........0x8
>>                 init_rep_mtu_cap........0x4
>>                 vl_stall_life...........0xF2
>>                 vl_enforce..............0x40
>>                 m_key_violations........0x0
>>                 p_key_violations........0x0
>>                 q_key_violations........0x0
>>                 guid_cap................0x0
>>                 client_reregister.......0x0
>>                 subnet_timeout..........0x0
>>                 resp_time_value.........0x0
>>                 error_threshold.........0x88
>> Mar 20 14:08:04 281553 [44606960] -> Capabilities Mask:
>> Mar 20 14:08:04 281561 [4780B960] -> SMP dump:
>>                 base_ver................0x1
>>                 mgmt_class..............0x81
>>                 class_ver...............0x1
>>                 method..................0x81 (SubnGetResp)
>>                 D bit...................0x1
>>                 status..................0x1C00
>>                 hop_ptr.................0x0
>>                 hop_count...............0x4
>>                 trans_id................0x61ef0
>>                 attr_id.................0x15 (PortInfo)
>>                 resv....................0x0
>>                 attr_mod................0x17
>>                 m_key...................0x0000000000000000
>>                 dr_slid.................0xFFFF
>>                 dr_dlid.................0xFFFF
>> 
>>                 Initial path: [0][1][11][1][6]
>>                 Return path:  [0][9][18][D][3]
>>                 Reserved:     [0][0][0][0][0][0][0]
>> 
>>                 00 00 00 00 00 00 00 00   00 00 00 00 00 00 00 00
>> 
>>                 00 00 00 00 00 00 00 00   00 00 00 00 03 03 03 02
>> 
>>                 14 52 00 11 40 40 00 08   08 04 F2 40 00 00 00 00
>> 
>>                 00 00 00 00 00 88 00 00   00 00 00 00 00 00 00 00
>> 
>> Mar 20 14:08:04 281572 [44606960] -> osm_pi_rcv_process_set: Received error 
>> status 0x1c for SetResp() during ACTIVE transition
>> Mar 20 14:08:04 281590 [44606960] -> PortInfo dump:
>>                 port number.............0x18
>>                 node_guid...............0x0005ad00000281a7
>>                 port_guid...............0x0005ad00000281a7
>>                 m_key...................0x0000000000000000
>>                 subnet_prefix...........0x0000000000000000
>>                 base_lid................0x0
>>                 master_sm_base_lid......0x0
>>                 capability_mask.........0x0
>>                 diag_code...............0x0
>>                 m_key_lease_period......0x0
>>                 local_port_num..........0x1
>>                 link_width_enabled......0x3
>>                 link_width_supported....0x3
>>                 link_width_active.......0x2
>>                 link_speed_supported....0x1
>>                 port_state..............ACTIVE
>>                 state_info2.............0x52
>>                 m_key_protect_bits......0x0
>>                 lmc.....................0x0
>>                 link_speed..............0x11
>>                 mtu_smsl................0x40
>>                 vl_cap_init_type........0x40
>>                 vl_high_limit...........0x0
>>                 vl_arb_high_cap.........0x8
>>                 vl_arb_low_cap..........0x8
>>                 init_rep_mtu_cap........0x4
>>                 vl_stall_life...........0xF2
>>                 vl_enforce..............0x40
>>                 m_key_violations........0x0
>>                 p_key_violations........0x0
>>                 q_key_violations........0x0
>>                 guid_cap................0x0
>>                 client_reregister.......0x0
>>                 subnet_timeout..........0x0
>>                 resp_time_value.........0x0
>>                 error_threshold.........0x88
>> Mar 20 14:08:04 281600 [44606960] -> Capabilities Mask:
>> Mar 20 14:08:04 281623 [44606960] -> osm_pi_rcv_process_set: Received error 
>> status 0x1c for SetResp() during ACTIVE transition
>> Mar 20 14:08:04 281626 [44606960] -> PortInfo dump:
>>                 port number.............0x16
>>                 node_guid...............0x0005ad00000281b3
>>                 port_guid...............0x0005ad00000281b3
>>                 m_key...................0x0000000000000000
>>                 subnet_prefix...........0x0000000000000000
>>                 base_lid................0x0
>>                 master_sm_base_lid......0x0
>>                 capability_mask.........0x0
>>                 diag_code...............0x0
>>                 m_key_lease_period......0x0
>>                 local_port_num..........0x3
>>                 link_width_enabled......0x3
>>                 link_width_supported....0x3
>>                 link_width_active.......0x2
>>                 link_speed_supported....0x1
>>                 port_state..............ACTIVE
>>                 state_info2.............0x52
>>                 m_key_protect_bits......0x0
>>                 lmc.....................0x0
>>                 link_speed..............0x11
>>                 mtu_smsl................0x40
>>                 vl_cap_init_type........0x40
>>                 vl_high_limit...........0x0
>>                 vl_arb_high_cap.........0x8
>>                 vl_arb_low_cap..........0x8
>>                 init_rep_mtu_cap........0x4
>>                 vl_stall_life...........0xF2
>>                 vl_enforce..............0x40
>>                 m_key_violations........0x0
>>                 p_key_violations........0x0
>>                 q_key_violations........0x0
>>                 guid_cap................0x0
>>                 client_reregister.......0x0
>>                 subnet_timeout..........0x0
>>                 resp_time_value.........0x0
>>                 error_threshold.........0x88
>> Mar 20 14:08:04 281635 [44606960] -> Capabilities Mask:
>> Mar 20 14:08:04 281637 [4780B960] -> __osm_sm_mad_ctrl_rcv_callback: ERR 
>> 3111: Error status = 0x1C00
>> Mar 20 14:08:04 281652 [44606960] -> osm_pi_rcv_process_set: Received error 
>> status 0x1c for SetResp() during ACTIVE transition
>> Mar 20 14:08:04 281663 [44606960] -> PortInfo dump:
>>                 port number.............0x17
>>                 node_guid...............0x0005ad00000281b3
>>                 port_guid...............0x0005ad00000281b3
>>                 m_key...................0x0000000000000000
>>                 subnet_prefix...........0x0000000000000000
>>                 base_lid................0x0
>>                 master_sm_base_lid......0x0
>>                 capability_mask.........0x0
>>                 diag_code...............0x0
>>                 m_key_lease_period......0x0
>>                 local_port_num..........0x3
>>                 link_width_enabled......0x3
>>                 link_width_supported....0x3
>>                 link_width_active.......0x2
>>                 link_speed_supported....0x1
>>                 port_state..............ACTIVE
>>                 state_info2.............0x52
>>                 m_key_protect_bits......0x0
>>                 lmc.....................0x0
>>                 link_speed..............0x11
>>                 mtu_smsl................0x40
>>                 vl_cap_init_type........0x40
>>                 vl_high_limit...........0x0
>>                 vl_arb_high_cap.........0x8
>>                 vl_arb_low_cap..........0x8
>>                 init_rep_mtu_cap........0x4
>>                 vl_stall_life...........0xF2
>>                 vl_enforce..............0x40
>>                 m_key_violations........0x0
>>                 p_key_violations........0x0
>>                 q_key_violations........0x0
>>                 guid_cap................0x0
>>                 client_reregister.......0x0
>>                 subnet_timeout..........0x0
>>                 resp_time_value.........0x0
>>                 error_threshold.........0x88
>> Mar 20 14:08:04 281673 [44606960] -> Capabilities Mask:
>> Mar 20 14:08:04 281675 [4780B960] -> SMP dump:
>>                 base_ver................0x1
>>                 mgmt_class..............0x81
>>                 class_ver...............0x1
>>                 method..................0x81 (SubnGetResp)
>>                 D bit...................0x1
>>                 status..................0x1C00
>>                 hop_ptr.................0x0
>>                 hop_count...............0x4
>>                 trans_id................0x61ef1
>>                 attr_id.................0x15 (PortInfo)
>>                 resv....................0x0
>>                 attr_mod................0x18
>>                 m_key...................0x0000000000000000
>>                 dr_slid.................0xFFFF
>>                 dr_dlid.................0xFFFF
>> 
>>                 Initial path: [0][1][11][1][6]
>>                 Return path:  [0][9][18][D][3]
>>                 Reserved:     [0][0][0][0][0][0][0]
>> 
>>                 00 00 00 00 00 00 00 00   00 00 00 00 00 00 00 00
>> 
>>                 00 00 00 00 00 00 00 00   00 00 00 00 03 03 03 02
>> 
>>                 14 52 00 11 40 40 00 08   08 04 F2 40 00 00 00 00
>> 
>>                 00 00 00 00 00 88 00 00   00 00 00 00 00 00 00 00
>> 
>> Mar 20 14:08:04 281721 [41E02960] -> osm_pi_rcv_process_set: Received error 
>> status 0x1c for SetResp() during ACTIVE transition
>> Mar 20 14:08:04 281726 [41E02960] -> PortInfo dump:
>>                 port number.............0x18
>>                 node_guid...............0x0005ad00000281b3
>>                 port_guid...............0x0005ad00000281b3
>>                 m_key...................0x0000000000000000
>>                 subnet_prefix...........0x0000000000000000
>>                 base_lid................0x0
>>                 master_sm_base_lid......0x0
>>                 capability_mask.........0x0
>>                 diag_code...............0x0
>>                 m_key_lease_period......0x0
>>                 local_port_num..........0x3
>>                 link_width_enabled......0x3
>>                 link_width_supported....0x3
>>                 link_width_active.......0x2
>>                 link_speed_supported....0x1
>>                 port_state..............ACTIVE
>>                 state_info2.............0x52
>>                 m_key_protect_bits......0x0
>>                 lmc.....................0x0
>>                 link_speed..............0x11
>>                 mtu_smsl................0x40
>>                 vl_cap_init_type........0x40
>>                 vl_high_limit...........0x0
>>                 vl_arb_high_cap.........0x8
>>                 vl_arb_low_cap..........0x8
>>                 init_rep_mtu_cap........0x4
>>                 vl_stall_life...........0xF2
>>                 vl_enforce..............0x40
>>                 m_key_violations........0x0
>>                 p_key_violations........0x0
>>                 q_key_violations........0x0
>>                 guid_cap................0x0
>>                 client_reregister.......0x0
>>                 subnet_timeout..........0x0
>>                 resp_time_value.........0x0
>>                 error_threshold.........0x88
>> Mar 20 14:08:04 281736 [41E02960] -> Capabilities Mask:
>> Mar 20 14:08:04 287136 [44606960] -> SUBNET UP
>> Mar 20 14:08:04 711595 [43C05960] -> osm_ucast_mgr_process: Min Hop Tables 
>> configured on all switches
>> Mar 20 14:08:04 766488 [45A08960] -> SUBNET UP
>> Mar 20 14:08:19 947200 [43204960] -> __osm_trap_rcv_process_request: 
>> Received Generic Notice type:0x01 num:128 Producer:2 from LID:0x0152 
>> TID:0x0000000000000000
>> Mar 20 14:08:19 947479 [43204960] -> osm_report_notice: Reporting Generic 
>> Notice type:1 num:128 from LID:0x0152 
>> GID:0xfe80000000000000,0x0005ad0000027c84
>> Mar 20 14:08:20 086909 [41E02960] -> __osm_trap_rcv_process_request: 
>> Received Generic Notice type:0x01 num:128 Producer:2 from LID:0x0152 
>> TID:0x0000000000000001
>> Mar 20 14:08:20 087084 [41E02960] -> osm_report_notice: Reporting Generic 
>> Notice type:1 num:128 from LID:0x0152 
>> GID:0xfe80000000000000,0x0005ad0000027c84
>> Mar 20 14:08:20 108865 [41401960] -> __osm_trap_rcv_process_request: 
>> Received Generic Notice type:0x01 num:128 Producer:2 from LID:0x0152 
>> TID:0x0000000000000002
>> Mar 20 14:08:20 109210 [41401960] -> osm_report_notice: Reporting Generic 
>> Notice type:1 num:128 from LID:0x0152 
>> GID:0xfe80000000000000,0x0005ad0000027c84
>> Mar 20 14:08:20 109996 [41E02960] -> __osm_trap_rcv_process_request: 
>> Received Generic Notice type:0x01 num:128 Producer:2 from LID:0x0152 
>> TID:0x0000000000000003
>> Mar 20 14:08:20 110407 [41E02960] -> osm_report_notice: Reporting Generic 
>> Notice type:1 num:128 from LID:0x0152 
>> GID:0xfe80000000000000,0x0005ad0000027c84
>> Mar 20 14:08:20 222523 [45A08960] -> __osm_trap_rcv_process_request: 
>> Received Generic Notice type:0x01 num:128 Producer:2 from LID:0x0152 
>> TID:0x0000000000000004
>> Mar 20 14:08:20 222613 [45A08960] -> osm_report_notice: Reporting Generic 
>> Notice type:1 num:128 from LID:0x0152 
>> GID:0xfe80000000000000,0x0005ad0000027c84
>> Mar 20 14:08:20 404596 [41401960] -> __osm_trap_rcv_process_request: 
>> Received Generic Notice type:0x01 num:128 Producer:2 from LID:0x0152 
>> TID:0x0000000000000005
>> Mar 20 14:08:20 404698 [41401960] -> osm_report_notice: Reporting Generic 
>> Notice type:1 num:128 from LID:0x0152 
>> GID:0xfe80000000000000,0x0005ad0000027c84
>> Mar 20 14:08:20 476804 [45007960] -> __osm_trap_rcv_process_request: 
>> Received Generic Notice type:0x01 num:128 Producer:2 from LID:0x0152 
>> TID:0x0000000000000006
>> Mar 20 14:08:20 476897 [45007960] -> osm_report_notice: Reporting Generic 
>> Notice type:1 num:128 from LID:0x0152 
>> GID:0xfe80000000000000,0x0005ad0000027c84
>> Mar 20 14:08:20 572434 [44606960] -> __osm_trap_rcv_process_request: 
>> Received Generic Notice type:0x01 num:128 Producer:2 from LID:0x0152 
>> TID:0x0000000000000007
>> Mar 20 14:08:20 572520 [44606960] -> osm_report_notice: Reporting Generic 
>> Notice type:1 num:128 from LID:0x0152 
>> GID:0xfe80000000000000,0x0005ad0000027c84
>> Mar 20 14:08:20 621715 [42803960] -> osm_report_notice: Reporting Generic 
>> Notice type:3 num:65 from LID:0x0092 
>> GID:0xfe80000000000000,0x0005ad0000024bbb
>> Mar 20 14:08:20 621726 [42803960] -> Removed port with 
>> GUID:0x0005ad00000249d3 LID range [0xB1,0xB1] of node:saguaro-23-2 HCA-1
>> Mar 20 14:08:20 656232 [42803960] -> osm_ucast_mgr_process: Min Hop Tables 
>> configured on all switches
>> Mar 20 14:08:20 698700 [44606960] -> __osm_trap_rcv_process_request: 
>> Received Generic Notice type:0x01 num:128 Producer:2 from LID:0x0152 
>> TID:0x0000000000000008
>> Mar 20 14:08:20 698794 [44606960] -> osm_report_notice: Reporting Generic 
>> Notice type:1 num:128 from LID:0x0152 
>> GID:0xfe80000000000000,0x0005ad0000027c84
>> Mar 20 14:08:20 708598 [41401960] -> __osm_trap_rcv_process_request: 
>> Received Generic Notice type:0x01 num:128 Producer:2 from LID:0x0152 
>> TID:0x0000000000000009
>> Mar 20 14:08:20 708698 [41401960] -> osm_report_notice: Reporting Generic 
>> Notice type:1 num:128 from LID:0x0152 
>> GID:0xfe80000000000000,0x0005ad0000027c84
>> Mar 20 14:08:20 713653 [45007960] -> SUBNET UP
>> Mar 20 14:08:20 730554 [44606960] -> __osm_trap_rcv_process_request: 
>> Received Generic Notice type:0x01 num:128 Producer:2 from LID:0x0152 
>> TID:0x000000000000000a
>> Mar 20 14:08:20 730697 [44606960] -> osm_report_notice: Reporting Generic 
>> Notice type:1 num:128 from LID:0x0152 
>> GID:0xfe80000000000000,0x0005ad0000027c84
>> Mar 20 14:08:20 754139 [45007960] -> __osm_trap_rcv_process_request: 
>> Received Generic Notice type:0x01 num:128 Producer:2 from LID:0x0152 
>> TID:0x000000000000000b
>> Mar 20 14:08:20 754251 [45007960] -> __osm_trap_rcv_process_request: ERR 
>> 3804: Received trap 11 times consecutively
>> Mar 20 14:08:20 947339 [41401960] -> __osm_trap_rcv_process_request: 
>> Received Generic Notice type:0x01 num:128 Producer:2 from LID:0x0152 
>> TID:0x000000000000000c
>> Mar 20 14:08:20 947426 [41401960] -> __osm_trap_rcv_process_request: ERR 
>> 3804: Received trap 12 times consecutively
>> Mar 20 14:08:20 975965 [45A08960] -> __osm_trap_rcv_process_request: 
>> Received Generic Notice type:0x01 num:128 Producer:2 from LID:0x0152 
>> TID:0x000000000000000d
>> Mar 20 14:08:20 976024 [45A08960] -> __osm_trap_rcv_process_request: ERR 
>> 3804: Received trap 13 times consecutively
>> Mar 20 14:08:20 997569 [43C05960] -> __osm_trap_rcv_process_request: 
>> Received Generic Notice type:0x01 num:128 Producer:2 from LID:0x0152 
>> TID:0x000000000000000e
>> Mar 20 14:08:20 997648 [43C05960] -> __osm_trap_rcv_process_request: ERR 
>> 3804: Received trap 14 times consecutively
>> Mar 20 14:08:21 019465 [44606960] -> __osm_trap_rcv_process_request: 
>> Received Generic Notice type:0x01 num:128 Producer:2 from LID:0x0152 
>> TID:0x000000000000000f
>> Mar 20 14:08:21 019512 [44606960] -> __osm_trap_rcv_process_request: ERR 
>> 3804: Received trap 15 times consecutively
>> Mar 20 14:08:21 064967 [43204960] -> __osm_trap_rcv_process_request: 
>> Received Generic Notice type:0x01 num:128 Producer:2 from LID:0x0152 
>> TID:0x0000000000000010
>> Mar 20 14:08:21 065009 [43204960] -> __osm_trap_rcv_process_request: ERR 
>> 3804: Received trap 16 times consecutively
>> Mar 20 14:08:21 082838 [41401960] -> __osm_trap_rcv_process_request: 
>> Received Generic Notice type:0x01 num:128 Producer:2 from LID:0x0152 
>> TID:0x0000000000000011
>> Mar 20 14:08:21 082877 [41401960] -> __osm_trap_rcv_process_request: ERR 
>> 3804: Received trap 17 times consecutively
>> Mar 20 14:08:21 100567 [43204960] -> __osm_trap_rcv_process_request: 
>> Received Generic Notice type:0x01 num:128 Producer:2 from LID:0x0152 
>> TID:0x0000000000000012
>> Mar 20 14:08:21 100619 [43204960] -> __osm_trap_rcv_process_request: ERR 
>> 3804: Received trap 18 times consecutively
>> Mar 20 14:08:21 188128 [43C05960] -> osm_report_notice: Reporting Generic 
>> Notice type:3 num:65 from LID:0x0092 
>> GID:0xfe80000000000000,0x0005ad0000024bbb
>> Mar 20 14:08:21 188144 [43C05960] -> Removed port with 
>> GUID:0x0005ad000002510b LID range [0xB5,0xB5] of node:saguaro-23-6 HCA-1
>> Mar 20 14:08:21 188166 [43C05960] -> osm_report_notice: Reporting Generic 
>> Notice type:3 num:65 from LID:0x0092 
>> GID:0xfe80000000000000,0x0005ad0000024bbb
>> Mar 20 14:08:21 188172 [43C05960] -> Removed port with 
>> GUID:0x0005ad0000024d47 LID range [0xB6,0xB6] of node:saguaro-23-7 HCA-1
>> Mar 20 14:08:21 188194 [43C05960] -> osm_report_notice: Reporting Generic 
>> Notice type:3 num:65 from LID:0x0092 
>> GID:0xfe80000000000000,0x0005ad0000024bbb
>> Mar 20 14:08:21 188199 [43C05960] -> Removed port with 
>> GUID:0x0005ad0000024d8b LID range [0xB7,0xB7] of node:saguaro-23-8 HCA-1
>> Mar 20 14:08:21 192421 [41E02960] -> osm_report_notice: Reporting Generic 
>> Notice type:3 num:64 from LID:0x0092 
>> GID:0xfe80000000000000,0x0005ad0000024bbb
>> Mar 20 14:08:21 192436 [41E02960] -> Discovered new port with 
>> GUID:0x0005ad00000249d3 LID range [0xB1,0xB1] of node:saguaro-23-2 HCA-1
>> Mar 20 14:08:21 208455 [41401960] -> __osm_trap_rcv_process_request: 
>> Received Generic Notice type:0x01 num:128 Producer:2 from LID:0x0152 
>> TID:0x0000000000000013
>> Mar 20 14:08:21 208499 [41401960] -> __osm_trap_rcv_process_request: ERR 
>> 3804: Received trap 19 times consecutively
>> Mar 20 14:08:21 223240 [41E02960] -> osm_ucast_mgr_process: Min Hop Tables 
>> configured on all switches
>> Mar 20 14:08:21 394585 [45007960] -> __osm_trap_rcv_process_request: 
>> Received Generic Notice type:0x01 num:128 Producer:2 from LID:0x0152 
>> TID:0x0000000000000014
>> Mar 20 14:08:21 394665 [45007960] -> __osm_trap_rcv_process_request: ERR 
>> 3804: Received trap 20 times consecutively
>> Mar 20 14:08:21 419333 [41E02960] -> __osm_trap_rcv_process_request: 
>> Received Generic Notice type:0x01 num:128 Producer:2 from LID:0x0152 
>> TID:0x0000000000000015
>> Mar 20 14:08:21 419393 [41E02960] -> __osm_trap_rcv_process_request: ERR 
>> 3804: Received trap 21 times consecutively
>> Mar 20 14:08:21 441228 [43204960] -> __osm_trap_rcv_process_request: 
>> Received Generic Notice type:0x01 num:128 Producer:2 from LID:0x0152 
>> TID:0x0000000000000016
>> Mar 20 14:08:21 441276 [43204960] -> __osm_trap_rcv_process_request: ERR 
>> 3804: Received trap 22 times consecutively
>> Mar 20 14:08:21 462915 [44606960] -> __osm_trap_rcv_process_request: 
>> Received Generic Notice type:0x01 num:128 Producer:2 from LID:0x0152 
>> TID:0x0000000000000017
>> Mar 20 14:08:21 462968 [44606960] -> __osm_trap_rcv_process_request: ERR 
>> 3804: Received trap 23 times consecutively
>> Mar 20 14:08:21 475440 [45007960] -> SUBNET UP
>> Mar 20 14:08:21 674045 [44606960] -> __osm_trap_rcv_process_request: 
>> Received Generic Notice type:0x01 num:128 Producer:2 from LID:0x0152 
>> TID:0x0000000000000018
>> Mar 20 14:08:21 674084 [43204960] -> __osm_trap_rcv_process_request: 
>> Received Generic Notice type:0x01 num:128 Producer:2 from LID:0x0148 
>> TID:0x000000000000004b
>> Mar 20 14:08:21 674137 [44606960] -> __osm_trap_rcv_process_request: ERR 
>> 3804: Received trap 24 times consecutively
>> Mar 20 14:08:21 674294 [43204960] -> osm_report_notice: Reporting Generic 
>> Notice type:1 num:128 from LID:0x0148 
>> GID:0xfe80000000000000,0x0005ad00000281b3
>> Mar 20 14:08:21 965885 [43204960] -> __osm_trap_rcv_process_request: 
>> Received Generic Notice type:0x01 num:128 Producer:2 from LID:0x0148 
>> TID:0x000000000000004c
>> Mar 20 14:08:21 965992 [43204960] -> osm_report_notice: Reporting Generic 
>> Notice type:1 num:128 from LID:0x0148 
>> GID:0xfe80000000000000,0x0005ad00000281b3
>> Mar 20 14:08:22 092378 [41401960] -> osm_report_notice: Reporting Generic 
>> Notice type:3 num:65 from LID:0x0092 
>> GID:0xfe80000000000000,0x0005ad0000024bbb
>> Mar 20 14:08:22 092395 [41401960] -> Removed port with 
>> GUID:0x0005ad0000024977 LID range [0xA9,0xA9] of node:saguaro-22-4 HCA-1
>> Mar 20 14:08:22 092415 [41401960] -> osm_report_notice: Reporting Generic 
>> Notice type:3 num:65 from LID:0x0092 
>> GID:0xfe80000000000000,0x0005ad0000024bbb
>> Mar 20 14:08:22 092420 [41401960] -> Removed port with 
>> GUID:0x0005ad0000024c9b LID range [0xA7,0xA7] of node:saguaro-22-2 HCA-1
>> Mar 20 14:08:22 092444 [41401960] -> osm_report_notice: Reporting Generic 
>> Notice type:3 num:65 from LID:0x0092 
>> GID:0xfe80000000000000,0x0005ad0000024bbb
>> Mar 20 14:08:22 092449 [41401960] -> Removed port with 
>> GUID:0x0005ad0000024feb LID range [0x153,0x153] of node:saguaro-22-5 HCA-1
>> Mar 20 14:08:22 092625 [41401960] -> osm_drop_mgr_process: ERR 0108: Unknown 
>> remote side for node 0x0005ad00000281b3 port 22. Adding to light sweep 
>> sampling list
>> Mar 20 14:08:22 092655 [41401960] -> Directed Path Dump of 4 hop path:
>>                 Path = [0][1][11][1][4]
>> Mar 20 14:08:22 092663 [41401960] -> osm_drop_mgr_process: ERR 0108: Unknown 
>> remote side for node 0x0005ad00000281b3 port 23. Adding to light sweep 
>> sampling list
>> Mar 20 14:08:22 092672 [41401960] -> Directed Path Dump of 4 hop path:
>>                 Path = [0][1][11][1][4]
>> Mar 20 14:08:22 096789 [41E02960] -> osm_report_notice: Reporting Generic 
>> Notice type:3 num:64 from LID:0x0092 
>> GID:0xfe80000000000000,0x0005ad0000024bbb
>> Mar 20 14:08:22 096801 [41E02960] -> Discovered new port with 
>> GUID:0x0005ad000002510b LID range [0xB5,0xB5] of node:saguaro-23-6 HCA-1
>> Mar 20 14:08:22 096805 [41E02960] -> osm_report_notice: Reporting Generic 
>> Notice type:3 num:64 from LID:0x0092 
>> GID:0xfe80000000000000,0x0005ad0000024bbb
>> Mar 20 14:08:22 096810 [41E02960] -> Discovered new port with 
>> GUID:0x0005ad0000024d47 LID range [0xB6,0xB6] of node:saguaro-23-7 HCA-1
>> Mar 20 14:08:22 096814 [41E02960] -> osm_report_notice: Reporting Generic 
>> Notice type:3 num:64 from LID:0x0092 
>> GID:0xfe80000000000000,0x0005ad0000024bbb
>> Mar 20 14:08:22 096819 [41E02960] -> Discovered new port with 
>> GUID:0x0005ad0000024d8b LID range [0xB7,0xB7] of node:saguaro-23-8 HCA-1
>> Mar 20 14:08:22 127266 [41E02960] -> osm_ucast_mgr_process: Min Hop Tables 
>> configured on all switches
>> Mar 20 14:08:22 184734 [45007960] -> SUBNET UP
>> Mar 20 14:08:22 541974 [41401960] -> osm_report_notice: Reporting Generic 
>> Notice type:3 num:64 from LID:0x0092 
>> GID:0xfe80000000000000,0x0005ad0000024bbb
>> Mar 20 14:08:22 541985 [41401960] -> Discovered new port with 
>> GUID:0x0005ad0000024c9b LID range [0xA7,0xA7] of node:saguaro-22-2 HCA-1
>> Mar 20 14:08:22 541989 [41401960] -> osm_report_notice: Reporting Generic 
>> Notice type:3 num:64 from LID:0x0092 
>> GID:0xfe80000000000000,0x0005ad0000024bbb
>> Mar 20 14:08:22 541995 [41401960] -> Discovered new port with 
>> GUID:0x0005ad0000024977 LID range [0xA9,0xA9] of node:saguaro-22-4 HCA-1
>> Mar 20 14:08:22 541998 [41401960] -> osm_report_notice: Reporting Generic 
>> Notice type:3 num:64 from LID:0x0092 
>> GID:0xfe80000000000000,0x0005ad0000024bbb
>> Mar 20 14:08:22 542003 [41401960] -> Discovered new port with 
>> GUID:0x0005ad0000024feb LID range [0x153,0x153] of node:saguaro-22-5 HCA-1
>> Mar 20 14:08:22 572711 [41401960] -> osm_ucast_mgr_process: Min Hop Tables 
>> configured on all switches
>> Mar 20 14:08:22 611570 [41401960] -> __osm_trap_rcv_process_request: 
>> Received Generic Notice type:0x01 num:128 Producer:2 from LID:0x0148 
>> TID:0x000000000000004d
>> Mar 20 14:08:22 611751 [41401960] -> osm_report_notice: Reporting Generic 
>> Notice type:1 num:128 from LID:0x0148 
>> GID:0xfe80000000000000,0x0005ad00000281b3
>> Mar 20 14:08:22 611770 [44606960] -> __osm_trap_rcv_process_request: 
>> Received Generic Notice type:0x01 num:128 Producer:2 from LID:0x001B 
>> TID:0x000000000000005e
>> Mar 20 14:08:22 612060 [44606960] -> osm_report_notice: Reporting Generic 
>> Notice type:1 num:128 from LID:0x001B 
>> GID:0xfe80000000000000,0x0005ad00000281a7
>> Mar 20 14:08:22 623766 [4780B960] -> __osm_sm_mad_ctrl_rcv_callback: ERR 
>> 3111: Error status = 0x1C00
>> Mar 20 14:08:22 623814 [4780B960] -> SMP dump:
>>                 base_ver................0x1
>>                 mgmt_class..............0x81
>>                 class_ver...............0x1
>>                 method..................0x81 (SubnGetResp)
>>                 D bit...................0x1
>>                 status..................0x1C00
>>                 hop_ptr.................0x0
>>                 hop_count...............0x4
>>                 trans_id................0x66134
>>                 attr_id.................0x15 (PortInfo)
>>                 resv....................0x0
>>                 attr_mod................0x16
>>                 m_key...................0x0000000000000000
>>                 dr_slid.................0xFFFF
>>                 dr_dlid.................0xFFFF
>> 
>>                 Initial path: [0][1][11][1][5]
>>                 Return path:  [0][9][18][D][2]
>>                 Reserved:     [0][0][0][0][0][0][0]
>> 
>>                 00 00 00 00 00 00 00 00   00 00 00 00 00 00 00 00
>> 
>>                 00 00 00 00 00 00 00 00   00 00 00 00 02 03 03 02
>> 
>>                 14 52 00 11 40 40 00 08   08 04 F2 40 00 00 00 00
>> 
>>                 00 00 00 00 00 88 00 00   00 00 00 00 00 00 00 00
>> 
>> Mar 20 14:08:22 623876 [45007960] -> osm_pi_rcv_process_set: Received error 
>> status 0x1c for SetResp() during ACTIVE transition
>> Mar 20 14:08:22 623888 [45007960] -> PortInfo dump:
>>                 port number.............0x16
>>                 node_guid...............0x0005ad00000281b3
>>                 port_guid...............0x0005ad00000281b3
>>                 m_key...................0x0000000000000000
>>                 subnet_prefix...........0x0000000000000000
>>                 base_lid................0x0
>>                 master_sm_base_lid......0x0
>>                 capability_mask.........0x0
>>                 diag_code...............0x0
>>                 m_key_lease_period......0x0
>>                 local_port_num..........0x2
>>                 link_width_enabled......0x3
>>                 link_width_supported....0x3
>>                 link_width_active.......0x2
>>                 link_speed_supported....0x1
>>                 port_state..............ACTIVE
>>                 state_info2.............0x52
>>                 m_key_protect_bits......0x0
>>                 lmc.....................0x0
>>                 link_speed..............0x11
>>                 mtu_smsl................0x40
>>                 vl_cap_init_type........0x40
>>                 vl_high_limit...........0x0
>>                 vl_arb_high_cap.........0x8
>>                 vl_arb_low_cap..........0x8
>>                 init_rep_mtu_cap........0x4
>>                 vl_stall_life...........0xF2
>>                 vl_enforce..............0x40
>>                 m_key_violations........0x0
>>                 p_key_violations........0x0
>>                 q_key_violations........0x0
>>                 guid_cap................0x0
>>                 client_reregister.......0x0
>>                 subnet_timeout..........0x0
>>                 resp_time_value.........0x0
>>                 error_threshold.........0x88
>> Mar 20 14:08:22 623907 [45007960] -> Capabilities Mask:
>> Mar 20 14:08:22 623945 [4780B960] -> __osm_sm_mad_ctrl_rcv_callback: ERR 
>> 3111: Error status = 0x1C00
>> Mar 20 14:08:22 623973 [4780B960] -> SMP dump:
>>                 base_ver................0x1
>>                 mgmt_class..............0x81
>>                 class_ver...............0x1
>>                 method..................0x81 (SubnGetResp)
>>                 D bit...................0x1
>>                 status..................0x1C00
>>                 hop_ptr.................0x0
>>                 hop_count...............0x4
>>                 trans_id................0x66135
>>                 attr_id.................0x15 (PortInfo)
>>                 resv....................0x0
>>                 attr_mod................0x17
>>                 m_key...................0x0000000000000000
>>                 dr_slid.................0xFFFF
>>                 dr_dlid.................0xFFFF
>> 
>>                 Initial path: [0][1][11][1][5]
>>                 Return path:  [0][9][18][D][2]
>>                 Reserved:     [0][0][0][0][0][0][0]
>> 
>>                 00 00 00 00 00 00 00 00   00 00 00 00 00 00 00 00
>> 
>>                 00 00 00 00 00 00 00 00   00 00 00 00 02 03 03 02
>> 
>>                 14 52 00 11 40 40 00 08   08 04 F2 40 00 00 00 00
>> 
>>                 00 00 00 00 00 88 00 00   00 00 00 00 00 00 00 00
>> 
>> Mar 20 14:08:22 624051 [44606960] -> osm_pi_rcv_process_set: Received error 
>> status 0x1c for SetResp() during ACTIVE transition
>> Mar 20 14:08:22 624056 [44606960] -> PortInfo dump:
>>                 port number.............0x17
>>                 node_guid...............0x0005ad00000281b3
>>                 port_guid...............0x0005ad00000281b3
>>                 m_key...................0x0000000000000000
>>                 subnet_prefix...........0x0000000000000000
>>                 base_lid................0x0
>>                 master_sm_base_lid......0x0
>>                 capability_mask.........0x0
>>                 diag_code...............0x0
>>                 m_key_lease_period......0x0
>>                 local_port_num..........0x2
>>                 link_width_enabled......0x3
>>                 link_width_supported....0x3
>>                 link_width_active.......0x2
>>                 link_speed_supported....0x1
>>                 port_state..............ACTIVE
>>                 state_info2.............0x52
>>                 m_key_protect_bits......0x0
>>                 lmc.....................0x0
>>                 link_speed..............0x11
>>                 mtu_smsl................0x40
>>                 vl_cap_init_type........0x40
>>                 vl_high_limit...........0x0
>>                 vl_arb_high_cap.........0x8
>>                 vl_arb_low_cap..........0x8
>>                 init_rep_mtu_cap........0x4
>>                 vl_stall_life...........0xF2
>>                 vl_enforce..............0x40
>>                 m_key_violations........0x0
>>                 p_key_violations........0x0
>>                 q_key_violations........0x0
>>                 guid_cap................0x0
>>                 client_reregister.......0x0
>>                 subnet_timeout..........0x0
>>                 resp_time_value.........0x0
>>                 error_threshold.........0x88
>> Mar 20 14:08:22 624069 [44606960] -> Capabilities Mask:
>> Mar 20 14:08:22 629289 [45A08960] -> SUBNET UP
>> Mar 20 14:08:22 712180 [43204960] -> __osm_trap_rcv_process_request: 
>> Received Generic Notice type:0x01 num:128 Producer:2 from LID:0x0152 
>> TID:0x0000000000000018
>> Mar 20 14:08:22 712238 [43204960] -> __osm_trap_rcv_process_request: ERR 
>> 3804: Received trap 25 times consecutively
>> Mar 20 14:08:22 869303 [43C05960] -> __osm_trap_rcv_process_request: 
>> Received Generic Notice type:0x01 num:128 Producer:2 from LID:0x001B 
>> TID:0x000000000000005f
>> Mar 20 14:08:22 869527 [43C05960] -> osm_report_notice: Reporting Generic 
>> Notice type:1 num:128 from LID:0x001B 
>> GID:0xfe80000000000000,0x0005ad00000281a7
>> Mar 20 14:08:22 892522 [45A08960] -> __osm_trap_rcv_process_request: 
>> Received Generic Notice type:0x01 num:128 Producer:2 from LID:0x0148 
>> TID:0x000000000000004e
>> Mar 20 14:08:22 892707 [45A08960] -> osm_report_notice: Reporting Generic 
>> Notice type:1 num:128 from LID:0x0148 
>> GID:0xfe80000000000000,0x0005ad00000281b3
>> Mar 20 14:08:22 957086 [42803960] -> __osm_trap_rcv_process_request: 
>> Received Generic Notice type:0x01 num:128 Producer:2 from LID:0x001B 
>> TID:0x0000000000000060
>> Mar 20 14:08:22 957189 [42803960] -> osm_report_notice: Reporting Generic 
>> Notice type:1 num:128 from LID:0x001B 
>> GID:0xfe80000000000000,0x0005ad00000281a7
>> Mar 20 14:08:23 080551 [41E02960] -> __osm_trap_rcv_process_request: 
>> Received Generic Notice type:0x01 num:128 Producer:2 from LID:0x001B 
>> TID:0x0000000000000061
>> Mar 20 14:08:23 080621 [41E02960] -> osm_report_notice: Reporting Generic 
>> Notice type:1 num:128 from LID:0x001B 
>> GID:0xfe80000000000000,0x0005ad00000281a7
>> Mar 20 14:08:23 102292 [45007960] -> __osm_trap_rcv_process_request: 
>> Received Generic Notice type:0x01 num:128 Producer:2 from LID:0x001B 
>> TID:0x0000000000000062
>> Mar 20 14:08:23 102372 [45007960] -> osm_report_notice: Reporting Generic 
>> Notice type:1 num:128 from LID:0x001B 
>> GID:0xfe80000000000000,0x0005ad00000281a7
>> Mar 20 14:08:23 124176 [43C05960] -> __osm_trap_rcv_process_request: 
>> Received Generic Notice type:0x01 num:128 Producer:2 from LID:0x001B 
>> TID:0x0000000000000063
>> Mar 20 14:08:23 124278 [43C05960] -> osm_report_notice: Reporting Generic 
>> Notice type:1 num:128 from LID:0x001B 
>> GID:0xfe80000000000000,0x0005ad00000281a7
>> Mar 20 14:08:23 285320 [42803960] -> __osm_trap_rcv_process_request: 
>> Received Generic Notice type:0x01 num:128 Producer:2 from LID:0x001B 
>> TID:0x0000000000000064
>> Mar 20 14:08:23 285393 [42803960] -> osm_report_notice: Reporting Generic 
>> Notice type:1 num:128 from LID:0x001B 
>> GID:0xfe80000000000000,0x0005ad00000281a7
>> Mar 20 14:08:23 403309 [41401960] -> __osm_trap_rcv_process_request: 
>> Received Generic Notice type:0x01 num:128 Producer:2 from LID:0x001B 
>> TID:0x0000000000000065
>> Mar 20 14:08:23 403388 [41401960] -> osm_report_notice: Reporting Generic 
>> Notice type:1 num:128 from LID:0x001B 
>> GID:0xfe80000000000000,0x0005ad00000281a7
>> Mar 20 14:08:23 425052 [45A08960] -> __osm_trap_rcv_process_request: 
>> Received Generic Notice type:0x01 num:128 Producer:2 from LID:0x001B 
>> TID:0x0000000000000066
>> Mar 20 14:08:23 425117 [45A08960] -> osm_report_notice: Reporting Generic 
>> Notice type:1 num:128 from LID:0x001B 
>> GID:0xfe80000000000000,0x0005ad00000281a7
>> Mar 20 14:08:23 447189 [41E02960] -> __osm_trap_rcv_process_request: 
>> Received Generic Notice type:0x01 num:128 Producer:2 from LID:0x001B 
>> TID:0x0000000000000067
>> Mar 20 14:08:23 447266 [41E02960] -> osm_report_notice: Reporting Generic 
>> Notice type:1 num:128 from LID:0x001B 
>> GID:0xfe80000000000000,0x0005ad00000281a7
>> Mar 20 14:08:23 535175 [44606960] -> osm_ucast_mgr_process: Min Hop Tables 
>> configured on all switches
>> Mar 20 14:08:23 595127 [41401960] -> SUBNET UP
>> Mar 20 14:08:23 750323 [41E02960] -> __osm_trap_rcv_process_request: 
>> Received Generic Notice type:0x01 num:128 Producer:2 from LID:0x0152 
>> TID:0x0000000000000018
>> Mar 20 14:08:23 750432 [41E02960] -> __osm_trap_rcv_process_request: ERR 
>> 3804: Received trap 26 times consecutively
>> Mar 20 14:08:23 960490 [42803960] -> osm_ucast_mgr_process: Min Hop Tables 
>> configured on all switches
>> Mar 20 14:08:24 014256 [4780B960] -> __osm_sm_mad_ctrl_rcv_callback: ERR 
>> 3111: Error status = 0x1C00
>> Mar 20 14:08:24 014323 [4780B960] -> SMP dump:
>>                 base_ver................0x1
>>                 mgmt_class..............0x81
>>                 class_ver...............0x1
>>                 method..................0x81 (SubnGetResp)
>>                 D bit...................0x1
>>                 status..................0x1C00
>>                 hop_ptr.................0x0
>>                 hop_count...............0x4
>>                 trans_id................0x67b9d
>>                 attr_id.................0x15 (PortInfo)
>>                 resv....................0x0
>>                 attr_mod................0x18
>>                 m_key...................0x0000000000000000
>>                 dr_slid.................0xFFFF
>>                 dr_dlid.................0xFFFF
>> 
>>                 Initial path: [0][1][11][1][6]
>>                 Return path:  [0][9][18][D][3]
>>                 Reserved:     [0][0][0][0][0][0][0]
>> 
>>                 00 00 00 00 00 00 00 00   00 00 00 00 00 00 00 00
>> 
>>                 00 00 00 00 00 00 00 00   00 00 00 00 03 03 03 02
>> 
>>                 14 52 00 11 40 40 00 08   08 04 F2 40 00 00 00 00
>> 
>>                 00 00 00 00 00 88 00 00   00 00 00 00 00 00 00 00
>> 
>> Mar 20 14:08:24 014398 [41401960] -> osm_pi_rcv_process_set: Received error 
>> status 0x1c for SetResp() during ACTIVE transition
>> Mar 20 14:08:24 014408 [41401960] -> PortInfo dump:
>>                 port number.............0x18
>>                 node_guid...............0x0005ad00000281b3
>>                 port_guid...............0x0005ad00000281b3
>>                 m_key...................0x0000000000000000
>>                 subnet_prefix...........0x0000000000000000
>>                 base_lid................0x0
>>                 master_sm_base_lid......0x0
>>                 capability_mask.........0x0
>>                 diag_code...............0x0
>>                 m_key_lease_period......0x0
>>                 local_port_num..........0x3
>>                 link_width_enabled......0x3
>>                 link_width_supported....0x3
>>                 link_width_active.......0x2
>>                 link_speed_supported....0x1
>>                 port_state..............ACTIVE
>>                 state_info2.............0x52
>>                 m_key_protect_bits......0x0
>>                 lmc.....................0x0
>>                 link_speed..............0x11
>>                 mtu_smsl................0x40
>>                 vl_cap_init_type........0x40
>>                 vl_high_limit...........0x0
>>                 vl_arb_high_cap.........0x8
>>                 vl_arb_low_cap..........0x8
>>                 init_rep_mtu_cap........0x4
>>                 vl_stall_life...........0xF2
>>                 vl_enforce..............0x40
>>                 m_key_violations........0x0
>>                 p_key_violations........0x0
>>                 q_key_violations........0x0
>>                 guid_cap................0x0
>>                 client_reregister.......0x0
>>                 subnet_timeout..........0x0
>>                 resp_time_value.........0x0
>>                 error_threshold.........0x88
>> Mar 20 14:08:24 014422 [41401960] -> Capabilities Mask:
>> Mar 20 14:08:24 019479 [41401960] -> SUBNET UP
>> Mar 20 14:11:00 201308 [43204960] -> __osm_trap_rcv_process_request: 
>> Received Generic Notice type:0x01 num:128 Producer:2 from LID:0x001F 
>> TID:0x0000000000000018
>> Mar 20 14:11:00 201580 [43204960] -> osm_report_notice: Reporting Generic 
>> Notice type:1 num:128 from LID:0x001F 
>> GID:0xfe80000000000000,0x0005ad0000027c56
>> Mar 20 14:11:00 554517 [41E02960] -> osm_report_notice: Reporting Generic 
>> Notice type:3 num:65 from LID:0x0092 
>> GID:0xfe80000000000000,0x0005ad0000024bbb
>> Mar 20 14:11:00 554538 [41E02960] -> Removed port with 
>> GUID:0x0005ad000002516f LID range [0xBA,0xBA] of node:saguaro-24-1 HCA-1
>> Mar 20 14:11:00 589140 [41E02960] -> osm_ucast_mgr_process: Min Hop Tables 
>> configured on all switches
>> Mar 20 14:11:00 641315 [45A08960] -> SUBNET UP
>> Mar 20 14:14:16 904140 [41E02960] -> __osm_trap_rcv_process_request: 
>> Received Generic Notice type:0x01 num:128 Producer:2 from LID:0x001B 
>> TID:0x0000000000000068
>> Mar 20 14:14:16 904369 [41E02960] -> osm_report_notice: Reporting Generic 
>> Notice type:1 num:128 from LID:0x001B 
>> GID:0xfe80000000000000,0x0005ad00000281a7
>> Mar 20 14:14:16 904462 [45007960] -> __osm_trap_rcv_process_request: 
>> Received Generic Notice type:0x01 num:128 Producer:2 from LID:0x0148 
>> TID:0x000000000000004f
>> Mar 20 14:14:16 904600 [45007960] -> osm_report_notice: Reporting Generic 
>> Notice type:1 num:128 from LID:0x0148 
>> GID:0xfe80000000000000,0x0005ad00000281b3
>> Mar 20 14:14:17 210726 [41401960] -> osm_report_notice: Reporting Generic 
>> Notice type:3 num:65 from LID:0x0092 
>> GID:0xfe80000000000000,0x0005ad0000024bbb
>> Mar 20 14:14:17 210747 [41401960] -> Removed port with 
>> GUID:0x0005ad0000024e0b LID range [0xB3,0xB3] of node:saguaro-23-4 HCA-1
>> Mar 20 14:14:17 210796 [41401960] -> osm_report_notice: Reporting Generic 
>> Notice type:3 num:65 from LID:0x0092 
>> GID:0xfe80000000000000,0x0005ad0000024bbb
>> Mar 20 14:14:17 210802 [41401960] -> Removed port with 
>> GUID:0x0005ad000002510b LID range [0xB5,0xB5] of node:saguaro-23-6 HCA-1
>> Mar 20 14:14:17 210818 [41401960] -> osm_report_notice: Reporting Generic 
>> Notice type:3 num:65 from LID:0x0092 
>> GID:0xfe80000000000000,0x0005ad0000024bbb
>> Mar 20 14:14:17 210836 [41401960] -> Removed port with 
>> GUID:0x0005ad000002511b LID range [0xA6,0xA6] of node:saguaro-22-1 HCA-1
>> Mar 20 14:14:17 210864 [41401960] -> osm_report_notice: Reporting Generic 
>> Notice type:3 num:65 from LID:0x0092 
>> GID:0xfe80000000000000,0x0005ad0000024bbb
>> Mar 20 14:14:17 210869 [41401960] -> Removed port with 
>> GUID:0x0005ad0000025043 LID range [0xB4,0xB4] of node:saguaro-23-5 HCA-1
>> Mar 20 14:14:17 210885 [41401960] -> osm_report_notice: Reporting Generic 
>> Notice type:3 num:65 from LID:0x0092 
>> GID:0xfe80000000000000,0x0005ad0000024bbb
>> Mar 20 14:14:17 210890 [41401960] -> Removed port with 
>> GUID:0x0005ad0000024d47 LID range [0xB6,0xB6] of node:saguaro-23-7 HCA-1
>> Mar 20 14:14:17 210908 [41401960] -> osm_report_notice: Reporting Generic 
>> Notice type:3 num:65 from LID:0x0092 
>> GID:0xfe80000000000000,0x0005ad0000024bbb
>> Mar 20 14:14:17 210913 [41401960] -> Removed port with 
>> GUID:0x0005ad0000024d6b LID range [0xB8,0xB8] of node:saguaro-23-9 HCA-1
>> Mar 20 14:14:17 210931 [41401960] -> osm_report_notice: Reporting Generic 
>> Notice type:3 num:65 from LID:0x0092 
>> GID:0xfe80000000000000,0x0005ad0000024bbb
>> Mar 20 14:14:17 210936 [41401960] -> Removed port with 
>> GUID:0x0005ad0000024977 LID range [0xA9,0xA9] of node:saguaro-22-4 HCA-1
>> Mar 20 14:14:17 211090 [41401960] -> osm_report_notice: Reporting Generic 
>> Notice type:3 num:65 from LID:0x0092 
>> GID:0xfe80000000000000,0x0005ad0000024bbb
>> Mar 20 14:14:17 211096 [41401960] -> Removed port with 
>> GUID:0x0005ad0000027c84 LID range [0x152,0x152] of node:Topspin Switch TS120
>> Mar 20 14:14:17 211127 [41401960] -> osm_report_notice: Reporting Generic 
>> Notice type:3 num:65 from LID:0x0092 
>> GID:0xfe80000000000000,0x0005ad0000024bbb
>> Mar 20 14:14:17 211133 [41401960] -> Removed port with 
>> GUID:0x0005ad0000024d8b LID range [0xB7,0xB7] of node:saguaro-23-8 HCA-1
>> Mar 20 14:14:17 211147 [41401960] -> osm_report_notice: Reporting Generic 
>> Notice type:3 num:65 from LID:0x0092 
>> GID:0xfe80000000000000,0x0005ad0000024bbb
>> Mar 20 14:14:17 211153 [41401960] -> Removed port with 
>> GUID:0x0005ad000002498f LID range [0xA8,0xA8] of node:saguaro-22-3 HCA-1
>> Mar 20 14:14:17 211169 [41401960] -> osm_report_notice: Reporting Generic 
>> Notice type:3 num:65 from LID:0x0092 
>> GID:0xfe80000000000000,0x0005ad0000024bbb
>> Mar 20 14:14:17 211174 [41401960] -> Removed port with 
>> GUID:0x0005ad0000024c9b LID range [0xA7,0xA7] of node:saguaro-22-2 HCA-1
>> Mar 20 14:14:17 211189 [41401960] -> osm_report_notice: Reporting Generic 
>> Notice type:3 num:65 from LID:0x0092 
>> GID:0xfe80000000000000,0x0005ad0000024bbb
>> Mar 20 14:14:17 211194 [41401960] -> Removed port with 
>> GUID:0x0005ad0000024da7 LID range [0xB0,0xB0] of node:saguaro-23-1 HCA-1
>> Mar 20 14:14:17 211212 [41401960] -> osm_report_notice: Reporting Generic 
>> Notice type:3 num:65 from LID:0x0092 
>> GID:0xfe80000000000000,0x0005ad0000024bbb
>> Mar 20 14:14:17 211216 [41401960] -> Removed port with 
>> GUID:0x0005ad0000024cbb LID range [0xB2,0xB2] of node:saguaro-23-3 HCA-1
>> Mar 20 14:14:17 211232 [41401960] -> osm_report_notice: Reporting Generic 
>> Notice type:3 num:65 from LID:0x0092 
>> GID:0xfe80000000000000,0x0005ad0000024bbb
>> Mar 20 14:14:17 211237 [41401960] -> Removed port with 
>> GUID:0x0005ad00000249d3 LID range [0xB1,0xB1] of node:saguaro-23-2 HCA-1
>> Mar 20 14:14:17 211253 [41401960] -> osm_report_notice: Reporting Generic 
>> Notice type:3 num:65 from LID:0x0092 
>> GID:0xfe80000000000000,0x0005ad0000024bbb
>> Mar 20 14:14:17 211317 [41401960] -> Removed port with 
>> GUID:0x0005ad0000024feb LID range [0x153,0x153] of node:saguaro-22-5 HCA-1
>> Mar 20 14:14:17 211333 [41401960] -> osm_report_notice: Reporting Generic 
>> Notice type:3 num:65 from LID:0x0092 
>> GID:0xfe80000000000000,0x0005ad0000024bbb
>> Mar 20 14:14:17 211338 [41401960] -> Removed port with 
>> GUID:0x0005ad0000024afb LID range [0xA5,0xA5] of node:saguaro-22-0 HCA-1
>> Mar 20 14:14:17 244432 [41401960] -> osm_ucast_mgr_process: Min Hop Tables 
>> configured on all switches
>> Mar 20 14:14:17 292747 [42803960] -> SUBNET UP
>> Mar 20 14:14:17 698554 [45A08960] -> osm_ucast_mgr_process: Min Hop Tables 
>> configured on all switches
>> Mar 20 14:14:17 750419 [44606960] -> SUBNET UP
>> Mar 20 14:15:11 300343 [41401960] -> __osm_trap_rcv_process_request: 
>> Received Generic Notice type:0x01 num:128 Producer:2 from LID:0x0148 
>> TID:0x0000000000000050
>> Mar 20 14:15:11 300577 [41401960] -> osm_report_notice: Reporting Generic 
>> Notice type:1 num:128 from LID:0x0148 
>> GID:0xfe80000000000000,0x0005ad00000281b3
>> Mar 20 14:15:11 306375 [45A08960] -> __osm_trap_rcv_process_request: 
>> Received Generic Notice type:0x01 num:128 Producer:2 from LID:0x001B 
>> TID:0x0000000000000069
>> Mar 20 14:15:11 306439 [42803960] -> __osm_trap_rcv_process_request: 
>> Received Generic Notice type:0x01 num:128 Producer:2 from LID:0x0148 
>> TID:0x0000000000000051
>> Mar 20 14:15:11 306487 [45A08960] -> osm_report_notice: Reporting Generic 
>> Notice type:1 num:128 from LID:0x001B 
>> GID:0xfe80000000000000,0x0005ad00000281a7
>> Mar 20 14:15:11 306514 [42803960] -> osm_report_notice: Reporting Generic 
>> Notice type:1 num:128 from LID:0x0148 
>> GID:0xfe80000000000000,0x0005ad00000281b3
>> Mar 20 14:15:11 312487 [43204960] -> __osm_trap_rcv_process_request: 
>> Received Generic Notice type:0x01 num:128 Producer:2 from LID:0x001B 
>> TID:0x000000000000006a
>> Mar 20 14:15:11 312581 [43204960] -> osm_report_notice: Reporting Generic 
>> Notice type:1 num:128 from LID:0x001B 
>> GID:0xfe80000000000000,0x0005ad00000281a7
>> Mar 20 14:15:11 636546 [45007960] -> osm_report_notice: Reporting Generic 
>> Notice type:3 num:64 from LID:0x0092 
>> GID:0xfe80000000000000,0x0005ad0000024bbb
>> Mar 20 14:15:11 636559 [45007960] -> Discovered new port with 
>> GUID:0x0005ad0000027c84 LID range [0x152,0x152] of node:Topspin Switch TS120
>> Mar 20 14:15:11 636565 [45007960] -> osm_report_notice: Reporting Generic 
>> Notice type:3 num:64 from LID:0x0092 
>> GID:0xfe80000000000000,0x0005ad0000024bbb
>> Mar 20 14:15:11 636572 [45007960] -> Discovered new port with 
>> GUID:0x0005ad0000024da7 LID range [0xB0,0xB0] of node:saguaro-23-1 HCA-1
>> Mar 20 14:15:11 636577 [45007960] -> osm_report_notice: Reporting Generic 
>> Notice type:3 num:64 from LID:0x0092 
>> GID:0xfe80000000000000,0x0005ad0000024bbb
>> Mar 20 14:15:11 636584 [45007960] -> Discovered new port with 
>> GUID:0x0005ad00000249d3 LID range [0xB1,0xB1] of node:saguaro-23-2 HCA-1
>> Mar 20 14:15:11 636589 [45007960] -> osm_report_notice: Reporting Generic 
>> Notice type:3 num:64 from LID:0x0092 
>> GID:0xfe80000000000000,0x0005ad0000024bbb
>> Mar 20 14:15:11 636595 [45007960] -> Discovered new port with 
>> GUID:0x0005ad0000024cbb LID range [0xB2,0xB2] of node:saguaro-23-3 HCA-1
>> Mar 20 14:15:11 636600 [45007960] -> osm_report_notice: Reporting Generic 
>> Notice type:3 num:64 from LID:0x0092 
>> GID:0xfe80000000000000,0x0005ad0000024bbb
>> Mar 20 14:15:11 636606 [45007960] -> Discovered new port with 
>> GUID:0x0005ad0000024e0b LID range [0xB3,0xB3] of node:saguaro-23-4 HCA-1
>> Mar 20 14:15:11 636612 [45007960] -> osm_report_notice: Reporting Generic 
>> Notice type:3 num:64 from LID:0x0092 
>> GID:0xfe80000000000000,0x0005ad0000024bbb
>> Mar 20 14:15:11 636618 [45007960] -> Discovered new port with 
>> GUID:0x0005ad0000025043 LID range [0xB4,0xB4] of node:saguaro-23-5 HCA-1
>> Mar 20 14:15:11 636623 [45007960] -> osm_report_notice: Reporting Generic 
>> Notice type:3 num:64 from LID:0x0092 
>> GID:0xfe80000000000000,0x0005ad0000024bbb
>> Mar 20 14:15:11 636629 [45007960] -> Discovered new port with 
>> GUID:0x0005ad000002510b LID range [0xB5,0xB5] of node:saguaro-23-6 HCA-1
>> Mar 20 14:15:11 636634 [45007960] -> osm_report_notice: Reporting Generic 
>> Notice type:3 num:64 from LID:0x0092 
>> GID:0xfe80000000000000,0x0005ad0000024bbb
>> Mar 20 14:15:11 636641 [45007960] -> Discovered new port with 
>> GUID:0x0005ad0000024d47 LID range [0xB6,0xB6] of node:saguaro-23-7 HCA-1
>> Mar 20 14:15:11 636646 [45007960] -> osm_report_notice: Reporting Generic 
>> Notice type:3 num:64 from LID:0x0092 
>> GID:0xfe80000000000000,0x0005ad0000024bbb
>> Mar 20 14:15:11 636652 [45007960] -> Discovered new port with 
>> GUID:0x0005ad0000024d8b LID range [0xB7,0xB7] of node:saguaro-23-8 HCA-1
>> Mar 20 14:15:11 636657 [45007960] -> osm_report_notice: Reporting Generic 
>> Notice type:3 num:64 from LID:0x0092 
>> GID:0xfe80000000000000,0x0005ad0000024bbb
>> Mar 20 14:15:11 636663 [45007960] -> Discovered new port with 
>> GUID:0x0005ad0000024d6b LID range [0xB8,0xB8] of node:saguaro-23-9 HCA-1
>> Mar 20 14:15:11 636668 [45007960] -> osm_report_notice: Reporting Generic 
>> Notice type:3 num:64 from LID:0x0092 
>> GID:0xfe80000000000000,0x0005ad0000024bbb
>> Mar 20 14:15:11 636675 [45007960] -> Discovered new port with 
>> GUID:0x0005ad0000024afb LID range [0xA5,0xA5] of node:saguaro-22-0 HCA-1
>> Mar 20 14:15:11 636680 [45007960] -> osm_report_notice: Reporting Generic 
>> Notice type:3 num:64 from LID:0x0092 
>> GID:0xfe80000000000000,0x0005ad0000024bbb
>> Mar 20 14:15:11 636686 [45007960] -> Discovered new port with 
>> GUID:0x0005ad000002511b LID range [0xA6,0xA6] of node:saguaro-22-1 HCA-1
>> Mar 20 14:15:11 636691 [45007960] -> osm_report_notice: Reporting Generic 
>> Notice type:3 num:64 from LID:0x0092 
>> GID:0xfe80000000000000,0x0005ad0000024bbb
>> Mar 20 14:15:11 636698 [45007960] -> Discovered new port with 
>> GUID:0x0005ad0000024c9b LID range [0xA7,0xA7] of node:saguaro-22-2 HCA-1
>> Mar 20 14:15:11 636703 [45007960] -> osm_report_notice: Reporting Generic 
>> Notice type:3 num:64 from LID:0x0092 
>> GID:0xfe80000000000000,0x0005ad0000024bbb
>> Mar 20 14:15:11 636709 [45007960] -> Discovered new port with 
>> GUID:0x0005ad000002498f LID range [0xA8,0xA8] of node:saguaro-22-3 HCA-1
>> Mar 20 14:15:11 636742 [45007960] -> osm_report_notice: Reporting Generic 
>> Notice type:3 num:64 from LID:0x0092 
>> GID:0xfe80000000000000,0x0005ad0000024bbb
>> Mar 20 14:15:11 636750 [45007960] -> Discovered new port with 
>> GUID:0x0005ad0000024977 LID range [0xA9,0xA9] of node:saguaro-22-4 HCA-1
>> Mar 20 14:15:11 636755 [45007960] -> osm_report_notice: Reporting Generic 
>> Notice type:3 num:64 from LID:0x0092 
>> GID:0xfe80000000000000,0x0005ad0000024bbb
>> Mar 20 14:15:11 636761 [45007960] -> Discovered new port with 
>> GUID:0x0005ad0000024feb LID range [0x153,0x153] of node:saguaro-22-5 HCA-1
>> Mar 20 14:15:11 667436 [45007960] -> osm_ucast_mgr_process: Min Hop Tables 
>> configured on all switches
>> Mar 20 14:15:11 731917 [4780B960] -> __osm_sm_mad_ctrl_rcv_callback: ERR 
>> 3111: Error status = 0x1C00
>> Mar 20 14:15:11 732017 [4780B960] -> SMP dump:
>>                 base_ver................0x1
>>                 mgmt_class..............0x81
>>                 class_ver...............0x1
>>                 method..................0x81 (SubnGetResp)
>>                 D bit...................0x1
>>                 status..................0x1C00
>>                 hop_ptr.................0x0
>>                 hop_count...............0x4
>>                 trans_id................0x6b507
>>                 attr_id.................0x15 (PortInfo)
>>                 resv....................0x0
>>                 attr_mod................0x13
>>                 m_key...................0x0000000000000000
>>                 dr_slid.................0xFFFF
>>                 dr_dlid.................0xFFFF
>> 
>>                 Initial path: [0][1][16][1][4]
>>                 Return path:  [0][9][13][D][1]
>>                 Reserved:     [0][0][0][0][0][0][0]
>> 
>>                 00 00 00 00 00 00 00 00   00 00 00 00 00 00 00 00
>> 
>>                 00 00 00 00 00 00 00 00   00 00 00 00 01 03 03 02
>> 
>>                 14 52 00 11 40 40 00 08   08 04 F2 40 00 00 00 00
>> 
>>                 00 00 00 00 00 88 00 00   00 00 00 00 00 00 00 00
>> 
>> Mar 20 14:15:11 732102 [41401960] -> osm_pi_rcv_process_set: Received error 
>> status 0x1c for SetResp() during ACTIVE transition
>> Mar 20 14:15:11 732106 [41401960] -> PortInfo dump:
>>                 port number.............0x13
>>                 node_guid...............0x0005ad00000281a7
>>                 port_guid...............0x0005ad00000281a7
>>                 m_key...................0x0000000000000000
>>                 subnet_prefix...........0x0000000000000000
>>                 base_lid................0x0
>>                 master_sm_base_lid......0x0
>>                 capability_mask.........0x0
>>                 diag_code...............0x0
>>                 m_key_lease_period......0x0
>>                 local_port_num..........0x1
>>                 link_width_enabled......0x3
>>                 link_width_supported....0x3
>>                 link_width_active.......0x2
>>                 link_speed_supported....0x1
>>                 port_state..............ACTIVE
>>                 state_info2.............0x52
>>                 m_key_protect_bits......0x0
>>                 lmc.....................0x0
>>                 link_speed..............0x11
>>                 mtu_smsl................0x40
>>                 vl_cap_init_type........0x40
>>                 vl_high_limit...........0x0
>>                 vl_arb_high_cap.........0x8
>>                 vl_arb_low_cap..........0x8
>>                 init_rep_mtu_cap........0x4
>>                 vl_stall_life...........0xF2
>>                 vl_enforce..............0x40
>>                 m_key_violations........0x0
>>                 p_key_violations........0x0
>>                 q_key_violations........0x0
>>                 guid_cap................0x0
>>                 client_reregister.......0x0
>>                 subnet_timeout..........0x0
>>                 resp_time_value.........0x0
>>                 error_threshold.........0x88
>> Mar 20 14:15:11 732128 [41401960] -> Capabilities Mask:
>> Mar 20 14:15:11 732160 [4780B960] -> __osm_sm_mad_ctrl_rcv_callback: ERR 
>> 3111: Error status = 0x1C00
>> Mar 20 14:15:11 732185 [4780B960] -> SMP dump:
>>                 base_ver................0x1
>>                 mgmt_class..............0x81
>>                 class_ver...............0x1
>>                 method..................0x81 (SubnGetResp)
>>                 D bit...................0x1
>>                 status..................0x1C00
>>                 hop_ptr.................0x0
>>                 hop_count...............0x4
>>                 trans_id................0x6b508
>>                 attr_id.................0x15 (PortInfo)
>>                 resv....................0x0
>>                 attr_mod................0x16
>>                 m_key...................0x0000000000000000
>>                 dr_slid.................0xFFFF
>>                 dr_dlid.................0xFFFF
>> 
>>                 Initial path: [0][1][16][1][4]
>>                 Return path:  [0][9][13][D][1]
>>                 Reserved:     [0][0][0][0][0][0][0]
>> 
>>                 00 00 00 00 00 00 00 00   00 00 00 00 00 00 00 00
>> 
>>                 00 00 00 00 00 00 00 00   00 00 00 00 01 03 03 02
>> 
>>                 14 52 00 11 40 40 00 08   08 04 F2 40 00 00 00 00
>> 
>>                 00 00 00 00 00 88 00 00   00 00 00 00 00 00 00 00
>> 
>> Mar 20 14:15:11 732254 [44606960] -> osm_pi_rcv_process_set: Received error 
>> status 0x1c for SetResp() during ACTIVE transition
>> Mar 20 14:15:11 732258 [44606960] -> PortInfo dump:
>>                 port number.............0x16
>>                 node_guid...............0x0005ad00000281a7
>>                 port_guid...............0x0005ad00000281a7
>>                 m_key...................0x0000000000000000
>>                 subnet_prefix...........0x0000000000000000
>>                 base_lid................0x0
>>                 master_sm_base_lid......0x0
>>                 capability_mask.........0x0
>>                 diag_code...............0x0
>>                 m_key_lease_period......0x0
>>                 local_port_num..........0x1
>>                 link_width_enabled......0x3
>>                 link_width_supported....0x3
>>                 link_width_active.......0x2
>>                 link_speed_supported....0x1
>>                 port_state..............ACTIVE
>>                 state_info2.............0x52
>>                 m_key_protect_bits......0x0
>>                 lmc.....................0x0
>>                 link_speed..............0x11
>>                 mtu_smsl................0x40
>>                 vl_cap_init_type........0x40
>>                 vl_high_limit...........0x0
>>                 vl_arb_high_cap.........0x8
>>                 vl_arb_low_cap..........0x8
>>                 init_rep_mtu_cap........0x4
>>                 vl_stall_life...........0xF2
>>                 vl_enforce..............0x40
>>                 m_key_violations........0x0
>>                 p_key_violations........0x0
>>                 q_key_violations........0x0
>>                 guid_cap................0x0
>>                 client_reregister.......0x0
>>                 subnet_timeout..........0x0
>>                 resp_time_value.........0x0
>>                 error_threshold.........0x88
>> Mar 20 14:15:11 732269 [44606960] -> Capabilities Mask:
>> Mar 20 14:15:11 732300 [4780B960] -> __osm_sm_mad_ctrl_rcv_callback: ERR 
>> 3111: Error status = 0x1C00
>> Mar 20 14:15:11 732334 [4780B960] -> SMP dump:
>>                 base_ver................0x1
>>                 mgmt_class..............0x81
>>                 class_ver...............0x1
>>                 method..................0x81 (SubnGetResp)
>>                 D bit...................0x1
>>                 status..................0x1C00
>>                 hop_ptr.................0x0
>>                 hop_count...............0x4
>>                 trans_id................0x6b509
>>                 attr_id.................0x15 (PortInfo)
>>                 resv....................0x0
>>                 attr_mod................0x17
>>                 m_key...................0x0000000000000000
>>                 dr_slid.................0xFFFF
>>                 dr_dlid.................0xFFFF
>> 
>>                 Initial path: [0][1][16][1][4]
>>                 Return path:  [0][9][13][D][1]
>>                 Reserved:     [0][0][0][0][0][0][0]
>> 
>>                 00 00 00 00 00 00 00 00   00 00 00 00 00 00 00 00
>> 
>>                 00 00 00 00 00 00 00 00   00 00 00 00 01 03 03 02
>> 
>>                 14 52 00 11 40 40 00 08   08 04 F2 40 00 00 00 00
>> 
>>                 00 00 00 00 00 88 00 00   00 00 00 00 00 00 00 00
>> 
>> Mar 20 14:15:11 732420 [4780B960] -> __osm_sm_mad_ctrl_rcv_callback: ERR 
>> 3111: Error status = 0x1C00
>> Mar 20 14:15:11 732419 [45007960] -> osm_pi_rcv_process_set: Received error 
>> status 0x1c for SetResp() during ACTIVE transition
>> Mar 20 14:15:11 732451 [4780B960] -> SMP dump:
>>                 base_ver................0x1
>>                 mgmt_class..............0x81
>>                 class_ver...............0x1
>>                 method..................0x81 (SubnGetResp)
>>                 D bit...................0x1
>>                 status..................0x1C00
>>                 hop_ptr.................0x0
>>                 hop_count...............0x4
>>                 trans_id................0x6b50a
>>                 attr_id.................0x15 (PortInfo)
>>                 resv....................0x0
>>                 attr_mod................0x18
>>                 m_key...................0x0000000000000000
>>                 dr_slid.................0xFFFF
>>                 dr_dlid.................0xFFFF
>> 
>>                 Initial path: [0][1][16][1][4]
>>                 Return path:  [0][9][13][D][1]
>>                 Reserved:     [0][0][0][0][0][0][0]
>> 
>>                 00 00 00 00 00 00 00 00   00 00 00 00 00 00 00 00
>> 
>>                 00 00 00 00 00 00 00 00   00 00 00 00 01 03 03 02
>> 
>>                 14 52 00 11 40 40 00 08   08 04 F2 40 00 00 00 00
>> 
>>                 00 00 00 00 00 88 00 00   00 00 00 00 00 00 00 00
>> 
>> Mar 20 14:15:11 732447 [45007960] -> PortInfo dump:
>>                 port number.............0x17
>>                 node_guid...............0x0005ad00000281a7
>>                 port_guid...............0x0005ad00000281a7
>>                 m_key...................0x0000000000000000
>>                 subnet_prefix...........0x0000000000000000
>>                 base_lid................0x0
>>                 master_sm_base_lid......0x0
>>                 capability_mask.........0x0
>>                 diag_code...............0x0
>>                 m_key_lease_period......0x0
>>                 local_port_num..........0x1
>>                 link_width_enabled......0x3
>>                 link_width_supported....0x3
>>                 link_width_active.......0x2
>>                 link_speed_supported....0x1
>>                 port_state..............ACTIVE
>>                 state_info2.............0x52
>>                 m_key_protect_bits......0x0
>>                 lmc.....................0x0
>>                 link_speed..............0x11
>>                 mtu_smsl................0x40
>>                 vl_cap_init_type........0x40
>>                 vl_high_limit...........0x0
>>                 vl_arb_high_cap.........0x8
>>                 vl_arb_low_cap..........0x8
>>                 init_rep_mtu_cap........0x4
>>                 vl_stall_life...........0xF2
>>                 vl_enforce..............0x40
>>                 m_key_violations........0x0
>>                 p_key_violations........0x0
>>                 q_key_violations........0x0
>>                 guid_cap................0x0
>>                 client_reregister.......0x0
>>                 subnet_timeout..........0x0
>>                 resp_time_value.........0x0
>>                 error_threshold.........0x88
>> Mar 20 14:15:11 732471 [45007960] -> Capabilities Mask:
>> Mar 20 14:15:11 732511 [45007960] -> osm_pi_rcv_process_set: Received error 
>> status 0x1c for SetResp() during ACTIVE transition
>> Mar 20 14:15:11 732516 [45007960] -> PortInfo dump:
>>                 port number.............0x18
>>                 node_guid...............0x0005ad00000281a7
>>                 port_guid...............0x0005ad00000281a7
>>                 m_key...................0x0000000000000000
>>                 subnet_prefix...........0x0000000000000000
>>                 base_lid................0x0
>>                 master_sm_base_lid......0x0
>>                 capability_mask.........0x0
>>                 diag_code...............0x0
>>                 m_key_lease_period......0x0
>>                 local_port_num..........0x1
>>                 link_width_enabled......0x3
>>                 link_width_supported....0x3
>>                 link_width_active.......0x2
>>                 link_speed_supported....0x1
>>                 port_state..............ACTIVE
>>                 state_info2.............0x52
>>                 m_key_protect_bits......0x0
>>                 lmc.....................0x0
>>                 link_speed..............0x11
>>                 mtu_smsl................0x40
>>                 vl_cap_init_type........0x40
>>                 vl_high_limit...........0x0
>>                 vl_arb_high_cap.........0x8
>>                 vl_arb_low_cap..........0x8
>>                 init_rep_mtu_cap........0x4
>>                 vl_stall_life...........0xF2
>>                 vl_enforce..............0x40
>>                 m_key_violations........0x0
>>                 p_key_violations........0x0
>>                 q_key_violations........0x0
>>                 guid_cap................0x0
>>                 client_reregister.......0x0
>>                 subnet_timeout..........0x0
>>                 resp_time_value.........0x0
>>                 error_threshold.........0x88
>> Mar 20 14:15:11 732529 [45007960] -> Capabilities Mask:
>> Mar 20 14:15:11 732556 [4780B960] -> __osm_sm_mad_ctrl_rcv_callback: ERR 
>> 3111: Error status = 0x1C00
>> Mar 20 14:15:11 732591 [4780B960] -> SMP dump:
>>                 base_ver................0x1
>>                 mgmt_class..............0x81
>>                 class_ver...............0x1
>>                 method..................0x81 (SubnGetResp)
>>                 D bit...................0x1
>>                 status..................0x1C00
>>                 hop_ptr.................0x0
>>                 hop_count...............0x4
>>                 trans_id................0x6b50b
>>                 attr_id.................0x15 (PortInfo)
>>                 resv....................0x0
>>                 attr_mod................0x16
>>                 m_key...................0x0000000000000000
>>                 dr_slid.................0xFFFF
>>                 dr_dlid.................0xFFFF
>> 
>>                 Initial path: [0][1][11][2][5]
>>                 Return path:  [0][9][18][E][2]
>>                 Reserved:     [0][0][0][0][0][0][0]
>> 
>>                 00 00 00 00 00 00 00 00   00 00 00 00 00 00 00 00
>> 
>>                 00 00 00 00 00 00 00 00   00 00 00 00 02 03 03 02
>> 
>>                 14 52 00 11 40 40 00 08   08 04 F2 40 00 00 00 00
>> 
>>                 00 00 00 00 00 88 00 00   00 00 00 00 00 00 00 00
>> 
>> Mar 20 14:15:11 732653 [43204960] -> osm_pi_rcv_process_set: Received error 
>> status 0x1c for SetResp() during ACTIVE transition
>> Mar 20 14:15:11 732662 [43204960] -> PortInfo dump:
>>                 port number.............0x16
>>                 node_guid...............0x0005ad00000281b3
>>                 port_guid...............0x0005ad00000281b3
>>                 m_key...................0x0000000000000000
>>                 subnet_prefix...........0x0000000000000000
>>                 base_lid................0x0
>>                 master_sm_base_lid......0x0
>>                 capability_mask.........0x0
>>                 diag_code...............0x0
>>                 m_key_lease_period......0x0
>>                 local_port_num..........0x2
>>                 link_width_enabled......0x3
>>                 link_width_supported....0x3
>>                 link_width_active.......0x2
>>                 link_speed_supported....0x1
>>                 port_state..............ACTIVE
>>                 state_info2.............0x52
>>                 m_key_protect_bits......0x0
>>                 lmc.....................0x0
>>                 link_speed..............0x11
>>                 mtu_smsl................0x40
>>                 vl_cap_init_type........0x40
>>                 vl_high_limit...........0x0
>>                 vl_arb_high_cap.........0x8
>>                 vl_arb_low_cap..........0x8
>>                 init_rep_mtu_cap........0x4
>>                 vl_stall_life...........0xF2
>>                 vl_enforce..............0x40
>>                 m_key_violations........0x0
>>                 p_key_violations........0x0
>>                 q_key_violations........0x0
>>                 guid_cap................0x0
>>                 client_reregister.......0x0
>>                 subnet_timeout..........0x0
>>                 resp_time_value.........0x0
>>                 error_threshold.........0x88
>> Mar 20 14:15:11 732673 [43204960] -> Capabilities Mask:
>> Mar 20 14:15:11 732705 [4780B960] -> __osm_sm_mad_ctrl_rcv_callback: ERR 
>> 3111: Error status = 0x1C00
>> Mar 20 14:15:11 732739 [4780B960] -> SMP dump:
>>                 base_ver................0x1
>>                 mgmt_class..............0x81
>>                 class_ver...............0x1
>>                 method..................0x81 (SubnGetResp)
>>                 D bit...................0x1
>>                 status..................0x1C00
>>                 hop_ptr.................0x0
>>                 hop_count...............0x4
>>                 trans_id................0x6b50c
>>                 attr_id.................0x15 (PortInfo)
>>                 resv....................0x0
>>                 attr_mod................0x17
>>                 m_key...................0x0000000000000000
>>                 dr_slid.................0xFFFF
>>                 dr_dlid.................0xFFFF
>> 
>>                 Initial path: [0][1][11][2][5]
>>                 Return path:  [0][9][18][E][2]
>>                 Reserved:     [0][0][0][0][0][0][0]
>> 
>>                 00 00 00 00 00 00 00 00   00 00 00 00 00 00 00 00
>> 
>>                 00 00 00 00 00 00 00 00   00 00 00 00 02 03 03 02
>> 
>>                 14 52 00 11 40 40 00 08   08 04 F2 40 00 00 00 00
>> 
>>                 00 00 00 00 00 88 00 00   00 00 00 00 00 00 00 00
>> 
>> Mar 20 14:15:11 732809 [4780B960] -> __osm_sm_mad_ctrl_rcv_callback: ERR 
>> 3111: Error status = 0x1C00
>> Mar 20 14:15:11 732805 [41E02960] -> osm_pi_rcv_process_set: Received error 
>> status 0x1c for SetResp() during ACTIVE transition
>> Mar 20 14:15:11 732839 [4780B960] -> SMP dump:
>>                 base_ver................0x1
>>                 mgmt_class..............0x81
>>                 class_ver...............0x1
>>                 method..................0x81 (SubnGetResp)
>>                 D bit...................0x1
>>                 status..................0x1C00
>>                 hop_ptr.................0x0
>>                 hop_count...............0x4
>>                 trans_id................0x6b50d
>>                 attr_id.................0x15 (PortInfo)
>>                 resv....................0x0
>>                 attr_mod................0x18
>>                 m_key...................0x0000000000000000
>>                 dr_slid.................0xFFFF
>>                 dr_dlid.................0xFFFF
>> 
>>                 Initial path: [0][1][11][2][5]
>>                 Return path:  [0][9][18][E][2]
>>                 Reserved:     [0][0][0][0][0][0][0]
>> 
>>                 00 00 00 00 00 00 00 00   00 00 00 00 00 00 00 00
>> 
>>                 00 00 00 00 00 00 00 00   00 00 00 00 02 03 03 02
>> 
>>                 14 52 00 11 40 40 00 08   08 04 F2 40 00 00 00 00
>> 
>>                 00 00 00 00 00 88 00 00   00 00 00 00 00 00 00 00
>> 
>> Mar 20 14:15:11 732837 [41E02960] -> PortInfo dump:
>>                 port number.............0x17
>>                 node_guid...............0x0005ad00000281b3
>>                 port_guid...............0x0005ad00000281b3
>>                 m_key...................0x0000000000000000
>>                 subnet_prefix...........0x0000000000000000
>>                 base_lid................0x0
>>                 master_sm_base_lid......0x0
>>                 capability_mask.........0x0
>>                 diag_code...............0x0
>>                 m_key_lease_period......0x0
>>                 local_port_num..........0x2
>>                 link_width_enabled......0x3
>>                 link_width_supported....0x3
>>                 link_width_active.......0x2
>>                 link_speed_supported....0x1
>>                 port_state..............ACTIVE
>>                 state_info2.............0x52
>>                 m_key_protect_bits......0x0
>>                 lmc.....................0x0
>>                 link_speed..............0x11
>>                 mtu_smsl................0x40
>>                 vl_cap_init_type........0x40
>>                 vl_high_limit...........0x0
>>                 vl_arb_high_cap.........0x8
>>                 vl_arb_low_cap..........0x8
>>                 init_rep_mtu_cap........0x4
>>                 vl_stall_life...........0xF2
>>                 vl_enforce..............0x40
>>                 m_key_violations........0x0
>>                 p_key_violations........0x0
>>                 q_key_violations........0x0
>>                 guid_cap................0x0
>>                 client_reregister.......0x0
>>                 subnet_timeout..........0x0
>>                 resp_time_value.........0x0
>>                 error_threshold.........0x88
>> Mar 20 14:15:11 732856 [41E02960] -> Capabilities Mask:
>> Mar 20 14:15:11 732898 [41E02960] -> osm_pi_rcv_process_set: Received error 
>> status 0x1c for SetResp() during ACTIVE transition
>> Mar 20 14:15:11 732911 [41E02960] -> PortInfo dump:
>>                 port number.............0x18
>>                 node_guid...............0x0005ad00000281b3
>>                 port_guid...............0x0005ad00000281b3
>>                 m_key...................0x0000000000000000
>>                 subnet_prefix...........0x0000000000000000
>>                 base_lid................0x0
>>                 master_sm_base_lid......0x0
>>                 capability_mask.........0x0
>>                 diag_code...............0x0
>>                 m_key_lease_period......0x0
>>                 local_port_num..........0x2
>>                 link_width_enabled......0x3
>>                 link_width_supported....0x3
>>                 link_width_active.......0x2
>>                 link_speed_supported....0x1
>>                 port_state..............ACTIVE
>>                 state_info2.............0x52
>>                 m_key_protect_bits......0x0
>>                 lmc.....................0x0
>>                 link_speed..............0x11
>>                 mtu_smsl................0x40
>>                 vl_cap_init_type........0x40
>>                 vl_high_limit...........0x0
>>                 vl_arb_high_cap.........0x8
>>                 vl_arb_low_cap..........0x8
>>                 init_rep_mtu_cap........0x4
>>                 vl_stall_life...........0xF2
>>                 vl_enforce..............0x40
>>                 m_key_violations........0x0
>>                 p_key_violations........0x0
>>                 q_key_violations........0x0
>>                 guid_cap................0x0
>>                 client_reregister.......0x0
>>                 subnet_timeout..........0x0
>>                 resp_time_value.........0x0
>>                 error_threshold.........0x88
>> Mar 20 14:15:11 732925 [41E02960] -> Capabilities Mask:
>> Mar 20 14:15:11 738354 [45A08960] -> SUBNET UP
>> Mar 20 14:15:12 115658 [44606960] -> osm_ucast_mgr_process: Min Hop Tables 
>> configured on all switches
>> Mar 20 14:15:12 172029 [44606960] -> SUBNET UP
>> Mar 20 14:15:27 277617 [41401960] -> __osm_trap_rcv_process_request: 
>> Received Generic Notice type:0x01 num:128 Producer:2 from LID:0x0152 
>> TID:0x0000000000000000
>> Mar 20 14:15:27 277863 [41401960] -> osm_report_notice: Reporting Generic 
>> Notice type:1 num:128 from LID:0x0152 
>> GID:0xfe80000000000000,0x0005ad0000027c84
>> Mar 20 14:15:27 510410 [43C05960] -> __osm_trap_rcv_process_request: 
>> Received Generic Notice type:0x01 num:128 Producer:2 from LID:0x0152 
>> TID:0x0000000000000001
>> Mar 20 14:15:27 510626 [43C05960] -> osm_report_notice: Reporting Generic 
>> Notice type:1 num:128 from LID:0x0152 
>> GID:0xfe80000000000000,0x0005ad0000027c84
>> Mar 20 14:15:27 532239 [41E02960] -> __osm_trap_rcv_process_request: 
>> Received Generic Notice type:0x01 num:128 Producer:2 from LID:0x0152 
>> TID:0x0000000000000002
>> Mar 20 14:15:27 532443 [41E02960] -> osm_report_notice: Reporting Generic 
>> Notice type:1 num:128 from LID:0x0152 
>> GID:0xfe80000000000000,0x0005ad0000027c84
>> Mar 20 14:15:27 533517 [45A08960] -> __osm_trap_rcv_process_request: 
>> Received Generic Notice type:0x01 num:128 Producer:2 from LID:0x0152 
>> TID:0x0000000000000003
>> Mar 20 14:15:27 533612 [45A08960] -> osm_report_notice: Reporting Generic 
>> Notice type:1 num:128 from LID:0x0152 
>> GID:0xfe80000000000000,0x0005ad0000027c84
>> Mar 20 14:15:27 591171 [41401960] -> osm_report_notice: Reporting Generic 
>> Notice type:3 num:65 from LID:0x0092 
>> GID:0xfe80000000000000,0x0005ad0000024bbb
>> Mar 20 14:15:27 591185 [41401960] -> Removed port with 
>> GUID:0x0005ad0000024da7 LID range [0xB0,0xB0] of node:saguaro-23-1 HCA-1
>> Mar 20 14:15:27 591206 [41401960] -> osm_report_notice: Reporting Generic 
>> Notice type:3 num:65 from LID:0x0092 
>> GID:0xfe80000000000000,0x0005ad0000024bbb
>> Mar 20 14:15:27 591211 [41401960] -> Removed port with 
>> GUID:0x0005ad00000249d3 LID range [0xB1,0xB1] of node:saguaro-23-2 HCA-1
>> Mar 20 14:15:27 625811 [41401960] -> osm_ucast_mgr_process: Min Hop Tables 
>> configured on all switches
>> Mar 20 14:15:27 668356 [41401960] -> __osm_trap_rcv_process_request: 
>> Received Generic Notice type:0x01 num:128 Producer:2 from LID:0x0152 
>> TID:0x0000000000000004
>> Mar 20 14:15:27 668485 [41401960] -> osm_report_notice: Reporting Generic 
>> Notice type:1 num:128 from LID:0x0152 
>> GID:0xfe80000000000000,0x0005ad0000027c84
>> Mar 20 14:15:27 682282 [43204960] -> SUBNET UP
>> Mar 20 14:15:27 737313 [41E02960] -> __osm_trap_rcv_process_request: 
>> Received Generic Notice type:0x01 num:128 Producer:2 from LID:0x0152 
>> TID:0x0000000000000005
>> Mar 20 14:15:27 737387 [41E02960] -> osm_report_notice: Reporting Generic 
>> Notice type:1 num:128 from LID:0x0152 
>> GID:0xfe80000000000000,0x0005ad0000027c84
>> Mar 20 14:15:27 809341 [42803960] -> __osm_trap_rcv_process_request: 
>> Received Generic Notice type:0x01 num:128 Producer:2 from LID:0x0152 
>> TID:0x0000000000000006
>> Mar 20 14:15:27 809813 [42803960] -> osm_report_notice: Reporting Generic 
>> Notice type:1 num:128 from LID:0x0152 
>> GID:0xfe80000000000000,0x0005ad0000027c84
>> Mar 20 14:15:27 998181 [45007960] -> __osm_trap_rcv_process_request: 
>> Received Generic Notice type:0x01 num:128 Producer:2 from LID:0x0152 
>> TID:0x0000000000000007
>> Mar 20 14:15:27 998331 [45007960] -> osm_report_notice: Reporting Generic 
>> Notice type:1 num:128 from LID:0x0152 
>> GID:0xfe80000000000000,0x0005ad0000027c84
>> Mar 20 14:15:28 012193 [45007960] -> __osm_trap_rcv_process_request: 
>> Received Generic Notice type:0x01 num:128 Producer:2 from LID:0x0152 
>> TID:0x0000000000000008
>> Mar 20 14:15:28 012277 [45007960] -> osm_report_notice: Reporting Generic 
>> Notice type:1 num:128 from LID:0x0152 
>> GID:0xfe80000000000000,0x0005ad0000027c84
>> Mar 20 14:15:28 496329 [43204960] -> __osm_trap_rcv_process_request: 
>> Received Generic Notice type:0x01 num:128 Producer:2 from LID:0x0152 
>> TID:0x0000000000000009
>> Mar 20 14:15:28 496422 [43204960] -> osm_report_notice: Reporting Generic 
>> Notice type:1 num:128 from LID:0x0152 
>> GID:0xfe80000000000000,0x0005ad0000027c84
>> Mar 20 14:15:28 624912 [43C05960] -> osm_report_notice: Reporting Generic 
>> Notice type:3 num:65 from LID:0x0092 
>> GID:0xfe80000000000000,0x0005ad0000024bbb
>> Mar 20 14:15:28 624940 [43C05960] -> Removed port with 
>> GUID:0x0005ad0000024e0b LID range [0xB3,0xB3] of node:saguaro-23-4 HCA-1
>> Mar 20 14:15:28 624965 [43C05960] -> osm_report_notice: Reporting Generic 
>> Notice type:3 num:65 from LID:0x0092 
>> GID:0xfe80000000000000,0x0005ad0000024bbb
>> Mar 20 14:15:28 624972 [43C05960] -> Removed port with 
>> GUID:0x0005ad0000025043 LID range [0xB4,0xB4] of node:saguaro-23-5 HCA-1
>> Mar 20 14:15:28 625001 [43C05960] -> osm_report_notice: Reporting Generic 
>> Notice type:3 num:65 from LID:0x0092 
>> GID:0xfe80000000000000,0x0005ad0000024bbb
>> Mar 20 14:15:28 625008 [43C05960] -> Removed port with 
>> GUID:0x0005ad0000024cbb LID range [0xB2,0xB2] of node:saguaro-23-3 HCA-1
>> Mar 20 14:15:28 629507 [42803960] -> osm_report_notice: Reporting Generic 
>> Notice type:3 num:64 from LID:0x0092 
>> GID:0xfe80000000000000,0x0005ad0000024bbb
>> Mar 20 14:15:28 629518 [42803960] -> Discovered new port with 
>> GUID:0x0005ad00000249d3 LID range [0xB1,0xB1] of node:saguaro-23-2 HCA-1
>> Mar 20 14:15:28 649776 [43204960] -> __osm_trap_rcv_process_request: 
>> Received Generic Notice type:0x01 num:128 Producer:2 from LID:0x0152 
>> TID:0x000000000000000a
>> Mar 20 14:15:28 660297 [42803960] -> osm_ucast_mgr_process: Min Hop Tables 
>> configured on all switches
>> Mar 20 14:15:28 699777 [43204960] -> osm_report_notice: Reporting Generic 
>> Notice type:1 num:128 from LID:0x0152 
>> GID:0xfe80000000000000,0x0005ad0000027c84
>> Mar 20 14:15:28 716354 [41E02960] -> SUBNET UP
>> Mar 20 14:15:28 744686 [45007960] -> __osm_trap_rcv_process_request: 
>> Received Generic Notice type:0x01 num:128 Producer:2 from LID:0x0152 
>> TID:0x000000000000000b
>> Mar 20 14:15:28 744857 [45007960] -> __osm_trap_rcv_process_request: ERR 
>> 3804: Received trap 11 times consecutively
>> Mar 20 14:15:28 811329 [45A08960] -> __osm_trap_rcv_process_request: 
>> Received Generic Notice type:0x01 num:128 Producer:2 from LID:0x0152 
>> TID:0x000000000000000c
>> Mar 20 14:15:28 811392 [45A08960] -> __osm_trap_rcv_process_request: ERR 
>> 3804: Received trap 12 times consecutively
>> Mar 20 14:15:28 999808 [45007960] -> __osm_trap_rcv_process_request: 
>> Received Generic Notice type:0x01 num:128 Producer:2 from LID:0x0152 
>> TID:0x000000000000000d
>> Mar 20 14:15:28 999881 [45007960] -> __osm_trap_rcv_process_request: ERR 
>> 3804: Received trap 13 times consecutively
>> Mar 20 14:15:29 029918 [43C05960] -> __osm_trap_rcv_process_request: 
>> Received Generic Notice type:0x01 num:128 Producer:2 from LID:0x0152 
>> TID:0x000000000000000e
>> Mar 20 14:15:29 029969 [43C05960] -> __osm_trap_rcv_process_request: ERR 
>> 3804: Received trap 14 times consecutively
>> Mar 20 14:15:29 031783 [45A08960] -> __osm_trap_rcv_process_request: 
>> Received Generic Notice type:0x01 num:128 Producer:2 from LID:0x0148 
>> TID:0x0000000000000052
>> Mar 20 14:15:29 031900 [45A08960] -> osm_report_notice: Reporting Generic 
>> Notice type:1 num:128 from LID:0x0148 
>> GID:0xfe80000000000000,0x0005ad00000281b3
>> Mar 20 14:15:29 037646 [44606960] -> osm_report_notice: Reporting Generic 
>> Notice type:3 num:65 from LID:0x0092 
>> GID:0xfe80000000000000,0x0005ad0000024bbb
>> Mar 20 14:15:29 037662 [44606960] -> Removed port with 
>> GUID:0x0005ad0000024d6b LID range [0xB8,0xB8] of node:saguaro-23-9 HCA-1
>> Mar 20 14:15:29 037683 [44606960] -> osm_report_notice: Reporting Generic 
>> Notice type:3 num:65 from LID:0x0092 
>> GID:0xfe80000000000000,0x0005ad0000024bbb
>> Mar 20 14:15:29 037690 [44606960] -> Removed port with 
>> GUID:0x0005ad0000024977 LID range [0xA9,0xA9] of node:saguaro-22-4 HCA-1
>> Mar 20 14:15:29 037721 [44606960] -> osm_report_notice: Reporting Generic 
>> Notice type:3 num:65 from LID:0x0092 
>> GID:0xfe80000000000000,0x0005ad0000024bbb
>> Mar 20 14:15:29 037726 [44606960] -> Removed port with 
>> GUID:0x0005ad000002498f LID range [0xA8,0xA8] of node:saguaro-22-3 HCA-1
>> Mar 20 14:15:29 037741 [44606960] -> osm_report_notice: Reporting Generic 
>> Notice type:3 num:65 from LID:0x0092 
>> GID:0xfe80000000000000,0x0005ad0000024bbb
>> Mar 20 14:15:29 037746 [44606960] -> Removed port with 
>> GUID:0x0005ad0000024c9b LID range [0xA7,0xA7] of node:saguaro-22-2 HCA-1
>> Mar 20 14:15:29 037766 [44606960] -> osm_report_notice: Reporting Generic 
>> Notice type:3 num:65 from LID:0x0092 
>> GID:0xfe80000000000000,0x0005ad0000024bbb
>> Mar 20 14:15:29 037771 [44606960] -> Removed port with 
>> GUID:0x0005ad0000024afb LID range [0xA5,0xA5] of node:saguaro-22-0 HCA-1
>> Mar 20 14:15:29 361560 [42803960] -> __osm_trap_rcv_process_request: 
>> Received Generic Notice type:0x01 num:128 Producer:2 from LID:0x0148 
>> TID:0x0000000000000053
>> Mar 20 14:15:29 361622 [42803960] -> osm_report_notice: Reporting Generic 
>> Notice type:1 num:128 from LID:0x0148 
>> GID:0xfe80000000000000,0x0005ad00000281b3
>> Mar 20 14:15:29 433665 [43204960] -> osm_report_notice: Reporting Generic 
>> Notice type:3 num:64 from LID:0x0092 
>> GID:0xfe80000000000000,0x0005ad0000024bbb
>> Mar 20 14:15:29 433674 [43204960] -> Discovered new port with 
>> GUID:0x0005ad0000024da7 LID range [0xB0,0xB0] of node:saguaro-23-1 HCA-1
>> Mar 20 14:15:29 433680 [43204960] -> osm_report_notice: Reporting Generic 
>> Notice type:3 num:64 from LID:0x0092 
>> GID:0xfe80000000000000,0x0005ad0000024bbb
>> Mar 20 14:15:29 433687 [43204960] -> Discovered new port with 
>> GUID:0x0005ad0000024cbb LID range [0xB2,0xB2] of node:saguaro-23-3 HCA-1
>> Mar 20 14:15:29 433692 [43204960] -> osm_report_notice: Reporting Generic 
>> Notice type:3 num:64 from LID:0x0092 
>> GID:0xfe80000000000000,0x0005ad0000024bbb
>> Mar 20 14:15:29 433698 [43204960] -> Discovered new port with 
>> GUID:0x0005ad0000024e0b LID range [0xB3,0xB3] of node:saguaro-23-4 HCA-1
>> Mar 20 14:15:29 433703 [43204960] -> osm_report_notice: Reporting Generic 
>> Notice type:3 num:64 from LID:0x0092 
>> GID:0xfe80000000000000,0x0005ad0000024bbb
>> Mar 20 14:15:29 433709 [43204960] -> Discovered new port with 
>> GUID:0x0005ad0000025043 LID range [0xB4,0xB4] of node:saguaro-23-5 HCA-1
>> Mar 20 14:15:29 464434 [43204960] -> osm_ucast_mgr_process: Min Hop Tables 
>> configured on all switches
>> Mar 20 14:15:29 522011 [42803960] -> SUBNET UP
>> Mar 20 14:15:29 699605 [41E02960] -> __osm_trap_rcv_process_request: 
>> Received Generic Notice type:0x01 num:128 Producer:2 from LID:0x001B 
>> TID:0x000000000000006b
>> Mar 20 14:15:29 699782 [41E02960] -> osm_report_notice: Reporting Generic 
>> Notice type:1 num:128 from LID:0x001B 
>> GID:0xfe80000000000000,0x0005ad00000281a7
>> Mar 20 14:15:29 701115 [45A08960] -> __osm_trap_rcv_process_request: 
>> Received Generic Notice type:0x01 num:128 Producer:2 from LID:0x0148 
>> TID:0x0000000000000054
>> Mar 20 14:15:29 701301 [45A08960] -> osm_report_notice: Reporting Generic 
>> Notice type:1 num:128 from LID:0x0148 
>> GID:0xfe80000000000000,0x0005ad00000281b3
>> Mar 20 14:15:29 818974 [41E02960] -> __osm_trap_rcv_process_request: 
>> Received Generic Notice type:0x01 num:128 Producer:2 from LID:0x0148 
>> TID:0x0000000000000055
>> Mar 20 14:15:29 819054 [41E02960] -> osm_report_notice: Reporting Generic 
>> Notice type:1 num:128 from LID:0x0148 
>> GID:0xfe80000000000000,0x0005ad00000281b3
>> Mar 20 14:15:29 992006 [41E02960] -> __osm_trap_rcv_process_request: 
>> Received Generic Notice type:0x01 num:128 Producer:2 from LID:0x0148 
>> TID:0x0000000000000056
>> Mar 20 14:15:29 992080 [41E02960] -> osm_report_notice: Reporting Generic 
>> Notice type:1 num:128 from LID:0x0148 
>> GID:0xfe80000000000000,0x0005ad00000281b3
>> Mar 20 14:15:30 184132 [44606960] -> __osm_trap_rcv_process_request: 
>> Received Generic Notice type:0x01 num:128 Producer:2 from LID:0x001B 
>> TID:0x000000000000006c
>> Mar 20 14:15:30 184205 [44606960] -> osm_report_notice: Reporting Generic 
>> Notice type:1 num:128 from LID:0x001B 
>> GID:0xfe80000000000000,0x0005ad00000281a7
>> Mar 20 14:15:30 207030 [43204960] -> __osm_trap_rcv_process_request: 
>> Received Generic Notice type:0x01 num:128 Producer:2 from LID:0x0148 
>> TID:0x0000000000000057
>> Mar 20 14:15:30 207101 [43204960] -> osm_report_notice: Reporting Generic 
>> Notice type:1 num:128 from LID:0x0148 
>> GID:0xfe80000000000000,0x0005ad00000281b3
>> Mar 20 14:15:30 250541 [43C05960] -> __osm_trap_rcv_process_request: 
>> Received Generic Notice type:0x01 num:128 Producer:2 from LID:0x001B 
>> TID:0x000000000000006d
>> Mar 20 14:15:30 250635 [43C05960] -> osm_report_notice: Reporting Generic 
>> Notice type:1 num:128 from LID:0x001B 
>> GID:0xfe80000000000000,0x0005ad00000281a7
>> Mar 20 14:15:30 317366 [45A08960] -> osm_drop_mgr_process: ERR 0108: Unknown 
>> remote side for node 0x0005ad00000281a7 port 22. Adding to light sweep 
>> sampling list
>> Mar 20 14:15:30 317409 [45A08960] -> Directed Path Dump of 4 hop path:
>>                 Path = [0][1][17][1][4]
>> Mar 20 14:15:30 494183 [41401960] -> __osm_trap_rcv_process_request: 
>> Received Generic Notice type:0x01 num:128 Producer:2 from LID:0x001B 
>> TID:0x000000000000006e
>> Mar 20 14:15:30 494247 [41401960] -> osm_report_notice: Reporting Generic 
>> Notice type:1 num:128 from LID:0x001B 
>> GID:0xfe80000000000000,0x0005ad00000281a7
>> Mar 20 14:15:30 521869 [43C05960] -> osm_report_notice: Reporting Generic 
>> Notice type:3 num:64 from LID:0x0092 
>> GID:0xfe80000000000000,0x0005ad0000024bbb
>> Mar 20 14:15:30 521879 [43C05960] -> Discovered new port with 
>> GUID:0x0005ad0000024d6b LID range [0xB8,0xB8] of node:saguaro-23-9 HCA-1
>> Mar 20 14:15:30 521885 [43C05960] -> osm_report_notice: Reporting Generic 
>> Notice type:3 num:64 from LID:0x0092 
>> GID:0xfe80000000000000,0x0005ad0000024bbb
>> Mar 20 14:15:30 521891 [43C05960] -> Discovered new port with 
>> GUID:0x0005ad0000024afb LID range [0xA5,0xA5] of node:saguaro-22-0 HCA-1
>> Mar 20 14:15:30 521896 [43C05960] -> osm_report_notice: Reporting Generic 
>> Notice type:3 num:64 from LID:0x0092 
>> GID:0xfe80000000000000,0x0005ad0000024bbb
>> Mar 20 14:15:30 521903 [43C05960] -> Discovered new port with 
>> GUID:0x0005ad0000024c9b LID range [0xA7,0xA7] of node:saguaro-22-2 HCA-1
>> Mar 20 14:15:30 521908 [43C05960] -> osm_report_notice: Reporting Generic 
>> Notice type:3 num:64 from LID:0x0092 
>> GID:0xfe80000000000000,0x0005ad0000024bbb
>> Mar 20 14:15:30 521914 [43C05960] -> Discovered new port with 
>> GUID:0x0005ad000002498f LID range [0xA8,0xA8] of node:saguaro-22-3 HCA-1
>> Mar 20 14:15:30 521919 [43C05960] -> osm_report_notice: Reporting Generic 
>> Notice type:3 num:64 from LID:0x0092 
>> GID:0xfe80000000000000,0x0005ad0000024bbb
>> Mar 20 14:15:30 521926 [43C05960] -> Discovered new port with 
>> GUID:0x0005ad0000024977 LID range [0xA9,0xA9] of node:saguaro-22-4 HCA-1
>> Mar 20 14:15:30 552581 [43C05960] -> osm_ucast_mgr_process: Min Hop Tables 
>> configured on all switches
>> Mar 20 14:15:30 553014 [45A08960] -> __osm_trap_rcv_process_request: 
>> Received Generic Notice type:0x01 num:128 Producer:2 from LID:0x001B 
>> TID:0x000000000000006f
>> Mar 20 14:15:30 592863 [45A08960] -> osm_report_notice: Reporting Generic 
>> Notice type:1 num:128 from LID:0x001B 
>> GID:0xfe80000000000000,0x0005ad00000281a7
>> Mar 20 14:15:30 607595 [4780B960] -> __osm_sm_mad_ctrl_rcv_callback: ERR 
>> 3111: Error status = 0x1C00
>> Mar 20 14:15:30 607666 [4780B960] -> SMP dump:
>>                 base_ver................0x1
>>                 mgmt_class..............0x81
>>                 class_ver...............0x1
>>                 method..................0x81 (SubnGetResp)
>>                 D bit...................0x1
>>                 status..................0x1C00
>>                 hop_ptr.................0x0
>>                 hop_count...............0x4
>>                 trans_id................0x6f744
>>                 attr_id.................0x15 (PortInfo)
>>                 resv....................0x0
>>                 attr_mod................0x16
>>                 m_key...................0x0000000000000000
>>                 dr_slid.................0xFFFF
>>                 dr_dlid.................0xFFFF
>> 
>>                 Initial path: [0][1][14][1][6]
>>                 Return path:  [0][9][15][D][3]
>>                 Reserved:     [0][0][0][0][0][0][0]
>> 
>>                 00 00 00 00 00 00 00 00   00 00 00 00 00 00 00 00
>> 
>>                 00 00 00 00 00 00 00 00   00 00 00 00 03 03 03 02
>> 
>>                 14 52 00 11 40 40 00 08   08 04 F2 40 00 00 00 00
>> 
>>                 00 00 00 00 00 88 00 00   00 00 00 00 00 00 00 00
>> 
>> Mar 20 14:15:30 607770 [44606960] -> osm_pi_rcv_process_set: Received error 
>> status 0x1c for SetResp() during ACTIVE transition
>> Mar 20 14:15:30 607777 [44606960] -> PortInfo dump:
>>                 port number.............0x16
>>                 node_guid...............0x0005ad00000281b3
>>                 port_guid...............0x0005ad00000281b3
>>                 m_key...................0x0000000000000000
>>                 subnet_prefix...........0x0000000000000000
>>                 base_lid................0x0
>>                 master_sm_base_lid......0x0
>>                 capability_mask.........0x0
>>                 diag_code...............0x0
>>                 m_key_lease_period......0x0
>>                 local_port_num..........0x3
>>                 link_width_enabled......0x3
>>                 link_width_supported....0x3
>>                 link_width_active.......0x2
>>                 link_speed_supported....0x1
>>                 port_state..............ACTIVE
>>                 state_info2.............0x52
>>                 m_key_protect_bits......0x0
>>                 lmc.....................0x0
>>                 link_speed..............0x11
>>                 mtu_smsl................0x40
>>                 vl_cap_init_type........0x40
>>                 vl_high_limit...........0x0
>>                 vl_arb_high_cap.........0x8
>>                 vl_arb_low_cap..........0x8
>>                 init_rep_mtu_cap........0x4
>>                 vl_stall_life...........0xF2
>>                 vl_enforce..............0x40
>>                 m_key_violations........0x0
>>                 p_key_violations........0x0
>>                 q_key_violations........0x0
>>                 guid_cap................0x0
>>                 client_reregister.......0x0
>>                 subnet_timeout..........0x0
>>                 resp_time_value.........0x0
>>                 error_threshold.........0x88
>> Mar 20 14:15:30 607794 [44606960] -> Capabilities Mask:
>> Mar 20 14:15:30 607914 [4780B960] -> __osm_sm_mad_ctrl_rcv_callback: ERR 
>> 3111: Error status = 0x1C00
>> Mar 20 14:15:30 607958 [4780B960] -> SMP dump:
>>                 base_ver................0x1
>>                 mgmt_class..............0x81
>>                 class_ver...............0x1
>>                 method..................0x81 (SubnGetResp)
>>                 D bit...................0x1
>>                 status..................0x1C00
>>                 hop_ptr.................0x0
>>                 hop_count...............0x4
>>                 trans_id................0x6f745
>>                 attr_id.................0x15 (PortInfo)
>>                 resv....................0x0
>>                 attr_mod................0x17
>>                 m_key...................0x0000000000000000
>>                 dr_slid.................0xFFFF
>>                 dr_dlid.................0xFFFF
>> 
>>                 Initial path: [0][1][14][1][6]
>>                 Return path:  [0][9][15][D][3]
>>                 Reserved:     [0][0][0][0][0][0][0]
>> 
>>                 00 00 00 00 00 00 00 00   00 00 00 00 00 00 00 00
>> 
>>                 00 00 00 00 00 00 00 00   00 00 00 00 03 03 03 02
>> 
>>                 14 52 00 11 40 40 00 08   08 04 F2 40 00 00 00 00
>> 
>>                 00 00 00 00 00 88 00 00   00 00 00 00 00 00 00 00
>> 
>> Mar 20 14:15:30 608014 [43204960] -> osm_pi_rcv_process_set: Received error 
>> status 0x1c for SetResp() during ACTIVE transition
>> Mar 20 14:15:30 608018 [43204960] -> PortInfo dump:
>>                 port number.............0x17
>>                 node_guid...............0x0005ad00000281b3
>>                 port_guid...............0x0005ad00000281b3
>>                 m_key...................0x0000000000000000
>>                 subnet_prefix...........0x0000000000000000
>>                 base_lid................0x0
>>                 master_sm_base_lid......0x0
>>                 capability_mask.........0x0
>>                 diag_code...............0x0
>>                 m_key_lease_period......0x0
>>                 local_port_num..........0x3
>>                 link_width_enabled......0x3
>>                 link_width_supported....0x3
>>                 link_width_active.......0x2
>>                 link_speed_supported....0x1
>>                 port_state..............ACTIVE
>>                 state_info2.............0x52
>>                 m_key_protect_bits......0x0
>>                 lmc.....................0x0
>>                 link_speed..............0x11
>>                 mtu_smsl................0x40
>>                 vl_cap_init_type........0x40
>>                 vl_high_limit...........0x0
>>                 vl_arb_high_cap.........0x8
>>                 vl_arb_low_cap..........0x8
>>                 init_rep_mtu_cap........0x4
>>                 vl_stall_life...........0xF2
>>                 vl_enforce..............0x40
>>                 m_key_violations........0x0
>>                 p_key_violations........0x0
>>                 q_key_violations........0x0
>>                 guid_cap................0x0
>>                 client_reregister.......0x0
>>                 subnet_timeout..........0x0
>>                 resp_time_value.........0x0
>>                 error_threshold.........0x88
>> Mar 20 14:15:30 608031 [43204960] -> Capabilities Mask:
>> Mar 20 14:15:30 613309 [41E02960] -> SUBNET UP
>> Mar 20 14:15:30 995108 [41401960] -> osm_ucast_mgr_process: Min Hop Tables 
>> configured on all switches
>> Mar 20 14:15:31 050102 [4780B960] -> __osm_sm_mad_ctrl_rcv_callback: ERR 
>> 3111: Error status = 0x1C00
>> Mar 20 14:15:31 050180 [4780B960] -> SMP dump:
>>                 base_ver................0x1
>>                 mgmt_class..............0x81
>>                 class_ver...............0x1
>>                 method..................0x81 (SubnGetResp)
>>                 D bit...................0x1
>>                 status..................0x1C00
>>                 hop_ptr.................0x0
>>                 hop_count...............0x4
>>                 trans_id................0x70486
>>                 attr_id.................0x15 (PortInfo)
>>                 resv....................0x0
>>                 attr_mod................0x18
>>                 m_key...................0x0000000000000000
>>                 dr_slid.................0xFFFF
>>                 dr_dlid.................0xFFFF
>> 
>>                 Initial path: [0][1][11][1][4]
>>                 Return path:  [0][9][18][D][1]
>>                 Reserved:     [0][0][0][0][0][0][0]
>> 
>>                 00 00 00 00 00 00 00 00   00 00 00 00 00 00 00 00
>> 
>>                 00 00 00 00 00 00 00 00   00 00 00 00 01 03 03 02
>> 
>>                 14 52 00 11 40 40 00 08   08 04 F2 40 00 00 00 00
>> 
>>                 00 00 00 00 00 88 00 00   00 00 00 00 00 00 00 00
>> 
>> Mar 20 14:15:31 050233 [45A08960] -> osm_pi_rcv_process_set: Received error 
>> status 0x1c for SetResp() during ACTIVE transition
>> Mar 20 14:15:31 050238 [45A08960] -> PortInfo dump:
>>                 port number.............0x18
>>                 node_guid...............0x0005ad00000281b3
>>                 port_guid...............0x0005ad00000281b3
>>                 m_key...................0x0000000000000000
>>                 subnet_prefix...........0x0000000000000000
>>                 base_lid................0x0
>>                 master_sm_base_lid......0x0
>>                 capability_mask.........0x0
>>                 diag_code...............0x0
>>                 m_key_lease_period......0x0
>>                 local_port_num..........0x1
>>                 link_width_enabled......0x3
>>                 link_width_supported....0x3
>>                 link_width_active.......0x2
>>                 link_speed_supported....0x1
>>                 port_state..............ACTIVE
>>                 state_info2.............0x52
>>                 m_key_protect_bits......0x0
>>                 lmc.....................0x0
>>                 link_speed..............0x11
>>                 mtu_smsl................0x40
>>                 vl_cap_init_type........0x40
>>                 vl_high_limit...........0x0
>>                 vl_arb_high_cap.........0x8
>>                 vl_arb_low_cap..........0x8
>>                 init_rep_mtu_cap........0x4
>>                 vl_stall_life...........0xF2
>>                 vl_enforce..............0x40
>>                 m_key_violations........0x0
>>                 p_key_violations........0x0
>>                 q_key_violations........0x0
>>                 guid_cap................0x0
>>                 client_reregister.......0x0
>>                 subnet_timeout..........0x0
>>                 resp_time_value.........0x0
>>                 error_threshold.........0x88
>> Mar 20 14:15:31 050251 [45A08960] -> Capabilities Mask:
>> Mar 20 14:15:31 055273 [42803960] -> SUBNET UP
>> Mar 20 14:15:31 106129 [41401960] -> __osm_trap_rcv_process_request: 
>> Received Generic Notice type:0x01 num:128 Producer:2 from LID:0x0152 
>> TID:0x000000000000000e
>> Mar 20 14:15:31 106193 [41401960] -> __osm_trap_rcv_process_request: ERR 
>> 3804: Received trap 15 times consecutively
>> Mar 20 14:17:18 456260 [43204960] -> __osm_trap_rcv_process_request: 
>> Received Generic Notice type:0x01 num:128 Producer:2 from LID:0x0148 
>> TID:0x0000000000000058
>> Mar 20 14:17:18 456512 [43204960] -> osm_report_notice: Reporting Generic 
>> Notice type:1 num:128 from LID:0x0148 
>> GID:0xfe80000000000000,0x0005ad00000281b3
>> Mar 20 14:17:18 456649 [41E02960] -> __osm_trap_rcv_process_request: 
>> Received Generic Notice type:0x01 num:128 Producer:2 from LID:0x001B 
>> TID:0x0000000000000070
>> Mar 20 14:17:18 456761 [41E02960] -> osm_report_notice: Reporting Generic 
>> Notice type:1 num:128 from LID:0x001B 
>> GID:0xfe80000000000000,0x0005ad00000281a7
>> Mar 20 14:17:18 769730 [45007960] -> osm_report_notice: Reporting Generic 
>> Notice type:3 num:65 from LID:0x0092 
>> GID:0xfe80000000000000,0x0005ad0000024bbb
>> Mar 20 14:17:18 769751 [45007960] -> Removed port with 
>> GUID:0x0005ad0000024e0b LID range [0xB3,0xB3] of node:saguaro-23-4 HCA-1
>> Mar 20 14:17:18 769773 [45007960] -> osm_report_notice: Reporting Generic 
>> Notice type:3 num:65 from LID:0x0092 
>> GID:0xfe80000000000000,0x0005ad0000024bbb
>> Mar 20 14:17:18 769780 [45007960] -> Removed port with 
>> GUID:0x0005ad000002510b LID range [0xB5,0xB5] of node:saguaro-23-6 HCA-1
>> Mar 20 14:17:18 769803 [45007960] -> osm_report_notice: Reporting Generic 
>> Notice type:3 num:65 from LID:0x0092 
>> GID:0xfe80000000000000,0x0005ad0000024bbb
>> Mar 20 14:17:18 769809 [45007960] -> Removed port with 
>> GUID:0x0005ad000002511b LID range [0xA6,0xA6] of node:saguaro-22-1 HCA-1
>> Mar 20 14:17:18 769832 [45007960] -> osm_report_notice: Reporting Generic 
>> Notice type:3 num:65 from LID:0x0092 
>> GID:0xfe80000000000000,0x0005ad0000024bbb
>> Mar 20 14:17:18 769838 [45007960] -> Removed port with 
>> GUID:0x0005ad0000025043 LID range [0xB4,0xB4] of node:saguaro-23-5 HCA-1
>> Mar 20 14:17:18 769858 [45007960] -> osm_report_notice: Reporting Generic 
>> Notice type:3 num:65 from LID:0x0092 
>> GID:0xfe80000000000000,0x0005ad0000024bbb
>> Mar 20 14:17:18 769865 [45007960] -> Removed port with 
>> GUID:0x0005ad0000024d47 LID range [0xB6,0xB6] of node:saguaro-23-7 HCA-1
>> Mar 20 14:17:18 769888 [45007960] -> osm_report_notice: Reporting Generic 
>> Notice type:3 num:65 from LID:0x0092 
>> GID:0xfe80000000000000,0x0005ad0000024bbb
>> Mar 20 14:17:18 769895 [45007960] -> Removed port with 
>> GUID:0x0005ad0000024d6b LID range [0xB8,0xB8] of node:saguaro-23-9 HCA-1
>> Mar 20 14:17:18 769927 [45007960] -> osm_report_notice: Reporting Generic 
>> Notice type:3 num:65 from LID:0x0092 
>> GID:0xfe80000000000000,0x0005ad0000024bbb
>> Mar 20 14:17:18 769932 [45007960] -> Removed port with 
>> GUID:0x0005ad0000024977 LID range [0xA9,0xA9] of node:saguaro-22-4 HCA-1
>> Mar 20 14:17:18 770075 [45007960] -> osm_report_notice: Reporting Generic 
>> Notice type:3 num:65 from LID:0x0092 
>> GID:0xfe80000000000000,0x0005ad0000024bbb
>> Mar 20 14:17:18 770081 [45007960] -> Removed port with 
>> GUID:0x0005ad0000027c84 LID range [0x152,0x152] of node:Topspin Switch TS120
>> Mar 20 14:17:18 770109 [45007960] -> osm_report_notice: Reporting Generic 
>> Notice type:3 num:65 from LID:0x0092 
>> GID:0xfe80000000000000,0x0005ad0000024bbb
>> Mar 20 14:17:18 770114 [45007960] -> Removed port with 
>> GUID:0x0005ad0000024d8b LID range [0xB7,0xB7] of node:saguaro-23-8 HCA-1
>> Mar 20 14:17:18 770130 [45007960] -> osm_report_notice: Reporting Generic 
>> Notice type:3 num:65 from LID:0x0092 
>> GID:0xfe80000000000000,0x0005ad0000024bbb
>> Mar 20 14:17:18 770135 [45007960] -> Removed port with 
>> GUID:0x0005ad000002498f LID range [0xA8,0xA8] of node:saguaro-22-3 HCA-1
>> Mar 20 14:17:18 770150 [45007960] -> osm_report_notice: Reporting Generic 
>> Notice type:3 num:65 from LID:0x0092 
>> GID:0xfe80000000000000,0x0005ad0000024bbb
>> Mar 20 14:17:18 770155 [45007960] -> Removed port with 
>> GUID:0x0005ad0000024c9b LID range [0xA7,0xA7] of node:saguaro-22-2 HCA-1
>> Mar 20 14:17:18 770171 [45007960] -> osm_report_notice: Reporting Generic 
>> Notice type:3 num:65 from LID:0x0092 
>> GID:0xfe80000000000000,0x0005ad0000024bbb
>> Mar 20 14:17:18 770176 [45007960] -> Removed port with 
>> GUID:0x0005ad0000024da7 LID range [0xB0,0xB0] of node:saguaro-23-1 HCA-1
>> Mar 20 14:17:18 770193 [45007960] -> osm_report_notice: Reporting Generic 
>> Notice type:3 num:65 from LID:0x0092 
>> GID:0xfe80000000000000,0x0005ad0000024bbb
>> Mar 20 14:17:18 770198 [45007960] -> Removed port with 
>> GUID:0x0005ad0000024cbb LID range [0xB2,0xB2] of node:saguaro-23-3 HCA-1
>> Mar 20 14:17:18 770216 [45007960] -> osm_report_notice: Reporting Generic 
>> Notice type:3 num:65 from LID:0x0092 
>> GID:0xfe80000000000000,0x0005ad0000024bbb
>> Mar 20 14:17:18 770221 [45007960] -> Removed port with 
>> GUID:0x0005ad00000249d3 LID range [0xB1,0xB1] of node:saguaro-23-2 HCA-1
>> Mar 20 14:17:18 770238 [45007960] -> osm_report_notice: Reporting Generic 
>> Notice type:3 num:65 from LID:0x0092 
>> GID:0xfe80000000000000,0x0005ad0000024bbb
>> Mar 20 14:17:18 770301 [45007960] -> Removed port with 
>> GUID:0x0005ad0000024feb LID range [0x153,0x153] of node:saguaro-22-5 HCA-1
>> Mar 20 14:17:18 770318 [45007960] -> osm_report_notice: Reporting Generic 
>> Notice type:3 num:65 from LID:0x0092 
>> GID:0xfe80000000000000,0x0005ad0000024bbb
>> Mar 20 14:17:18 770323 [45007960] -> Removed port with 
>> GUID:0x0005ad0000024afb LID range [0xA5,0xA5] of node:saguaro-22-0 HCA-1
>> Mar 20 14:17:18 803377 [45007960] -> osm_ucast_mgr_process: Min Hop Tables 
>> configured on all switches
>> Mar 20 14:17:18 855545 [44606960] -> SUBNET UP
>> Mar 20 14:17:19 249722 [43204960] -> osm_ucast_mgr_process: Min Hop Tables 
>> configured on all switches
>> Mar 20 14:17:19 300999 [45A08960] -> SUBNET UP
>> Mar 20 14:18:11 663850 [43C05960] -> __osm_trap_rcv_process_request: 
>> Received Generic Notice type:0x01 num:128 Producer:2 from LID:0x0148 
>> TID:0x0000000000000059
>> Mar 20 14:18:11 664195 [43C05960] -> osm_report_notice: Reporting Generic 
>> Notice type:1 num:128 from LID:0x0148 
>> GID:0xfe80000000000000,0x0005ad00000281b3
>> Mar 20 14:18:11 670836 [41E02960] -> __osm_trap_rcv_process_request: 
>> Received Generic Notice type:0x01 num:128 Producer:2 from LID:0x001B 
>> TID:0x0000000000000071
>> Mar 20 14:18:11 670964 [41401960] -> __osm_trap_rcv_process_request: 
>> Received Generic Notice type:0x01 num:128 Producer:2 from LID:0x0148 
>> TID:0x000000000000005a
>> Mar 20 14:18:11 671199 [41E02960] -> osm_report_notice: Reporting Generic 
>> Notice type:1 num:128 from LID:0x001B 
>> GID:0xfe80000000000000,0x0005ad00000281a7
>> Mar 20 14:18:11 672933 [41401960] -> osm_report_notice: Reporting Generic 
>> Notice type:1 num:128 from LID:0x0148 
>> GID:0xfe80000000000000,0x0005ad00000281b3
>> Mar 20 14:18:11 677654 [44606960] -> __osm_trap_rcv_process_request: 
>> Received Generic Notice type:0x01 num:128 Producer:2 from LID:0x001B 
>> TID:0x0000000000000072
>> Mar 20 14:18:11 677826 [44606960] -> osm_report_notice: Reporting Generic 
>> Notice type:1 num:128 from LID:0x001B 
>> GID:0xfe80000000000000,0x0005ad00000281a7
>> Mar 20 14:18:12 026661 [44606960] -> osm_report_notice: Reporting Generic 
>> Notice type:3 num:64 from LID:0x0092 
>> GID:0xfe80000000000000,0x0005ad0000024bbb
>> Mar 20 14:18:12 026675 [44606960] -> Discovered new port with 
>> GUID:0x0005ad0000027c84 LID range [0x152,0x152] of node:Topspin Switch TS120
>> Mar 20 14:18:12 026681 [44606960] -> osm_report_notice: Reporting Generic 
>> Notice type:3 num:64 from LID:0x0092 
>> GID:0xfe80000000000000,0x0005ad0000024bbb
>> Mar 20 14:18:12 026688 [44606960] -> Discovered new port with 
>> GUID:0x0005ad0000024da7 LID range [0xB0,0xB0] of node:saguaro-23-1 HCA-1
>> Mar 20 14:18:12 026693 [44606960] -> osm_report_notice: Reporting Generic 
>> Notice type:3 num:64 from LID:0x0092 
>> GID:0xfe80000000000000,0x0005ad0000024bbb
>> Mar 20 14:18:12 026700 [44606960] -> Discovered new port with 
>> GUID:0x0005ad00000249d3 LID range [0xB1,0xB1] of node:saguaro-23-2 HCA-1
>> Mar 20 14:18:12 026705 [44606960] -> osm_report_notice: Reporting Generic 
>> Notice type:3 num:64 from LID:0x0092 
>> GID:0xfe80000000000000,0x0005ad0000024bbb
>> Mar 20 14:18:12 026711 [44606960] -> Discovered new port with 
>> GUID:0x0005ad0000024cbb LID range [0xB2,0xB2] of node:saguaro-23-3 HCA-1
>> Mar 20 14:18:12 026716 [44606960] -> osm_report_notice: Reporting Generic 
>> Notice type:3 num:64 from LID:0x0092 
>> GID:0xfe80000000000000,0x0005ad0000024bbb
>> Mar 20 14:18:12 026723 [44606960] -> Discovered new port with 
>> GUID:0x0005ad0000024e0b LID range [0xB3,0xB3] of node:saguaro-23-4 HCA-1
>> Mar 20 14:18:12 026727 [44606960] -> osm_report_notice: Reporting Generic 
>> Notice type:3 num:64 from LID:0x0092 
>> GID:0xfe80000000000000,0x0005ad0000024bbb
>> Mar 20 14:18:12 026740 [44606960] -> Discovered new port with 
>> GUID:0x0005ad0000025043 LID range [0xB4,0xB4] of node:saguaro-23-5 HCA-1
>> Mar 20 14:18:12 026745 [44606960] -> osm_report_notice: Reporting Generic 
>> Notice type:3 num:64 from LID:0x0092 
>> GID:0xfe80000000000000,0x0005ad0000024bbb
>> Mar 20 14:18:12 026751 [44606960] -> Discovered new port with 
>> GUID:0x0005ad000002510b LID range [0xB5,0xB5] of node:saguaro-23-6 HCA-1
>> Mar 20 14:18:12 026758 [44606960] -> osm_report_notice: Reporting Generic 
>> Notice type:3 num:64 from LID:0x0092 
>> GID:0xfe80000000000000,0x0005ad0000024bbb
>> Mar 20 14:18:12 026764 [44606960] -> Discovered new port with 
>> GUID:0x0005ad0000024d47 LID range [0xB6,0xB6] of node:saguaro-23-7 HCA-1
>> Mar 20 14:18:12 026769 [44606960] -> osm_report_notice: Reporting Generic 
>> Notice type:3 num:64 from LID:0x0092 
>> GID:0xfe80000000000000,0x0005ad0000024bbb
>> Mar 20 14:18:12 026776 [44606960] -> Discovered new port with 
>> GUID:0x0005ad0000024d8b LID range [0xB7,0xB7] of node:saguaro-23-8 HCA-1
>> Mar 20 14:18:12 026781 [44606960] -> osm_report_notice: Reporting Generic 
>> Notice type:3 num:64 from LID:0x0092 
>> GID:0xfe80000000000000,0x0005ad0000024bbb
>> Mar 20 14:18:12 026787 [44606960] -> Discovered new port with 
>> GUID:0x0005ad0000024d6b LID range [0xB8,0xB8] of node:saguaro-23-9 HCA-1
>> Mar 20 14:18:12 026792 [44606960] -> osm_report_notice: Reporting Generic 
>> Notice type:3 num:64 from LID:0x0092 
>> GID:0xfe80000000000000,0x0005ad0000024bbb
>> Mar 20 14:18:12 026798 [44606960] -> Discovered new port with 
>> GUID:0x0005ad0000024afb LID range [0xA5,0xA5] of node:saguaro-22-0 HCA-1
>> Mar 20 14:18:12 026803 [44606960] -> osm_report_notice: Reporting Generic 
>> Notice type:3 num:64 from LID:0x0092 
>> GID:0xfe80000000000000,0x0005ad0000024bbb
>> Mar 20 14:18:12 026809 [44606960] -> Discovered new port with 
>> GUID:0x0005ad000002511b LID range [0xA6,0xA6] of node:saguaro-22-1 HCA-1
>> Mar 20 14:18:12 026814 [44606960] -> osm_report_notice: Reporting Generic 
>> Notice type:3 num:64 from LID:0x0092 
>> GID:0xfe80000000000000,0x0005ad0000024bbb
>> Mar 20 14:18:12 026821 [44606960] -> Discovered new port with 
>> GUID:0x0005ad0000024c9b LID range [0xA7,0xA7] of node:saguaro-22-2 HCA-1
>> Mar 20 14:18:12 026826 [44606960] -> osm_report_notice: Reporting Generic 
>> Notice type:3 num:64 from LID:0x0092 
>> GID:0xfe80000000000000,0x0005ad0000024bbb
>> Mar 20 14:18:12 026832 [44606960] -> Discovered new port with 
>> GUID:0x0005ad000002498f LID range [0xA8,0xA8] of node:saguaro-22-3 HCA-1
>> Mar 20 14:18:12 026869 [44606960] -> osm_report_notice: Reporting Generic 
>> Notice type:3 num:64 from LID:0x0092 
>> GID:0xfe80000000000000,0x0005ad0000024bbb
>> Mar 20 14:18:12 026877 [44606960] -> Discovered new port with 
>> GUID:0x0005ad0000024977 LID range [0xA9,0xA9] of node:saguaro-22-4 HCA-1
>> Mar 20 14:18:12 026882 [44606960] -> osm_report_notice: Reporting Generic 
>> Notice type:3 num:64 from LID:0x0092 
>> GID:0xfe80000000000000,0x0005ad0000024bbb
>> Mar 20 14:18:12 026888 [44606960] -> Discovered new port with 
>> GUID:0x0005ad0000024feb LID range [0x153,0x153] of node:saguaro-22-5 HCA-1
>> Mar 20 14:18:12 057534 [44606960] -> osm_ucast_mgr_process: Min Hop Tables 
>> configured on all switches
>> Mar 20 14:18:12 133316 [4780B960] -> __osm_sm_mad_ctrl_rcv_callback: ERR 
>> 3111: Error status = 0x1C00
>> Mar 20 14:18:12 133419 [4780B960] -> SMP dump:
>>                 base_ver................0x1
>>                 mgmt_class..............0x81
>>                 class_ver...............0x1
>>                 method..................0x81 (SubnGetResp)
>>                 D bit...................0x1
>>                 status..................0x1C00
>>                 hop_ptr.................0x0
>>                 hop_count...............0x4
>>                 trans_id................0x72d97
>>                 attr_id.................0x15 (PortInfo)
>>                 resv....................0x0
>>                 attr_mod................0x16
>>                 m_key...................0x0000000000000000
>>                 dr_slid.................0xFFFF
>>                 dr_dlid.................0xFFFF
>> 
>>                 Initial path: [0][1][14][3][6]
>>                 Return path:  [0][9][15][F][3]
>>                 Reserved:     [0][0][0][0][0][0][0]
>> 
>>                 00 00 00 00 00 00 00 00   00 00 00 00 00 00 00 00
>> 
>>                 00 00 00 00 00 00 00 00   00 00 00 00 03 03 03 02
>> 
>>                 14 52 00 11 40 40 00 08   08 04 F2 40 00 00 00 00
>> 
>>                 00 00 00 00 00 88 00 00   00 00 00 00 00 00 00 00
>> 
>> Mar 20 14:18:12 133466 [4780B960] -> __osm_sm_mad_ctrl_rcv_callback: ERR 
>> 3111: Error status = 0x1C00
>> Mar 20 14:18:12 133467 [43204960] -> osm_pi_rcv_process_set: Received error 
>> status 0x1c for SetResp() during ACTIVE transition
>> Mar 20 14:18:12 133478 [43204960] -> PortInfo dump:
>>                 port number.............0x16
>>                 node_guid...............0x0005ad00000281b3
>>                 port_guid...............0x0005ad00000281b3
>>                 m_key...................0x0000000000000000
>>                 subnet_prefix...........0x0000000000000000
>>                 base_lid................0x0
>>                 master_sm_base_lid......0x0
>>                 capability_mask.........0x0
>>                 diag_code...............0x0
>>                 m_key_lease_period......0x0
>>                 local_port_num..........0x3
>>                 link_width_enabled......0x3
>>                 link_width_supported....0x3
>>                 link_width_active.......0x2
>>                 link_speed_supported....0x1
>>                 port_state..............ACTIVE
>>                 state_info2.............0x52
>>                 m_key_protect_bits......0x0
>>                 lmc.....................0x0
>>                 link_speed..............0x11
>>                 mtu_smsl................0x40
>>                 vl_cap_init_type........0x40
>>                 vl_high_limit...........0x0
>>                 vl_arb_high_cap.........0x8
>>                 vl_arb_low_cap..........0x8
>>                 init_rep_mtu_cap........0x4
>>                 vl_stall_life...........0xF2
>>                 vl_enforce..............0x40
>>                 m_key_violations........0x0
>>                 p_key_violations........0x0
>>                 q_key_violations........0x0
>>                 guid_cap................0x0
>>                 client_reregister.......0x0
>>                 subnet_timeout..........0x0
>>                 resp_time_value.........0x0
>>                 error_threshold.........0x88
>> Mar 20 14:18:12 133490 [4780B960] -> SMP dump:
>>                 base_ver................0x1
>>                 mgmt_class..............0x81
>>                 class_ver...............0x1
>>                 method..................0x81 (SubnGetResp)
>>                 D bit...................0x1
>>                 status..................0x1C00
>>                 hop_ptr.................0x0
>>                 hop_count...............0x4
>>                 trans_id................0x72d98
>>                 attr_id.................0x15 (PortInfo)
>>                 resv....................0x0
>>                 attr_mod................0x17
>>                 m_key...................0x0000000000000000
>>                 dr_slid.................0xFFFF
>>                 dr_dlid.................0xFFFF
>> 
>>                 Initial path: [0][1][14][3][6]
>>                 Return path:  [0][9][15][F][3]
>>                 Reserved:     [0][0][0][0][0][0][0]
>> 
>>                 00 00 00 00 00 00 00 00   00 00 00 00 00 00 00 00
>> 
>>                 00 00 00 00 00 00 00 00   00 00 00 00 03 03 03 02
>> 
>>                 14 52 00 11 40 40 00 08   08 04 F2 40 00 00 00 00
>> 
>>                 00 00 00 00 00 88 00 00   00 00 00 00 00 00 00 00
>> 
>> Mar 20 14:18:12 133493 [43204960] -> Capabilities Mask:
>> Mar 20 14:18:12 133566 [45A08960] -> osm_pi_rcv_process_set: Received error 
>> status 0x1c for SetResp() during ACTIVE transition
>> Mar 20 14:18:12 133595 [45A08960] -> PortInfo dump:
>>                 port number.............0x17
>>                 node_guid...............0x0005ad00000281b3
>>                 port_guid...............0x0005ad00000281b3
>>                 m_key...................0x0000000000000000
>>                 subnet_prefix...........0x0000000000000000
>>                 base_lid................0x0
>>                 master_sm_base_lid......0x0
>>                 capability_mask.........0x0
>>                 diag_code...............0x0
>>                 m_key_lease_period......0x0
>>                 local_port_num..........0x3
>>                 link_width_enabled......0x3
>>                 link_width_supported....0x3
>>                 link_width_active.......0x2
>>                 link_speed_supported....0x1
>>                 port_state..............ACTIVE
>>                 state_info2.............0x52
>>                 m_key_protect_bits......0x0
>>                 lmc.....................0x0
>>                 link_speed..............0x11
>>                 mtu_smsl................0x40
>>                 vl_cap_init_type........0x40
>>                 vl_high_limit...........0x0
>>                 vl_arb_high_cap.........0x8
>>                 vl_arb_low_cap..........0x8
>>                 init_rep_mtu_cap........0x4
>>                 vl_stall_life...........0xF2
>>                 vl_enforce..............0x40
>>                 m_key_violations........0x0
>>                 p_key_violations........0x0
>>                 q_key_violations........0x0
>>                 guid_cap................0x0
>>                 client_reregister.......0x0
>>                 subnet_timeout..........0x0
>>                 resp_time_value.........0x0
>>                 error_threshold.........0x88
>> Mar 20 14:18:12 133614 [45A08960] -> Capabilities Mask:
>> Mar 20 14:18:12 133583 [4780B960] -> __osm_sm_mad_ctrl_rcv_callback: ERR 
>> 3111: Error status = 0x1C00
>> Mar 20 14:18:12 133671 [4780B960] -> SMP dump:
>>                 base_ver................0x1
>>                 mgmt_class..............0x81
>>                 class_ver...............0x1
>>                 method..................0x81 (SubnGetResp)
>>                 D bit...................0x1
>>                 status..................0x1C00
>>                 hop_ptr.................0x0
>>                 hop_count...............0x4
>>                 trans_id................0x72d99
>>                 attr_id.................0x15 (PortInfo)
>>                 resv....................0x0
>>                 attr_mod................0x18
>>                 m_key...................0x0000000000000000
>>                 dr_slid.................0xFFFF
>>                 dr_dlid.................0xFFFF
>> 
>>                 Initial path: [0][1][14][3][6]
>>                 Return path:  [0][9][15][F][3]
>>                 Reserved:     [0][0][0][0][0][0][0]
>> 
>>                 00 00 00 00 00 00 00 00   00 00 00 00 00 00 00 00
>> 
>>                 00 00 00 00 00 00 00 00   00 00 00 00 03 03 03 02
>> 
>>                 14 52 00 11 40 40 00 08   08 04 F2 40 00 00 00 00
>> 
>>                 00 00 00 00 00 88 00 00   00 00 00 00 00 00 00 00
>> 
>> Mar 20 14:18:12 133760 [43C05960] -> osm_pi_rcv_process_set: Received error 
>> status 0x1c for SetResp() during ACTIVE transition
>> Mar 20 14:18:12 133788 [43C05960] -> PortInfo dump:
>>                 port number.............0x18
>>                 node_guid...............0x0005ad00000281b3
>>                 port_guid...............0x0005ad00000281b3
>>                 m_key...................0x0000000000000000
>>                 subnet_prefix...........0x0000000000000000
>>                 base_lid................0x0
>>                 master_sm_base_lid......0x0
>>                 capability_mask.........0x0
>>                 diag_code...............0x0
>>                 m_key_lease_period......0x0
>>                 local_port_num..........0x3
>>                 link_width_enabled......0x3
>>                 link_width_supported....0x3
>>                 link_width_active.......0x2
>>                 link_speed_supported....0x1
>>                 port_state..............ACTIVE
>>                 state_info2.............0x52
>>                 m_key_protect_bits......0x0
>>                 lmc.....................0x0
>>                 link_speed..............0x11
>>                 mtu_smsl................0x40
>>                 vl_cap_init_type........0x40
>>                 vl_high_limit...........0x0
>>                 vl_arb_high_cap.........0x8
>>                 vl_arb_low_cap..........0x8
>>                 init_rep_mtu_cap........0x4
>>                 vl_stall_life...........0xF2
>>                 vl_enforce..............0x40
>>                 m_key_violations........0x0
>>                 p_key_violations........0x0
>>                 q_key_violations........0x0
>>                 guid_cap................0x0
>>                 client_reregister.......0x0
>>                 subnet_timeout..........0x0
>>                 resp_time_value.........0x0
>>                 error_threshold.........0x88
>> Mar 20 14:18:12 133807 [43C05960] -> Capabilities Mask:
>> Mar 20 14:18:12 139330 [41401960] -> SUBNET UP
>> Mar 20 14:18:12 496444 [45A08960] -> osm_ucast_mgr_process: Min Hop Tables 
>> configured on all switches
>> Mar 20 14:18:12 558965 [41401960] -> SUBNET UP
>> Mar 20 14:18:27 748551 [43C05960] -> __osm_trap_rcv_process_request: 
>> Received Generic Notice type:0x01 num:128 Producer:2 from LID:0x0152 
>> TID:0x0000000000000000
>> Mar 20 14:18:27 748795 [43C05960] -> osm_report_notice: Reporting Generic 
>> Notice type:1 num:128 from LID:0x0152 
>> GID:0xfe80000000000000,0x0005ad0000027c84
>> Mar 20 14:18:27 888669 [44606960] -> __osm_trap_rcv_process_request: 
>> Received Generic Notice type:0x01 num:128 Producer:2 from LID:0x0152 
>> TID:0x0000000000000001
>> Mar 20 14:18:27 888902 [44606960] -> osm_report_notice: Reporting Generic 
>> Notice type:1 num:128 from LID:0x0152 
>> GID:0xfe80000000000000,0x0005ad0000027c84
>> Mar 20 14:18:27 910605 [44606960] -> __osm_trap_rcv_process_request: 
>> Received Generic Notice type:0x01 num:128 Producer:2 from LID:0x0152 
>> TID:0x0000000000000002
>> Mar 20 14:18:27 910710 [44606960] -> osm_report_notice: Reporting Generic 
>> Notice type:1 num:128 from LID:0x0152 
>> GID:0xfe80000000000000,0x0005ad0000027c84
>> Mar 20 14:18:27 911951 [41E02960] -> __osm_trap_rcv_process_request: 
>> Received Generic Notice type:0x01 num:128 Producer:2 from LID:0x0152 
>> TID:0x0000000000000003
>> Mar 20 14:18:27 912119 [41E02960] -> osm_report_notice: Reporting Generic 
>> Notice type:1 num:128 from LID:0x0152 
>> GID:0xfe80000000000000,0x0005ad0000027c84
>> Mar 20 14:18:28 012957 [45A08960] -> __osm_trap_rcv_process_request: 
>> Received Generic Notice type:0x01 num:128 Producer:2 from LID:0x0152 
>> TID:0x0000000000000004
>> Mar 20 14:18:28 013058 [45A08960] -> osm_report_notice: Reporting Generic 
>> Notice type:1 num:128 from LID:0x0152 
>> GID:0xfe80000000000000,0x0005ad0000027c84
>> Mar 20 14:18:28 075266 [43C05960] -> __osm_trap_rcv_process_request: 
>> Received Generic Notice type:0x01 num:128 Producer:2 from LID:0x0152 
>> TID:0x0000000000000005
>> Mar 20 14:18:28 075397 [43C05960] -> osm_report_notice: Reporting Generic 
>> Notice type:1 num:128 from LID:0x0152 
>> GID:0xfe80000000000000,0x0005ad0000027c84
>> Mar 20 14:18:28 259000 [41E02960] -> __osm_trap_rcv_process_request: 
>> Received Generic Notice type:0x01 num:128 Producer:2 from LID:0x0152 
>> TID:0x0000000000000006
>> Mar 20 14:18:28 259121 [41E02960] -> osm_report_notice: Reporting Generic 
>> Notice type:1 num:128 from LID:0x0152 
>> GID:0xfe80000000000000,0x0005ad0000027c84
>> Mar 20 14:18:28 308865 [42803960] -> __osm_trap_rcv_process_request: 
>> Received Generic Notice type:0x01 num:128 Producer:2 from LID:0x0152 
>> TID:0x0000000000000007
>> Mar 20 14:18:28 309000 [42803960] -> osm_report_notice: Reporting Generic 
>> Notice type:1 num:128 from LID:0x0152 
>> GID:0xfe80000000000000,0x0005ad0000027c84
>> Mar 20 14:18:28 330606 [45007960] -> __osm_trap_rcv_process_request: 
>> Received Generic Notice type:0x01 num:128 Producer:2 from LID:0x0152 
>> TID:0x0000000000000008
>> Mar 20 14:18:28 330714 [45007960] -> osm_report_notice: Reporting Generic 
>> Notice type:1 num:128 from LID:0x0152 
>> GID:0xfe80000000000000,0x0005ad0000027c84
>> Mar 20 14:18:28 444107 [45A08960] -> __osm_trap_rcv_process_request: 
>> Received Generic Notice type:0x01 num:128 Producer:2 from LID:0x0152 
>> TID:0x0000000000000009
>> Mar 20 14:18:28 444191 [45A08960] -> osm_report_notice: Reporting Generic 
>> Notice type:1 num:128 from LID:0x0152 
>> GID:0xfe80000000000000,0x0005ad0000027c84
>> Mar 20 14:18:28 466156 [44606960] -> __osm_trap_rcv_process_request: 
>> Received Generic Notice type:0x01 num:128 Producer:2 from LID:0x0152 
>> TID:0x000000000000000a
>> Mar 20 14:18:28 466234 [44606960] -> osm_report_notice: Reporting Generic 
>> Notice type:1 num:128 from LID:0x0152 
>> GID:0xfe80000000000000,0x0005ad0000027c84
>> Mar 20 14:18:28 478021 [43C05960] -> __osm_trap_rcv_process_request: 
>> Received Generic Notice type:0x01 num:128 Producer:2 from LID:0x0152 
>> TID:0x000000000000000b
>> Mar 20 14:18:28 478070 [43C05960] -> __osm_trap_rcv_process_request: ERR 
>> 3804: Received trap 11 times consecutively
>> Mar 20 14:18:28 489091 [43204960] -> osm_report_notice: Reporting Generic 
>> Notice type:3 num:65 from LID:0x0092 
>> GID:0xfe80000000000000,0x0005ad0000024bbb
>> Mar 20 14:18:28 489106 [43204960] -> Removed port with 
>> GUID:0x0005ad00000249d3 LID range [0xB1,0xB1] of node:saguaro-23-2 HCA-1
>> Mar 20 14:18:28 521430 [42803960] -> __osm_trap_rcv_process_request: 
>> Received Generic Notice type:0x01 num:128 Producer:2 from LID:0x0152 
>> TID:0x000000000000000c
>> Mar 20 14:18:28 521499 [42803960] -> __osm_trap_rcv_process_request: ERR 
>> 3804: Received trap 12 times consecutively
>> Mar 20 14:18:28 523658 [43204960] -> osm_ucast_mgr_process: Min Hop Tables 
>> configured on all switches
>> Mar 20 14:18:28 580295 [43204960] -> SUBNET UP
>> Mar 20 14:18:28 611805 [43204960] -> __osm_trap_rcv_process_request: 
>> Received Generic Notice type:0x01 num:128 Producer:2 from LID:0x0152 
>> TID:0x000000000000000d
>> Mar 20 14:18:28 611893 [43204960] -> __osm_trap_rcv_process_request: ERR 
>> 3804: Received trap 13 times consecutively
>> Mar 20 14:18:28 661292 [45A08960] -> __osm_trap_rcv_process_request: 
>> Received Generic Notice type:0x01 num:128 Producer:2 from LID:0x0152 
>> TID:0x000000000000000e
>> Mar 20 14:18:28 661351 [45A08960] -> __osm_trap_rcv_process_request: ERR 
>> 3804: Received trap 14 times consecutively
>> Mar 20 14:18:28 871670 [44606960] -> __osm_trap_rcv_process_request: 
>> Received Generic Notice type:0x01 num:128 Producer:2 from LID:0x0152 
>> TID:0x000000000000000f
>> Mar 20 14:18:28 871732 [44606960] -> __osm_trap_rcv_process_request: ERR 
>> 3804: Received trap 15 times consecutively
>> Mar 20 14:18:28 934440 [43204960] -> __osm_trap_rcv_process_request: 
>> Received Generic Notice type:0x01 num:128 Producer:2 from LID:0x0152 
>> TID:0x0000000000000010
>> Mar 20 14:18:28 934505 [43204960] -> __osm_trap_rcv_process_request: ERR 
>> 3804: Received trap 16 times consecutively
>> Mar 20 14:18:28 941281 [45A08960] -> osm_report_notice: Reporting Generic 
>> Notice type:3 num:65 from LID:0x0092 
>> GID:0xfe80000000000000,0x0005ad0000024bbb
>> Mar 20 14:18:28 941303 [45A08960] -> Removed port with 
>> GUID:0x0005ad000002510b LID range [0xB5,0xB5] of node:saguaro-23-6 HCA-1
>> Mar 20 14:18:28 941329 [45A08960] -> osm_report_notice: Reporting Generic 
>> Notice type:3 num:65 from LID:0x0092 
>> GID:0xfe80000000000000,0x0005ad0000024bbb
>> Mar 20 14:18:28 941336 [45A08960] -> Removed port with 
>> GUID:0x0005ad0000025043 LID range [0xB4,0xB4] of node:saguaro-23-5 HCA-1
>> Mar 20 14:18:28 941356 [45A08960] -> osm_report_notice: Reporting Generic 
>> Notice type:3 num:65 from LID:0x0092 
>> GID:0xfe80000000000000,0x0005ad0000024bbb
>> Mar 20 14:18:28 941363 [45A08960] -> Removed port with 
>> GUID:0x0005ad0000024d47 LID range [0xB6,0xB6] of node:saguaro-23-7 HCA-1
>> Mar 20 14:18:28 941388 [45A08960] -> osm_report_notice: Reporting Generic 
>> Notice type:3 num:65 from LID:0x0092 
>> GID:0xfe80000000000000,0x0005ad0000024bbb
>> Mar 20 14:18:28 941395 [45A08960] -> Removed port with 
>> GUID:0x0005ad0000024d8b LID range [0xB7,0xB7] of node:saguaro-23-8 HCA-1
>> Mar 20 14:18:28 941420 [45A08960] -> osm_report_notice: Reporting Generic 
>> Notice type:3 num:65 from LID:0x0092 
>> GID:0xfe80000000000000,0x0005ad0000024bbb
>> Mar 20 14:18:28 941426 [45A08960] -> Removed port with 
>> GUID:0x0005ad0000024cbb LID range [0xB2,0xB2] of node:saguaro-23-3 HCA-1
>> Mar 20 14:18:28 945507 [45A08960] -> osm_report_notice: Reporting Generic 
>> Notice type:3 num:64 from LID:0x0092 
>> GID:0xfe80000000000000,0x0005ad0000024bbb
>> Mar 20 14:18:28 945515 [45A08960] -> Discovered new port with 
>> GUID:0x0005ad00000249d3 LID range [0xB1,0xB1] of node:saguaro-23-2 HCA-1
>> Mar 20 14:18:28 956576 [44606960] -> __osm_trap_rcv_process_request: 
>> Received Generic Notice type:0x01 num:128 Producer:2 from LID:0x0152 
>> TID:0x0000000000000011
>> Mar 20 14:18:28 956665 [44606960] -> __osm_trap_rcv_process_request: ERR 
>> 3804: Received trap 17 times consecutively
>> Mar 20 14:18:28 976211 [45A08960] -> osm_ucast_mgr_process: Min Hop Tables 
>> configured on all switches
>> Mar 20 14:18:29 033513 [42803960] -> SUBNET UP
>> Mar 20 14:18:29 071283 [41401960] -> __osm_trap_rcv_process_request: 
>> Received Generic Notice type:0x01 num:128 Producer:2 from LID:0x0152 
>> TID:0x0000000000000012
>> Mar 20 14:18:29 071345 [41401960] -> __osm_trap_rcv_process_request: ERR 
>> 3804: Received trap 18 times consecutively
>> Mar 20 14:18:29 352103 [44606960] -> __osm_trap_rcv_process_request: 
>> Received Generic Notice type:0x01 num:128 Producer:2 from LID:0x0152 
>> TID:0x0000000000000013
>> Mar 20 14:18:29 352155 [44606960] -> __osm_trap_rcv_process_request: ERR 
>> 3804: Received trap 19 times consecutively
>> Mar 20 14:18:29 376386 [41E02960] -> __osm_trap_rcv_process_request: 
>> Received Generic Notice type:0x01 num:128 Producer:2 from LID:0x0152 
>> TID:0x0000000000000014
>> Mar 20 14:18:29 376461 [41E02960] -> __osm_trap_rcv_process_request: ERR 
>> 3804: Received trap 20 times consecutively
>> Mar 20 14:18:29 420228 [43204960] -> __osm_trap_rcv_process_request: 
>> Received Generic Notice type:0x01 num:128 Producer:2 from LID:0x0152 
>> TID:0x0000000000000015
>> Mar 20 14:18:29 420282 [43204960] -> __osm_trap_rcv_process_request: ERR 
>> 3804: Received trap 21 times consecutively
>> Mar 20 14:18:29 421294 [43C05960] -> __osm_trap_rcv_process_request: 
>> Received Generic Notice type:0x01 num:128 Producer:2 from LID:0x0152 
>> TID:0x0000000000000016
>> Mar 20 14:18:29 421345 [43C05960] -> __osm_trap_rcv_process_request: ERR 
>> 3804: Received trap 22 times consecutively
>> Mar 20 14:18:29 461135 [45A08960] -> __osm_trap_rcv_process_request: 
>> Received Generic Notice type:0x01 num:128 Producer:2 from LID:0x0152 
>> TID:0x0000000000000017
>> Mar 20 14:18:29 461179 [45A08960] -> __osm_trap_rcv_process_request: ERR 
>> 3804: Received trap 23 times consecutively
>> Mar 20 14:18:29 633008 [45007960] -> __osm_trap_rcv_process_request: 
>> Received Generic Notice type:0x01 num:128 Producer:2 from LID:0x0152 
>> TID:0x0000000000000018
>> Mar 20 14:18:29 633050 [42803960] -> __osm_trap_rcv_process_request: 
>> Received Generic Notice type:0x01 num:128 Producer:2 from LID:0x0148 
>> TID:0x000000000000005b
>> Mar 20 14:18:29 633062 [45007960] -> __osm_trap_rcv_process_request: ERR 
>> 3804: Received trap 24 times consecutively
>> Mar 20 14:18:29 633350 [42803960] -> osm_report_notice: Reporting Generic 
>> Notice type:1 num:128 from LID:0x0148 
>> GID:0xfe80000000000000,0x0005ad00000281b3
>> Mar 20 14:18:29 733039 [45A08960] -> __osm_trap_rcv_process_request: 
>> Received Generic Notice type:0x01 num:128 Producer:2 from LID:0x0148 
>> TID:0x000000000000005c
>> Mar 20 14:18:29 733238 [45A08960] -> osm_report_notice: Reporting Generic 
>> Notice type:1 num:128 from LID:0x0148 
>> GID:0xfe80000000000000,0x0005ad00000281b3
>> Mar 20 14:18:29 947440 [44606960] -> osm_report_notice: Reporting Generic 
>> Notice type:3 num:64 from LID:0x0092 
>> GID:0xfe80000000000000,0x0005ad0000024bbb
>> Mar 20 14:18:29 947452 [44606960] -> Discovered new port with 
>> GUID:0x0005ad0000024cbb LID range [0xB2,0xB2] of node:saguaro-23-3 HCA-1
>> Mar 20 14:18:29 947457 [44606960] -> osm_report_notice: Reporting Generic 
>> Notice type:3 num:64 from LID:0x0092 
>> GID:0xfe80000000000000,0x0005ad0000024bbb
>> Mar 20 14:18:29 947462 [44606960] -> Discovered new port with 
>> GUID:0x0005ad0000025043 LID range [0xB4,0xB4] of node:saguaro-23-5 HCA-1
>> Mar 20 14:18:29 947465 [44606960] -> osm_report_notice: Reporting Generic 
>> Notice type:3 num:64 from LID:0x0092 
>> GID:0xfe80000000000000,0x0005ad0000024bbb
>> Mar 20 14:18:29 947470 [44606960] -> Discovered new port with 
>> GUID:0x0005ad000002510b LID range [0xB5,0xB5] of node:saguaro-23-6 HCA-1
>> Mar 20 14:18:29 947474 [44606960] -> osm_report_notice: Reporting Generic 
>> Notice type:3 num:64 from LID:0x0092 
>> GID:0xfe80000000000000,0x0005ad0000024bbb
>> Mar 20 14:18:29 947479 [44606960] -> Discovered new port with 
>> GUID:0x0005ad0000024d47 LID range [0xB6,0xB6] of node:saguaro-23-7 HCA-1
>> Mar 20 14:18:29 947482 [44606960] -> osm_report_notice: Reporting Generic 
>> Notice type:3 num:64 from LID:0x0092 
>> GID:0xfe80000000000000,0x0005ad0000024bbb
>> Mar 20 14:18:29 947487 [44606960] -> Discovered new port with 
>> GUID:0x0005ad0000024d8b LID range [0xB7,0xB7] of node:saguaro-23-8 HCA-1
>> Mar 20 14:18:29 978182 [44606960] -> osm_ucast_mgr_process: Min Hop Tables 
>> configured on all switches
>> Mar 20 14:18:30 027730 [4780B960] -> __osm_sm_mad_ctrl_rcv_callback: ERR 
>> 3111: Error status = 0x1C00
>> Mar 20 14:18:30 027819 [4780B960] -> SMP dump:
>>                 base_ver................0x1
>>                 mgmt_class..............0x81
>>                 class_ver...............0x1
>>                 method..................0x81 (SubnGetResp)
>>                 D bit...................0x1
>>                 status..................0x1C00
>>                 hop_ptr.................0x0
>>                 hop_count...............0x4
>>                 trans_id................0x762b8
>>                 attr_id.................0x15 (PortInfo)
>>                 resv....................0x0
>>                 attr_mod................0x16
>>                 m_key...................0x0000000000000000
>>                 dr_slid.................0xFFFF
>>                 dr_dlid.................0xFFFF
>> 
>>                 Initial path: [0][1][11][1][4]
>>                 Return path:  [0][9][18][D][1]
>>                 Reserved:     [0][0][0][0][0][0][0]
>> 
>>                 00 00 00 00 00 00 00 00   00 00 00 00 00 00 00 00
>> 
>>                 00 00 00 00 00 00 00 00   00 00 00 00 01 03 03 02
>> 
>>                 14 52 00 11 40 40 00 08   08 04 F2 40 00 00 00 00
>> 
>>                 00 00 00 00 00 88 00 00   00 00 00 00 00 00 00 00
>> 
>> Mar 20 14:18:30 027897 [41401960] -> osm_pi_rcv_process_set: Received error 
>> status 0x1c for SetResp() during ACTIVE transition
>> Mar 20 14:18:30 027901 [41401960] -> PortInfo dump:
>>                 port number.............0x16
>>                 node_guid...............0x0005ad00000281b3
>>                 port_guid...............0x0005ad00000281b3
>>                 m_key...................0x0000000000000000
>>                 subnet_prefix...........0x0000000000000000
>>                 base_lid................0x0
>>                 master_sm_base_lid......0x0
>>                 capability_mask.........0x0
>>                 diag_code...............0x0
>>                 m_key_lease_period......0x0
>>                 local_port_num..........0x1
>>                 link_width_enabled......0x3
>>                 link_width_supported....0x3
>>                 link_width_active.......0x2
>>                 link_speed_supported....0x1
>>                 port_state..............ACTIVE
>>                 state_info2.............0x52
>>                 m_key_protect_bits......0x0
>>                 lmc.....................0x0
>>                 link_speed..............0x11
>>                 mtu_smsl................0x40
>>                 vl_cap_init_type........0x40
>>                 vl_high_limit...........0x0
>>                 vl_arb_high_cap.........0x8
>>                 vl_arb_low_cap..........0x8
>>                 init_rep_mtu_cap........0x4
>>                 vl_stall_life...........0xF2
>>                 vl_enforce..............0x40
>>                 m_key_violations........0x0
>>                 p_key_violations........0x0
>>                 q_key_violations........0x0
>>                 guid_cap................0x0
>>                 client_reregister.......0x0
>>                 subnet_timeout..........0x0
>>                 resp_time_value.........0x0
>>                 error_threshold.........0x88
>> Mar 20 14:18:30 027914 [41401960] -> Capabilities Mask:
>> Mar 20 14:18:30 027993 [4780B960] -> __osm_sm_mad_ctrl_rcv_callback: ERR 
>> 3111: Error status = 0x1C00
>> Mar 20 14:18:30 028043 [4780B960] -> SMP dump:
>>                 base_ver................0x1
>>                 mgmt_class..............0x81
>>                 class_ver...............0x1
>>                 method..................0x81 (SubnGetResp)
>>                 D bit...................0x1
>>                 status..................0x1C00
>>                 hop_ptr.................0x0
>>                 hop_count...............0x4
>>                 trans_id................0x762b9
>>                 attr_id.................0x15 (PortInfo)
>>                 resv....................0x0
>>                 attr_mod................0x17
>>                 m_key...................0x0000000000000000
>>                 dr_slid.................0xFFFF
>>                 dr_dlid.................0xFFFF
>> 
>>                 Initial path: [0][1][11][1][4]
>>                 Return path:  [0][9][18][D][1]
>>                 Reserved:     [0][0][0][0][0][0][0]
>> 
>>                 00 00 00 00 00 00 00 00   00 00 00 00 00 00 00 00
>> 
>>                 00 00 00 00 00 00 00 00   00 00 00 00 01 03 03 02
>> 
>>                 14 52 00 11 40 40 00 08   08 04 F2 40 00 00 00 00
>> 
>>                 00 00 00 00 00 88 00 00   00 00 00 00 00 00 00 00
>> 
>> Mar 20 14:18:30 028098 [45A08960] -> osm_pi_rcv_process_set: Received error 
>> status 0x1c for SetResp() during ACTIVE transition
>> Mar 20 14:18:30 028109 [45A08960] -> PortInfo dump:
>>                 port number.............0x17
>>                 node_guid...............0x0005ad00000281b3
>>                 port_guid...............0x0005ad00000281b3
>>                 m_key...................0x0000000000000000
>>                 subnet_prefix...........0x0000000000000000
>>                 base_lid................0x0
>>                 master_sm_base_lid......0x0
>>                 capability_mask.........0x0
>>                 diag_code...............0x0
>>                 m_key_lease_period......0x0
>>                 local_port_num..........0x1
>>                 link_width_enabled......0x3
>>                 link_width_supported....0x3
>>                 link_width_active.......0x2
>>                 link_speed_supported....0x1
>>                 port_state..............ACTIVE
>>                 state_info2.............0x52
>>                 m_key_protect_bits......0x0
>>                 lmc.....................0x0
>>                 link_speed..............0x11
>>                 mtu_smsl................0x40
>>                 vl_cap_init_type........0x40
>>                 vl_high_limit...........0x0
>>                 vl_arb_high_cap.........0x8
>>                 vl_arb_low_cap..........0x8
>>                 init_rep_mtu_cap........0x4
>>                 vl_stall_life...........0xF2
>>                 vl_enforce..............0x40
>>                 m_key_violations........0x0
>>                 p_key_violations........0x0
>>                 q_key_violations........0x0
>>                 guid_cap................0x0
>>                 client_reregister.......0x0
>>                 subnet_timeout..........0x0
>>                 resp_time_value.........0x0
>>                 error_threshold.........0x88
>> Mar 20 14:18:30 028124 [45A08960] -> Capabilities Mask:
>> Mar 20 14:18:30 033824 [44606960] -> SUBNET UP
>> Mar 20 14:18:30 418497 [43C05960] -> osm_report_notice: Reporting Generic 
>> Notice type:3 num:65 from LID:0x0092 
>> GID:0xfe80000000000000,0x0005ad0000024bbb
>> Mar 20 14:18:30 418522 [43C05960] -> Removed port with 
>> GUID:0x0005ad0000024feb LID range [0x153,0x153] of node:saguaro-22-5 HCA-1
>> Mar 20 14:18:30 453167 [43C05960] -> osm_ucast_mgr_process: Min Hop Tables 
>> configured on all switches
>> Mar 20 14:18:30 494719 [41E02960] -> __osm_trap_rcv_process_request: 
>> Received Generic Notice type:0x01 num:128 Producer:2 from LID:0x0148 
>> TID:0x000000000000005d
>> Mar 20 14:18:30 494877 [41E02960] -> osm_report_notice: Reporting Generic 
>> Notice type:1 num:128 from LID:0x0148 
>> GID:0xfe80000000000000,0x0005ad00000281b3
>> Mar 20 14:18:30 662496 [44606960] -> __osm_trap_rcv_process_request: 
>> Received Generic Notice type:0x01 num:128 Producer:2 from LID:0x001B 
>> TID:0x0000000000000073
>> Mar 20 14:18:30 662564 [44606960] -> osm_report_notice: Reporting Generic 
>> Notice type:1 num:128 from LID:0x001B 
>> GID:0xfe80000000000000,0x0005ad00000281a7
>> Mar 20 14:18:30 662645 [43C05960] -> __osm_trap_rcv_process_request: 
>> Received Generic Notice type:0x01 num:128 Producer:2 from LID:0x0148 
>> TID:0x000000000000005e
>> Mar 20 14:18:30 662759 [43C05960] -> osm_report_notice: Reporting Generic 
>> Notice type:1 num:128 from LID:0x0148 
>> GID:0xfe80000000000000,0x0005ad00000281b3
>> Mar 20 14:18:30 707085 [42803960] -> __osm_trap_rcv_process_request: 
>> Received Generic Notice type:0x01 num:128 Producer:2 from LID:0x0148 
>> TID:0x000000000000005f
>> Mar 20 14:18:30 707179 [42803960] -> osm_report_notice: Reporting Generic 
>> Notice type:1 num:128 from LID:0x0148 
>> GID:0xfe80000000000000,0x0005ad00000281b3
>> Mar 20 14:18:30 728948 [41E02960] -> __osm_trap_rcv_process_request: 
>> Received Generic Notice type:0x01 num:128 Producer:2 from LID:0x0148 
>> TID:0x0000000000000060
>> Mar 20 14:18:30 729041 [41E02960] -> osm_report_notice: Reporting Generic 
>> Notice type:1 num:128 from LID:0x0148 
>> GID:0xfe80000000000000,0x0005ad00000281b3
>> Mar 20 14:18:30 872332 [45A08960] -> __osm_trap_rcv_process_request: 
>> Received Generic Notice type:0x01 num:128 Producer:2 from LID:0x0148 
>> TID:0x0000000000000061
>> Mar 20 14:18:30 872412 [45A08960] -> osm_report_notice: Reporting Generic 
>> Notice type:1 num:128 from LID:0x0148 
>> GID:0xfe80000000000000,0x0005ad00000281b3
>> Mar 20 14:18:30 899764 [45A08960] -> SUBNET UP
>> Mar 20 14:18:31 047423 [43C05960] -> __osm_trap_rcv_process_request: 
>> Received Generic Notice type:0x01 num:128 Producer:2 from LID:0x0148 
>> TID:0x0000000000000062
>> Mar 20 14:18:31 047611 [43C05960] -> osm_report_notice: Reporting Generic 
>> Notice type:1 num:128 from LID:0x0148 
>> GID:0xfe80000000000000,0x0005ad00000281b3
>> Mar 20 14:18:31 165201 [45A08960] -> __osm_trap_rcv_process_request: 
>> Received Generic Notice type:0x01 num:128 Producer:2 from LID:0x0148 
>> TID:0x0000000000000063
>> Mar 20 14:18:31 165272 [45A08960] -> osm_report_notice: Reporting Generic 
>> Notice type:1 num:128 from LID:0x0148 
>> GID:0xfe80000000000000,0x0005ad00000281b3
>> Mar 20 14:18:31 182461 [44606960] -> __osm_trap_rcv_process_request: 
>> Received Generic Notice type:0x01 num:128 Producer:2 from LID:0x001B 
>> TID:0x0000000000000074
>> Mar 20 14:18:31 182653 [44606960] -> osm_report_notice: Reporting Generic 
>> Notice type:1 num:128 from LID:0x001B 
>> GID:0xfe80000000000000,0x0005ad00000281a7
>> Mar 20 14:18:31 248834 [44606960] -> __osm_trap_rcv_process_request: 
>> Received Generic Notice type:0x01 num:128 Producer:2 from LID:0x001B 
>> TID:0x0000000000000075
>> Mar 20 14:18:31 248893 [44606960] -> osm_report_notice: Reporting Generic 
>> Notice type:1 num:128 from LID:0x001B 
>> GID:0xfe80000000000000,0x0005ad00000281a7
>> Mar 20 14:18:31 499830 [45A08960] -> __osm_trap_rcv_process_request: 
>> Received Generic Notice type:0x01 num:128 Producer:2 from LID:0x001B 
>> TID:0x0000000000000076
>> Mar 20 14:18:31 499908 [45A08960] -> osm_report_notice: Reporting Generic 
>> Notice type:1 num:128 from LID:0x001B 
>> GID:0xfe80000000000000,0x0005ad00000281a7
>> Mar 20 14:18:31 521824 [41401960] -> __osm_trap_rcv_process_request: 
>> Received Generic Notice type:0x01 num:128 Producer:2 from LID:0x001B 
>> TID:0x0000000000000077
>> Mar 20 14:18:31 521891 [41401960] -> osm_report_notice: Reporting Generic 
>> Notice type:1 num:128 from LID:0x001B 
>> GID:0xfe80000000000000,0x0005ad00000281a7
>> Mar 20 14:18:31 543713 [44606960] -> __osm_trap_rcv_process_request: 
>> Received Generic Notice type:0x01 num:128 Producer:2 from LID:0x001B 
>> TID:0x0000000000000078
>> Mar 20 14:18:31 543784 [44606960] -> osm_report_notice: Reporting Generic 
>> Notice type:1 num:128 from LID:0x001B 
>> GID:0xfe80000000000000,0x0005ad00000281a7
>> Mar 20 14:18:31 589490 [43C05960] -> osm_report_notice: Reporting Generic 
>> Notice type:3 num:64 from LID:0x0092 
>> GID:0xfe80000000000000,0x0005ad0000024bbb
>> Mar 20 14:18:31 589499 [43C05960] -> Discovered new port with 
>> GUID:0x0005ad0000024feb LID range [0x153,0x153] of node:saguaro-22-5 HCA-1
>> Mar 20 14:18:31 620166 [43C05960] -> osm_ucast_mgr_process: Min Hop Tables 
>> configured on all switches
>> Mar 20 14:18:31 672647 [4780B960] -> __osm_sm_mad_ctrl_rcv_callback: ERR 
>> 3111: Error status = 0x1C00
>> Mar 20 14:18:31 672739 [4780B960] -> SMP dump:
>>                 base_ver................0x1
>>                 mgmt_class..............0x81
>>                 class_ver...............0x1
>>                 method..................0x81 (SubnGetResp)
>>                 D bit...................0x1
>>                 status..................0x1C00
>>                 hop_ptr.................0x0
>>                 hop_count...............0x4
>>                 trans_id................0x77d11
>>                 attr_id.................0x15 (PortInfo)
>>                 resv....................0x0
>>                 attr_mod................0x16
>>                 m_key...................0x0000000000000000
>>                 dr_slid.................0xFFFF
>>                 dr_dlid.................0xFFFF
>> 
>>                 Initial path: [0][1][11][1][4]
>>                 Return path:  [0][9][18][D][1]
>>                 Reserved:     [0][0][0][0][0][0][0]
>> 
>>                 00 00 00 00 00 00 00 00   00 00 00 00 00 00 00 00
>> 
>>                 00 00 00 00 00 00 00 00   00 00 00 00 01 03 03 02
>> 
>>                 14 52 00 11 40 40 00 08   08 04 F2 40 00 00 00 00
>> 
>>                 00 00 00 00 00 88 00 00   00 00 00 00 00 00 00 00
>> 
>> Mar 20 14:18:31 672817 [43C05960] -> osm_pi_rcv_process_set: Received error 
>> status 0x1c for SetResp() during ACTIVE transition
>> Mar 20 14:18:31 672823 [4780B960] -> __osm_sm_mad_ctrl_rcv_callback: ERR 
>> 3111: Error status = 0x1C00
>> Mar 20 14:18:31 672833 [43C05960] -> PortInfo dump:
>>                 port number.............0x16
>>                 node_guid...............0x0005ad00000281b3
>>                 port_guid...............0x0005ad00000281b3
>>                 m_key...................0x0000000000000000
>>                 subnet_prefix...........0x0000000000000000
>>                 base_lid................0x0
>>                 master_sm_base_lid......0x0
>>                 capability_mask.........0x0
>>                 diag_code...............0x0
>>                 m_key_lease_period......0x0
>>                 local_port_num..........0x1
>>                 link_width_enabled......0x3
>>                 link_width_supported....0x3
>>                 link_width_active.......0x2
>>                 link_speed_supported....0x1
>>                 port_state..............ACTIVE
>>                 state_info2.............0x52
>>                 m_key_protect_bits......0x0
>>                 lmc.....................0x0
>>                 link_speed..............0x11
>>                 mtu_smsl................0x40
>>                 vl_cap_init_type........0x40
>>                 vl_high_limit...........0x0
>>                 vl_arb_high_cap.........0x8
>>                 vl_arb_low_cap..........0x8
>>                 init_rep_mtu_cap........0x4
>>                 vl_stall_life...........0xF2
>>                 vl_enforce..............0x40
>>                 m_key_violations........0x0
>>                 p_key_violations........0x0
>>                 q_key_violations........0x0
>>                 guid_cap................0x0
>>                 client_reregister.......0x0
>>                 subnet_timeout..........0x0
>>                 resp_time_value.........0x0
>>                 error_threshold.........0x88
>> Mar 20 14:18:31 672852 [43C05960] -> Capabilities Mask:
>> Mar 20 14:18:31 672861 [4780B960] -> SMP dump:
>>                 base_ver................0x1
>>                 mgmt_class..............0x81
>>                 class_ver...............0x1
>>                 method..................0x81 (SubnGetResp)
>>                 D bit...................0x1
>>                 status..................0x1C00
>>                 hop_ptr.................0x0
>>                 hop_count...............0x4
>>                 trans_id................0x77d12
>>                 attr_id.................0x15 (PortInfo)
>>                 resv....................0x0
>>                 attr_mod................0x17
>>                 m_key...................0x0000000000000000
>>                 dr_slid.................0xFFFF
>>                 dr_dlid.................0xFFFF
>> 
>>                 Initial path: [0][1][11][1][4]
>>                 Return path:  [0][9][18][D][1]
>>                 Reserved:     [0][0][0][0][0][0][0]
>> 
>>                 00 00 00 00 00 00 00 00   00 00 00 00 00 00 00 00
>> 
>>                 00 00 00 00 00 00 00 00   00 00 00 00 01 03 03 02
>> 
>>                 14 52 00 11 40 40 00 08   08 04 F2 40 00 00 00 00
>> 
>>                 00 00 00 00 00 88 00 00   00 00 00 00 00 00 00 00
>> 
>> Mar 20 14:18:31 672918 [45007960] -> osm_pi_rcv_process_set: Received error 
>> status 0x1c for SetResp() during ACTIVE transition
>> Mar 20 14:18:31 672922 [45007960] -> PortInfo dump:
>>                 port number.............0x17
>>                 node_guid...............0x0005ad00000281b3
>>                 port_guid...............0x0005ad00000281b3
>>                 m_key...................0x0000000000000000
>>                 subnet_prefix...........0x0000000000000000
>>                 base_lid................0x0
>>                 master_sm_base_lid......0x0
>>                 capability_mask.........0x0
>>                 diag_code...............0x0
>>                 m_key_lease_period......0x0
>>                 local_port_num..........0x1
>>                 link_width_enabled......0x3
>>                 link_width_supported....0x3
>>                 link_width_active.......0x2
>>                 link_speed_supported....0x1
>>                 port_state..............ACTIVE
>>                 state_info2.............0x52
>>                 m_key_protect_bits......0x0
>>                 lmc.....................0x0
>>                 link_speed..............0x11
>>                 mtu_smsl................0x40
>>                 vl_cap_init_type........0x40
>>                 vl_high_limit...........0x0
>>                 vl_arb_high_cap.........0x8
>>                 vl_arb_low_cap..........0x8
>>                 init_rep_mtu_cap........0x4
>>                 vl_stall_life...........0xF2
>>                 vl_enforce..............0x40
>>                 m_key_violations........0x0
>>                 p_key_violations........0x0
>>                 q_key_violations........0x0
>>                 guid_cap................0x0
>>                 client_reregister.......0x0
>>                 subnet_timeout..........0x0
>>                 resp_time_value.........0x0
>>                 error_threshold.........0x88
>> Mar 20 14:18:31 672936 [45007960] -> Capabilities Mask:
>> Mar 20 14:18:31 678085 [45A08960] -> SUBNET UP
>> Mar 20 14:18:31 723715 [41E02960] -> __osm_trap_rcv_process_request: 
>> Received Generic Notice type:0x01 num:128 Producer:2 from LID:0x0152 
>> TID:0x0000000000000018
>> Mar 20 14:18:31 723815 [41E02960] -> __osm_trap_rcv_process_request: ERR 
>> 3804: Received trap 25 times consecutively
>> Mar 20 14:18:32 061932 [41401960] -> osm_ucast_mgr_process: Min Hop Tables 
>> configured on all switches
>> Mar 20 14:18:32 113545 [4780B960] -> __osm_sm_mad_ctrl_rcv_callback: ERR 
>> 3111: Error status = 0x1C00
>> Mar 20 14:18:32 113610 [4780B960] -> SMP dump:
>>                 base_ver................0x1
>>                 mgmt_class..............0x81
>>                 class_ver...............0x1
>>                 method..................0x81 (SubnGetResp)
>>                 D bit...................0x1
>>                 status..................0x1C00
>>                 hop_ptr.................0x0
>>                 hop_count...............0x4
>>                 trans_id................0x78a4d
>>                 attr_id.................0x15 (PortInfo)
>>                 resv....................0x0
>>                 attr_mod................0x13
>>                 m_key...................0x0000000000000000
>>                 dr_slid.................0xFFFF
>>                 dr_dlid.................0xFFFF
>> 
>>                 Initial path: [0][1][15][4][4]
>>                 Return path:  [0][9][18][D][4]
>>                 Reserved:     [0][0][0][0][0][0][0]
>> 
>>                 00 00 00 00 00 00 00 00   00 00 00 00 00 00 00 00
>> 
>>                 00 00 00 00 00 00 00 00   00 00 00 00 04 03 03 02
>> 
>>                 14 52 00 11 40 40 00 08   08 04 F2 40 00 00 00 00
>> 
>>                 00 00 00 00 00 88 00 00   00 00 00 00 00 00 00 00
>> 
>> Mar 20 14:18:32 113712 [42803960] -> osm_pi_rcv_process_set: Received error 
>> status 0x1c for SetResp() during ACTIVE transition
>> Mar 20 14:18:32 113725 [4780B960] -> __osm_sm_mad_ctrl_rcv_callback: ERR 
>> 3111: Error status = 0x1C00
>> Mar 20 14:18:32 113730 [42803960] -> PortInfo dump:
>>                 port number.............0x13
>>                 node_guid...............0x0005ad00000281a7
>>                 port_guid...............0x0005ad00000281a7
>>                 m_key...................0x0000000000000000
>>                 subnet_prefix...........0x0000000000000000
>>                 base_lid................0x0
>>                 master_sm_base_lid......0x0
>>                 capability_mask.........0x0
>>                 diag_code...............0x0
>>                 m_key_lease_period......0x0
>>                 local_port_num..........0x4
>>                 link_width_enabled......0x3
>>                 link_width_supported....0x3
>>                 link_width_active.......0x2
>>                 link_speed_supported....0x1
>>                 port_state..............ACTIVE
>>                 state_info2.............0x52
>>                 m_key_protect_bits......0x0
>>                 lmc.....................0x0
>>                 link_speed..............0x11
>>                 mtu_smsl................0x40
>>                 vl_cap_init_type........0x40
>>                 vl_high_limit...........0x0
>>                 vl_arb_high_cap.........0x8
>>                 vl_arb_low_cap..........0x8
>>                 init_rep_mtu_cap........0x4
>>                 vl_stall_life...........0xF2
>>                 vl_enforce..............0x40
>>                 m_key_violations........0x0
>>                 p_key_violations........0x0
>>                 q_key_violations........0x0
>>                 guid_cap................0x0
>>                 client_reregister.......0x0
>>                 subnet_timeout..........0x0
>>                 resp_time_value.........0x0
>>                 error_threshold.........0x88
>> Mar 20 14:18:32 113745 [42803960] -> Capabilities Mask:
>> Mar 20 14:18:32 113751 [4780B960] -> SMP dump:
>>                 base_ver................0x1
>>                 mgmt_class..............0x81
>>                 class_ver...............0x1
>>                 method..................0x81 (SubnGetResp)
>>                 D bit...................0x1
>>                 status..................0x1C00
>>                 hop_ptr.................0x0
>>                 hop_count...............0x4
>>                 trans_id................0x78a4e
>>                 attr_id.................0x15 (PortInfo)
>>                 resv....................0x0
>>                 attr_mod................0x16
>>                 m_key...................0x0000000000000000
>>                 dr_slid.................0xFFFF
>>                 dr_dlid.................0xFFFF
>> 
>>                 Initial path: [0][1][15][4][4]
>>                 Return path:  [0][9][18][D][4]
>>                 Reserved:     [0][0][0][0][0][0][0]
>> 
>>                 00 00 00 00 00 00 00 00   00 00 00 00 00 00 00 00
>> 
>>                 00 00 00 00 00 00 00 00   00 00 00 00 04 03 03 02
>> 
>>                 14 52 00 11 40 40 00 08   08 04 F2 40 00 00 00 00
>> 
>>                 00 00 00 00 00 88 00 00   00 00 00 00 00 00 00 00
>> 
>> Mar 20 14:18:32 113803 [44606960] -> osm_pi_rcv_process_set: Received error 
>> status 0x1c for SetResp() during ACTIVE transition
>> Mar 20 14:18:32 113807 [44606960] -> PortInfo dump:
>>                 port number.............0x16
>>                 node_guid...............0x0005ad00000281a7
>>                 port_guid...............0x0005ad00000281a7
>>                 m_key...................0x0000000000000000
>>                 subnet_prefix...........0x0000000000000000
>>                 base_lid................0x0
>>                 master_sm_base_lid......0x0
>>                 capability_mask.........0x0
>>                 diag_code...............0x0
>>                 m_key_lease_period......0x0
>>                 local_port_num..........0x4
>>                 link_width_enabled......0x3
>>                 link_width_supported....0x3
>>                 link_width_active.......0x2
>>                 link_speed_supported....0x1
>>                 port_state..............ACTIVE
>>                 state_info2.............0x52
>>                 m_key_protect_bits......0x0
>>                 lmc.....................0x0
>>                 link_speed..............0x11
>>                 mtu_smsl................0x40
>>                 vl_cap_init_type........0x40
>>                 vl_high_limit...........0x0
>>                 vl_arb_high_cap.........0x8
>>                 vl_arb_low_cap..........0x8
>>                 init_rep_mtu_cap........0x4
>>                 vl_stall_life...........0xF2
>>                 vl_enforce..............0x40
>>                 m_key_violations........0x0
>>                 p_key_violations........0x0
>>                 q_key_violations........0x0
>>                 guid_cap................0x0
>>                 client_reregister.......0x0
>>                 subnet_timeout..........0x0
>>                 resp_time_value.........0x0
>>                 error_threshold.........0x88
>> Mar 20 14:18:32 113820 [44606960] -> Capabilities Mask:
>> Mar 20 14:18:32 113845 [4780B960] -> __osm_sm_mad_ctrl_rcv_callback: ERR 
>> 3111: Error status = 0x1C00
>> Mar 20 14:18:32 113907 [4780B960] -> SMP dump:
>>                 base_ver................0x1
>>                 mgmt_class..............0x81
>>                 class_ver...............0x1
>>                 method..................0x81 (SubnGetResp)
>>                 D bit...................0x1
>>                 status..................0x1C00
>>                 hop_ptr.................0x0
>>                 hop_count...............0x4
>>                 trans_id................0x78a4f
>>                 attr_id.................0x15 (PortInfo)
>>                 resv....................0x0
>>                 attr_mod................0x17
>>                 m_key...................0x0000000000000000
>>                 dr_slid.................0xFFFF
>>                 dr_dlid.................0xFFFF
>> 
>>                 Initial path: [0][1][15][4][4]
>>                 Return path:  [0][9][18][D][4]
>>                 Reserved:     [0][0][0][0][0][0][0]
>> 
>>                 00 00 00 00 00 00 00 00   00 00 00 00 00 00 00 00
>> 
>>                 00 00 00 00 00 00 00 00   00 00 00 00 04 03 03 02
>> 
>>                 14 52 00 11 40 40 00 08   08 04 F2 40 00 00 00 00
>> 
>>                 00 00 00 00 00 88 00 00   00 00 00 00 00 00 00 00
>> 
>> Mar 20 14:18:32 113958 [41E02960] -> osm_pi_rcv_process_set: Received error 
>> status 0x1c for SetResp() during ACTIVE transition
>> Mar 20 14:18:32 113963 [4780B960] -> __osm_sm_mad_ctrl_rcv_callback: ERR 
>> 3111: Error status = 0x1C00
>> Mar 20 14:18:32 113969 [41E02960] -> PortInfo dump:
>>                 port number.............0x17
>>                 node_guid...............0x0005ad00000281a7
>>                 port_guid...............0x0005ad00000281a7
>>                 m_key...................0x0000000000000000
>>                 subnet_prefix...........0x0000000000000000
>>                 base_lid................0x0
>>                 master_sm_base_lid......0x0
>>                 capability_mask.........0x0
>>                 diag_code...............0x0
>>                 m_key_lease_period......0x0
>>                 local_port_num..........0x4
>>                 link_width_enabled......0x3
>>                 link_width_supported....0x3
>>                 link_width_active.......0x2
>>                 link_speed_supported....0x1
>>                 port_state..............ACTIVE
>>                 state_info2.............0x52
>>                 m_key_protect_bits......0x0
>>                 lmc.....................0x0
>>                 link_speed..............0x11
>>                 mtu_smsl................0x40
>>                 vl_cap_init_type........0x40
>>                 vl_high_limit...........0x0
>>                 vl_arb_high_cap.........0x8
>>                 vl_arb_low_cap..........0x8
>>                 init_rep_mtu_cap........0x4
>>                 vl_stall_life...........0xF2
>>                 vl_enforce..............0x40
>>                 m_key_violations........0x0
>>                 p_key_violations........0x0
>>                 q_key_violations........0x0
>>                 guid_cap................0x0
>>                 client_reregister.......0x0
>>                 subnet_timeout..........0x0
>>                 resp_time_value.........0x0
>>                 error_threshold.........0x88
>> Mar 20 14:18:32 113986 [41E02960] -> Capabilities Mask:
>> Mar 20 14:18:32 113992 [4780B960] -> SMP dump:
>>                 base_ver................0x1
>>                 mgmt_class..............0x81
>>                 class_ver...............0x1
>>                 method..................0x81 (SubnGetResp)
>>                 D bit...................0x1
>>                 status..................0x1C00
>>                 hop_ptr.................0x0
>>                 hop_count...............0x4
>>                 trans_id................0x78a50
>>                 attr_id.................0x15 (PortInfo)
>>                 resv....................0x0
>>                 attr_mod................0x18
>>                 m_key...................0x0000000000000000
>>                 dr_slid.................0xFFFF
>>                 dr_dlid.................0xFFFF
>> 
>>                 Initial path: [0][1][15][4][4]
>>                 Return path:  [0][9][18][D][4]
>>                 Reserved:     [0][0][0][0][0][0][0]
>> 
>>                 00 00 00 00 00 00 00 00   00 00 00 00 00 00 00 00
>> 
>>                 00 00 00 00 00 00 00 00   00 00 00 00 04 03 03 02
>> 
>>                 14 52 00 11 40 40 00 08   08 04 F2 40 00 00 00 00
>> 
>>                 00 00 00 00 00 88 00 00   00 00 00 00 00 00 00 00
>> 
>> Mar 20 14:18:32 114052 [45A08960] -> osm_pi_rcv_process_set: Received error 
>> status 0x1c for SetResp() during ACTIVE transition
>> Mar 20 14:18:32 114066 [45A08960] -> PortInfo dump:
>>                 port number.............0x18
>>                 node_guid...............0x0005ad00000281a7
>>                 port_guid...............0x0005ad00000281a7
>>                 m_key...................0x0000000000000000
>>                 subnet_prefix...........0x0000000000000000
>>                 base_lid................0x0
>>                 master_sm_base_lid......0x0
>>                 capability_mask.........0x0
>>                 diag_code...............0x0
>>                 m_key_lease_period......0x0
>>                 local_port_num..........0x4
>>                 link_width_enabled......0x3
>>                 link_width_supported....0x3
>>                 link_width_active.......0x2
>>                 link_speed_supported....0x1
>>                 port_state..............ACTIVE
>>                 state_info2.............0x52
>>                 m_key_protect_bits......0x0
>>                 lmc.....................0x0
>>                 link_speed..............0x11
>>                 mtu_smsl................0x40
>>                 vl_cap_init_type........0x40
>>                 vl_high_limit...........0x0
>>                 vl_arb_high_cap.........0x8
>>                 vl_arb_low_cap..........0x8
>>                 init_rep_mtu_cap........0x4
>>                 vl_stall_life...........0xF2
>>                 vl_enforce..............0x40
>>                 m_key_violations........0x0
>>                 p_key_violations........0x0
>>                 q_key_violations........0x0
>>                 guid_cap................0x0
>>                 client_reregister.......0x0
>>                 subnet_timeout..........0x0
>>                 resp_time_value.........0x0
>>                 error_threshold.........0x88
>> Mar 20 14:18:32 114089 [45A08960] -> Capabilities Mask:
>> Mar 20 14:18:32 114052 [4780B960] -> __osm_sm_mad_ctrl_rcv_callback: ERR 
>> 3111: Error status = 0x1C00
>> Mar 20 14:18:32 114171 [4780B960] -> SMP dump:
>>                 base_ver................0x1
>>                 mgmt_class..............0x81
>>                 class_ver...............0x1
>>                 method..................0x81 (SubnGetResp)
>>                 D bit...................0x1
>>                 status..................0x1C00
>>                 hop_ptr.................0x0
>>                 hop_count...............0x4
>>                 trans_id................0x78a51
>>                 attr_id.................0x15 (PortInfo)
>>                 resv....................0x0
>>                 attr_mod................0x18
>>                 m_key...................0x0000000000000000
>>                 dr_slid.................0xFFFF
>>                 dr_dlid.................0xFFFF
>> 
>>                 Initial path: [0][1][13][1][6]
>>                 Return path:  [0][9][13][D][3]
>>                 Reserved:     [0][0][0][0][0][0][0]
>> 
>>                 00 00 00 00 00 00 00 00   00 00 00 00 00 00 00 00
>> 
>>                 00 00 00 00 00 00 00 00   00 00 00 00 03 03 03 02
>> 
>>                 14 52 00 11 40 40 00 08   08 04 F2 40 00 00 00 00
>> 
>>                 00 00 00 00 00 88 00 00   00 00 00 00 00 00 00 00
>> 
>> Mar 20 14:18:32 114224 [42803960] -> osm_pi_rcv_process_set: Received error 
>> status 0x1c for SetResp() during ACTIVE transition
>> Mar 20 14:18:32 114228 [42803960] -> PortInfo dump:
>>                 port number.............0x18
>>                 node_guid...............0x0005ad00000281b3
>>                 port_guid...............0x0005ad00000281b3
>>                 m_key...................0x0000000000000000
>>                 subnet_prefix...........0x0000000000000000
>>                 base_lid................0x0
>>                 master_sm_base_lid......0x0
>>                 capability_mask.........0x0
>>                 diag_code...............0x0
>>                 m_key_lease_period......0x0
>>                 local_port_num..........0x3
>>                 link_width_enabled......0x3
>>                 link_width_supported....0x3
>>                 link_width_active.......0x2
>>                 link_speed_supported....0x1
>>                 port_state..............ACTIVE
>>                 state_info2.............0x52
>>                 m_key_protect_bits......0x0
>>                 lmc.....................0x0
>>                 link_speed..............0x11
>>                 mtu_smsl................0x40
>>                 vl_cap_init_type........0x40
>>                 vl_high_limit...........0x0
>>                 vl_arb_high_cap.........0x8
>>                 vl_arb_low_cap..........0x8
>>                 init_rep_mtu_cap........0x4
>>                 vl_stall_life...........0xF2
>>                 vl_enforce..............0x40
>>                 m_key_violations........0x0
>>                 p_key_violations........0x0
>>                 q_key_violations........0x0
>>                 guid_cap................0x0
>>                 client_reregister.......0x0
>>                 subnet_timeout..........0x0
>>                 resp_time_value.........0x0
>>                 error_threshold.........0x88
>> Mar 20 14:18:32 114242 [42803960] -> Capabilities Mask:
>> Mar 20 14:18:32 119326 [42803960] -> SUBNET UP
>> Mar 20 14:23:02 506774 [41E02960] -> __osm_trap_rcv_process_request: 
>> Received Generic Notice type:0x01 num:128 Producer:2 from LID:0x0152 
>> TID:0x0000000000000019
>> Mar 20 14:23:02 507064 [41E02960] -> osm_report_notice: Reporting Generic 
>> Notice type:1 num:128 from LID:0x0152 
>> GID:0xfe80000000000000,0x0005ad0000027c84
>> Mar 20 14:23:02 861642 [43204960] -> osm_report_notice: Reporting Generic 
>> Notice type:3 num:64 from LID:0x0092 
>> GID:0xfe80000000000000,0x0005ad0000024bbb
>> Mar 20 14:23:02 861653 [43204960] -> Discovered new port with 
>> GUID:0x0005ad0000024b27 LID range [0xAF,0xAF] of node:Topspin IB-DC
>> Mar 20 14:23:02 893030 [43204960] -> osm_ucast_mgr_process: Min Hop Tables 
>> configured on all switches
>> Mar 20 14:23:02 943693 [4780B960] -> __osm_sm_mad_ctrl_rcv_callback: ERR 
>> 3111: Error status = 0x1C00
>> Mar 20 14:23:02 943765 [4780B960] -> SMP dump:
>>                 base_ver................0x1
>>                 mgmt_class..............0x81
>>                 class_ver...............0x1
>>                 method..................0x81 (SubnGetResp)
>>                 D bit...................0x1
>>                 status..................0x1C00
>>                 hop_ptr.................0x0
>>                 hop_count...............0x5
>>                 trans_id................0x79aff
>>                 attr_id.................0x15 (PortInfo)
>>                 resv....................0x0
>>                 attr_mod................0x1
>>                 m_key...................0x0000000000000000
>>                 dr_slid.................0xFFFF
>>                 dr_dlid.................0xFFFF
>> 
>>                 Initial path: [0][1][11][1][5][18]
>>                 Return path:  [0][9][18][D][2][13]
>>                 Reserved:     [0][0][0][0][0][0][0]
>> 
>>                 00 00 00 00 00 00 00 00   00 00 00 00 00 00 00 00
>> 
>>                 00 00 00 00 00 00 00 00   00 00 00 00 13 03 03 02
>> 
>>                 14 52 00 11 40 40 00 08   08 04 2C 4C 00 00 00 00
>> 
>>                 00 00 00 00 00 88 00 00   00 00 00 00 00 00 00 00
>> 
>> Mar 20 14:23:02 943854 [43C05960] -> osm_pi_rcv_process_set: Received error 
>> status 0x1c for SetResp() during ACTIVE transition
>> Mar 20 14:23:02 943870 [43C05960] -> PortInfo dump:
>>                 port number.............0x1
>>                 node_guid...............0x0005ad0000027c84
>>                 port_guid...............0x0005ad0000027c84
>>                 m_key...................0x0000000000000000
>>                 subnet_prefix...........0x0000000000000000
>>                 base_lid................0x0
>>                 master_sm_base_lid......0x0
>>                 capability_mask.........0x0
>>                 diag_code...............0x0
>>                 m_key_lease_period......0x0
>>                 local_port_num..........0x13
>>                 link_width_enabled......0x3
>>                 link_width_supported....0x3
>>                 link_width_active.......0x2
>>                 link_speed_supported....0x1
>>                 port_state..............ACTIVE
>>                 state_info2.............0x52
>>                 m_key_protect_bits......0x0
>>                 lmc.....................0x0
>>                 link_speed..............0x11
>>                 mtu_smsl................0x40
>>                 vl_cap_init_type........0x40
>>                 vl_high_limit...........0x0
>>                 vl_arb_high_cap.........0x8
>>                 vl_arb_low_cap..........0x8
>>                 init_rep_mtu_cap........0x4
>>                 vl_stall_life...........0x2C
>>                 vl_enforce..............0x4C
>>                 m_key_violations........0x0
>>                 p_key_violations........0x0
>>                 q_key_violations........0x0
>>                 guid_cap................0x0
>>                 client_reregister.......0x0
>>                 subnet_timeout..........0x0
>>                 resp_time_value.........0x0
>>                 error_threshold.........0x88
>> Mar 20 14:23:02 943886 [43C05960] -> Capabilities Mask:
>> Mar 20 14:23:02 948898 [43C05960] -> SUBNET UP
>> Mar 20 14:23:03 237496 [42803960] -> __osm_trap_rcv_process_request: 
>> Received Generic Notice type:0x04 num:144 Producer:1 from LID:0x00AF 
>> TID:0x0000000000000000
>> Mar 20 14:23:03 237710 [42803960] -> osm_report_notice: Reporting Generic 
>> Notice type:4 num:144 from LID:0x00AF 
>> GID:0xfe80000000000000,0x0005ad0000024b27
>> Mar 20 14:23:03 605548 [45007960] -> osm_ucast_mgr_process: Min Hop Tables 
>> configured on all switches
>> Mar 20 14:23:03 662757 [41401960] -> SUBNET UP
>> Mar 20 14:24:54 675782 [44606960] -> __osm_trap_rcv_process_request: 
>> Received Generic Notice type:0x01 num:128 Producer:2 from LID:0x001B 
>> TID:0x0000000000000079
>> Mar 20 14:24:54 676077 [44606960] -> osm_report_notice: Reporting Generic 
>> Notice type:1 num:128 from LID:0x001B 
>> GID:0xfe80000000000000,0x0005ad00000281a7
>> Mar 20 14:24:54 677026 [43204960] -> __osm_trap_rcv_process_request: 
>> Received Generic Notice type:0x01 num:128 Producer:2 from LID:0x0148 
>> TID:0x0000000000000064
>> Mar 20 14:24:54 677118 [43204960] -> osm_report_notice: Reporting Generic 
>> Notice type:1 num:128 from LID:0x0148 
>> GID:0xfe80000000000000,0x0005ad00000281b3
>> Mar 20 14:24:55 047478 [43204960] -> osm_report_notice: Reporting Generic 
>> Notice type:3 num:65 from LID:0x0092 
>> GID:0xfe80000000000000,0x0005ad0000024bbb
>> Mar 20 14:24:55 047501 [43204960] -> Removed port with 
>> GUID:0x0005ad0000024e0b LID range [0xB3,0xB3] of node:saguaro-23-4 HCA-1
>> Mar 20 14:24:55 047520 [43204960] -> osm_report_notice: Reporting Generic 
>> Notice type:3 num:65 from LID:0x0092 
>> GID:0xfe80000000000000,0x0005ad0000024bbb
>> Mar 20 14:24:55 047525 [43204960] -> Removed port with 
>> GUID:0x0005ad000002510b LID range [0xB5,0xB5] of node:saguaro-23-6 HCA-1
>> Mar 20 14:24:55 047541 [43204960] -> osm_report_notice: Reporting Generic 
>> Notice type:3 num:65 from LID:0x0092 
>> GID:0xfe80000000000000,0x0005ad0000024bbb
>> Mar 20 14:24:55 047546 [43204960] -> Removed port with 
>> GUID:0x0005ad000002511b LID range [0xA6,0xA6] of node:saguaro-22-1 HCA-1
>> Mar 20 14:24:55 047563 [43204960] -> osm_report_notice: Reporting Generic 
>> Notice type:3 num:65 from LID:0x0092 
>> GID:0xfe80000000000000,0x0005ad0000024bbb
>> Mar 20 14:24:55 047569 [43204960] -> Removed port with 
>> GUID:0x0005ad0000024b27 LID range [0xAF,0xAF] of node:Topspin IB-DC
>> Mar 20 14:24:55 047586 [43204960] -> osm_report_notice: Reporting Generic 
>> Notice type:3 num:65 from LID:0x0092 
>> GID:0xfe80000000000000,0x0005ad0000024bbb
>> Mar 20 14:24:55 047591 [43204960] -> Removed port with 
>> GUID:0x0005ad0000025043 LID range [0xB4,0xB4] of node:saguaro-23-5 HCA-1
>> Mar 20 14:24:55 047607 [43204960] -> osm_report_notice: Reporting Generic 
>> Notice type:3 num:65 from LID:0x0092 
>> GID:0xfe80000000000000,0x0005ad0000024bbb
>> Mar 20 14:24:55 047612 [43204960] -> Removed port with 
>> GUID:0x0005ad0000024d47 LID range [0xB6,0xB6] of node:saguaro-23-7 HCA-1
>> Mar 20 14:24:55 047630 [43204960] -> osm_report_notice: Reporting Generic 
>> Notice type:3 num:65 from LID:0x0092 
>> GID:0xfe80000000000000,0x0005ad0000024bbb
>> Mar 20 14:24:55 047635 [43204960] -> Removed port with 
>> GUID:0x0005ad0000024d6b LID range [0xB8,0xB8] of node:saguaro-23-9 HCA-1
>> Mar 20 14:24:55 047652 [43204960] -> osm_report_notice: Reporting Generic 
>> Notice type:3 num:65 from LID:0x0092 
>> GID:0xfe80000000000000,0x0005ad0000024bbb
>> Mar 20 14:24:55 047657 [43204960] -> Removed port with 
>> GUID:0x0005ad0000024977 LID range [0xA9,0xA9] of node:saguaro-22-4 HCA-1
>> Mar 20 14:24:55 047798 [43204960] -> osm_report_notice: Reporting Generic 
>> Notice type:3 num:65 from LID:0x0092 
>> GID:0xfe80000000000000,0x0005ad0000024bbb
>> Mar 20 14:24:55 047803 [43204960] -> Removed port with 
>> GUID:0x0005ad0000027c84 LID range [0x152,0x152] of node:Topspin Switch TS120
>> Mar 20 14:24:55 047836 [43204960] -> osm_report_notice: Reporting Generic 
>> Notice type:3 num:65 from LID:0x0092 
>> GID:0xfe80000000000000,0x0005ad0000024bbb
>> Mar 20 14:24:55 047842 [43204960] -> Removed port with 
>> GUID:0x0005ad0000024d8b LID range [0xB7,0xB7] of node:saguaro-23-8 HCA-1
>> Mar 20 14:24:55 047857 [43204960] -> osm_report_notice: Reporting Generic 
>> Notice type:3 num:65 from LID:0x0092 
>> GID:0xfe80000000000000,0x0005ad0000024bbb
>> Mar 20 14:24:55 047862 [43204960] -> Removed port with 
>> GUID:0x0005ad000002498f LID range [0xA8,0xA8] of node:saguaro-22-3 HCA-1
>> Mar 20 14:24:55 047877 [43204960] -> osm_report_notice: Reporting Generic 
>> Notice type:3 num:65 from LID:0x0092 
>> GID:0xfe80000000000000,0x0005ad0000024bbb
>> Mar 20 14:24:55 047882 [43204960] -> Removed port with 
>> GUID:0x0005ad0000024c9b LID range [0xA7,0xA7] of node:saguaro-22-2 HCA-1
>> Mar 20 14:24:55 047896 [43204960] -> osm_report_notice: Reporting Generic 
>> Notice type:3 num:65 from LID:0x0092 
>> GID:0xfe80000000000000,0x0005ad0000024bbb
>> Mar 20 14:24:55 047902 [43204960] -> Removed port with 
>> GUID:0x0005ad0000024da7 LID range [0xB0,0xB0] of node:saguaro-23-1 HCA-1
>> Mar 20 14:24:55 047918 [43204960] -> osm_report_notice: Reporting Generic 
>> Notice type:3 num:65 from LID:0x0092 
>> GID:0xfe80000000000000,0x0005ad0000024bbb
>> Mar 20 14:24:55 047923 [43204960] -> Removed port with 
>> GUID:0x0005ad0000024cbb LID range [0xB2,0xB2] of node:saguaro-23-3 HCA-1
>> Mar 20 14:24:55 047939 [43204960] -> osm_report_notice: Reporting Generic 
>> Notice type:3 num:65 from LID:0x0092 
>> GID:0xfe80000000000000,0x0005ad0000024bbb
>> Mar 20 14:24:55 047988 [43204960] -> Removed port with 
>> GUID:0x0005ad00000249d3 LID range [0xB1,0xB1] of node:saguaro-23-2 HCA-1
>> Mar 20 14:24:55 048005 [43204960] -> osm_report_notice: Reporting Generic 
>> Notice type:3 num:65 from LID:0x0092 
>> GID:0xfe80000000000000,0x0005ad0000024bbb
>> Mar 20 14:24:55 048010 [43204960] -> Removed port with 
>> GUID:0x0005ad0000024feb LID range [0x153,0x153] of node:saguaro-22-5 HCA-1
>> Mar 20 14:24:55 048025 [43204960] -> osm_report_notice: Reporting Generic 
>> Notice type:3 num:65 from LID:0x0092 
>> GID:0xfe80000000000000,0x0005ad0000024bbb
>> Mar 20 14:24:55 048030 [43204960] -> Removed port with 
>> GUID:0x0005ad0000024afb LID range [0xA5,0xA5] of node:saguaro-22-0 HCA-1
>> Mar 20 14:24:55 081006 [43204960] -> osm_ucast_mgr_process: Min Hop Tables 
>> configured on all switches
>> Mar 20 14:24:55 130875 [45A08960] -> SUBNET UP
>> Mar 20 14:24:55 484995 [42803960] -> osm_ucast_mgr_process: Min Hop Tables 
>> configured on all switches
>> Mar 20 14:24:55 535902 [42803960] -> SUBNET UP
>> Mar 20 14:25:48 653788 [43204960] -> __osm_trap_rcv_process_request: 
>> Received Generic Notice type:0x01 num:128 Producer:2 from LID:0x0148 
>> TID:0x0000000000000065
>> Mar 20 14:25:48 654009 [43204960] -> osm_report_notice: Reporting Generic 
>> Notice type:1 num:128 from LID:0x0148 
>> GID:0xfe80000000000000,0x0005ad00000281b3
>> Mar 20 14:25:48 659749 [45A08960] -> __osm_trap_rcv_process_request: 
>> Received Generic Notice type:0x01 num:128 Producer:2 from LID:0x001B 
>> TID:0x000000000000007a
>> Mar 20 14:25:48 659790 [42803960] -> __osm_trap_rcv_process_request: 
>> Received Generic Notice type:0x01 num:128 Producer:2 from LID:0x0148 
>> TID:0x0000000000000066
>> Mar 20 14:25:48 659814 [45A08960] -> osm_report_notice: Reporting Generic 
>> Notice type:1 num:128 from LID:0x001B 
>> GID:0xfe80000000000000,0x0005ad00000281a7
>> Mar 20 14:25:48 659963 [42803960] -> osm_report_notice: Reporting Generic 
>> Notice type:1 num:128 from LID:0x0148 
>> GID:0xfe80000000000000,0x0005ad00000281b3
>> Mar 20 14:25:48 665972 [41401960] -> __osm_trap_rcv_process_request: 
>> Received Generic Notice type:0x01 num:128 Producer:2 from LID:0x001B 
>> TID:0x000000000000007b
>> Mar 20 14:25:48 666050 [41401960] -> osm_report_notice: Reporting Generic 
>> Notice type:1 num:128 from LID:0x001B 
>> GID:0xfe80000000000000,0x0005ad00000281a7
>> Mar 20 14:25:49 025384 [41E02960] -> osm_report_notice: Reporting Generic 
>> Notice type:3 num:64 from LID:0x0092 
>> GID:0xfe80000000000000,0x0005ad0000024bbb
>> Mar 20 14:25:49 025396 [41E02960] -> Discovered new port with 
>> GUID:0x0005ad0000027c84 LID range [0x152,0x152] of node:Topspin Switch TS120
>> Mar 20 14:25:49 025401 [41E02960] -> osm_report_notice: Reporting Generic 
>> Notice type:3 num:64 from LID:0x0092 
>> GID:0xfe80000000000000,0x0005ad0000024bbb
>> Mar 20 14:25:49 025406 [41E02960] -> Discovered new port with 
>> GUID:0x0005ad0000024b27 LID range [0xAF,0xAF] of node:saguaro-23-0 HCA-1
>> Mar 20 14:25:49 025410 [41E02960] -> osm_report_notice: Reporting Generic 
>> Notice type:3 num:64 from LID:0x0092 
>> GID:0xfe80000000000000,0x0005ad0000024bbb
>> Mar 20 14:25:49 025416 [41E02960] -> Discovered new port with 
>> GUID:0x0005ad0000024da7 LID range [0xB0,0xB0] of node:saguaro-23-1 HCA-1
>> Mar 20 14:25:49 025420 [41E02960] -> osm_report_notice: Reporting Generic 
>> Notice type:3 num:64 from LID:0x0092 
>> GID:0xfe80000000000000,0x0005ad0000024bbb
>> Mar 20 14:25:49 025425 [41E02960] -> Discovered new port with 
>> GUID:0x0005ad00000249d3 LID range [0xB1,0xB1] of node:saguaro-23-2 HCA-1
>> Mar 20 14:25:49 025428 [41E02960] -> osm_report_notice: Reporting Generic 
>> Notice type:3 num:64 from LID:0x0092 
>> GID:0xfe80000000000000,0x0005ad0000024bbb
>> Mar 20 14:25:49 025433 [41E02960] -> Discovered new port with 
>> GUID:0x0005ad0000024cbb LID range [0xB2,0xB2] of node:saguaro-23-3 HCA-1
>> Mar 20 14:25:49 025437 [41E02960] -> osm_report_notice: Reporting Generic 
>> Notice type:3 num:64 from LID:0x0092 
>> GID:0xfe80000000000000,0x0005ad0000024bbb
>> Mar 20 14:25:49 025442 [41E02960] -> Discovered new port with 
>> GUID:0x0005ad0000024e0b LID range [0xB3,0xB3] of node:saguaro-23-4 HCA-1
>> Mar 20 14:25:49 025446 [41E02960] -> osm_report_notice: Reporting Generic 
>> Notice type:3 num:64 from LID:0x0092 
>> GID:0xfe80000000000000,0x0005ad0000024bbb
>> Mar 20 14:25:49 025451 [41E02960] -> Discovered new port with 
>> GUID:0x0005ad0000025043 LID range [0xB4,0xB4] of node:saguaro-23-5 HCA-1
>> Mar 20 14:25:49 025461 [41E02960] -> osm_report_notice: Reporting Generic 
>> Notice type:3 num:64 from LID:0x0092 
>> GID:0xfe80000000000000,0x0005ad0000024bbb
>> Mar 20 14:25:49 025466 [41E02960] -> Discovered new port with 
>> GUID:0x0005ad000002510b LID range [0xB5,0xB5] of node:saguaro-23-6 HCA-1
>> Mar 20 14:25:49 025470 [41E02960] -> osm_report_notice: Reporting Generic 
>> Notice type:3 num:64 from LID:0x0092 
>> GID:0xfe80000000000000,0x0005ad0000024bbb
>> Mar 20 14:25:49 025475 [41E02960] -> Discovered new port with 
>> GUID:0x0005ad0000024d47 LID range [0xB6,0xB6] of node:saguaro-23-7 HCA-1
>> Mar 20 14:25:49 025483 [41E02960] -> osm_report_notice: Reporting Generic 
>> Notice type:3 num:64 from LID:0x0092 
>> GID:0xfe80000000000000,0x0005ad0000024bbb
>> Mar 20 14:25:49 025488 [41E02960] -> Discovered new port with 
>> GUID:0x0005ad0000024d6b LID range [0xB8,0xB8] of node:saguaro-23-9 HCA-1
>> Mar 20 14:25:49 025491 [41E02960] -> osm_report_notice: Reporting Generic 
>> Notice type:3 num:64 from LID:0x0092 
>> GID:0xfe80000000000000,0x0005ad0000024bbb
>> Mar 20 14:25:49 025496 [41E02960] -> Discovered new port with 
>> GUID:0x0005ad0000024afb LID range [0xA5,0xA5] of node:saguaro-22-0 HCA-1
>> Mar 20 14:25:49 025500 [41E02960] -> osm_report_notice: Reporting Generic 
>> Notice type:3 num:64 from LID:0x0092 
>> GID:0xfe80000000000000,0x0005ad0000024bbb
>> Mar 20 14:25:49 025505 [41E02960] -> Discovered new port with 
>> GUID:0x0005ad000002511b LID range [0xA6,0xA6] of node:saguaro-22-1 HCA-1
>> Mar 20 14:25:49 025508 [41E02960] -> osm_report_notice: Reporting Generic 
>> Notice type:3 num:64 from LID:0x0092 
>> GID:0xfe80000000000000,0x0005ad0000024bbb
>> Mar 20 14:25:49 025513 [41E02960] -> Discovered new port with 
>> GUID:0x0005ad0000024c9b LID range [0xA7,0xA7] of node:saguaro-22-2 HCA-1
>> Mar 20 14:25:49 025517 [41E02960] -> osm_report_notice: Reporting Generic 
>> Notice type:3 num:64 from LID:0x0092 
>> GID:0xfe80000000000000,0x0005ad0000024bbb
>> Mar 20 14:25:49 025522 [41E02960] -> Discovered new port with 
>> GUID:0x0005ad000002498f LID range [0xA8,0xA8] of node:saguaro-22-3 HCA-1
>> Mar 20 14:25:49 025556 [41E02960] -> osm_report_notice: Reporting Generic 
>> Notice type:3 num:64 from LID:0x0092 
>> GID:0xfe80000000000000,0x0005ad0000024bbb
>> Mar 20 14:25:49 025562 [41E02960] -> Discovered new port with 
>> GUID:0x0005ad0000024977 LID range [0xA9,0xA9] of node:saguaro-22-4 HCA-1
>> Mar 20 14:25:49 025565 [41E02960] -> osm_report_notice: Reporting Generic 
>> Notice type:3 num:64 from LID:0x0092 
>> GID:0xfe80000000000000,0x0005ad0000024bbb
>> Mar 20 14:25:49 025570 [41E02960] -> Discovered new port with 
>> GUID:0x0005ad0000024feb LID range [0x153,0x153] of node:saguaro-22-5 HCA-1
>> Mar 20 14:25:49 025574 [41E02960] -> osm_report_notice: Reporting Generic 
>> Notice type:3 num:64 from LID:0x0092 
>> GID:0xfe80000000000000,0x0005ad0000024bbb
>> Mar 20 14:25:49 025579 [41E02960] -> Discovered new port with 
>> GUID:0x0005ad0000024d8b LID range [0xB7,0xB7] of node:saguaro-23-8 HCA-1
>> Mar 20 14:25:49 056324 [41E02960] -> osm_ucast_mgr_process: Min Hop Tables 
>> configured on all switches
>> Mar 20 14:25:49 126247 [4780B960] -> __osm_sm_mad_ctrl_rcv_callback: ERR 
>> 3111: Error status = 0x1C00
>> Mar 20 14:25:49 126356 [4780B960] -> SMP dump:
>>                 base_ver................0x1
>>                 mgmt_class..............0x81
>>                 class_ver...............0x1
>>                 method..................0x81 (SubnGetResp)
>>                 D bit...................0x1
>>                 status..................0x1C00
>>                 hop_ptr.................0x0
>>                 hop_count...............0x4
>>                 trans_id................0x7d165
>>                 attr_id.................0x15 (PortInfo)
>>                 resv....................0x0
>>                 attr_mod................0x13
>>                 m_key...................0x0000000000000000
>>                 dr_slid.................0xFFFF
>>                 dr_dlid.................0xFFFF
>> 
>>                 Initial path: [0][1][15][1][6]
>>                 Return path:  [0][9][18][D][3]
>>                 Reserved:     [0][0][0][0][0][0][0]
>> 
>>                 00 00 00 00 00 00 00 00   00 00 00 00 00 00 00 00
>> 
>>                 00 00 00 00 00 00 00 00   00 00 00 00 03 03 03 02
>> 
>>                 14 52 00 11 40 40 00 08   08 04 F2 40 00 00 00 00
>> 
>>                 00 00 00 00 00 88 00 00   00 00 00 00 00 00 00 00
>> 
>> Mar 20 14:25:49 126409 [4780B960] -> __osm_sm_mad_ctrl_rcv_callback: ERR 
>> 3111: Error status = 0x1C00
>> Mar 20 14:25:49 126442 [4780B960] -> SMP dump:
>>                 base_ver................0x1
>>                 mgmt_class..............0x81
>>                 class_ver...............0x1
>>                 method..................0x81 (SubnGetResp)
>>                 D bit...................0x1
>>                 status..................0x1C00
>>                 hop_ptr.................0x0
>>                 hop_count...............0x4
>>                 trans_id................0x7d166
>>                 attr_id.................0x15 (PortInfo)
>>                 resv....................0x0
>>                 attr_mod................0x16
>>                 m_key...................0x0000000000000000
>>                 dr_slid.................0xFFFF
>>                 dr_dlid.................0xFFFF
>> 
>>                 Initial path: [0][1][15][1][6]
>>                 Return path:  [0][9][18][D][3]
>>                 Reserved:     [0][0][0][0][0][0][0]
>> 
>>                 00 00 00 00 00 00 00 00   00 00 00 00 00 00 00 00
>> 
>>                 00 00 00 00 00 00 00 00   00 00 00 00 03 03 03 02
>> 
>>                 14 52 00 11 40 40 00 08   08 04 F2 40 00 00 00 00
>> 
>>                 00 00 00 00 00 88 00 00   00 00 00 00 00 00 00 00
>> 
>> Mar 20 14:25:49 126496 [4780B960] -> __osm_sm_mad_ctrl_rcv_callback: ERR 
>> 3111: Error status = 0x1C00
>> Mar 20 14:25:49 126489 [42803960] -> osm_pi_rcv_process_set: Received error 
>> status 0x1c for SetResp() during ACTIVE transition
>> Mar 20 14:25:49 126535 [4780B960] -> SMP dump:
>>                 base_ver................0x1
>>                 mgmt_class..............0x81
>>                 class_ver...............0x1
>>                 method..................0x81 (SubnGetResp)
>>                 D bit...................0x1
>>                 status..................0x1C00
>>                 hop_ptr.................0x0
>>                 hop_count...............0x4
>>                 trans_id................0x7d167
>>                 attr_id.................0x15 (PortInfo)
>>                 resv....................0x0
>>                 attr_mod................0x17
>>                 m_key...................0x0000000000000000
>>                 dr_slid.................0xFFFF
>>                 dr_dlid.................0xFFFF
>> 
>>                 Initial path: [0][1][15][1][6]
>>                 Return path:  [0][9][18][D][3]
>>                 Reserved:     [0][0][0][0][0][0][0]
>> 
>>                 00 00 00 00 00 00 00 00   00 00 00 00 00 00 00 00
>> 
>>                 00 00 00 00 00 00 00 00   00 00 00 00 03 03 03 02
>> 
>>                 14 52 00 11 40 40 00 08   08 04 F2 40 00 00 00 00
>> 
>>                 00 00 00 00 00 88 00 00   00 00 00 00 00 00 00 00
>> 
>> Mar 20 14:25:49 126526 [42803960] -> PortInfo dump:
>>                 port number.............0x13
>>                 node_guid...............0x0005ad00000281a7
>>                 port_guid...............0x0005ad00000281a7
>>                 m_key...................0x0000000000000000
>>                 subnet_prefix...........0x0000000000000000
>>                 base_lid................0x0
>>                 master_sm_base_lid......0x0
>>                 capability_mask.........0x0
>>                 diag_code...............0x0
>>                 m_key_lease_period......0x0
>>                 local_port_num..........0x3
>>                 link_width_enabled......0x3
>>                 link_width_supported....0x3
>>                 link_width_active.......0x2
>>                 link_speed_supported....0x1
>>                 port_state..............ACTIVE
>>                 state_info2.............0x52
>>                 m_key_protect_bits......0x0
>>                 lmc.....................0x0
>>                 link_speed..............0x11
>>                 mtu_smsl................0x40
>>                 vl_cap_init_type........0x40
>>                 vl_high_limit...........0x0
>>                 vl_arb_high_cap.........0x8
>>                 vl_arb_low_cap..........0x8
>>                 init_rep_mtu_cap........0x4
>>                 vl_stall_life...........0xF2
>>                 vl_enforce..............0x40
>>                 m_key_violations........0x0
>>                 p_key_violations........0x0
>>                 q_key_violations........0x0
>>                 guid_cap................0x0
>>                 client_reregister.......0x0
>>                 subnet_timeout..........0x0
>>                 resp_time_value.........0x0
>>                 error_threshold.........0x88
>> Mar 20 14:25:49 126567 [42803960] -> Capabilities Mask:
>> Mar 20 14:25:49 126613 [4780B960] -> __osm_sm_mad_ctrl_rcv_callback: ERR 
>> 3111: Error status = 0x1C00
>> Mar 20 14:25:49 126617 [42803960] -> osm_pi_rcv_process_set: Received error 
>> status 0x1c for SetResp() during ACTIVE transition
>> Mar 20 14:25:49 126658 [4780B960] -> SMP dump:
>>                 base_ver................0x1
>>                 mgmt_class..............0x81
>>                 class_ver...............0x1
>>                 method..................0x81 (SubnGetResp)
>>                 D bit...................0x1
>>                 status..................0x1C00
>>                 hop_ptr.................0x0
>>                 hop_count...............0x4
>>                 trans_id................0x7d168
>>                 attr_id.................0x15 (PortInfo)
>>                 resv....................0x0
>>                 attr_mod................0x18
>>                 m_key...................0x0000000000000000
>>                 dr_slid.................0xFFFF
>>                 dr_dlid.................0xFFFF
>> 
>>                 Initial path: [0][1][15][1][6]
>>                 Return path:  [0][9][18][D][3]
>>                 Reserved:     [0][0][0][0][0][0][0]
>> 
>>                 00 00 00 00 00 00 00 00   00 00 00 00 00 00 00 00
>> 
>>                 00 00 00 00 00 00 00 00   00 00 00 00 03 03 03 02
>> 
>>                 14 52 00 11 40 40 00 08   08 04 F2 40 00 00 00 00
>> 
>>                 00 00 00 00 00 88 00 00   00 00 00 00 00 00 00 00
>> 
>> Mar 20 14:25:49 126653 [42803960] -> PortInfo dump:
>>                 port number.............0x16
>>                 node_guid...............0x0005ad00000281a7
>>                 port_guid...............0x0005ad00000281a7
>>                 m_key...................0x0000000000000000
>>                 subnet_prefix...........0x0000000000000000
>>                 base_lid................0x0
>>                 master_sm_base_lid......0x0
>>                 capability_mask.........0x0
>>                 diag_code...............0x0
>>                 m_key_lease_period......0x0
>>                 local_port_num..........0x3
>>                 link_width_enabled......0x3
>>                 link_width_supported....0x3
>>                 link_width_active.......0x2
>>                 link_speed_supported....0x1
>>                 port_state..............ACTIVE
>>                 state_info2.............0x52
>>                 m_key_protect_bits......0x0
>>                 lmc.....................0x0
>>                 link_speed..............0x11
>>                 mtu_smsl................0x40
>>                 vl_cap_init_type........0x40
>>                 vl_high_limit...........0x0
>>                 vl_arb_high_cap.........0x8
>>                 vl_arb_low_cap..........0x8
>>                 init_rep_mtu_cap........0x4
>>                 vl_stall_life...........0xF2
>>                 vl_enforce..............0x40
>>                 m_key_violations........0x0
>>                 p_key_violations........0x0
>>                 q_key_violations........0x0
>>                 guid_cap................0x0
>>                 client_reregister.......0x0
>>                 subnet_timeout..........0x0
>>                 resp_time_value.........0x0
>>                 error_threshold.........0x88
>> Mar 20 14:25:49 126687 [42803960] -> Capabilities Mask:
>> Mar 20 14:25:49 126703 [43204960] -> osm_pi_rcv_process_set: Received error 
>> status 0x1c for SetResp() during ACTIVE transition
>> Mar 20 14:25:49 126709 [43204960] -> PortInfo dump:
>>                 port number.............0x18
>>                 node_guid...............0x0005ad00000281a7
>>                 port_guid...............0x0005ad00000281a7
>>                 m_key...................0x0000000000000000
>>                 subnet_prefix...........0x0000000000000000
>>                 base_lid................0x0
>>                 master_sm_base_lid......0x0
>>                 capability_mask.........0x0
>>                 diag_code...............0x0
>>                 m_key_lease_period......0x0
>>                 local_port_num..........0x3
>>                 link_width_enabled......0x3
>>                 link_width_supported....0x3
>>                 link_width_active.......0x2
>>                 link_speed_supported....0x1
>>                 port_state..............ACTIVE
>>                 state_info2.............0x52
>>                 m_key_protect_bits......0x0
>>                 lmc.....................0x0
>>                 link_speed..............0x11
>>                 mtu_smsl................0x40
>>                 vl_cap_init_type........0x40
>>                 vl_high_limit...........0x0
>>                 vl_arb_high_cap.........0x8
>>                 vl_arb_low_cap..........0x8
>>                 init_rep_mtu_cap........0x4
>>                 vl_stall_life...........0xF2
>>                 vl_enforce..............0x40
>>                 m_key_violations........0x0
>>                 p_key_violations........0x0
>>                 q_key_violations........0x0
>>                 guid_cap................0x0
>>                 client_reregister.......0x0
>>                 subnet_timeout..........0x0
>>                 resp_time_value.........0x0
>>                 error_threshold.........0x88
>> Mar 20 14:25:49 126744 [43204960] -> Capabilities Mask:
>> Mar 20 14:25:49 126765 [43C05960] -> osm_pi_rcv_process_set: Received error 
>> status 0x1c for SetResp() during ACTIVE transition
>> Mar 20 14:25:49 126770 [43C05960] -> PortInfo dump:
>>                 port number.............0x17
>>                 node_guid...............0x0005ad00000281a7
>>                 port_guid...............0x0005ad00000281a7
>>                 m_key...................0x0000000000000000
>>                 subnet_prefix...........0x0000000000000000
>>                 base_lid................0x0
>>                 master_sm_base_lid......0x0
>>                 capability_mask.........0x0
>>                 diag_code...............0x0
>>                 m_key_lease_period......0x0
>>                 local_port_num..........0x3
>>                 link_width_enabled......0x3
>>                 link_width_supported....0x3
>>                 link_width_active.......0x2
>>                 link_speed_supported....0x1
>>                 port_state..............ACTIVE
>>                 state_info2.............0x52
>>                 m_key_protect_bits......0x0
>>                 lmc.....................0x0
>>                 link_speed..............0x11
>>                 mtu_smsl................0x40
>>                 vl_cap_init_type........0x40
>>                 vl_high_limit...........0x0
>>                 vl_arb_high_cap.........0x8
>>                 vl_arb_low_cap..........0x8
>>                 init_rep_mtu_cap........0x4
>>                 vl_stall_life...........0xF2
>>                 vl_enforce..............0x40
>>                 m_key_violations........0x0
>>                 p_key_violations........0x0
>>                 q_key_violations........0x0
>>                 guid_cap................0x0
>>                 client_reregister.......0x0
>>                 subnet_timeout..........0x0
>>                 resp_time_value.........0x0
>>                 error_threshold.........0x88
>> Mar 20 14:25:49 126874 [43C05960] -> Capabilities Mask:
>> Mar 20 14:25:49 126975 [4780B960] -> __osm_sm_mad_ctrl_rcv_callback: ERR 
>> 3111: Error status = 0x1C00
>> Mar 20 14:25:49 127015 [4780B960] -> SMP dump:
>>                 base_ver................0x1
>>                 mgmt_class..............0x81
>>                 class_ver...............0x1
>>                 method..................0x81 (SubnGetResp)
>>                 D bit...................0x1
>>                 status..................0x1C00
>>                 hop_ptr.................0x0
>>                 hop_count...............0x4
>>                 trans_id................0x7d169
>>                 attr_id.................0x15 (PortInfo)
>>                 resv....................0x0
>>                 attr_mod................0x16
>>                 m_key...................0x0000000000000000
>>                 dr_slid.................0xFFFF
>>                 dr_dlid.................0xFFFF
>> 
>>                 Initial path: [0][1][13][1][6]
>>                 Return path:  [0][9][13][D][3]
>>                 Reserved:     [0][0][0][0][0][0][0]
>> 
>>                 00 00 00 00 00 00 00 00   00 00 00 00 00 00 00 00
>> 
>>                 00 00 00 00 00 00 00 00   00 00 00 00 03 03 03 02
>> 
>>                 14 52 00 11 40 40 00 08   08 04 F2 40 00 00 00 00
>> 
>>                 00 00 00 00 00 88 00 00   00 00 00 00 00 00 00 00
>> 
>> Mar 20 14:25:49 127066 [45A08960] -> osm_pi_rcv_process_set: Received error 
>> status 0x1c for SetResp() during ACTIVE transition
>> Mar 20 14:25:49 127072 [4780B960] -> __osm_sm_mad_ctrl_rcv_callback: ERR 
>> 3111: Error status = 0x1C00
>> Mar 20 14:25:49 127084 [45A08960] -> PortInfo dump:
>>                 port number.............0x16
>>                 node_guid...............0x0005ad00000281b3
>>                 port_guid...............0x0005ad00000281b3
>>                 m_key...................0x0000000000000000
>>                 subnet_prefix...........0x0000000000000000
>>                 base_lid................0x0
>>                 master_sm_base_lid......0x0
>>                 capability_mask.........0x0
>>                 diag_code...............0x0
>>                 m_key_lease_period......0x0
>>                 local_port_num..........0x3
>>                 link_width_enabled......0x3
>>                 link_width_supported....0x3
>>                 link_width_active.......0x2
>>                 link_speed_supported....0x1
>>                 port_state..............ACTIVE
>>                 state_info2.............0x52
>>                 m_key_protect_bits......0x0
>>                 lmc.....................0x0
>>                 link_speed..............0x11
>>                 mtu_smsl................0x40
>>                 vl_cap_init_type........0x40
>>                 vl_high_limit...........0x0
>>                 vl_arb_high_cap.........0x8
>>                 vl_arb_low_cap..........0x8
>>                 init_rep_mtu_cap........0x4
>>                 vl_stall_life...........0xF2
>>                 vl_enforce..............0x40
>>                 m_key_violations........0x0
>>                 p_key_violations........0x0
>>                 q_key_violations........0x0
>>                 guid_cap................0x0
>>                 client_reregister.......0x0
>>                 subnet_timeout..........0x0
>>                 resp_time_value.........0x0
>>                 error_threshold.........0x88
>> Mar 20 14:25:49 127103 [45A08960] -> Capabilities Mask:
>> Mar 20 14:25:49 127121 [4780B960] -> SMP dump:
>>                 base_ver................0x1
>>                 mgmt_class..............0x81
>>                 class_ver...............0x1
>>                 method..................0x81 (SubnGetResp)
>>                 D bit...................0x1
>>                 status..................0x1C00
>>                 hop_ptr.................0x0
>>                 hop_count...............0x4
>>                 trans_id................0x7d16a
>>                 attr_id.................0x15 (PortInfo)
>>                 resv....................0x0
>>                 attr_mod................0x17
>>                 m_key...................0x0000000000000000
>>                 dr_slid.................0xFFFF
>>                 dr_dlid.................0xFFFF
>> 
>>                 Initial path: [0][1][13][1][6]
>>                 Return path:  [0][9][13][D][3]
>>                 Reserved:     [0][0][0][0][0][0][0]
>> 
>>                 00 00 00 00 00 00 00 00   00 00 00 00 00 00 00 00
>> 
>>                 00 00 00 00 00 00 00 00   00 00 00 00 03 03 03 02
>> 
>>                 14 52 00 11 40 40 00 08   08 04 F2 40 00 00 00 00
>> 
>>                 00 00 00 00 00 88 00 00   00 00 00 00 00 00 00 00
>> 
>> Mar 20 14:25:49 127188 [4780B960] -> __osm_sm_mad_ctrl_rcv_callback: ERR 
>> 3111: Error status = 0x1C00
>> Mar 20 14:25:49 127220 [4780B960] -> SMP dump:
>>                 base_ver................0x1
>>                 mgmt_class..............0x81
>>                 class_ver...............0x1
>>                 method..................0x81 (SubnGetResp)
>>                 D bit...................0x1
>>                 status..................0x1C00
>>                 hop_ptr.................0x0
>>                 hop_count...............0x4
>>                 trans_id................0x7d16b
>>                 attr_id.................0x15 (PortInfo)
>>                 resv....................0x0
>>                 attr_mod................0x18
>>                 m_key...................0x0000000000000000
>>                 dr_slid.................0xFFFF
>>                 dr_dlid.................0xFFFF
>> 
>>                 Initial path: [0][1][13][1][6]
>>                 Return path:  [0][9][13][D][3]
>>                 Reserved:     [0][0][0][0][0][0][0]
>> 
>>                 00 00 00 00 00 00 00 00   00 00 00 00 00 00 00 00
>> 
>>                 00 00 00 00 00 00 00 00   00 00 00 00 03 03 03 02
>> 
>>                 14 52 00 11 40 40 00 08   08 04 F2 40 00 00 00 00
>> 
>>                 00 00 00 00 00 88 00 00   00 00 00 00 00 00 00 00
>> 
>> Mar 20 14:25:49 127326 [44606960] -> osm_pi_rcv_process_set: Received error 
>> status 0x1c for SetResp() during ACTIVE transition
>> Mar 20 14:25:49 127339 [44606960] -> PortInfo dump:
>>                 port number.............0x17
>>                 node_guid...............0x0005ad00000281b3
>>                 port_guid...............0x0005ad00000281b3
>>                 m_key...................0x0000000000000000
>>                 subnet_prefix...........0x0000000000000000
>>                 base_lid................0x0
>>                 master_sm_base_lid......0x0
>>                 capability_mask.........0x0
>>                 diag_code...............0x0
>>                 m_key_lease_period......0x0
>>                 local_port_num..........0x3
>>                 link_width_enabled......0x3
>>                 link_width_supported....0x3
>>                 link_width_active.......0x2
>>                 link_speed_supported....0x1
>>                 port_state..............ACTIVE
>>                 state_info2.............0x52
>>                 m_key_protect_bits......0x0
>>                 lmc.....................0x0
>>                 link_speed..............0x11
>>                 mtu_smsl................0x40
>>                 vl_cap_init_type........0x40
>>                 vl_high_limit...........0x0
>>                 vl_arb_high_cap.........0x8
>>                 vl_arb_low_cap..........0x8
>>                 init_rep_mtu_cap........0x4
>>                 vl_stall_life...........0xF2
>>                 vl_enforce..............0x40
>>                 m_key_violations........0x0
>>                 p_key_violations........0x0
>>                 q_key_violations........0x0
>>                 guid_cap................0x0
>>                 client_reregister.......0x0
>>                 subnet_timeout..........0x0
>>                 resp_time_value.........0x0
>>                 error_threshold.........0x88
>> Mar 20 14:25:49 127357 [44606960] -> Capabilities Mask:
>> Mar 20 14:25:49 127378 [45007960] -> osm_pi_rcv_process_set: Received error 
>> status 0x1c for SetResp() during ACTIVE transition
>> Mar 20 14:25:49 127397 [45007960] -> PortInfo dump:
>>                 port number.............0x18
>>                 node_guid...............0x0005ad00000281b3
>>                 port_guid...............0x0005ad00000281b3
>>                 m_key...................0x0000000000000000
>>                 subnet_prefix...........0x0000000000000000
>>                 base_lid................0x0
>>                 master_sm_base_lid......0x0
>>                 capability_mask.........0x0
>>                 diag_code...............0x0
>>                 m_key_lease_period......0x0
>>                 local_port_num..........0x3
>>                 link_width_enabled......0x3
>>                 link_width_supported....0x3
>>                 link_width_active.......0x2
>>                 link_speed_supported....0x1
>>                 port_state..............ACTIVE
>>                 state_info2.............0x52
>>                 m_key_protect_bits......0x0
>>                 lmc.....................0x0
>>                 link_speed..............0x11
>>                 mtu_smsl................0x40
>>                 vl_cap_init_type........0x40
>>                 vl_high_limit...........0x0
>>                 vl_arb_high_cap.........0x8
>>                 vl_arb_low_cap..........0x8
>>                 init_rep_mtu_cap........0x4
>>                 vl_stall_life...........0xF2
>>                 vl_enforce..............0x40
>>                 m_key_violations........0x0
>>                 p_key_violations........0x0
>>                 q_key_violations........0x0
>>                 guid_cap................0x0
>>                 client_reregister.......0x0
>>                 subnet_timeout..........0x0
>>                 resp_time_value.........0x0
>>                 error_threshold.........0x88
>> Mar 20 14:25:49 127410 [45007960] -> Capabilities Mask:
>> Mar 20 14:25:49 132961 [43204960] -> SUBNET UP
>> Mar 20 14:25:49 523879 [44606960] -> osm_ucast_mgr_process: Min Hop Tables 
>> configured on all switches
>> Mar 20 14:25:49 580522 [42803960] -> SUBNET UP
>> Mar 20 14:26:04 718574 [43C05960] -> __osm_trap_rcv_process_request: 
>> Received Generic Notice type:0x01 num:128 Producer:2 from LID:0x0152 
>> TID:0x0000000000000000
>> Mar 20 14:26:04 718819 [43C05960] -> osm_report_notice: Reporting Generic 
>> Notice type:1 num:128 from LID:0x0152 
>> GID:0xfe80000000000000,0x0005ad0000027c84
>> Mar 20 14:26:04 836781 [45A08960] -> __osm_trap_rcv_process_request: 
>> Received Generic Notice type:0x01 num:128 Producer:2 from LID:0x0152 
>> TID:0x0000000000000001
>> Mar 20 14:26:04 836881 [45A08960] -> osm_report_notice: Reporting Generic 
>> Notice type:1 num:128 from LID:0x0152 
>> GID:0xfe80000000000000,0x0005ad0000027c84
>> Mar 20 14:26:04 858762 [45A08960] -> __osm_trap_rcv_process_request: 
>> Received Generic Notice type:0x01 num:128 Producer:2 from LID:0x0152 
>> TID:0x0000000000000002
>> Mar 20 14:26:04 860242 [45A08960] -> osm_report_notice: Reporting Generic 
>> Notice type:1 num:128 from LID:0x0152 
>> GID:0xfe80000000000000,0x0005ad0000027c84
>> Mar 20 14:26:04 997451 [45007960] -> __osm_trap_rcv_process_request: 
>> Received Generic Notice type:0x01 num:128 Producer:2 from LID:0x0152 
>> TID:0x0000000000000003
>> Mar 20 14:26:04 997647 [45007960] -> osm_report_notice: Reporting Generic 
>> Notice type:1 num:128 from LID:0x0152 
>> GID:0xfe80000000000000,0x0005ad0000027c84
>> Mar 20 14:26:05 180722 [43204960] -> __osm_trap_rcv_process_request: 
>> Received Generic Notice type:0x01 num:128 Producer:2 from LID:0x0152 
>> TID:0x0000000000000004
>> Mar 20 14:26:05 180855 [43204960] -> osm_report_notice: Reporting Generic 
>> Notice type:1 num:128 from LID:0x0152 
>> GID:0xfe80000000000000,0x0005ad0000027c84
>> Mar 20 14:26:05 209122 [41401960] -> __osm_trap_rcv_process_request: 
>> Received Generic Notice type:0x01 num:128 Producer:2 from LID:0x0152 
>> TID:0x0000000000000005
>> Mar 20 14:26:05 209200 [41401960] -> osm_report_notice: Reporting Generic 
>> Notice type:1 num:128 from LID:0x0152 
>> GID:0xfe80000000000000,0x0005ad0000027c84
>> Mar 20 14:26:05 347419 [45A08960] -> __osm_trap_rcv_process_request: 
>> Received Generic Notice type:0x01 num:128 Producer:2 from LID:0x0152 
>> TID:0x0000000000000006
>> Mar 20 14:26:05 347488 [45A08960] -> osm_report_notice: Reporting Generic 
>> Notice type:1 num:128 from LID:0x0152 
>> GID:0xfe80000000000000,0x0005ad0000027c84
>> Mar 20 14:26:05 378670 [42803960] -> __osm_trap_rcv_process_request: 
>> Received Generic Notice type:0x01 num:128 Producer:2 from LID:0x0152 
>> TID:0x0000000000000007
>> Mar 20 14:26:05 378739 [42803960] -> osm_report_notice: Reporting Generic 
>> Notice type:1 num:128 from LID:0x0152 
>> GID:0xfe80000000000000,0x0005ad0000027c84
>> Mar 20 14:26:05 409112 [41401960] -> osm_report_notice: Reporting Generic 
>> Notice type:3 num:65 from LID:0x0092 
>> GID:0xfe80000000000000,0x0005ad0000024bbb
>> Mar 20 14:26:05 409121 [41401960] -> Removed port with 
>> GUID:0x0005ad00000249d3 LID range [0xB1,0xB1] of node:saguaro-23-2 HCA-1
>> Mar 20 14:26:05 443639 [41401960] -> osm_ucast_mgr_process: Min Hop Tables 
>> configured on all switches
>> Mar 20 14:26:05 483503 [45007960] -> __osm_trap_rcv_process_request: 
>> Received Generic Notice type:0x01 num:128 Producer:2 from LID:0x0152 
>> TID:0x0000000000000008
>> Mar 20 14:26:05 486002 [45007960] -> osm_report_notice: Reporting Generic 
>> Notice type:1 num:128 from LID:0x0152 
>> GID:0xfe80000000000000,0x0005ad0000027c84
>> Mar 20 14:26:05 499183 [44606960] -> SUBNET UP
>> Mar 20 14:26:05 499856 [43C05960] -> __osm_trap_rcv_process_request: 
>> Received Generic Notice type:0x01 num:128 Producer:2 from LID:0x0152 
>> TID:0x0000000000000009
>> Mar 20 14:26:05 499941 [43C05960] -> osm_report_notice: Reporting Generic 
>> Notice type:1 num:128 from LID:0x0152 
>> GID:0xfe80000000000000,0x0005ad0000027c84
>> Mar 20 14:26:05 521857 [43204960] -> __osm_trap_rcv_process_request: 
>> Received Generic Notice type:0x01 num:128 Producer:2 from LID:0x0152 
>> TID:0x000000000000000a
>> Mar 20 14:26:05 521971 [43204960] -> osm_report_notice: Reporting Generic 
>> Notice type:1 num:128 from LID:0x0152 
>> GID:0xfe80000000000000,0x0005ad0000027c84
>> Mar 20 14:26:05 532569 [41401960] -> __osm_trap_rcv_process_request: 
>> Received Generic Notice type:0x01 num:128 Producer:2 from LID:0x0152 
>> TID:0x000000000000000b
>> Mar 20 14:26:05 532624 [41401960] -> __osm_trap_rcv_process_request: ERR 
>> 3804: Received trap 11 times consecutively
>> Mar 20 14:26:05 633813 [43204960] -> __osm_trap_rcv_process_request: 
>> Received Generic Notice type:0x01 num:128 Producer:2 from LID:0x0152 
>> TID:0x000000000000000c
>> Mar 20 14:26:05 633869 [43204960] -> __osm_trap_rcv_process_request: ERR 
>> 3804: Received trap 12 times consecutively
>> Mar 20 14:26:05 655421 [41401960] -> __osm_trap_rcv_process_request: 
>> Received Generic Notice type:0x01 num:128 Producer:2 from LID:0x0152 
>> TID:0x000000000000000d
>> Mar 20 14:26:05 655501 [41401960] -> __osm_trap_rcv_process_request: ERR 
>> 3804: Received trap 13 times consecutively
>> Mar 20 14:26:05 702652 [42803960] -> __osm_trap_rcv_process_request: 
>> Received Generic Notice type:0x01 num:128 Producer:2 from LID:0x0152 
>> TID:0x000000000000000e
>> Mar 20 14:26:05 702745 [42803960] -> __osm_trap_rcv_process_request: ERR 
>> 3804: Received trap 14 times consecutively
>> Mar 20 14:26:05 817201 [43204960] -> osm_report_notice: Reporting Generic 
>> Notice type:3 num:65 from LID:0x0092 
>> GID:0xfe80000000000000,0x0005ad0000024bbb
>> Mar 20 14:26:05 817216 [43204960] -> Removed port with 
>> GUID:0x0005ad000002510b LID range [0xB5,0xB5] of node:saguaro-23-6 HCA-1
>> Mar 20 14:26:05 817235 [43204960] -> osm_report_notice: Reporting Generic 
>> Notice type:3 num:65 from LID:0x0092 
>> GID:0xfe80000000000000,0x0005ad0000024bbb
>> Mar 20 14:26:05 817241 [43204960] -> Removed port with 
>> GUID:0x0005ad0000024d47 LID range [0xB6,0xB6] of node:saguaro-23-7 HCA-1
>> Mar 20 14:26:05 817259 [43204960] -> osm_report_notice: Reporting Generic 
>> Notice type:3 num:65 from LID:0x0092 
>> GID:0xfe80000000000000,0x0005ad0000024bbb
>> Mar 20 14:26:05 817264 [43204960] -> Removed port with 
>> GUID:0x0005ad0000024d8b LID range [0xB7,0xB7] of node:saguaro-23-8 HCA-1
>> Mar 20 14:26:05 821322 [41E02960] -> osm_report_notice: Reporting Generic 
>> Notice type:3 num:64 from LID:0x0092 
>> GID:0xfe80000000000000,0x0005ad0000024bbb
>> Mar 20 14:26:05 821330 [41E02960] -> Discovered new port with 
>> GUID:0x0005ad00000249d3 LID range [0xB1,0xB1] of node:saguaro-23-2 HCA-1
>> Mar 20 14:26:05 847950 [45007960] -> __osm_trap_rcv_process_request: 
>> Received Generic Notice type:0x01 num:128 Producer:2 from LID:0x0152 
>> TID:0x000000000000000f
>> Mar 20 14:26:05 848031 [45007960] -> __osm_trap_rcv_process_request: ERR 
>> 3804: Received trap 15 times consecutively
>> Mar 20 14:26:05 852036 [41E02960] -> osm_ucast_mgr_process: Min Hop Tables 
>> configured on all switches
>> Mar 20 14:26:05 893954 [45007960] -> __osm_trap_rcv_process_request: 
>> Received Generic Notice type:0x01 num:128 Producer:2 from LID:0x0152 
>> TID:0x0000000000000010
>> Mar 20 14:26:05 894021 [45007960] -> __osm_trap_rcv_process_request: ERR 
>> 3804: Received trap 16 times consecutively
>> Mar 20 14:26:05 910489 [44606960] -> SUBNET UP
>> Mar 20 14:26:05 999993 [43C05960] -> __osm_trap_rcv_process_request: 
>> Received Generic Notice type:0x01 num:128 Producer:2 from LID:0x0152 
>> TID:0x0000000000000011
>> Mar 20 14:26:06 000039 [43C05960] -> __osm_trap_rcv_process_request: ERR 
>> 3804: Received trap 17 times consecutively
>> Mar 20 14:26:06 021880 [45A08960] -> __osm_trap_rcv_process_request: 
>> Received Generic Notice type:0x01 num:128 Producer:2 from LID:0x0152 
>> TID:0x0000000000000012
>> Mar 20 14:26:06 021970 [45A08960] -> __osm_trap_rcv_process_request: ERR 
>> 3804: Received trap 18 times consecutively
>> Mar 20 14:26:06 043912 [44606960] -> __osm_trap_rcv_process_request: 
>> Received Generic Notice type:0x01 num:128 Producer:2 from LID:0x0152 
>> TID:0x0000000000000013
>> Mar 20 14:26:06 044001 [44606960] -> __osm_trap_rcv_process_request: ERR 
>> 3804: Received trap 19 times consecutively
>> Mar 20 14:26:06 052878 [44606960] -> __osm_trap_rcv_process_request: 
>> Received Generic Notice type:0x01 num:128 Producer:2 from LID:0x0152 
>> TID:0x0000000000000014
>> Mar 20 14:26:06 052975 [44606960] -> __osm_trap_rcv_process_request: ERR 
>> 3804: Received trap 20 times consecutively
>> Mar 20 14:26:06 147560 [42803960] -> __osm_trap_rcv_process_request: 
>> Received Generic Notice type:0x01 num:128 Producer:2 from LID:0x0152 
>> TID:0x0000000000000015
>> Mar 20 14:26:06 147616 [42803960] -> __osm_trap_rcv_process_request: ERR 
>> 3804: Received trap 21 times consecutively
>> Mar 20 14:26:06 158945 [41401960] -> __osm_trap_rcv_process_request: 
>> Received Generic Notice type:0x01 num:128 Producer:2 from LID:0x0152 
>> TID:0x0000000000000016
>> Mar 20 14:26:06 158978 [41401960] -> __osm_trap_rcv_process_request: ERR 
>> 3804: Received trap 22 times consecutively
>> Mar 20 14:26:06 346046 [44606960] -> __osm_trap_rcv_process_request: 
>> Received Generic Notice type:0x01 num:128 Producer:2 from LID:0x0152 
>> TID:0x0000000000000017
>> Mar 20 14:26:06 346106 [44606960] -> __osm_trap_rcv_process_request: ERR 
>> 3804: Received trap 23 times consecutively
>> Mar 20 14:26:06 405311 [43204960] -> __osm_trap_rcv_process_request: 
>> Received Generic Notice type:0x01 num:128 Producer:2 from LID:0x0152 
>> TID:0x0000000000000018
>> Mar 20 14:26:06 405349 [43204960] -> __osm_trap_rcv_process_request: ERR 
>> 3804: Received trap 24 times consecutively
>> Mar 20 14:26:06 632882 [45007960] -> __osm_trap_rcv_process_request: 
>> Received Generic Notice type:0x01 num:128 Producer:2 from LID:0x0152 
>> TID:0x0000000000000019
>> Mar 20 14:26:06 632923 [45007960] -> __osm_trap_rcv_process_request: ERR 
>> 3804: Received trap 25 times consecutively
>> Mar 20 14:26:06 634031 [43C05960] -> __osm_trap_rcv_process_request: 
>> Received Generic Notice type:0x01 num:128 Producer:2 from LID:0x0148 
>> TID:0x0000000000000067
>> Mar 20 14:26:06 634110 [43C05960] -> osm_report_notice: Reporting Generic 
>> Notice type:1 num:128 from LID:0x0148 
>> GID:0xfe80000000000000,0x0005ad00000281b3
>> Mar 20 14:26:06 883831 [45007960] -> __osm_trap_rcv_process_request: 
>> Received Generic Notice type:0x01 num:128 Producer:2 from LID:0x0152 
>> TID:0x000000000000001a
>> Mar 20 14:26:06 883879 [45007960] -> __osm_trap_rcv_process_request: ERR 
>> 3804: Received trap 26 times consecutively
>> Mar 20 14:26:06 885475 [43C05960] -> __osm_trap_rcv_process_request: 
>> Received Generic Notice type:0x01 num:128 Producer:2 from LID:0x0148 
>> TID:0x0000000000000068
>> Mar 20 14:26:06 885560 [43C05960] -> osm_report_notice: Reporting Generic 
>> Notice type:1 num:128 from LID:0x0148 
>> GID:0xfe80000000000000,0x0005ad00000281b3
>> Mar 20 14:26:06 982877 [43204960] -> __osm_trap_rcv_process_request: 
>> Received Generic Notice type:0x01 num:128 Producer:2 from LID:0x0152 
>> TID:0x000000000000001b
>> Mar 20 14:26:06 982926 [43204960] -> __osm_trap_rcv_process_request: ERR 
>> 3804: Received trap 27 times consecutively
>> Mar 20 14:26:06 992809 [41E02960] -> __osm_trap_rcv_process_request: 
>> Received Generic Notice type:0x01 num:128 Producer:2 from LID:0x0148 
>> TID:0x0000000000000069
>> Mar 20 14:26:06 992871 [41E02960] -> osm_report_notice: Reporting Generic 
>> Notice type:1 num:128 from LID:0x0148 
>> GID:0xfe80000000000000,0x0005ad00000281b3
>> Mar 20 14:26:06 992909 [41E02960] -> __osm_trap_rcv_process_request: 
>> Received Generic Notice type:0x01 num:128 Producer:2 from LID:0x0152 
>> TID:0x000000000000001c
>> Mar 20 14:26:06 992943 [41E02960] -> __osm_trap_rcv_process_request: ERR 
>> 3804: Received trap 28 times consecutively
>> Mar 20 14:26:06 993058 [41E02960] -> osm_report_notice: Reporting Generic 
>> Notice type:3 num:64 from LID:0x0092 
>> GID:0xfe80000000000000,0x0005ad0000024bbb
>> Mar 20 14:26:06 993065 [41E02960] -> Discovered new port with 
>> GUID:0x0005ad000002510b LID range [0xB5,0xB5] of node:saguaro-23-6 HCA-1
>> Mar 20 14:26:06 993069 [41E02960] -> osm_report_notice: Reporting Generic 
>> Notice type:3 num:64 from LID:0x0092 
>> GID:0xfe80000000000000,0x0005ad0000024bbb
>> Mar 20 14:26:06 993074 [41E02960] -> Discovered new port with 
>> GUID:0x0005ad0000024d47 LID range [0xB6,0xB6] of node:saguaro-23-7 HCA-1
>> Mar 20 14:26:07 023890 [41E02960] -> osm_ucast_mgr_process: Min Hop Tables 
>> configured on all switches
>> Mar 20 14:26:07 085081 [41E02960] -> SUBNET UP
>> Mar 20 14:26:07 348105 [45A08960] -> __osm_trap_rcv_process_request: 
>> Received Generic Notice type:0x01 num:128 Producer:2 from LID:0x0152 
>> TID:0x000000000000001d
>> Mar 20 14:26:07 348218 [45A08960] -> __osm_trap_rcv_process_request: ERR 
>> 3804: Received trap 29 times consecutively
>> Mar 20 14:26:07 348958 [45A08960] -> __osm_trap_rcv_process_request: 
>> Received Generic Notice type:0x01 num:128 Producer:2 from LID:0x0148 
>> TID:0x000000000000006a
>> Mar 20 14:26:07 349041 [45A08960] -> osm_report_notice: Reporting Generic 
>> Notice type:1 num:128 from LID:0x0148 
>> GID:0xfe80000000000000,0x0005ad00000281b3
>> Mar 20 14:26:07 540871 [41401960] -> __osm_trap_rcv_process_request: 
>> Received Generic Notice type:0x01 num:128 Producer:2 from LID:0x0148 
>> TID:0x000000000000006b
>> Mar 20 14:26:07 540983 [41401960] -> osm_report_notice: Reporting Generic 
>> Notice type:1 num:128 from LID:0x0148 
>> GID:0xfe80000000000000,0x0005ad00000281b3
>> Mar 20 14:26:07 541063 [43204960] -> __osm_trap_rcv_process_request: 
>> Received Generic Notice type:0x01 num:128 Producer:2 from LID:0x001B 
>> TID:0x000000000000007c
>> Mar 20 14:26:07 541131 [43204960] -> osm_report_notice: Reporting Generic 
>> Notice type:1 num:128 from LID:0x001B 
>> GID:0xfe80000000000000,0x0005ad00000281a7
>> Mar 20 14:26:07 585394 [43C05960] -> __osm_trap_rcv_process_request: 
>> Received Generic Notice type:0x01 num:128 Producer:2 from LID:0x0148 
>> TID:0x000000000000006c
>> Mar 20 14:26:07 585464 [43C05960] -> osm_report_notice: Reporting Generic 
>> Notice type:1 num:128 from LID:0x0148 
>> GID:0xfe80000000000000,0x0005ad00000281b3
>> Mar 20 14:26:07 607406 [45A08960] -> __osm_trap_rcv_process_request: 
>> Received Generic Notice type:0x01 num:128 Producer:2 from LID:0x0148 
>> TID:0x000000000000006d
>> Mar 20 14:26:07 607486 [45A08960] -> osm_report_notice: Reporting Generic 
>> Notice type:1 num:128 from LID:0x0148 
>> GID:0xfe80000000000000,0x0005ad00000281b3
>> Mar 20 14:26:07 850410 [42803960] -> __osm_trap_rcv_process_request: 
>> Received Generic Notice type:0x01 num:128 Producer:2 from LID:0x0148 
>> TID:0x000000000000006e
>> Mar 20 14:26:07 850483 [42803960] -> osm_report_notice: Reporting Generic 
>> Notice type:1 num:128 from LID:0x0148 
>> GID:0xfe80000000000000,0x0005ad00000281b3
>> Mar 20 14:26:07 956365 [41E02960] -> __osm_trap_rcv_process_request: 
>> Received Generic Notice type:0x01 num:128 Producer:2 from LID:0x001B 
>> TID:0x000000000000007d
>> Mar 20 14:26:07 956404 [42803960] -> osm_report_notice: Reporting Generic 
>> Notice type:3 num:64 from LID:0x0092 
>> GID:0xfe80000000000000,0x0005ad0000024bbb
>> Mar 20 14:26:07 956413 [42803960] -> Discovered new port with 
>> GUID:0x0005ad0000024d8b LID range [0xB7,0xB7] of node:saguaro-23-8 HCA-1
>> Mar 20 14:26:07 987136 [42803960] -> osm_ucast_mgr_process: Min Hop Tables 
>> configured on all switches
>> Mar 20 14:26:08 018887 [41E02960] -> osm_report_notice: Reporting Generic 
>> Notice type:1 num:128 from LID:0x001B 
>> GID:0xfe80000000000000,0x0005ad00000281a7
>> Mar 20 14:26:08 032634 [4780B960] -> __osm_sm_mad_ctrl_rcv_callback: ERR 
>> 3111: Error status = 0x1C00
>> Mar 20 14:26:08 032679 [4780B960] -> SMP dump:
>>                 base_ver................0x1
>>                 mgmt_class..............0x81
>>                 class_ver...............0x1
>>                 method..................0x81 (SubnGetResp)
>>                 D bit...................0x1
>>                 status..................0x1C00
>>                 hop_ptr.................0x0
>>                 hop_count...............0x4
>>                 trans_id................0x813ce
>>                 attr_id.................0x15 (PortInfo)
>>                 resv....................0x0
>>                 attr_mod................0x16
>>                 m_key...................0x0000000000000000
>>                 dr_slid.................0xFFFF
>>                 dr_dlid.................0xFFFF
>> 
>>                 Initial path: [0][1][12][4][5]
>>                 Return path:  [0][9][14][D][5]
>>                 Reserved:     [0][0][0][0][0][0][0]
>> 
>>                 00 00 00 00 00 00 00 00   00 00 00 00 00 00 00 00
>> 
>>                 00 00 00 00 00 00 00 00   00 00 00 00 05 03 03 02
>> 
>>                 14 52 00 11 40 40 00 08   08 04 F2 40 00 00 00 00
>> 
>>                 00 00 00 00 00 88 00 00   00 00 00 00 00 00 00 00
>> 
>> Mar 20 14:26:08 032749 [41E02960] -> osm_pi_rcv_process_set: Received error 
>> status 0x1c for SetResp() during ACTIVE transition
>> Mar 20 14:26:08 032757 [41E02960] -> PortInfo dump:
>>                 port number.............0x16
>>                 node_guid...............0x0005ad00000281b3
>>                 port_guid...............0x0005ad00000281b3
>>                 m_key...................0x0000000000000000
>>                 subnet_prefix...........0x0000000000000000
>>                 base_lid................0x0
>>                 master_sm_base_lid......0x0
>>                 capability_mask.........0x0
>>                 diag_code...............0x0
>>                 m_key_lease_period......0x0
>>                 local_port_num..........0x5
>>                 link_width_enabled......0x3
>>                 link_width_supported....0x3
>>                 link_width_active.......0x2
>>                 link_speed_supported....0x1
>>                 port_state..............ACTIVE
>>                 state_info2.............0x52
>>                 m_key_protect_bits......0x0
>>                 lmc.....................0x0
>>                 link_speed..............0x11
>>                 mtu_smsl................0x40
>>                 vl_cap_init_type........0x40
>>                 vl_high_limit...........0x0
>>                 vl_arb_high_cap.........0x8
>>                 vl_arb_low_cap..........0x8
>>                 init_rep_mtu_cap........0x4
>>                 vl_stall_life...........0xF2
>>                 vl_enforce..............0x40
>>                 m_key_violations........0x0
>>                 p_key_violations........0x0
>>                 q_key_violations........0x0
>>                 guid_cap................0x0
>>                 client_reregister.......0x0
>>                 subnet_timeout..........0x0
>>                 resp_time_value.........0x0
>>                 error_threshold.........0x88
>> Mar 20 14:26:08 032774 [41E02960] -> Capabilities Mask:
>> Mar 20 14:26:08 033119 [4780B960] -> __osm_sm_mad_ctrl_rcv_callback: ERR 
>> 3111: Error status = 0x1C00
>> Mar 20 14:26:08 033154 [4780B960] -> SMP dump:
>>                 base_ver................0x1
>>                 mgmt_class..............0x81
>>                 class_ver...............0x1
>>                 method..................0x81 (SubnGetResp)
>>                 D bit...................0x1
>>                 status..................0x1C00
>>                 hop_ptr.................0x0
>>                 hop_count...............0x4
>>                 trans_id................0x813cf
>>                 attr_id.................0x15 (PortInfo)
>>                 resv....................0x0
>>                 attr_mod................0x17
>>                 m_key...................0x0000000000000000
>>                 dr_slid.................0xFFFF
>>                 dr_dlid.................0xFFFF
>> 
>>                 Initial path: [0][1][12][4][5]
>>                 Return path:  [0][9][14][D][5]
>>                 Reserved:     [0][0][0][0][0][0][0]
>> 
>>                 00 00 00 00 00 00 00 00   00 00 00 00 00 00 00 00
>> 
>>                 00 00 00 00 00 00 00 00   00 00 00 00 05 03 03 02
>> 
>>                 14 52 00 11 40 40 00 08   08 04 F2 40 00 00 00 00
>> 
>>                 00 00 00 00 00 88 00 00   00 00 00 00 00 00 00 00
>> 
>> Mar 20 14:26:08 033202 [43C05960] -> osm_pi_rcv_process_set: Received error 
>> status 0x1c for SetResp() during ACTIVE transition
>> Mar 20 14:26:08 033213 [43C05960] -> PortInfo dump:
>>                 port number.............0x17
>>                 node_guid...............0x0005ad00000281b3
>>                 port_guid...............0x0005ad00000281b3
>>                 m_key...................0x0000000000000000
>>                 subnet_prefix...........0x0000000000000000
>>                 base_lid................0x0
>>                 master_sm_base_lid......0x0
>>                 capability_mask.........0x0
>>                 diag_code...............0x0
>>                 m_key_lease_period......0x0
>>                 local_port_num..........0x5
>>                 link_width_enabled......0x3
>>                 link_width_supported....0x3
>>                 link_width_active.......0x2
>>                 link_speed_supported....0x1
>>                 port_state..............ACTIVE
>>                 state_info2.............0x52
>>                 m_key_protect_bits......0x0
>>                 lmc.....................0x0
>>                 link_speed..............0x11
>>                 mtu_smsl................0x40
>>                 vl_cap_init_type........0x40
>>                 vl_high_limit...........0x0
>>                 vl_arb_high_cap.........0x8
>>                 vl_arb_low_cap..........0x8
>>                 init_rep_mtu_cap........0x4
>>                 vl_stall_life...........0xF2
>>                 vl_enforce..............0x40
>>                 m_key_violations........0x0
>>                 p_key_violations........0x0
>>                 q_key_violations........0x0
>>                 guid_cap................0x0
>>                 client_reregister.......0x0
>>                 subnet_timeout..........0x0
>>                 resp_time_value.........0x0
>>                 error_threshold.........0x88
>> Mar 20 14:26:08 033231 [43C05960] -> Capabilities Mask:
>> Mar 20 14:26:08 038497 [45A08960] -> SUBNET UP
>> Mar 20 14:26:08 055480 [43C05960] -> __osm_trap_rcv_process_request: 
>> Received Generic Notice type:0x01 num:128 Producer:2 from LID:0x001B 
>> TID:0x000000000000007e
>> Mar 20 14:26:08 055587 [43C05960] -> osm_report_notice: Reporting Generic 
>> Notice type:1 num:128 from LID:0x001B 
>> GID:0xfe80000000000000,0x0005ad00000281a7
>> Mar 20 14:26:08 372288 [43204960] -> __osm_trap_rcv_process_request: 
>> Received Generic Notice type:0x01 num:128 Producer:2 from LID:0x001B 
>> TID:0x000000000000007f
>> Mar 20 14:26:08 376158 [42803960] -> osm_ucast_mgr_process: Min Hop Tables 
>> configured on all switches
>> Mar 20 14:26:08 418607 [44606960] -> __osm_trap_rcv_process_request: 
>> Received Generic Notice type:0x01 num:128 Producer:2 from LID:0x001B 
>> TID:0x0000000000000080
>> Mar 20 14:26:08 420668 [43204960] -> osm_report_notice: Reporting Generic 
>> Notice type:1 num:128 from LID:0x001B 
>> GID:0xfe80000000000000,0x0005ad00000281a7
>> Mar 20 14:26:08 420714 [44606960] -> osm_report_notice: Reporting Generic 
>> Notice type:1 num:128 from LID:0x001B 
>> GID:0xfe80000000000000,0x0005ad00000281a7
>> Mar 20 14:26:08 430046 [4780B960] -> __osm_sm_mad_ctrl_rcv_callback: ERR 
>> 3111: Error status = 0x1C00
>> Mar 20 14:26:08 430147 [4780B960] -> SMP dump:
>>                 base_ver................0x1
>>                 mgmt_class..............0x81
>>                 class_ver...............0x1
>>                 method..................0x81 (SubnGetResp)
>>                 D bit...................0x1
>>                 status..................0x1C00
>>                 hop_ptr.................0x0
>>                 hop_count...............0x4
>>                 trans_id................0x820fa
>>                 attr_id.................0x15 (PortInfo)
>>                 resv....................0x0
>>                 attr_mod................0x16
>>                 m_key...................0x0000000000000000
>>                 dr_slid.................0xFFFF
>>                 dr_dlid.................0xFFFF
>> 
>>                 Initial path: [0][1][15][1][4]
>>                 Return path:  [0][9][18][D][1]
>>                 Reserved:     [0][0][0][0][0][0][0]
>> 
>>                 00 00 00 00 00 00 00 00   00 00 00 00 00 00 00 00
>> 
>>                 00 00 00 00 00 00 00 00   00 00 00 00 01 03 03 02
>> 
>>                 14 52 00 11 40 40 00 08   08 04 F2 40 00 00 00 00
>> 
>>                 00 00 00 00 00 88 00 00   00 00 00 00 00 00 00 00
>> 
>> Mar 20 14:26:08 430236 [4780B960] -> __osm_sm_mad_ctrl_rcv_callback: ERR 
>> 3111: Error status = 0x1C00
>> Mar 20 14:26:08 430236 [43C05960] -> osm_pi_rcv_process_set: Received error 
>> status 0x1c for SetResp() during ACTIVE transition
>> Mar 20 14:26:08 430267 [4780B960] -> SMP dump:
>>                 base_ver................0x1
>>                 mgmt_class..............0x81
>>                 class_ver...............0x1
>>                 method..................0x81 (SubnGetResp)
>>                 D bit...................0x1
>>                 status..................0x1C00
>>                 hop_ptr.................0x0
>>                 hop_count...............0x4
>>                 trans_id................0x820fb
>>                 attr_id.................0x15 (PortInfo)
>>                 resv....................0x0
>>                 attr_mod................0x18
>>                 m_key...................0x0000000000000000
>>                 dr_slid.................0xFFFF
>>                 dr_dlid.................0xFFFF
>> 
>>                 Initial path: [0][1][12][1][6]
>>                 Return path:  [0][9][14][D][3]
>>                 Reserved:     [0][0][0][0][0][0][0]
>> 
>>                 00 00 00 00 00 00 00 00   00 00 00 00 00 00 00 00
>> 
>>                 00 00 00 00 00 00 00 00   00 00 00 00 03 03 03 02
>> 
>>                 14 52 00 11 40 40 00 08   08 04 F2 40 00 00 00 00
>> 
>>                 00 00 00 00 00 88 00 00   00 00 00 00 00 00 00 00
>> 
>> Mar 20 14:26:08 430262 [43C05960] -> PortInfo dump:
>>                 port number.............0x16
>>                 node_guid...............0x0005ad00000281a7
>>                 port_guid...............0x0005ad00000281a7
>>                 m_key...................0x0000000000000000
>>                 subnet_prefix...........0x0000000000000000
>>                 base_lid................0x0
>>                 master_sm_base_lid......0x0
>>                 capability_mask.........0x0
>>                 diag_code...............0x0
>>                 m_key_lease_period......0x0
>>                 local_port_num..........0x1
>>                 link_width_enabled......0x3
>>                 link_width_supported....0x3
>>                 link_width_active.......0x2
>>                 link_speed_supported....0x1
>>                 port_state..............ACTIVE
>>                 state_info2.............0x52
>>                 m_key_protect_bits......0x0
>>                 lmc.....................0x0
>>                 link_speed..............0x11
>>                 mtu_smsl................0x40
>>                 vl_cap_init_type........0x40
>>                 vl_high_limit...........0x0
>>                 vl_arb_high_cap.........0x8
>>                 vl_arb_low_cap..........0x8
>>                 init_rep_mtu_cap........0x4
>>                 vl_stall_life...........0xF2
>>                 vl_enforce..............0x40
>>                 m_key_violations........0x0
>>                 p_key_violations........0x0
>>                 q_key_violations........0x0
>>                 guid_cap................0x0
>>                 client_reregister.......0x0
>>                 subnet_timeout..........0x0
>>                 resp_time_value.........0x0
>>                 error_threshold.........0x88
>> Mar 20 14:26:08 430286 [43C05960] -> Capabilities Mask:
>> Mar 20 14:26:08 430350 [43C05960] -> osm_pi_rcv_process_set: Received error 
>> status 0x1c for SetResp() during ACTIVE transition
>> Mar 20 14:26:08 430362 [43C05960] -> PortInfo dump:
>>                 port number.............0x18
>>                 node_guid...............0x0005ad00000281b3
>>                 port_guid...............0x0005ad00000281b3
>>                 m_key...................0x0000000000000000
>>                 subnet_prefix...........0x0000000000000000
>>                 base_lid................0x0
>>                 master_sm_base_lid......0x0
>>                 capability_mask.........0x0
>>                 diag_code...............0x0
>>                 m_key_lease_period......0x0
>>                 local_port_num..........0x3
>>                 link_width_enabled......0x3
>>                 link_width_supported....0x3
>>                 link_width_active.......0x2
>>                 link_speed_supported....0x1
>>                 port_state..............ACTIVE
>>                 state_info2.............0x52
>>                 m_key_protect_bits......0x0
>>                 lmc.....................0x0
>>                 link_speed..............0x11
>>                 mtu_smsl................0x40
>>                 vl_cap_init_type........0x40
>>                 vl_high_limit...........0x0
>>                 vl_arb_high_cap.........0x8
>>                 vl_arb_low_cap..........0x8
>>                 init_rep_mtu_cap........0x4
>>                 vl_stall_life...........0xF2
>>                 vl_enforce..............0x40
>>                 m_key_violations........0x0
>>                 p_key_violations........0x0
>>                 q_key_violations........0x0
>>                 guid_cap................0x0
>>                 client_reregister.......0x0
>>                 subnet_timeout..........0x0
>>                 resp_time_value.........0x0
>>                 error_threshold.........0x88
>> Mar 20 14:26:08 430375 [43C05960] -> Capabilities Mask:
>> Mar 20 14:26:08 435317 [43C05960] -> SUBNET UP
>> Mar 20 14:26:08 583769 [41401960] -> __osm_trap_rcv_process_request: 
>> Received Generic Notice type:0x01 num:128 Producer:2 from LID:0x0152 
>> TID:0x000000000000001e
>> Mar 20 14:26:08 583903 [41401960] -> __osm_trap_rcv_process_request: ERR 
>> 3804: Received trap 30 times consecutively
>> Mar 20 14:26:08 854841 [41E02960] -> osm_ucast_mgr_process: Min Hop Tables 
>> configured on all switches
>> Mar 20 14:26:08 913273 [4780B960] -> __osm_sm_mad_ctrl_rcv_callback: ERR 
>> 3111: Error status = 0x1C00
>> Mar 20 14:26:08 913349 [4780B960] -> SMP dump:
>>                 base_ver................0x1
>>                 mgmt_class..............0x81
>>                 class_ver...............0x1
>>                 method..................0x81 (SubnGetResp)
>>                 D bit...................0x1
>>                 status..................0x1C00
>>                 hop_ptr.................0x0
>>                 hop_count...............0x4
>>                 trans_id................0x82e32
>>                 attr_id.................0x15 (PortInfo)
>>                 resv....................0x0
>>                 attr_mod................0x13
>>                 m_key...................0x0000000000000000
>>                 dr_slid.................0xFFFF
>>                 dr_dlid.................0xFFFF
>> 
>>                 Initial path: [0][1][17][2][5]
>>                 Return path:  [0][9][14][E][2]
>>                 Reserved:     [0][0][0][0][0][0][0]
>> 
>>                 00 00 00 00 00 00 00 00   00 00 00 00 00 00 00 00
>> 
>>                 00 00 00 00 00 00 00 00   00 00 00 00 02 03 03 02
>> 
>>                 14 52 00 11 40 40 00 08   08 04 F2 40 00 00 00 00
>> 
>>                 00 00 00 00 00 88 00 00   00 00 00 00 00 00 00 00
>> 
>> Mar 20 14:26:08 913415 [45A08960] -> osm_pi_rcv_process_set: Received error 
>> status 0x1c for SetResp() during ACTIVE transition
>> Mar 20 14:26:08 913432 [45A08960] -> PortInfo dump:
>>                 port number.............0x13
>>                 node_guid...............0x0005ad00000281a7
>>                 port_guid...............0x0005ad00000281a7
>>                 m_key...................0x0000000000000000
>>                 subnet_prefix...........0x0000000000000000
>>                 base_lid................0x0
>>                 master_sm_base_lid......0x0
>>                 capability_mask.........0x0
>>                 diag_code...............0x0
>>                 m_key_lease_period......0x0
>>                 local_port_num..........0x2
>>                 link_width_enabled......0x3
>>                 link_width_supported....0x3
>>                 link_width_active.......0x2
>>                 link_speed_supported....0x1
>>                 port_state..............ACTIVE
>>                 state_info2.............0x52
>>                 m_key_protect_bits......0x0
>>                 lmc.....................0x0
>>                 link_speed..............0x11
>>                 mtu_smsl................0x40
>>                 vl_cap_init_type........0x40
>>                 vl_high_limit...........0x0
>>                 vl_arb_high_cap.........0x8
>>                 vl_arb_low_cap..........0x8
>>                 init_rep_mtu_cap........0x4
>>                 vl_stall_life...........0xF2
>>                 vl_enforce..............0x40
>>                 m_key_violations........0x0
>>                 p_key_violations........0x0
>>                 q_key_violations........0x0
>>                 guid_cap................0x0
>>                 client_reregister.......0x0
>>                 subnet_timeout..........0x0
>>                 resp_time_value.........0x0
>>                 error_threshold.........0x88
>> Mar 20 14:26:08 913449 [45A08960] -> Capabilities Mask:
>> Mar 20 14:26:08 913598 [4780B960] -> __osm_sm_mad_ctrl_rcv_callback: ERR 
>> 3111: Error status = 0x1C00
>> Mar 20 14:26:08 913676 [4780B960] -> SMP dump:
>>                 base_ver................0x1
>>                 mgmt_class..............0x81
>>                 class_ver...............0x1
>>                 method..................0x81 (SubnGetResp)
>>                 D bit...................0x1
>>                 status..................0x1C00
>>                 hop_ptr.................0x0
>>                 hop_count...............0x4
>>                 trans_id................0x82e33
>>                 attr_id.................0x15 (PortInfo)
>>                 resv....................0x0
>>                 attr_mod................0x17
>>                 m_key...................0x0000000000000000
>>                 dr_slid.................0xFFFF
>>                 dr_dlid.................0xFFFF
>> 
>>                 Initial path: [0][1][17][2][5]
>>                 Return path:  [0][9][14][E][2]
>>                 Reserved:     [0][0][0][0][0][0][0]
>> 
>>                 00 00 00 00 00 00 00 00   00 00 00 00 00 00 00 00
>> 
>>                 00 00 00 00 00 00 00 00   00 00 00 00 02 03 03 02
>> 
>>                 14 52 00 11 40 40 00 08   08 04 F2 40 00 00 00 00
>> 
>>                 00 00 00 00 00 88 00 00   00 00 00 00 00 00 00 00
>> 
>> Mar 20 14:26:08 913727 [43C05960] -> osm_pi_rcv_process_set: Received error 
>> status 0x1c for SetResp() during ACTIVE transition
>> Mar 20 14:26:08 913732 [4780B960] -> __osm_sm_mad_ctrl_rcv_callback: ERR 
>> 3111: Error status = 0x1C00
>> Mar 20 14:26:08 913734 [43C05960] -> PortInfo dump:
>>                 port number.............0x17
>>                 node_guid...............0x0005ad00000281a7
>>                 port_guid...............0x0005ad00000281a7
>>                 m_key...................0x0000000000000000
>>                 subnet_prefix...........0x0000000000000000
>>                 base_lid................0x0
>>                 master_sm_base_lid......0x0
>>                 capability_mask.........0x0
>>                 diag_code...............0x0
>>                 m_key_lease_period......0x0
>>                 local_port_num..........0x2
>>                 link_width_enabled......0x3
>>                 link_width_supported....0x3
>>                 link_width_active.......0x2
>>                 link_speed_supported....0x1
>>                 port_state..............ACTIVE
>>                 state_info2.............0x52
>>                 m_key_protect_bits......0x0
>>                 lmc.....................0x0
>>                 link_speed..............0x11
>>                 mtu_smsl................0x40
>>                 vl_cap_init_type........0x40
>>                 vl_high_limit...........0x0
>>                 vl_arb_high_cap.........0x8
>>                 vl_arb_low_cap..........0x8
>>                 init_rep_mtu_cap........0x4
>>                 vl_stall_life...........0xF2
>>                 vl_enforce..............0x40
>>                 m_key_violations........0x0
>>                 p_key_violations........0x0
>>                 q_key_violations........0x0
>>                 guid_cap................0x0
>>                 client_reregister.......0x0
>>                 subnet_timeout..........0x0
>>                 resp_time_value.........0x0
>>                 error_threshold.........0x88
>> Mar 20 14:26:08 913752 [43C05960] -> Capabilities Mask:
>> Mar 20 14:26:08 913766 [4780B960] -> SMP dump:
>>                 base_ver................0x1
>>                 mgmt_class..............0x81
>>                 class_ver...............0x1
>>                 method..................0x81 (SubnGetResp)
>>                 D bit...................0x1
>>                 status..................0x1C00
>>                 hop_ptr.................0x0
>>                 hop_count...............0x4
>>                 trans_id................0x82e34
>>                 attr_id.................0x15 (PortInfo)
>>                 resv....................0x0
>>                 attr_mod................0x18
>>                 m_key...................0x0000000000000000
>>                 dr_slid.................0xFFFF
>>                 dr_dlid.................0xFFFF
>> 
>>                 Initial path: [0][1][17][2][5]
>>                 Return path:  [0][9][14][E][2]
>>                 Reserved:     [0][0][0][0][0][0][0]
>> 
>>                 00 00 00 00 00 00 00 00   00 00 00 00 00 00 00 00
>> 
>>                 00 00 00 00 00 00 00 00   00 00 00 00 02 03 03 02
>> 
>>                 14 52 00 11 40 40 00 08   08 04 F2 40 00 00 00 00
>> 
>>                 00 00 00 00 00 88 00 00   00 00 00 00 00 00 00 00
>> 
>> Mar 20 14:26:08 913828 [41E02960] -> osm_pi_rcv_process_set: Received error 
>> status 0x1c for SetResp() during ACTIVE transition
>> Mar 20 14:26:08 913833 [41E02960] -> PortInfo dump:
>>                 port number.............0x18
>>                 node_guid...............0x0005ad00000281a7
>>                 port_guid...............0x0005ad00000281a7
>>                 m_key...................0x0000000000000000
>>                 subnet_prefix...........0x0000000000000000
>>                 base_lid................0x0
>>                 master_sm_base_lid......0x0
>>                 capability_mask.........0x0
>>                 diag_code...............0x0
>>                 m_key_lease_period......0x0
>>                 local_port_num..........0x2
>>                 link_width_enabled......0x3
>>                 link_width_supported....0x3
>>                 link_width_active.......0x2
>>                 link_speed_supported....0x1
>>                 port_state..............ACTIVE
>>                 state_info2.............0x52
>>                 m_key_protect_bits......0x0
>>                 lmc.....................0x0
>>                 link_speed..............0x11
>>                 mtu_smsl................0x40
>>                 vl_cap_init_type........0x40
>>                 vl_high_limit...........0x0
>>                 vl_arb_high_cap.........0x8
>>                 vl_arb_low_cap..........0x8
>>                 init_rep_mtu_cap........0x4
>>                 vl_stall_life...........0xF2
>>                 vl_enforce..............0x40
>>                 m_key_violations........0x0
>>                 p_key_violations........0x0
>>                 q_key_violations........0x0
>>                 guid_cap................0x0
>>                 client_reregister.......0x0
>>                 subnet_timeout..........0x0
>>                 resp_time_value.........0x0
>>                 error_threshold.........0x88
>> Mar 20 14:26:08 913848 [41E02960] -> Capabilities Mask:
>> Mar 20 14:26:08 918887 [41E02960] -> SUBNET UP
>> Mar 20 14:26:48 657517 [41E02960] -> __osm_trap_rcv_process_request: 
>> Received Generic Notice type:0x01 num:128 Producer:2 from LID:0x0148 
>> TID:0x000000000000006f
>> Mar 20 14:26:48 657779 [41E02960] -> osm_report_notice: Reporting Generic 
>> Notice type:1 num:128 from LID:0x0148 
>> GID:0xfe80000000000000,0x0005ad00000281b3
>> Mar 20 14:26:48 658393 [43204960] -> __osm_trap_rcv_process_request: 
>> Received Generic Notice type:0x01 num:128 Producer:2 from LID:0x001B 
>> TID:0x0000000000000081
>> Mar 20 14:26:48 658465 [43204960] -> osm_report_notice: Reporting Generic 
>> Notice type:1 num:128 from LID:0x001B 
>> GID:0xfe80000000000000,0x0005ad00000281a7
>> Mar 20 14:26:48 979610 [41401960] -> osm_report_notice: Reporting Generic 
>> Notice type:3 num:65 from LID:0x0092 
>> GID:0xfe80000000000000,0x0005ad0000024bbb
>> Mar 20 14:26:48 979629 [41401960] -> Removed port with 
>> GUID:0x0005ad0000024e0b LID range [0xB3,0xB3] of node:saguaro-23-4 HCA-1
>> Mar 20 14:26:48 979652 [41401960] -> osm_report_notice: Reporting Generic 
>> Notice type:3 num:65 from LID:0x0092 
>> GID:0xfe80000000000000,0x0005ad0000024bbb
>> Mar 20 14:26:48 979660 [41401960] -> Removed port with 
>> GUID:0x0005ad000002510b LID range [0xB5,0xB5] of node:saguaro-23-6 HCA-1
>> Mar 20 14:26:48 979682 [41401960] -> osm_report_notice: Reporting Generic 
>> Notice type:3 num:65 from LID:0x0092 
>> GID:0xfe80000000000000,0x0005ad0000024bbb
>> Mar 20 14:26:48 979688 [41401960] -> Removed port with 
>> GUID:0x0005ad000002511b LID range [0xA6,0xA6] of node:saguaro-22-1 HCA-1
>> Mar 20 14:26:48 979721 [41401960] -> osm_report_notice: Reporting Generic 
>> Notice type:3 num:65 from LID:0x0092 
>> GID:0xfe80000000000000,0x0005ad0000024bbb
>> Mar 20 14:26:48 979727 [41401960] -> Removed port with 
>> GUID:0x0005ad0000024b27 LID range [0xAF,0xAF] of node:saguaro-23-0 HCA-1
>> Mar 20 14:26:48 979770 [41401960] -> osm_report_notice: Reporting Generic 
>> Notice type:3 num:65 from LID:0x0092 
>> GID:0xfe80000000000000,0x0005ad0000024bbb
>> Mar 20 14:26:48 979782 [41401960] -> Removed port with 
>> GUID:0x0005ad0000025043 LID range [0xB4,0xB4] of node:saguaro-23-5 HCA-1
>> Mar 20 14:26:48 979799 [41401960] -> osm_report_notice: Reporting Generic 
>> Notice type:3 num:65 from LID:0x0092 
>> GID:0xfe80000000000000,0x0005ad0000024bbb
>> Mar 20 14:26:48 979804 [41401960] -> Removed port with 
>> GUID:0x0005ad0000024d47 LID range [0xB6,0xB6] of node:saguaro-23-7 HCA-1
>> Mar 20 14:26:48 979822 [41401960] -> osm_report_notice: Reporting Generic 
>> Notice type:3 num:65 from LID:0x0092 
>> GID:0xfe80000000000000,0x0005ad0000024bbb
>> Mar 20 14:26:48 979827 [41401960] -> Removed port with 
>> GUID:0x0005ad0000024d6b LID range [0xB8,0xB8] of node:saguaro-23-9 HCA-1
>> Mar 20 14:26:48 979845 [41401960] -> osm_report_notice: Reporting Generic 
>> Notice type:3 num:65 from LID:0x0092 
>> GID:0xfe80000000000000,0x0005ad0000024bbb
>> Mar 20 14:26:48 979849 [41401960] -> Removed port with 
>> GUID:0x0005ad0000024977 LID range [0xA9,0xA9] of node:saguaro-22-4 HCA-1
>> Mar 20 14:26:48 980028 [41401960] -> osm_report_notice: Reporting Generic 
>> Notice type:3 num:65 from LID:0x0092 
>> GID:0xfe80000000000000,0x0005ad0000024bbb
>> Mar 20 14:26:48 980033 [41401960] -> Removed port with 
>> GUID:0x0005ad0000027c84 LID range [0x152,0x152] of node:Topspin Switch TS120
>> Mar 20 14:26:48 980061 [41401960] -> osm_report_notice: Reporting Generic 
>> Notice type:3 num:65 from LID:0x0092 
>> GID:0xfe80000000000000,0x0005ad0000024bbb
>> Mar 20 14:26:48 980066 [41401960] -> Removed port with 
>> GUID:0x0005ad0000024d8b LID range [0xB7,0xB7] of node:saguaro-23-8 HCA-1
>> Mar 20 14:26:48 980081 [41401960] -> osm_report_notice: Reporting Generic 
>> Notice type:3 num:65 from LID:0x0092 
>> GID:0xfe80000000000000,0x0005ad0000024bbb
>> Mar 20 14:26:48 980087 [41401960] -> Removed port with 
>> GUID:0x0005ad000002498f LID range [0xA8,0xA8] of node:saguaro-22-3 HCA-1
>> Mar 20 14:26:48 980102 [41401960] -> osm_report_notice: Reporting Generic 
>> Notice type:3 num:65 from LID:0x0092 
>> GID:0xfe80000000000000,0x0005ad0000024bbb
>> Mar 20 14:26:48 980107 [41401960] -> Removed port with 
>> GUID:0x0005ad0000024c9b LID range [0xA7,0xA7] of node:saguaro-22-2 HCA-1
>> Mar 20 14:26:48 980122 [41401960] -> osm_report_notice: Reporting Generic 
>> Notice type:3 num:65 from LID:0x0092 
>> GID:0xfe80000000000000,0x0005ad0000024bbb
>> Mar 20 14:26:48 980127 [41401960] -> Removed port with 
>> GUID:0x0005ad0000024da7 LID range [0xB0,0xB0] of node:saguaro-23-1 HCA-1
>> Mar 20 14:26:48 980143 [41401960] -> osm_report_notice: Reporting Generic 
>> Notice type:3 num:65 from LID:0x0092 
>> GID:0xfe80000000000000,0x0005ad0000024bbb
>> Mar 20 14:26:48 980148 [41401960] -> Removed port with 
>> GUID:0x0005ad0000024cbb LID range [0xB2,0xB2] of node:saguaro-23-3 HCA-1
>> Mar 20 14:26:48 980163 [41401960] -> osm_report_notice: Reporting Generic 
>> Notice type:3 num:65 from LID:0x0092 
>> GID:0xfe80000000000000,0x0005ad0000024bbb
>> Mar 20 14:26:48 980239 [41401960] -> Removed port with 
>> GUID:0x0005ad00000249d3 LID range [0xB1,0xB1] of node:saguaro-23-2 HCA-1
>> Mar 20 14:26:48 980256 [41401960] -> osm_report_notice: Reporting Generic 
>> Notice type:3 num:65 from LID:0x0092 
>> GID:0xfe80000000000000,0x0005ad0000024bbb
>> Mar 20 14:26:48 980261 [41401960] -> Removed port with 
>> GUID:0x0005ad0000024feb LID range [0x153,0x153] of node:saguaro-22-5 HCA-1
>> Mar 20 14:26:48 980365 [41401960] -> osm_report_notice: Reporting Generic 
>> Notice type:3 num:65 from LID:0x0092 
>> GID:0xfe80000000000000,0x0005ad0000024bbb
>> Mar 20 14:26:48 980371 [41401960] -> Removed port with 
>> GUID:0x0005ad0000024afb LID range [0xA5,0xA5] of node:saguaro-22-0 HCA-1
>> Mar 20 14:26:49 013365 [41401960] -> osm_ucast_mgr_process: Min Hop Tables 
>> configured on all switches
>> Mar 20 14:26:49 065887 [43C05960] -> SUBNET UP
>> Mar 20 14:26:49 407010 [44606960] -> osm_ucast_mgr_process: Min Hop Tables 
>> configured on all switches
>> Mar 20 14:26:49 459477 [44606960] -> SUBNET UP
>> Mar 20 14:27:42 754098 [45007960] -> __osm_trap_rcv_process_request: 
>> Received Generic Notice type:0x01 num:128 Producer:2 from LID:0x0148 
>> TID:0x0000000000000070
>> Mar 20 14:27:42 754349 [45007960] -> osm_report_notice: Reporting Generic 
>> Notice type:1 num:128 from LID:0x0148 
>> GID:0xfe80000000000000,0x0005ad00000281b3
>> Mar 20 14:27:42 760115 [43C05960] -> __osm_trap_rcv_process_request: 
>> Received Generic Notice type:0x01 num:128 Producer:2 from LID:0x0148 
>> TID:0x0000000000000071
>> Mar 20 14:27:42 760178 [44606960] -> __osm_trap_rcv_process_request: 
>> Received Generic Notice type:0x01 num:128 Producer:2 from LID:0x001B 
>> TID:0x0000000000000082
>> Mar 20 14:27:42 760236 [43C05960] -> osm_report_notice: Reporting Generic 
>> Notice type:1 num:128 from LID:0x0148 
>> GID:0xfe80000000000000,0x0005ad00000281b3
>> Mar 20 14:27:42 760406 [44606960] -> osm_report_notice: Reporting Generic 
>> Notice type:1 num:128 from LID:0x001B 
>> GID:0xfe80000000000000,0x0005ad00000281a7
>> Mar 20 14:27:42 766931 [41E02960] -> __osm_trap_rcv_process_request: 
>> Received Generic Notice type:0x01 num:128 Producer:2 from LID:0x001B 
>> TID:0x0000000000000083
>> Mar 20 14:27:42 767049 [41E02960] -> osm_report_notice: Reporting Generic 
>> Notice type:1 num:128 from LID:0x001B 
>> GID:0xfe80000000000000,0x0005ad00000281a7
>> Mar 20 14:27:43 085327 [43C05960] -> osm_report_notice: Reporting Generic 
>> Notice type:3 num:64 from LID:0x0092 
>> GID:0xfe80000000000000,0x0005ad0000024bbb
>> Mar 20 14:27:43 085345 [43C05960] -> Discovered new port with 
>> GUID:0x0005ad0000027c84 LID range [0x152,0x152] of node:Topspin Switch TS120
>> Mar 20 14:27:43 085349 [43C05960] -> osm_report_notice: Reporting Generic 
>> Notice type:3 num:64 from LID:0x0092 
>> GID:0xfe80000000000000,0x0005ad0000024bbb
>> Mar 20 14:27:43 085355 [43C05960] -> Discovered new port with 
>> GUID:0x0005ad00000249d3 LID range [0xB1,0xB1] of node:saguaro-23-2 HCA-1
>> Mar 20 14:27:43 085359 [43C05960] -> osm_report_notice: Reporting Generic 
>> Notice type:3 num:64 from LID:0x0092 
>> GID:0xfe80000000000000,0x0005ad0000024bbb
>> Mar 20 14:27:43 085364 [43C05960] -> Discovered new port with 
>> GUID:0x0005ad0000025043 LID range [0xB4,0xB4] of node:saguaro-23-5 HCA-1
>> Mar 20 14:27:43 085368 [43C05960] -> osm_report_notice: Reporting Generic 
>> Notice type:3 num:64 from LID:0x0092 
>> GID:0xfe80000000000000,0x0005ad0000024bbb
>> Mar 20 14:27:43 085373 [43C05960] -> Discovered new port with 
>> GUID:0x0005ad000002510b LID range [0xB5,0xB5] of node:saguaro-23-6 HCA-1
>> Mar 20 14:27:43 085377 [43C05960] -> osm_report_notice: Reporting Generic 
>> Notice type:3 num:64 from LID:0x0092 
>> GID:0xfe80000000000000,0x0005ad0000024bbb
>> Mar 20 14:27:43 085382 [43C05960] -> Discovered new port with 
>> GUID:0x0005ad0000024b27 LID range [0xAF,0xAF] of node:saguaro-23-0 HCA-1
>> Mar 20 14:27:43 085386 [43C05960] -> osm_report_notice: Reporting Generic 
>> Notice type:3 num:64 from LID:0x0092 
>> GID:0xfe80000000000000,0x0005ad0000024bbb
>> Mar 20 14:27:43 085390 [43C05960] -> Discovered new port with 
>> GUID:0x0005ad0000024da7 LID range [0xB0,0xB0] of node:saguaro-23-1 HCA-1
>> Mar 20 14:27:43 085394 [43C05960] -> osm_report_notice: Reporting Generic 
>> Notice type:3 num:64 from LID:0x0092 
>> GID:0xfe80000000000000,0x0005ad0000024bbb
>> Mar 20 14:27:43 085399 [43C05960] -> Discovered new port with 
>> GUID:0x0005ad0000024cbb LID range [0xB2,0xB2] of node:saguaro-23-3 HCA-1
>> Mar 20 14:27:43 085403 [43C05960] -> osm_report_notice: Reporting Generic 
>> Notice type:3 num:64 from LID:0x0092 
>> GID:0xfe80000000000000,0x0005ad0000024bbb
>> Mar 20 14:27:43 085407 [43C05960] -> Discovered new port with 
>> GUID:0x0005ad0000024afb LID range [0xA5,0xA5] of node:saguaro-22-0 HCA-1
>> Mar 20 14:27:43 085411 [43C05960] -> osm_report_notice: Reporting Generic 
>> Notice type:3 num:64 from LID:0x0092 
>> GID:0xfe80000000000000,0x0005ad0000024bbb
>> Mar 20 14:27:43 085416 [43C05960] -> Discovered new port with 
>> GUID:0x0005ad0000024e0b LID range [0xB3,0xB3] of node:saguaro-23-4 HCA-1
>> Mar 20 14:27:43 085420 [43C05960] -> osm_report_notice: Reporting Generic 
>> Notice type:3 num:64 from LID:0x0092 
>> GID:0xfe80000000000000,0x0005ad0000024bbb
>> Mar 20 14:27:43 085425 [43C05960] -> Discovered new port with 
>> GUID:0x0005ad0000024d47 LID range [0xB6,0xB6] of node:saguaro-23-7 HCA-1
>> Mar 20 14:27:43 085428 [43C05960] -> osm_report_notice: Reporting Generic 
>> Notice type:3 num:64 from LID:0x0092 
>> GID:0xfe80000000000000,0x0005ad0000024bbb
>> Mar 20 14:27:43 085433 [43C05960] -> Discovered new port with 
>> GUID:0x0005ad0000024d8b LID range [0xB7,0xB7] of node:saguaro-23-8 HCA-1
>> Mar 20 14:27:43 085437 [43C05960] -> osm_report_notice: Reporting Generic 
>> Notice type:3 num:64 from LID:0x0092 
>> GID:0xfe80000000000000,0x0005ad0000024bbb
>> Mar 20 14:27:43 085442 [43C05960] -> Discovered new port with 
>> GUID:0x0005ad0000024d6b LID range [0xB8,0xB8] of node:saguaro-23-9 HCA-1
>> Mar 20 14:27:43 085446 [43C05960] -> osm_report_notice: Reporting Generic 
>> Notice type:3 num:64 from LID:0x0092 
>> GID:0xfe80000000000000,0x0005ad0000024bbb
>> Mar 20 14:27:43 085450 [43C05960] -> Discovered new port with 
>> GUID:0x0005ad000002511b LID range [0xA6,0xA6] of node:saguaro-22-1 HCA-1
>> Mar 20 14:27:43 085454 [43C05960] -> osm_report_notice: Reporting Generic 
>> Notice type:3 num:64 from LID:0x0092 
>> GID:0xfe80000000000000,0x0005ad0000024bbb
>> Mar 20 14:27:43 085459 [43C05960] -> Discovered new port with 
>> GUID:0x0005ad0000024c9b LID range [0xA7,0xA7] of node:saguaro-22-2 HCA-1
>> Mar 20 14:27:43 085511 [43C05960] -> osm_report_notice: Reporting Generic 
>> Notice type:3 num:64 from LID:0x0092 
>> GID:0xfe80000000000000,0x0005ad0000024bbb
>> Mar 20 14:27:43 085517 [43C05960] -> Discovered new port with 
>> GUID:0x0005ad000002498f LID range [0xA8,0xA8] of node:saguaro-22-3 HCA-1
>> Mar 20 14:27:43 085521 [43C05960] -> osm_report_notice: Reporting Generic 
>> Notice type:3 num:64 from LID:0x0092 
>> GID:0xfe80000000000000,0x0005ad0000024bbb
>> Mar 20 14:27:43 085526 [43C05960] -> Discovered new port with 
>> GUID:0x0005ad0000024feb LID range [0x153,0x153] of node:saguaro-22-5 HCA-1
>> Mar 20 14:27:43 085530 [43C05960] -> osm_report_notice: Reporting Generic 
>> Notice type:3 num:64 from LID:0x0092 
>> GID:0xfe80000000000000,0x0005ad0000024bbb
>> Mar 20 14:27:43 085534 [43C05960] -> Discovered new port with 
>> GUID:0x0005ad0000024977 LID range [0xA9,0xA9] of node:saguaro-22-4 HCA-1
>> Mar 20 14:27:43 116308 [43C05960] -> osm_ucast_mgr_process: Min Hop Tables 
>> configured on all switches
>> Mar 20 14:27:43 179935 [4780B960] -> __osm_sm_mad_ctrl_rcv_callback: ERR 
>> 3111: Error status = 0x1C00
>> Mar 20 14:27:43 179980 [4780B960] -> SMP dump:
>>                 base_ver................0x1
>>                 mgmt_class..............0x81
>>                 class_ver...............0x1
>>                 method..................0x81 (SubnGetResp)
>>                 D bit...................0x1
>>                 status..................0x1C00
>>                 hop_ptr.................0x0
>>                 hop_count...............0x4
>>                 trans_id................0x85669
>>                 attr_id.................0x15 (PortInfo)
>>                 resv....................0x0
>>                 attr_mod................0x13
>>                 m_key...................0x0000000000000000
>>                 dr_slid.................0xFFFF
>>                 dr_dlid.................0xFFFF
>> 
>>                 Initial path: [0][1][16][1][4]
>>                 Return path:  [0][9][13][D][1]
>>                 Reserved:     [0][0][0][0][0][0][0]
>> 
>>                 00 00 00 00 00 00 00 00   00 00 00 00 00 00 00 00
>> 
>>                 00 00 00 00 00 00 00 00   00 00 00 00 01 03 03 02
>> 
>>                 14 52 00 11 40 40 00 08   08 04 F2 40 00 00 00 00
>> 
>>                 00 00 00 00 00 88 00 00   00 00 00 00 00 00 00 00
>> 
>> Mar 20 14:27:43 180019 [41401960] -> osm_pi_rcv_process_set: Received error 
>> status 0x1c for SetResp() during ACTIVE transition
>> Mar 20 14:27:43 180037 [41401960] -> PortInfo dump:
>>                 port number.............0x13
>>                 node_guid...............0x0005ad00000281a7
>>                 port_guid...............0x0005ad00000281a7
>>                 m_key...................0x0000000000000000
>>                 subnet_prefix...........0x0000000000000000
>>                 base_lid................0x0
>>                 master_sm_base_lid......0x0
>>                 capability_mask.........0x0
>>                 diag_code...............0x0
>>                 m_key_lease_period......0x0
>>                 local_port_num..........0x1
>>                 link_width_enabled......0x3
>>                 link_width_supported....0x3
>>                 link_width_active.......0x2
>>                 link_speed_supported....0x1
>>                 port_state..............ACTIVE
>>                 state_info2.............0x52
>>                 m_key_protect_bits......0x0
>>                 lmc.....................0x0
>>                 link_speed..............0x11
>>                 mtu_smsl................0x40
>>                 vl_cap_init_type........0x40
>>                 vl_high_limit...........0x0
>>                 vl_arb_high_cap.........0x8
>>                 vl_arb_low_cap..........0x8
>>                 init_rep_mtu_cap........0x4
>>                 vl_stall_life...........0xF2
>>                 vl_enforce..............0x40
>>                 m_key_violations........0x0
>>                 p_key_violations........0x0
>>                 q_key_violations........0x0
>>                 guid_cap................0x0
>>                 client_reregister.......0x0
>>                 subnet_timeout..........0x0
>>                 resp_time_value.........0x0
>>                 error_threshold.........0x88
>> Mar 20 14:27:43 180050 [41401960] -> Capabilities Mask:
>> Mar 20 14:27:43 180092 [4780B960] -> __osm_sm_mad_ctrl_rcv_callback: ERR 
>> 3111: Error status = 0x1C00
>> Mar 20 14:27:43 180137 [4780B960] -> SMP dump:
>>                 base_ver................0x1
>>                 mgmt_class..............0x81
>>                 class_ver...............0x1
>>                 method..................0x81 (SubnGetResp)
>>                 D bit...................0x1
>>                 status..................0x1C00
>>                 hop_ptr.................0x0
>>                 hop_count...............0x4
>>                 trans_id................0x8566a
>>                 attr_id.................0x15 (PortInfo)
>>                 resv....................0x0
>>                 attr_mod................0x16
>>                 m_key...................0x0000000000000000
>>                 dr_slid.................0xFFFF
>>                 dr_dlid.................0xFFFF
>> 
>>                 Initial path: [0][1][16][1][4]
>>                 Return path:  [0][9][13][D][1]
>>                 Reserved:     [0][0][0][0][0][0][0]
>> 
>>                 00 00 00 00 00 00 00 00   00 00 00 00 00 00 00 00
>> 
>>                 00 00 00 00 00 00 00 00   00 00 00 00 01 03 03 02
>> 
>>                 14 52 00 11 40 40 00 08   08 04 F2 40 00 00 00 00
>> 
>>                 00 00 00 00 00 88 00 00   00 00 00 00 00 00 00 00
>> 
>> Mar 20 14:27:43 180185 [44606960] -> osm_pi_rcv_process_set: Received error 
>> status 0x1c for SetResp() during ACTIVE transition
>> Mar 20 14:27:43 180189 [44606960] -> PortInfo dump:
>>                 port number.............0x16
>>                 node_guid...............0x0005ad00000281a7
>>                 port_guid...............0x0005ad00000281a7
>>                 m_key...................0x0000000000000000
>>                 subnet_prefix...........0x0000000000000000
>>                 base_lid................0x0
>>                 master_sm_base_lid......0x0
>>                 capability_mask.........0x0
>>                 diag_code...............0x0
>>                 m_key_lease_period......0x0
>>                 local_port_num..........0x1
>>                 link_width_enabled......0x3
>>                 link_width_supported....0x3
>>                 link_width_active.......0x2
>>                 link_speed_supported....0x1
>>                 port_state..............ACTIVE
>>                 state_info2.............0x52
>>                 m_key_protect_bits......0x0
>>                 lmc.....................0x0
>>                 link_speed..............0x11
>>                 mtu_smsl................0x40
>>                 vl_cap_init_type........0x40
>>                 vl_high_limit...........0x0
>>                 vl_arb_high_cap.........0x8
>>                 vl_arb_low_cap..........0x8
>>                 init_rep_mtu_cap........0x4
>>                 vl_stall_life...........0xF2
>>                 vl_enforce..............0x40
>>                 m_key_violations........0x0
>>                 p_key_violations........0x0
>>                 q_key_violations........0x0
>>                 guid_cap................0x0
>>                 client_reregister.......0x0
>>                 subnet_timeout..........0x0
>>                 resp_time_value.........0x0
>>                 error_threshold.........0x88
>> Mar 20 14:27:43 180199 [44606960] -> Capabilities Mask:
>> Mar 20 14:27:43 180239 [4780B960] -> __osm_sm_mad_ctrl_rcv_callback: ERR 
>> 3111: Error status = 0x1C00
>> Mar 20 14:27:43 180263 [4780B960] -> SMP dump:
>>                 base_ver................0x1
>>                 mgmt_class..............0x81
>>                 class_ver...............0x1
>>                 method..................0x81 (SubnGetResp)
>>                 D bit...................0x1
>>                 status..................0x1C00
>>                 hop_ptr.................0x0
>>                 hop_count...............0x4
>>                 trans_id................0x8566b
>>                 attr_id.................0x15 (PortInfo)
>>                 resv....................0x0
>>                 attr_mod................0x17
>>                 m_key...................0x0000000000000000
>>                 dr_slid.................0xFFFF
>>                 dr_dlid.................0xFFFF
>> 
>>                 Initial path: [0][1][16][1][4]
>>                 Return path:  [0][9][13][D][1]
>>                 Reserved:     [0][0][0][0][0][0][0]
>> 
>>                 00 00 00 00 00 00 00 00   00 00 00 00 00 00 00 00
>> 
>>                 00 00 00 00 00 00 00 00   00 00 00 00 01 03 03 02
>> 
>>                 14 52 00 11 40 40 00 08   08 04 F2 40 00 00 00 00
>> 
>>                 00 00 00 00 00 88 00 00   00 00 00 00 00 00 00 00
>> 
>> Mar 20 14:27:43 180307 [4780B960] -> __osm_sm_mad_ctrl_rcv_callback: ERR 
>> 3111: Error status = 0x1C00
>> Mar 20 14:27:43 180319 [42803960] -> osm_pi_rcv_process_set: Received error 
>> status 0x1c for SetResp() during ACTIVE transition
>> Mar 20 14:27:43 180332 [4780B960] -> SMP dump:
>>                 base_ver................0x1
>>                 mgmt_class..............0x81
>>                 class_ver...............0x1
>>                 method..................0x81 (SubnGetResp)
>>                 D bit...................0x1
>>                 status..................0x1C00
>>                 hop_ptr.................0x0
>>                 hop_count...............0x4
>>                 trans_id................0x8566c
>>                 attr_id.................0x15 (PortInfo)
>>                 resv....................0x0
>>                 attr_mod................0x18
>>                 m_key...................0x0000000000000000
>>                 dr_slid.................0xFFFF
>>                 dr_dlid.................0xFFFF
>> 
>>                 Initial path: [0][1][16][1][4]
>>                 Return path:  [0][9][13][D][1]
>>                 Reserved:     [0][0][0][0][0][0][0]
>> 
>>                 00 00 00 00 00 00 00 00   00 00 00 00 00 00 00 00
>> 
>>                 00 00 00 00 00 00 00 00   00 00 00 00 01 03 03 02
>> 
>>                 14 52 00 11 40 40 00 08   08 04 F2 40 00 00 00 00
>> 
>>                 00 00 00 00 00 88 00 00   00 00 00 00 00 00 00 00
>> 
>> Mar 20 14:27:43 180336 [42803960] -> PortInfo dump:
>>                 port number.............0x17
>>                 node_guid...............0x0005ad00000281a7
>>                 port_guid...............0x0005ad00000281a7
>>                 m_key...................0x0000000000000000
>>                 subnet_prefix...........0x0000000000000000
>>                 base_lid................0x0
>>                 master_sm_base_lid......0x0
>>                 capability_mask.........0x0
>>                 diag_code...............0x0
>>                 m_key_lease_period......0x0
>>                 local_port_num..........0x1
>>                 link_width_enabled......0x3
>>                 link_width_supported....0x3
>>                 link_width_active.......0x2
>>                 link_speed_supported....0x1
>>                 port_state..............ACTIVE
>>                 state_info2.............0x52
>>                 m_key_protect_bits......0x0
>>                 lmc.....................0x0
>>                 link_speed..............0x11
>>                 mtu_smsl................0x40
>>                 vl_cap_init_type........0x40
>>                 vl_high_limit...........0x0
>>                 vl_arb_high_cap.........0x8
>>                 vl_arb_low_cap..........0x8
>>                 init_rep_mtu_cap........0x4
>>                 vl_stall_life...........0xF2
>>                 vl_enforce..............0x40
>>                 m_key_violations........0x0
>>                 p_key_violations........0x0
>>                 q_key_violations........0x0
>>                 guid_cap................0x0
>>                 client_reregister.......0x0
>>                 subnet_timeout..........0x0
>>                 resp_time_value.........0x0
>>                 error_threshold.........0x88
>> Mar 20 14:27:43 180364 [42803960] -> Capabilities Mask:
>> Mar 20 14:27:43 180389 [42803960] -> osm_pi_rcv_process_set: Received error 
>> status 0x1c for SetResp() during ACTIVE transition
>> Mar 20 14:27:43 180410 [4780B960] -> __osm_sm_mad_ctrl_rcv_callback: ERR 
>> 3111: Error status = 0x1C00
>> Mar 20 14:27:43 180392 [42803960] -> PortInfo dump:
>>                 port number.............0x18
>>                 node_guid...............0x0005ad00000281a7
>>                 port_guid...............0x0005ad00000281a7
>>                 m_key...................0x0000000000000000
>>                 subnet_prefix...........0x0000000000000000
>>                 base_lid................0x0
>>                 master_sm_base_lid......0x0
>>                 capability_mask.........0x0
>>                 diag_code...............0x0
>>                 m_key_lease_period......0x0
>>                 local_port_num..........0x1
>>                 link_width_enabled......0x3
>>                 link_width_supported....0x3
>>                 link_width_active.......0x2
>>                 link_speed_supported....0x1
>>                 port_state..............ACTIVE
>>                 state_info2.............0x52
>>                 m_key_protect_bits......0x0
>>                 lmc.....................0x0
>>                 link_speed..............0x11
>>                 mtu_smsl................0x40
>>                 vl_cap_init_type........0x40
>>                 vl_high_limit...........0x0
>>                 vl_arb_high_cap.........0x8
>>                 vl_arb_low_cap..........0x8
>>                 init_rep_mtu_cap........0x4
>>                 vl_stall_life...........0xF2
>>                 vl_enforce..............0x40
>>                 m_key_violations........0x0
>>                 p_key_violations........0x0
>>                 q_key_violations........0x0
>>                 guid_cap................0x0
>>                 client_reregister.......0x0
>>                 subnet_timeout..........0x0
>>                 resp_time_value.........0x0
>>                 error_threshold.........0x88
>> Mar 20 14:27:43 180415 [42803960] -> Capabilities Mask:
>> Mar 20 14:27:43 180436 [4780B960] -> SMP dump:
>>                 base_ver................0x1
>>                 mgmt_class..............0x81
>>                 class_ver...............0x1
>>                 method..................0x81 (SubnGetResp)
>>                 D bit...................0x1
>>                 status..................0x1C00
>>                 hop_ptr.................0x0
>>                 hop_count...............0x4
>>                 trans_id................0x8566d
>>                 attr_id.................0x15 (PortInfo)
>>                 resv....................0x0
>>                 attr_mod................0x16
>>                 m_key...................0x0000000000000000
>>                 dr_slid.................0xFFFF
>>                 dr_dlid.................0xFFFF
>> 
>>                 Initial path: [0][1][11][2][5]
>>                 Return path:  [0][9][18][E][2]
>>                 Reserved:     [0][0][0][0][0][0][0]
>> 
>>                 00 00 00 00 00 00 00 00   00 00 00 00 00 00 00 00
>> 
>>                 00 00 00 00 00 00 00 00   00 00 00 00 02 03 03 02
>> 
>>                 14 52 00 11 40 40 00 08   08 04 F2 40 00 00 00 00
>> 
>>                 00 00 00 00 00 88 00 00   00 00 00 00 00 00 00 00
>> 
>> Mar 20 14:27:43 180490 [41E02960] -> osm_pi_rcv_process_set: Received error 
>> status 0x1c for SetResp() during ACTIVE transition
>> Mar 20 14:27:43 180494 [41E02960] -> PortInfo dump:
>>                 port number.............0x16
>>                 node_guid...............0x0005ad00000281b3
>>                 port_guid...............0x0005ad00000281b3
>>                 m_key...................0x0000000000000000
>>                 subnet_prefix...........0x0000000000000000
>>                 base_lid................0x0
>>                 master_sm_base_lid......0x0
>>                 capability_mask.........0x0
>>                 diag_code...............0x0
>>                 m_key_lease_period......0x0
>>                 local_port_num..........0x2
>>                 link_width_enabled......0x3
>>                 link_width_supported....0x3
>>                 link_width_active.......0x2
>>                 link_speed_supported....0x1
>>                 port_state..............ACTIVE
>>                 state_info2.............0x52
>>                 m_key_protect_bits......0x0
>>                 lmc.....................0x0
>>                 link_speed..............0x11
>>                 mtu_smsl................0x40
>>                 vl_cap_init_type........0x40
>>                 vl_high_limit...........0x0
>>                 vl_arb_high_cap.........0x8
>>                 vl_arb_low_cap..........0x8
>>                 init_rep_mtu_cap........0x4
>>                 vl_stall_life...........0xF2
>>                 vl_enforce..............0x40
>>                 m_key_violations........0x0
>>                 p_key_violations........0x0
>>                 q_key_violations........0x0
>>                 guid_cap................0x0
>>                 client_reregister.......0x0
>>                 subnet_timeout..........0x0
>>                 resp_time_value.........0x0
>>                 error_threshold.........0x88
>> Mar 20 14:27:43 180504 [41E02960] -> Capabilities Mask:
>> Mar 20 14:27:43 180536 [4780B960] -> __osm_sm_mad_ctrl_rcv_callback: ERR 
>> 3111: Error status = 0x1C00
>> Mar 20 14:27:43 180560 [4780B960] -> SMP dump:
>>                 base_ver................0x1
>>                 mgmt_class..............0x81
>>                 class_ver...............0x1
>>                 method..................0x81 (SubnGetResp)
>>                 D bit...................0x1
>>                 status..................0x1C00
>>                 hop_ptr.................0x0
>>                 hop_count...............0x4
>>                 trans_id................0x8566e
>>                 attr_id.................0x15 (PortInfo)
>>                 resv....................0x0
>>                 attr_mod................0x17
>>                 m_key...................0x0000000000000000
>>                 dr_slid.................0xFFFF
>>                 dr_dlid.................0xFFFF
>> 
>>                 Initial path: [0][1][11][2][5]
>>                 Return path:  [0][9][18][E][2]
>>                 Reserved:     [0][0][0][0][0][0][0]
>> 
>>                 00 00 00 00 00 00 00 00   00 00 00 00 00 00 00 00
>> 
>>                 00 00 00 00 00 00 00 00   00 00 00 00 02 03 03 02
>> 
>>                 14 52 00 11 40 40 00 08   08 04 F2 40 00 00 00 00
>> 
>>                 00 00 00 00 00 88 00 00   00 00 00 00 00 00 00 00
>> 
>> Mar 20 14:27:43 180606 [45007960] -> osm_pi_rcv_process_set: Received error 
>> status 0x1c for SetResp() during ACTIVE transition
>> Mar 20 14:27:43 180615 [4780B960] -> __osm_sm_mad_ctrl_rcv_callback: ERR 
>> 3111: Error status = 0x1C00
>> Mar 20 14:27:43 180634 [45007960] -> PortInfo dump:
>>                 port number.............0x17
>>                 node_guid...............0x0005ad00000281b3
>>                 port_guid...............0x0005ad00000281b3
>>                 m_key...................0x0000000000000000
>>                 subnet_prefix...........0x0000000000000000
>>                 base_lid................0x0
>>                 master_sm_base_lid......0x0
>>                 capability_mask.........0x0
>>                 diag_code...............0x0
>>                 m_key_lease_period......0x0
>>                 local_port_num..........0x2
>>                 link_width_enabled......0x3
>>                 link_width_supported....0x3
>>                 link_width_active.......0x2
>>                 link_speed_supported....0x1
>>                 port_state..............ACTIVE
>>                 state_info2.............0x52
>>                 m_key_protect_bits......0x0
>>                 lmc.....................0x0
>>                 link_speed..............0x11
>>                 mtu_smsl................0x40
>>                 vl_cap_init_type........0x40
>>                 vl_high_limit...........0x0
>>                 vl_arb_high_cap.........0x8
>>                 vl_arb_low_cap..........0x8
>>                 init_rep_mtu_cap........0x4
>>                 vl_stall_life...........0xF2
>>                 vl_enforce..............0x40
>>                 m_key_violations........0x0
>>                 p_key_violations........0x0
>>                 q_key_violations........0x0
>>                 guid_cap................0x0
>>                 client_reregister.......0x0
>>                 subnet_timeout..........0x0
>>                 resp_time_value.........0x0
>>                 error_threshold.........0x88
>> Mar 20 14:27:43 180657 [45007960] -> Capabilities Mask:
>> Mar 20 14:27:43 180678 [4780B960] -> SMP dump:
>>                 base_ver................0x1
>>                 mgmt_class..............0x81
>>                 class_ver...............0x1
>>                 method..................0x81 (SubnGetResp)
>>                 D bit...................0x1
>>                 status..................0x1C00
>>                 hop_ptr.................0x0
>>                 hop_count...............0x4
>>                 trans_id................0x8566f
>>                 attr_id.................0x15 (PortInfo)
>>                 resv....................0x0
>>                 attr_mod................0x18
>>                 m_key...................0x0000000000000000
>>                 dr_slid.................0xFFFF
>>                 dr_dlid.................0xFFFF
>> 
>>                 Initial path: [0][1][11][2][5]
>>                 Return path:  [0][9][18][E][2]
>>                 Reserved:     [0][0][0][0][0][0][0]
>> 
>>                 00 00 00 00 00 00 00 00   00 00 00 00 00 00 00 00
>> 
>>                 00 00 00 00 00 00 00 00   00 00 00 00 02 03 03 02
>> 
>>                 14 52 00 11 40 40 00 08   08 04 F2 40 00 00 00 00
>> 
>>                 00 00 00 00 00 88 00 00   00 00 00 00 00 00 00 00
>> 
>> Mar 20 14:27:43 180769 [43C05960] -> osm_pi_rcv_process_set: Received error 
>> status 0x1c for SetResp() during ACTIVE transition
>> Mar 20 14:27:43 180775 [43C05960] -> PortInfo dump:
>>                 port number.............0x18
>>                 node_guid...............0x0005ad00000281b3
>>                 port_guid...............0x0005ad00000281b3
>>                 m_key...................0x0000000000000000
>>                 subnet_prefix...........0x0000000000000000
>>                 base_lid................0x0
>>                 master_sm_base_lid......0x0
>>                 capability_mask.........0x0
>>                 diag_code...............0x0
>>                 m_key_lease_period......0x0
>>                 local_port_num..........0x2
>>                 link_width_enabled......0x3
>>                 link_width_supported....0x3
>>                 link_width_active.......0x2
>>                 link_speed_supported....0x1
>>                 port_state..............ACTIVE
>>                 state_info2.............0x52
>>                 m_key_protect_bits......0x0
>>                 lmc.....................0x0
>>                 link_speed..............0x11
>>                 mtu_smsl................0x40
>>                 vl_cap_init_type........0x40
>>                 vl_high_limit...........0x0
>>                 vl_arb_high_cap.........0x8
>>                 vl_arb_low_cap..........0x8
>>                 init_rep_mtu_cap........0x4
>>                 vl_stall_life...........0xF2
>>                 vl_enforce..............0x40
>>                 m_key_violations........0x0
>>                 p_key_violations........0x0
>>                 q_key_violations........0x0
>>                 guid_cap................0x0
>>                 client_reregister.......0x0
>>                 subnet_timeout..........0x0
>>                 resp_time_value.........0x0
>>                 error_threshold.........0x88
>> Mar 20 14:27:43 180789 [43C05960] -> Capabilities Mask:
>> Mar 20 14:27:43 186228 [43204960] -> SUBNET UP
>> Mar 20 14:27:43 557268 [45A08960] -> osm_ucast_mgr_process: Min Hop Tables 
>> configured on all switches
>> Mar 20 14:27:43 611082 [45A08960] -> SUBNET UP
>> Mar 20 14:27:58 852744 [45007960] -> __osm_trap_rcv_process_request: 
>> Received Generic Notice type:0x01 num:128 Producer:2 from LID:0x0152 
>> TID:0x0000000000000000
>> Mar 20 14:27:58 852982 [45007960] -> osm_report_notice: Reporting Generic 
>> Notice type:1 num:128 from LID:0x0152 
>> GID:0xfe80000000000000,0x0005ad0000027c84
>> Mar 20 14:27:58 970772 [43204960] -> __osm_trap_rcv_process_request: 
>> Received Generic Notice type:0x01 num:128 Producer:2 from LID:0x0152 
>> TID:0x0000000000000001
>> Mar 20 14:27:58 970864 [43204960] -> osm_report_notice: Reporting Generic 
>> Notice type:1 num:128 from LID:0x0152 
>> GID:0xfe80000000000000,0x0005ad0000027c84
>> Mar 20 14:27:58 992628 [41E02960] -> __osm_trap_rcv_process_request: 
>> Received Generic Notice type:0x01 num:128 Producer:2 from LID:0x0152 
>> TID:0x0000000000000002
>> Mar 20 14:27:58 992712 [41E02960] -> osm_report_notice: Reporting Generic 
>> Notice type:1 num:128 from LID:0x0152 
>> GID:0xfe80000000000000,0x0005ad0000027c84
>> Mar 20 14:27:59 132331 [42803960] -> __osm_trap_rcv_process_request: 
>> Received Generic Notice type:0x01 num:128 Producer:2 from LID:0x0152 
>> TID:0x0000000000000003
>> Mar 20 14:27:59 132484 [42803960] -> osm_report_notice: Reporting Generic 
>> Notice type:1 num:128 from LID:0x0152 
>> GID:0xfe80000000000000,0x0005ad0000027c84
>> Mar 20 14:27:59 314893 [41E02960] -> __osm_trap_rcv_process_request: 
>> Received Generic Notice type:0x01 num:128 Producer:2 from LID:0x0152 
>> TID:0x0000000000000004
>> Mar 20 14:27:59 315006 [41E02960] -> osm_report_notice: Reporting Generic 
>> Notice type:1 num:128 from LID:0x0152 
>> GID:0xfe80000000000000,0x0005ad0000027c84
>> Mar 20 14:27:59 343241 [42803960] -> __osm_trap_rcv_process_request: 
>> Received Generic Notice type:0x01 num:128 Producer:2 from LID:0x0152 
>> TID:0x0000000000000005
>> Mar 20 14:27:59 343320 [42803960] -> osm_report_notice: Reporting Generic 
>> Notice type:1 num:128 from LID:0x0152 
>> GID:0xfe80000000000000,0x0005ad0000027c84
>> Mar 20 14:27:59 481698 [45007960] -> __osm_trap_rcv_process_request: 
>> Received Generic Notice type:0x01 num:128 Producer:2 from LID:0x0152 
>> TID:0x0000000000000006
>> Mar 20 14:27:59 481775 [45007960] -> osm_report_notice: Reporting Generic 
>> Notice type:1 num:128 from LID:0x0152 
>> GID:0xfe80000000000000,0x0005ad0000027c84
>> Mar 20 14:27:59 512746 [45A08960] -> __osm_trap_rcv_process_request: 
>> Received Generic Notice type:0x01 num:128 Producer:2 from LID:0x0152 
>> TID:0x0000000000000007
>> Mar 20 14:27:59 512853 [45A08960] -> osm_report_notice: Reporting Generic 
>> Notice type:1 num:128 from LID:0x0152 
>> GID:0xfe80000000000000,0x0005ad0000027c84
>> Mar 20 14:27:59 548851 [41E02960] -> osm_report_notice: Reporting Generic 
>> Notice type:3 num:65 from LID:0x0092 
>> GID:0xfe80000000000000,0x0005ad0000024bbb
>> Mar 20 14:27:59 548861 [41E02960] -> Removed port with 
>> GUID:0x0005ad00000249d3 LID range [0xB1,0xB1] of node:saguaro-23-2 HCA-1
>> Mar 20 14:27:59 583414 [41E02960] -> osm_ucast_mgr_process: Min Hop Tables 
>> configured on all switches
>> Mar 20 14:27:59 583817 [43C05960] -> __osm_trap_rcv_process_request: 
>> Received Generic Notice type:0x01 num:128 Producer:2 from LID:0x0152 
>> TID:0x0000000000000008
>> Mar 20 14:27:59 623971 [43C05960] -> osm_report_notice: Reporting Generic 
>> Notice type:1 num:128 from LID:0x0152 
>> GID:0xfe80000000000000,0x0005ad0000027c84
>> Mar 20 14:27:59 626182 [42803960] -> __osm_trap_rcv_process_request: 
>> Received Generic Notice type:0x01 num:128 Producer:2 from LID:0x0152 
>> TID:0x0000000000000009
>> Mar 20 14:27:59 626329 [42803960] -> osm_report_notice: Reporting Generic 
>> Notice type:1 num:128 from LID:0x0152 
>> GID:0xfe80000000000000,0x0005ad0000027c84
>> Mar 20 14:27:59 634080 [41E02960] -> __osm_trap_rcv_process_request: 
>> Received Generic Notice type:0x01 num:128 Producer:2 from LID:0x0152 
>> TID:0x000000000000000a
>> Mar 20 14:27:59 634442 [41E02960] -> osm_report_notice: Reporting Generic 
>> Notice type:1 num:128 from LID:0x0152 
>> GID:0xfe80000000000000,0x0005ad0000027c84
>> Mar 20 14:27:59 641962 [45A08960] -> SUBNET UP
>> Mar 20 14:27:59 656231 [41401960] -> __osm_trap_rcv_process_request: 
>> Received Generic Notice type:0x01 num:128 Producer:2 from LID:0x0152 
>> TID:0x000000000000000b
>> Mar 20 14:27:59 656307 [41401960] -> __osm_trap_rcv_process_request: ERR 
>> 3804: Received trap 11 times consecutively
>> Mar 20 14:27:59 689788 [41E02960] -> __osm_trap_rcv_process_request: 
>> Received Generic Notice type:0x01 num:128 Producer:2 from LID:0x0152 
>> TID:0x000000000000000c
>> Mar 20 14:27:59 690249 [41E02960] -> __osm_trap_rcv_process_request: ERR 
>> 3804: Received trap 12 times consecutively
>> Mar 20 14:27:59 758521 [42803960] -> __osm_trap_rcv_process_request: 
>> Received Generic Notice type:0x01 num:128 Producer:2 from LID:0x0152 
>> TID:0x000000000000000d
>> Mar 20 14:27:59 758646 [42803960] -> __osm_trap_rcv_process_request: ERR 
>> 3804: Received trap 13 times consecutively
>> Mar 20 14:27:59 970740 [43204960] -> __osm_trap_rcv_process_request: 
>> Received Generic Notice type:0x01 num:128 Producer:2 from LID:0x0152 
>> TID:0x000000000000000e
>> Mar 20 14:27:59 970812 [43204960] -> __osm_trap_rcv_process_request: ERR 
>> 3804: Received trap 14 times consecutively
>> Mar 20 14:27:59 985557 [41E02960] -> osm_report_notice: Reporting Generic 
>> Notice type:3 num:65 from LID:0x0092 
>> GID:0xfe80000000000000,0x0005ad0000024bbb
>> Mar 20 14:27:59 985577 [41E02960] -> Removed port with 
>> GUID:0x0005ad000002510b LID range [0xB5,0xB5] of node:saguaro-23-6 HCA-1
>> Mar 20 14:27:59 985601 [41E02960] -> osm_report_notice: Reporting Generic 
>> Notice type:3 num:65 from LID:0x0092 
>> GID:0xfe80000000000000,0x0005ad0000024bbb
>> Mar 20 14:27:59 985615 [41E02960] -> Removed port with 
>> GUID:0x0005ad0000024d47 LID range [0xB6,0xB6] of node:saguaro-23-7 HCA-1
>> Mar 20 14:27:59 985649 [41E02960] -> osm_report_notice: Reporting Generic 
>> Notice type:3 num:65 from LID:0x0092 
>> GID:0xfe80000000000000,0x0005ad0000024bbb
>> Mar 20 14:27:59 985656 [41E02960] -> Removed port with 
>> GUID:0x0005ad0000024d8b LID range [0xB7,0xB7] of node:saguaro-23-8 HCA-1
>> Mar 20 14:27:59 989767 [42803960] -> osm_report_notice: Reporting Generic 
>> Notice type:3 num:64 from LID:0x0092 
>> GID:0xfe80000000000000,0x0005ad0000024bbb
>> Mar 20 14:27:59 989787 [42803960] -> Discovered new port with 
>> GUID:0x0005ad00000249d3 LID range [0xB1,0xB1] of node:saguaro-23-2 HCA-1
>> Mar 20 14:28:00 014445 [43C05960] -> __osm_trap_rcv_process_request: 
>> Received Generic Notice type:0x01 num:128 Producer:2 from LID:0x0152 
>> TID:0x000000000000000f
>> Mar 20 14:28:00 014524 [43C05960] -> __osm_trap_rcv_process_request: ERR 
>> 3804: Received trap 15 times consecutively
>> Mar 20 14:28:00 020896 [42803960] -> osm_ucast_mgr_process: Min Hop Tables 
>> configured on all switches
>> Mar 20 14:28:00 086824 [43204960] -> SUBNET UP
>> Mar 20 14:28:00 124057 [45007960] -> __osm_trap_rcv_process_request: 
>> Received Generic Notice type:0x01 num:128 Producer:2 from LID:0x0152 
>> TID:0x0000000000000010
>> Mar 20 14:28:00 124108 [45007960] -> __osm_trap_rcv_process_request: ERR 
>> 3804: Received trap 16 times consecutively
>> Mar 20 14:28:00 131596 [41401960] -> __osm_trap_rcv_process_request: 
>> Received Generic Notice type:0x01 num:128 Producer:2 from LID:0x0152 
>> TID:0x0000000000000011
>> Mar 20 14:28:00 131643 [41401960] -> __osm_trap_rcv_process_request: ERR 
>> 3804: Received trap 17 times consecutively
>> Mar 20 14:28:00 412484 [43C05960] -> __osm_trap_rcv_process_request: 
>> Received Generic Notice type:0x01 num:128 Producer:2 from LID:0x0152 
>> TID:0x0000000000000012
>> Mar 20 14:28:00 412528 [43C05960] -> __osm_trap_rcv_process_request: ERR 
>> 3804: Received trap 18 times consecutively
>> Mar 20 14:28:00 436877 [44606960] -> __osm_trap_rcv_process_request: 
>> Received Generic Notice type:0x01 num:128 Producer:2 from LID:0x0152 
>> TID:0x0000000000000013
>> Mar 20 14:28:00 436921 [44606960] -> __osm_trap_rcv_process_request: ERR 
>> 3804: Received trap 19 times consecutively
>> Mar 20 14:28:00 458745 [42803960] -> __osm_trap_rcv_process_request: 
>> Received Generic Notice type:0x01 num:128 Producer:2 from LID:0x0152 
>> TID:0x0000000000000014
>> Mar 20 14:28:00 458816 [42803960] -> __osm_trap_rcv_process_request: ERR 
>> 3804: Received trap 20 times consecutively
>> Mar 20 14:28:00 480551 [41E02960] -> __osm_trap_rcv_process_request: 
>> Received Generic Notice type:0x01 num:128 Producer:2 from LID:0x0152 
>> TID:0x0000000000000015
>> Mar 20 14:28:00 480599 [41E02960] -> __osm_trap_rcv_process_request: ERR 
>> 3804: Received trap 21 times consecutively
>> Mar 20 14:28:00 695340 [45A08960] -> __osm_trap_rcv_process_request: 
>> Received Generic Notice type:0x01 num:128 Producer:2 from LID:0x0152 
>> TID:0x0000000000000016
>> Mar 20 14:28:00 695386 [45A08960] -> __osm_trap_rcv_process_request: ERR 
>> 3804: Received trap 22 times consecutively
>> Mar 20 14:28:00 695726 [43204960] -> __osm_trap_rcv_process_request: 
>> Received Generic Notice type:0x01 num:128 Producer:2 from LID:0x0148 
>> TID:0x0000000000000072
>> Mar 20 14:28:00 695886 [43204960] -> osm_report_notice: Reporting Generic 
>> Notice type:1 num:128 from LID:0x0148 
>> GID:0xfe80000000000000,0x0005ad00000281b3
>> Mar 20 14:28:00 719764 [41401960] -> __osm_trap_rcv_process_request: 
>> Received Generic Notice type:0x01 num:128 Producer:2 from LID:0x0152 
>> TID:0x0000000000000017
>> Mar 20 14:28:00 719825 [41401960] -> __osm_trap_rcv_process_request: ERR 
>> 3804: Received trap 23 times consecutively
>> Mar 20 14:28:00 743680 [43204960] -> __osm_trap_rcv_process_request: 
>> Received Generic Notice type:0x01 num:128 Producer:2 from LID:0x0152 
>> TID:0x0000000000000018
>> Mar 20 14:28:00 743775 [43204960] -> __osm_trap_rcv_process_request: ERR 
>> 3804: Received trap 24 times consecutively
>> Mar 20 14:28:00 763599 [45007960] -> __osm_trap_rcv_process_request: 
>> Received Generic Notice type:0x01 num:128 Producer:2 from LID:0x0152 
>> TID:0x0000000000000019
>> Mar 20 14:28:00 763654 [45007960] -> __osm_trap_rcv_process_request: ERR 
>> 3804: Received trap 25 times consecutively
>> Mar 20 14:28:00 813393 [43C05960] -> __osm_trap_rcv_process_request: 
>> Received Generic Notice type:0x01 num:128 Producer:2 from LID:0x0152 
>> TID:0x000000000000001a
>> Mar 20 14:28:00 813473 [43C05960] -> __osm_trap_rcv_process_request: ERR 
>> 3804: Received trap 26 times consecutively
>> Mar 20 14:28:00 831287 [45A08960] -> __osm_trap_rcv_process_request: 
>> Received Generic Notice type:0x01 num:128 Producer:2 from LID:0x0152 
>> TID:0x000000000000001b
>> Mar 20 14:28:00 831302 [44606960] -> __osm_trap_rcv_process_request: 
>> Received Generic Notice type:0x01 num:128 Producer:2 from LID:0x0148 
>> TID:0x0000000000000073
>> Mar 20 14:28:00 831383 [45A08960] -> __osm_trap_rcv_process_request: ERR 
>> 3804: Received trap 27 times consecutively
>> Mar 20 14:28:00 831424 [44606960] -> osm_report_notice: Reporting Generic 
>> Notice type:1 num:128 from LID:0x0148 
>> GID:0xfe80000000000000,0x0005ad00000281b3
>> Mar 20 14:28:00 841593 [41E02960] -> __osm_trap_rcv_process_request: 
>> Received Generic Notice type:0x01 num:128 Producer:2 from LID:0x0152 
>> TID:0x000000000000001c
>> Mar 20 14:28:00 841644 [41E02960] -> __osm_trap_rcv_process_request: ERR 
>> 3804: Received trap 28 times consecutively
>> Mar 20 14:28:01 050511 [41E02960] -> osm_report_notice: Reporting Generic 
>> Notice type:3 num:64 from LID:0x0092 
>> GID:0xfe80000000000000,0x0005ad0000024bbb
>> Mar 20 14:28:01 050529 [41E02960] -> Discovered new port with 
>> GUID:0x0005ad000002510b LID range [0xB5,0xB5] of node:saguaro-23-6 HCA-1
>> Mar 20 14:28:01 050535 [41E02960] -> osm_report_notice: Reporting Generic 
>> Notice type:3 num:64 from LID:0x0092 
>> GID:0xfe80000000000000,0x0005ad0000024bbb
>> Mar 20 14:28:01 050542 [41E02960] -> Discovered new port with 
>> GUID:0x0005ad0000024d47 LID range [0xB6,0xB6] of node:saguaro-23-7 HCA-1
>> Mar 20 14:28:01 050547 [41E02960] -> osm_report_notice: Reporting Generic 
>> Notice type:3 num:64 from LID:0x0092 
>> GID:0xfe80000000000000,0x0005ad0000024bbb
>> Mar 20 14:28:01 050554 [41E02960] -> Discovered new port with 
>> GUID:0x0005ad0000024d8b LID range [0xB7,0xB7] of node:saguaro-23-8 HCA-1
>> Mar 20 14:28:01 081322 [41E02960] -> osm_ucast_mgr_process: Min Hop Tables 
>> configured on all switches
>> Mar 20 14:28:01 142873 [43204960] -> SUBNET UP
>> Mar 20 14:28:01 460275 [44606960] -> __osm_trap_rcv_process_request: 
>> Received Generic Notice type:0x01 num:128 Producer:2 from LID:0x0152 
>> TID:0x000000000000001d
>> Mar 20 14:28:01 460358 [44606960] -> __osm_trap_rcv_process_request: ERR 
>> 3804: Received trap 29 times consecutively
>> Mar 20 14:28:01 488474 [45007960] -> osm_ucast_mgr_process: Min Hop Tables 
>> configured on all switches
>> Mar 20 14:28:01 538634 [4780B960] -> __osm_sm_mad_ctrl_rcv_callback: ERR 
>> 3111: Error status = 0x1C00
>> Mar 20 14:28:01 538712 [4780B960] -> SMP dump:
>>                 base_ver................0x1
>>                 mgmt_class..............0x81
>>                 class_ver...............0x1
>>                 method..................0x81 (SubnGetResp)
>>                 D bit...................0x1
>>                 status..................0x1C00
>>                 hop_ptr.................0x0
>>                 hop_count...............0x4
>>                 trans_id................0x898d1
>>                 attr_id.................0x15 (PortInfo)
>>                 resv....................0x0
>>                 attr_mod................0x16
>>                 m_key...................0x0000000000000000
>>                 dr_slid.................0xFFFF
>>                 dr_dlid.................0xFFFF
>> 
>>                 Initial path: [0][1][11][1][6]
>>                 Return path:  [0][9][18][D][3]
>>                 Reserved:     [0][0][0][0][0][0][0]
>> 
>>                 00 00 00 00 00 00 00 00   00 00 00 00 00 00 00 00
>> 
>>                 00 00 00 00 00 00 00 00   00 00 00 00 03 03 03 02
>> 
>>                 11 42 00 11 40 40 00 08   08 04 F2 40 00 00 00 00
>> 
>>                 00 00 00 00 00 88 00 00   00 00 00 00 00 00 00 00
>> 
>> Mar 20 14:28:01 538752 [42803960] -> osm_pi_rcv_process_set: ERR 0F10: 
>> Received error status for SetResp()
>> Mar 20 14:28:01 538758 [4780B960] -> __osm_sm_mad_ctrl_rcv_callback: ERR 
>> 3111: Error status = 0x1C00
>> Mar 20 14:28:01 538767 [42803960] -> PortInfo dump:
>>                 port number.............0x16
>>                 node_guid...............0x0005ad00000281b3
>>                 port_guid...............0x0005ad00000281b3
>>                 m_key...................0x0000000000000000
>>                 subnet_prefix...........0x0000000000000000
>>                 base_lid................0x0
>>                 master_sm_base_lid......0x0
>>                 capability_mask.........0x0
>>                 diag_code...............0x0
>>                 m_key_lease_period......0x0
>>                 local_port_num..........0x3
>>                 link_width_enabled......0x3
>>                 link_width_supported....0x3
>>                 link_width_active.......0x2
>>                 link_speed_supported....0x1
>>                 port_state..............DOWN
>>                 state_info2.............0x42
>>                 m_key_protect_bits......0x0
>>                 lmc.....................0x0
>>                 link_speed..............0x11
>>                 mtu_smsl................0x40
>>                 vl_cap_init_type........0x40
>>                 vl_high_limit...........0x0
>>                 vl_arb_high_cap.........0x8
>>                 vl_arb_low_cap..........0x8
>>                 init_rep_mtu_cap........0x4
>>                 vl_stall_life...........0xF2
>>                 vl_enforce..............0x40
>>                 m_key_violations........0x0
>>                 p_key_violations........0x0
>>                 q_key_violations........0x0
>>                 guid_cap................0x0
>>                 client_reregister.......0x0
>>                 subnet_timeout..........0x0
>>                 resp_time_value.........0x0
>>                 error_threshold.........0x88
>> Mar 20 14:28:01 538795 [4780B960] -> SMP dump:
>>                 base_ver................0x1
>>                 mgmt_class..............0x81
>>                 class_ver...............0x1
>>                 method..................0x81 (SubnGetResp)
>>                 D bit...................0x1
>>                 status..................0x1C00
>>                 hop_ptr.................0x0
>>                 hop_count...............0x4
>>                 trans_id................0x898d2
>>                 attr_id.................0x15 (PortInfo)
>>                 resv....................0x0
>>                 attr_mod................0x17
>>                 m_key...................0x0000000000000000
>>                 dr_slid.................0xFFFF
>>                 dr_dlid.................0xFFFF
>> 
>>                 Initial path: [0][1][11][1][6]
>>                 Return path:  [0][9][18][D][3]
>>                 Reserved:     [0][0][0][0][0][0][0]
>> 
>>                 00 00 00 00 00 00 00 00   00 00 00 00 00 00 00 00
>> 
>>                 00 00 00 00 00 00 00 00   00 00 00 00 03 03 03 02
>> 
>>                 11 42 00 11 40 40 00 08   08 04 F2 40 00 00 00 00
>> 
>>                 00 00 00 00 00 88 00 00   00 00 00 00 00 00 00 00
>> 
>> Mar 20 14:28:01 538810 [42803960] -> Capabilities Mask:
>> Mar 20 14:28:01 538849 [42803960] -> osm_pi_rcv_process_set: ERR 0F10: 
>> Received error status for SetResp()
>> Mar 20 14:28:01 538856 [42803960] -> PortInfo dump:
>>                 port number.............0x17
>>                 node_guid...............0x0005ad00000281b3
>>                 port_guid...............0x0005ad00000281b3
>>                 m_key...................0x0000000000000000
>>                 subnet_prefix...........0x0000000000000000
>>                 base_lid................0x0
>>                 master_sm_base_lid......0x0
>>                 capability_mask.........0x0
>>                 diag_code...............0x0
>>                 m_key_lease_period......0x0
>>                 local_port_num..........0x3
>>                 link_width_enabled......0x3
>>                 link_width_supported....0x3
>>                 link_width_active.......0x2
>>                 link_speed_supported....0x1
>>                 port_state..............DOWN
>>                 state_info2.............0x42
>>                 m_key_protect_bits......0x0
>>                 lmc.....................0x0
>>                 link_speed..............0x11
>>                 mtu_smsl................0x40
>>                 vl_cap_init_type........0x40
>>                 vl_high_limit...........0x0
>>                 vl_arb_high_cap.........0x8
>>                 vl_arb_low_cap..........0x8
>>                 init_rep_mtu_cap........0x4
>>                 vl_stall_life...........0xF2
>>                 vl_enforce..............0x40
>>                 m_key_violations........0x0
>>                 p_key_violations........0x0
>>                 q_key_violations........0x0
>>                 guid_cap................0x0
>>                 client_reregister.......0x0
>>                 subnet_timeout..........0x0
>>                 resp_time_value.........0x0
>>                 error_threshold.........0x88
>> Mar 20 14:28:01 538871 [42803960] -> Capabilities Mask:
>> Mar 20 14:28:01 539658 [4780B960] -> __osm_sm_mad_ctrl_rcv_callback: ERR 
>> 3111: Error status = 0x1C00
>> Mar 20 14:28:01 539696 [4780B960] -> SMP dump:
>>                 base_ver................0x1
>>                 mgmt_class..............0x81
>>                 class_ver...............0x1
>>                 method..................0x81 (SubnGetResp)
>>                 D bit...................0x1
>>                 status..................0x1C00
>>                 hop_ptr.................0x0
>>                 hop_count...............0x5
>>                 trans_id................0x898d3
>>                 attr_id.................0x15 (PortInfo)
>>                 resv....................0x0
>>                 attr_mod................0x11
>>                 m_key...................0x0000000000000000
>>                 dr_slid.................0xFFFF
>>                 dr_dlid.................0xFFFF
>> 
>>                 Initial path: [0][1][15][1][4][17]
>>                 Return path:  [0][9][18][D][1][16]
>>                 Reserved:     [0][0][0][0][0][0][0]
>> 
>>                 00 00 00 00 00 00 00 00   00 00 00 00 00 00 00 00
>> 
>>                 00 00 00 00 00 00 00 00   00 00 00 00 16 02 03 02
>> 
>>                 11 42 00 11 40 40 00 08   08 04 F2 40 00 00 00 00
>> 
>>                 00 00 00 00 00 88 00 00   00 00 00 00 00 00 00 00
>> 
>> Mar 20 14:28:01 539778 [45A08960] -> osm_pi_rcv_process_set: Received error 
>> status 0x1c for SetResp() during ACTIVE transition
>> Mar 20 14:28:01 539784 [4780B960] -> __osm_sm_mad_ctrl_rcv_callback: ERR 
>> 3111: Error status = 0x1C00
>> Mar 20 14:28:01 539798 [45A08960] -> PortInfo dump:
>>                 port number.............0x11
>>                 node_guid...............0x0005ad0000027c84
>>                 port_guid...............0x0005ad0000027c84
>>                 m_key...................0x0000000000000000
>>                 subnet_prefix...........0x0000000000000000
>>                 base_lid................0x0
>>                 master_sm_base_lid......0x0
>>                 capability_mask.........0x0
>>                 diag_code...............0x0
>>                 m_key_lease_period......0x0
>>                 local_port_num..........0x16
>>                 link_width_enabled......0x2
>>                 link_width_supported....0x3
>>                 link_width_active.......0x2
>>                 link_speed_supported....0x1
>>                 port_state..............DOWN
>>                 state_info2.............0x42
>>                 m_key_protect_bits......0x0
>>                 lmc.....................0x0
>>                 link_speed..............0x11
>>                 mtu_smsl................0x40
>>                 vl_cap_init_type........0x40
>>                 vl_high_limit...........0x0
>>                 vl_arb_high_cap.........0x8
>>                 vl_arb_low_cap..........0x8
>>                 init_rep_mtu_cap........0x4
>>                 vl_stall_life...........0xF2
>>                 vl_enforce..............0x40
>>                 m_key_violations........0x0
>>                 p_key_violations........0x0
>>                 q_key_violations........0x0
>>                 guid_cap................0x0
>>                 client_reregister.......0x0
>>                 subnet_timeout..........0x0
>>                 resp_time_value.........0x0
>>                 error_threshold.........0x88
>> Mar 20 14:28:01 539834 [45A08960] -> Capabilities Mask:
>> Mar 20 14:28:01 539844 [4780B960] -> SMP dump:
>>                 base_ver................0x1
>>                 mgmt_class..............0x81
>>                 class_ver...............0x1
>>                 method..................0x81 (SubnGetResp)
>>                 D bit...................0x1
>>                 status..................0x1C00
>>                 hop_ptr.................0x0
>>                 hop_count...............0x5
>>                 trans_id................0x898d4
>>                 attr_id.................0x15 (PortInfo)
>>                 resv....................0x0
>>                 attr_mod................0x12
>>                 m_key...................0x0000000000000000
>>                 dr_slid.................0xFFFF
>>                 dr_dlid.................0xFFFF
>> 
>>                 Initial path: [0][1][15][1][4][17]
>>                 Return path:  [0][9][18][D][1][16]
>>                 Reserved:     [0][0][0][0][0][0][0]
>> 
>>                 00 00 00 00 00 00 00 00   00 00 00 00 00 00 00 00
>> 
>>                 00 00 00 00 00 00 00 00   00 00 00 00 16 02 03 02
>> 
>>                 11 42 00 11 40 40 00 08   08 04 F2 40 00 00 00 00
>> 
>>                 00 00 00 00 00 88 00 00   00 00 00 00 00 00 00 00
>> 
>> Mar 20 14:28:01 539903 [45007960] -> osm_pi_rcv_process_set: Received error 
>> status 0x1c for SetResp() during ACTIVE transition
>> Mar 20 14:28:01 539908 [45007960] -> PortInfo dump:
>>                 port number.............0x12
>>                 node_guid...............0x0005ad0000027c84
>>                 port_guid...............0x0005ad0000027c84
>>                 m_key...................0x0000000000000000
>>                 subnet_prefix...........0x0000000000000000
>>                 base_lid................0x0
>>                 master_sm_base_lid......0x0
>>                 capability_mask.........0x0
>>                 diag_code...............0x0
>>                 m_key_lease_period......0x0
>>                 local_port_num..........0x16
>>                 link_width_enabled......0x2
>>                 link_width_supported....0x3
>>                 link_width_active.......0x2
>>                 link_speed_supported....0x1
>>                 port_state..............DOWN
>>                 state_info2.............0x42
>>                 m_key_protect_bits......0x0
>>                 lmc.....................0x0
>>                 link_speed..............0x11
>>                 mtu_smsl................0x40
>>                 vl_cap_init_type........0x40
>>                 vl_high_limit...........0x0
>>                 vl_arb_high_cap.........0x8
>>                 vl_arb_low_cap..........0x8
>>                 init_rep_mtu_cap........0x4
>>                 vl_stall_life...........0xF2
>>                 vl_enforce..............0x40
>>                 m_key_violations........0x0
>>                 p_key_violations........0x0
>>                 q_key_violations........0x0
>>                 guid_cap................0x0
>>                 client_reregister.......0x0
>>                 subnet_timeout..........0x0
>>                 resp_time_value.........0x0
>>                 error_threshold.........0x88
>> Mar 20 14:28:01 539924 [45007960] -> Capabilities Mask:
>> Mar 20 14:28:01 545091 [45007960] -> SUBNET UP
>> Mar 20 14:28:01 652647 [43204960] -> __osm_trap_rcv_process_request: 
>> Received Generic Notice type:0x01 num:128 Producer:2 from LID:0x001B 
>> TID:0x0000000000000084
>> Mar 20 14:28:01 652864 [43204960] -> osm_report_notice: Reporting Generic 
>> Notice type:1 num:128 from LID:0x001B 
>> GID:0xfe80000000000000,0x0005ad00000281a7
>> Mar 20 14:28:01 879631 [44606960] -> __osm_trap_rcv_process_request: 
>> Received Generic Notice type:0x01 num:128 Producer:2 from LID:0x0148 
>> TID:0x0000000000000074
>> Mar 20 14:28:01 880104 [44606960] -> osm_report_notice: Reporting Generic 
>> Notice type:1 num:128 from LID:0x0148 
>> GID:0xfe80000000000000,0x0005ad00000281b3
>> Mar 20 14:28:01 962839 [44606960] -> __osm_trap_rcv_process_request: 
>> Received Generic Notice type:0x01 num:128 Producer:2 from LID:0x0148 
>> TID:0x0000000000000075
>> Mar 20 14:28:01 965155 [44606960] -> osm_report_notice: Reporting Generic 
>> Notice type:1 num:128 from LID:0x0148 
>> GID:0xfe80000000000000,0x0005ad00000281b3
>> Mar 20 14:28:02 006432 [41401960] -> __osm_trap_rcv_process_request: 
>> Received Generic Notice type:0x01 num:128 Producer:2 from LID:0x001B 
>> TID:0x0000000000000085
>> Mar 20 14:28:02 030610 [42803960] -> osm_ucast_mgr_process: Min Hop Tables 
>> configured on all switches
>> Mar 20 14:28:02 068999 [41401960] -> osm_report_notice: Reporting Generic 
>> Notice type:1 num:128 from LID:0x001B 
>> GID:0xfe80000000000000,0x0005ad00000281a7
>> Mar 20 14:28:02 081130 [4780B960] -> __osm_sm_mad_ctrl_rcv_callback: ERR 
>> 3111: Error status = 0x1C00
>> Mar 20 14:28:02 081198 [4780B960] -> SMP dump:
>>                 base_ver................0x1
>>                 mgmt_class..............0x81
>>                 class_ver...............0x1
>>                 method..................0x81 (SubnGetResp)
>>                 D bit...................0x1
>>                 status..................0x1C00
>>                 hop_ptr.................0x0
>>                 hop_count...............0x4
>>                 trans_id................0x8a604
>>                 attr_id.................0x15 (PortInfo)
>>                 resv....................0x0
>>                 attr_mod................0x16
>>                 m_key...................0x0000000000000000
>>                 dr_slid.................0xFFFF
>>                 dr_dlid.................0xFFFF
>> 
>>                 Initial path: [0][1][11][4][4]
>>                 Return path:  [0][9][18][D][4]
>>                 Reserved:     [0][0][0][0][0][0][0]
>> 
>>                 00 00 00 00 00 00 00 00   00 00 00 00 00 00 00 00
>> 
>>                 00 00 00 00 00 00 00 00   00 00 00 00 04 03 03 02
>> 
>>                 14 52 00 11 40 40 00 08   08 04 F2 40 00 00 00 00
>> 
>>                 00 00 00 00 00 88 00 00   00 00 00 00 00 00 00 00
>> 
>> Mar 20 14:28:02 081249 [43204960] -> osm_pi_rcv_process_set: Received error 
>> status 0x1c for SetResp() during ACTIVE transition
>> Mar 20 14:28:02 081257 [43204960] -> PortInfo dump:
>>                 port number.............0x16
>>                 node_guid...............0x0005ad00000281b3
>>                 port_guid...............0x0005ad00000281b3
>>                 m_key...................0x0000000000000000
>>                 subnet_prefix...........0x0000000000000000
>>                 base_lid................0x0
>>                 master_sm_base_lid......0x0
>>                 capability_mask.........0x0
>>                 diag_code...............0x0
>>                 m_key_lease_period......0x0
>>                 local_port_num..........0x4
>>                 link_width_enabled......0x3
>>                 link_width_supported....0x3
>>                 link_width_active.......0x2
>>                 link_speed_supported....0x1
>>                 port_state..............ACTIVE
>>                 state_info2.............0x52
>>                 m_key_protect_bits......0x0
>>                 lmc.....................0x0
>>                 link_speed..............0x11
>>                 mtu_smsl................0x40
>>                 vl_cap_init_type........0x40
>>                 vl_high_limit...........0x0
>>                 vl_arb_high_cap.........0x8
>>                 vl_arb_low_cap..........0x8
>>                 init_rep_mtu_cap........0x4
>>                 vl_stall_life...........0xF2
>>                 vl_enforce..............0x40
>>                 m_key_violations........0x0
>>                 p_key_violations........0x0
>>                 q_key_violations........0x0
>>                 guid_cap................0x0
>>                 client_reregister.......0x0
>>                 subnet_timeout..........0x0
>>                 resp_time_value.........0x0
>>                 error_threshold.........0x88
>> Mar 20 14:28:02 081275 [43204960] -> Capabilities Mask:
>> Mar 20 14:28:02 081650 [4780B960] -> __osm_sm_mad_ctrl_rcv_callback: ERR 
>> 3111: Error status = 0x1C00
>> Mar 20 14:28:02 081713 [4780B960] -> SMP dump:
>>                 base_ver................0x1
>>                 mgmt_class..............0x81
>>                 class_ver...............0x1
>>                 method..................0x81 (SubnGetResp)
>>                 D bit...................0x1
>>                 status..................0x1C00
>>                 hop_ptr.................0x0
>>                 hop_count...............0x4
>>                 trans_id................0x8a605
>>                 attr_id.................0x15 (PortInfo)
>>                 resv....................0x0
>>                 attr_mod................0x17
>>                 m_key...................0x0000000000000000
>>                 dr_slid.................0xFFFF
>>                 dr_dlid.................0xFFFF
>> 
>>                 Initial path: [0][1][11][4][4]
>>                 Return path:  [0][9][18][D][4]
>>                 Reserved:     [0][0][0][0][0][0][0]
>> 
>>                 00 00 00 00 00 00 00 00   00 00 00 00 00 00 00 00
>> 
>>                 00 00 00 00 00 00 00 00   00 00 00 00 04 03 03 02
>> 
>>                 14 52 00 11 40 40 00 08   08 04 F2 40 00 00 00 00
>> 
>>                 00 00 00 00 00 88 00 00   00 00 00 00 00 00 00 00
>> 
>> Mar 20 14:28:02 081782 [43C05960] -> osm_pi_rcv_process_set: Received error 
>> status 0x1c for SetResp() during ACTIVE transition
>> Mar 20 14:28:02 081787 [43C05960] -> PortInfo dump:
>>                 port number.............0x17
>>                 node_guid...............0x0005ad00000281b3
>>                 port_guid...............0x0005ad00000281b3
>>                 m_key...................0x0000000000000000
>>                 subnet_prefix...........0x0000000000000000
>>                 base_lid................0x0
>>                 master_sm_base_lid......0x0
>>                 capability_mask.........0x0
>>                 diag_code...............0x0
>>                 m_key_lease_period......0x0
>>                 local_port_num..........0x4
>>                 link_width_enabled......0x3
>>                 link_width_supported....0x3
>>                 link_width_active.......0x2
>>                 link_speed_supported....0x1
>>                 port_state..............ACTIVE
>>                 state_info2.............0x52
>>                 m_key_protect_bits......0x0
>>                 lmc.....................0x0
>>                 link_speed..............0x11
>>                 mtu_smsl................0x40
>>                 vl_cap_init_type........0x40
>>                 vl_high_limit...........0x0
>>                 vl_arb_high_cap.........0x8
>>                 vl_arb_low_cap..........0x8
>>                 init_rep_mtu_cap........0x4
>>                 vl_stall_life...........0xF2
>>                 vl_enforce..............0x40
>>                 m_key_violations........0x0
>>                 p_key_violations........0x0
>>                 q_key_violations........0x0
>>                 guid_cap................0x0
>>                 client_reregister.......0x0
>>                 subnet_timeout..........0x0
>>                 resp_time_value.........0x0
>>                 error_threshold.........0x88
>> Mar 20 14:28:02 081802 [43C05960] -> Capabilities Mask:
>> Mar 20 14:28:02 087435 [45A08960] -> SUBNET UP
>> Mar 20 14:28:02 170696 [41401960] -> __osm_trap_rcv_process_request: 
>> Received Generic Notice type:0x01 num:128 Producer:2 from LID:0x001B 
>> TID:0x0000000000000086
>> Mar 20 14:28:02 170819 [41401960] -> osm_report_notice: Reporting Generic 
>> Notice type:1 num:128 from LID:0x001B 
>> GID:0xfe80000000000000,0x0005ad00000281a7
>> Mar 20 14:28:02 459228 [41E02960] -> osm_ucast_mgr_process: Min Hop Tables 
>> configured on all switches
>> Mar 20 14:28:02 500761 [43C05960] -> __osm_trap_rcv_process_request: 
>> Received Generic Notice type:0x01 num:128 Producer:2 from LID:0x001B 
>> TID:0x0000000000000087
>> Mar 20 14:28:02 500979 [43C05960] -> osm_report_notice: Reporting Generic 
>> Notice type:1 num:128 from LID:0x001B 
>> GID:0xfe80000000000000,0x0005ad00000281a7
>> Mar 20 14:28:02 510190 [4780B960] -> __osm_sm_mad_ctrl_rcv_callback: ERR 
>> 3111: Error status = 0x1C00
>> Mar 20 14:28:02 510258 [4780B960] -> SMP dump:
>>                 base_ver................0x1
>>                 mgmt_class..............0x81
>>                 class_ver...............0x1
>>                 method..................0x81 (SubnGetResp)
>>                 D bit...................0x1
>>                 status..................0x1C00
>>                 hop_ptr.................0x0
>>                 hop_count...............0x4
>>                 trans_id................0x8b330
>>                 attr_id.................0x15 (PortInfo)
>>                 resv....................0x0
>>                 attr_mod................0x16
>>                 m_key...................0x0000000000000000
>>                 dr_slid.................0xFFFF
>>                 dr_dlid.................0xFFFF
>> 
>>                 Initial path: [0][1][17][1][5]
>>                 Return path:  [0][9][14][D][2]
>>                 Reserved:     [0][0][0][0][0][0][0]
>> 
>>                 00 00 00 00 00 00 00 00   00 00 00 00 00 00 00 00
>> 
>>                 00 00 00 00 00 00 00 00   00 00 00 00 02 03 03 02
>> 
>>                 14 52 00 11 40 40 00 08   08 04 F2 40 00 00 00 00
>> 
>>                 00 00 00 00 00 88 00 00   00 00 00 00 00 00 00 00
>> 
>> Mar 20 14:28:02 510366 [4780B960] -> __osm_sm_mad_ctrl_rcv_callback: ERR 
>> 3111: Error status = 0x1C00
>> Mar 20 14:28:02 510370 [45007960] -> osm_pi_rcv_process_set: Received error 
>> status 0x1c for SetResp() during ACTIVE transition
>> Mar 20 14:28:02 510384 [45007960] -> PortInfo dump:
>>                 port number.............0x16
>>                 node_guid...............0x0005ad00000281a7
>>                 port_guid...............0x0005ad00000281a7
>>                 m_key...................0x0000000000000000
>>                 subnet_prefix...........0x0000000000000000
>>                 base_lid................0x0
>>                 master_sm_base_lid......0x0
>>                 capability_mask.........0x0
>>                 diag_code...............0x0
>>                 m_key_lease_period......0x0
>>                 local_port_num..........0x2
>>                 link_width_enabled......0x3
>>                 link_width_supported....0x3
>>                 link_width_active.......0x2
>>                 link_speed_supported....0x1
>>                 port_state..............ACTIVE
>>                 state_info2.............0x52
>>                 m_key_protect_bits......0x0
>>                 lmc.....................0x0
>>                 link_speed..............0x11
>>                 mtu_smsl................0x40
>>                 vl_cap_init_type........0x40
>>                 vl_high_limit...........0x0
>>                 vl_arb_high_cap.........0x8
>>                 vl_arb_low_cap..........0x8
>>                 init_rep_mtu_cap........0x4
>>                 vl_stall_life...........0xF2
>>                 vl_enforce..............0x40
>>                 m_key_violations........0x0
>>                 p_key_violations........0x0
>>                 q_key_violations........0x0
>>                 guid_cap................0x0
>>                 client_reregister.......0x0
>>                 subnet_timeout..........0x0
>>                 resp_time_value.........0x0
>>                 error_threshold.........0x88
>> Mar 20 14:28:02 510394 [4780B960] -> SMP dump:
>>                 base_ver................0x1
>>                 mgmt_class..............0x81
>>                 class_ver...............0x1
>>                 method..................0x81 (SubnGetResp)
>>                 D bit...................0x1
>>                 status..................0x1C00
>>                 hop_ptr.................0x0
>>                 hop_count...............0x4
>>                 trans_id................0x8b331
>>                 attr_id.................0x15 (PortInfo)
>>                 resv....................0x0
>>                 attr_mod................0x18
>>                 m_key...................0x0000000000000000
>>                 dr_slid.................0xFFFF
>>                 dr_dlid.................0xFFFF
>> 
>>                 Initial path: [0][1][11][3][6]
>>                 Return path:  [0][9][18][F][3]
>>                 Reserved:     [0][0][0][0][0][0][0]
>> 
>>                 00 00 00 00 00 00 00 00   00 00 00 00 00 00 00 00
>> 
>>                 00 00 00 00 00 00 00 00   00 00 00 00 03 03 03 02
>> 
>>                 14 52 00 11 40 40 00 08   08 04 F2 40 00 00 00 00
>> 
>>                 00 00 00 00 00 88 00 00   00 00 00 00 00 00 00 00
>> 
>> Mar 20 14:28:02 510398 [45007960] -> Capabilities Mask:
>> Mar 20 14:28:02 510481 [41401960] -> osm_pi_rcv_process_set: Received error 
>> status 0x1c for SetResp() during ACTIVE transition
>> Mar 20 14:28:02 510491 [41401960] -> PortInfo dump:
>>                 port number.............0x18
>>                 node_guid...............0x0005ad00000281b3
>>                 port_guid...............0x0005ad00000281b3
>>                 m_key...................0x0000000000000000
>>                 subnet_prefix...........0x0000000000000000
>>                 base_lid................0x0
>>                 master_sm_base_lid......0x0
>>                 capability_mask.........0x0
>>                 diag_code...............0x0
>>                 m_key_lease_period......0x0
>>                 local_port_num..........0x3
>>                 link_width_enabled......0x3
>>                 link_width_supported....0x3
>>                 link_width_active.......0x2
>>                 link_speed_supported....0x1
>>                 port_state..............ACTIVE
>>                 state_info2.............0x52
>>                 m_key_protect_bits......0x0
>>                 lmc.....................0x0
>>                 link_speed..............0x11
>>                 mtu_smsl................0x40
>>                 vl_cap_init_type........0x40
>>                 vl_high_limit...........0x0
>>                 vl_arb_high_cap.........0x8
>>                 vl_arb_low_cap..........0x8
>>                 init_rep_mtu_cap........0x4
>>                 vl_stall_life...........0xF2
>>                 vl_enforce..............0x40
>>                 m_key_violations........0x0
>>                 p_key_violations........0x0
>>                 q_key_violations........0x0
>>                 guid_cap................0x0
>>                 client_reregister.......0x0
>>                 subnet_timeout..........0x0
>>                 resp_time_value.........0x0
>>                 error_threshold.........0x88
>> Mar 20 14:28:02 510509 [41401960] -> Capabilities Mask:
>> Mar 20 14:28:02 510511 [42803960] -> __osm_trap_rcv_process_request: 
>> Received Generic Notice type:0x01 num:128 Producer:2 from LID:0x001B 
>> TID:0x0000000000000088
>> Mar 20 14:28:02 515576 [41401960] -> SUBNET UP
>> Mar 20 14:28:02 515695 [42803960] -> osm_report_notice: Reporting Generic 
>> Notice type:1 num:128 from LID:0x001B 
>> GID:0xfe80000000000000,0x0005ad00000281a7
>> Mar 20 14:28:02 532552 [45A08960] -> __osm_trap_rcv_process_request: 
>> Received Generic Notice type:0x01 num:128 Producer:2 from LID:0x001B 
>> TID:0x0000000000000089
>> Mar 20 14:28:02 538569 [45A08960] -> osm_report_notice: Reporting Generic 
>> Notice type:1 num:128 from LID:0x001B 
>> GID:0xfe80000000000000,0x0005ad00000281a7
>> Mar 20 14:28:02 695997 [43204960] -> __osm_trap_rcv_process_request: 
>> Received Generic Notice type:0x01 num:128 Producer:2 from LID:0x0152 
>> TID:0x000000000000001e
>> Mar 20 14:28:02 696096 [43204960] -> __osm_trap_rcv_process_request: ERR 
>> 3804: Received trap 30 times consecutively
>> Mar 20 14:28:02 918226 [45007960] -> osm_ucast_mgr_process: Min Hop Tables 
>> configured on all switches
>> Mar 20 14:28:02 975244 [43204960] -> SUBNET UP
>> Mar 20 14:28:03 325494 [43204960] -> osm_ucast_mgr_process: Min Hop Tables 
>> configured on all switches
>> Mar 20 14:28:03 379145 [41401960] -> SUBNET UP
>> Mar 20 14:29:12 561841 [41401960] -> __osm_trap_rcv_process_request: 
>> Received Generic Notice type:0x01 num:128 Producer:2 from LID:0x001F 
>> TID:0x0000000000000019
>> Mar 20 14:29:12 562033 [41401960] -> osm_report_notice: Reporting Generic 
>> Notice type:1 num:128 from LID:0x001F 
>> GID:0xfe80000000000000,0x0005ad0000027c56
>> Mar 20 14:29:12 562751 [42803960] -> __osm_trap_rcv_process_request: 
>> Received Generic Notice type:0x01 num:128 Producer:2 from LID:0x001F 
>> TID:0x000000000000001a
>> Mar 20 14:29:12 562902 [42803960] -> osm_report_notice: Reporting Generic 
>> Notice type:1 num:128 from LID:0x001F 
>> GID:0xfe80000000000000,0x0005ad0000027c56
>> Mar 20 14:29:12 571346 [42803960] -> __osm_trap_rcv_process_request: 
>> Received Generic Notice type:0x01 num:128 Producer:2 from LID:0x0084 
>> TID:0x0000000000000018
>> Mar 20 14:29:12 571569 [42803960] -> osm_report_notice: Reporting Generic 
>> Notice type:1 num:128 from LID:0x0084 
>> GID:0xfe80000000000000,0x0005ad0000027c70
>> Mar 20 14:29:12 914371 [42803960] -> __osm_trap_rcv_process_request: 
>> Received Generic Notice type:0x01 num:128 Producer:2 from LID:0x001F 
>> TID:0x000000000000001b
>> Mar 20 14:29:12 916287 [44606960] -> osm_report_notice: Reporting Generic 
>> Notice type:3 num:64 from LID:0x0092 
>> GID:0xfe80000000000000,0x0005ad0000024bbb
>> Mar 20 14:29:12 916297 [44606960] -> Discovered new port with 
>> GUID:0x0005ad000002502f LID range [0x2,0x2] of node:Topspin IB-DC
>> Mar 20 14:29:12 946985 [44606960] -> osm_ucast_mgr_process: Min Hop Tables 
>> configured on all switches
>> Mar 20 14:29:12 976839 [42803960] -> osm_report_notice: Reporting Generic 
>> Notice type:1 num:128 from LID:0x001F 
>> GID:0xfe80000000000000,0x0005ad0000027c56
>> Mar 20 14:29:12 987963 [4780B960] -> __osm_sm_mad_ctrl_rcv_callback: ERR 
>> 3111: Error status = 0x1C00
>> Mar 20 14:29:12 988004 [4780B960] -> SMP dump:
>>                 base_ver................0x1
>>                 mgmt_class..............0x81
>>                 class_ver...............0x1
>>                 method..................0x81 (SubnGetResp)
>>                 D bit...................0x1
>>                 status..................0x1C00
>>                 hop_ptr.................0x0
>>                 hop_count...............0x5
>>                 trans_id................0x8dbb2
>>                 attr_id.................0x15 (PortInfo)
>>                 resv....................0x0
>>                 attr_mod................0xD
>>                 m_key...................0x0000000000000000
>>                 dr_slid.................0xFFFF
>>                 dr_dlid.................0xFFFF
>> 
>>                 Initial path: [0][1][11][2][4][D]
>>                 Return path:  [0][9][18][E][1][12]
>>                 Reserved:     [0][0][0][0][0][0][0]
>> 
>>                 00 00 00 00 00 00 00 00   00 00 00 00 00 00 00 00
>> 
>>                 00 00 00 00 00 00 00 00   00 00 00 00 12 03 03 02
>> 
>>                 14 52 00 11 40 40 00 08   08 04 2C 4C 00 00 00 00
>> 
>>                 00 00 00 00 00 88 00 00   00 00 00 00 00 00 00 00
>> 
>> Mar 20 14:29:12 988078 [43C05960] -> osm_pi_rcv_process_set: Received error 
>> status 0x1c for SetResp() during ACTIVE transition
>> Mar 20 14:29:12 988089 [43C05960] -> PortInfo dump:
>>                 port number.............0xD
>>                 node_guid...............0x0005ad0000027c70
>>                 port_guid...............0x0005ad0000027c70
>>                 m_key...................0x0000000000000000
>>                 subnet_prefix...........0x0000000000000000
>>                 base_lid................0x0
>>                 master_sm_base_lid......0x0
>>                 capability_mask.........0x0
>>                 diag_code...............0x0
>>                 m_key_lease_period......0x0
>>                 local_port_num..........0x12
>>                 link_width_enabled......0x3
>>                 link_width_supported....0x3
>>                 link_width_active.......0x2
>>                 link_speed_supported....0x1
>>                 port_state..............ACTIVE
>>                 state_info2.............0x52
>>                 m_key_protect_bits......0x0
>>                 lmc.....................0x0
>>                 link_speed..............0x11
>>                 mtu_smsl................0x40
>>                 vl_cap_init_type........0x40
>>                 vl_high_limit...........0x0
>>                 vl_arb_high_cap.........0x8
>>                 vl_arb_low_cap..........0x8
>>                 init_rep_mtu_cap........0x4
>>                 vl_stall_life...........0x2C
>>                 vl_enforce..............0x4C
>>                 m_key_violations........0x0
>>                 p_key_violations........0x0
>>                 q_key_violations........0x0
>>                 guid_cap................0x0
>>                 client_reregister.......0x0
>>                 subnet_timeout..........0x0
>>                 resp_time_value.........0x0
>>                 error_threshold.........0x88
>> Mar 20 14:29:12 988105 [43C05960] -> Capabilities Mask:
>> Mar 20 14:29:12 993136 [44606960] -> SUBNET UP
>> Mar 20 14:29:13 300755 [41E02960] -> __osm_trap_rcv_process_request: 
>> Received Generic Notice type:0x04 num:144 Producer:1 from LID:0x0002 
>> TID:0x0000000000000000
>> Mar 20 14:29:13 300874 [41E02960] -> osm_report_notice: Reporting Generic 
>> Notice type:4 num:144 from LID:0x0002 
>> GID:0xfe80000000000000,0x0005ad000002502f
>> Mar 20 14:29:13 338077 [41E02960] -> osm_report_notice: Reporting Generic 
>> Notice type:3 num:64 from LID:0x0092 
>> GID:0xfe80000000000000,0x0005ad0000024bbb
>> Mar 20 14:29:13 338099 [41E02960] -> Discovered new port with 
>> GUID:0x0005ad000002516f LID range [0xBA,0xBA] of node:Topspin IB-DC
>> Mar 20 14:29:13 368879 [41E02960] -> osm_ucast_mgr_process: Min Hop Tables 
>> configured on all switches
>> Mar 20 14:29:13 431763 [4780B960] -> __osm_sm_mad_ctrl_rcv_callback: ERR 
>> 3111: Error status = 0x1C00
>> Mar 20 14:29:13 431806 [4780B960] -> SMP dump:
>>                 base_ver................0x1
>>                 mgmt_class..............0x81
>>                 class_ver...............0x1
>>                 method..................0x81 (SubnGetResp)
>>                 D bit...................0x1
>>                 status..................0x1C00
>>                 hop_ptr.................0x0
>>                 hop_count...............0x5
>>                 trans_id................0x8e8e9
>>                 attr_id.................0x15 (PortInfo)
>>                 resv....................0x0
>>                 attr_mod................0xA
>>                 m_key...................0x0000000000000000
>>                 dr_slid.................0xFFFF
>>                 dr_dlid.................0xFFFF
>> 
>>                 Initial path: [0][1][14][1][6][12]
>>                 Return path:  [0][9][15][D][3][11]
>>                 Reserved:     [0][0][0][0][0][0][0]
>> 
>>                 00 00 00 00 00 00 00 00   00 00 00 00 00 00 00 00
>> 
>>                 00 00 00 00 00 00 00 00   00 00 00 00 11 03 03 02
>> 
>>                 14 52 00 11 40 40 00 08   08 04 2C 4C 00 00 00 00
>> 
>>                 00 00 00 00 00 88 00 00   00 00 00 00 00 00 00 00
>> 
>> Mar 20 14:29:13 432093 [43204960] -> osm_pi_rcv_process_set: Received error 
>> status 0x1c for SetResp() during ACTIVE transition
>> Mar 20 14:29:13 432116 [43204960] -> PortInfo dump:
>>                 port number.............0xA
>>                 node_guid...............0x0005ad0000027c56
>>                 port_guid...............0x0005ad0000027c56
>>                 m_key...................0x0000000000000000
>>                 subnet_prefix...........0x0000000000000000
>>                 base_lid................0x0
>>                 master_sm_base_lid......0x0
>>                 capability_mask.........0x0
>>                 diag_code...............0x0
>>                 m_key_lease_period......0x0
>>                 local_port_num..........0x11
>>                 link_width_enabled......0x3
>>                 link_width_supported....0x3
>>                 link_width_active.......0x2
>>                 link_speed_supported....0x1
>>                 port_state..............ACTIVE
>>                 state_info2.............0x52
>>                 m_key_protect_bits......0x0
>>                 lmc.....................0x0
>>                 link_speed..............0x11
>>                 mtu_smsl................0x40
>>                 vl_cap_init_type........0x40
>>                 vl_high_limit...........0x0
>>                 vl_arb_high_cap.........0x8
>>                 vl_arb_low_cap..........0x8
>>                 init_rep_mtu_cap........0x4
>>                 vl_stall_life...........0x2C
>>                 vl_enforce..............0x4C
>>                 m_key_violations........0x0
>>                 p_key_violations........0x0
>>                 q_key_violations........0x0
>>                 guid_cap................0x0
>>                 client_reregister.......0x0
>>                 subnet_timeout..........0x0
>>                 resp_time_value.........0x0
>>                 error_threshold.........0x88
>> Mar 20 14:29:13 432135 [43204960] -> Capabilities Mask:
>> Mar 20 14:29:13 437219 [45007960] -> SUBNET UP
>> Mar 20 14:29:13 690992 [42803960] -> __osm_trap_rcv_process_request: 
>> Received Generic Notice type:0x04 num:144 Producer:1 from LID:0x00BA 
>> TID:0x0000000000000000
>> Mar 20 14:29:13 691128 [42803960] -> osm_report_notice: Reporting Generic 
>> Notice type:4 num:144 from LID:0x00BA 
>> GID:0xfe80000000000000,0x0005ad000002516f
>> Mar 20 14:29:13 835017 [44606960] -> osm_ucast_mgr_process: Min Hop Tables 
>> configured on all switches
>> Mar 20 14:29:13 891082 [42803960] -> SUBNET UP
>> Mar 20 14:29:14 235714 [42803960] -> osm_ucast_mgr_process: Min Hop Tables 
>> configured on all switches
>> Mar 20 14:29:14 289127 [41E02960] -> SUBNET UP
>> Mar 20 14:29:17 689267 [43204960] -> __osm_trap_rcv_process_request: 
>> Received Generic Notice type:0x01 num:128 Producer:2 from LID:0x001B 
>> TID:0x000000000000008a
>> Mar 20 14:29:17 689511 [43204960] -> osm_report_notice: Reporting Generic 
>> Notice type:1 num:128 from LID:0x001B 
>> GID:0xfe80000000000000,0x0005ad00000281a7
>> Mar 20 14:29:17 689975 [42803960] -> __osm_trap_rcv_process_request: 
>> Received Generic Notice type:0x01 num:128 Producer:2 from LID:0x0148 
>> TID:0x0000000000000076
>> Mar 20 14:29:17 690097 [42803960] -> osm_report_notice: Reporting Generic 
>> Notice type:1 num:128 from LID:0x0148 
>> GID:0xfe80000000000000,0x0005ad00000281b3
>> Mar 20 14:29:18 025237 [44606960] -> osm_report_notice: Reporting Generic 
>> Notice type:3 num:65 from LID:0x0092 
>> GID:0xfe80000000000000,0x0005ad0000024bbb
>> Mar 20 14:29:18 025255 [44606960] -> Removed port with 
>> GUID:0x0005ad0000024e0b LID range [0xB3,0xB3] of node:saguaro-23-4 HCA-1
>> Mar 20 14:29:18 025273 [44606960] -> osm_report_notice: Reporting Generic 
>> Notice type:3 num:65 from LID:0x0092 
>> GID:0xfe80000000000000,0x0005ad0000024bbb
>> Mar 20 14:29:18 025279 [44606960] -> Removed port with 
>> GUID:0x0005ad000002510b LID range [0xB5,0xB5] of node:saguaro-23-6 HCA-1
>> Mar 20 14:29:18 025296 [44606960] -> osm_report_notice: Reporting Generic 
>> Notice type:3 num:65 from LID:0x0092 
>> GID:0xfe80000000000000,0x0005ad0000024bbb
>> Mar 20 14:29:18 025300 [44606960] -> Removed port with 
>> GUID:0x0005ad000002511b LID range [0xA6,0xA6] of node:saguaro-22-1 HCA-1
>> Mar 20 14:29:18 025317 [44606960] -> osm_report_notice: Reporting Generic 
>> Notice type:3 num:65 from LID:0x0092 
>> GID:0xfe80000000000000,0x0005ad0000024bbb
>> Mar 20 14:29:18 025323 [44606960] -> Removed port with 
>> GUID:0x0005ad0000024b27 LID range [0xAF,0xAF] of node:saguaro-23-0 HCA-1
>> Mar 20 14:29:18 025340 [44606960] -> osm_report_notice: Reporting Generic 
>> Notice type:3 num:65 from LID:0x0092 
>> GID:0xfe80000000000000,0x0005ad0000024bbb
>> Mar 20 14:29:18 025345 [44606960] -> Removed port with 
>> GUID:0x0005ad0000025043 LID range [0xB4,0xB4] of node:saguaro-23-5 HCA-1
>> Mar 20 14:29:18 025362 [44606960] -> osm_report_notice: Reporting Generic 
>> Notice type:3 num:65 from LID:0x0092 
>> GID:0xfe80000000000000,0x0005ad0000024bbb
>> Mar 20 14:29:18 025367 [44606960] -> Removed port with 
>> GUID:0x0005ad0000024d47 LID range [0xB6,0xB6] of node:saguaro-23-7 HCA-1
>> Mar 20 14:29:18 025385 [44606960] -> osm_report_notice: Reporting Generic 
>> Notice type:3 num:65 from LID:0x0092 
>> GID:0xfe80000000000000,0x0005ad0000024bbb
>> Mar 20 14:29:18 025390 [44606960] -> Removed port with 
>> GUID:0x0005ad0000024d6b LID range [0xB8,0xB8] of node:saguaro-23-9 HCA-1
>> Mar 20 14:29:18 025406 [44606960] -> osm_report_notice: Reporting Generic 
>> Notice type:3 num:65 from LID:0x0092 
>> GID:0xfe80000000000000,0x0005ad0000024bbb
>> Mar 20 14:29:18 025411 [44606960] -> Removed port with 
>> GUID:0x0005ad0000024977 LID range [0xA9,0xA9] of node:saguaro-22-4 HCA-1
>> Mar 20 14:29:18 025571 [44606960] -> osm_report_notice: Reporting Generic 
>> Notice type:3 num:65 from LID:0x0092 
>> GID:0xfe80000000000000,0x0005ad0000024bbb
>> Mar 20 14:29:18 025576 [44606960] -> Removed port with 
>> GUID:0x0005ad0000027c84 LID range [0x152,0x152] of node:Topspin Switch TS120
>> Mar 20 14:29:18 025612 [44606960] -> osm_report_notice: Reporting Generic 
>> Notice type:3 num:65 from LID:0x0092 
>> GID:0xfe80000000000000,0x0005ad0000024bbb
>> Mar 20 14:29:18 025619 [44606960] -> Removed port with 
>> GUID:0x0005ad0000024d8b LID range [0xB7,0xB7] of node:saguaro-23-8 HCA-1
>> Mar 20 14:29:18 025634 [44606960] -> osm_report_notice: Reporting Generic 
>> Notice type:3 num:65 from LID:0x0092 
>> GID:0xfe80000000000000,0x0005ad0000024bbb
>> Mar 20 14:29:18 025639 [44606960] -> Removed port with 
>> GUID:0x0005ad000002498f LID range [0xA8,0xA8] of node:saguaro-22-3 HCA-1
>> Mar 20 14:29:18 025655 [44606960] -> osm_report_notice: Reporting Generic 
>> Notice type:3 num:65 from LID:0x0092 
>> GID:0xfe80000000000000,0x0005ad0000024bbb
>> Mar 20 14:29:18 025660 [44606960] -> Removed port with 
>> GUID:0x0005ad0000024c9b LID range [0xA7,0xA7] of node:saguaro-22-2 HCA-1
>> Mar 20 14:29:18 025678 [44606960] -> osm_report_notice: Reporting Generic 
>> Notice type:3 num:65 from LID:0x0092 
>> GID:0xfe80000000000000,0x0005ad0000024bbb
>> Mar 20 14:29:18 025683 [44606960] -> Removed port with 
>> GUID:0x0005ad0000024da7 LID range [0xB0,0xB0] of node:saguaro-23-1 HCA-1
>> Mar 20 14:29:18 025700 [44606960] -> osm_report_notice: Reporting Generic 
>> Notice type:3 num:65 from LID:0x0092 
>> GID:0xfe80000000000000,0x0005ad0000024bbb
>> Mar 20 14:29:18 025705 [44606960] -> Removed port with 
>> GUID:0x0005ad0000024cbb LID range [0xB2,0xB2] of node:saguaro-23-3 HCA-1
>> Mar 20 14:29:18 025721 [44606960] -> osm_report_notice: Reporting Generic 
>> Notice type:3 num:65 from LID:0x0092 
>> GID:0xfe80000000000000,0x0005ad0000024bbb
>> Mar 20 14:29:18 025777 [44606960] -> Removed port with 
>> GUID:0x0005ad00000249d3 LID range [0xB1,0xB1] of node:saguaro-23-2 HCA-1
>> Mar 20 14:29:18 025794 [44606960] -> osm_report_notice: Reporting Generic 
>> Notice type:3 num:65 from LID:0x0092 
>> GID:0xfe80000000000000,0x0005ad0000024bbb
>> Mar 20 14:29:18 025800 [44606960] -> Removed port with 
>> GUID:0x0005ad0000024feb LID range [0x153,0x153] of node:saguaro-22-5 HCA-1
>> Mar 20 14:29:18 025816 [44606960] -> osm_report_notice: Reporting Generic 
>> Notice type:3 num:65 from LID:0x0092 
>> GID:0xfe80000000000000,0x0005ad0000024bbb
>> Mar 20 14:29:18 025821 [44606960] -> Removed port with 
>> GUID:0x0005ad0000024afb LID range [0xA5,0xA5] of node:saguaro-22-0 HCA-1
>> Mar 20 14:29:18 058968 [44606960] -> osm_ucast_mgr_process: Min Hop Tables 
>> configured on all switches
>> Mar 20 14:29:18 112970 [43C05960] -> SUBNET UP
>> Mar 20 14:29:18 511156 [45007960] -> osm_ucast_mgr_process: Min Hop Tables 
>> configured on all switches
>> Mar 20 14:29:18 573846 [41E02960] -> SUBNET UP
>> Mar 20 14:30:11 599965 [45A08960] -> __osm_trap_rcv_process_request: 
>> Received Generic Notice type:0x01 num:128 Producer:2 from LID:0x0148 
>> TID:0x0000000000000077
>> Mar 20 14:30:11 600182 [45A08960] -> osm_report_notice: Reporting Generic 
>> Notice type:1 num:128 from LID:0x0148 
>> GID:0xfe80000000000000,0x0005ad00000281b3
>> Mar 20 14:30:11 606044 [44606960] -> __osm_trap_rcv_process_request: 
>> Received Generic Notice type:0x01 num:128 Producer:2 from LID:0x0148 
>> TID:0x0000000000000078
>> Mar 20 14:30:11 606078 [43C05960] -> __osm_trap_rcv_process_request: 
>> Received Generic Notice type:0x01 num:128 Producer:2 from LID:0x001B 
>> TID:0x000000000000008b
>> Mar 20 14:30:11 606178 [44606960] -> osm_report_notice: Reporting Generic 
>> Notice type:1 num:128 from LID:0x0148 
>> GID:0xfe80000000000000,0x0005ad00000281b3
>> Mar 20 14:30:11 606207 [43C05960] -> osm_report_notice: Reporting Generic 
>> Notice type:1 num:128 from LID:0x001B 
>> GID:0xfe80000000000000,0x0005ad00000281a7
>> Mar 20 14:30:11 612375 [42803960] -> __osm_trap_rcv_process_request: 
>> Received Generic Notice type:0x01 num:128 Producer:2 from LID:0x001B 
>> TID:0x000000000000008c
>> Mar 20 14:30:11 612499 [42803960] -> osm_report_notice: Reporting Generic 
>> Notice type:1 num:128 from LID:0x001B 
>> GID:0xfe80000000000000,0x0005ad00000281a7
>> Mar 20 14:30:11 947057 [45007960] -> osm_report_notice: Reporting Generic 
>> Notice type:3 num:64 from LID:0x0092 
>> GID:0xfe80000000000000,0x0005ad0000024bbb
>> Mar 20 14:30:11 947074 [45007960] -> Discovered new port with 
>> GUID:0x0005ad0000027c84 LID range [0x152,0x152] of node:Topspin Switch TS120
>> Mar 20 14:30:11 947079 [45007960] -> osm_report_notice: Reporting Generic 
>> Notice type:3 num:64 from LID:0x0092 
>> GID:0xfe80000000000000,0x0005ad0000024bbb
>> Mar 20 14:30:11 947084 [45007960] -> Discovered new port with 
>> GUID:0x0005ad0000024b27 LID range [0xAF,0xAF] of node:saguaro-23-0 HCA-1
>> Mar 20 14:30:11 947088 [45007960] -> osm_report_notice: Reporting Generic 
>> Notice type:3 num:64 from LID:0x0092 
>> GID:0xfe80000000000000,0x0005ad0000024bbb
>> Mar 20 14:30:11 947093 [45007960] -> Discovered new port with 
>> GUID:0x0005ad0000024da7 LID range [0xB0,0xB0] of node:saguaro-23-1 HCA-1
>> Mar 20 14:30:11 947097 [45007960] -> osm_report_notice: Reporting Generic 
>> Notice type:3 num:64 from LID:0x0092 
>> GID:0xfe80000000000000,0x0005ad0000024bbb
>> Mar 20 14:30:11 947102 [45007960] -> Discovered new port with 
>> GUID:0x0005ad00000249d3 LID range [0xB1,0xB1] of node:saguaro-23-2 HCA-1
>> Mar 20 14:30:11 947106 [45007960] -> osm_report_notice: Reporting Generic 
>> Notice type:3 num:64 from LID:0x0092 
>> GID:0xfe80000000000000,0x0005ad0000024bbb
>> Mar 20 14:30:11 947118 [45007960] -> Discovered new port with 
>> GUID:0x0005ad0000024cbb LID range [0xB2,0xB2] of node:saguaro-23-3 HCA-1
>> Mar 20 14:30:11 947138 [45007960] -> osm_report_notice: Reporting Generic 
>> Notice type:3 num:64 from LID:0x0092 
>> GID:0xfe80000000000000,0x0005ad0000024bbb
>> Mar 20 14:30:11 947143 [45007960] -> Discovered new port with 
>> GUID:0x0005ad0000024e0b LID range [0xB3,0xB3] of node:saguaro-23-4 HCA-1
>> Mar 20 14:30:11 947148 [45007960] -> osm_report_notice: Reporting Generic 
>> Notice type:3 num:64 from LID:0x0092 
>> GID:0xfe80000000000000,0x0005ad0000024bbb
>> Mar 20 14:30:11 947153 [45007960] -> Discovered new port with 
>> GUID:0x0005ad0000025043 LID range [0xB4,0xB4] of node:saguaro-23-5 HCA-1
>> Mar 20 14:30:11 947157 [45007960] -> osm_report_notice: Reporting Generic 
>> Notice type:3 num:64 from LID:0x0092 
>> GID:0xfe80000000000000,0x0005ad0000024bbb
>> Mar 20 14:30:11 947162 [45007960] -> Discovered new port with 
>> GUID:0x0005ad000002510b LID range [0xB5,0xB5] of node:saguaro-23-6 HCA-1
>> Mar 20 14:30:11 947166 [45007960] -> osm_report_notice: Reporting Generic 
>> Notice type:3 num:64 from LID:0x0092 
>> GID:0xfe80000000000000,0x0005ad0000024bbb
>> Mar 20 14:30:11 947170 [45007960] -> Discovered new port with 
>> GUID:0x0005ad0000024d47 LID range [0xB6,0xB6] of node:saguaro-23-7 HCA-1
>> Mar 20 14:30:11 947174 [45007960] -> osm_report_notice: Reporting Generic 
>> Notice type:3 num:64 from LID:0x0092 
>> GID:0xfe80000000000000,0x0005ad0000024bbb
>> Mar 20 14:30:11 947179 [45007960] -> Discovered new port with 
>> GUID:0x0005ad0000024d8b LID range [0xB7,0xB7] of node:saguaro-23-8 HCA-1
>> Mar 20 14:30:11 947183 [45007960] -> osm_report_notice: Reporting Generic 
>> Notice type:3 num:64 from LID:0x0092 
>> GID:0xfe80000000000000,0x0005ad0000024bbb
>> Mar 20 14:30:11 947188 [45007960] -> Discovered new port with 
>> GUID:0x0005ad0000024d6b LID range [0xB8,0xB8] of node:saguaro-23-9 HCA-1
>> Mar 20 14:30:11 947191 [45007960] -> osm_report_notice: Reporting Generic 
>> Notice type:3 num:64 from LID:0x0092 
>> GID:0xfe80000000000000,0x0005ad0000024bbb
>> Mar 20 14:30:11 947196 [45007960] -> Discovered new port with 
>> GUID:0x0005ad0000024afb LID range [0xA5,0xA5] of node:saguaro-22-0 HCA-1
>> Mar 20 14:30:11 947200 [45007960] -> osm_report_notice: Reporting Generic 
>> Notice type:3 num:64 from LID:0x0092 
>> GID:0xfe80000000000000,0x0005ad0000024bbb
>> Mar 20 14:30:11 947205 [45007960] -> Discovered new port with 
>> GUID:0x0005ad000002511b LID range [0xA6,0xA6] of node:saguaro-22-1 HCA-1
>> Mar 20 14:30:11 947209 [45007960] -> osm_report_notice: Reporting Generic 
>> Notice type:3 num:64 from LID:0x0092 
>> GID:0xfe80000000000000,0x0005ad0000024bbb
>> Mar 20 14:30:11 947214 [45007960] -> Discovered new port with 
>> GUID:0x0005ad0000024c9b LID range [0xA7,0xA7] of node:saguaro-22-2 HCA-1
>> Mar 20 14:30:11 947282 [45007960] -> osm_report_notice: Reporting Generic 
>> Notice type:3 num:64 from LID:0x0092 
>> GID:0xfe80000000000000,0x0005ad0000024bbb
>> Mar 20 14:30:11 947288 [45007960] -> Discovered new port with 
>> GUID:0x0005ad000002498f LID range [0xA8,0xA8] of node:saguaro-22-3 HCA-1
>> Mar 20 14:30:11 947291 [45007960] -> osm_report_notice: Reporting Generic 
>> Notice type:3 num:64 from LID:0x0092 
>> GID:0xfe80000000000000,0x0005ad0000024bbb
>> Mar 20 14:30:11 947296 [45007960] -> Discovered new port with 
>> GUID:0x0005ad0000024977 LID range [0xA9,0xA9] of node:saguaro-22-4 HCA-1
>> Mar 20 14:30:11 947300 [45007960] -> osm_report_notice: Reporting Generic 
>> Notice type:3 num:64 from LID:0x0092 
>> GID:0xfe80000000000000,0x0005ad0000024bbb
>> Mar 20 14:30:11 947305 [45007960] -> Discovered new port with 
>> GUID:0x0005ad0000024feb LID range [0x153,0x153] of node:saguaro-22-5 HCA-1
>> Mar 20 14:30:11 978149 [45007960] -> osm_ucast_mgr_process: Min Hop Tables 
>> configured on all switches
>> Mar 20 14:30:12 042474 [4780B960] -> __osm_sm_mad_ctrl_rcv_callback: ERR 
>> 3111: Error status = 0x1C00
>> Mar 20 14:30:12 042577 [4780B960] -> SMP dump:
>>                 base_ver................0x1
>>                 mgmt_class..............0x81
>>                 class_ver...............0x1
>>                 method..................0x81 (SubnGetResp)
>>                 D bit...................0x1
>>                 status..................0x1C00
>>                 hop_ptr.................0x0
>>                 hop_count...............0x4
>>                 trans_id................0x92b38
>>                 attr_id.................0x15 (PortInfo)
>>                 resv....................0x0
>>                 attr_mod................0x13
>>                 m_key...................0x0000000000000000
>>                 dr_slid.................0xFFFF
>>                 dr_dlid.................0xFFFF
>> 
>>                 Initial path: [0][1][16][1][5]
>>                 Return path:  [0][9][13][D][2]
>>                 Reserved:     [0][0][0][0][0][0][0]
>> 
>>                 00 00 00 00 00 00 00 00   00 00 00 00 00 00 00 00
>> 
>>                 00 00 00 00 00 00 00 00   00 00 00 00 02 03 03 02
>> 
>>                 14 52 00 11 40 40 00 08   08 04 F2 40 00 00 00 00
>> 
>>                 00 00 00 00 00 88 00 00   00 00 00 00 00 00 00 00
>> 
>> Mar 20 14:30:12 042668 [45007960] -> osm_pi_rcv_process_set: Received error 
>> status 0x1c for SetResp() during ACTIVE transition
>> Mar 20 14:30:12 042676 [4780B960] -> __osm_sm_mad_ctrl_rcv_callback: ERR 
>> 3111: Error status = 0x1C00
>> Mar 20 14:30:12 042682 [45007960] -> PortInfo dump:
>>                 port number.............0x13
>>                 node_guid...............0x0005ad00000281a7
>>                 port_guid...............0x0005ad00000281a7
>>                 m_key...................0x0000000000000000
>>                 subnet_prefix...........0x0000000000000000
>>                 base_lid................0x0
>>                 master_sm_base_lid......0x0
>>                 capability_mask.........0x0
>>                 diag_code...............0x0
>>                 m_key_lease_period......0x0
>>                 local_port_num..........0x2
>>                 link_width_enabled......0x3
>>                 link_width_supported....0x3
>>                 link_width_active.......0x2
>>                 link_speed_supported....0x1
>>                 port_state..............ACTIVE
>>                 state_info2.............0x52
>>                 m_key_protect_bits......0x0
>>                 lmc.....................0x0
>>                 link_speed..............0x11
>>                 mtu_smsl................0x40
>>                 vl_cap_init_type........0x40
>>                 vl_high_limit...........0x0
>>                 vl_arb_high_cap.........0x8
>>                 vl_arb_low_cap..........0x8
>>                 init_rep_mtu_cap........0x4
>>                 vl_stall_life...........0xF2
>>                 vl_enforce..............0x40
>>                 m_key_violations........0x0
>>                 p_key_violations........0x0
>>                 q_key_violations........0x0
>>                 guid_cap................0x0
>>                 client_reregister.......0x0
>>                 subnet_timeout..........0x0
>>                 resp_time_value.........0x0
>>                 error_threshold.........0x88
>> Mar 20 14:30:12 042701 [45007960] -> Capabilities Mask:
>> Mar 20 14:30:12 042714 [4780B960] -> SMP dump:
>>                 base_ver................0x1
>>                 mgmt_class..............0x81
>>                 class_ver...............0x1
>>                 method..................0x81 (SubnGetResp)
>>                 D bit...................0x1
>>                 status..................0x1C00
>>                 hop_ptr.................0x0
>>                 hop_count...............0x4
>>                 trans_id................0x92b39
>>                 attr_id.................0x15 (PortInfo)
>>                 resv....................0x0
>>                 attr_mod................0x16
>>                 m_key...................0x0000000000000000
>>                 dr_slid.................0xFFFF
>>                 dr_dlid.................0xFFFF
>> 
>>                 Initial path: [0][1][16][1][5]
>>                 Return path:  [0][9][13][D][2]
>>                 Reserved:     [0][0][0][0][0][0][0]
>> 
>>                 00 00 00 00 00 00 00 00   00 00 00 00 00 00 00 00
>> 
>>                 00 00 00 00 00 00 00 00   00 00 00 00 02 03 03 02
>> 
>>                 14 52 00 11 40 40 00 08   08 04 F2 40 00 00 00 00
>> 
>>                 00 00 00 00 00 88 00 00   00 00 00 00 00 00 00 00
>> 
>> Mar 20 14:30:12 042845 [41401960] -> osm_pi_rcv_process_set: Received error 
>> status 0x1c for SetResp() during ACTIVE transition
>> Mar 20 14:30:12 042856 [4780B960] -> __osm_sm_mad_ctrl_rcv_callback: ERR 
>> 3111: Error status = 0x1C00
>> Mar 20 14:30:12 042851 [41401960] -> PortInfo dump:
>>                 port number.............0x16
>>                 node_guid...............0x0005ad00000281a7
>>                 port_guid...............0x0005ad00000281a7
>>                 m_key...................0x0000000000000000
>>                 subnet_prefix...........0x0000000000000000
>>                 base_lid................0x0
>>                 master_sm_base_lid......0x0
>>                 capability_mask.........0x0
>>                 diag_code...............0x0
>>                 m_key_lease_period......0x0
>>                 local_port_num..........0x2
>>                 link_width_enabled......0x3
>>                 link_width_supported....0x3
>>                 link_width_active.......0x2
>>                 link_speed_supported....0x1
>>                 port_state..............ACTIVE
>>                 state_info2.............0x52
>>                 m_key_protect_bits......0x0
>>                 lmc.....................0x0
>>                 link_speed..............0x11
>>                 mtu_smsl................0x40
>>                 vl_cap_init_type........0x40
>>                 vl_high_limit...........0x0
>>                 vl_arb_high_cap.........0x8
>>                 vl_arb_low_cap..........0x8
>>                 init_rep_mtu_cap........0x4
>>                 vl_stall_life...........0xF2
>>                 vl_enforce..............0x40
>>                 m_key_violations........0x0
>>                 p_key_violations........0x0
>>                 q_key_violations........0x0
>>                 guid_cap................0x0
>>                 client_reregister.......0x0
>>                 subnet_timeout..........0x0
>>                 resp_time_value.........0x0
>>                 error_threshold.........0x88
>> Mar 20 14:30:12 042897 [41401960] -> Capabilities Mask:
>> Mar 20 14:30:12 042907 [4780B960] -> SMP dump:
>>                 base_ver................0x1
>>                 mgmt_class..............0x81
>>                 class_ver...............0x1
>>                 method..................0x81 (SubnGetResp)
>>                 D bit...................0x1
>>                 status..................0x1C00
>>                 hop_ptr.................0x0
>>                 hop_count...............0x4
>>                 trans_id................0x92b3a
>>                 attr_id.................0x15 (PortInfo)
>>                 resv....................0x0
>>                 attr_mod................0x17
>>                 m_key...................0x0000000000000000
>>                 dr_slid.................0xFFFF
>>                 dr_dlid.................0xFFFF
>> 
>>                 Initial path: [0][1][16][1][5]
>>                 Return path:  [0][9][13][D][2]
>>                 Reserved:     [0][0][0][0][0][0][0]
>> 
>>                 00 00 00 00 00 00 00 00   00 00 00 00 00 00 00 00
>> 
>>                 00 00 00 00 00 00 00 00   00 00 00 00 02 03 03 02
>> 
>>                 14 52 00 11 40 40 00 08   08 04 F2 40 00 00 00 00
>> 
>>                 00 00 00 00 00 88 00 00   00 00 00 00 00 00 00 00
>> 
>> Mar 20 14:30:12 043013 [43204960] -> osm_pi_rcv_process_set: Received error 
>> status 0x1c for SetResp() during ACTIVE transition
>> Mar 20 14:30:12 043015 [4780B960] -> __osm_sm_mad_ctrl_rcv_callback: ERR 
>> 3111: Error status = 0x1C00
>> Mar 20 14:30:12 043038 [43204960] -> PortInfo dump:
>>                 port number.............0x17
>>                 node_guid...............0x0005ad00000281a7
>>                 port_guid...............0x0005ad00000281a7
>>                 m_key...................0x0000000000000000
>>                 subnet_prefix...........0x0000000000000000
>>                 base_lid................0x0
>>                 master_sm_base_lid......0x0
>>                 capability_mask.........0x0
>>                 diag_code...............0x0
>>                 m_key_lease_period......0x0
>>                 local_port_num..........0x2
>>                 link_width_enabled......0x3
>>                 link_width_supported....0x3
>>                 link_width_active.......0x2
>>                 link_speed_supported....0x1
>>                 port_state..............ACTIVE
>>                 state_info2.............0x52
>>                 m_key_protect_bits......0x0
>>                 lmc.....................0x0
>>                 link_speed..............0x11
>>                 mtu_smsl................0x40
>>                 vl_cap_init_type........0x40
>>                 vl_high_limit...........0x0
>>                 vl_arb_high_cap.........0x8
>>                 vl_arb_low_cap..........0x8
>>                 init_rep_mtu_cap........0x4
>>                 vl_stall_life...........0xF2
>>                 vl_enforce..............0x40
>>                 m_key_violations........0x0
>>                 p_key_violations........0x0
>>                 q_key_violations........0x0
>>                 guid_cap................0x0
>>                 client_reregister.......0x0
>>                 subnet_timeout..........0x0
>>                 resp_time_value.........0x0
>>                 error_threshold.........0x88
>> Mar 20 14:30:12 043090 [43204960] -> Capabilities Mask:
>> Mar 20 14:30:12 043094 [4780B960] -> SMP dump:
>>                 base_ver................0x1
>>                 mgmt_class..............0x81
>>                 class_ver...............0x1
>>                 method..................0x81 (SubnGetResp)
>>                 D bit...................0x1
>>                 status..................0x1C00
>>                 hop_ptr.................0x0
>>                 hop_count...............0x4
>>                 trans_id................0x92b3b
>>                 attr_id.................0x15 (PortInfo)
>>                 resv....................0x0
>>                 attr_mod................0x18
>>                 m_key...................0x0000000000000000
>>                 dr_slid.................0xFFFF
>>                 dr_dlid.................0xFFFF
>> 
>>                 Initial path: [0][1][16][1][5]
>>                 Return path:  [0][9][13][D][2]
>>                 Reserved:     [0][0][0][0][0][0][0]
>> 
>>                 00 00 00 00 00 00 00 00   00 00 00 00 00 00 00 00
>> 
>>                 00 00 00 00 00 00 00 00   00 00 00 00 02 03 03 02
>> 
>>                 14 52 00 11 40 40 00 08   08 04 F2 40 00 00 00 00
>> 
>>                 00 00 00 00 00 88 00 00   00 00 00 00 00 00 00 00
>> 
>> Mar 20 14:30:12 043173 [44606960] -> osm_pi_rcv_process_set: Received error 
>> status 0x1c for SetResp() during ACTIVE transition
>> Mar 20 14:30:12 043178 [44606960] -> PortInfo dump:
>>                 port number.............0x18
>>                 node_guid...............0x0005ad00000281a7
>>                 port_guid...............0x0005ad00000281a7
>>                 m_key...................0x0000000000000000
>>                 subnet_prefix...........0x0000000000000000
>>                 base_lid................0x0
>>                 master_sm_base_lid......0x0
>>                 capability_mask.........0x0
>>                 diag_code...............0x0
>>                 m_key_lease_period......0x0
>>                 local_port_num..........0x2
>>                 link_width_enabled......0x3
>>                 link_width_supported....0x3
>>                 link_width_active.......0x2
>>                 link_speed_supported....0x1
>>                 port_state..............ACTIVE
>>                 state_info2.............0x52
>>                 m_key_protect_bits......0x0
>>                 lmc.....................0x0
>>                 link_speed..............0x11
>>                 mtu_smsl................0x40
>>                 vl_cap_init_type........0x40
>>                 vl_high_limit...........0x0
>>                 vl_arb_high_cap.........0x8
>>                 vl_arb_low_cap..........0x8
>>                 init_rep_mtu_cap........0x4
>>                 vl_stall_life...........0xF2
>>                 vl_enforce..............0x40
>>                 m_key_violations........0x0
>>                 p_key_violations........0x0
>>                 q_key_violations........0x0
>>                 guid_cap................0x0
>>                 client_reregister.......0x0
>>                 subnet_timeout..........0x0
>>                 resp_time_value.........0x0
>>                 error_threshold.........0x88
>> Mar 20 14:30:12 043191 [44606960] -> Capabilities Mask:
>> Mar 20 14:30:12 043222 [4780B960] -> __osm_sm_mad_ctrl_rcv_callback: ERR 
>> 3111: Error status = 0x1C00
>> Mar 20 14:30:12 043247 [4780B960] -> SMP dump:
>>                 base_ver................0x1
>>                 mgmt_class..............0x81
>>                 class_ver...............0x1
>>                 method..................0x81 (SubnGetResp)
>>                 D bit...................0x1
>>                 status..................0x1C00
>>                 hop_ptr.................0x0
>>                 hop_count...............0x4
>>                 trans_id................0x92b3c
>>                 attr_id.................0x15 (PortInfo)
>>                 resv....................0x0
>>                 attr_mod................0x16
>>                 m_key...................0x0000000000000000
>>                 dr_slid.................0xFFFF
>>                 dr_dlid.................0xFFFF
>> 
>>                 Initial path: [0][1][12][1][4]
>>                 Return path:  [0][9][14][D][1]
>>                 Reserved:     [0][0][0][0][0][0][0]
>> 
>>                 00 00 00 00 00 00 00 00   00 00 00 00 00 00 00 00
>> 
>>                 00 00 00 00 00 00 00 00   00 00 00 00 01 03 03 02
>> 
>>                 14 52 00 11 40 40 00 08   08 04 F2 40 00 00 00 00
>> 
>>                 00 00 00 00 00 88 00 00   00 00 00 00 00 00 00 00
>> 
>> Mar 20 14:30:12 043318 [4780B960] -> __osm_sm_mad_ctrl_rcv_callback: ERR 
>> 3111: Error status = 0x1C00
>> Mar 20 14:30:12 043314 [41E02960] -> osm_pi_rcv_process_set: Received error 
>> status 0x1c for SetResp() during ACTIVE transition
>> Mar 20 14:30:12 043357 [4780B960] -> SMP dump:
>>                 base_ver................0x1
>>                 mgmt_class..............0x81
>>                 class_ver...............0x1
>>                 method..................0x81 (SubnGetResp)
>>                 D bit...................0x1
>>                 status..................0x1C00
>>                 hop_ptr.................0x0
>>                 hop_count...............0x4
>>                 trans_id................0x92b3d
>>                 attr_id.................0x15 (PortInfo)
>>                 resv....................0x0
>>                 attr_mod................0x17
>>                 m_key...................0x0000000000000000
>>                 dr_slid.................0xFFFF
>>                 dr_dlid.................0xFFFF
>> 
>>                 Initial path: [0][1][12][1][4]
>>                 Return path:  [0][9][14][D][1]
>>                 Reserved:     [0][0][0][0][0][0][0]
>> 
>>                 00 00 00 00 00 00 00 00   00 00 00 00 00 00 00 00
>> 
>>                 00 00 00 00 00 00 00 00   00 00 00 00 01 03 03 02
>> 
>>                 14 52 00 11 40 40 00 08   08 04 F2 40 00 00 00 00
>> 
>>                 00 00 00 00 00 88 00 00   00 00 00 00 00 00 00 00
>> 
>> Mar 20 14:30:12 043367 [41E02960] -> PortInfo dump:
>>                 port number.............0x16
>>                 node_guid...............0x0005ad00000281b3
>>                 port_guid...............0x0005ad00000281b3
>>                 m_key...................0x0000000000000000
>>                 subnet_prefix...........0x0000000000000000
>>                 base_lid................0x0
>>                 master_sm_base_lid......0x0
>>                 capability_mask.........0x0
>>                 diag_code...............0x0
>>                 m_key_lease_period......0x0
>>                 local_port_num..........0x1
>>                 link_width_enabled......0x3
>>                 link_width_supported....0x3
>>                 link_width_active.......0x2
>>                 link_speed_supported....0x1
>>                 port_state..............ACTIVE
>>                 state_info2.............0x52
>>                 m_key_protect_bits......0x0
>>                 lmc.....................0x0
>>                 link_speed..............0x11
>>                 mtu_smsl................0x40
>>                 vl_cap_init_type........0x40
>>                 vl_high_limit...........0x0
>>                 vl_arb_high_cap.........0x8
>>                 vl_arb_low_cap..........0x8
>>                 init_rep_mtu_cap........0x4
>>                 vl_stall_life...........0xF2
>>                 vl_enforce..............0x40
>>                 m_key_violations........0x0
>>                 p_key_violations........0x0
>>                 q_key_violations........0x0
>>                 guid_cap................0x0
>>                 client_reregister.......0x0
>>                 subnet_timeout..........0x0
>>                 resp_time_value.........0x0
>>                 error_threshold.........0x88
>> Mar 20 14:30:12 043422 [41E02960] -> Capabilities Mask:
>> Mar 20 14:30:12 043513 [43C05960] -> osm_pi_rcv_process_set: Received error 
>> status 0x1c for SetResp() during ACTIVE transition
>> Mar 20 14:30:12 043518 [4780B960] -> __osm_sm_mad_ctrl_rcv_callback: ERR 
>> 3111: Error status = 0x1C00
>> Mar 20 14:30:12 043519 [43C05960] -> PortInfo dump:
>>                 port number.............0x17
>>                 node_guid...............0x0005ad00000281b3
>>                 port_guid...............0x0005ad00000281b3
>>                 m_key...................0x0000000000000000
>>                 subnet_prefix...........0x0000000000000000
>>                 base_lid................0x0
>>                 master_sm_base_lid......0x0
>>                 capability_mask.........0x0
>>                 diag_code...............0x0
>>                 m_key_lease_period......0x0
>>                 local_port_num..........0x1
>>                 link_width_enabled......0x3
>>                 link_width_supported....0x3
>>                 link_width_active.......0x2
>>                 link_speed_supported....0x1
>>                 port_state..............ACTIVE
>>                 state_info2.............0x52
>>                 m_key_protect_bits......0x0
>>                 lmc.....................0x0
>>                 link_speed..............0x11
>>                 mtu_smsl................0x40
>>                 vl_cap_init_type........0x40
>>                 vl_high_limit...........0x0
>>                 vl_arb_high_cap.........0x8
>>                 vl_arb_low_cap..........0x8
>>                 init_rep_mtu_cap........0x4
>>                 vl_stall_life...........0xF2
>>                 vl_enforce..............0x40
>>                 m_key_violations........0x0
>>                 p_key_violations........0x0
>>                 q_key_violations........0x0
>>                 guid_cap................0x0
>>                 client_reregister.......0x0
>>                 subnet_timeout..........0x0
>>                 resp_time_value.........0x0
>>                 error_threshold.........0x88
>> Mar 20 14:30:12 043535 [43C05960] -> Capabilities Mask:
>> Mar 20 14:30:12 043553 [4780B960] -> SMP dump:
>>                 base_ver................0x1
>>                 mgmt_class..............0x81
>>                 class_ver...............0x1
>>                 method..................0x81 (SubnGetResp)
>>                 D bit...................0x1
>>                 status..................0x1C00
>>                 hop_ptr.................0x0
>>                 hop_count...............0x4
>>                 trans_id................0x92b3e
>>                 attr_id.................0x15 (PortInfo)
>>                 resv....................0x0
>>                 attr_mod................0x18
>>                 m_key...................0x0000000000000000
>>                 dr_slid.................0xFFFF
>>                 dr_dlid.................0xFFFF
>> 
>>                 Initial path: [0][1][12][1][4]
>>                 Return path:  [0][9][14][D][1]
>>                 Reserved:     [0][0][0][0][0][0][0]
>> 
>>                 00 00 00 00 00 00 00 00   00 00 00 00 00 00 00 00
>> 
>>                 00 00 00 00 00 00 00 00   00 00 00 00 01 03 03 02
>> 
>>                 14 52 00 11 40 40 00 08   08 04 F2 40 00 00 00 00
>> 
>>                 00 00 00 00 00 88 00 00   00 00 00 00 00 00 00 00
>> 
>> Mar 20 14:30:12 043658 [42803960] -> osm_pi_rcv_process_set: Received error 
>> status 0x1c for SetResp() during ACTIVE transition
>> Mar 20 14:30:12 043663 [42803960] -> PortInfo dump:
>>                 port number.............0x18
>>                 node_guid...............0x0005ad00000281b3
>>                 port_guid...............0x0005ad00000281b3
>>                 m_key...................0x0000000000000000
>>                 subnet_prefix...........0x0000000000000000
>>                 base_lid................0x0
>>                 master_sm_base_lid......0x0
>>                 capability_mask.........0x0
>>                 diag_code...............0x0
>>                 m_key_lease_period......0x0
>>                 local_port_num..........0x1
>>                 link_width_enabled......0x3
>>                 link_width_supported....0x3
>>                 link_width_active.......0x2
>>                 link_speed_supported....0x1
>>                 port_state..............ACTIVE
>>                 state_info2.............0x52
>>                 m_key_protect_bits......0x0
>>                 lmc.....................0x0
>>                 link_speed..............0x11
>>                 mtu_smsl................0x40
>>                 vl_cap_init_type........0x40
>>                 vl_high_limit...........0x0
>>                 vl_arb_high_cap.........0x8
>>                 vl_arb_low_cap..........0x8
>>                 init_rep_mtu_cap........0x4
>>                 vl_stall_life...........0xF2
>>                 vl_enforce..............0x40
>>                 m_key_violations........0x0
>>                 p_key_violations........0x0
>>                 q_key_violations........0x0
>>                 guid_cap................0x0
>>                 client_reregister.......0x0
>>                 subnet_timeout..........0x0
>>                 resp_time_value.........0x0
>>                 error_threshold.........0x88
>> Mar 20 14:30:12 043678 [42803960] -> Capabilities Mask:
>> Mar 20 14:30:12 049088 [43204960] -> SUBNET UP
>> Mar 20 14:30:12 442903 [43C05960] -> osm_ucast_mgr_process: Min Hop Tables 
>> configured on all switches
>> Mar 20 14:30:12 497312 [45007960] -> SUBNET UP
>> Mar 20 14:30:27 571421 [43C05960] -> __osm_trap_rcv_process_request: 
>> Received Generic Notice type:0x01 num:128 Producer:2 from LID:0x0152 
>> TID:0x0000000000000000
>> Mar 20 14:30:27 571674 [43C05960] -> osm_report_notice: Reporting Generic 
>> Notice type:1 num:128 from LID:0x0152 
>> GID:0xfe80000000000000,0x0005ad0000027c84
>> Mar 20 14:30:27 782498 [45A08960] -> __osm_trap_rcv_process_request: 
>> Received Generic Notice type:0x01 num:128 Producer:2 from LID:0x0152 
>> TID:0x0000000000000001
>> Mar 20 14:30:27 782616 [45A08960] -> osm_report_notice: Reporting Generic 
>> Notice type:1 num:128 from LID:0x0152 
>> GID:0xfe80000000000000,0x0005ad0000027c84
>> Mar 20 14:30:27 804302 [42803960] -> __osm_trap_rcv_process_request: 
>> Received Generic Notice type:0x01 num:128 Producer:2 from LID:0x0152 
>> TID:0x0000000000000002
>> Mar 20 14:30:27 805103 [42803960] -> osm_report_notice: Reporting Generic 
>> Notice type:1 num:128 from LID:0x0152 
>> GID:0xfe80000000000000,0x0005ad0000027c84
>> Mar 20 14:30:27 924983 [41E02960] -> __osm_trap_rcv_process_request: 
>> Received Generic Notice type:0x01 num:128 Producer:2 from LID:0x0152 
>> TID:0x0000000000000003
>> Mar 20 14:30:27 925088 [41E02960] -> osm_report_notice: Reporting Generic 
>> Notice type:1 num:128 from LID:0x0152 
>> GID:0xfe80000000000000,0x0005ad0000027c84
>> Mar 20 14:30:27 934314 [43204960] -> osm_report_notice: Reporting Generic 
>> Notice type:3 num:65 from LID:0x0092 
>> GID:0xfe80000000000000,0x0005ad0000024bbb
>> Mar 20 14:30:27 934327 [43204960] -> Removed port with 
>> GUID:0x0005ad00000249d3 LID range [0xB1,0xB1] of node:saguaro-23-2 HCA-1
>> Mar 20 14:30:27 969077 [43204960] -> osm_ucast_mgr_process: Min Hop Tables 
>> configured on all switches
>> Mar 20 14:30:28 017451 [41E02960] -> SUBNET UP
>> Mar 20 14:30:28 030947 [45007960] -> __osm_trap_rcv_process_request: 
>> Received Generic Notice type:0x01 num:128 Producer:2 from LID:0x0152 
>> TID:0x0000000000000004
>> Mar 20 14:30:28 031177 [45007960] -> osm_report_notice: Reporting Generic 
>> Notice type:1 num:128 from LID:0x0152 
>> GID:0xfe80000000000000,0x0005ad0000027c84
>> Mar 20 14:30:28 120040 [42803960] -> __osm_trap_rcv_process_request: 
>> Received Generic Notice type:0x01 num:128 Producer:2 from LID:0x0152 
>> TID:0x0000000000000005
>> Mar 20 14:30:28 120190 [42803960] -> osm_report_notice: Reporting Generic 
>> Notice type:1 num:128 from LID:0x0152 
>> GID:0xfe80000000000000,0x0005ad0000027c84
>> Mar 20 14:30:28 148805 [41E02960] -> __osm_trap_rcv_process_request: 
>> Received Generic Notice type:0x01 num:128 Producer:2 from LID:0x0152 
>> TID:0x0000000000000006
>> Mar 20 14:30:28 149108 [41E02960] -> osm_report_notice: Reporting Generic 
>> Notice type:1 num:128 from LID:0x0152 
>> GID:0xfe80000000000000,0x0005ad0000027c84
>> Mar 20 14:30:28 170453 [41401960] -> __osm_trap_rcv_process_request: 
>> Received Generic Notice type:0x01 num:128 Producer:2 from LID:0x0152 
>> TID:0x0000000000000007
>> Mar 20 14:30:28 170971 [41401960] -> osm_report_notice: Reporting Generic 
>> Notice type:1 num:128 from LID:0x0152 
>> GID:0xfe80000000000000,0x0005ad0000027c84
>> Mar 20 14:30:28 336861 [43C05960] -> osm_report_notice: Reporting Generic 
>> Notice type:3 num:65 from LID:0x0092 
>> GID:0xfe80000000000000,0x0005ad0000024bbb
>> Mar 20 14:30:28 336884 [43C05960] -> Removed port with 
>> GUID:0x0005ad0000024e0b LID range [0xB3,0xB3] of node:saguaro-23-4 HCA-1
>> Mar 20 14:30:28 336910 [43C05960] -> osm_report_notice: Reporting Generic 
>> Notice type:3 num:65 from LID:0x0092 
>> GID:0xfe80000000000000,0x0005ad0000024bbb
>> Mar 20 14:30:28 336916 [43C05960] -> Removed port with 
>> GUID:0x0005ad0000025043 LID range [0xB4,0xB4] of node:saguaro-23-5 HCA-1
>> Mar 20 14:30:28 336945 [43C05960] -> osm_report_notice: Reporting Generic 
>> Notice type:3 num:65 from LID:0x0092 
>> GID:0xfe80000000000000,0x0005ad0000024bbb
>> Mar 20 14:30:28 336951 [43C05960] -> Removed port with 
>> GUID:0x0005ad0000024cbb LID range [0xB2,0xB2] of node:saguaro-23-3 HCA-1
>> Mar 20 14:30:28 371497 [43C05960] -> osm_ucast_mgr_process: Min Hop Tables 
>> configured on all switches
>> Mar 20 14:30:28 410709 [41401960] -> __osm_trap_rcv_process_request: 
>> Received Generic Notice type:0x01 num:128 Producer:2 from LID:0x0152 
>> TID:0x0000000000000008
>> Mar 20 14:30:28 410894 [41401960] -> osm_report_notice: Reporting Generic 
>> Notice type:1 num:128 from LID:0x0152 
>> GID:0xfe80000000000000,0x0005ad0000027c84
>> Mar 20 14:30:28 415926 [43C05960] -> __osm_trap_rcv_process_request: 
>> Received Generic Notice type:0x01 num:128 Producer:2 from LID:0x0152 
>> TID:0x0000000000000009
>> Mar 20 14:30:28 419624 [43C05960] -> osm_report_notice: Reporting Generic 
>> Notice type:1 num:128 from LID:0x0152 
>> GID:0xfe80000000000000,0x0005ad0000027c84
>> Mar 20 14:30:28 426978 [45A08960] -> SUBNET UP
>> Mar 20 14:30:28 438003 [41E02960] -> __osm_trap_rcv_process_request: 
>> Received Generic Notice type:0x01 num:128 Producer:2 from LID:0x0152 
>> TID:0x000000000000000a
>> Mar 20 14:30:28 438182 [41E02960] -> osm_report_notice: Reporting Generic 
>> Notice type:1 num:128 from LID:0x0152 
>> GID:0xfe80000000000000,0x0005ad0000027c84
>> Mar 20 14:30:28 470141 [41E02960] -> __osm_trap_rcv_process_request: 
>> Received Generic Notice type:0x01 num:128 Producer:2 from LID:0x0152 
>> TID:0x000000000000000b
>> Mar 20 14:30:28 470197 [41E02960] -> __osm_trap_rcv_process_request: ERR 
>> 3804: Received trap 11 times consecutively
>> Mar 20 14:30:28 652535 [42803960] -> __osm_trap_rcv_process_request: 
>> Received Generic Notice type:0x01 num:128 Producer:2 from LID:0x0152 
>> TID:0x000000000000000c
>> Mar 20 14:30:28 652623 [42803960] -> __osm_trap_rcv_process_request: ERR 
>> 3804: Received trap 12 times consecutively
>> Mar 20 14:30:28 681514 [43C05960] -> __osm_trap_rcv_process_request: 
>> Received Generic Notice type:0x01 num:128 Producer:2 from LID:0x0152 
>> TID:0x000000000000000d
>> Mar 20 14:30:28 681636 [43C05960] -> __osm_trap_rcv_process_request: ERR 
>> 3804: Received trap 13 times consecutively
>> Mar 20 14:30:28 703052 [44606960] -> __osm_trap_rcv_process_request: 
>> Received Generic Notice type:0x01 num:128 Producer:2 from LID:0x0152 
>> TID:0x000000000000000e
>> Mar 20 14:30:28 703092 [44606960] -> __osm_trap_rcv_process_request: ERR 
>> 3804: Received trap 14 times consecutively
>> Mar 20 14:30:28 724753 [43C05960] -> __osm_trap_rcv_process_request: 
>> Received Generic Notice type:0x01 num:128 Producer:2 from LID:0x0152 
>> TID:0x000000000000000f
>> Mar 20 14:30:28 724809 [43C05960] -> __osm_trap_rcv_process_request: ERR 
>> 3804: Received trap 15 times consecutively
>> Mar 20 14:30:28 855519 [42803960] -> __osm_trap_rcv_process_request: 
>> Received Generic Notice type:0x01 num:128 Producer:2 from LID:0x0152 
>> TID:0x0000000000000010
>> Mar 20 14:30:28 855671 [42803960] -> __osm_trap_rcv_process_request: ERR 
>> 3804: Received trap 16 times consecutively
>> Mar 20 14:30:28 877289 [45A08960] -> __osm_trap_rcv_process_request: 
>> Received Generic Notice type:0x01 num:128 Producer:2 from LID:0x0152 
>> TID:0x0000000000000011
>> Mar 20 14:30:28 877354 [45A08960] -> __osm_trap_rcv_process_request: ERR 
>> 3804: Received trap 17 times consecutively
>> Mar 20 14:30:28 899021 [45A08960] -> __osm_trap_rcv_process_request: 
>> Received Generic Notice type:0x01 num:128 Producer:2 from LID:0x0152 
>> TID:0x0000000000000012
>> Mar 20 14:30:28 899062 [45A08960] -> __osm_trap_rcv_process_request: ERR 
>> 3804: Received trap 18 times consecutively
>> Mar 20 14:30:29 006886 [45007960] -> __osm_trap_rcv_process_request: 
>> Received Generic Notice type:0x01 num:128 Producer:2 from LID:0x0152 
>> TID:0x0000000000000013
>> Mar 20 14:30:29 006950 [45007960] -> __osm_trap_rcv_process_request: ERR 
>> 3804: Received trap 19 times consecutively
>> Mar 20 14:30:29 099965 [44606960] -> __osm_trap_rcv_process_request: 
>> Received Generic Notice type:0x01 num:128 Producer:2 from LID:0x0152 
>> TID:0x0000000000000014
>> Mar 20 14:30:29 100020 [44606960] -> __osm_trap_rcv_process_request: ERR 
>> 3804: Received trap 20 times consecutively
>> Mar 20 14:30:29 146532 [41E02960] -> __osm_trap_rcv_process_request: 
>> Received Generic Notice type:0x01 num:128 Producer:2 from LID:0x0152 
>> TID:0x0000000000000015
>> Mar 20 14:30:29 146578 [41E02960] -> __osm_trap_rcv_process_request: ERR 
>> 3804: Received trap 21 times consecutively
>> Mar 20 14:30:29 356891 [43C05960] -> __osm_trap_rcv_process_request: 
>> Received Generic Notice type:0x01 num:128 Producer:2 from LID:0x0152 
>> TID:0x0000000000000016
>> Mar 20 14:30:29 356938 [43C05960] -> __osm_trap_rcv_process_request: ERR 
>> 3804: Received trap 22 times consecutively
>> Mar 20 14:30:29 383112 [43204960] -> __osm_trap_rcv_process_request: 
>> Received Generic Notice type:0x01 num:128 Producer:2 from LID:0x0152 
>> TID:0x0000000000000017
>> Mar 20 14:30:29 383157 [43204960] -> __osm_trap_rcv_process_request: ERR 
>> 3804: Received trap 23 times consecutively
>> Mar 20 14:30:29 383710 [41401960] -> __osm_trap_rcv_process_request: 
>> Received Generic Notice type:0x01 num:128 Producer:2 from LID:0x0148 
>> TID:0x0000000000000079
>> Mar 20 14:30:29 383790 [41401960] -> osm_report_notice: Reporting Generic 
>> Notice type:1 num:128 from LID:0x0148 
>> GID:0xfe80000000000000,0x0005ad00000281b3
>> Mar 20 14:30:29 407890 [42803960] -> __osm_trap_rcv_process_request: 
>> Received Generic Notice type:0x01 num:128 Producer:2 from LID:0x0152 
>> TID:0x0000000000000018
>> Mar 20 14:30:29 407935 [42803960] -> __osm_trap_rcv_process_request: ERR 
>> 3804: Received trap 24 times consecutively
>> Mar 20 14:30:29 429653 [45A08960] -> __osm_trap_rcv_process_request: 
>> Received Generic Notice type:0x01 num:128 Producer:2 from LID:0x0152 
>> TID:0x0000000000000019
>> Mar 20 14:30:29 429700 [45A08960] -> __osm_trap_rcv_process_request: ERR 
>> 3804: Received trap 25 times consecutively
>> Mar 20 14:30:29 451352 [45007960] -> __osm_trap_rcv_process_request: 
>> Received Generic Notice type:0x01 num:128 Producer:2 from LID:0x0152 
>> TID:0x000000000000001a
>> Mar 20 14:30:29 451401 [45007960] -> __osm_trap_rcv_process_request: ERR 
>> 3804: Received trap 26 times consecutively
>> Mar 20 14:30:29 479843 [4780B960] -> umad_receiver: ERR 5409: send completed 
>> with error (method=0x1 attr=0x11 trans_id=0x124ef00095cbf) -- dropping
>> Mar 20 14:30:29 479855 [4780B960] -> umad_receiver: ERR 5411: DR SMP
>> Mar 20 14:30:29 479865 [4780B960] -> __osm_sm_mad_ctrl_send_err_cb: ERR 
>> 3113: MAD completed in error (IB_TIMEOUT)
>> Mar 20 14:30:29 479901 [4780B960] -> SMP dump:
>>                 base_ver................0x1
>>                 mgmt_class..............0x81
>>                 class_ver...............0x1
>>                 method..................0x1 (SubnGet)
>>                 D bit...................0x0
>>                 status..................0x0
>>                 hop_ptr.................0x0
>>                 hop_count...............0x6
>>                 trans_id................0x95cbf
>>                 attr_id.................0x11 (NodeInfo)
>>                 resv....................0x0
>>                 attr_mod................0x0
>>                 m_key...................0x0000000000000000
>>                 dr_slid.................0xFFFF
>>                 dr_dlid.................0xFFFF
>> 
>>                 Initial path: [0][1][11][1][5][17][C]
>>                 Return path:  [0][0][0][0][0][0][0]
>>                 Reserved:     [0][0][0][0][0][0][0]
>> 
>>                 00 00 00 00 00 00 00 00   00 00 00 00 00 00 00 00
>> 
>>                 00 00 00 00 00 00 00 00   00 00 00 00 00 00 00 00
>> 
>>                 00 00 00 00 00 00 00 00   00 00 00 00 00 00 00 00
>> 
>>                 00 00 00 00 00 00 00 00   00 00 00 00 00 00 00 00
>> 
>> Mar 20 14:30:29 480017 [44606960] -> osm_report_notice: Reporting Generic 
>> Notice type:3 num:65 from LID:0x0092 
>> GID:0xfe80000000000000,0x0005ad0000024bbb
>> Mar 20 14:30:29 480030 [44606960] -> Removed port with 
>> GUID:0x0005ad000002510b LID range [0xB5,0xB5] of node:saguaro-23-6 HCA-1
>> Mar 20 14:30:29 480092 [44606960] -> osm_report_notice: Reporting Generic 
>> Notice type:3 num:65 from LID:0x0092 
>> GID:0xfe80000000000000,0x0005ad0000024bbb
>> Mar 20 14:30:29 480099 [44606960] -> Removed port with 
>> GUID:0x0005ad000002511b LID range [0xA6,0xA6] of node:saguaro-22-1 HCA-1
>> Mar 20 14:30:29 480121 [44606960] -> osm_report_notice: Reporting Generic 
>> Notice type:3 num:65 from LID:0x0092 
>> GID:0xfe80000000000000,0x0005ad0000024bbb
>> Mar 20 14:30:29 480128 [44606960] -> Removed port with 
>> GUID:0x0005ad0000024d47 LID range [0xB6,0xB6] of node:saguaro-23-7 HCA-1
>> Mar 20 14:30:29 480152 [44606960] -> osm_report_notice: Reporting Generic 
>> Notice type:3 num:65 from LID:0x0092 
>> GID:0xfe80000000000000,0x0005ad0000024bbb
>> Mar 20 14:30:29 480160 [44606960] -> Removed port with 
>> GUID:0x0005ad0000024d8b LID range [0xB7,0xB7] of node:saguaro-23-8 HCA-1
>> Mar 20 14:30:29 480325 [44606960] -> osm_drop_mgr_process: ERR 0108: Unknown 
>> remote side for node 0x0005ad0000027c84 port 12. Adding to light sweep 
>> sampling list
>> Mar 20 14:30:29 480343 [44606960] -> Directed Path Dump of 5 hop path:
>>                 Path = [0][1][11][1][5][17]
>> Mar 20 14:30:29 665327 [41E02960] -> __osm_trap_rcv_process_request: 
>> Received Generic Notice type:0x01 num:128 Producer:2 from LID:0x0148 
>> TID:0x000000000000007a
>> Mar 20 14:30:29 665355 [43C05960] -> __osm_trap_rcv_process_request: 
>> Received Generic Notice type:0x01 num:128 Producer:2 from LID:0x0152 
>> TID:0x000000000000001b
>> Mar 20 14:30:29 665397 [41E02960] -> osm_report_notice: Reporting Generic 
>> Notice type:1 num:128 from LID:0x0148 
>> GID:0xfe80000000000000,0x0005ad00000281b3
>> Mar 20 14:30:29 665404 [43C05960] -> __osm_trap_rcv_process_request: ERR 
>> 3804: Received trap 27 times consecutively
>> Mar 20 14:30:29 680658 [45A08960] -> osm_report_notice: Reporting Generic 
>> Notice type:3 num:64 from LID:0x0092 
>> GID:0xfe80000000000000,0x0005ad0000024bbb
>> Mar 20 14:30:29 680668 [45A08960] -> Discovered new port with 
>> GUID:0x0005ad00000249d3 LID range [0xB1,0xB1] of node:saguaro-23-2 HCA-1
>> Mar 20 14:30:29 680672 [45A08960] -> osm_report_notice: Reporting Generic 
>> Notice type:3 num:64 from LID:0x0092 
>> GID:0xfe80000000000000,0x0005ad0000024bbb
>> Mar 20 14:30:29 680678 [45A08960] -> Discovered new port with 
>> GUID:0x0005ad0000024cbb LID range [0xB2,0xB2] of node:saguaro-23-3 HCA-1
>> Mar 20 14:30:29 680681 [45A08960] -> osm_report_notice: Reporting Generic 
>> Notice type:3 num:64 from LID:0x0092 
>> GID:0xfe80000000000000,0x0005ad0000024bbb
>> Mar 20 14:30:29 680686 [45A08960] -> Discovered new port with 
>> GUID:0x0005ad0000024e0b LID range [0xB3,0xB3] of node:saguaro-23-4 HCA-1
>> Mar 20 14:30:29 680690 [45A08960] -> osm_report_notice: Reporting Generic 
>> Notice type:3 num:64 from LID:0x0092 
>> GID:0xfe80000000000000,0x0005ad0000024bbb
>> Mar 20 14:30:29 680695 [45A08960] -> Discovered new port with 
>> GUID:0x0005ad0000025043 LID range [0xB4,0xB4] of node:saguaro-23-5 HCA-1
>> Mar 20 14:30:29 711542 [45A08960] -> osm_ucast_mgr_process: Min Hop Tables 
>> configured on all switches
>> Mar 20 14:30:29 768280 [41401960] -> SUBNET UP
>> Mar 20 14:30:30 113195 [45A08960] -> osm_report_notice: Reporting Generic 
>> Notice type:3 num:64 from LID:0x0092 
>> GID:0xfe80000000000000,0x0005ad0000024bbb
>> Mar 20 14:30:30 113206 [45A08960] -> Discovered new port with 
>> GUID:0x0005ad000002510b LID range [0xB5,0xB5] of node:saguaro-23-6 HCA-1
>> Mar 20 14:30:30 113211 [45A08960] -> osm_report_notice: Reporting Generic 
>> Notice type:3 num:64 from LID:0x0092 
>> GID:0xfe80000000000000,0x0005ad0000024bbb
>> Mar 20 14:30:30 113216 [45A08960] -> Discovered new port with 
>> GUID:0x0005ad0000024d47 LID range [0xB6,0xB6] of node:saguaro-23-7 HCA-1
>> Mar 20 14:30:30 113220 [45A08960] -> osm_report_notice: Reporting Generic 
>> Notice type:3 num:64 from LID:0x0092 
>> GID:0xfe80000000000000,0x0005ad0000024bbb
>> Mar 20 14:30:30 113225 [45A08960] -> Discovered new port with 
>> GUID:0x0005ad0000024d8b LID range [0xB7,0xB7] of node:saguaro-23-8 HCA-1
>> Mar 20 14:30:30 113228 [45A08960] -> osm_report_notice: Reporting Generic 
>> Notice type:3 num:64 from LID:0x0092 
>> GID:0xfe80000000000000,0x0005ad0000024bbb
>> Mar 20 14:30:30 113233 [45A08960] -> Discovered new port with 
>> GUID:0x0005ad000002511b LID range [0xA6,0xA6] of node:saguaro-22-1 HCA-1
>> Mar 20 14:30:30 144149 [45A08960] -> osm_ucast_mgr_process: Min Hop Tables 
>> configured on all switches
>> Mar 20 14:30:30 195765 [4780B960] -> __osm_sm_mad_ctrl_rcv_callback: ERR 
>> 3111: Error status = 0x1C00
>> Mar 20 14:30:30 195850 [4780B960] -> SMP dump:
>>                 base_ver................0x1
>>                 mgmt_class..............0x81
>>                 class_ver...............0x1
>>                 method..................0x81 (SubnGetResp)
>>                 D bit...................0x1
>>                 status..................0x1C00
>>                 hop_ptr.................0x0
>>                 hop_count...............0x4
>>                 trans_id................0x96dcd
>>                 attr_id.................0x15 (PortInfo)
>>                 resv....................0x0
>>                 attr_mod................0x16
>>                 m_key...................0x0000000000000000
>>                 dr_slid.................0xFFFF
>>                 dr_dlid.................0xFFFF
>> 
>>                 Initial path: [0][1][14][2][4]
>>                 Return path:  [0][9][15][E][1]
>>                 Reserved:     [0][0][0][0][0][0][0]
>> 
>>                 00 00 00 00 00 00 00 00   00 00 00 00 00 00 00 00
>> 
>>                 00 00 00 00 00 00 00 00   00 00 00 00 01 03 03 02
>> 
>>                 14 52 00 11 40 40 00 08   08 04 F2 40 00 00 00 00
>> 
>>                 00 00 00 00 00 88 00 00   00 00 00 00 00 00 00 00
>> 
>> Mar 20 14:30:30 195929 [43C05960] -> osm_pi_rcv_process_set: Received error 
>> status 0x1c for SetResp() during ACTIVE transition
>> Mar 20 14:30:30 195942 [43C05960] -> PortInfo dump:
>>                 port number.............0x16
>>                 node_guid...............0x0005ad00000281b3
>>                 port_guid...............0x0005ad00000281b3
>>                 m_key...................0x0000000000000000
>>                 subnet_prefix...........0x0000000000000000
>>                 base_lid................0x0
>>                 master_sm_base_lid......0x0
>>                 capability_mask.........0x0
>>                 diag_code...............0x0
>>                 m_key_lease_period......0x0
>>                 local_port_num..........0x1
>>                 link_width_enabled......0x3
>>                 link_width_supported....0x3
>>                 link_width_active.......0x2
>>                 link_speed_supported....0x1
>>                 port_state..............ACTIVE
>>                 state_info2.............0x52
>>                 m_key_protect_bits......0x0
>>                 lmc.....................0x0
>>                 link_speed..............0x11
>>                 mtu_smsl................0x40
>>                 vl_cap_init_type........0x40
>>                 vl_high_limit...........0x0
>>                 vl_arb_high_cap.........0x8
>>                 vl_arb_low_cap..........0x8
>>                 init_rep_mtu_cap........0x4
>>                 vl_stall_life...........0xF2
>>                 vl_enforce..............0x40
>>                 m_key_violations........0x0
>>                 p_key_violations........0x0
>>                 q_key_violations........0x0
>>                 guid_cap................0x0
>>                 client_reregister.......0x0
>>                 subnet_timeout..........0x0
>>                 resp_time_value.........0x0
>>                 error_threshold.........0x88
>> Mar 20 14:30:30 195968 [43C05960] -> Capabilities Mask:
>> Mar 20 14:30:30 196144 [4780B960] -> __osm_sm_mad_ctrl_rcv_callback: ERR 
>> 3111: Error status = 0x1C00
>> Mar 20 14:30:30 196179 [4780B960] -> SMP dump:
>>                 base_ver................0x1
>>                 mgmt_class..............0x81
>>                 class_ver...............0x1
>>                 method..................0x81 (SubnGetResp)
>>                 D bit...................0x1
>>                 status..................0x1C00
>>                 hop_ptr.................0x0
>>                 hop_count...............0x4
>>                 trans_id................0x96dce
>>                 attr_id.................0x15 (PortInfo)
>>                 resv....................0x0
>>                 attr_mod................0x17
>>                 m_key...................0x0000000000000000
>>                 dr_slid.................0xFFFF
>>                 dr_dlid.................0xFFFF
>> 
>>                 Initial path: [0][1][14][2][4]
>>                 Return path:  [0][9][15][E][1]
>>                 Reserved:     [0][0][0][0][0][0][0]
>> 
>>                 00 00 00 00 00 00 00 00   00 00 00 00 00 00 00 00
>> 
>>                 00 00 00 00 00 00 00 00   00 00 00 00 01 03 03 02
>> 
>>                 14 52 00 11 40 40 00 08   08 04 F2 40 00 00 00 00
>> 
>>                 00 00 00 00 00 88 00 00   00 00 00 00 00 00 00 00
>> 
>> Mar 20 14:30:30 196248 [43204960] -> osm_pi_rcv_process_set: Received error 
>> status 0x1c for SetResp() during ACTIVE transition
>> Mar 20 14:30:30 196254 [43204960] -> PortInfo dump:
>>                 port number.............0x17
>>                 node_guid...............0x0005ad00000281b3
>>                 port_guid...............0x0005ad00000281b3
>>                 m_key...................0x0000000000000000
>>                 subnet_prefix...........0x0000000000000000
>>                 base_lid................0x0
>>                 master_sm_base_lid......0x0
>>                 capability_mask.........0x0
>>                 diag_code...............0x0
>>                 m_key_lease_period......0x0
>>                 local_port_num..........0x1
>>                 link_width_enabled......0x3
>>                 link_width_supported....0x3
>>                 link_width_active.......0x2
>>                 link_speed_supported....0x1
>>                 port_state..............ACTIVE
>>                 state_info2.............0x52
>>                 m_key_protect_bits......0x0
>>                 lmc.....................0x0
>>                 link_speed..............0x11
>>                 mtu_smsl................0x40
>>                 vl_cap_init_type........0x40
>>                 vl_high_limit...........0x0
>>                 vl_arb_high_cap.........0x8
>>                 vl_arb_low_cap..........0x8
>>                 init_rep_mtu_cap........0x4
>>                 vl_stall_life...........0xF2
>>                 vl_enforce..............0x40
>>                 m_key_violations........0x0
>>                 p_key_violations........0x0
>>                 q_key_violations........0x0
>>                 guid_cap................0x0
>>                 client_reregister.......0x0
>>                 subnet_timeout..........0x0
>>                 resp_time_value.........0x0
>>                 error_threshold.........0x88
>> Mar 20 14:30:30 196269 [43204960] -> Capabilities Mask:
>> Mar 20 14:30:30 201633 [45007960] -> SUBNET UP
>> Mar 20 14:30:30 278051 [43C05960] -> __osm_trap_rcv_process_request: 
>> Received Generic Notice type:0x01 num:128 Producer:2 from LID:0x0152 
>> TID:0x000000000000001c
>> Mar 20 14:30:30 278107 [43C05960] -> __osm_trap_rcv_process_request: ERR 
>> 3804: Received trap 28 times consecutively
>> Mar 20 14:30:30 278656 [41E02960] -> __osm_trap_rcv_process_request: 
>> Received Generic Notice type:0x01 num:128 Producer:2 from LID:0x0148 
>> TID:0x000000000000007b
>> Mar 20 14:30:30 278871 [41E02960] -> osm_report_notice: Reporting Generic 
>> Notice type:1 num:128 from LID:0x0148 
>> GID:0xfe80000000000000,0x0005ad00000281b3
>> Mar 20 14:30:30 279653 [45007960] -> __osm_trap_rcv_process_request: 
>> Received Generic Notice type:0x01 num:128 Producer:2 from LID:0x001B 
>> TID:0x000000000000008d
>> Mar 20 14:30:30 279765 [45007960] -> osm_report_notice: Reporting Generic 
>> Notice type:1 num:128 from LID:0x001B 
>> GID:0xfe80000000000000,0x0005ad00000281a7
>> Mar 20 14:30:30 568539 [43C05960] -> __osm_trap_rcv_process_request: 
>> Received Generic Notice type:0x01 num:128 Producer:2 from LID:0x001B 
>> TID:0x000000000000008e
>> Mar 20 14:30:30 568617 [43C05960] -> osm_report_notice: Reporting Generic 
>> Notice type:1 num:128 from LID:0x001B 
>> GID:0xfe80000000000000,0x0005ad00000281a7
>> Mar 20 14:30:30 607916 [45A08960] -> __osm_trap_rcv_process_request: 
>> Received Generic Notice type:0x01 num:128 Producer:2 from LID:0x0148 
>> TID:0x000000000000007c
>> Mar 20 14:30:30 625139 [44606960] -> osm_ucast_mgr_process: Min Hop Tables 
>> configured on all switches
>> Mar 20 14:30:30 663838 [45A08960] -> osm_report_notice: Reporting Generic 
>> Notice type:1 num:128 from LID:0x0148 
>> GID:0xfe80000000000000,0x0005ad00000281b3
>> Mar 20 14:30:30 664569 [44606960] -> __osm_trap_rcv_process_request: 
>> Received Generic Notice type:0x01 num:128 Producer:2 from LID:0x001B 
>> TID:0x000000000000008f
>> Mar 20 14:30:30 664747 [44606960] -> osm_report_notice: Reporting Generic 
>> Notice type:1 num:128 from LID:0x001B 
>> GID:0xfe80000000000000,0x0005ad00000281a7
>> Mar 20 14:30:30 679262 [45A08960] -> SUBNET UP
>> Mar 20 14:30:30 784024 [43204960] -> __osm_trap_rcv_process_request: 
>> Received Generic Notice type:0x01 num:128 Producer:2 from LID:0x001B 
>> TID:0x0000000000000090
>> Mar 20 14:30:30 784123 [43204960] -> osm_report_notice: Reporting Generic 
>> Notice type:1 num:128 from LID:0x001B 
>> GID:0xfe80000000000000,0x0005ad00000281a7
>> Mar 20 14:30:30 804217 [41401960] -> __osm_trap_rcv_process_request: 
>> Received Generic Notice type:0x01 num:128 Producer:2 from LID:0x001B 
>> TID:0x0000000000000091
>> Mar 20 14:30:30 807807 [41401960] -> osm_report_notice: Reporting Generic 
>> Notice type:1 num:128 from LID:0x001B 
>> GID:0xfe80000000000000,0x0005ad00000281a7
>> Mar 20 14:30:30 825500 [45A08960] -> __osm_trap_rcv_process_request: 
>> Received Generic Notice type:0x01 num:128 Producer:2 from LID:0x001B 
>> TID:0x0000000000000092
>> Mar 20 14:30:30 825600 [45A08960] -> osm_report_notice: Reporting Generic 
>> Notice type:1 num:128 from LID:0x001B 
>> GID:0xfe80000000000000,0x0005ad00000281a7
>> Mar 20 14:30:30 988887 [43C05960] -> __osm_trap_rcv_process_request: 
>> Received Generic Notice type:0x01 num:128 Producer:2 from LID:0x001B 
>> TID:0x0000000000000093
>> Mar 20 14:30:30 988978 [43C05960] -> osm_report_notice: Reporting Generic 
>> Notice type:1 num:128 from LID:0x001B 
>> GID:0xfe80000000000000,0x0005ad00000281a7
>> Mar 20 14:30:31 059298 [41401960] -> osm_ucast_mgr_process: Min Hop Tables 
>> configured on all switches
>> Mar 20 14:30:31 106840 [41E02960] -> __osm_trap_rcv_process_request: 
>> Received Generic Notice type:0x01 num:128 Producer:2 from LID:0x001B 
>> TID:0x0000000000000094
>> Mar 20 14:30:31 111335 [41E02960] -> osm_report_notice: Reporting Generic 
>> Notice type:1 num:128 from LID:0x001B 
>> GID:0xfe80000000000000,0x0005ad00000281a7
>> Mar 20 14:30:31 112465 [4780B960] -> __osm_sm_mad_ctrl_rcv_callback: ERR 
>> 3111: Error status = 0x1C00
>> Mar 20 14:30:31 112497 [4780B960] -> SMP dump:
>>                 base_ver................0x1
>>                 mgmt_class..............0x81
>>                 class_ver...............0x1
>>                 method..................0x81 (SubnGetResp)
>>                 D bit...................0x1
>>                 status..................0x1C00
>>                 hop_ptr.................0x0
>>                 hop_count...............0x4
>>                 trans_id................0x98837
>>                 attr_id.................0x15 (PortInfo)
>>                 resv....................0x0
>>                 attr_mod................0x18
>>                 m_key...................0x0000000000000000
>>                 dr_slid.................0xFFFF
>>                 dr_dlid.................0xFFFF
>> 
>>                 Initial path: [0][1][16][1][5]
>>                 Return path:  [0][9][13][D][2]
>>                 Reserved:     [0][0][0][0][0][0][0]
>> 
>>                 00 00 00 00 00 00 00 00   00 00 00 00 00 00 00 00
>> 
>>                 00 00 00 00 00 00 00 00   00 00 00 00 02 03 03 02
>> 
>>                 11 42 00 11 40 40 00 08   08 04 F2 40 00 00 00 00
>> 
>>                 00 00 00 00 00 88 00 00   00 00 00 00 00 00 00 00
>> 
>> Mar 20 14:30:31 112593 [44606960] -> osm_pi_rcv_process_set: ERR 0F10: 
>> Received error status for SetResp()
>> Mar 20 14:30:31 112627 [44606960] -> PortInfo dump:
>>                 port number.............0x18
>>                 node_guid...............0x0005ad00000281a7
>>                 port_guid...............0x0005ad00000281a7
>>                 m_key...................0x0000000000000000
>>                 subnet_prefix...........0x0000000000000000
>>                 base_lid................0x0
>>                 master_sm_base_lid......0x0
>>                 capability_mask.........0x0
>>                 diag_code...............0x0
>>                 m_key_lease_period......0x0
>>                 local_port_num..........0x2
>>                 link_width_enabled......0x3
>>                 link_width_supported....0x3
>>                 link_width_active.......0x2
>>                 link_speed_supported....0x1
>>                 port_state..............DOWN
>>                 state_info2.............0x42
>>                 m_key_protect_bits......0x0
>>                 lmc.....................0x0
>>                 link_speed..............0x11
>>                 mtu_smsl................0x40
>>                 vl_cap_init_type........0x40
>>                 vl_high_limit...........0x0
>>                 vl_arb_high_cap.........0x8
>>                 vl_arb_low_cap..........0x8
>>                 init_rep_mtu_cap........0x4
>>                 vl_stall_life...........0xF2
>>                 vl_enforce..............0x40
>>                 m_key_violations........0x0
>>                 p_key_violations........0x0
>>                 q_key_violations........0x0
>>                 guid_cap................0x0
>>                 client_reregister.......0x0
>>                 subnet_timeout..........0x0
>>                 resp_time_value.........0x0
>>                 error_threshold.........0x88
>> Mar 20 14:30:31 112673 [44606960] -> Capabilities Mask:
>> Mar 20 14:30:31 113808 [4780B960] -> __osm_sm_mad_ctrl_rcv_callback: ERR 
>> 3111: Error status = 0x1C00
>> Mar 20 14:30:31 113838 [4780B960] -> SMP dump:
>>                 base_ver................0x1
>>                 mgmt_class..............0x81
>>                 class_ver...............0x1
>>                 method..................0x81 (SubnGetResp)
>>                 D bit...................0x1
>>                 status..................0x1C00
>>                 hop_ptr.................0x0
>>                 hop_count...............0x4
>>                 trans_id................0x9883e
>>                 attr_id.................0x15 (PortInfo)
>>                 resv....................0x0
>>                 attr_mod................0x18
>>                 m_key...................0x0000000000000000
>>                 dr_slid.................0xFFFF
>>                 dr_dlid.................0xFFFF
>> 
>>                 Initial path: [0][1][11][1][4]
>>                 Return path:  [0][9][18][D][1]
>>                 Reserved:     [0][0][0][0][0][0][0]
>> 
>>                 00 00 00 00 00 00 00 00   00 00 00 00 00 00 00 00
>> 
>>                 00 00 00 00 00 00 00 00   00 00 00 00 01 03 03 02
>> 
>>                 14 52 00 11 40 40 00 08   08 04 F2 40 00 00 00 00
>> 
>>                 00 00 00 00 00 88 00 00   00 00 00 00 00 00 00 00
>> 
>> Mar 20 14:30:31 113925 [43204960] -> osm_pi_rcv_process_set: Received error 
>> status 0x1c for SetResp() during ACTIVE transition
>> Mar 20 14:30:31 113930 [43204960] -> PortInfo dump:
>>                 port number.............0x18
>>                 node_guid...............0x0005ad00000281b3
>>                 port_guid...............0x0005ad00000281b3
>>                 m_key...................0x0000000000000000
>>                 subnet_prefix...........0x0000000000000000
>>                 base_lid................0x0
>>                 master_sm_base_lid......0x0
>>                 capability_mask.........0x0
>>                 diag_code...............0x0
>>                 m_key_lease_period......0x0
>>                 local_port_num..........0x1
>>                 link_width_enabled......0x3
>>                 link_width_supported....0x3
>>                 link_width_active.......0x2
>>                 link_speed_supported....0x1
>>                 port_state..............ACTIVE
>>                 state_info2.............0x52
>>                 m_key_protect_bits......0x0
>>                 lmc.....................0x0
>>                 link_speed..............0x11
>>                 mtu_smsl................0x40
>>                 vl_cap_init_type........0x40
>>                 vl_high_limit...........0x0
>>                 vl_arb_high_cap.........0x8
>>                 vl_arb_low_cap..........0x8
>>                 init_rep_mtu_cap........0x4
>>                 vl_stall_life...........0xF2
>>                 vl_enforce..............0x40
>>                 m_key_violations........0x0
>>                 p_key_violations........0x0
>>                 q_key_violations........0x0
>>                 guid_cap................0x0
>>                 client_reregister.......0x0
>>                 subnet_timeout..........0x0
>>                 resp_time_value.........0x0
>>                 error_threshold.........0x88
>> Mar 20 14:30:31 113946 [43204960] -> Capabilities Mask:
>> Mar 20 14:30:31 119007 [43204960] -> SUBNET UP
>> Mar 20 14:30:31 128758 [45007960] -> __osm_trap_rcv_process_request: 
>> Received Generic Notice type:0x01 num:128 Producer:2 from LID:0x001B 
>> TID:0x0000000000000095
>> Mar 20 14:30:31 128851 [45007960] -> osm_report_notice: Reporting Generic 
>> Notice type:1 num:128 from LID:0x001B 
>> GID:0xfe80000000000000,0x0005ad00000281a7
>> Mar 20 14:30:31 150370 [44606960] -> __osm_trap_rcv_process_request: 
>> Received Generic Notice type:0x01 num:128 Producer:2 from LID:0x001B 
>> TID:0x0000000000000096
>> Mar 20 14:30:31 150468 [44606960] -> osm_report_notice: Reporting Generic 
>> Notice type:1 num:128 from LID:0x001B 
>> GID:0xfe80000000000000,0x0005ad00000281a7
>> Mar 20 14:30:31 316422 [41401960] -> __osm_trap_rcv_process_request: 
>> Received Generic Notice type:0x01 num:128 Producer:2 from LID:0x0152 
>> TID:0x000000000000001c
>> Mar 20 14:30:31 316498 [41401960] -> __osm_trap_rcv_process_request: ERR 
>> 3804: Received trap 29 times consecutively
>> 
>> _______________________________________________
>> general mailing list
>> general at lists.openfabrics.org
>> http://lists.openfabrics.org/cgi-bin/mailman/listinfo/general
>> 
>> To unsubscribe, please visit 
>> http://openib.org/mailman/listinfo/openib-general
>