[ofa-general] osm error messages
Hal Rosenstock
halr at voltaire.com
Wed Mar 21 12:53:23 PDT 2007
On Wed, 2007-03-21 at 13:29, Douglas Fuller wrote:
> I'm seeing some sporadic error activity from OpnSM (OFED 1.1; osm.log
> below) that ay correlate with some ob failures -- I'm trying to get to the
> bottom of this.
>
> Before seeing this, I isolatedand disabled with ibportstate) what ppeared
> to be a ba internal port in one of our core switches. That leads me to
> suspectI have a switchmisbehaving somwhere.
>
> Without any other intervention, things seem to check out (wth
> ibdiagnet/ibchecknet). Any thought? Need any more nformatin?
Is something bouncing your subnet or was this just what ibportstate did
? It could be if this was a core switch.
Also, you may have some SMAs which have gone nonresponsive to SMPs
(IB_TIMEOUTs) but the links are up. I can't be sure not knowing what the
exact scenario was. If you do, you will like want to chase these and do
something about them if you haven't already.
All the messages relating to ACTIVE -> ACTIVE transition can be ignored.
Also, it looks like something is removing characters in the log.
-- Hal
> Thanks,
> --Doug Fuller
>
> Mar 19 18:8:50 000354 [AB000160] -> OpenSM Rev:openib-2.0.5 OpnIBsvn
> Exported revision
> Mar 19 18:28:50 000466 [AB00160] -> OpenSM Rev:openib-2.0.5 OpenIB svn
> Exported revision
> Mar 19 18:28:50 007666 [AB000160] -> om_vendor_bind: Binding to port
> 0x5ad0000024bb
> Mar 19 18:28:50 011279 [AB000160] ->osm_vendor_bind: Binding to port
> 0x5ad0000024bbb
> Mar 19 18:2850 438326 [44606960] -> Entering MASTER stte
> Mar 19 18:28:50 438628 [4606960] -> osm_report_notice: Reporting Generic
> Notice type:3 num:66 from LID:0x0000
> GID:0xfe8000000000000,0x0005ad000024bbb
> Mar 19 1828:50 438661 [44606960] -> osm_report_notice: Reporting Generic
> Notice type:3 num:6 from LID:0x0000
> GID:0xf8000000000000,0x0005ad0000024bbb
> Mar 1 18:28:50 504176 [41401960] -> osm_cast_mgr_process: Min Hop Tabes
> onfigured on all switches
> Mar 19 18:28:50 639453 [44606960] -> SUNET UP
> Mar 19 18:28:50 853613 [1E02960 -> __osm_trap_rcv_process_reqest:
> Received Generic Notice type:0x04 num:144 Producer:1 from LID:0x0092
> TID:0x000000000000018
> Mar 19 18:28:5 853813 [41E0960] -> osm_report_notice: Reporting Generic
> Notice typ:4 num:144 from LID:0x0092
> GID:0xfe8000000000000,0x0005ad0000024bbb
> Mar 19 18:28:51 273470 [4460960] -> osm_ucast_mgr_process: Min HopTables
> configured on all switches
> Mar 19 18:28:51 33730 [43C05960] -> SUBNET UP
> Mar 19 18:3033 565682 [4320490] -> __osm_trap_rcv_process_request:
> Received Generic Notice type:0x1 num:128 Poducer:2 from LID:0x0001
> TID:0x0000000000000019
> Mar 19 18:30:33 565958 [43204960] -> sm_report_notice: Reporting Generic
> Noicetype:1 num:128 from ID:0x0001
> GID:0xfe80000000000000,0x005ad0000027c6
> Mar 19 18:30:33 963901 [41401960] > osm_report_notice: Reporting Generic
> Notice typ:3 num:64 fro LID:x0092
> GID:0xfe80000000000000,0x005ad0000024bbb
> Mar 19 18:30:33 963914 [41401960] -> Discovered nw port with
> GUID:0x0005ad00000297b LI range [0x3,0x37] of node:saguro-14-9 HCA-1
> Mar 19 18:30:33 994698 [4401960] > om_ucast_mgr_process: Min Hp Tables
> configured n all switches
> Mar 19 18:30:34 054763 [45A08960]-> UBNET UP
> Mar 19 18:30:34 351397 [43C0560] -> __osm_trap_rcv_process_request:
> Received Generi Ntice type:0x04 num:144 Producer:1 fomLID:0x0037
> TID:0x000000000000000
> Mar 19 18:30:34 351615 [43C05960] -> osm_report_notice Reportig Generic
> otice type:4 num:144 from LID:0x0037
> GID:0xfe80000000000000,0x0005ad00002497b
> ar 19 18:30:34 777488 [45A08960] -> osm_ucast_mgr_process:Min Hop Tables
> configured onall switces
> Mar 19 18:30:34 832664 [4A08960] -> SUBNET UP
> Mar 19 18:32:27 476136 [45A08960] -> __osm_trap_cv_process_request:
> Received Generic Notice typ:0x01 num:128 Producer:2 from LID:0x0148
> TID:0x000000000000002b
> Mar 19 18:32:27 476340 [43204960] ->__osm_trap_rcv_process_request:
> Reeivd Gneric Notice type:0x01 num:128 Producer:2 from LID:0x001B
> TID:0x000000000000037
> Mar 19 18:32:2 476389 [45A08960] -> osm_report_notice: Reporting Generic
> Notice type: num:128 from LID:0x0148
> GID:0xfe8000000000000,0x0005ad00000281b3
> Mar 19 18:2:27 47485 [43204960] -> osm_report_ntice: Reporting Generic
> Notice tye:1 num:128 from LID:0x001B
> GID:0xfe80000000000000,0x0005ad0000081a7Mar 19 18:32:27 817617 [42803960] -> osm_report_notice: Reporting Generic
> Notice type:3 nm:65 frm LID:0x0092
> GID:0xfe80000000000000,0x005ad000024bbb
> Mar 19 18:32:27 817637 [4280396] -> Remove port with
> GUID:0x0005ad0000024e0b LID range [0xB3,xB3] of node:saguaro-23-4 HCA-1
> Mar19 18:32:27 817655 [42803960] -> sm_report_notice: Reporting Generic
> Notice type:3 num:65 from LID:0x0092
> GID:0xfe80000000000000,00005ad0000024bbb
> Mar 9 18:32:27 81766 [42803960] -> Remove port with
> GUID:0x0005ad000002510b LID range [0xB5,0xB5] of node:saguaro-23-6 HCA-1
> Mar 1 18:32:2 817694 [42803960] -> osm_report_notice: Reporting Generic
> Ntice type:3 num:65 from LID:0x0092
> GID:0xfe80000000000000,00005ad0000024bbb
> Mar 19 18:3:7 81769 [42803960] -> Removed port wth
> GUID0x0005ad000002511b LID range [0xA6,0xA6] of node:saguaro-22-1 HCA-1
> Mar 19 18:3227 817716 42803960] -> osm_report_ntice: ReportingGeneric
> Notice type:3 num:65 from LID:0x0092
> GID:0xfe80000000000000,0x0005ad0000024bbb
> Mar 19 18:32:27 81771 [42803960] -> Remved port with
> GUID0x0005ad0000024b27 LID range [0xAF,0xAF] of node:sguaro-23-0 HCA-1
> Mar 19 18:32:27 817738 [4280390] -> osm_report_notice: Reporting Generic
> Notic type:3 num:65 from LID:0x0092
> GID:0xfe80000000000000,0x0005ad0000024bbb
> Mar 19 18:32:27 817743 [4203960] -> Removed ort with
> GUID:0x000ad0000025043 LID range [0xB4,0xB4] of node:saguaro-23-5 HCA-1
> Mar 19 18:32:27 817758 [42803960] - osm_report_notice: Reporing Generic
> Notice type:3 num:65 from LID:0x0092
> GID:0xfe8000000000000,0x0005ad0000024bbb
> Mar 19 18:32:27 817763 [42803960 -> Removed port with
> GUID:0x0005ad000024d7 LID range [0xB6,0xB6] of node:saguar-23-7 HCA-1
> Mar 19 18:32:27 817780 [42803960] -> osm_rport_notice: Reporting Generic
> Notice type:3 nu:65 fromLID:0x0092
> GID:0xfe80000000000000,0x0005ad0000024bbb
> Mar 19 18:32:27 17785 [42803960] -> Remved port with
> GUID:0x0005ad0000024d6bLID range [0xB8,0xB8] of node:saguao-23-9 HCA-1
> Mar 19 18:32:27 817803 [480396] -> osm_report_notce: Reporting Generic
> Notice type:3 num:65 from LID:0x0092
> GID:0xfe80000000000000,0x0005ad000024bbb
> Mar 19 18:32:27 817808 [4283960] -> Removed por with
> GUID:0x0005ad0000024977 LID rane [0xA9,0xA9] of node:saguaro-22-4 HCA1
> Mar 19 18:32:27 817932 [42803960] -> osm_report_notice: Rporting Generic
> Notice type:3 num:65 from LID:0x0092
> GID:0xfe80000000000000,0x0005ad000024bbb
> Mar 19 18:32:27 817938 [4280390] -> Removed port with
> GID:0x0005ad0000027c84 LID range [0x15,0x152] of node:Topspin Switch TS20
> Mar 19 18:32:27 817970 [42803960] -> osm_report_notice: Reporing Generic
> Notice type:3 num:65 from LID:0x0092
> GID:0xfe80000000000000,0x0005ad000024bbb
> Mar 19 18:32:27 817977 [4280360] -> Removed port with
> GUID0x0005ad0000024d8b LID range [0xB7,0xB7] of node:saguaro-23-8 HCA-1Mar 19 1:32:27 817992 [42803960] -> osm_report_notice: Reporting Generic
> Notice tye:3 num:65 from LID:0x0092
> GID:0xfe80000000000000,0x0005ad000004bbb
> Mar 19 18:32:27 817997 [42803960 -> Removed port with
> GUID:0x0005ad00000249f LID range [0xA8,0xA8] of node:saguaro-22-3 HCA-1
> Ma 19 18:32:27 81811 [42803960] -> osm_report_notice: Reporting Generic
> Notice type:3 num:65 from LID:0x0092
> GID:0xe80000000000000,0x0005ad0000024bb
> Mar 9 18:32:27 818016 [42803960] - Removed port with
> UID:0x0005ad0000024c9b LID range [0xA7,0xA7] of node:saguaro-2-2 HCA-1
> Mar 1 18:32:27 818032 [42803960] -> osm_report_notice: Reporing Generic
> Notice type:3 num:65 from LID:0x0092
> GID:0xfe80000000000000,0x0005ad0000024bbb
> a 19 18:32:27 818037 [42803960] -> Rmoved port with
> GUID:0x0005ad000004da7 LID range [0xB0,0xB0] of node:saguaro-23-1 HCA-1
> Mar 19 1:32:27 818054 [4280360] -> osm_repot_notice: Reporting Generic
> Notice typ:3 num:65 from LID:0x0092
> GID:0xfe80000000000000,x0005ad0000024bbb
> Mar19 18:32:27 818115 [42803960] -> Remoed port wth
> GUID:0x0005ad0000024cbb LID range [xB2,0xB2] of node:saguaro-23-3 HCA-1
> Mar 1 18:3:27 81812 [42803960] -> osm_report_notice: Reorting Generic
> Notie type:3 num:65 from LID:0x0092
> GID:0xfe80000000000000,0x0005ad000002bbb
> Mar1918:32:27 818137 [42803960] -> Removedport with
> GUID:0x0005ad00000249d3 LID range [0xB1,0B1] of node:saguaro-23-2 HCA-1
> Mar 19 8:32:2 818153 [4280360] -> osm_report_notice: Reporting Generic
> Notice type:3 num:65 from LID:0x0092
> GID:0xfe800000000000,0x0005ad0000024bbb
> Mar 19 1832:7 818158 [42803960] -> Removedpot with
> GUID:0x0005ad0000024feb LID range [0x153,0x153] of node:saguaro-2-5 HCA-1
> Mar 19 18:32:7818173 [42803960] -> osm_report_notic: Reporting Generic
> Notice type:3 num:65 from LID:0x0092
> GID:0xfe80000000000000,0x0005ad000024bbb
> Mar 19 18:322 818178 [42803960 -> Removed port wih
> GUID:0x0005ad0000024afb LID rnge [0xA5,0xA5] of node:saguaro-22-0 HCA-1
> Mar 19 183227 851249[42803960] -> osm_ucast_mgr_procss: Min Hop Tables
> configured on all sitches
> Mar 19 18:32:27 898524 [43204960] -> SUBNET UP
> Mar 19 8:32:2828664 [45007960] -> osm_ucast_mgr_proessMin Hop Tables
> confiured on all switches
> Mar 19 18:32:28 341659 [4466960] -> SUBNET UP
> Mar 19 1:33:21814615 [41E0296] -> __osm_trap_rcv_process_request:
> Rceived Gneric Notice type:0x01 num:128 Produce:2 from LID:0x0148
> TID:0x00000000000002c
> Mar 19 8:33:21 81484 [4E02960]-> osm_report_otice: Reporting Generi
> Notice type:1 num:128 fom LID:0x0148
> GID:0xfe8000000000000,0x0005ad0000281b3
> Mar19 18:3321 820835 [41E02960] -> __osm_tap_rcv_process_request:
> Received Genei Notce type:0x01 num:128 Producer:2 fro LI:0x001B
> TID:0x000000000000008
> Ma 19 18:33:21 82090 [41E02960] -> smreport_notice: Repoting Generic
> Notice tye:1 num:128 fromLID:0x001B
> GD:0xfe80000000000000,0x0005ad0000281a7
> ar 19 18:33:21 82038 [41E02960] -> __osm_trap_rcv_processrequest:
> Received Generic Notce tpe:0x num:128 Producer:2 from LID:00148
> TID0x00000000000002d
> Mar 19 18:33:21 820992 [1E060] -> osm_report_notice:Reporting Gnric
> Notice type:1 num:128from LID:0x0148
> GID:0xfe8000000000000,0x005ad00000281b3
> Mar 19 18:3:21 826779 [4102960] -> __osm_trap_rcv_process_reqest:
> Receivd Generic Notice type0x0 num:128 Prducr:2 from LID:0x001B
> TID0x000000000000039
> Mar 19 18:33:21 82742 [41E02960] - om_report_notice: Reporting Generc
> Ntice type:1 num:128 from LID:0x001B
> GID:0xfe8000000000000,0x0005a00000281a7Mar 19 18:33:22 164580 [45007960 > osm_drop_mgr_process: ERR 0108: Ukown
> remote side fo node 0x0005a0000027c84 port 18. Addi to light sweep
> sampling list
> Mar 191:3:22 164654 [45007960] -> Direced PathDump of 5 hop path:
> Path = [0][1][11][1][5][17]
> Mar19 18:33:22 164712 [45007960] -> osm_op_mgrprocess: ERR 0108: Unknown
> reote sde for node 0x0005ad00000281b3 port2. Adding to light swep
> sampling lit
> Mar 19 18:33:22164724 45007960] -> irected Path Dump of hop path:
> Path = [0][1[11][1][5]
> Mar 9 18:33:22 173634 [43C0960] ->osm_report_notic: Reporting Generic
> Notice type:3 num:64 fromID0x0092
> GID:0xfe80000000000000,0x005ad0000024bbb
> Mar 19 18:33:22 17365 [43C05960] -> iscovered e port with
> GUID:0x0005ad000027c4 LIDrange [0x152,0x152] of node:Toppin Switch TS120
> Mar 19 18:33:22 17365 [43C05960] -> osm_reportnotice:RepotingGeneric
> Notice type:3 num:64from LID:0x0092
> GID:0xfe8000000000000,0x05ad0000024bbb
> Mar 19 18:33:22 173662 [43C05960] -> icovered newpot with
> GUID:0x005ad0000024b27 ID rage [0xAF,0xAF] o ode:saguaro-23- HCA-1
> Mar 19 18:33:22 17366 [43C05960] -> osm_report_notice: Reprting Gneric
> otice type: num:64 from LID:0x0092
> GI0xfe80000000000000,0x0005ad0000024bbbMar19 18:33:22 173671[43C05960] -> Discovered new port with
> GUID:0x005ad0000024da7 LID rage 0xB00xB0] ofnode:saguar-23-1 HCA-1
> Mar 19 18:33:22 173675 [43C0596] -> osm_report_notice: Rerting Generic
> Notice typ:3 num:64 fro LID:0x0092
> GID:0xfe800000000000,0x0005ad0000024bbb
> Mar 9 18:33:22173680 [43C05960] -> Discoveed new port withGUID:0x005ad00000249d3 LD range [0xB1,0xB1] of node:saguaro-2-2 HC1
> Mar 1918:33:22 173684[43C05960] ->osm_report_notice: Reporting Generic
> Notice type:3 num:6 fro LI:0x002
> GID:0xfe80000000000000,0x005ad000002bb
> Mar 19 18:33:22 173689 [43C05960] -> Dicovered new port with
> GUI:0x0005ad000024cbb LID range [0xB,0xB] of node:saguaro-23-3 HCA-1
> Mar 19 1:33:22 173693 [3C0960] ->osm_report_notice: Rporting Gneric
> Notice type:3 num:64 fro LID:0x0092
> GID:0xfe000000000000,0x005ad0000024bbb
> Mar 19 18:33:22 173697 43C0596] -> Discoverednew port with
> GUID:0x005ad000024e0b LI range [0xB3,0xB3] of nod:saguaro23-4 HCA-1
> Mar 19 1833:22 173701 [43C5960] -> os_report_otice: Reporting Genric
> Notice type:3 num:4 from LID:0x0092
> GID:0xfe80000000000000,0x005d0000024bbb
> Mr 19 1833:22 173706 [43C05960] -> Dscovered new port with
> GUID:0x0005ad00025043 LID ange [0xB4,0xB4] ofnode:saguaro-23-5 HA-1
> Mar 1 18:33:2 173710 43C05960] -> osm_repo_notice: Reporting Generic
> Notice type:3 nm:64 from LID:0x0092
> ID:0xfe800000000000,0x0005ad0000024bbb
> Ma 1918:33:22 173715 [43C0596] -> Discverednew port with
> GUID:0x0005ad00002510b LID range [0xB5,0xB5 of node:saguaro-23-6 HCA1
> Mar 19 18:3:2 173719 [3C05960] -> osm_report_ntic: Reportin Generic
> Notice type:3 num:64 from LID:0x0092
> GID0xfe8000000000000,0x0005ad0000024bbb
> Mar 19 18:33:22 1723 [43C05960]-> Disoveed new port wth
> GUID:0x0005ad000002447 LID range [0xB6,0xB6] of node:saguaro-23-7 HCA-1Mar 1 18:33:22 173727 43C05960] -> osm_rert_notie: Reporting Generic
> Notice type:3 num:64 from LID:0x0092
> GID:0fe80000000000000,0x0005ad000004bbb
> Mar 9 18:33:22 17373 [43C05960] - Discoverednew port wit
> GUID:0x0005ad000024d8bLID range [0xB7,0xB7] of node:saguaro-23-8 HCA-1
> Mr 19 18:33:22 173736 [4C05960] -> os_report_notice:Reporting Generic
> Notic type:3 num:64 from LID:0x0092
> GID:0xf80000000000000,0x0005ad0000024bbb
> Mar 19 18:33:22 173741 43C05960] ->Dscovered ne port with
> GID:0x0005ad0000024d6b LI range [0xB8,0xB8] ofnode:saguaro-23-9 HCA-1
> Mar19 18:33:22 173744 [43C0960] -> osm_report_notice: Reorting Generic
> otice type:3 num:64 from LID:0x0092
> GI:0xfe8000000000000,0x0005ad0000024bbb
> Mar 19 18:3:22 173749 [4305960] -> Discovered new prt with
> GUI:0x0005d0000024afb LID rnge [0xA5,0xA5] of noe:saguaro-22-0 HCA-1
> Mar 19 1833:22 13753 [43C05960] -> om_report_notice: Reporing Generic
> Notice type:3 num:64 from LID:0x002
> GID:0xfe8000000000000,0x0005ad00004bbb
> Mar 19 18:33:22 17758 [43C0596] -> Discovered new portwith
> GUID:00005ad000002511b LID rang [0xA6,0A6] f node:saguao-22-1 HCA-1
> Mar 19 18:33:22 173762 [43C05960] -> osm_reort_notice: Reportin Geneic
> Notice type:3 num:64 from LID:0x0092
> GID:0xfe80000000000000,0x005ad000024bbb
> Mar 19 18:33:22 17376 [3C0960] -> Discovered new port wihGUID:0x0005ad0000024c9b LID range [xA7,0xA7] of node:saguaro-222 HCA-1
> Mar 19 8:33:22 173770 [43C05960] -> osm_report_notice Reporting Gneric
> Notie type:3 nm:64 from LID:0x0092
> GID:0xfe80000000000000,0x0005ad000024bbb
> Mar 1918:33:22 173830 43C0590]-> Discovered new port with
> ID:0x0005ad000002498f LID range [0xA,0xA8]of node:saguaro-22-3 HCA-1
> Mar 9 18:33:22 173834 [43C05960] -> osm_eport_notice: Reporting Geneic
> Notice type:3num:64 from LID:0x0092
> GD:0xfe80000000000000,0x0005ad0000024bb
> Mar 1 18:33:22 173839 [43C05960] -> Discovered new port ith
> GUI:0x005ad0000024977 LID range [xA9,0A9 of node:saguaro-22-4 HCA-1
> Mar 19 18:33:22 173843 [43C05960] ->osm_report_notice: Reporting GenericNotice ype: num:64 from LID:0x0092
> GID:0xfe0000000000000,0x0005ad0000024bbb
> Mar 9 8:33:22 173848 [43C05960] -> Discovered new port with
> GUD:00005ad0000024feb LID range [0x153,x13 of node:saguaro-22-5 HCA-1
> Mar 19 18:33:22 204620 [43C05960] -> osm_cast_mgr_process: Min Hop Tablesconfgued on all switches
> Mar 19 18:33:22 278567 [45A08960] -> SUNET UP
> Mar19 18:33:22 664286 [4141960] -> osm_ucast_mgr_process: Min Hop Table
> configured on all switces
> Mr 19 1833:22 734270 [45007960] -> SUBNET UP
> Mar 19 1833:37 650358 [41401960] -> __osm_trap_cv_process_request
> Rceived Geneic Noice type:0x01 num:128 Producer:2 from LID:0x0152
> TID0x0000000000000000
> Mar 19 18:33:37 65058 [41401960] -> os_report_notice Reporting Generic
> Noticetype:1 num:28 from LID:0x0152GID:0xfe8000000000000,0x005ad0000027c84
> Mar 19 18:33:37 927263 [45A08960] -> __osm_rap_rcv_procs_request:
> Received Generic otice tye:0x01 num:128 Producer:2 fro LID:0x0152
> TID:0x000000000000001
> Mar 19 18:33:37 927420 [45A090] -> osm_report_notice: Reportig Geeric
> Notice type:1 num:128from LID:0x0152
> GD:0xfe80000000000000,0x0005ad0000027c84
> Mar 19 18:3:37 95572 [4A0896] -> __osm_trap_rcv_process_rquest:
> Received Generic Notice type001 num:128Producer:2 from LID:x0152
> TID:0x000000000000002
> Mar 1918:33:37 955657 [45A08960] -> osreprt_notice: Reporting Generic
> Noticetype:1num:128 from LD:0x0152
> GID:0xfe80000000000000,0x0005ad000002c84
> Mar 1 18:33:37 97718 [44606960] -> _osm_tap_rcv_process_request:
> Receivd Generic Notice type:0x01 nu:128 Produr:2 from LID:0x0152
> TID:00000000000000003
> Mar 19 18:33:3 97740 [44606960] -> osm_report_notice: Rporing Geneic
> Notice type:1 num:128 frm LID:0x0152
> GID:0xfe800000000000,0x0005d0000027c84
> Mar 19 18:3337 999319 [41E02960] -> __osm_trap_rc_process_rquest:
> Received Gneri Notice type:0x01 num:128 Producer:2 rom LID:0x052
> TID:0x000000000000004
> Mar 19 18:33:37 999447 [41E02960] > sm_report_notice: Reporting Generic
> otice type:1 num:128 from LID:x152GID:xfe800000000000000x000ad0000027c84
> Mar 19 18:33:38 045171 [4606960] -> __osm_trap_rcvprcess_request:
> Received Gneric Notice type:0x0 num:128 Producer:2 frm LID:0x0152
> TID:0x000000000000005
> Mar 9 183:38 045271 [44606960] -> osm_report_notice: Reporting Generic
> Ntice ype:1 num:18 from LID:0x05
> GID:0xfe800000000000000x0005ad000027c84
> Mar 19 18:33:38 06305 [432060] -> __osm_trap_rcv_process_request:
> Received Generic Notice typ:0x01 nu:128 Producer:2 from ID:0x052
> TID:0x000000000000006
> Mar 1918:3:3 063102 [43204960] -> osm_reprt_notice: Rporting Generic
> Notice type:1 num:128 from LID:0x0152ID:0xfe8000000000000,0x0005a0000027c4
> ar 9 18:3338 182624 [2803960] -> __osm_trap_rcv_process_request:
> Received Generic Notice type:0x01num:12 Produer2from LID:0x0152
> TID:0x000000000000007
> Mar 19 18:3338 18720 [4280360 -> osm_report_notice: Reporting Generic
> Notice type:1 num:128 from LID:0x05
> GID:xfe800000000000,0x0005ad000007c84
> Mr 19 18:33:38 19435 [44606960] -> __osm_trap_rcv_process_request:
> Reeived Generic Notice tpe:0x0 num:128 Prducer:2 from LID:0x012
> TID:x0000000000000008
> Mar 1918:33:38 194209[44606960] -> osm_reportnotice: Reporting Generic
> Notic type:1 num:28 from LID:0x152
> GID:0xfe000000000000,0x0005ad000007c4
> Mar 1 18:33:38 379421 [43C05960] -> _om_trap_rcv_process_request:
> Received Generi Notice type:0x01 num:12Producr:2 from LID:0x0152
> TID:0x000000000000009
> Mar 19 18:33:38 37959 [4305960] -> osm_report_otice: Reporting eneric
> Ntice type:1 num:128 from LID:0x0152
> GD:0xfe800000000000,0x0005ad0000027c84
> Mar 19 1833:3 07685 [41401960] -> __osm_trap_rcv_rocss_request:Received GenericNotice type:0x01 num:128 Producer:2from LID:0x0152
> TID:0x0000000000000a
> Mar 19 18:33:38 47758 [4140190] -> osm_report_notice: eprting Generic
> Notice type:1 num:128 rom LID:0x0152
> GID:0xfe8000000000000,0000ad0000027c84
> Mar 1 18:33:8 429658 [4A08960] -> __osm_trap_rcv_pocess_request:
> Received Generic Ntice type:001 num:128 Producer:2 fm LID0x0152
> TID:0x000000000000000bMa 19 8:33:8 429700 [45A08960] -> __osm_trap_rcv_process_reqest: ERR
> 3804: Received trap 11 ties oecutiveyMar 19 18:33:38 544177 [45007960] - __osm_trap_rcv_process_reuest:
> Received Generic Notice type:0x0 num128 Prodcer:2 from LID:0x152
> ID:0x000000000000000c
> Mar 9 18:3338 544221 [4507960] -> __osm_trp_rv_process_request: ERR
> 304: Received trap 12 times consecutiely
> Mar 1918:33:8 545235 [4280960] ->osm_repot_ntice:Reporting Generic
> Notice type:3 num:65 from LID:0x0092
> GID:0xfe800000000000,0x0005ad0000024b
> Mar 19 18:3338 54247 [42803960] -> Removed port with
> GUID:0x0005ad00024b27 LID range 0xAF,0xAF] of node:sauaro-23-0 HCA-1
> Mar 19 18:33:3 545278 [42803960] -> osmreort_notice: Reporing Generic
> Noticetype3 num:65 from LD:0x0092
> GI:0xfe80000000000000,0x0005ad0000024bbb
> Mar 19 18:33:38 54586 [428030] -> Removd port with
> GUI:0x0005ad000024da7 LID range [0xB0,0x0] of node:saguao-23-1 HC-1
> Mar 19 18:33:38 545312 42803960] ->osm_report_noice: Reporting Generic
> Notice ype: num:65 from LID:0x0092
> GID:0xfe800000000000,0x0005ad0000024bb
> Mar 19 18:33:38 545318 [42803960] -> Reoved portwth
> GUID:0x0005ad00000249d3 LID rang [xB1,0B1] of node:saguaro-23-2 HA-1
> Mar 19 8:3:38 580005 [42803960 -> osm_ucast_mgr_process: Min Hop Tabes
> configured on all swiches
> Mar 19 18:33:38 66849[43C0590] -> SUBNET UP
> Mar 19 18:33:38 648520[45A08960] -> __om_tra_rcv_process_reques:
> ReceivedGeneric Notice type:0x01 num:128 Producer:2 from LID:0x015
> D:0x00000000000000d
> Mar 19 18:33:38 48616 [45A08960] -> __osm_trap_cv_process_requet: ERR
> 3804: eceived trap 13 tmes consecutiely
> Mar 19 18:3338 676891[41E02960 -> __osm_trap_rcv_rocess_request:
> Recived Genei Notice type:0x01 num:128 Producer:2 fo LID:0x0152
> TID:0x0000000000000e
> Mar 19 18:33:38 67670 [4102960] -> __osm_trap_rcv_proces_request: ERR
> 3804: Rceived trap 14 tes cosecutively
> Mar 19 18:33:38 698797 [4460960] ->__osm_trap_rcv_prcessrequest:
> Received Geneic Notice type:0x1 num:128 Producer:2frm LID:0x0152
> TD:0x00000000000000f
> Mr 19 18:33:8 69860 [44606960] -> __osm_trap_rcv_process_equest: ERR
> 3804: Receved trap 15 times consecutivey
> Mar 19 18:33:38 20538 [43C05960] -> __s_trap_rcv_proces_request:
> Received Generic Notce type:0x01 num:128 Poducer:2from ID:0x0152
> TID:0x0000000000000010Mar 19 18:33:38 720612 [43C0960] -> __osm_trp_rcv_process_reqest: RR
> 3804: eceived trap 16 times onsecutively
> ar 19 18:33:38 921253 [42803960] > __osm_trap_rcv_process_equest:
> eceived Generic Notice type:x01 num:128 Producer:2 from LID0x012
> TIDx0000000000000011
> Mar 19 18:33:8 92130 [42803960] -> __osm_trap_rcv_procss_reques: ERR
> 3804: Recived trap 17 imes consecutively
> Mar 198:33:38 97418 [43C05960] -> __osm_trap_rcv_proess_reqest:
> Received Generic Notice ype:0x01 num:128 Producer:2 rom LID:0x152
> TID:0x0000000000000012
> a19 18:33:38 967479 [43C05960] > __os_trap_rcv_process_request: RR
> 3804: Received trap 18 times onsecutively
> Mar 19 18:33:38 98519 [483960] -> _osm_trap_rcv_processreques:
> ReceivedGeneric Notice type:0x01 num:128 Producer:2 from LID:0x015
> TID:0x000000000000013
> Mar 19 18:33:3 98955[2803960] -> __osm_trap_rcv_process_rquest: ERR3804: Receivedtrap 19 times consecutively
> Mar 9 18:33:38 998342 [43204960] -> __osm_trap_rcv_poces_request:
> ecived Generic Notice type0x01 num:128 Poducer:2 from LID:0x0152
> TD:0x000000000000014
> Mar 19 18:33:38 998380 [4320496] -> __osm_ap_rcv_process_request ERR
> 384: Received trap 20 times consecutively
> ar 19 18:33:3 039293 43204960] -> __osm_trap_rcv_process_request:
> Recived Gneric Notice type:0x0 num:128 Producr:2 frm LID:0x0152
> TID:x0000000000000015
> Mar9 18:33:39 039334 [43204960] -> __os_trap_rcv_process_requs: ERR
> 3804:Received trap 21 times consecutively
> Mar19 18:33:39 061060 [3204960] -> __osm_trap_rcv_process_request:
> Receid Generic Notice type:01 num:128 Producer:2 from LID:0x0152TID:0x000000000000016
> Mar 19 18:3:3906108 [43204960] -> __osm_trap_rcv_prcess_request: ERR
> 3804: Reeied tra 22 times consecutivel
> Mar 19 18:33:39 079032 [41E02960] -> __osm_tra_rcv_process_request:
> Received eneric Notice type:0x01 num:128 Prducer2 from LID:0x0152
> TID:0x000000000000017
> Mar 19 18:33:39 07972 [4E02960] -> _osm_trap_rcv_proces_request: ERR
> 3804: Receied trap23 times consecutively
> Mar 19 18:3:9 146006 [41E0960] -> osm_report_notice: Reporting Gneric
> Notice ype:3 num:65 from LID:0x0092
> GD:0xfe8000000000000,0x0005ad0000024bbMar 19 18:33:39 146018 [4E02960] - Removed portwith
> GUID:0x005ad000002511b LID range [0xA6,xA6] of node:saguaro-2-1 HCA-1
> Mar 19 18:33:39 14604 [41E02960] -> osm_eport_notice: Reporting Generic
> Noticetype:3 num:65 from LID:0x0092
> GID:0xfe80000000000000,0x005ad0000024bb
> Mar 19 18:33:39 146050 [41E02960] -> Rmoved port with
> UID:0x0005d0000024db LID range [0xB80xB8] of node:saguaro-23-9 HCA-1
> Mar 19 18:33:39 146082 [41E2960] -> osm_report_notice Reporting Generic
> Notic type:3 num:65 from LID:0x0092
> GID:0xfe80000000000000,0x0005ad000024bb
> Mar 19 18:33:39 146089 [41E02960] -> Removed port wth
> GUID:00005ad0000024afb LID range 0xA,0xA5] of node:saguaro-22-0 HCA-1
> Mar 19 18:33:39 150720 [4140190 -> osm_report_notice: Reporting Gneric
> Notie type:3 num:64 from LID:0x0092
> GID:0xfe80000000000000,0x0005ad00000bbb
> Mar 19 18:33:39 150732 [41401960] -> Discovered new port with
> GUI0x0005ad0000024b27 LID rage [0xAF,0xAF] of node:saguaro-23-0 HCA-1
> Mar 19 18:33:39 150736 [4140160] -> osm_report_notice: Reporting Geneic
> Notice ype:3 num:64 from LID:0x0092
> GID:0xfe0000000000000,0x0005d000024bbb
> Mar 19 18:33:39 150742 [41401960] -> Discovered new port with
> GID:0x000ad0000024da LID range [0xB0,0xB0] of node:saguaro-23-1 HCA-1
> Mar 19 18:33:39 150745 [4141960] -> osm_report_notice: Reporting Geneic
> Notice tpe:3num:64 from LID:0x0092
> GID:0xfe8000000000000,0x0005ad000024bbb
> Mar 19 18:3:39 150750 [41401960] -> Discovered new port with
> UID:0x0005ad0000249d3 LID range [0xB1,0xB1] of node:saguaro-23-2 HA-1
> Mar 19 18:33:39 181553 [4141960] -> osm_ucast_mgr_process: Min Ho Tables
> configured on al switches
> Mar 19 18:33:39 218130 [43C5960] -> __osm_trap_rcv_process_request:
> Received eneric Notice type:0x01 num:128 Producer:2 from LID:00152
> TID:0x000000000000018
> Mar 19 18:33:39 218197 [43C05960] -> __osm_trap_rcv_process_request: ERR
> 3804: Receivd trap 24 times consecutively
> Mar 1918:33:39 375407 [42803960] -> __osm_trap_rcv_process_request:
> Received Generc Notice type:0x01 um:128 Producer:2 from LID:0x0152
> TID:0x000000000000019
> Mar 19 18:3339 375456 [42803960] -> __osm_trap_rcv_process_request: ERR
> 3804: Received tra 25 times consecutvely
> Mar 19 18:33:39 375588 [43C05960]-> __osm_trap_rcv_proces_request:
> Received Generic Notic type:0x01 num:128 Producer:2 from LID:0x0148
> TID:0x00000000000002e
> Mr 19 18:33:39 375630 [43C05960] -> osm_report_notice Reporting Generic
> Notice type:1 num:128 fro LID:0x0148
> GID:0xfe80000000000000,0x0005ad00000281b
> Mar 19 18:33:39 637844 [41401960] -> UBNET UP
> Mar 19 18:33:9 664805 [45A08960] -> __osm_trap_rcv_process_request:
> Received Generc Notce type:0x01 num128 Producer:2 from LID:0x0148
> TID:0x000000000000002f
> Mar 19 18:33:39 66490 [45A08960] -> osm_report_notice: Reporting Generic
> Notice type:1 num:128 from LID:0x0148
> GID:xfe800000000000000x0005ad00000281b3
> ar 19 18:33:39 666276 [45A08960] -> __osm_trap_rcvprocess_request:
> Received Generic Notice type:x01 num:128 Producer:2 from LID:0x001B
> TID:0x00000000000003a
> Mar 19 18:33:39 666364 [45A08960] -> osm_report_notice Reporting Generic
> Notice type1 num:128 from LID:0x001B
> GID:0xfe8000000000000,0x0005ad00000281a7Mar 19 18:33:39 710546 [41E02960] -> __osm_trap_rcv_proces_request
> Received Generic Notice type:0x01 num:128 Producer2 from LID:0x014
> TID:0x000000000000003
> Mar 19 18:33:39 710642 [41E02960] -> osm_report_notice Reporting Generic
> Notice type:1 num:28 from LID:0x0148
> ID:0xfe80000000000000,0x0005ad00000281b3
> Mar 19 18:33:39 732425 [41E0960] ->__osm_trap_rcvprocess_request:
> Received Generic Notice type:0x01 num:128 Producer:2 from ID:0x0148
> TID:0x0000000000000031
> Mar 19 18:33:39732514 41E02960] -> osm_report_notice: Reporing Generic
> Notice type:1num:128 from LID:0x0148
> GID:0xfe8000000000000,0x0005ad00000281b3
> Mar 1 18:33:39 784151 [43204960] -> __osm_trap_rcv_process_request:
> ReceivedGeneric Notice type:0x01 num:128 Producer:2 from LID:0x0148
> TID:000000000000032
> Mar 19 18:33:39 784269 [43204960] -> osm_report_notice: Reporting neric
> Notice type:1 nu:128 from LID:x0148
> GID:0xfe80000000000000,0x0005ad00000281b3
> Mar 19 18:33:39 824170 [42803960] -> __osm_trap_rcv_procss_request:
> Received Generic Notice type:0x01 num:128 Produer:2 from LID:0x001B
> TID:0x00000000000003b
> Mar 19 18:33:39 824443 [42803960] -> osm_repot_notice: Reporting Generic
> Notice tye:1 num:128 frm LID:0x001B
> GID:0xfe80000000000000,0x0005ad00000281a7
> Mar 19 18:33:40 015052 [44606960] - osm_report_notice: Reporting Generic
> Notice type:3num:64 from ID:0x0092
> GID:0xfe80000000000000,0x0005ad0000024bbb
> Mar 9 18:33:40 015070[44606960] - Discovered new port with
> GUID:0x0005ad0000024d6b LID range [0xB80xB8] of node:saguaro-23-9 HCA-1
> Mar 19 18:33:40 015074 [44606960] -> osm_repot_notice: Reporting Generic
> Notice type3 num:64 from LID:0x0092
> GID:0fe80000000000000,0x0005ad0000024bbb
> Mar 19 18:33:40 015080 [44606960] -> Discovered new port wit
> GUID:0x0005ad000024afb LID range [0xA5,0xA5] of node:saguaro-22-0 HCA-1
> Mar 19 18:3:40 015083 [44606960] -> osm_repor_notice: Reporting Generic
> Notice type:3 num:64 from LID:0x002
> GID:0xfe80000000000000,0x0005ad000002bbb
> Mar 19 18:33:40 015088 [44606960] -> Discovered new port with
> GUID:00005ad000002511b LID range [0xA6,0xA6] of node:saguaro-22-1 HCA-1
> Mar19 18:33:40 046164 [44606960] -> osm_ucast_mgr_process: Min Hop Tables
> configured on all switchesMar 19 18:33:40 106627 [42803960] -> SBNET UP
> Mar 19 18:33:40 145952 [45007960] -> __osm_trap_rcv_process_request:
> Received Generic Notice type:0x01 um:128 Producer:2 from LID:0x0148
> TID:0x0000000000000033
> Mar 19 18:3340 146076 [45007960] -> osm_report_notice: Reporting Generic
> Notice type:1 num:128 from LID:0x018
> GID:0xfe80000000000000,0x0005ad00000281b3
> Mar 19 18:33:40 146486 [44606960] -> __osm_trap_rcv_process_request:Received Generic Notice ype:0x01 num:128 Producer:2 from LID:0x001B
> ID:0x000000000000003c
> Mar 19 18:33:40 146611 [44606960] -> osm_report_notice: Reporting Generic
> otice type:1 num:128 from LID:0x001B
> GID:0xfe80000000000000,0x0005ad00000281a7
> Mar 19 18:33:40 306176 [41401960]-> __osm_trap_rcv_process_request:
> Received Generic Notice type:0x01 um:128 Producer:2 from LID:0x001B
> TID:0x000000000000003d
> Mar 19 18:33:40 306270 [41401960] -> os_report_notice: Reporting Generic
> Notice type:1 num:128 from LID:0x001B
> GID:0xfe80000000000000,0x0005ad00000281a
> Mar 19 18:33:40 420009 [43C05960] -> __osm_trap_rcv_process_rquest:Received Generic Notice typ:0x01 num:128 Producer:2 from LID:0x0152
> TID:0x0000000000000019
> Mar 918:33:4420071 [43C05960] -> __osm_trap_rcv_process_request: ERR
> 3804: Received trap 26 times conecutively
> r 19 18:33:40 433566 [4280390] -> __osm_trap_rcv_process_request:
> Reeived Geneic Notice type:0x01 num:128 Producer:2 from LID:0x0152
> TID:0x00000000000001a
> Mar 19 1:33:40 433596 [42803960] -> __osm_traprcv_proess_request: ERR
> 3804: Received trap 27 times consecutively
> Mar 19 18:33:40 434996 [45007960] -> _osm_trap_cv_process_request:
> Received Generic otice type:0x01 num:128 Producer:2 from LID:0x001BTID:0x000000000000003e
> Mar 19 18:33:40 435041 [4500790] -> osm_reportotice: Reporting Generic
> Notice ype:1 num:128 from LID:0x001B
> GID:0xfe80000000000000,0x0005ad0000281a7
> Mar 19 18:33:40 485454 [3204960 -> osm_ucast_mgr_procss: Mi Hop Tables
> configured on all switches
> Mar 19 18:33:40 528816 [43C05960] -> __osm_trap_cv_process_requet:
> Received Generic Noice type:0x01 num:128 Producer:2 from LID:0x001B
> TID:0x000000000000003f
> Mar 19 18:33:40 528960 [43C05960] -> osm_reort_notie: Reporting Generic
> Notice type:1 nu:128 from LID:0x001B
> GID:0xfe80000000000000,0x0005ad00000281a7
> Mar 19 18:33:40 546019 [42803960] -> SUBNT UP
> Mar 19 18:33:40 551048 [42803960] -> __osm_trap_rcv_process_request:
> Received Generic Notice type:0x01 num:128 Producer:2 from LID:0x0148
> TID:x0000000000000034
> Mar 19 18:33:40 55119 [42803960] -> osm_report_notice: Reporting Generic
> Notice typ:1 num:128 from LID:0x0148
> GID:0xfe8000000000000,0x0005ad00000281b3
> Mar 19 18:33:40 594994 [44606960] -> __osm_trap_rcv_process_request:
> Received Generic Notice type:0x01 num:128 Producer2 from LID:0x001B
> TID:0x0000000000000040
> Mar 19 18:33:40 595074 [44606960] -> osm_report_notice: Reporting Generic
> Notice type:1 num:128 from LD:0x001B
> GID:0xfe80000000000000,0x0005ad00000281a7
> Mar 19 18:33:40 83973 [43204960] -> __osm_trap_rcv_process_request:
> Received Generic Notice type:0x01 num:128 Prodcer:2 from LID:0x001B
> TID:0x0000000000000041
> Mar 19 18:33:40 840064 [43204960] -> osm_report_notice: Reporting Gneric
> Notice type:1 num:128 from LID:0x001B
> GID:0xfe80000000000000,0x005ad0000281a7
> Mar 19 18:33:40 861973 [43204960] -> __osm_trap_rcv_process_request:
> Received Genric Notice type:0x01 num:128 Producer: from LID:0x001B
> TID:0x0000000000000042
> Mar 19 18:33:40 862075 [43204960]-> osm_report_notice: Reporting Generic
> Ntice type:1 num:128 from LID:0x001B
> GID:0xfe80000000000000,0x005ad00000281a7
> Mar 19 18:33:40 83777 [43204960] -> __osm_trap_rcv_process_request:
> Received Generic otice type:0x01 num:128 Producr:2 from LID:0x001B
> TID:0x0000000000000043
> Mar 19 18:33:40 907658 [4803960] -> osm_ucast_mgr_process: Min Hop Tables
> configured on all switches
> Mar 19 18:33:40 947974 [43204960] -> osm_report_notice: Reporting Generic
> Notice type:1 num:128 from LID:0x001B
> GID:0xfe80000000000000,0x0005ad00000281a7
> Mar 19 18:33:40 965203 [45007960] -> SUBNET UP
> Mar 19 18:33:41 350582 [45007960] -> osm_ucast_mgr_process: Min Hop Tables
> configured on all switches
> Mar 19 18:33:41 417662 [43204960] -> SUBNET UP
> Mar 19 18:33:41 571156 [45A08960] -> __osm_trap_rc_process_request:
> Received Generic Notice type:0x01 num:128 Producer:2 from LID:0x0152
> TID:0x000000000000001b
> Mar 19 18:33:41 571256 [45A08960] -> __osm_trap_rcv_process_request: ERR
> 3804: Received trap 28 times consecutively
> Mar 19 18:35:50 971684 [43C05960] -> __osm_trap_rcv_process_request:
> Received Generic Notice type:0x01 num:128 Producer:2 from LID:0x0148
> TID:0x0000000000000035
> Mar 19 18:35:50 971926 [43C05960] -> osm_report_notice: Reporting Generic
> Notice type:1 num:128 from LID:0x0148
> GID:0xfe80000000000000,0x0005ad00000281b3
> Mar 19 18:35:50 972301 [45007960] -> __osm_trap_rcv_process_request:
> Received Generic Notice type:0x01 num:128 Producer:2 from LID:0x001B
> TID:0x0000000000000044
> Mar 19 18:35:50 972378 [45007960] -> osm_report_notice: Reporting Generic
> Notice type:1 num:128 from LID:0x001B
> GID:0xfe80000000000000,0x0005ad00000281a7
> Mar 19 18:35:51 342826 [43204960] -> osm_report_notice: Reporting Generic
> Notice type:3 num:65 from LID:0x0092
> GID:0xfe80000000000000,0x0005ad0000024bbb
> Mar 19 18:35:51 342845 [43204960] -> Removed port with
> GUID:0x0005ad0000024e0b LID range [0xB3,0xB3] of node:saguaro-23-4 HCA-1
> Mar 19 18:35:51 342866 [43204960] -> osm_report_notice: Reporting Generic
> Notice type:3 num:65 from LID:0x0092
> GID:0xfe80000000000000,0x0005ad0000024bbb
> Mar 19 18:35:51 342873 [43204960] -> Removed port with
> GUID:0x0005ad000002510b LID range [0xB5,0xB5] of node:saguaro-23-6 HCA-1
> Mar 19 18:35:51 342895 [43204960] -> osm_report_notice: Reporting Generic
> Notice type:3 num:65 from LID:0x0092
> GID:0xfe80000000000000,0x0005ad0000024bbb
> Mar 19 18:35:51 342901 [43204960] -> Removed port with
> GUID:0x0005ad000002511b LID range [0xA6,0xA6] of node:saguaro-22-1 HCA-1
> Mar 19 18:35:51 342923 [43204960] -> osm_report_notice: Reporting Generic
> Notice type:3 num:65 from LID:0x0092
> GID:0xfe80000000000000,0x0005ad0000024bbb
> Mar 19 18:35:51 342930 [43204960] -> Removed port with
> GUID:0x0005ad0000024b27 LID range [0xAF,0xAF] of node:saguaro-23-0 HCA-1
> Mar 19 18:35:51 342968 [43204960] -> osm_report_notice: Reporting Generic
> Notice type:3 num:65 from LID:0x0092
> GID:0xfe80000000000000,0x0005ad0000024bbb
> Mar 19 18:35:51 342972 [43204960] -> Removed port with
> GUID:0x0005ad0000025043 LID range [0xB4,0xB4] of node:saguaro-23-5 HCA-1
> Mar 19 18:35:51 342989 [43204960] -> osm_report_notice: Reporting Generic
> Notice type:3 num:65 from LID:0x0092
> GID:0xfe80000000000000,0x0005ad0000024bbb
> Mar 19 18:35:51 342994 [43204960] -> Removed port with
> GUID:0x0005ad0000024d47 LID range [0xB6,0xB6] of node:saguaro-23-7 HCA-1
> Mar 19 18:35:51 343011 [43204960] -> osm_report_notice: Reporting Generic
> Notice type:3 num:65 from LID:0x0092
> GID:0xfe80000000000000,0x0005ad0000024bbb
> Mar 19 18:35:51 343016 [43204960] -> Removed port with
> GUID:0x0005ad0000024d6b LID range [0xB8,0xB8] of node:saguaro-23-9 HCA-1
> Mar 19 18:35:51 343033 [43204960] -> osm_report_notice: Reporting Generic
> Notice type:3 num:65 from LID:0x0092
> GID:0xfe80000000000000,0x0005ad0000024bbb
> Mar 19 18:35:51 343038 [43204960] -> Removed port with
> GUID:0x0005ad0000024977 LID range [0xA9,0xA9] of node:saguaro-22-4 HCA-1
> Mar 19 18:35:51 343189 [43204960] -> osm_report_notice: Reporting Generic
> Notice type:3 num:65 from LID:0x0092
> GID:0xfe80000000000000,0x0005ad0000024bbb
> Mar 19 18:35:51 343194 [43204960] -> Removed port with
> GUID:0x0005ad0000027c84 LID range [0x152,0x152] of node:Topspin Switch TS120
> Mar 19 18:35:51 343234 [43204960] -> osm_report_notice: Reporting Generic
> Notice type:3 num:65 from LID:0x0092
> GID:0xfe80000000000000,0x0005ad0000024bbb
> Mar 19 18:35:51 343239 [43204960] -> Removed port with
> GUID:0x0005ad0000024d8b LID range [0xB7,0xB7] of node:saguaro-23-8 HCA-1
> Mar 19 18:35:51 343253 [43204960] -> osm_report_notice: Reporting Generic
> Notice type:3 num:65 from LID:0x0092
> GID:0xfe80000000000000,0x0005ad0000024bbb
> Mar 19 18:35:51 343258 [43204960] -> Removed port with
> GUID:0x0005ad000002498f LID range [0xA8,0xA8] of node:saguaro-22-3 HCA-1
> Mar 19 18:35:51 343273 [43204960] -> osm_report_notice: Reporting Generic
> Notice type:3 num:65 from LID:0x0092
> GID:0xfe80000000000000,0x0005ad0000024bbb
> Mar 19 18:35:51 343278 [43204960] -> Removed port with
> GUID:0x0005ad0000024c9b LID range [0xA7,0xA7] of node:saguaro-22-2 HCA-1
> Mar 19 18:35:51 343293 [43204960] -> osm_report_notice: Reporting Generic
> Notice type:3 num:65 from LID:0x0092
> GID:0xfe80000000000000,0x0005ad0000024bbb
> Mar 19 18:35:51 343298 [43204960] -> Removed port with
> GUID:0x0005ad0000024da7 LID range [0xB0,0xB0] of node:saguaro-23-1 HCA-1
> Mar 19 18:35:51 343314 [43204960] -> osm_report_notice: Reporting Generic
> Notice type:3 num:65 from LID:0x0092
> GID:0xfe80000000000000,0x0005ad0000024bbb
> Mar 19 18:35:51 343319 [43204960] -> Removed port with
> GUID:0x0005ad0000024cbb LID range [0xB2,0xB2] of node:saguaro-23-3 HCA-1
> Mar 19 18:35:51 343334 [43204960] -> osm_report_notice: Reporting Generic
> Notice type:3 num:65 from LID:0x0092
> GID:0xfe80000000000000,0x0005ad0000024bbb
> Mar 19 18:35:51 343393 [43204960] -> Removed port with
> GUID:0x0005ad00000249d3 LID range [0xB1,0xB1] of node:saguaro-23-2 HCA-1
> Mar 19 18:35:51 343410 [43204960] -> osm_report_notice: Reporting Generic
> Notice type:3 num:65 from LID:0x0092
> GID:0xfe80000000000000,0x0005ad0000024bbb
> Mar 19 18:35:51 343415 [43204960] -> Removed port with
> GUID:0x0005ad0000024feb LID range [0x153,0x153] of node:saguaro-22-5 HCA-1
> Mar 19 18:35:51 343430 [43204960] -> osm_report_notice: Reporting Generic
> Notice type:3 num:65 from LID:0x0092
> GID:0xfe80000000000000,0x0005ad0000024bbb
> Mar 19 18:35:51 343435 [43204960] -> Removed port with
> GUID:0x0005ad0000024afb LID range [0xA5,0xA5] of node:saguaro-22-0 HCA-1
> Mar 19 18:35:51 376525 [43204960] -> osm_ucast_mgr_process: Min Hop Tables
> configured on all switches
> Mar 19 18:35:51 433087 [43204960] -> SUBNET UP
> Mar 19 18:35:51 849193 [44606960] -> osm_ucast_mgr_process: Min Hop Tables
> configured on all switches
> Mar 19 18:35:51 901399 [42803960] -> SUBNET UP
> Mar 19 18:36:44 359407 [42803960] -> __osm_trap_rcv_process_request:
> Received Generic Notice type:0x01 num:128 Producer:2 from LID:0x0148
> TID:0x0000000000000036
> Mar 19 18:36:44 359652 [42803960] -> osm_report_notice: Reporting Generic
> Notice type:1 num:128 from LID:0x0148
> GID:0xfe80000000000000,0x0005ad00000281b3
> Mar 19 18:36:44 365352 [42803960] -> __osm_trap_rcv_process_request:
> Received Generic Notice type:0x01 num:128 Producer:2 from LID:0x0148
> TID:0x0000000000000037
> Mar 19 18:36:44 365427 [42803960] -> osm_report_notice: Reporting Generic
> Notice type:1 num:128 from LID:0x0148
> GID:0xfe80000000000000,0x0005ad00000281b3
> Mar 19 18:36:44 365432 [43204960] -> __osm_trap_rcv_process_request:
> Received Generic Notice type:0x01 num:128 Producer:2 from LID:0x001B
> TID:0x0000000000000045
> Mar 19 18:36:44 365567 [43204960] -> osm_report_notice: Reporting Generic
> Notice type:1 num:128 from LID:0x001B
> GID:0xfe80000000000000,0x0005ad00000281a7
> Mar 19 18:36:44 371481 [44606960] -> __osm_trap_rcv_process_request:
> Received Generic Notice type:0x01 num:128 Producer:2 from LID:0x001B
> TID:0x0000000000000046
> Mar 19 18:36:44 371591 [44606960] -> osm_report_notice: Reporting Generic
> Notice type:1 num:128 from LID:0x001B
> GID:0xfe80000000000000,0x0005ad00000281a7
> Mar 19 18:36:44 711649 [43204960] -> osm_drop_mgr_process: ERR 0108: Unknown
> remote side for node 0x0005ad0000027c84 port 19. Adding to light sweep
> sampling list
> Mar 19 18:36:44 711691 [43204960] -> Directed Path Dump of 5 hop path:
> Path = [0][1][11][1][6][18]
> Mar 19 18:36:44 711738 [43204960] -> osm_drop_mgr_process: ERR 0108: Unknown
> remote side for node 0x0005ad00000281b3 port 24. Adding to light sweep
> sampling list
> Mar 19 18:36:44 711748 [43204960] -> Directed Path Dump of 4 hop path:
> Path = [0][1][11][1][6]
> Mar 19 18:36:44 721719 [43204960] -> osm_report_notice: Reporting Generic
> Notice type:3 num:64 from LID:0x0092
> GID:0xfe80000000000000,0x0005ad0000024bbb
> Mar 19 18:36:44 721730 [43204960] -> Discovered new port with
> GUID:0x0005ad0000027c84 LID range [0x152,0x152] of node:Topspin Switch TS120
> Mar 19 18:36:44 721736 [43204960] -> osm_report_notice: Reporting Generic
> Notice type:3 num:64 from LID:0x0092
> GID:0xfe80000000000000,0x0005ad0000024bbb
> Mar 19 18:36:44 721744 [43204960] -> Discovered new port with
> GUID:0x0005ad0000024da7 LID range [0xB0,0xB0] of node:saguaro-23-1 HCA-1
> Mar 19 18:36:44 721749 [43204960] -> osm_report_notice: Reporting Generic
> Notice type:3 num:64 from LID:0x0092
> GID:0xfe80000000000000,0x0005ad0000024bbb
> Mar 19 18:36:44 721756 [43204960] -> Discovered new port with
> GUID:0x0005ad00000249d3 LID range [0xB1,0xB1] of node:saguaro-23-2 HCA-1
> Mar 19 18:36:44 721761 [43204960] -> osm_report_notice: Reporting Generic
> Notice type:3 num:64 from LID:0x0092
> GID:0xfe80000000000000,0x0005ad0000024bbb
> Mar 19 18:36:44 721767 [43204960] -> Discovered new port with
> GUID:0x0005ad0000024b27 LID range [0xAF,0xAF] of node:saguaro-23-0 HCA-1
> Mar 19 18:36:44 721772 [43204960] -> osm_report_notice: Reporting Generic
> Notice type:3 num:64 from LID:0x0092
> GID:0xfe80000000000000,0x0005ad0000024bbb
> Mar 19 18:36:44 721779 [43204960] -> Discovered new port with
> GUID:0x0005ad0000024cbb LID range [0xB2,0xB2] of node:saguaro-23-3 HCA-1
> Mar 19 18:36:44 721784 [43204960] -> osm_report_notice: Reporting Generic
> Notice type:3 num:64 from LID:0x0092
> GID:0xfe80000000000000,0x0005ad0000024bbb
> Mar 19 18:36:44 721790 [43204960] -> Discovered new port with
> GUID:0x0005ad0000024e0b LID range [0xB3,0xB3] of node:saguaro-23-4 HCA-1
> Mar 19 18:36:44 721795 [43204960] -> osm_report_notice: Reporting Generic
> Notice type:3 num:64 from LID:0x0092
> GID:0xfe80000000000000,0x0005ad0000024bbb
> Mar 19 18:36:44 721802 [43204960] -> Discovered new port with
> GUID:0x0005ad000002510b LID range [0xB5,0xB5] of node:saguaro-23-6 HCA-1
> Mar 19 18:36:44 721826 [43204960] -> osm_report_notice: Reporting Generic
> Notice type:3 num:64 from LID:0x0092
> GID:0xfe80000000000000,0x0005ad0000024bbb
> Mar 19 18:36:44 721831 [43204960] -> Discovered new port with
> GUID:0x0005ad0000025043 LID range [0xB4,0xB4] of node:saguaro-23-5 HCA-1
> Mar 19 18:36:44 721845 [43204960] -> osm_report_notice: Reporting Generic
> Notice type:3 num:64 from LID:0x0092
> GID:0xfe80000000000000,0x0005ad0000024bbb
> Mar 19 18:36:44 721850 [43204960] -> Discovered new port with
> GUID:0x0005ad0000024d47 LID range [0xB6,0xB6] of node:saguaro-23-7 HCA-1
> Mar 19 18:36:44 721854 [43204960] -> osm_report_notice: Reporting Generic
> Notice type:3 num:64 from LID:0x0092
> GID:0xfe80000000000000,0x0005ad0000024bbb
> Mar 19 18:36:44 721859 [43204960] -> Discovered new port with
> GUID:0x0005ad0000024d8b LID range [0xB7,0xB7] of node:saguaro-23-8 HCA-1
> Mar 19 18:36:44 721862 [43204960] -> osm_report_notice: Reporting Generic
> Notice type:3 num:64 from LID:0x0092
> GID:0xfe80000000000000,0x0005ad0000024bbb
> Mar 19 18:36:44 721867 [43204960] -> Discovered new port with
> GUID:0x0005ad0000024d6b LID range [0xB8,0xB8] of node:saguaro-23-9 HCA-1
> Mar 19 18:36:44 721871 [43204960] -> osm_report_notice: Reporting Generic
> Notice type:3 num:64 from LID:0x0092
> GID:0xfe80000000000000,0x0005ad0000024bbb
> Mar 19 18:36:44 721876 [43204960] -> Discovered new port with
> GUID:0x0005ad0000024afb LID range [0xA5,0xA5] of node:saguaro-22-0 HCA-1
> Mar 19 18:36:44 721880 [43204960] -> osm_report_notice: Reporting Generic
> Notice type:3 num:64 from LID:0x0092
> GID:0xfe80000000000000,0x0005ad0000024bbb
> Mar 19 18:36:44 721884 [43204960] -> Discovered new port with
> GUID:0x0005ad000002511b LID range [0xA6,0xA6] of node:saguaro-22-1 HCA-1
> Mar 19 18:36:44 721888 [43204960] -> osm_report_notice: Reporting Generic
> Notice type:3 num:64 from LID:0x0092
> GID:0xfe80000000000000,0x0005ad0000024bbb
> Mar 19 18:36:44 721893 [43204960] -> Discovered new port with
> GUID:0x0005ad0000024c9b LID range [0xA7,0xA7] of node:saguaro-22-2 HCA-1
> Mar 19 18:36:44 721897 [43204960] -> osm_report_notice: Reporting Generic
> Notice type:3 num:64 from LID:0x0092
> GID:0xfe80000000000000,0x0005ad0000024bbb
> Mar 19 18:36:44 721923 [43204960] -> Discovered new port with
> GUID:0x0005ad000002498f LID range [0xA8,0xA8] of node:saguaro-22-3 HCA-1
> Mar 19 18:36:44 721927 [43204960] -> osm_report_notice: Reporting Generic
> Notice type:3 num:64 from LID:0x0092
> GID:0xfe80000000000000,0x0005ad0000024bbb
> Mar 19 18:36:44 721932 [43204960] -> Discovered new port with
> GUID:0x0005ad0000024977 LID range [0xA9,0xA9] of node:saguaro-22-4 HCA-1
> Mar 19 18:36:44 721936 [43204960] -> osm_report_notice: Reporting Generic
> Notice type:3 num:64 from LID:0x0092
> GID:0xfe80000000000000,0x0005ad0000024bbb
> Mar 19 18:36:44 721941 [43204960] -> Discovered new port with
> GUID:0x0005ad0000024feb LID range [0x153,0x153] of node:saguaro-22-5 HCA-1
> Mar 19 18:36:44 752683 [43204960] -> osm_ucast_mgr_process: Min Hop Tables
> configured on all switches
> Mar 19 18:36:44 820881 [43C05960] -> SUBNET UP
> Mar 19 18:36:45 198990 [44606960] -> osm_ucast_mgr_process: Min Hop Tables
> configured on all switches
> Mar 19 18:36:45 258878 [44606960] -> SUBNET UP
> Mar 19 18:37:00 446068 [45A08960] -> __osm_trap_rcv_process_request:
> Received Generic Notice type:0x01 num:128 Producer:2 from LID:0x0152
> TID:0x0000000000000000
> Mar 19 18:37:00 446346 [45A08960] -> osm_report_notice: Reporting Generic
> Notice type:1 num:128 from LID:0x0152
> GID:0xfe80000000000000,0x0005ad0000027c84
> Mar 19 18:37:00 564122 [41401960] -> __osm_trap_rcv_process_request:
> Received Generic Notice type:0x01 num:128 Producer:2 from LID:0x0152
> TID:0x0000000000000001
> Mar 19 18:37:00 564810 [41401960] -> osm_report_notice: Reporting Generic
> Notice type:1 num:128 from LID:0x0152
> GID:0xfe80000000000000,0x0005ad0000027c84
> Mar 19 18:37:00 589920 [45007960] -> __osm_trap_rcv_process_request:
> Received Generic Notice type:0x01 num:128 Producer:2 from LID:0x0152
> TID:0x0000000000000002
> Mar 19 18:37:00 590067 [45007960] -> osm_report_notice: Reporting Generic
> Notice type:1 num:128 from LID:0x0152
> GID:0xfe80000000000000,0x0005ad0000027c84
> Mar 19 18:37:00 611770 [41E02960] -> __osm_trap_rcv_process_request:
> Received Generic Notice type:0x01 num:128 Producer:2 from LID:0x0152
> TID:0x0000000000000003
> Mar 19 18:37:00 611916 [41E02960] -> osm_report_notice: Reporting Generic
> Notice type:1 num:128 from LID:0x0152
> GID:0xfe80000000000000,0x0005ad0000027c84
> Mar 19 18:37:00 800652 [42803960] -> __osm_trap_rcv_process_request:
> Received Generic Notice type:0x01 num:128 Producer:2 from LID:0x0152
> TID:0x0000000000000004
> Mar 19 18:37:00 817995 [45007960] -> osm_ucast_mgr_process: Min Hop Tables
> configured on all switches
> Mar 19 18:37:00 861575 [42803960] -> osm_report_notice: Reporting Generic
> Notice type:1 num:128 from LID:0x0152
> GID:0xfe80000000000000,0x0005ad0000027c84
> Mar 19 18:37:00 983908 [42803960] -> __osm_trap_rcv_process_request:
> Received Generic Notice type:0x01 num:128 Producer:2 from LID:0x0152
> TID:0x0000000000000005
> Mar 19 18:37:00 984004 [42803960] -> osm_report_notice: Reporting Generic
> Notice type:1 num:128 from LID:0x0152
> GID:0xfe80000000000000,0x0005ad0000027c84
> Mar 19 18:37:01 012195 [44606960] -> __osm_trap_rcv_process_request:
> Received Generic Notice type:0x01 num:128 Producer:2 from LID:0x0152
> TID:0x0000000000000006
> Mar 19 18:37:01 012283 [44606960] -> osm_report_notice: Reporting Generic
> Notice type:1 num:128 from LID:0x0152
> GID:0xfe80000000000000,0x0005ad0000027c84
> Mar 19 18:37:01 034177 [43204960] -> __osm_trap_rcv_process_request:
> Received Generic Notice type:0x01 num:128 Producer:2 from LID:0x0152
> TID:0x0000000000000007
> Mar 19 18:37:01 034272 [43204960] -> osm_report_notice: Reporting Generic
> Notice type:1 num:128 from LID:0x0152
> GID:0xfe80000000000000,0x0005ad0000027c84
> Mar 19 18:37:01 056001 [41401960] -> __osm_trap_rcv_process_request:
> Received Generic Notice type:0x01 num:128 Producer:2 from LID:0x0152
> TID:0x0000000000000008
> Mar 19 18:37:01 056068 [41401960] -> osm_report_notice: Reporting Generic
> Notice type:1 num:128 from LID:0x0152
> GID:0xfe80000000000000,0x0005ad0000027c84
> Mar 19 18:37:01 074341 [43204960] -> SUBNET UP
> Mar 19 18:37:01 252871 [43204960] -> __osm_trap_rcv_process_request:
> Received Generic Notice type:0x01 num:128 Producer:2 from LID:0x0152
> TID:0x0000000000000009
> Mar 19 18:37:01 253037 [43204960] -> osm_report_notice: Reporting Generic
> Notice type:1 num:128 from LID:0x0152
> GID:0xfe80000000000000,0x0005ad0000027c84
> Mar 19 18:37:01 303407 [44606960] -> __osm_trap_rcv_process_request:
> Received Generic Notice type:0x01 num:128 Producer:2 from LID:0x0152
> TID:0x000000000000000a
> Mar 19 18:37:01 303490 [44606960] -> osm_report_notice: Reporting Generic
> Notice type:1 num:128 from LID:0x0152
> GID:0xfe80000000000000,0x0005ad0000027c84
> Mar 19 18:37:01 325057 [41E02960] -> __osm_trap_rcv_process_request:
> Received Generic Notice type:0x01 num:128 Producer:2 from LID:0x0152
> TID:0x000000000000000b
> Mar 19 18:37:01 325160 [41E02960] -> __osm_trap_rcv_process_request: ERR
> 3804: Received trap 11 times consecutively
> Mar 19 18:37:01 334059 [43204960] -> __osm_trap_rcv_process_request:
> Received Generic Notice type:0x01 num:128 Producer:2 from LID:0x0152
> TID:0x000000000000000c
> Mar 19 18:37:01 334118 [43204960] -> __osm_trap_rcv_process_request: ERR
> 3804: Received trap 12 times consecutively
> Mar 19 18:37:01 474293 [45007960] -> osm_report_notice: Reporting Generic
> Notice type:3 num:65 from LID:0x0092
> GID:0xfe80000000000000,0x0005ad0000024bbb
> Mar 19 18:37:01 474317 [45007960] -> Removed port with
> GUID:0x0005ad000002510b LID range [0xB5,0xB5] of node:saguaro-23-6 HCA-1
> Mar 19 18:37:01 474341 [45007960] -> osm_report_notice: Reporting Generic
> Notice type:3 num:65 from LID:0x0092
> GID:0xfe80000000000000,0x0005ad0000024bbb
> Mar 19 18:37:01 474348 [45007960] -> Removed port with
> GUID:0x0005ad0000024d47 LID range [0xB6,0xB6] of node:saguaro-23-7 HCA-1
> Mar 19 18:37:01 474371 [45007960] -> osm_report_notice: Reporting Generic
> Notice type:3 num:65 from LID:0x0092
> GID:0xfe80000000000000,0x0005ad0000024bbb
> Mar 19 18:37:01 474378 [45007960] -> Removed port with
> GUID:0x0005ad0000024d8b LID range [0xB7,0xB7] of node:saguaro-23-8 HCA-1
> Mar 19 18:37:01 509205 [45007960] -> osm_ucast_mgr_process: Min Hop Tables
> configured on all switches
> Mar 19 18:37:01 557110 [45A08960] -> __osm_trap_rcv_process_request:
> Received Generic Notice type:0x01 num:128 Producer:2 from LID:0x0152
> TID:0x000000000000000d
> Mar 19 18:37:01 557172 [45A08960] -> __osm_trap_rcv_process_request: ERR
> 3804: Received trap 13 times consecutively
> Mar 19 18:37:01 565676 [43C05960] -> SUBNET UP
> Mar 19 18:37:01 576199 [41401960] -> __osm_trap_rcv_process_request:
> Received Generic Notice type:0x01 num:128 Producer:2 from LID:0x0152
> TID:0x000000000000000e
> Mar 19 18:37:01 576270 [41401960] -> __osm_trap_rcv_process_request: ERR
> 3804: Received trap 14 times consecutively
> Mar 19 18:37:01 599713 [41E02960] -> __osm_trap_rcv_process_request:
> Received Generic Notice type:0x01 num:128 Producer:2 from LID:0x0152
> TID:0x000000000000000f
> Mar 19 18:37:01 599779 [41E02960] -> __osm_trap_rcv_process_request: ERR
> 3804: Received trap 15 times consecutively
> Mar 19 18:37:01 707096 [45007960] -> __osm_trap_rcv_process_request:
> Received Generic Notice type:0x01 num:128 Producer:2 from LID:0x0152
> TID:0x0000000000000010
> Mar 19 18:37:01 707150 [45007960] -> __osm_trap_rcv_process_request: ERR
> 3804: Received trap 16 times consecutively
> Mar 19 18:37:01 921406 [45A08960] -> osm_report_notice: Reporting Generic
> Notice type:3 num:65 from LID:0x0092
> GID:0xfe80000000000000,0x0005ad0000024bbb
> Mar 19 18:37:01 921423 [45A08960] -> Removed port with
> GUID:0x0005ad000002511b LID range [0xA6,0xA6] of node:saguaro-22-1 HCA-1
> Mar 19 18:37:01 921448 [45A08960] -> osm_report_notice: Reporting Generic
> Notice type:3 num:65 from LID:0x0092
> GID:0xfe80000000000000,0x0005ad0000024bbb
> Mar 19 18:37:01 921455 [45A08960] -> Removed port with
> GUID:0x0005ad0000024d6b LID range [0xB8,0xB8] of node:saguaro-23-9 HCA-1
> Mar 19 18:37:01 921495 [45A08960] -> osm_report_notice: Reporting Generic
> Notice type:3 num:65 from LID:0x0092
> GID:0xfe80000000000000,0x0005ad0000024bbb
> Mar 19 18:37:01 921502 [45A08960] -> Removed port with
> GUID:0x0005ad0000024afb LID range [0xA5,0xA5] of node:saguaro-22-0 HCA-1
> Mar 19 18:37:01 925845 [41E02960] -> osm_report_notice: Reporting Generic
> Notice type:3 num:64 from LID:0x0092
> GID:0xfe80000000000000,0x0005ad0000024bbb
> Mar 19 18:37:01 925855 [41E02960] -> Discovered new port with
> GUID:0x0005ad000002510b LID range [0xB5,0xB5] of node:saguaro-23-6 HCA-1
> Mar 19 18:37:01 925859 [41E02960] -> osm_report_notice: Reporting Generic
> Notice type:3 num:64 from LID:0x0092
> GID:0xfe80000000000000,0x0005ad0000024bbb
> Mar 19 18:37:01 925864 [41E02960] -> Discovered new port with
> GUID:0x0005ad0000024d47 LID range [0xB6,0xB6] of node:saguaro-23-7 HCA-1
> Mar 19 18:37:01 925868 [41E02960] -> osm_report_notice: Reporting Generic
> Notice type:3 num:64 from LID:0x0092
> GID:0xfe80000000000000,0x0005ad0000024bbb
> Mar 19 18:37:01 925873 [41E02960] -> Discovered new port with
> GUID:0x0005ad0000024d8b LID range [0xB7,0xB7] of node:saguaro-23-8 HCA-1
> Mar 19 18:37:01 956691 [41E02960] -> osm_ucast_mgr_process: Min Hop Tables
> configured on all switches
> Mar 19 18:37:01 999372 [43204960] -> __osm_trap_rcv_process_request:
> Received Generic Notice type:0x01 num:128 Producer:2 from LID:0x0152
> TID:0x0000000000000011
> Mar 19 18:37:01 999470 [43204960] -> __osm_trap_rcv_process_request: ERR
> 3804: Received trap 17 times consecutively
> Mar 19 18:37:02 012194 [41E02960] -> __osm_trap_rcv_process_request:
> Received Generic Notice type:0x01 num:128 Producer:2 from LID:0x0152
> TID:0x0000000000000012
> Mar 19 18:37:02 012256 [41E02960] -> __osm_trap_rcv_process_request: ERR
> 3804: Received trap 18 times consecutively
> Mar 19 18:37:02 014327 [41401960] -> SUBNET UP
> Mar 19 18:37:02 034202 [44606960] -> __osm_trap_rcv_process_request:
> Received Generic Notice type:0x01 num:128 Producer:2 from LID:0x0152
> TID:0x0000000000000013
> Mar 19 18:37:02 034250 [44606960] -> __osm_trap_rcv_process_request: ERR
> 3804: Received trap 19 times consecutively
> Mar 19 18:37:02 056015 [45A08960] -> __osm_trap_rcv_process_request:
> Received Generic Notice type:0x01 num:128 Producer:2 from LID:0x0152
> TID:0x0000000000000014
> Mar 19 18:37:02 056060 [45A08960] -> __osm_trap_rcv_process_request: ERR
> 3804: Received trap 20 times consecutively
> Mar 19 18:37:02 270731 [43204960] -> __osm_trap_rcv_process_request:
> Received Generic Notice type:0x01 num:128 Producer:2 from LID:0x0152
> TID:0x0000000000000015
> Mar 19 18:37:02 270777 [43204960] -> __osm_trap_rcv_process_request: ERR
> 3804: Received trap 21 times consecutively
> Mar 19 18:37:02 271169 [43C05960] -> __osm_trap_rcv_process_request:
> Received Generic Notice type:0x01 num:128 Producer:2 from LID:0x0148
> TID:0x0000000000000038
> Mar 19 18:37:02 271347 [43C05960] -> osm_report_notice: Reporting Generic
> Notice type:1 num:128 from LID:0x0148
> GID:0xfe80000000000000,0x0005ad00000281b3
> Mar 19 18:37:02 462374 [41E02960] -> __osm_trap_rcv_process_request:
> Received Generic Notice type:0x01 num:128 Producer:2 from LID:0x0148
> TID:0x0000000000000039
> Mar 19 18:37:02 462511 [41E02960] -> osm_report_notice: Reporting Generic
> Notice type:1 num:128 from LID:0x0148
> GID:0xfe80000000000000,0x0005ad00000281b3
> Mar 19 18:37:02 496247 [45007960] -> __osm_trap_rcv_process_request:
> Received Generic Notice type:0x01 num:128 Producer:2 from LID:0x0148
> TID:0x000000000000003a
> Mar 19 18:37:02 496310 [45007960] -> osm_report_notice: Reporting Generic
> Notice type:1 num:128 from LID:0x0148
> GID:0xfe80000000000000,0x0005ad00000281b3
> Mar 19 18:37:02 624890 [45A08960] -> osm_report_notice: Reporting Generic
> Notice type:3 num:64 from LID:0x0092
> GID:0xfe80000000000000,0x0005ad0000024bbb
> Mar 19 18:37:02 624902 [45A08960] -> Discovered new port with
> GUID:0x0005ad0000024d6b LID range [0xB8,0xB8] of node:saguaro-23-9 HCA-1
> Mar 19 18:37:02 624908 [45A08960] -> osm_report_notice: Reporting Generic
> Notice type:3 num:64 from LID:0x0092
> GID:0xfe80000000000000,0x0005ad0000024bbb
> Mar 19 18:37:02 624914 [45A08960] -> Discovered new port with
> GUID:0x0005ad0000024afb LID range [0xA5,0xA5] of node:saguaro-22-0 HCA-1
> Mar 19 18:37:02 624919 [45A08960] -> osm_report_notice: Reporting Generic
> Notice type:3 num:64 from LID:0x0092
> GID:0xfe80000000000000,0x0005ad0000024bbb
> Mar 19 18:37:02 624926 [45A08960] -> Discovered new port with
> GUID:0x0005ad000002511b LID range [0xA6,0xA6] of node:saguaro-22-1 HCA-1
> Mar 19 18:37:02 655848 [45A08960] -> osm_ucast_mgr_process: Min Hop Tables
> configured on all switches
> Mar 19 18:37:02 709115 [42803960] -> SUBNET UP
> Mar 19 18:37:03 082995 [44606960] -> __osm_trap_rcv_process_request:
> Received Generic Notice type:0x01 num:128 Producer:2 from LID:0x0148
> TID:0x000000000000003b
> Mar 19 18:37:03 106373 [43204960] -> osm_ucast_mgr_process: Min Hop Tables
> configured on all switches
> Mar 19 18:37:03 136757 [44606960] -> osm_report_notice: Reporting Generic
> Notice type:1 num:128 from LID:0x0148
> GID:0xfe80000000000000,0x0005ad00000281b3
> Mar 19 18:37:03 178027 [41401960] -> __osm_trap_rcv_process_request:
> Received Generic Notice type:0x01 num:128 Producer:2 from LID:0x001B
> TID:0x0000000000000047
> Mar 19 18:37:03 178064 [43C05960] -> __osm_trap_rcv_process_request:
> Received Generic Notice type:0x01 num:128 Producer:2 from LID:0x0148
> TID:0x000000000000003c
> Mar 19 18:37:03 178139 [41401960] -> osm_report_notice: Reporting Generic
> Notice type:1 num:128 from LID:0x001B
> GID:0xfe80000000000000,0x0005ad00000281a7
> Mar 19 18:37:03 178160 [43C05960] -> osm_report_notice: Reporting Generic
> Notice type:1 num:128 from LID:0x0148
> GID:0xfe80000000000000,0x0005ad00000281b3
> Mar 19 18:37:03 315226 [41E02960] -> __osm_trap_rcv_process_request:
> Received Generic Notice type:0x01 num:128 Producer:2 from LID:0x0152
> TID:0x0000000000000015
> Mar 19 18:37:03 315289 [41E02960] -> __osm_trap_rcv_process_request: ERR
> 3804: Received trap 22 times consecutively
> Mar 19 18:37:03 341474 [41E02960] -> __osm_trap_rcv_process_request:
> Received Generic Notice type:0x01 num:128 Producer:2 from LID:0x0152
> TID:0x0000000000000016
> Mar 19 18:37:03 341549 [41E02960] -> __osm_trap_rcv_process_request: ERR
> 3804: Received trap 23 times consecutively
> Mar 19 18:37:03 341616 [41E02960] -> __osm_trap_rcv_process_request:
> Received Generic Notice type:0x01 num:128 Producer:2 from LID:0x0148
> TID:0x000000000000003d
> Mar 19 18:37:03 342446 [41E02960] -> osm_report_notice: Reporting Generic
> Notice type:1 num:128 from LID:0x0148
> GID:0xfe80000000000000,0x0005ad00000281b3
> Mar 19 18:37:03 343169 [4780B960] -> __osm_sm_mad_ctrl_rcv_callback: ERR
> 3111: Error status = 0x1C00
> Mar 19 18:37:03 343262 [4780B960] -> SMP dump:
> base_ver................0x1
> mgmt_class..............0x81
> class_ver...............0x1
> method..................0x81 (SubnGetResp)
> D bit...................0x1
> status..................0x1C00
> hop_ptr.................0x0
> hop_count...............0x5
> trans_id................0x14d08
> attr_id.................0x15 (PortInfo)
> resv....................0x0
> attr_mod................0x11
> m_key...................0x0000000000000000
> dr_slid.................0xFFFF
> dr_dlid.................0xFFFF
>
> Initial path: [0][1][11][1][6][16]
> Return path: [0][9][18][D][3][11]
> Reserved: [0][0][0][0][0][0][0]
>
> 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
>
> 00 00 00 00 00 00 00 00 00 00 00 00 11 02 03 02
>
> 12 52 00 11 40 40 00 08 08 04 F2 40 00 00 00 00
>
> 00 00 00 00 00 88 00 00 00 00 00 00 00 00 00 00
>
> Mar 19 18:37:03 343371 [4780B960] -> __osm_sm_mad_ctrl_rcv_callback: ERR
> 3111: Error status = 0x1C00
> Mar 19 18:37:03 343364 [45007960] -> osm_pi_rcv_process_set: Received error
> status 0x1c for SetResp() during ACTIVE transition
> Mar 19 18:37:03 343415 [4780B960] -> SMP dump:
> base_ver................0x1
> mgmt_class..............0x81
> class_ver...............0x1
> method..................0x81 (SubnGetResp)
> D bit...................0x1
> status..................0x1C00
> hop_ptr.................0x0
> hop_count...............0x5
> trans_id................0x14d09
> attr_id.................0x15 (PortInfo)
> resv....................0x0
> attr_mod................0x12
> m_key...................0x0000000000000000
> dr_slid.................0xFFFF
> dr_dlid.................0xFFFF
>
> Initial path: [0][1][11][1][6][16]
> Return path: [0][9][18][D][3][11]
> Reserved: [0][0][0][0][0][0][0]
>
> 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
>
> 00 00 00 00 00 00 00 00 00 00 00 00 11 02 03 02
>
> 12 52 00 11 40 40 00 08 08 04 F2 40 00 00 00 00
>
> 00 00 00 00 00 88 00 00 00 00 00 00 00 00 00 00
>
> Mar 19 18:37:03 343409 [45007960] -> PortInfo dump:
> port number.............0x11
> node_guid...............0x0005ad0000027c84
> port_guid...............0x0005ad0000027c84
> m_key...................0x0000000000000000
> subnet_prefix...........0x0000000000000000
> base_lid................0x0
> master_sm_base_lid......0x0
> capability_mask.........0x0
> diag_code...............0x0
> m_key_lease_period......0x0
> local_port_num..........0x11
> link_width_enabled......0x2
> link_width_supported....0x3
> link_width_active.......0x2
> link_speed_supported....0x1
> port_state..............INIT
> state_info2.............0x52
> m_key_protect_bits......0x0
> lmc.....................0x0
> link_speed..............0x11
> mtu_smsl................0x40
> vl_cap_init_type........0x40
> vl_high_limit...........0x0
> vl_arb_high_cap.........0x8
> vl_arb_low_cap..........0x8
> init_rep_mtu_cap........0x4
> vl_stall_life...........0xF2
> vl_enforce..............0x40
> m_key_violations........0x0
> p_key_violations........0x0
> q_key_violations........0x0
> guid_cap................0x0
> client_reregister.......0x0
> subnet_timeout..........0x0
> resp_time_value.........0x0
> error_threshold.........0x88
> Mar 19 18:37:03 343481 [45007960] -> Capabilities Mask:
> Mar 19 18:37:03 343532 [45007960] -> osm_pi_rcv_process_set: Received error
> status 0x1c for SetResp() during ACTIVE transition
> Mar 19 18:37:03 343537 [45007960] -> PortInfo dump:
> port number.............0x12
> node_guid...............0x0005ad0000027c84
> port_guid...............0x0005ad0000027c84
> m_key...................0x0000000000000000
> subnet_prefix...........0x0000000000000000
> base_lid................0x0
> master_sm_base_lid......0x0
> capability_mask.........0x0
> diag_code...............0x0
> m_key_lease_period......0x0
> local_port_num..........0x11
> link_width_enabled......0x2
> link_width_supported....0x3
> link_width_active.......0x2
> link_speed_supported....0x1
> port_state..............INIT
> state_info2.............0x52
> m_key_protect_bits......0x0
> lmc.....................0x0
> link_speed..............0x11
> mtu_smsl................0x40
> vl_cap_init_type........0x40
> vl_high_limit...........0x0
> vl_arb_high_cap.........0x8
> vl_arb_low_cap..........0x8
> init_rep_mtu_cap........0x4
> vl_stall_life...........0xF2
> vl_enforce..............0x40
> m_key_violations........0x0
> p_key_violations........0x0
> q_key_violations........0x0
> guid_cap................0x0
> client_reregister.......0x0
> subnet_timeout..........0x0
> resp_time_value.........0x0
> error_threshold.........0x88
> Mar 19 18:37:03 343555 [45007960] -> Capabilities Mask:
> Mar 19 18:37:03 348684 [45007960] -> SUBNET UP
> Mar 19 18:37:03 461748 [44606960] -> __osm_trap_rcv_process_request:
> Received Generic Notice type:0x01 num:128 Producer:2 from LID:0x001B
> TID:0x0000000000000048
> Mar 19 18:37:03 461958 [44606960] -> osm_report_notice: Reporting Generic
> Notice type:1 num:128 from LID:0x001B
> GID:0xfe80000000000000,0x0005ad00000281a7
> Mar 19 18:37:03 484827 [43C05960] -> __osm_trap_rcv_process_request:
> Received Generic Notice type:0x01 num:128 Producer:2 from LID:0x0148
> TID:0x000000000000003e
> Mar 19 18:37:03 486448 [43C05960] -> osm_report_notice: Reporting Generic
> Notice type:1 num:128 from LID:0x0148
> GID:0xfe80000000000000,0x0005ad00000281b3
> Mar 19 18:37:03 528040 [43204960] -> __osm_trap_rcv_process_request:
> Received Generic Notice type:0x01 num:128 Producer:2 from LID:0x001B
> TID:0x0000000000000049
> Mar 19 18:37:03 528154 [43204960] -> osm_report_notice: Reporting Generic
> Notice type:1 num:128 from LID:0x001B
> GID:0xfe80000000000000,0x0005ad00000281a7
> Mar 19 18:37:03 580196 [43C05960] -> __osm_trap_rcv_process_request:
> Received Generic Notice type:0x01 num:128 Producer:2 from LID:0x001B
> TID:0x000000000000004a
> Mar 19 18:37:03 580534 [43C05960] -> osm_report_notice: Reporting Generic
> Notice type:1 num:128 from LID:0x001B
> GID:0xfe80000000000000,0x0005ad00000281a7
> Mar 19 18:37:03 599784 [44606960] -> __osm_trap_rcv_process_request:
> Received Generic Notice type:0x01 num:128 Producer:2 from LID:0x001B
> TID:0x000000000000004b
> Mar 19 18:37:03 599879 [44606960] -> osm_report_notice: Reporting Generic
> Notice type:1 num:128 from LID:0x001B
> GID:0xfe80000000000000,0x0005ad00000281a7
> Mar 19 18:37:03 621883 [45A08960] -> __osm_trap_rcv_process_request:
> Received Generic Notice type:0x01 num:128 Producer:2 from LID:0x001B
> TID:0x000000000000004c
> Mar 19 18:37:03 621940 [45A08960] -> osm_report_notice: Reporting Generic
> Notice type:1 num:128 from LID:0x001B
> GID:0xfe80000000000000,0x0005ad00000281a7
> Mar 19 18:37:03 707894 [43C05960] -> osm_ucast_mgr_process: Min Hop Tables
> configured on all switches
> Mar 19 18:37:03 764678 [43204960] -> SUBNET UP
> Mar 19 18:37:03 783783 [41401960] -> __osm_trap_rcv_process_request:
> Received Generic Notice type:0x01 num:128 Producer:2 from LID:0x001B
> TID:0x000000000000004d
> Mar 19 18:37:03 783844 [41401960] -> osm_report_notice: Reporting Generic
> Notice type:1 num:128 from LID:0x001B
> GID:0xfe80000000000000,0x0005ad00000281a7
> Mar 19 18:37:04 000228 [43204960] -> __osm_trap_rcv_process_request:
> Received Generic Notice type:0x01 num:128 Producer:2 from LID:0x001B
> TID:0x000000000000004e
> Mar 19 18:37:04 000628 [43204960] -> osm_report_notice: Reporting Generic
> Notice type:1 num:128 from LID:0x001B
> GID:0xfe80000000000000,0x0005ad00000281a7
> Mar 19 18:37:04 022198 [43C05960] -> __osm_trap_rcv_process_request:
> Received Generic Notice type:0x01 num:128 Producer:2 from LID:0x001B
> TID:0x000000000000004f
> Mar 19 18:37:04 022299 [43C05960] -> osm_report_notice: Reporting Generic
> Notice type:1 num:128 from LID:0x001B
> GID:0xfe80000000000000,0x0005ad00000281a7
> Mar 19 18:37:04 043985 [42803960] -> __osm_trap_rcv_process_request:
> Received Generic Notice type:0x01 num:128 Producer:2 from LID:0x001B
> TID:0x0000000000000050
> Mar 19 18:37:04 044052 [42803960] -> osm_report_notice: Reporting Generic
> Notice type:1 num:128 from LID:0x001B
> GID:0xfe80000000000000,0x0005ad00000281a7
> Mar 19 18:37:04 155809 [45A08960] -> osm_ucast_mgr_process: Min Hop Tables
> configured on all switches
> Mar 19 18:37:04 210448 [41401960] -> SUBNET UP
> Mar 19 18:37:04 504490 [43204960] -> __osm_trap_rcv_process_request:
> Received Generic Notice type:0x01 num:128 Producer:2 from LID:0x0152
> TID:0x0000000000000017
> Mar 19 18:37:04 504569 [43204960] -> __osm_trap_rcv_process_request: ERR
> 3804: Received trap 24 times consecutively
> Mar 19 18:37:04 570084 [42803960] -> osm_ucast_mgr_process: Min Hop Tables
> configured on all switches
> Mar 19 18:37:04 626298 [43C05960] -> SUBNET UP
> Mar 19 18:37:54 424084 [41E02960] -> __osm_trap_rcv_process_request:
> Received Generic Notice type:0x01 num:128 Producer:2 from LID:0x001B
> TID:0x0000000000000051
> Mar 19 18:37:54 424430 [41E02960] -> osm_report_notice: Reporting Generic
> Notice type:1 num:128 from LID:0x001B
> GID:0xfe80000000000000,0x0005ad00000281a7
> Mar 19 18:37:54 424457 [41E02960] -> __osm_trap_rcv_process_request:
> Received Generic Notice type:0x01 num:128 Producer:2 from LID:0x0148
> TID:0x000000000000003f
> Mar 19 18:37:54 424522 [41E02960] -> osm_report_notice: Reporting Generic
> Notice type:1 num:128 from LID:0x0148
> GID:0xfe80000000000000,0x0005ad00000281b3
> Mar 19 18:37:54 722515 [44606960] -> osm_report_notice: Reporting Generic
> Notice type:3 num:65 from LID:0x0092
> GID:0xfe80000000000000,0x0005ad0000024bbb
> Mar 19 18:37:54 722536 [44606960] -> Removed port with
> GUID:0x0005ad0000024e0b LID range [0xB3,0xB3] of node:saguaro-23-4 HCA-1
> Mar 19 18:37:54 722558 [44606960] -> osm_report_notice: Reporting Generic
> Notice type:3 num:65 from LID:0x0092
> GID:0xfe80000000000000,0x0005ad0000024bbb
> Mar 19 18:37:54 722565 [44606960] -> Removed port with
> GUID:0x0005ad000002510b LID range [0xB5,0xB5] of node:saguaro-23-6 HCA-1
> Mar 19 18:37:54 722587 [44606960] -> osm_report_notice: Reporting Generic
> Notice type:3 num:65 from LID:0x0092
> GID:0xfe80000000000000,0x0005ad0000024bbb
> Mar 19 18:37:54 722594 [44606960] -> Removed port with
> GUID:0x0005ad000002511b LID range [0xA6,0xA6] of node:saguaro-22-1 HCA-1
> Mar 19 18:37:54 722636 [44606960] -> osm_report_notice: Reporting Generic
> Notice type:3 num:65 from LID:0x0092
> GID:0xfe80000000000000,0x0005ad0000024bbb
> Mar 19 18:37:54 722641 [44606960] -> Removed port with
> GUID:0x0005ad0000024b27 LID range [0xAF,0xAF] of node:saguaro-23-0 HCA-1
> Mar 19 18:37:54 722658 [44606960] -> osm_report_notice: Reporting Generic
> Notice type:3 num:65 from LID:0x0092
> GID:0xfe80000000000000,0x0005ad0000024bbb
> Mar 19 18:37:54 722663 [44606960] -> Removed port with
> GUID:0x0005ad0000025043 LID range [0xB4,0xB4] of node:saguaro-23-5 HCA-1
> Mar 19 18:37:54 722679 [44606960] -> osm_report_notice: Reporting Generic
> Notice type:3 num:65 from LID:0x0092
> GID:0xfe80000000000000,0x0005ad0000024bbb
> Mar 19 18:37:54 722684 [44606960] -> Removed port with
> GUID:0x0005ad0000024d47 LID range [0xB6,0xB6] of node:saguaro-23-7 HCA-1
> Mar 19 18:37:54 722701 [44606960] -> osm_report_notice: Reporting Generic
> Notice type:3 num:65 from LID:0x0092
> GID:0xfe80000000000000,0x0005ad0000024bbb
> Mar 19 18:37:54 722706 [44606960] -> Removed port with
> GUID:0x0005ad0000024d6b LID range [0xB8,0xB8] of node:saguaro-23-9 HCA-1
> Mar 19 18:37:54 722723 [44606960] -> osm_report_notice: Reporting Generic
> Notice type:3 num:65 from LID:0x0092
> GID:0xfe80000000000000,0x0005ad0000024bbb
> Mar 19 18:37:54 722728 [44606960] -> Removed port with
> GUID:0x0005ad0000024977 LID range [0xA9,0xA9] of node:saguaro-22-4 HCA-1
> Mar 19 18:37:54 722875 [44606960] -> osm_report_notice: Reporting Generic
> Notice type:3 num:65 from LID:0x0092
> GID:0xfe80000000000000,0x0005ad0000024bbb
> Mar 19 18:37:54 722880 [44606960] -> Removed port with
> GUID:0x0005ad0000027c84 LID range [0x152,0x152] of node:Topspin Switch TS120
> Mar 19 18:37:54 722909 [44606960] -> osm_report_notice: Reporting Generic
> Notice type:3 num:65 from LID:0x0092
> GID:0xfe80000000000000,0x0005ad0000024bbb
> Mar 19 18:37:54 722915 [44606960] -> Removed port with
> GUID:0x0005ad0000024d8b LID range [0xB7,0xB7] of node:saguaro-23-8 HCA-1
> Mar 19 18:37:54 722929 [44606960] -> osm_report_notice: Reporting Generic
> Notice type:3 num:65 from LID:0x0092
> GID:0xfe80000000000000,0x0005ad0000024bbb
> Mar 19 18:37:54 722934 [44606960] -> Removed port with
> GUID:0x0005ad000002498f LID range [0xA8,0xA8] of node:saguaro-22-3 HCA-1
> Mar 19 18:37:54 722949 [44606960] -> osm_report_notice: Reporting Generic
> Notice type:3 num:65 from LID:0x0092
> GID:0xfe80000000000000,0x0005ad0000024bbb
> Mar 19 18:37:54 722955 [44606960] -> Removed port with
> GUID:0x0005ad0000024c9b LID range [0xA7,0xA7] of node:saguaro-22-2 HCA-1
> Mar 19 18:37:54 722970 [44606960] -> osm_report_notice: Reporting Generic
> Notice type:3 num:65 from LID:0x0092
> GID:0xfe80000000000000,0x0005ad0000024bbb
> Mar 19 18:37:54 722975 [44606960] -> Removed port with
> GUID:0x0005ad0000024da7 LID range [0xB0,0xB0] of node:saguaro-23-1 HCA-1
> Mar 19 18:37:54 722992 [44606960] -> osm_report_notice: Reporting Generic
> Notice type:3 num:65 from LID:0x0092
> GID:0xfe80000000000000,0x0005ad0000024bbb
> Mar 19 18:37:54 722997 [44606960] -> Removed port with
> GUID:0x0005ad0000024cbb LID range [0xB2,0xB2] of node:saguaro-23-3 HCA-1
> Mar 19 18:37:54 723012 [44606960] -> osm_report_notice: Reporting Generic
> Notice type:3 num:65 from LID:0x0092
> GID:0xfe80000000000000,0x0005ad0000024bbb
> Mar 19 18:37:54 723073 [44606960] -> Removed port with
> GUID:0x0005ad00000249d3 LID range [0xB1,0xB1] of node:saguaro-23-2 HCA-1
> Mar 19 18:37:54 723090 [44606960] -> osm_report_notice: Reporting Generic
> Notice type:3 num:65 from LID:0x0092
> GID:0xfe80000000000000,0x0005ad0000024bbb
> Mar 19 18:37:54 723095 [44606960] -> Removed port with
> GUID:0x0005ad0000024feb LID range [0x153,0x153] of node:saguaro-22-5 HCA-1
> Mar 19 18:37:54 723111 [44606960] -> osm_report_notice: Reporting Generic
> Notice type:3 num:65 from LID:0x0092
> GID:0xfe80000000000000,0x0005ad0000024bbb
> Mar 19 18:37:54 723116 [44606960] -> Removed port with
> GUID:0x0005ad0000024afb LID range [0xA5,0xA5] of node:saguaro-22-0 HCA-1
> Mar 19 18:37:54 756302 [44606960] -> osm_ucast_mgr_process: Min Hop Tables
> configured on all switches
> Mar 19 18:37:54 806787 [45A08960] -> SUBNET UP
> Mar 19 18:37:55 149566 [41E02960] -> osm_ucast_mgr_process: Min Hop Tables
> configured on all switches
> Mar 19 18:37:55 198855 [41401960] -> SUBNET UP
> Mar 19 18:38:48 131054 [41E02960] -> __osm_trap_rcv_process_request:
> Received Generic Notice type:0x01 num:128 Producer:2 from LID:0x0148
> TID:0x0000000000000040
> Mar 19 18:38:48 131349 [41E02960] -> osm_report_notice: Reporting Generic
> Notice type:1 num:128 from LID:0x0148
> GID:0xfe80000000000000,0x0005ad00000281b3
> Mar 19 18:38:48 137230 [44606960] -> __osm_trap_rcv_process_request:
> Received Generic Notice type:0x01 num:128 Producer:2 from LID:0x001B
> TID:0x0000000000000052
> Mar 19 18:38:48 137268 [45007960] -> __osm_trap_rcv_process_request:
> Received Generic Notice type:0x01 num:128 Producer:2 from LID:0x0148
> TID:0x0000000000000041
> Mar 19 18:38:48 137395 [44606960] -> osm_report_notice: Reporting Generic
> Notice type:1 num:128 from LID:0x001B
> GID:0xfe80000000000000,0x0005ad00000281a7
> Mar 19 18:38:48 137432 [45007960] -> osm_report_notice: Reporting Generic
> Notice type:1 num:128 from LID:0x0148
> GID:0xfe80000000000000,0x0005ad00000281b3
> Mar 19 18:38:48 143370 [45A08960] -> __osm_trap_rcv_process_request:
> Received Generic Notice type:0x01 num:128 Producer:2 from LID:0x001B
> TID:0x0000000000000053
> Mar 19 18:38:48 144327 [45A08960] -> osm_report_notice: Reporting Generic
> Notice type:1 num:128 from LID:0x001B
> GID:0xfe80000000000000,0x0005ad00000281a7
> Mar 19 18:38:48 529052 [41E02960] -> osm_report_notice: Reporting Generic
> Notice type:3 num:64 from LID:0x0092
> GID:0xfe80000000000000,0x0005ad0000024bbb
> Mar 19 18:38:48 529065 [41E02960] -> Discovered new port with
> GUID:0x0005ad0000027c84 LID range [0x152,0x152] of node:Topspin Switch TS120
> Mar 19 18:38:48 529071 [41E02960] -> osm_report_notice: Reporting Generic
> Notice type:3 num:64 from LID:0x0092
> GID:0xfe80000000000000,0x0005ad0000024bbb
> Mar 19 18:38:48 529078 [41E02960] -> Discovered new port with
> GUID:0x0005ad0000024b27 LID range [0xAF,0xAF] of node:saguaro-23-0 HCA-1
> Mar 19 18:38:48 529083 [41E02960] -> osm_report_notice: Reporting Generic
> Notice type:3 num:64 from LID:0x0092
> GID:0xfe80000000000000,0x0005ad0000024bbb
> Mar 19 18:38:48 529090 [41E02960] -> Discovered new port with
> GUID:0x0005ad00000249d3 LID range [0xB1,0xB1] of node:saguaro-23-2 HCA-1
> Mar 19 18:38:48 529095 [41E02960] -> osm_report_notice: Reporting Generic
> Notice type:3 num:64 from LID:0x0092
> GID:0xfe80000000000000,0x0005ad0000024bbb
> Mar 19 18:38:48 529101 [41E02960] -> Discovered new port with
> GUID:0x0005ad0000024da7 LID range [0xB0,0xB0] of node:saguaro-23-1 HCA-1
> Mar 19 18:38:48 529106 [41E02960] -> osm_report_notice: Reporting Generic
> Notice type:3 num:64 from LID:0x0092
> GID:0xfe80000000000000,0x0005ad0000024bbb
> Mar 19 18:38:48 529113 [41E02960] -> Discovered new port with
> GUID:0x0005ad0000024cbb LID range [0xB2,0xB2] of node:saguaro-23-3 HCA-1
> Mar 19 18:38:48 529118 [41E02960] -> osm_report_notice: Reporting Generic
> Notice type:3 num:64 from LID:0x0092
> GID:0xfe80000000000000,0x0005ad0000024bbb
> Mar 19 18:38:48 529124 [41E02960] -> Discovered new port with
> GUID:0x0005ad0000024e0b LID range [0xB3,0xB3] of node:saguaro-23-4 HCA-1
> Mar 19 18:38:48 529129 [41E02960] -> osm_report_notice: Reporting Generic
> Notice type:3 num:64 from LID:0x0092
> GID:0xfe80000000000000,0x0005ad0000024bbb
> Mar 19 18:38:48 529136 [41E02960] -> Discovered new port with
> GUID:0x0005ad0000025043 LID range [0xB4,0xB4] of node:saguaro-23-5 HCA-1
> Mar 19 18:38:48 529141 [41E02960] -> osm_report_notice: Reporting Generic
> Notice type:3 num:64 from LID:0x0092
> GID:0xfe80000000000000,0x0005ad0000024bbb
> Mar 19 18:38:48 529147 [41E02960] -> Discovered new port with
> GUID:0x0005ad000002510b LID range [0xB5,0xB5] of node:saguaro-23-6 HCA-1
> Mar 19 18:38:48 529152 [41E02960] -> osm_report_notice: Reporting Generic
> Notice type:3 num:64 from LID:0x0092
> GID:0xfe80000000000000,0x0005ad0000024bbb
> Mar 19 18:38:48 529159 [41E02960] -> Discovered new port with
> GUID:0x0005ad0000024d47 LID range [0xB6,0xB6] of node:saguaro-23-7 HCA-1
> Mar 19 18:38:48 529164 [41E02960] -> osm_report_notice: Reporting Generic
> Notice type:3 num:64 from LID:0x0092
> GID:0xfe80000000000000,0x0005ad0000024bbb
> Mar 19 18:38:48 529170 [41E02960] -> Discovered new port with
> GUID:0x0005ad0000024d8b LID range [0xB7,0xB7] of node:saguaro-23-8 HCA-1
> Mar 19 18:38:48 529175 [41E02960] -> osm_report_notice: Reporting Generic
> Notice type:3 num:64 from LID:0x0092
> GID:0xfe80000000000000,0x0005ad0000024bbb
> Mar 19 18:38:48 529182 [41E02960] -> Discovered new port with
> GUID:0x0005ad0000024d6b LID range [0xB8,0xB8] of node:saguaro-23-9 HCA-1
> Mar 19 18:38:48 529186 [41E02960] -> osm_report_notice: Reporting Generic
> Notice type:3 num:64 from LID:0x0092
> GID:0xfe80000000000000,0x0005ad0000024bbb
> Mar 19 18:38:48 529193 [41E02960] -> Discovered new port with
> GUID:0x0005ad0000024afb LID range [0xA5,0xA5] of node:saguaro-22-0 HCA-1
> Mar 19 18:38:48 529198 [41E02960] -> osm_report_notice: Reporting Generic
> Notice type:3 num:64 from LID:0x0092
> GID:0xfe80000000000000,0x0005ad0000024bbb
> Mar 19 18:38:48 529204 [41E02960] -> Discovered new port with
> GUID:0x0005ad000002511b LID range [0xA6,0xA6] of node:saguaro-22-1 HCA-1
> Mar 19 18:38:48 529209 [41E02960] -> osm_report_notice: Reporting Generic
> Notice type:3 num:64 from LID:0x0092
> GID:0xfe80000000000000,0x0005ad0000024bbb
> Mar 19 18:38:48 529216 [41E02960] -> Discovered new port with
> GUID:0x0005ad0000024c9b LID range [0xA7,0xA7] of node:saguaro-22-2 HCA-1
> Mar 19 18:38:48 529271 [41E02960] -> osm_report_notice: Reporting Generic
> Notice type:3 num:64 from LID:0x0092
> GID:0xfe80000000000000,0x0005ad0000024bbb
> Mar 19 18:38:48 529277 [41E02960] -> Discovered new port with
> GUID:0x0005ad000002498f LID range [0xA8,0xA8] of node:saguaro-22-3 HCA-1
> Mar 19 18:38:48 529281 [41E02960] -> osm_report_notice: Reporting Generic
> Notice type:3 num:64 from LID:0x0092
> GID:0xfe80000000000000,0x0005ad0000024bbb
> Mar 19 18:38:48 529286 [41E02960] -> Discovered new port with
> GUID:0x0005ad0000024977 LID range [0xA9,0xA9] of node:saguaro-22-4 HCA-1
> Mar 19 18:38:48 529290 [41E02960] -> osm_report_notice: Reporting Generic
> Notice type:3 num:64 from LID:0x0092
> GID:0xfe80000000000000,0x0005ad0000024bbb
> Mar 19 18:38:48 529294 [41E02960] -> Discovered new port with
> GUID:0x0005ad0000024feb LID range [0x153,0x153] of node:saguaro-22-5 HCA-1
> Mar 19 18:38:48 560082 [41E02960] -> osm_ucast_mgr_process: Min Hop Tables
> configured on all switches
> Mar 19 18:38:48 630464 [43204960] -> SUBNET UP
> Mar 19 18:38:49 018498 [44606960] -> osm_ucast_mgr_process: Min Hop Tables
> configured on all switches
> Mar 19 18:38:49 073355 [45007960] -> SUBNET UP
> Mar 19 18:39:04 189829 [45007960] -> __osm_trap_rcv_process_request:
> Received Generic Notice type:0x01 num:128 Producer:2 from LID:0x0152
> TID:0x0000000000000000
> Mar 19 18:39:04 190072 [45007960] -> osm_report_notice: Reporting Generic
> Notice type:1 num:128 from LID:0x0152
> GID:0xfe80000000000000,0x0005ad0000027c84
> Mar 19 18:39:04 307827 [44606960] -> __osm_trap_rcv_process_request:
> Received Generic Notice type:0x01 num:128 Producer:2 from LID:0x0152
> TID:0x0000000000000001
> Mar 19 18:39:04 307940 [44606960] -> osm_report_notice: Reporting Generic
> Notice type:1 num:128 from LID:0x0152
> GID:0xfe80000000000000,0x0005ad0000027c84
> Mar 19 18:39:04 330104 [44606960] -> __osm_trap_rcv_process_request:
> Received Generic Notice type:0x01 num:128 Producer:2 from LID:0x0152
> TID:0x0000000000000002
> Mar 19 18:39:04 330210 [44606960] -> osm_report_notice: Reporting Generic
> Notice type:1 num:128 from LID:0x0152
> GID:0xfe80000000000000,0x0005ad0000027c84
> Mar 19 18:39:04 468676 [41401960] -> __osm_trap_rcv_process_request:
> Received Generic Notice type:0x01 num:128 Producer:2 from LID:0x0152
> TID:0x0000000000000003
> Mar 19 18:39:04 468758 [41401960] -> osm_report_notice: Reporting Generic
> Notice type:1 num:128 from LID:0x0152
> GID:0xfe80000000000000,0x0005ad0000027c84
> Mar 19 18:39:04 680305 [42803960] -> __osm_trap_rcv_process_request:
> Received Generic Notice type:0x01 num:128 Producer:2 from LID:0x0152
> TID:0x0000000000000004
> Mar 19 18:39:04 680400 [42803960] -> osm_report_notice: Reporting Generic
> Notice type:1 num:128 from LID:0x0152
> GID:0xfe80000000000000,0x0005ad0000027c84
> Mar 19 18:39:04 702144 [41E02960] -> __osm_trap_rcv_process_request:
> Received Generic Notice type:0x01 num:128 Producer:2 from LID:0x0152
> TID:0x0000000000000005
> Mar 19 18:39:04 702286 [41E02960] -> osm_report_notice: Reporting Generic
> Notice type:1 num:128 from LID:0x0152
> GID:0xfe80000000000000,0x0005ad0000027c84
> Mar 19 18:39:04 704346 [43204960] -> osm_report_notice: Reporting Generic
> Notice type:3 num:65 from LID:0x0092
> GID:0xfe80000000000000,0x0005ad0000024bbb
> Mar 19 18:39:04 704354 [43204960] -> Removed port with
> GUID:0x0005ad00000249d3 LID range [0xB1,0xB1] of node:saguaro-23-2 HCA-1
> Mar 19 18:39:04 739059 [43204960] -> osm_ucast_mgr_process: Min Hop Tables
> configured on all switches
> Mar 19 18:39:04 739896 [41E02960] -> __osm_trap_rcv_process_request:
> Received Generic Notice type:0x01 num:128 Producer:2 from LID:0x0152
> TID:0x0000000000000006
> Mar 19 18:39:04 783807 [41E02960] -> osm_report_notice: Reporting Generic
> Notice type:1 num:128 from LID:0x0152
> GID:0xfe80000000000000,0x0005ad0000027c84
> Mar 19 18:39:04 797411 [44606960] -> SUBNET UP
> Mar 19 18:39:04 849970 [42803960] -> __osm_trap_rcv_process_request:
> Received Generic Notice type:0x01 num:128 Producer:2 from LID:0x0152
> TID:0x0000000000000007
> Mar 19 18:39:04 850195 [42803960] -> osm_report_notice: Reporting Generic
> Notice type:1 num:128 from LID:0x0152
> GID:0xfe80000000000000,0x0005ad0000027c84
> Mar 19 18:39:04 853735 [43C05960] -> __osm_trap_rcv_process_request:
> Received Generic Notice type:0x01 num:128 Producer:2 from LID:0x0152
> TID:0x0000000000000008
> Mar 19 18:39:04 853809 [43C05960] -> osm_report_notice: Reporting Generic
> Notice type:1 num:128 from LID:0x0152
> GID:0xfe80000000000000,0x0005ad0000027c84
> Mar 19 18:39:04 897727 [43C05960] -> __osm_trap_rcv_process_request:
> Received Generic Notice type:0x01 num:128 Producer:2 from LID:0x0152
> TID:0x0000000000000009
> Mar 19 18:39:04 897860 [43C05960] -> osm_report_notice: Reporting Generic
> Notice type:1 num:128 from LID:0x0152
> GID:0xfe80000000000000,0x0005ad0000027c84
> Mar 19 18:39:04 901577 [41401960] -> __osm_trap_rcv_process_request:
> Received Generic Notice type:0x01 num:128 Producer:2 from LID:0x0152
> TID:0x000000000000000a
> Mar 19 18:39:04 901719 [41401960] -> osm_report_notice: Reporting Generic
> Notice type:1 num:128 from LID:0x0152
> GID:0xfe80000000000000,0x0005ad0000027c84
> Mar 19 18:39:04 923271 [45007960] -> __osm_trap_rcv_process_request:
> Received Generic Notice type:0x01 num:128 Producer:2 from LID:0x0152
> TID:0x000000000000000b
> Mar 19 18:39:04 923377 [45007960] -> __osm_trap_rcv_process_request: ERR
> 3804: Received trap 11 times consecutively
> Mar 19 18:39:05 106246 [45007960] -> __osm_trap_rcv_process_request:
> Received Generic Notice type:0x01 num:128 Producer:2 from LID:0x0152
> TID:0x000000000000000c
> Mar 19 18:39:05 106314 [45007960] -> __osm_trap_rcv_process_request: ERR
> 3804: Received trap 12 times consecutively
> Mar 19 18:39:05 178215 [44606960] -> __osm_trap_rcv_process_request:
> Received Generic Notice type:0x01 num:128 Producer:2 from LID:0x0152
> TID:0x000000000000000d
> Mar 19 18:39:05 178258 [44606960] -> __osm_trap_rcv_process_request: ERR
> 3804: Received trap 13 times consecutively
> Mar 19 18:39:05 272913 [42803960] -> __osm_trap_rcv_process_request:
> Received Generic Notice type:0x01 num:128 Producer:2 from LID:0x0152
> TID:0x000000000000000e
> Mar 19 18:39:05 272983 [42803960] -> __osm_trap_rcv_process_request: ERR
> 3804: Received trap 14 times consecutively
> Mar 19 18:39:05 339633 [43204960] -> __osm_trap_rcv_process_request:
> Received Generic Notice type:0x01 num:128 Producer:2 from LID:0x0152
> TID:0x000000000000000f
> Mar 19 18:39:05 339679 [43204960] -> __osm_trap_rcv_process_request: ERR
> 3804: Received trap 15 times consecutively
> Mar 19 18:39:05 469093 [41401960] -> __osm_trap_rcv_process_request:
> Received Generic Notice type:0x01 num:128 Producer:2 from LID:0x0152
> TID:0x0000000000000010
> Mar 19 18:39:05 469145 [41401960] -> __osm_trap_rcv_process_request: ERR
> 3804: Received trap 16 times consecutively
> Mar 19 18:39:05 484587 [44606960] -> __osm_trap_rcv_process_request:
> Received Generic Notice type:0x01 num:128 Producer:2 from LID:0x0152
> TID:0x0000000000000011
> Mar 19 18:39:05 484633 [44606960] -> __osm_trap_rcv_process_request: ERR
> 3804: Received trap 17 times consecutively
> Mar 19 18:39:05 574251 [43C05960] -> __osm_trap_rcv_process_request:
> Received Generic Notice type:0x01 num:128 Producer:2 from LID:0x0152
> TID:0x0000000000000012
> Mar 19 18:39:05 574301 [43C05960] -> __osm_trap_rcv_process_request: ERR
> 3804: Received trap 18 times consecutively
> Mar 19 18:39:05 602665 [41E02960] -> __osm_trap_rcv_process_request:
> Received Generic Notice type:0x01 num:128 Producer:2 from LID:0x0152
> TID:0x0000000000000013
> Mar 19 18:39:05 602700 [41E02960] -> __osm_trap_rcv_process_request: ERR
> 3804: Received trap 19 times consecutively
> Mar 19 18:39:05 646331 [45007960] -> __osm_trap_rcv_process_request:
> Received Generic Notice type:0x01 num:128 Producer:2 from LID:0x0152
> TID:0x0000000000000014
> Mar 19 18:39:05 646369 [45007960] -> __osm_trap_rcv_process_request: ERR
> 3804: Received trap 20 times consecutively
> Mar 19 18:39:05 834613 [41E02960] -> __osm_trap_rcv_process_request:
> Received Generic Notice type:0x01 num:128 Producer:2 from LID:0x0152
> TID:0x0000000000000015
> Mar 19 18:39:05 834685 [41E02960] -> __osm_trap_rcv_process_request: ERR
> 3804: Received trap 21 times consecutively
> Mar 19 18:39:05 851128 [45007960] -> __osm_trap_rcv_process_request:
> Received Generic Notice type:0x01 num:128 Producer:2 from LID:0x0152
> TID:0x0000000000000016
> Mar 19 18:39:05 851166 [45007960] -> __osm_trap_rcv_process_request: ERR
> 3804: Received trap 22 times consecutively
> Mar 19 18:39:05 875540 [45A08960] -> __osm_trap_rcv_process_request:
> Received Generic Notice type:0x01 num:128 Producer:2 from LID:0x0152
> TID:0x0000000000000017
> Mar 19 18:39:05 875592 [45A08960] -> __osm_trap_rcv_process_request: ERR
> 3804: Received trap 23 times consecutively
> Mar 19 18:39:05 897378 [42803960] -> __osm_trap_rcv_process_request:
> Received Generic Notice type:0x01 num:128 Producer:2 from LID:0x0152
> TID:0x0000000000000018
> Mar 19 18:39:05 897424 [42803960] -> __osm_trap_rcv_process_request: ERR
> 3804: Received trap 24 times consecutively
> Mar 19 18:39:05 907232 [4780B960] -> umad_receiver: ERR 5409: send completed
> with error (method=0x1 attr=0x15 trans_id=0x124ef0001c2fe) -- dropping
> Mar 19 18:39:05 907249 [4780B960] -> umad_receiver: ERR 5411: DR SMP
> Mar 19 18:39:05 907259 [4780B960] -> __osm_sm_mad_ctrl_send_err_cb: ERR
> 3113: MAD completed in error (IB_TIMEOUT)
> Mar 19 18:39:05 907295 [4780B960] -> SMP dump:
> base_ver................0x1
> mgmt_class..............0x81
> class_ver...............0x1
> method..................0x1 (SubnGet)
> D bit...................0x0
> status..................0x0
> hop_ptr.................0x0
> hop_count...............0x6
> trans_id................0x1c2fe
> attr_id.................0x15 (PortInfo)
> resv....................0x0
> attr_mod................0x1
> m_key...................0x0000000000000000
> dr_slid.................0xFFFF
> dr_dlid.................0xFFFF
>
> Initial path: [0][1][11][1][6][16][8]
> Return path: [0][0][0][0][0][0][0]
> Reserved: [0][0][0][0][0][0][0]
>
> 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
>
> 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
>
> 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
>
> 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
>
> Mar 19 18:39:05 907372 [41401960] -> osm_report_notice: Reporting Generic
> Notice type:3 num:65 from LID:0x0092
> GID:0xfe80000000000000,0x0005ad0000024bbb
> Mar 19 18:39:05 907384 [41401960] -> Removed port with
> GUID:0x0005ad0000024e0b LID range [0xB3,0xB3] of node:saguaro-23-4 HCA-1
> Mar 19 18:39:05 907407 [41401960] -> osm_report_notice: Reporting Generic
> Notice type:3 num:65 from LID:0x0092
> GID:0xfe80000000000000,0x0005ad0000024bbb
> Mar 19 18:39:05 907414 [41401960] -> Removed port with
> GUID:0x0005ad0000025043 LID range [0xB4,0xB4] of node:saguaro-23-5 HCA-1
> Mar 19 18:39:05 907480 [41401960] -> osm_report_notice: Reporting Generic
> Notice type:3 num:65 from LID:0x0092
> GID:0xfe80000000000000,0x0005ad0000024bbb
> Mar 19 18:39:05 907485 [41401960] -> Removed port with
> GUID:0x0005ad0000024cbb LID range [0xB2,0xB2] of node:saguaro-23-3 HCA-1
> Mar 19 18:39:05 907577 [41401960] -> osm_report_notice: Reporting Generic
> Notice type:3 num:65 from LID:0x0092
> GID:0xfe80000000000000,0x0005ad0000024bbb
> Mar 19 18:39:05 907582 [41401960] -> Removed port with
> GUID:0x0005ad0000024d47 LID range [0xB6,0xB6] of node:saguaro-23-7 HCA-1
> Mar 19 18:39:05 907618 [41401960] -> osm_drop_mgr_process: ERR 0108: Unknown
> remote side for node 0x0005ad0000027c84 port 8. Adding to light sweep
> sampling list
> Mar 19 18:39:05 907657 [41401960] -> Directed Path Dump of 5 hop path:
> Path = [0][1][11][1][6][16]
> Mar 19 18:39:05 911559 [43204960] -> osm_report_notice: Reporting Generic
> Notice type:3 num:64 from LID:0x0092
> GID:0xfe80000000000000,0x0005ad0000024bbb
> Mar 19 18:39:05 911572 [43204960] -> Discovered new port with
> GUID:0x0005ad00000249d3 LID range [0xB1,0xB1] of node:saguaro-23-2 HCA-1
> Mar 19 18:39:05 927229 [43C05960] -> __osm_trap_rcv_process_request:
> Received Generic Notice type:0x01 num:128 Producer:2 from LID:0x0152
> TID:0x0000000000000019
> Mar 19 18:39:05 927285 [43C05960] -> __osm_trap_rcv_process_request: ERR
> 3804: Received trap 25 times consecutively
> Mar 19 18:39:05 942538 [43204960] -> osm_ucast_mgr_process: Min Hop Tables
> configured on all switches
> Mar 19 18:39:06 000027 [41E02960] -> SUBNET UP
> Mar 19 18:39:06 130255 [43204960] -> __osm_trap_rcv_process_request:
> Received Generic Notice type:0x01 num:128 Producer:2 from LID:0x0152
> TID:0x000000000000001a
> Mar 19 18:39:06 130308 [43204960] -> __osm_trap_rcv_process_request: ERR
> 3804: Received trap 26 times consecutively
> Mar 19 18:39:06 131922 [42803960] -> __osm_trap_rcv_process_request:
> Received Generic Notice type:0x01 num:128 Producer:2 from LID:0x0148
> TID:0x0000000000000042
> Mar 19 18:39:06 132063 [42803960] -> osm_report_notice: Reporting Generic
> Notice type:1 num:128 from LID:0x0148
> GID:0xfe80000000000000,0x0005ad00000281b3
> Mar 19 18:39:06 154579 [43C05960] -> __osm_trap_rcv_process_request:
> Received Generic Notice type:0x01 num:128 Producer:2 from LID:0x0152
> TID:0x000000000000001b
> Mar 19 18:39:06 154681 [43C05960] -> __osm_trap_rcv_process_request: ERR
> 3804: Received trap 27 times consecutively
> Mar 19 18:39:06 176248 [44606960] -> __osm_trap_rcv_process_request:
> Received Generic Notice type:0x01 num:128 Producer:2 from LID:0x0152
> TID:0x000000000000001c
> Mar 19 18:39:06 176304 [44606960] -> __osm_trap_rcv_process_request: ERR
> 3804: Received trap 28 times consecutively
> Mar 19 18:39:06 198132 [44606960] -> __osm_trap_rcv_process_request:
> Received Generic Notice type:0x01 num:128 Producer:2 from LID:0x0152
> TID:0x000000000000001d
> Mar 19 18:39:06 198195 [44606960] -> __osm_trap_rcv_process_request: ERR
> 3804: Received trap 29 times consecutively
> Mar 19 18:39:06 230022 [43C05960] -> __osm_trap_rcv_process_request:
> Received Generic Notice type:0x01 num:128 Producer:2 from LID:0x0152
> TID:0x000000000000001e
> Mar 19 18:39:06 230108 [43C05960] -> __osm_trap_rcv_process_request: ERR
> 3804: Received trap 30 times consecutively
> Mar 19 18:39:06 230229 [43204960] -> __osm_trap_rcv_process_request:
> Received Generic Notice type:0x01 num:128 Producer:2 from LID:0x0148
> TID:0x0000000000000043
> Mar 19 18:39:06 230311 [43204960] -> osm_report_notice: Reporting Generic
> Notice type:1 num:128 from LID:0x0148
> GID:0xfe80000000000000,0x0005ad00000281b3
> Mar 19 18:39:06 399543 [43C05960] -> osm_report_notice: Reporting Generic
> Notice type:3 num:64 from LID:0x0092
> GID:0xfe80000000000000,0x0005ad0000024bbb
> Mar 19 18:39:06 399556 [43C05960] -> Discovered new port with
> GUID:0x0005ad0000024cbb LID range [0xB2,0xB2] of node:saguaro-23-3 HCA-1
> Mar 19 18:39:06 399562 [43C05960] -> osm_report_notice: Reporting Generic
> Notice type:3 num:64 from LID:0x0092
> GID:0xfe80000000000000,0x0005ad0000024bbb
> Mar 19 18:39:06 399569 [43C05960] -> Discovered new port with
> GUID:0x0005ad0000024e0b LID range [0xB3,0xB3] of node:saguaro-23-4 HCA-1
> Mar 19 18:39:06 399574 [43C05960] -> osm_report_notice: Reporting Generic
> Notice type:3 num:64 from LID:0x0092
> GID:0xfe80000000000000,0x0005ad0000024bbb
> Mar 19 18:39:06 399580 [43C05960] -> Discovered new port with
> GUID:0x0005ad0000025043 LID range [0xB4,0xB4] of node:saguaro-23-5 HCA-1
> Mar 19 18:39:06 399585 [43C05960] -> osm_report_notice: Reporting Generic
> Notice type:3 num:64 from LID:0x0092
> GID:0xfe80000000000000,0x0005ad0000024bbb
> Mar 19 18:39:06 399592 [43C05960] -> Discovered new port with
> GUID:0x0005ad0000024d47 LID range [0xB6,0xB6] of node:saguaro-23-7 HCA-1
> Mar 19 18:39:06 430598 [43C05960] -> osm_ucast_mgr_process: Min Hop Tables
> configured on all switches
> Mar 19 18:39:06 494689 [44606960] -> SUBNET UP
> Mar 19 18:39:06 837303 [43204960] -> __osm_trap_rcv_process_request:
> Received Generic Notice type:0x01 num:128 Producer:2 from LID:0x0152
> TID:0x000000000000001f
> Mar 19 18:39:06 837446 [43204960] -> __osm_trap_rcv_process_request: ERR
> 3804: Received trap 31 times consecutively
> Mar 19 18:39:06 838528 [43C05960] -> __osm_trap_rcv_process_request:
> Received Generic Notice type:0x01 num:128 Producer:2 from LID:0x0148
> TID:0x0000000000000044
> Mar 19 18:39:06 838636 [43C05960] -> osm_report_notice: Reporting Generic
> Notice type:1 num:128 from LID:0x0148
> GID:0xfe80000000000000,0x0005ad00000281b3
> Mar 19 18:39:06 876308 [43C05960] -> osm_ucast_mgr_process: Min Hop Tables
> configured on all switches
> Mar 19 18:39:07 028376 [45A08960] -> __osm_trap_rcv_process_request:
> Received Generic Notice type:0x01 num:128 Producer:2 from LID:0x0152
> TID:0x0000000000000020
> Mar 19 18:39:07 028459 [45A08960] -> __osm_trap_rcv_process_request: ERR
> 3804: Received trap 32 times consecutively
> Mar 19 18:39:07 028545 [43204960] -> __osm_trap_rcv_process_request:
> Received Generic Notice type:0x01 num:128 Producer:2 from LID:0x0148
> TID:0x0000000000000045
> Mar 19 18:39:07 028652 [43204960] -> osm_report_notice: Reporting Generic
> Notice type:1 num:128 from LID:0x0148
> GID:0xfe80000000000000,0x0005ad00000281b3
> Mar 19 18:39:07 030190 [45007960] -> __osm_trap_rcv_process_request:
> Received Generic Notice type:0x01 num:128 Producer:2 from LID:0x001B
> TID:0x0000000000000054
> Mar 19 18:39:07 030277 [45007960] -> osm_report_notice: Reporting Generic
> Notice type:1 num:128 from LID:0x001B
> GID:0xfe80000000000000,0x0005ad00000281a7
> Mar 19 18:39:07 096812 [41401960] -> __osm_trap_rcv_process_request:
> Received Generic Notice type:0x01 num:128 Producer:2 from LID:0x0148
> TID:0x0000000000000046
> Mar 19 18:39:07 096959 [41401960] -> osm_report_notice: Reporting Generic
> Notice type:1 num:128 from LID:0x0148
> GID:0xfe80000000000000,0x0005ad00000281b3
> Mar 19 18:39:07 111719 [4780B960] -> __osm_sm_mad_ctrl_rcv_callback: ERR
> 3111: Error status = 0x1C00
> Mar 19 18:39:07 111759 [4780B960] -> SMP dump:
> base_ver................0x1
> mgmt_class..............0x81
> class_ver...............0x1
> method..................0x81 (SubnGetResp)
> D bit...................0x1
> status..................0x1C00
> hop_ptr.................0x0
> hop_count...............0x5
> trans_id................0x1dfac
> attr_id.................0x15 (PortInfo)
> resv....................0x0
> attr_mod................0x11
> m_key...................0x0000000000000000
> dr_slid.................0xFFFF
> dr_dlid.................0xFFFF
>
> Initial path: [0][1][11][1][4][16]
> Return path: [0][9][18][D][1][11]
> Reserved: [0][0][0][0][0][0][0]
>
> 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
>
> 00 00 00 00 00 00 00 00 00 00 00 00 11 02 03 02
>
> 12 52 00 11 40 40 00 08 08 04 F2 40 00 00 00 00
>
> 00 00 00 00 00 88 00 00 00 00 00 00 00 00 00 00
>
> Mar 19 18:39:07 111810 [41E02960] -> osm_pi_rcv_process_set: Received error
> status 0x1c for SetResp() during ACTIVE transition
> Mar 19 18:39:07 111814 [4780B960] -> __osm_sm_mad_ctrl_rcv_callback: ERR
> 3111: Error status = 0x1C00
> Mar 19 18:39:07 111831 [41E02960] -> PortInfo dump:
> port number.............0x11
> node_guid...............0x0005ad0000027c84
> port_guid...............0x0005ad0000027c84
> m_key...................0x0000000000000000
> subnet_prefix...........0x0000000000000000
> base_lid................0x0
> master_sm_base_lid......0x0
> capability_mask.........0x0
> diag_code...............0x0
> m_key_lease_period......0x0
> local_port_num..........0x11
> link_width_enabled......0x2
> link_width_supported....0x3
> link_width_active.......0x2
> link_speed_supported....0x1
> port_state..............INIT
> state_info2.............0x52
> m_key_protect_bits......0x0
> lmc.....................0x0
> link_speed..............0x11
> mtu_smsl................0x40
> vl_cap_init_type........0x40
> vl_high_limit...........0x0
> vl_arb_high_cap.........0x8
> vl_arb_low_cap..........0x8
> init_rep_mtu_cap........0x4
> vl_stall_life...........0xF2
> vl_enforce..............0x40
> m_key_violations........0x0
> p_key_violations........0x0
> q_key_violations........0x0
> guid_cap................0x0
> client_reregister.......0x0
> subnet_timeout..........0x0
> resp_time_value.........0x0
> error_threshold.........0x88
> Mar 19 18:39:07 111868 [41E02960] -> Capabilities Mask:
> Mar 19 18:39:07 111844 [4780B960] -> SMP dump:
> base_ver................0x1
> mgmt_class..............0x81
> class_ver...............0x1
> method..................0x81 (SubnGetResp)
> D bit...................0x1
> status..................0x1C00
> hop_ptr.................0x0
> hop_count...............0x5
> trans_id................0x1dfad
> attr_id.................0x15 (PortInfo)
> resv....................0x0
> attr_mod................0x12
> m_key...................0x0000000000000000
> dr_slid.................0xFFFF
> dr_dlid.................0xFFFF
>
> Initial path: [0][1][11][1][4][16]
> Return path: [0][9][18][D][1][11]
> Reserved: [0][0][0][0][0][0][0]
>
> 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
>
> 00 00 00 00 00 00 00 00 00 00 00 00 11 02 03 02
>
> 12 52 00 11 40 40 00 08 08 04 F2 40 00 00 00 00
>
> 00 00 00 00 00 88 00 00 00 00 00 00 00 00 00 00
>
> Mar 19 18:39:07 112011 [41401960] -> osm_pi_rcv_process_set: Received error
> status 0x1c for SetResp() during ACTIVE transition
> Mar 19 18:39:07 112018 [41401960] -> PortInfo dump:
> port number.............0x12
> node_guid...............0x0005ad0000027c84
> port_guid...............0x0005ad0000027c84
> m_key...................0x0000000000000000
> subnet_prefix...........0x0000000000000000
> base_lid................0x0
> master_sm_base_lid......0x0
> capability_mask.........0x0
> diag_code...............0x0
> m_key_lease_period......0x0
> local_port_num..........0x11
> link_width_enabled......0x2
> link_width_supported....0x3
> link_width_active.......0x2
> link_speed_supported....0x1
> port_state..............INIT
> state_info2.............0x52
> m_key_protect_bits......0x0
> lmc.....................0x0
> link_speed..............0x11
> mtu_smsl................0x40
> vl_cap_init_type........0x40
> vl_high_limit...........0x0
> vl_arb_high_cap.........0x8
> vl_arb_low_cap..........0x8
> init_rep_mtu_cap........0x4
> vl_stall_life...........0xF2
> vl_enforce..............0x40
> m_key_violations........0x0
> p_key_violations........0x0
> q_key_violations........0x0
> guid_cap................0x0
> client_reregister.......0x0
> subnet_timeout..........0x0
> resp_time_value.........0x0
> error_threshold.........0x88
> Mar 19 18:39:07 112034 [41401960] -> Capabilities Mask:
> Mar 19 18:39:07 117211 [45A08960] -> SUBNET UP
> Mar 19 18:39:07 354540 [41E02960] -> __osm_trap_rcv_process_request:
> Received Generic Notice type:0x01 num:128 Producer:2 from LID:0x0148
> TID:0x0000000000000047
> Mar 19 18:39:07 354686 [41E02960] -> osm_report_notice: Reporting Generic
> Notice type:1 num:128 from LID:0x0148
> GID:0xfe80000000000000,0x0005ad00000281b3
> Mar 19 18:39:07 383453 [42803960] -> __osm_trap_rcv_process_request:
> Received Generic Notice type:0x01 num:128 Producer:2 from LID:0x001B
> TID:0x0000000000000055
> Mar 19 18:39:07 383530 [42803960] -> osm_report_notice: Reporting Generic
> Notice type:1 num:128 from LID:0x001B
> GID:0xfe80000000000000,0x0005ad00000281a7
> Mar 19 18:39:07 497601 [42803960] -> osm_ucast_mgr_process: Min Hop Tables
> configured on all switches
> Mar 19 18:39:07 548184 [43204960] -> __osm_trap_rcv_process_request:
> Received Generic Notice type:0x01 num:128 Producer:2 from LID:0x001B
> TID:0x0000000000000056
> Mar 19 18:39:07 548217 [43C05960] -> SUBNET UP
> Mar 19 18:39:07 548427 [43204960] -> osm_report_notice: Reporting Generic
> Notice type:1 num:128 from LID:0x001B
> GID:0xfe80000000000000,0x0005ad00000281a7
> Mar 19 18:39:07 878403 [45007960] -> __osm_trap_rcv_process_request:
> Received Generic Notice type:0x01 num:128 Producer:2 from LID:0x001B
> TID:0x0000000000000057
> Mar 19 18:39:07 887312 [45A08960] -> __osm_trap_rcv_process_request:
> Received Generic Notice type:0x01 num:128 Producer:2 from LID:0x001B
> TID:0x0000000000000058
> Mar 19 18:39:07 888156 [41E02960] -> osm_ucast_mgr_process: Min Hop Tables
> configured on all switches
> Mar 19 18:39:07 929819 [45007960] -> osm_report_notice: Reporting Generic
> Notice type:1 num:128 from LID:0x001B
> GID:0xfe80000000000000,0x0005ad00000281a7
> Mar 19 18:39:07 929834 [45A08960] -> osm_report_notice: Reporting Generic
> Notice type:1 num:128 from LID:0x001B
> GID:0xfe80000000000000,0x0005ad00000281a7
> Mar 19 18:39:07 931166 [45007960] -> __osm_trap_rcv_process_request:
> Received Generic Notice type:0x01 num:128 Producer:2 from LID:0x001B
> TID:0x0000000000000059
> Mar 19 18:39:07 931288 [45007960] -> osm_report_notice: Reporting Generic
> Notice type:1 num:128 from LID:0x001B
> GID:0xfe80000000000000,0x0005ad00000281a7
> Mar 19 18:39:07 946406 [42803960] -> SUBNET UP
> Mar 19 18:39:08 073735 [41E02960] -> __osm_trap_rcv_process_request:
> Received Generic Notice type:0x01 num:128 Producer:2 from LID:0x0152
> TID:0x0000000000000020
> Mar 19 18:39:08 073811 [41E02960] -> __osm_trap_rcv_process_request: ERR
> 3804: Received trap 33 times consecutively
> Mar 19 18:39:08 400790 [43204960] -> osm_ucast_mgr_process: Min Hop Tables
> configured on all switches
> Mar 19 18:39:08 467925 [45A08960] -> SUBNET UP
> Mar 19 20:24:07 009911 [42803960] -> __osm_trap_rcv_process_request:
> Received Generic Notice type:0x01 num:128 Producer:2 from LID:0x0020
> TID:0x0000000000000020
> Mar 19 20:24:07 010153 [42803960] -> osm_report_notice: Reporting Generic
> Notice type:1 num:128 from LID:0x0020
> GID:0xfe80000000000000,0x0005ad00000281ad
> Mar 19 20:24:07 010966 [41401960] -> __osm_trap_rcv_process_request:
> Received Generic Notice type:0x01 num:128 Producer:2 from LID:0x0001
> TID:0x000000000000001a
> Mar 19 20:24:07 011064 [41401960] -> osm_report_notice: Reporting Generic
> Notice type:1 num:128 from LID:0x0001
> GID:0xfe80000000000000,0x0005ad0000027c6a
> Mar 19 20:24:07 390927 [43204960] -> osm_ucast_mgr_process: Min Hop Tables
> configured on all switches
> Mar 19 20:24:07 453747 [43204960] -> SUBNET UP
> Mar 19 20:24:07 839927 [45007960] -> osm_ucast_mgr_process: Min Hop Tables
> configured on all switches
> Mar 19 20:24:07 895694 [45A08960] -> SUBNET UP
> Mar 19 20:24:08 049066 [41E02960] -> __osm_trap_rcv_process_request:
> Received Generic Notice type:0x01 num:128 Producer:2 from LID:0x0001
> TID:0x000000000000001a
> Mar 19 20:24:08 049322 [41E02960] -> osm_report_notice: Reporting Generic
> Notice type:1 num:128 from LID:0x0001
> GID:0xfe80000000000000,0x0005ad0000027c6a
> Mar 19 20:24:08 433979 [42803960] -> osm_ucast_mgr_process: Min Hop Tables
> configured on all switches
> Mar 19 20:24:08 487950 [43204960] -> SUBNET UP
> Mar 19 20:26:28 608381 [42803960] -> __osm_trap_rcv_process_request:
> Received Generic Notice type:0x01 num:128 Producer:2 from LID:0x0020
> TID:0x0000000000000021
> Mar 19 20:26:28 608406 [44606960] -> __osm_trap_rcv_process_request:
> Received Generic Notice type:0x01 num:128 Producer:2 from LID:0x0001
> TID:0x000000000000001b
> Mar 19 20:26:28 608685 [42803960] -> osm_report_notice: Reporting Generic
> Notice type:1 num:128 from LID:0x0020
> GID:0xfe80000000000000,0x0005ad00000281ad
> Mar 19 20:26:28 608693 [44606960] -> osm_report_notice: Reporting Generic
> Notice type:1 num:128 from LID:0x0001
> GID:0xfe80000000000000,0x0005ad0000027c6a
> Mar 19 20:26:28 972140 [44606960] -> osm_ucast_mgr_process: Min Hop Tables
> configured on all switches
> Mar 19 20:26:29 028682 [43C05960] -> SUBNET UP
> Mar 19 20:26:29 399649 [43204960] -> osm_ucast_mgr_process: Min Hop Tables
> configured on all switches
> Mar 19 20:26:29 465737 [45007960] -> SUBNET UP
> Mar 19 21:30:38 775260 [45007960] -> __osm_trap_rcv_process_request:
> Received Generic Notice type:0x01 num:128 Producer:2 from LID:0x0146
> TID:0x000000000000002f
> Mar 19 21:30:38 775533 [45007960] -> osm_report_notice: Reporting Generic
> Notice type:1 num:128 from LID:0x0146
> GID:0xfe80000000000000,0x0005ad00000281b6
> Mar 19 21:30:38 777083 [45007960] -> __osm_trap_rcv_process_request:
> Received Generic Notice type:0x01 num:128 Producer:2 from LID:0x0143
> TID:0x0000000000000037
> Mar 19 21:30:38 777242 [45007960] -> osm_report_notice: Reporting Generic
> Notice type:1 num:128 from LID:0x0143
> GID:0xfe80000000000000,0x0005ad00000281b9
> Mar 19 21:30:39 144779 [43C05960] -> osm_ucast_mgr_process: Min Hop Tables
> configured on all switches
> Mar 19 21:30:39 200635 [43204960] -> SUBNET UP
> Mar 19 21:30:39 536003 [43C05960] -> osm_ucast_mgr_process: Min Hop Tables
> configured on all switches
> Mar 19 21:30:39 591216 [42803960] -> SUBNET UP
> Mar 20 14:06:48 971082 [41401960] -> __osm_trap_rcv_process_request:
> Received Generic Notice type:0x01 num:128 Producer:2 from LID:0x0152
> TID:0x0000000000000021
> Mar 20 14:06:48 971376 [41401960] -> osm_report_notice: Reporting Generic
> Notice type:1 num:128 from LID:0x0152
> GID:0xfe80000000000000,0x0005ad0000027c84
> Mar 20 14:06:49 346734 [42803960] -> osm_report_notice: Reporting Generic
> Notice type:3 num:65 from LID:0x0092
> GID:0xfe80000000000000,0x0005ad0000024bbb
> Mar 20 14:06:49 346761 [42803960] -> Removed port with
> GUID:0x0005ad0000024b27 LID range [0xAF,0xAF] of node:saguaro-23-0 HCA-1
> Mar 20 14:06:49 381394 [42803960] -> osm_ucast_mgr_process: Min Hop Tables
> configured on all switches
> Mar 20 14:06:49 440803 [43204960] -> SUBNET UP
> Mar 20 14:07:09 098449 [44606960] -> __osm_trap_rcv_process_request:
> Received Generic Notice type:0x01 num:128 Producer:2 from LID:0x0148
> TID:0x0000000000000048
> Mar 20 14:07:09 098708 [44606960] -> osm_report_notice: Reporting Generic
> Notice type:1 num:128 from LID:0x0148
> GID:0xfe80000000000000,0x0005ad00000281b3
> Mar 20 14:07:09 098733 [41E02960] -> __osm_trap_rcv_process_request:
> Received Generic Notice type:0x01 num:128 Producer:2 from LID:0x001B
> TID:0x000000000000005a
> Mar 20 14:07:09 098777 [41E02960] -> osm_report_notice: Reporting Generic
> Notice type:1 num:128 from LID:0x001B
> GID:0xfe80000000000000,0x0005ad00000281a7
> Mar 20 14:07:09 417844 [42803960] -> osm_report_notice: Reporting Generic
> Notice type:3 num:65 from LID:0x0092
> GID:0xfe80000000000000,0x0005ad0000024bbb
> Mar 20 14:07:09 417862 [42803960] -> Removed port with
> GUID:0x0005ad0000024e0b LID range [0xB3,0xB3] of node:saguaro-23-4 HCA-1
> Mar 20 14:07:09 417879 [42803960] -> osm_report_notice: Reporting Generic
> Notice type:3 num:65 from LID:0x0092
> GID:0xfe80000000000000,0x0005ad0000024bbb
> Mar 20 14:07:09 417885 [42803960] -> Removed port with
> GUID:0x0005ad000002510b LID range [0xB5,0xB5] of node:saguaro-23-6 HCA-1
> Mar 20 14:07:09 417902 [42803960] -> osm_report_notice: Reporting Generic
> Notice type:3 num:65 from LID:0x0092
> GID:0xfe80000000000000,0x0005ad0000024bbb
> Mar 20 14:07:09 417907 [42803960] -> Removed port with
> GUID:0x0005ad000002511b LID range [0xA6,0xA6] of node:saguaro-22-1 HCA-1
> Mar 20 14:07:09 417924 [42803960] -> osm_report_notice: Reporting Generic
> Notice type:3 num:65 from LID:0x0092
> GID:0xfe80000000000000,0x0005ad0000024bbb
> Mar 20 14:07:09 417929 [42803960] -> Removed port with
> GUID:0x0005ad0000025043 LID range [0xB4,0xB4] of node:saguaro-23-5 HCA-1
> Mar 20 14:07:09 417945 [42803960] -> osm_report_notice: Reporting Generic
> Notice type:3 num:65 from LID:0x0092
> GID:0xfe80000000000000,0x0005ad0000024bbb
> Mar 20 14:07:09 417951 [42803960] -> Removed port with
> GUID:0x0005ad0000024d47 LID range [0xB6,0xB6] of node:saguaro-23-7 HCA-1
> Mar 20 14:07:09 417967 [42803960] -> osm_report_notice: Reporting Generic
> Notice type:3 num:65 from LID:0x0092
> GID:0xfe80000000000000,0x0005ad0000024bbb
> Mar 20 14:07:09 417973 [42803960] -> Removed port with
> GUID:0x0005ad0000024d6b LID range [0xB8,0xB8] of node:saguaro-23-9 HCA-1
> Mar 20 14:07:09 417989 [42803960] -> osm_report_notice: Reporting Generic
> Notice type:3 num:65 from LID:0x0092
> GID:0xfe80000000000000,0x0005ad0000024bbb
> Mar 20 14:07:09 417994 [42803960] -> Removed port with
> GUID:0x0005ad0000024977 LID range [0xA9,0xA9] of node:saguaro-22-4 HCA-1
> Mar 20 14:07:09 418131 [42803960] -> osm_report_notice: Reporting Generic
> Notice type:3 num:65 from LID:0x0092
> GID:0xfe80000000000000,0x0005ad0000024bbb
> Mar 20 14:07:09 418137 [42803960] -> Removed port with
> GUID:0x0005ad0000027c84 LID range [0x152,0x152] of node:Topspin Switch TS120
> Mar 20 14:07:09 418168 [42803960] -> osm_report_notice: Reporting Generic
> Notice type:3 num:65 from LID:0x0092
> GID:0xfe80000000000000,0x0005ad0000024bbb
> Mar 20 14:07:09 418173 [42803960] -> Removed port with
> GUID:0x0005ad0000024d8b LID range [0xB7,0xB7] of node:saguaro-23-8 HCA-1
> Mar 20 14:07:09 418188 [42803960] -> osm_report_notice: Reporting Generic
> Notice type:3 num:65 from LID:0x0092
> GID:0xfe80000000000000,0x0005ad0000024bbb
> Mar 20 14:07:09 418193 [42803960] -> Removed port with
> GUID:0x0005ad000002498f LID range [0xA8,0xA8] of node:saguaro-22-3 HCA-1
> Mar 20 14:07:09 418207 [42803960] -> osm_report_notice: Reporting Generic
> Notice type:3 num:65 from LID:0x0092
> GID:0xfe80000000000000,0x0005ad0000024bbb
> Mar 20 14:07:09 418212 [42803960] -> Removed port with
> GUID:0x0005ad0000024c9b LID range [0xA7,0xA7] of node:saguaro-22-2 HCA-1
> Mar 20 14:07:09 418227 [42803960] -> osm_report_notice: Reporting Generic
> Notice type:3 num:65 from LID:0x0092
> GID:0xfe80000000000000,0x0005ad0000024bbb
> Mar 20 14:07:09 418232 [42803960] -> Removed port with
> GUID:0x0005ad0000024da7 LID range [0xB0,0xB0] of node:saguaro-23-1 HCA-1
> Mar 20 14:07:09 418248 [42803960] -> osm_report_notice: Reporting Generic
> Notice type:3 num:65 from LID:0x0092
> GID:0xfe80000000000000,0x0005ad0000024bbb
> Mar 20 14:07:09 418253 [42803960] -> Removed port with
> GUID:0x0005ad0000024cbb LID range [0xB2,0xB2] of node:saguaro-23-3 HCA-1
> Mar 20 14:07:09 418285 [42803960] -> osm_report_notice: Reporting Generic
> Notice type:3 num:65 from LID:0x0092
> GID:0xfe80000000000000,0x0005ad0000024bbb
> Mar 20 14:07:09 418290 [42803960] -> Removed port with
> GUID:0x0005ad00000249d3 LID range [0xB1,0xB1] of node:saguaro-23-2 HCA-1
> Mar 20 14:07:09 418306 [42803960] -> osm_report_notice: Reporting Generic
> Notice type:3 num:65 from LID:0x0092
> GID:0xfe80000000000000,0x0005ad0000024bbb
> Mar 20 14:07:09 418362 [42803960] -> Removed port with
> GUID:0x0005ad0000024feb LID range [0x153,0x153] of node:saguaro-22-5 HCA-1
> Mar 20 14:07:09 418378 [42803960] -> osm_report_notice: Reporting Generic
> Notice type:3 num:65 from LID:0x0092
> GID:0xfe80000000000000,0x0005ad0000024bbb
> Mar 20 14:07:09 418383 [42803960] -> Removed port with
> GUID:0x0005ad0000024afb LID range [0xA5,0xA5] of node:saguaro-22-0 HCA-1
> Mar 20 14:07:09 451317 [42803960] -> osm_ucast_mgr_process: Min Hop Tables
> configured on all switches
> Mar 20 14:07:09 502755 [41401960] -> SUBNET UP
> Mar 20 14:07:09 902534 [41E02960] -> osm_ucast_mgr_process: Min Hop Tables
> configured on all switches
> Mar 20 14:07:09 955229 [45A08960] -> SUBNET UP
> Mar 20 14:08:03 850926 [45A08960] -> __osm_trap_rcv_process_request:
> Received Generic Notice type:0x01 num:128 Producer:2 from LID:0x0148
> TID:0x0000000000000049
> Mar 20 14:08:03 851134 [45A08960] -> osm_report_notice: Reporting Generic
> Notice type:1 num:128 from LID:0x0148
> GID:0xfe80000000000000,0x0005ad00000281b3
> Mar 20 14:08:03 856880 [43204960] -> __osm_trap_rcv_process_request:
> Received Generic Notice type:0x01 num:128 Producer:2 from LID:0x0148
> TID:0x000000000000004a
> Mar 20 14:08:03 856955 [43204960] -> osm_report_notice: Reporting Generic
> Notice type:1 num:128 from LID:0x0148
> GID:0xfe80000000000000,0x0005ad00000281b3
> Mar 20 14:08:03 866819 [42803960] -> __osm_trap_rcv_process_request:
> Received Generic Notice type:0x01 num:128 Producer:2 from LID:0x001B
> TID:0x000000000000005b
> Mar 20 14:08:03 866977 [42803960] -> osm_report_notice: Reporting Generic
> Notice type:1 num:128 from LID:0x001B
> GID:0xfe80000000000000,0x0005ad00000281a7
> Mar 20 14:08:03 963024 [45A08960] -> __osm_trap_rcv_process_request:
> Received Generic Notice type:0x01 num:128 Producer:2 from LID:0x001B
> TID:0x000000000000005c
> Mar 20 14:08:03 963130 [45A08960] -> osm_report_notice: Reporting Generic
> Notice type:1 num:128 from LID:0x001B
> GID:0xfe80000000000000,0x0005ad00000281a7
> Mar 20 14:08:04 106856 [43C05960] -> __osm_trap_rcv_process_request:
> Received Generic Notice type:0x01 num:128 Producer:2 from LID:0x001B
> TID:0x000000000000005d
> Mar 20 14:08:04 106995 [43C05960] -> osm_report_notice: Reporting Generic
> Notice type:1 num:128 from LID:0x001B
> GID:0xfe80000000000000,0x0005ad00000281a7
> Mar 20 14:08:04 193747 [44606960] -> osm_report_notice: Reporting Generic
> Notice type:3 num:64 from LID:0x0092
> GID:0xfe80000000000000,0x0005ad0000024bbb
> Mar 20 14:08:04 193766 [44606960] -> Discovered new port with
> GUID:0x0005ad0000027c84 LID range [0x152,0x152] of node:Topspin Switch TS120
> Mar 20 14:08:04 193771 [44606960] -> osm_report_notice: Reporting Generic
> Notice type:3 num:64 from LID:0x0092
> GID:0xfe80000000000000,0x0005ad0000024bbb
> Mar 20 14:08:04 193777 [44606960] -> Discovered new port with
> GUID:0x0005ad0000024da7 LID range [0xB0,0xB0] of node:saguaro-23-1 HCA-1
> Mar 20 14:08:04 193781 [44606960] -> osm_report_notice: Reporting Generic
> Notice type:3 num:64 from LID:0x0092
> GID:0xfe80000000000000,0x0005ad0000024bbb
> Mar 20 14:08:04 193786 [44606960] -> Discovered new port with
> GUID:0x0005ad00000249d3 LID range [0xB1,0xB1] of node:saguaro-23-2 HCA-1
> Mar 20 14:08:04 193790 [44606960] -> osm_report_notice: Reporting Generic
> Notice type:3 num:64 from LID:0x0092
> GID:0xfe80000000000000,0x0005ad0000024bbb
> Mar 20 14:08:04 193795 [44606960] -> Discovered new port with
> GUID:0x0005ad0000024cbb LID range [0xB2,0xB2] of node:saguaro-23-3 HCA-1
> Mar 20 14:08:04 193799 [44606960] -> osm_report_notice: Reporting Generic
> Notice type:3 num:64 from LID:0x0092
> GID:0xfe80000000000000,0x0005ad0000024bbb
> Mar 20 14:08:04 193804 [44606960] -> Discovered new port with
> GUID:0x0005ad0000024e0b LID range [0xB3,0xB3] of node:saguaro-23-4 HCA-1
> Mar 20 14:08:04 193808 [44606960] -> osm_report_notice: Reporting Generic
> Notice type:3 num:64 from LID:0x0092
> GID:0xfe80000000000000,0x0005ad0000024bbb
> Mar 20 14:08:04 193813 [44606960] -> Discovered new port with
> GUID:0x0005ad0000025043 LID range [0xB4,0xB4] of node:saguaro-23-5 HCA-1
> Mar 20 14:08:04 193817 [44606960] -> osm_report_notice: Reporting Generic
> Notice type:3 num:64 from LID:0x0092
> GID:0xfe80000000000000,0x0005ad0000024bbb
> Mar 20 14:08:04 193822 [44606960] -> Discovered new port with
> GUID:0x0005ad000002510b LID range [0xB5,0xB5] of node:saguaro-23-6 HCA-1
> Mar 20 14:08:04 193826 [44606960] -> osm_report_notice: Reporting Generic
> Notice type:3 num:64 from LID:0x0092
> GID:0xfe80000000000000,0x0005ad0000024bbb
> Mar 20 14:08:04 193830 [44606960] -> Discovered new port with
> GUID:0x0005ad0000024d8b LID range [0xB7,0xB7] of node:saguaro-23-8 HCA-1
> Mar 20 14:08:04 193834 [44606960] -> osm_report_notice: Reporting Generic
> Notice type:3 num:64 from LID:0x0092
> GID:0xfe80000000000000,0x0005ad0000024bbb
> Mar 20 14:08:04 193839 [44606960] -> Discovered new port with
> GUID:0x0005ad0000024d47 LID range [0xB6,0xB6] of node:saguaro-23-7 HCA-1
> Mar 20 14:08:04 193843 [44606960] -> osm_report_notice: Reporting Generic
> Notice type:3 num:64 from LID:0x0092
> GID:0xfe80000000000000,0x0005ad0000024bbb
> Mar 20 14:08:04 193848 [44606960] -> Discovered new port with
> GUID:0x0005ad0000024d6b LID range [0xB8,0xB8] of node:saguaro-23-9 HCA-1
> Mar 20 14:08:04 193852 [44606960] -> osm_report_notice: Reporting Generic
> Notice type:3 num:64 from LID:0x0092
> GID:0xfe80000000000000,0x0005ad0000024bbb
> Mar 20 14:08:04 193857 [44606960] -> Discovered new port with
> GUID:0x0005ad0000024afb LID range [0xA5,0xA5] of node:saguaro-22-0 HCA-1
> Mar 20 14:08:04 193861 [44606960] -> osm_report_notice: Reporting Generic
> Notice type:3 num:64 from LID:0x0092
> GID:0xfe80000000000000,0x0005ad0000024bbb
> Mar 20 14:08:04 193866 [44606960] -> Discovered new port with
> GUID:0x0005ad000002511b LID range [0xA6,0xA6] of node:saguaro-22-1 HCA-1
> Mar 20 14:08:04 193870 [44606960] -> osm_report_notice: Reporting Generic
> Notice type:3 num:64 from LID:0x0092
> GID:0xfe80000000000000,0x0005ad0000024bbb
> Mar 20 14:08:04 193874 [44606960] -> Discovered new port with
> GUID:0x0005ad0000024c9b LID range [0xA7,0xA7] of node:saguaro-22-2 HCA-1
> Mar 20 14:08:04 193878 [44606960] -> osm_report_notice: Reporting Generic
> Notice type:3 num:64 from LID:0x0092
> GID:0xfe80000000000000,0x0005ad0000024bbb
> Mar 20 14:08:04 193883 [44606960] -> Discovered new port with
> GUID:0x0005ad000002498f LID range [0xA8,0xA8] of node:saguaro-22-3 HCA-1
> Mar 20 14:08:04 193938 [44606960] -> osm_report_notice: Reporting Generic
> Notice type:3 num:64 from LID:0x0092
> GID:0xfe80000000000000,0x0005ad0000024bbb
> Mar 20 14:08:04 193944 [44606960] -> Discovered new port with
> GUID:0x0005ad0000024977 LID range [0xA9,0xA9] of node:saguaro-22-4 HCA-1
> Mar 20 14:08:04 193948 [44606960] -> osm_report_notice: Reporting Generic
> Notice type:3 num:64 from LID:0x0092
> GID:0xfe80000000000000,0x0005ad0000024bbb
> Mar 20 14:08:04 193953 [44606960] -> Discovered new port with
> GUID:0x0005ad0000024feb LID range [0x153,0x153] of node:saguaro-22-5 HCA-1
> Mar 20 14:08:04 224695 [44606960] -> osm_ucast_mgr_process: Min Hop Tables
> configured on all switches
> Mar 20 14:08:04 281046 [4780B960] -> __osm_sm_mad_ctrl_rcv_callback: ERR
> 3111: Error status = 0x1C00
> Mar 20 14:08:04 281106 [4780B960] -> SMP dump:
> base_ver................0x1
> mgmt_class..............0x81
> class_ver...............0x1
> method..................0x81 (SubnGetResp)
> D bit...................0x1
> status..................0x1C00
> hop_ptr.................0x0
> hop_count...............0x4
> trans_id................0x61eec
> attr_id.................0x15 (PortInfo)
> resv....................0x0
> attr_mod................0x13
> m_key...................0x0000000000000000
> dr_slid.................0xFFFF
> dr_dlid.................0xFFFF
>
> Initial path: [0][1][17][2][4]
> Return path: [0][9][14][E][1]
> Reserved: [0][0][0][0][0][0][0]
>
> 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
>
> 00 00 00 00 00 00 00 00 00 00 00 00 01 03 03 02
>
> 14 52 00 11 40 40 00 08 08 04 F2 40 00 00 00 00
>
> 00 00 00 00 00 88 00 00 00 00 00 00 00 00 00 00
>
> Mar 20 14:08:04 281154 [41401960] -> osm_pi_rcv_process_set: Received error
> status 0x1c for SetResp() during ACTIVE transition
> Mar 20 14:08:04 281159 [41401960] -> PortInfo dump:
> port number.............0x13
> node_guid...............0x0005ad00000281a7
> port_guid...............0x0005ad00000281a7
> m_key...................0x0000000000000000
> subnet_prefix...........0x0000000000000000
> base_lid................0x0
> master_sm_base_lid......0x0
> capability_mask.........0x0
> diag_code...............0x0
> m_key_lease_period......0x0
> local_port_num..........0x1
> link_width_enabled......0x3
> link_width_supported....0x3
> link_width_active.......0x2
> link_speed_supported....0x1
> port_state..............ACTIVE
> state_info2.............0x52
> m_key_protect_bits......0x0
> lmc.....................0x0
> link_speed..............0x11
> mtu_smsl................0x40
> vl_cap_init_type........0x40
> vl_high_limit...........0x0
> vl_arb_high_cap.........0x8
> vl_arb_low_cap..........0x8
> init_rep_mtu_cap........0x4
> vl_stall_life...........0xF2
> vl_enforce..............0x40
> m_key_violations........0x0
> p_key_violations........0x0
> q_key_violations........0x0
> guid_cap................0x0
> client_reregister.......0x0
> subnet_timeout..........0x0
> resp_time_value.........0x0
> error_threshold.........0x88
> Mar 20 14:08:04 281172 [41401960] -> Capabilities Mask:
> Mar 20 14:08:04 281187 [4780B960] -> __osm_sm_mad_ctrl_rcv_callback: ERR
> 3111: Error status = 0x1C00
> Mar 20 14:08:04 281213 [4780B960] -> SMP dump:
> base_ver................0x1
> mgmt_class..............0x81
> class_ver...............0x1
> method..................0x81 (SubnGetResp)
> D bit...................0x1
> status..................0x1C00
> hop_ptr.................0x0
> hop_count...............0x4
> trans_id................0x61eed
> attr_id.................0x15 (PortInfo)
> resv....................0x0
> attr_mod................0x17
> m_key...................0x0000000000000000
> dr_slid.................0xFFFF
> dr_dlid.................0xFFFF
>
> Initial path: [0][1][17][2][4]
> Return path: [0][9][14][E][1]
> Reserved: [0][0][0][0][0][0][0]
>
> 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
>
> 00 00 00 00 00 00 00 00 00 00 00 00 01 03 03 02
>
> 14 52 00 11 40 40 00 08 08 04 F2 40 00 00 00 00
>
> 00 00 00 00 00 88 00 00 00 00 00 00 00 00 00 00
>
> Mar 20 14:08:04 281279 [4780B960] -> __osm_sm_mad_ctrl_rcv_callback: ERR
> 3111: Error status = 0x1C00
> Mar 20 14:08:04 281316 [4780B960] -> SMP dump:
> base_ver................0x1
> mgmt_class..............0x81
> class_ver...............0x1
> method..................0x81 (SubnGetResp)
> D bit...................0x1
> status..................0x1C00
> hop_ptr.................0x0
> hop_count...............0x4
> trans_id................0x61eee
> attr_id.................0x15 (PortInfo)
> resv....................0x0
> attr_mod................0x18
> m_key...................0x0000000000000000
> dr_slid.................0xFFFF
> dr_dlid.................0xFFFF
>
> Initial path: [0][1][17][2][4]
> Return path: [0][9][14][E][1]
> Reserved: [0][0][0][0][0][0][0]
>
> 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
>
> 00 00 00 00 00 00 00 00 00 00 00 00 01 03 03 02
>
> 14 52 00 11 40 40 00 08 08 04 F2 40 00 00 00 00
>
> 00 00 00 00 00 88 00 00 00 00 00 00 00 00 00 00
>
> Mar 20 14:08:04 281392 [4780B960] -> __osm_sm_mad_ctrl_rcv_callback: ERR
> 3111: Error status = 0x1C00
> Mar 20 14:08:04 281416 [4780B960] -> SMP dump:
> base_ver................0x1
> mgmt_class..............0x81
> class_ver...............0x1
> method..................0x81 (SubnGetResp)
> D bit...................0x1
> status..................0x1C00
> hop_ptr.................0x0
> hop_count...............0x4
> trans_id................0x61eef
> attr_id.................0x15 (PortInfo)
> resv....................0x0
> attr_mod................0x16
> m_key...................0x0000000000000000
> dr_slid.................0xFFFF
> dr_dlid.................0xFFFF
>
> Initial path: [0][1][11][1][6]
> Return path: [0][9][18][D][3]
> Reserved: [0][0][0][0][0][0][0]
>
> 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
>
> 00 00 00 00 00 00 00 00 00 00 00 00 03 03 03 02
>
> 14 52 00 11 40 40 00 08 08 04 F2 40 00 00 00 00
>
> 00 00 00 00 00 88 00 00 00 00 00 00 00 00 00 00
>
> Mar 20 14:08:04 281515 [44606960] -> osm_pi_rcv_process_set: Received error
> status 0x1c for SetResp() during ACTIVE transition
> Mar 20 14:08:04 281522 [4780B960] -> __osm_sm_mad_ctrl_rcv_callback: ERR
> 3111: Error status = 0x1C00
> Mar 20 14:08:04 281542 [44606960] -> PortInfo dump:
> port number.............0x17
> node_guid...............0x0005ad00000281a7
> port_guid...............0x0005ad00000281a7
> m_key...................0x0000000000000000
> subnet_prefix...........0x0000000000000000
> base_lid................0x0
> master_sm_base_lid......0x0
> capability_mask.........0x0
> diag_code...............0x0
> m_key_lease_period......0x0
> local_port_num..........0x1
> link_width_enabled......0x3
> link_width_supported....0x3
> link_width_active.......0x2
> link_speed_supported....0x1
> port_state..............ACTIVE
> state_info2.............0x52
> m_key_protect_bits......0x0
> lmc.....................0x0
> link_speed..............0x11
> mtu_smsl................0x40
> vl_cap_init_type........0x40
> vl_high_limit...........0x0
> vl_arb_high_cap.........0x8
> vl_arb_low_cap..........0x8
> init_rep_mtu_cap........0x4
> vl_stall_life...........0xF2
> vl_enforce..............0x40
> m_key_violations........0x0
> p_key_violations........0x0
> q_key_violations........0x0
> guid_cap................0x0
> client_reregister.......0x0
> subnet_timeout..........0x0
> resp_time_value.........0x0
> error_threshold.........0x88
> Mar 20 14:08:04 281553 [44606960] -> Capabilities Mask:
> Mar 20 14:08:04 281561 [4780B960] -> SMP dump:
> base_ver................0x1
> mgmt_class..............0x81
> class_ver...............0x1
> method..................0x81 (SubnGetResp)
> D bit...................0x1
> status..................0x1C00
> hop_ptr.................0x0
> hop_count...............0x4
> trans_id................0x61ef0
> attr_id.................0x15 (PortInfo)
> resv....................0x0
> attr_mod................0x17
> m_key...................0x0000000000000000
> dr_slid.................0xFFFF
> dr_dlid.................0xFFFF
>
> Initial path: [0][1][11][1][6]
> Return path: [0][9][18][D][3]
> Reserved: [0][0][0][0][0][0][0]
>
> 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
>
> 00 00 00 00 00 00 00 00 00 00 00 00 03 03 03 02
>
> 14 52 00 11 40 40 00 08 08 04 F2 40 00 00 00 00
>
> 00 00 00 00 00 88 00 00 00 00 00 00 00 00 00 00
>
> Mar 20 14:08:04 281572 [44606960] -> osm_pi_rcv_process_set: Received error
> status 0x1c for SetResp() during ACTIVE transition
> Mar 20 14:08:04 281590 [44606960] -> PortInfo dump:
> port number.............0x18
> node_guid...............0x0005ad00000281a7
> port_guid...............0x0005ad00000281a7
> m_key...................0x0000000000000000
> subnet_prefix...........0x0000000000000000
> base_lid................0x0
> master_sm_base_lid......0x0
> capability_mask.........0x0
> diag_code...............0x0
> m_key_lease_period......0x0
> local_port_num..........0x1
> link_width_enabled......0x3
> link_width_supported....0x3
> link_width_active.......0x2
> link_speed_supported....0x1
> port_state..............ACTIVE
> state_info2.............0x52
> m_key_protect_bits......0x0
> lmc.....................0x0
> link_speed..............0x11
> mtu_smsl................0x40
> vl_cap_init_type........0x40
> vl_high_limit...........0x0
> vl_arb_high_cap.........0x8
> vl_arb_low_cap..........0x8
> init_rep_mtu_cap........0x4
> vl_stall_life...........0xF2
> vl_enforce..............0x40
> m_key_violations........0x0
> p_key_violations........0x0
> q_key_violations........0x0
> guid_cap................0x0
> client_reregister.......0x0
> subnet_timeout..........0x0
> resp_time_value.........0x0
> error_threshold.........0x88
> Mar 20 14:08:04 281600 [44606960] -> Capabilities Mask:
> Mar 20 14:08:04 281623 [44606960] -> osm_pi_rcv_process_set: Received error
> status 0x1c for SetResp() during ACTIVE transition
> Mar 20 14:08:04 281626 [44606960] -> PortInfo dump:
> port number.............0x16
> node_guid...............0x0005ad00000281b3
> port_guid...............0x0005ad00000281b3
> m_key...................0x0000000000000000
> subnet_prefix...........0x0000000000000000
> base_lid................0x0
> master_sm_base_lid......0x0
> capability_mask.........0x0
> diag_code...............0x0
> m_key_lease_period......0x0
> local_port_num..........0x3
> link_width_enabled......0x3
> link_width_supported....0x3
> link_width_active.......0x2
> link_speed_supported....0x1
> port_state..............ACTIVE
> state_info2.............0x52
> m_key_protect_bits......0x0
> lmc.....................0x0
> link_speed..............0x11
> mtu_smsl................0x40
> vl_cap_init_type........0x40
> vl_high_limit...........0x0
> vl_arb_high_cap.........0x8
> vl_arb_low_cap..........0x8
> init_rep_mtu_cap........0x4
> vl_stall_life...........0xF2
> vl_enforce..............0x40
> m_key_violations........0x0
> p_key_violations........0x0
> q_key_violations........0x0
> guid_cap................0x0
> client_reregister.......0x0
> subnet_timeout..........0x0
> resp_time_value.........0x0
> error_threshold.........0x88
> Mar 20 14:08:04 281635 [44606960] -> Capabilities Mask:
> Mar 20 14:08:04 281637 [4780B960] -> __osm_sm_mad_ctrl_rcv_callback: ERR
> 3111: Error status = 0x1C00
> Mar 20 14:08:04 281652 [44606960] -> osm_pi_rcv_process_set: Received error
> status 0x1c for SetResp() during ACTIVE transition
> Mar 20 14:08:04 281663 [44606960] -> PortInfo dump:
> port number.............0x17
> node_guid...............0x0005ad00000281b3
> port_guid...............0x0005ad00000281b3
> m_key...................0x0000000000000000
> subnet_prefix...........0x0000000000000000
> base_lid................0x0
> master_sm_base_lid......0x0
> capability_mask.........0x0
> diag_code...............0x0
> m_key_lease_period......0x0
> local_port_num..........0x3
> link_width_enabled......0x3
> link_width_supported....0x3
> link_width_active.......0x2
> link_speed_supported....0x1
> port_state..............ACTIVE
> state_info2.............0x52
> m_key_protect_bits......0x0
> lmc.....................0x0
> link_speed..............0x11
> mtu_smsl................0x40
> vl_cap_init_type........0x40
> vl_high_limit...........0x0
> vl_arb_high_cap.........0x8
> vl_arb_low_cap..........0x8
> init_rep_mtu_cap........0x4
> vl_stall_life...........0xF2
> vl_enforce..............0x40
> m_key_violations........0x0
> p_key_violations........0x0
> q_key_violations........0x0
> guid_cap................0x0
> client_reregister.......0x0
> subnet_timeout..........0x0
> resp_time_value.........0x0
> error_threshold.........0x88
> Mar 20 14:08:04 281673 [44606960] -> Capabilities Mask:
> Mar 20 14:08:04 281675 [4780B960] -> SMP dump:
> base_ver................0x1
> mgmt_class..............0x81
> class_ver...............0x1
> method..................0x81 (SubnGetResp)
> D bit...................0x1
> status..................0x1C00
> hop_ptr.................0x0
> hop_count...............0x4
> trans_id................0x61ef1
> attr_id.................0x15 (PortInfo)
> resv....................0x0
> attr_mod................0x18
> m_key...................0x0000000000000000
> dr_slid.................0xFFFF
> dr_dlid.................0xFFFF
>
> Initial path: [0][1][11][1][6]
> Return path: [0][9][18][D][3]
> Reserved: [0][0][0][0][0][0][0]
>
> 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
>
> 00 00 00 00 00 00 00 00 00 00 00 00 03 03 03 02
>
> 14 52 00 11 40 40 00 08 08 04 F2 40 00 00 00 00
>
> 00 00 00 00 00 88 00 00 00 00 00 00 00 00 00 00
>
> Mar 20 14:08:04 281721 [41E02960] -> osm_pi_rcv_process_set: Received error
> status 0x1c for SetResp() during ACTIVE transition
> Mar 20 14:08:04 281726 [41E02960] -> PortInfo dump:
> port number.............0x18
> node_guid...............0x0005ad00000281b3
> port_guid...............0x0005ad00000281b3
> m_key...................0x0000000000000000
> subnet_prefix...........0x0000000000000000
> base_lid................0x0
> master_sm_base_lid......0x0
> capability_mask.........0x0
> diag_code...............0x0
> m_key_lease_period......0x0
> local_port_num..........0x3
> link_width_enabled......0x3
> link_width_supported....0x3
> link_width_active.......0x2
> link_speed_supported....0x1
> port_state..............ACTIVE
> state_info2.............0x52
> m_key_protect_bits......0x0
> lmc.....................0x0
> link_speed..............0x11
> mtu_smsl................0x40
> vl_cap_init_type........0x40
> vl_high_limit...........0x0
> vl_arb_high_cap.........0x8
> vl_arb_low_cap..........0x8
> init_rep_mtu_cap........0x4
> vl_stall_life...........0xF2
> vl_enforce..............0x40
> m_key_violations........0x0
> p_key_violations........0x0
> q_key_violations........0x0
> guid_cap................0x0
> client_reregister.......0x0
> subnet_timeout..........0x0
> resp_time_value.........0x0
> error_threshold.........0x88
> Mar 20 14:08:04 281736 [41E02960] -> Capabilities Mask:
> Mar 20 14:08:04 287136 [44606960] -> SUBNET UP
> Mar 20 14:08:04 711595 [43C05960] -> osm_ucast_mgr_process: Min Hop Tables
> configured on all switches
> Mar 20 14:08:04 766488 [45A08960] -> SUBNET UP
> Mar 20 14:08:19 947200 [43204960] -> __osm_trap_rcv_process_request:
> Received Generic Notice type:0x01 num:128 Producer:2 from LID:0x0152
> TID:0x0000000000000000
> Mar 20 14:08:19 947479 [43204960] -> osm_report_notice: Reporting Generic
> Notice type:1 num:128 from LID:0x0152
> GID:0xfe80000000000000,0x0005ad0000027c84
> Mar 20 14:08:20 086909 [41E02960] -> __osm_trap_rcv_process_request:
> Received Generic Notice type:0x01 num:128 Producer:2 from LID:0x0152
> TID:0x0000000000000001
> Mar 20 14:08:20 087084 [41E02960] -> osm_report_notice: Reporting Generic
> Notice type:1 num:128 from LID:0x0152
> GID:0xfe80000000000000,0x0005ad0000027c84
> Mar 20 14:08:20 108865 [41401960] -> __osm_trap_rcv_process_request:
> Received Generic Notice type:0x01 num:128 Producer:2 from LID:0x0152
> TID:0x0000000000000002
> Mar 20 14:08:20 109210 [41401960] -> osm_report_notice: Reporting Generic
> Notice type:1 num:128 from LID:0x0152
> GID:0xfe80000000000000,0x0005ad0000027c84
> Mar 20 14:08:20 109996 [41E02960] -> __osm_trap_rcv_process_request:
> Received Generic Notice type:0x01 num:128 Producer:2 from LID:0x0152
> TID:0x0000000000000003
> Mar 20 14:08:20 110407 [41E02960] -> osm_report_notice: Reporting Generic
> Notice type:1 num:128 from LID:0x0152
> GID:0xfe80000000000000,0x0005ad0000027c84
> Mar 20 14:08:20 222523 [45A08960] -> __osm_trap_rcv_process_request:
> Received Generic Notice type:0x01 num:128 Producer:2 from LID:0x0152
> TID:0x0000000000000004
> Mar 20 14:08:20 222613 [45A08960] -> osm_report_notice: Reporting Generic
> Notice type:1 num:128 from LID:0x0152
> GID:0xfe80000000000000,0x0005ad0000027c84
> Mar 20 14:08:20 404596 [41401960] -> __osm_trap_rcv_process_request:
> Received Generic Notice type:0x01 num:128 Producer:2 from LID:0x0152
> TID:0x0000000000000005
> Mar 20 14:08:20 404698 [41401960] -> osm_report_notice: Reporting Generic
> Notice type:1 num:128 from LID:0x0152
> GID:0xfe80000000000000,0x0005ad0000027c84
> Mar 20 14:08:20 476804 [45007960] -> __osm_trap_rcv_process_request:
> Received Generic Notice type:0x01 num:128 Producer:2 from LID:0x0152
> TID:0x0000000000000006
> Mar 20 14:08:20 476897 [45007960] -> osm_report_notice: Reporting Generic
> Notice type:1 num:128 from LID:0x0152
> GID:0xfe80000000000000,0x0005ad0000027c84
> Mar 20 14:08:20 572434 [44606960] -> __osm_trap_rcv_process_request:
> Received Generic Notice type:0x01 num:128 Producer:2 from LID:0x0152
> TID:0x0000000000000007
> Mar 20 14:08:20 572520 [44606960] -> osm_report_notice: Reporting Generic
> Notice type:1 num:128 from LID:0x0152
> GID:0xfe80000000000000,0x0005ad0000027c84
> Mar 20 14:08:20 621715 [42803960] -> osm_report_notice: Reporting Generic
> Notice type:3 num:65 from LID:0x0092
> GID:0xfe80000000000000,0x0005ad0000024bbb
> Mar 20 14:08:20 621726 [42803960] -> Removed port with
> GUID:0x0005ad00000249d3 LID range [0xB1,0xB1] of node:saguaro-23-2 HCA-1
> Mar 20 14:08:20 656232 [42803960] -> osm_ucast_mgr_process: Min Hop Tables
> configured on all switches
> Mar 20 14:08:20 698700 [44606960] -> __osm_trap_rcv_process_request:
> Received Generic Notice type:0x01 num:128 Producer:2 from LID:0x0152
> TID:0x0000000000000008
> Mar 20 14:08:20 698794 [44606960] -> osm_report_notice: Reporting Generic
> Notice type:1 num:128 from LID:0x0152
> GID:0xfe80000000000000,0x0005ad0000027c84
> Mar 20 14:08:20 708598 [41401960] -> __osm_trap_rcv_process_request:
> Received Generic Notice type:0x01 num:128 Producer:2 from LID:0x0152
> TID:0x0000000000000009
> Mar 20 14:08:20 708698 [41401960] -> osm_report_notice: Reporting Generic
> Notice type:1 num:128 from LID:0x0152
> GID:0xfe80000000000000,0x0005ad0000027c84
> Mar 20 14:08:20 713653 [45007960] -> SUBNET UP
> Mar 20 14:08:20 730554 [44606960] -> __osm_trap_rcv_process_request:
> Received Generic Notice type:0x01 num:128 Producer:2 from LID:0x0152
> TID:0x000000000000000a
> Mar 20 14:08:20 730697 [44606960] -> osm_report_notice: Reporting Generic
> Notice type:1 num:128 from LID:0x0152
> GID:0xfe80000000000000,0x0005ad0000027c84
> Mar 20 14:08:20 754139 [45007960] -> __osm_trap_rcv_process_request:
> Received Generic Notice type:0x01 num:128 Producer:2 from LID:0x0152
> TID:0x000000000000000b
> Mar 20 14:08:20 754251 [45007960] -> __osm_trap_rcv_process_request: ERR
> 3804: Received trap 11 times consecutively
> Mar 20 14:08:20 947339 [41401960] -> __osm_trap_rcv_process_request:
> Received Generic Notice type:0x01 num:128 Producer:2 from LID:0x0152
> TID:0x000000000000000c
> Mar 20 14:08:20 947426 [41401960] -> __osm_trap_rcv_process_request: ERR
> 3804: Received trap 12 times consecutively
> Mar 20 14:08:20 975965 [45A08960] -> __osm_trap_rcv_process_request:
> Received Generic Notice type:0x01 num:128 Producer:2 from LID:0x0152
> TID:0x000000000000000d
> Mar 20 14:08:20 976024 [45A08960] -> __osm_trap_rcv_process_request: ERR
> 3804: Received trap 13 times consecutively
> Mar 20 14:08:20 997569 [43C05960] -> __osm_trap_rcv_process_request:
> Received Generic Notice type:0x01 num:128 Producer:2 from LID:0x0152
> TID:0x000000000000000e
> Mar 20 14:08:20 997648 [43C05960] -> __osm_trap_rcv_process_request: ERR
> 3804: Received trap 14 times consecutively
> Mar 20 14:08:21 019465 [44606960] -> __osm_trap_rcv_process_request:
> Received Generic Notice type:0x01 num:128 Producer:2 from LID:0x0152
> TID:0x000000000000000f
> Mar 20 14:08:21 019512 [44606960] -> __osm_trap_rcv_process_request: ERR
> 3804: Received trap 15 times consecutively
> Mar 20 14:08:21 064967 [43204960] -> __osm_trap_rcv_process_request:
> Received Generic Notice type:0x01 num:128 Producer:2 from LID:0x0152
> TID:0x0000000000000010
> Mar 20 14:08:21 065009 [43204960] -> __osm_trap_rcv_process_request: ERR
> 3804: Received trap 16 times consecutively
> Mar 20 14:08:21 082838 [41401960] -> __osm_trap_rcv_process_request:
> Received Generic Notice type:0x01 num:128 Producer:2 from LID:0x0152
> TID:0x0000000000000011
> Mar 20 14:08:21 082877 [41401960] -> __osm_trap_rcv_process_request: ERR
> 3804: Received trap 17 times consecutively
> Mar 20 14:08:21 100567 [43204960] -> __osm_trap_rcv_process_request:
> Received Generic Notice type:0x01 num:128 Producer:2 from LID:0x0152
> TID:0x0000000000000012
> Mar 20 14:08:21 100619 [43204960] -> __osm_trap_rcv_process_request: ERR
> 3804: Received trap 18 times consecutively
> Mar 20 14:08:21 188128 [43C05960] -> osm_report_notice: Reporting Generic
> Notice type:3 num:65 from LID:0x0092
> GID:0xfe80000000000000,0x0005ad0000024bbb
> Mar 20 14:08:21 188144 [43C05960] -> Removed port with
> GUID:0x0005ad000002510b LID range [0xB5,0xB5] of node:saguaro-23-6 HCA-1
> Mar 20 14:08:21 188166 [43C05960] -> osm_report_notice: Reporting Generic
> Notice type:3 num:65 from LID:0x0092
> GID:0xfe80000000000000,0x0005ad0000024bbb
> Mar 20 14:08:21 188172 [43C05960] -> Removed port with
> GUID:0x0005ad0000024d47 LID range [0xB6,0xB6] of node:saguaro-23-7 HCA-1
> Mar 20 14:08:21 188194 [43C05960] -> osm_report_notice: Reporting Generic
> Notice type:3 num:65 from LID:0x0092
> GID:0xfe80000000000000,0x0005ad0000024bbb
> Mar 20 14:08:21 188199 [43C05960] -> Removed port with
> GUID:0x0005ad0000024d8b LID range [0xB7,0xB7] of node:saguaro-23-8 HCA-1
> Mar 20 14:08:21 192421 [41E02960] -> osm_report_notice: Reporting Generic
> Notice type:3 num:64 from LID:0x0092
> GID:0xfe80000000000000,0x0005ad0000024bbb
> Mar 20 14:08:21 192436 [41E02960] -> Discovered new port with
> GUID:0x0005ad00000249d3 LID range [0xB1,0xB1] of node:saguaro-23-2 HCA-1
> Mar 20 14:08:21 208455 [41401960] -> __osm_trap_rcv_process_request:
> Received Generic Notice type:0x01 num:128 Producer:2 from LID:0x0152
> TID:0x0000000000000013
> Mar 20 14:08:21 208499 [41401960] -> __osm_trap_rcv_process_request: ERR
> 3804: Received trap 19 times consecutively
> Mar 20 14:08:21 223240 [41E02960] -> osm_ucast_mgr_process: Min Hop Tables
> configured on all switches
> Mar 20 14:08:21 394585 [45007960] -> __osm_trap_rcv_process_request:
> Received Generic Notice type:0x01 num:128 Producer:2 from LID:0x0152
> TID:0x0000000000000014
> Mar 20 14:08:21 394665 [45007960] -> __osm_trap_rcv_process_request: ERR
> 3804: Received trap 20 times consecutively
> Mar 20 14:08:21 419333 [41E02960] -> __osm_trap_rcv_process_request:
> Received Generic Notice type:0x01 num:128 Producer:2 from LID:0x0152
> TID:0x0000000000000015
> Mar 20 14:08:21 419393 [41E02960] -> __osm_trap_rcv_process_request: ERR
> 3804: Received trap 21 times consecutively
> Mar 20 14:08:21 441228 [43204960] -> __osm_trap_rcv_process_request:
> Received Generic Notice type:0x01 num:128 Producer:2 from LID:0x0152
> TID:0x0000000000000016
> Mar 20 14:08:21 441276 [43204960] -> __osm_trap_rcv_process_request: ERR
> 3804: Received trap 22 times consecutively
> Mar 20 14:08:21 462915 [44606960] -> __osm_trap_rcv_process_request:
> Received Generic Notice type:0x01 num:128 Producer:2 from LID:0x0152
> TID:0x0000000000000017
> Mar 20 14:08:21 462968 [44606960] -> __osm_trap_rcv_process_request: ERR
> 3804: Received trap 23 times consecutively
> Mar 20 14:08:21 475440 [45007960] -> SUBNET UP
> Mar 20 14:08:21 674045 [44606960] -> __osm_trap_rcv_process_request:
> Received Generic Notice type:0x01 num:128 Producer:2 from LID:0x0152
> TID:0x0000000000000018
> Mar 20 14:08:21 674084 [43204960] -> __osm_trap_rcv_process_request:
> Received Generic Notice type:0x01 num:128 Producer:2 from LID:0x0148
> TID:0x000000000000004b
> Mar 20 14:08:21 674137 [44606960] -> __osm_trap_rcv_process_request: ERR
> 3804: Received trap 24 times consecutively
> Mar 20 14:08:21 674294 [43204960] -> osm_report_notice: Reporting Generic
> Notice type:1 num:128 from LID:0x0148
> GID:0xfe80000000000000,0x0005ad00000281b3
> Mar 20 14:08:21 965885 [43204960] -> __osm_trap_rcv_process_request:
> Received Generic Notice type:0x01 num:128 Producer:2 from LID:0x0148
> TID:0x000000000000004c
> Mar 20 14:08:21 965992 [43204960] -> osm_report_notice: Reporting Generic
> Notice type:1 num:128 from LID:0x0148
> GID:0xfe80000000000000,0x0005ad00000281b3
> Mar 20 14:08:22 092378 [41401960] -> osm_report_notice: Reporting Generic
> Notice type:3 num:65 from LID:0x0092
> GID:0xfe80000000000000,0x0005ad0000024bbb
> Mar 20 14:08:22 092395 [41401960] -> Removed port with
> GUID:0x0005ad0000024977 LID range [0xA9,0xA9] of node:saguaro-22-4 HCA-1
> Mar 20 14:08:22 092415 [41401960] -> osm_report_notice: Reporting Generic
> Notice type:3 num:65 from LID:0x0092
> GID:0xfe80000000000000,0x0005ad0000024bbb
> Mar 20 14:08:22 092420 [41401960] -> Removed port with
> GUID:0x0005ad0000024c9b LID range [0xA7,0xA7] of node:saguaro-22-2 HCA-1
> Mar 20 14:08:22 092444 [41401960] -> osm_report_notice: Reporting Generic
> Notice type:3 num:65 from LID:0x0092
> GID:0xfe80000000000000,0x0005ad0000024bbb
> Mar 20 14:08:22 092449 [41401960] -> Removed port with
> GUID:0x0005ad0000024feb LID range [0x153,0x153] of node:saguaro-22-5 HCA-1
> Mar 20 14:08:22 092625 [41401960] -> osm_drop_mgr_process: ERR 0108: Unknown
> remote side for node 0x0005ad00000281b3 port 22. Adding to light sweep
> sampling list
> Mar 20 14:08:22 092655 [41401960] -> Directed Path Dump of 4 hop path:
> Path = [0][1][11][1][4]
> Mar 20 14:08:22 092663 [41401960] -> osm_drop_mgr_process: ERR 0108: Unknown
> remote side for node 0x0005ad00000281b3 port 23. Adding to light sweep
> sampling list
> Mar 20 14:08:22 092672 [41401960] -> Directed Path Dump of 4 hop path:
> Path = [0][1][11][1][4]
> Mar 20 14:08:22 096789 [41E02960] -> osm_report_notice: Reporting Generic
> Notice type:3 num:64 from LID:0x0092
> GID:0xfe80000000000000,0x0005ad0000024bbb
> Mar 20 14:08:22 096801 [41E02960] -> Discovered new port with
> GUID:0x0005ad000002510b LID range [0xB5,0xB5] of node:saguaro-23-6 HCA-1
> Mar 20 14:08:22 096805 [41E02960] -> osm_report_notice: Reporting Generic
> Notice type:3 num:64 from LID:0x0092
> GID:0xfe80000000000000,0x0005ad0000024bbb
> Mar 20 14:08:22 096810 [41E02960] -> Discovered new port with
> GUID:0x0005ad0000024d47 LID range [0xB6,0xB6] of node:saguaro-23-7 HCA-1
> Mar 20 14:08:22 096814 [41E02960] -> osm_report_notice: Reporting Generic
> Notice type:3 num:64 from LID:0x0092
> GID:0xfe80000000000000,0x0005ad0000024bbb
> Mar 20 14:08:22 096819 [41E02960] -> Discovered new port with
> GUID:0x0005ad0000024d8b LID range [0xB7,0xB7] of node:saguaro-23-8 HCA-1
> Mar 20 14:08:22 127266 [41E02960] -> osm_ucast_mgr_process: Min Hop Tables
> configured on all switches
> Mar 20 14:08:22 184734 [45007960] -> SUBNET UP
> Mar 20 14:08:22 541974 [41401960] -> osm_report_notice: Reporting Generic
> Notice type:3 num:64 from LID:0x0092
> GID:0xfe80000000000000,0x0005ad0000024bbb
> Mar 20 14:08:22 541985 [41401960] -> Discovered new port with
> GUID:0x0005ad0000024c9b LID range [0xA7,0xA7] of node:saguaro-22-2 HCA-1
> Mar 20 14:08:22 541989 [41401960] -> osm_report_notice: Reporting Generic
> Notice type:3 num:64 from LID:0x0092
> GID:0xfe80000000000000,0x0005ad0000024bbb
> Mar 20 14:08:22 541995 [41401960] -> Discovered new port with
> GUID:0x0005ad0000024977 LID range [0xA9,0xA9] of node:saguaro-22-4 HCA-1
> Mar 20 14:08:22 541998 [41401960] -> osm_report_notice: Reporting Generic
> Notice type:3 num:64 from LID:0x0092
> GID:0xfe80000000000000,0x0005ad0000024bbb
> Mar 20 14:08:22 542003 [41401960] -> Discovered new port with
> GUID:0x0005ad0000024feb LID range [0x153,0x153] of node:saguaro-22-5 HCA-1
> Mar 20 14:08:22 572711 [41401960] -> osm_ucast_mgr_process: Min Hop Tables
> configured on all switches
> Mar 20 14:08:22 611570 [41401960] -> __osm_trap_rcv_process_request:
> Received Generic Notice type:0x01 num:128 Producer:2 from LID:0x0148
> TID:0x000000000000004d
> Mar 20 14:08:22 611751 [41401960] -> osm_report_notice: Reporting Generic
> Notice type:1 num:128 from LID:0x0148
> GID:0xfe80000000000000,0x0005ad00000281b3
> Mar 20 14:08:22 611770 [44606960] -> __osm_trap_rcv_process_request:
> Received Generic Notice type:0x01 num:128 Producer:2 from LID:0x001B
> TID:0x000000000000005e
> Mar 20 14:08:22 612060 [44606960] -> osm_report_notice: Reporting Generic
> Notice type:1 num:128 from LID:0x001B
> GID:0xfe80000000000000,0x0005ad00000281a7
> Mar 20 14:08:22 623766 [4780B960] -> __osm_sm_mad_ctrl_rcv_callback: ERR
> 3111: Error status = 0x1C00
> Mar 20 14:08:22 623814 [4780B960] -> SMP dump:
> base_ver................0x1
> mgmt_class..............0x81
> class_ver...............0x1
> method..................0x81 (SubnGetResp)
> D bit...................0x1
> status..................0x1C00
> hop_ptr.................0x0
> hop_count...............0x4
> trans_id................0x66134
> attr_id.................0x15 (PortInfo)
> resv....................0x0
> attr_mod................0x16
> m_key...................0x0000000000000000
> dr_slid.................0xFFFF
> dr_dlid.................0xFFFF
>
> Initial path: [0][1][11][1][5]
> Return path: [0][9][18][D][2]
> Reserved: [0][0][0][0][0][0][0]
>
> 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
>
> 00 00 00 00 00 00 00 00 00 00 00 00 02 03 03 02
>
> 14 52 00 11 40 40 00 08 08 04 F2 40 00 00 00 00
>
> 00 00 00 00 00 88 00 00 00 00 00 00 00 00 00 00
>
> Mar 20 14:08:22 623876 [45007960] -> osm_pi_rcv_process_set: Received error
> status 0x1c for SetResp() during ACTIVE transition
> Mar 20 14:08:22 623888 [45007960] -> PortInfo dump:
> port number.............0x16
> node_guid...............0x0005ad00000281b3
> port_guid...............0x0005ad00000281b3
> m_key...................0x0000000000000000
> subnet_prefix...........0x0000000000000000
> base_lid................0x0
> master_sm_base_lid......0x0
> capability_mask.........0x0
> diag_code...............0x0
> m_key_lease_period......0x0
> local_port_num..........0x2
> link_width_enabled......0x3
> link_width_supported....0x3
> link_width_active.......0x2
> link_speed_supported....0x1
> port_state..............ACTIVE
> state_info2.............0x52
> m_key_protect_bits......0x0
> lmc.....................0x0
> link_speed..............0x11
> mtu_smsl................0x40
> vl_cap_init_type........0x40
> vl_high_limit...........0x0
> vl_arb_high_cap.........0x8
> vl_arb_low_cap..........0x8
> init_rep_mtu_cap........0x4
> vl_stall_life...........0xF2
> vl_enforce..............0x40
> m_key_violations........0x0
> p_key_violations........0x0
> q_key_violations........0x0
> guid_cap................0x0
> client_reregister.......0x0
> subnet_timeout..........0x0
> resp_time_value.........0x0
> error_threshold.........0x88
> Mar 20 14:08:22 623907 [45007960] -> Capabilities Mask:
> Mar 20 14:08:22 623945 [4780B960] -> __osm_sm_mad_ctrl_rcv_callback: ERR
> 3111: Error status = 0x1C00
> Mar 20 14:08:22 623973 [4780B960] -> SMP dump:
> base_ver................0x1
> mgmt_class..............0x81
> class_ver...............0x1
> method..................0x81 (SubnGetResp)
> D bit...................0x1
> status..................0x1C00
> hop_ptr.................0x0
> hop_count...............0x4
> trans_id................0x66135
> attr_id.................0x15 (PortInfo)
> resv....................0x0
> attr_mod................0x17
> m_key...................0x0000000000000000
> dr_slid.................0xFFFF
> dr_dlid.................0xFFFF
>
> Initial path: [0][1][11][1][5]
> Return path: [0][9][18][D][2]
> Reserved: [0][0][0][0][0][0][0]
>
> 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
>
> 00 00 00 00 00 00 00 00 00 00 00 00 02 03 03 02
>
> 14 52 00 11 40 40 00 08 08 04 F2 40 00 00 00 00
>
> 00 00 00 00 00 88 00 00 00 00 00 00 00 00 00 00
>
> Mar 20 14:08:22 624051 [44606960] -> osm_pi_rcv_process_set: Received error
> status 0x1c for SetResp() during ACTIVE transition
> Mar 20 14:08:22 624056 [44606960] -> PortInfo dump:
> port number.............0x17
> node_guid...............0x0005ad00000281b3
> port_guid...............0x0005ad00000281b3
> m_key...................0x0000000000000000
> subnet_prefix...........0x0000000000000000
> base_lid................0x0
> master_sm_base_lid......0x0
> capability_mask.........0x0
> diag_code...............0x0
> m_key_lease_period......0x0
> local_port_num..........0x2
> link_width_enabled......0x3
> link_width_supported....0x3
> link_width_active.......0x2
> link_speed_supported....0x1
> port_state..............ACTIVE
> state_info2.............0x52
> m_key_protect_bits......0x0
> lmc.....................0x0
> link_speed..............0x11
> mtu_smsl................0x40
> vl_cap_init_type........0x40
> vl_high_limit...........0x0
> vl_arb_high_cap.........0x8
> vl_arb_low_cap..........0x8
> init_rep_mtu_cap........0x4
> vl_stall_life...........0xF2
> vl_enforce..............0x40
> m_key_violations........0x0
> p_key_violations........0x0
> q_key_violations........0x0
> guid_cap................0x0
> client_reregister.......0x0
> subnet_timeout..........0x0
> resp_time_value.........0x0
> error_threshold.........0x88
> Mar 20 14:08:22 624069 [44606960] -> Capabilities Mask:
> Mar 20 14:08:22 629289 [45A08960] -> SUBNET UP
> Mar 20 14:08:22 712180 [43204960] -> __osm_trap_rcv_process_request:
> Received Generic Notice type:0x01 num:128 Producer:2 from LID:0x0152
> TID:0x0000000000000018
> Mar 20 14:08:22 712238 [43204960] -> __osm_trap_rcv_process_request: ERR
> 3804: Received trap 25 times consecutively
> Mar 20 14:08:22 869303 [43C05960] -> __osm_trap_rcv_process_request:
> Received Generic Notice type:0x01 num:128 Producer:2 from LID:0x001B
> TID:0x000000000000005f
> Mar 20 14:08:22 869527 [43C05960] -> osm_report_notice: Reporting Generic
> Notice type:1 num:128 from LID:0x001B
> GID:0xfe80000000000000,0x0005ad00000281a7
> Mar 20 14:08:22 892522 [45A08960] -> __osm_trap_rcv_process_request:
> Received Generic Notice type:0x01 num:128 Producer:2 from LID:0x0148
> TID:0x000000000000004e
> Mar 20 14:08:22 892707 [45A08960] -> osm_report_notice: Reporting Generic
> Notice type:1 num:128 from LID:0x0148
> GID:0xfe80000000000000,0x0005ad00000281b3
> Mar 20 14:08:22 957086 [42803960] -> __osm_trap_rcv_process_request:
> Received Generic Notice type:0x01 num:128 Producer:2 from LID:0x001B
> TID:0x0000000000000060
> Mar 20 14:08:22 957189 [42803960] -> osm_report_notice: Reporting Generic
> Notice type:1 num:128 from LID:0x001B
> GID:0xfe80000000000000,0x0005ad00000281a7
> Mar 20 14:08:23 080551 [41E02960] -> __osm_trap_rcv_process_request:
> Received Generic Notice type:0x01 num:128 Producer:2 from LID:0x001B
> TID:0x0000000000000061
> Mar 20 14:08:23 080621 [41E02960] -> osm_report_notice: Reporting Generic
> Notice type:1 num:128 from LID:0x001B
> GID:0xfe80000000000000,0x0005ad00000281a7
> Mar 20 14:08:23 102292 [45007960] -> __osm_trap_rcv_process_request:
> Received Generic Notice type:0x01 num:128 Producer:2 from LID:0x001B
> TID:0x0000000000000062
> Mar 20 14:08:23 102372 [45007960] -> osm_report_notice: Reporting Generic
> Notice type:1 num:128 from LID:0x001B
> GID:0xfe80000000000000,0x0005ad00000281a7
> Mar 20 14:08:23 124176 [43C05960] -> __osm_trap_rcv_process_request:
> Received Generic Notice type:0x01 num:128 Producer:2 from LID:0x001B
> TID:0x0000000000000063
> Mar 20 14:08:23 124278 [43C05960] -> osm_report_notice: Reporting Generic
> Notice type:1 num:128 from LID:0x001B
> GID:0xfe80000000000000,0x0005ad00000281a7
> Mar 20 14:08:23 285320 [42803960] -> __osm_trap_rcv_process_request:
> Received Generic Notice type:0x01 num:128 Producer:2 from LID:0x001B
> TID:0x0000000000000064
> Mar 20 14:08:23 285393 [42803960] -> osm_report_notice: Reporting Generic
> Notice type:1 num:128 from LID:0x001B
> GID:0xfe80000000000000,0x0005ad00000281a7
> Mar 20 14:08:23 403309 [41401960] -> __osm_trap_rcv_process_request:
> Received Generic Notice type:0x01 num:128 Producer:2 from LID:0x001B
> TID:0x0000000000000065
> Mar 20 14:08:23 403388 [41401960] -> osm_report_notice: Reporting Generic
> Notice type:1 num:128 from LID:0x001B
> GID:0xfe80000000000000,0x0005ad00000281a7
> Mar 20 14:08:23 425052 [45A08960] -> __osm_trap_rcv_process_request:
> Received Generic Notice type:0x01 num:128 Producer:2 from LID:0x001B
> TID:0x0000000000000066
> Mar 20 14:08:23 425117 [45A08960] -> osm_report_notice: Reporting Generic
> Notice type:1 num:128 from LID:0x001B
> GID:0xfe80000000000000,0x0005ad00000281a7
> Mar 20 14:08:23 447189 [41E02960] -> __osm_trap_rcv_process_request:
> Received Generic Notice type:0x01 num:128 Producer:2 from LID:0x001B
> TID:0x0000000000000067
> Mar 20 14:08:23 447266 [41E02960] -> osm_report_notice: Reporting Generic
> Notice type:1 num:128 from LID:0x001B
> GID:0xfe80000000000000,0x0005ad00000281a7
> Mar 20 14:08:23 535175 [44606960] -> osm_ucast_mgr_process: Min Hop Tables
> configured on all switches
> Mar 20 14:08:23 595127 [41401960] -> SUBNET UP
> Mar 20 14:08:23 750323 [41E02960] -> __osm_trap_rcv_process_request:
> Received Generic Notice type:0x01 num:128 Producer:2 from LID:0x0152
> TID:0x0000000000000018
> Mar 20 14:08:23 750432 [41E02960] -> __osm_trap_rcv_process_request: ERR
> 3804: Received trap 26 times consecutively
> Mar 20 14:08:23 960490 [42803960] -> osm_ucast_mgr_process: Min Hop Tables
> configured on all switches
> Mar 20 14:08:24 014256 [4780B960] -> __osm_sm_mad_ctrl_rcv_callback: ERR
> 3111: Error status = 0x1C00
> Mar 20 14:08:24 014323 [4780B960] -> SMP dump:
> base_ver................0x1
> mgmt_class..............0x81
> class_ver...............0x1
> method..................0x81 (SubnGetResp)
> D bit...................0x1
> status..................0x1C00
> hop_ptr.................0x0
> hop_count...............0x4
> trans_id................0x67b9d
> attr_id.................0x15 (PortInfo)
> resv....................0x0
> attr_mod................0x18
> m_key...................0x0000000000000000
> dr_slid.................0xFFFF
> dr_dlid.................0xFFFF
>
> Initial path: [0][1][11][1][6]
> Return path: [0][9][18][D][3]
> Reserved: [0][0][0][0][0][0][0]
>
> 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
>
> 00 00 00 00 00 00 00 00 00 00 00 00 03 03 03 02
>
> 14 52 00 11 40 40 00 08 08 04 F2 40 00 00 00 00
>
> 00 00 00 00 00 88 00 00 00 00 00 00 00 00 00 00
>
> Mar 20 14:08:24 014398 [41401960] -> osm_pi_rcv_process_set: Received error
> status 0x1c for SetResp() during ACTIVE transition
> Mar 20 14:08:24 014408 [41401960] -> PortInfo dump:
> port number.............0x18
> node_guid...............0x0005ad00000281b3
> port_guid...............0x0005ad00000281b3
> m_key...................0x0000000000000000
> subnet_prefix...........0x0000000000000000
> base_lid................0x0
> master_sm_base_lid......0x0
> capability_mask.........0x0
> diag_code...............0x0
> m_key_lease_period......0x0
> local_port_num..........0x3
> link_width_enabled......0x3
> link_width_supported....0x3
> link_width_active.......0x2
> link_speed_supported....0x1
> port_state..............ACTIVE
> state_info2.............0x52
> m_key_protect_bits......0x0
> lmc.....................0x0
> link_speed..............0x11
> mtu_smsl................0x40
> vl_cap_init_type........0x40
> vl_high_limit...........0x0
> vl_arb_high_cap.........0x8
> vl_arb_low_cap..........0x8
> init_rep_mtu_cap........0x4
> vl_stall_life...........0xF2
> vl_enforce..............0x40
> m_key_violations........0x0
> p_key_violations........0x0
> q_key_violations........0x0
> guid_cap................0x0
> client_reregister.......0x0
> subnet_timeout..........0x0
> resp_time_value.........0x0
> error_threshold.........0x88
> Mar 20 14:08:24 014422 [41401960] -> Capabilities Mask:
> Mar 20 14:08:24 019479 [41401960] -> SUBNET UP
> Mar 20 14:11:00 201308 [43204960] -> __osm_trap_rcv_process_request:
> Received Generic Notice type:0x01 num:128 Producer:2 from LID:0x001F
> TID:0x0000000000000018
> Mar 20 14:11:00 201580 [43204960] -> osm_report_notice: Reporting Generic
> Notice type:1 num:128 from LID:0x001F
> GID:0xfe80000000000000,0x0005ad0000027c56
> Mar 20 14:11:00 554517 [41E02960] -> osm_report_notice: Reporting Generic
> Notice type:3 num:65 from LID:0x0092
> GID:0xfe80000000000000,0x0005ad0000024bbb
> Mar 20 14:11:00 554538 [41E02960] -> Removed port with
> GUID:0x0005ad000002516f LID range [0xBA,0xBA] of node:saguaro-24-1 HCA-1
> Mar 20 14:11:00 589140 [41E02960] -> osm_ucast_mgr_process: Min Hop Tables
> configured on all switches
> Mar 20 14:11:00 641315 [45A08960] -> SUBNET UP
> Mar 20 14:14:16 904140 [41E02960] -> __osm_trap_rcv_process_request:
> Received Generic Notice type:0x01 num:128 Producer:2 from LID:0x001B
> TID:0x0000000000000068
> Mar 20 14:14:16 904369 [41E02960] -> osm_report_notice: Reporting Generic
> Notice type:1 num:128 from LID:0x001B
> GID:0xfe80000000000000,0x0005ad00000281a7
> Mar 20 14:14:16 904462 [45007960] -> __osm_trap_rcv_process_request:
> Received Generic Notice type:0x01 num:128 Producer:2 from LID:0x0148
> TID:0x000000000000004f
> Mar 20 14:14:16 904600 [45007960] -> osm_report_notice: Reporting Generic
> Notice type:1 num:128 from LID:0x0148
> GID:0xfe80000000000000,0x0005ad00000281b3
> Mar 20 14:14:17 210726 [41401960] -> osm_report_notice: Reporting Generic
> Notice type:3 num:65 from LID:0x0092
> GID:0xfe80000000000000,0x0005ad0000024bbb
> Mar 20 14:14:17 210747 [41401960] -> Removed port with
> GUID:0x0005ad0000024e0b LID range [0xB3,0xB3] of node:saguaro-23-4 HCA-1
> Mar 20 14:14:17 210796 [41401960] -> osm_report_notice: Reporting Generic
> Notice type:3 num:65 from LID:0x0092
> GID:0xfe80000000000000,0x0005ad0000024bbb
> Mar 20 14:14:17 210802 [41401960] -> Removed port with
> GUID:0x0005ad000002510b LID range [0xB5,0xB5] of node:saguaro-23-6 HCA-1
> Mar 20 14:14:17 210818 [41401960] -> osm_report_notice: Reporting Generic
> Notice type:3 num:65 from LID:0x0092
> GID:0xfe80000000000000,0x0005ad0000024bbb
> Mar 20 14:14:17 210836 [41401960] -> Removed port with
> GUID:0x0005ad000002511b LID range [0xA6,0xA6] of node:saguaro-22-1 HCA-1
> Mar 20 14:14:17 210864 [41401960] -> osm_report_notice: Reporting Generic
> Notice type:3 num:65 from LID:0x0092
> GID:0xfe80000000000000,0x0005ad0000024bbb
> Mar 20 14:14:17 210869 [41401960] -> Removed port with
> GUID:0x0005ad0000025043 LID range [0xB4,0xB4] of node:saguaro-23-5 HCA-1
> Mar 20 14:14:17 210885 [41401960] -> osm_report_notice: Reporting Generic
> Notice type:3 num:65 from LID:0x0092
> GID:0xfe80000000000000,0x0005ad0000024bbb
> Mar 20 14:14:17 210890 [41401960] -> Removed port with
> GUID:0x0005ad0000024d47 LID range [0xB6,0xB6] of node:saguaro-23-7 HCA-1
> Mar 20 14:14:17 210908 [41401960] -> osm_report_notice: Reporting Generic
> Notice type:3 num:65 from LID:0x0092
> GID:0xfe80000000000000,0x0005ad0000024bbb
> Mar 20 14:14:17 210913 [41401960] -> Removed port with
> GUID:0x0005ad0000024d6b LID range [0xB8,0xB8] of node:saguaro-23-9 HCA-1
> Mar 20 14:14:17 210931 [41401960] -> osm_report_notice: Reporting Generic
> Notice type:3 num:65 from LID:0x0092
> GID:0xfe80000000000000,0x0005ad0000024bbb
> Mar 20 14:14:17 210936 [41401960] -> Removed port with
> GUID:0x0005ad0000024977 LID range [0xA9,0xA9] of node:saguaro-22-4 HCA-1
> Mar 20 14:14:17 211090 [41401960] -> osm_report_notice: Reporting Generic
> Notice type:3 num:65 from LID:0x0092
> GID:0xfe80000000000000,0x0005ad0000024bbb
> Mar 20 14:14:17 211096 [41401960] -> Removed port with
> GUID:0x0005ad0000027c84 LID range [0x152,0x152] of node:Topspin Switch TS120
> Mar 20 14:14:17 211127 [41401960] -> osm_report_notice: Reporting Generic
> Notice type:3 num:65 from LID:0x0092
> GID:0xfe80000000000000,0x0005ad0000024bbb
> Mar 20 14:14:17 211133 [41401960] -> Removed port with
> GUID:0x0005ad0000024d8b LID range [0xB7,0xB7] of node:saguaro-23-8 HCA-1
> Mar 20 14:14:17 211147 [41401960] -> osm_report_notice: Reporting Generic
> Notice type:3 num:65 from LID:0x0092
> GID:0xfe80000000000000,0x0005ad0000024bbb
> Mar 20 14:14:17 211153 [41401960] -> Removed port with
> GUID:0x0005ad000002498f LID range [0xA8,0xA8] of node:saguaro-22-3 HCA-1
> Mar 20 14:14:17 211169 [41401960] -> osm_report_notice: Reporting Generic
> Notice type:3 num:65 from LID:0x0092
> GID:0xfe80000000000000,0x0005ad0000024bbb
> Mar 20 14:14:17 211174 [41401960] -> Removed port with
> GUID:0x0005ad0000024c9b LID range [0xA7,0xA7] of node:saguaro-22-2 HCA-1
> Mar 20 14:14:17 211189 [41401960] -> osm_report_notice: Reporting Generic
> Notice type:3 num:65 from LID:0x0092
> GID:0xfe80000000000000,0x0005ad0000024bbb
> Mar 20 14:14:17 211194 [41401960] -> Removed port with
> GUID:0x0005ad0000024da7 LID range [0xB0,0xB0] of node:saguaro-23-1 HCA-1
> Mar 20 14:14:17 211212 [41401960] -> osm_report_notice: Reporting Generic
> Notice type:3 num:65 from LID:0x0092
> GID:0xfe80000000000000,0x0005ad0000024bbb
> Mar 20 14:14:17 211216 [41401960] -> Removed port with
> GUID:0x0005ad0000024cbb LID range [0xB2,0xB2] of node:saguaro-23-3 HCA-1
> Mar 20 14:14:17 211232 [41401960] -> osm_report_notice: Reporting Generic
> Notice type:3 num:65 from LID:0x0092
> GID:0xfe80000000000000,0x0005ad0000024bbb
> Mar 20 14:14:17 211237 [41401960] -> Removed port with
> GUID:0x0005ad00000249d3 LID range [0xB1,0xB1] of node:saguaro-23-2 HCA-1
> Mar 20 14:14:17 211253 [41401960] -> osm_report_notice: Reporting Generic
> Notice type:3 num:65 from LID:0x0092
> GID:0xfe80000000000000,0x0005ad0000024bbb
> Mar 20 14:14:17 211317 [41401960] -> Removed port with
> GUID:0x0005ad0000024feb LID range [0x153,0x153] of node:saguaro-22-5 HCA-1
> Mar 20 14:14:17 211333 [41401960] -> osm_report_notice: Reporting Generic
> Notice type:3 num:65 from LID:0x0092
> GID:0xfe80000000000000,0x0005ad0000024bbb
> Mar 20 14:14:17 211338 [41401960] -> Removed port with
> GUID:0x0005ad0000024afb LID range [0xA5,0xA5] of node:saguaro-22-0 HCA-1
> Mar 20 14:14:17 244432 [41401960] -> osm_ucast_mgr_process: Min Hop Tables
> configured on all switches
> Mar 20 14:14:17 292747 [42803960] -> SUBNET UP
> Mar 20 14:14:17 698554 [45A08960] -> osm_ucast_mgr_process: Min Hop Tables
> configured on all switches
> Mar 20 14:14:17 750419 [44606960] -> SUBNET UP
> Mar 20 14:15:11 300343 [41401960] -> __osm_trap_rcv_process_request:
> Received Generic Notice type:0x01 num:128 Producer:2 from LID:0x0148
> TID:0x0000000000000050
> Mar 20 14:15:11 300577 [41401960] -> osm_report_notice: Reporting Generic
> Notice type:1 num:128 from LID:0x0148
> GID:0xfe80000000000000,0x0005ad00000281b3
> Mar 20 14:15:11 306375 [45A08960] -> __osm_trap_rcv_process_request:
> Received Generic Notice type:0x01 num:128 Producer:2 from LID:0x001B
> TID:0x0000000000000069
> Mar 20 14:15:11 306439 [42803960] -> __osm_trap_rcv_process_request:
> Received Generic Notice type:0x01 num:128 Producer:2 from LID:0x0148
> TID:0x0000000000000051
> Mar 20 14:15:11 306487 [45A08960] -> osm_report_notice: Reporting Generic
> Notice type:1 num:128 from LID:0x001B
> GID:0xfe80000000000000,0x0005ad00000281a7
> Mar 20 14:15:11 306514 [42803960] -> osm_report_notice: Reporting Generic
> Notice type:1 num:128 from LID:0x0148
> GID:0xfe80000000000000,0x0005ad00000281b3
> Mar 20 14:15:11 312487 [43204960] -> __osm_trap_rcv_process_request:
> Received Generic Notice type:0x01 num:128 Producer:2 from LID:0x001B
> TID:0x000000000000006a
> Mar 20 14:15:11 312581 [43204960] -> osm_report_notice: Reporting Generic
> Notice type:1 num:128 from LID:0x001B
> GID:0xfe80000000000000,0x0005ad00000281a7
> Mar 20 14:15:11 636546 [45007960] -> osm_report_notice: Reporting Generic
> Notice type:3 num:64 from LID:0x0092
> GID:0xfe80000000000000,0x0005ad0000024bbb
> Mar 20 14:15:11 636559 [45007960] -> Discovered new port with
> GUID:0x0005ad0000027c84 LID range [0x152,0x152] of node:Topspin Switch TS120
> Mar 20 14:15:11 636565 [45007960] -> osm_report_notice: Reporting Generic
> Notice type:3 num:64 from LID:0x0092
> GID:0xfe80000000000000,0x0005ad0000024bbb
> Mar 20 14:15:11 636572 [45007960] -> Discovered new port with
> GUID:0x0005ad0000024da7 LID range [0xB0,0xB0] of node:saguaro-23-1 HCA-1
> Mar 20 14:15:11 636577 [45007960] -> osm_report_notice: Reporting Generic
> Notice type:3 num:64 from LID:0x0092
> GID:0xfe80000000000000,0x0005ad0000024bbb
> Mar 20 14:15:11 636584 [45007960] -> Discovered new port with
> GUID:0x0005ad00000249d3 LID range [0xB1,0xB1] of node:saguaro-23-2 HCA-1
> Mar 20 14:15:11 636589 [45007960] -> osm_report_notice: Reporting Generic
> Notice type:3 num:64 from LID:0x0092
> GID:0xfe80000000000000,0x0005ad0000024bbb
> Mar 20 14:15:11 636595 [45007960] -> Discovered new port with
> GUID:0x0005ad0000024cbb LID range [0xB2,0xB2] of node:saguaro-23-3 HCA-1
> Mar 20 14:15:11 636600 [45007960] -> osm_report_notice: Reporting Generic
> Notice type:3 num:64 from LID:0x0092
> GID:0xfe80000000000000,0x0005ad0000024bbb
> Mar 20 14:15:11 636606 [45007960] -> Discovered new port with
> GUID:0x0005ad0000024e0b LID range [0xB3,0xB3] of node:saguaro-23-4 HCA-1
> Mar 20 14:15:11 636612 [45007960] -> osm_report_notice: Reporting Generic
> Notice type:3 num:64 from LID:0x0092
> GID:0xfe80000000000000,0x0005ad0000024bbb
> Mar 20 14:15:11 636618 [45007960] -> Discovered new port with
> GUID:0x0005ad0000025043 LID range [0xB4,0xB4] of node:saguaro-23-5 HCA-1
> Mar 20 14:15:11 636623 [45007960] -> osm_report_notice: Reporting Generic
> Notice type:3 num:64 from LID:0x0092
> GID:0xfe80000000000000,0x0005ad0000024bbb
> Mar 20 14:15:11 636629 [45007960] -> Discovered new port with
> GUID:0x0005ad000002510b LID range [0xB5,0xB5] of node:saguaro-23-6 HCA-1
> Mar 20 14:15:11 636634 [45007960] -> osm_report_notice: Reporting Generic
> Notice type:3 num:64 from LID:0x0092
> GID:0xfe80000000000000,0x0005ad0000024bbb
> Mar 20 14:15:11 636641 [45007960] -> Discovered new port with
> GUID:0x0005ad0000024d47 LID range [0xB6,0xB6] of node:saguaro-23-7 HCA-1
> Mar 20 14:15:11 636646 [45007960] -> osm_report_notice: Reporting Generic
> Notice type:3 num:64 from LID:0x0092
> GID:0xfe80000000000000,0x0005ad0000024bbb
> Mar 20 14:15:11 636652 [45007960] -> Discovered new port with
> GUID:0x0005ad0000024d8b LID range [0xB7,0xB7] of node:saguaro-23-8 HCA-1
> Mar 20 14:15:11 636657 [45007960] -> osm_report_notice: Reporting Generic
> Notice type:3 num:64 from LID:0x0092
> GID:0xfe80000000000000,0x0005ad0000024bbb
> Mar 20 14:15:11 636663 [45007960] -> Discovered new port with
> GUID:0x0005ad0000024d6b LID range [0xB8,0xB8] of node:saguaro-23-9 HCA-1
> Mar 20 14:15:11 636668 [45007960] -> osm_report_notice: Reporting Generic
> Notice type:3 num:64 from LID:0x0092
> GID:0xfe80000000000000,0x0005ad0000024bbb
> Mar 20 14:15:11 636675 [45007960] -> Discovered new port with
> GUID:0x0005ad0000024afb LID range [0xA5,0xA5] of node:saguaro-22-0 HCA-1
> Mar 20 14:15:11 636680 [45007960] -> osm_report_notice: Reporting Generic
> Notice type:3 num:64 from LID:0x0092
> GID:0xfe80000000000000,0x0005ad0000024bbb
> Mar 20 14:15:11 636686 [45007960] -> Discovered new port with
> GUID:0x0005ad000002511b LID range [0xA6,0xA6] of node:saguaro-22-1 HCA-1
> Mar 20 14:15:11 636691 [45007960] -> osm_report_notice: Reporting Generic
> Notice type:3 num:64 from LID:0x0092
> GID:0xfe80000000000000,0x0005ad0000024bbb
> Mar 20 14:15:11 636698 [45007960] -> Discovered new port with
> GUID:0x0005ad0000024c9b LID range [0xA7,0xA7] of node:saguaro-22-2 HCA-1
> Mar 20 14:15:11 636703 [45007960] -> osm_report_notice: Reporting Generic
> Notice type:3 num:64 from LID:0x0092
> GID:0xfe80000000000000,0x0005ad0000024bbb
> Mar 20 14:15:11 636709 [45007960] -> Discovered new port with
> GUID:0x0005ad000002498f LID range [0xA8,0xA8] of node:saguaro-22-3 HCA-1
> Mar 20 14:15:11 636742 [45007960] -> osm_report_notice: Reporting Generic
> Notice type:3 num:64 from LID:0x0092
> GID:0xfe80000000000000,0x0005ad0000024bbb
> Mar 20 14:15:11 636750 [45007960] -> Discovered new port with
> GUID:0x0005ad0000024977 LID range [0xA9,0xA9] of node:saguaro-22-4 HCA-1
> Mar 20 14:15:11 636755 [45007960] -> osm_report_notice: Reporting Generic
> Notice type:3 num:64 from LID:0x0092
> GID:0xfe80000000000000,0x0005ad0000024bbb
> Mar 20 14:15:11 636761 [45007960] -> Discovered new port with
> GUID:0x0005ad0000024feb LID range [0x153,0x153] of node:saguaro-22-5 HCA-1
> Mar 20 14:15:11 667436 [45007960] -> osm_ucast_mgr_process: Min Hop Tables
> configured on all switches
> Mar 20 14:15:11 731917 [4780B960] -> __osm_sm_mad_ctrl_rcv_callback: ERR
> 3111: Error status = 0x1C00
> Mar 20 14:15:11 732017 [4780B960] -> SMP dump:
> base_ver................0x1
> mgmt_class..............0x81
> class_ver...............0x1
> method..................0x81 (SubnGetResp)
> D bit...................0x1
> status..................0x1C00
> hop_ptr.................0x0
> hop_count...............0x4
> trans_id................0x6b507
> attr_id.................0x15 (PortInfo)
> resv....................0x0
> attr_mod................0x13
> m_key...................0x0000000000000000
> dr_slid.................0xFFFF
> dr_dlid.................0xFFFF
>
> Initial path: [0][1][16][1][4]
> Return path: [0][9][13][D][1]
> Reserved: [0][0][0][0][0][0][0]
>
> 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
>
> 00 00 00 00 00 00 00 00 00 00 00 00 01 03 03 02
>
> 14 52 00 11 40 40 00 08 08 04 F2 40 00 00 00 00
>
> 00 00 00 00 00 88 00 00 00 00 00 00 00 00 00 00
>
> Mar 20 14:15:11 732102 [41401960] -> osm_pi_rcv_process_set: Received error
> status 0x1c for SetResp() during ACTIVE transition
> Mar 20 14:15:11 732106 [41401960] -> PortInfo dump:
> port number.............0x13
> node_guid...............0x0005ad00000281a7
> port_guid...............0x0005ad00000281a7
> m_key...................0x0000000000000000
> subnet_prefix...........0x0000000000000000
> base_lid................0x0
> master_sm_base_lid......0x0
> capability_mask.........0x0
> diag_code...............0x0
> m_key_lease_period......0x0
> local_port_num..........0x1
> link_width_enabled......0x3
> link_width_supported....0x3
> link_width_active.......0x2
> link_speed_supported....0x1
> port_state..............ACTIVE
> state_info2.............0x52
> m_key_protect_bits......0x0
> lmc.....................0x0
> link_speed..............0x11
> mtu_smsl................0x40
> vl_cap_init_type........0x40
> vl_high_limit...........0x0
> vl_arb_high_cap.........0x8
> vl_arb_low_cap..........0x8
> init_rep_mtu_cap........0x4
> vl_stall_life...........0xF2
> vl_enforce..............0x40
> m_key_violations........0x0
> p_key_violations........0x0
> q_key_violations........0x0
> guid_cap................0x0
> client_reregister.......0x0
> subnet_timeout..........0x0
> resp_time_value.........0x0
> error_threshold.........0x88
> Mar 20 14:15:11 732128 [41401960] -> Capabilities Mask:
> Mar 20 14:15:11 732160 [4780B960] -> __osm_sm_mad_ctrl_rcv_callback: ERR
> 3111: Error status = 0x1C00
> Mar 20 14:15:11 732185 [4780B960] -> SMP dump:
> base_ver................0x1
> mgmt_class..............0x81
> class_ver...............0x1
> method..................0x81 (SubnGetResp)
> D bit...................0x1
> status..................0x1C00
> hop_ptr.................0x0
> hop_count...............0x4
> trans_id................0x6b508
> attr_id.................0x15 (PortInfo)
> resv....................0x0
> attr_mod................0x16
> m_key...................0x0000000000000000
> dr_slid.................0xFFFF
> dr_dlid.................0xFFFF
>
> Initial path: [0][1][16][1][4]
> Return path: [0][9][13][D][1]
> Reserved: [0][0][0][0][0][0][0]
>
> 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
>
> 00 00 00 00 00 00 00 00 00 00 00 00 01 03 03 02
>
> 14 52 00 11 40 40 00 08 08 04 F2 40 00 00 00 00
>
> 00 00 00 00 00 88 00 00 00 00 00 00 00 00 00 00
>
> Mar 20 14:15:11 732254 [44606960] -> osm_pi_rcv_process_set: Received error
> status 0x1c for SetResp() during ACTIVE transition
> Mar 20 14:15:11 732258 [44606960] -> PortInfo dump:
> port number.............0x16
> node_guid...............0x0005ad00000281a7
> port_guid...............0x0005ad00000281a7
> m_key...................0x0000000000000000
> subnet_prefix...........0x0000000000000000
> base_lid................0x0
> master_sm_base_lid......0x0
> capability_mask.........0x0
> diag_code...............0x0
> m_key_lease_period......0x0
> local_port_num..........0x1
> link_width_enabled......0x3
> link_width_supported....0x3
> link_width_active.......0x2
> link_speed_supported....0x1
> port_state..............ACTIVE
> state_info2.............0x52
> m_key_protect_bits......0x0
> lmc.....................0x0
> link_speed..............0x11
> mtu_smsl................0x40
> vl_cap_init_type........0x40
> vl_high_limit...........0x0
> vl_arb_high_cap.........0x8
> vl_arb_low_cap..........0x8
> init_rep_mtu_cap........0x4
> vl_stall_life...........0xF2
> vl_enforce..............0x40
> m_key_violations........0x0
> p_key_violations........0x0
> q_key_violations........0x0
> guid_cap................0x0
> client_reregister.......0x0
> subnet_timeout..........0x0
> resp_time_value.........0x0
> error_threshold.........0x88
> Mar 20 14:15:11 732269 [44606960] -> Capabilities Mask:
> Mar 20 14:15:11 732300 [4780B960] -> __osm_sm_mad_ctrl_rcv_callback: ERR
> 3111: Error status = 0x1C00
> Mar 20 14:15:11 732334 [4780B960] -> SMP dump:
> base_ver................0x1
> mgmt_class..............0x81
> class_ver...............0x1
> method..................0x81 (SubnGetResp)
> D bit...................0x1
> status..................0x1C00
> hop_ptr.................0x0
> hop_count...............0x4
> trans_id................0x6b509
> attr_id.................0x15 (PortInfo)
> resv....................0x0
> attr_mod................0x17
> m_key...................0x0000000000000000
> dr_slid.................0xFFFF
> dr_dlid.................0xFFFF
>
> Initial path: [0][1][16][1][4]
> Return path: [0][9][13][D][1]
> Reserved: [0][0][0][0][0][0][0]
>
> 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
>
> 00 00 00 00 00 00 00 00 00 00 00 00 01 03 03 02
>
> 14 52 00 11 40 40 00 08 08 04 F2 40 00 00 00 00
>
> 00 00 00 00 00 88 00 00 00 00 00 00 00 00 00 00
>
> Mar 20 14:15:11 732420 [4780B960] -> __osm_sm_mad_ctrl_rcv_callback: ERR
> 3111: Error status = 0x1C00
> Mar 20 14:15:11 732419 [45007960] -> osm_pi_rcv_process_set: Received error
> status 0x1c for SetResp() during ACTIVE transition
> Mar 20 14:15:11 732451 [4780B960] -> SMP dump:
> base_ver................0x1
> mgmt_class..............0x81
> class_ver...............0x1
> method..................0x81 (SubnGetResp)
> D bit...................0x1
> status..................0x1C00
> hop_ptr.................0x0
> hop_count...............0x4
> trans_id................0x6b50a
> attr_id.................0x15 (PortInfo)
> resv....................0x0
> attr_mod................0x18
> m_key...................0x0000000000000000
> dr_slid.................0xFFFF
> dr_dlid.................0xFFFF
>
> Initial path: [0][1][16][1][4]
> Return path: [0][9][13][D][1]
> Reserved: [0][0][0][0][0][0][0]
>
> 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
>
> 00 00 00 00 00 00 00 00 00 00 00 00 01 03 03 02
>
> 14 52 00 11 40 40 00 08 08 04 F2 40 00 00 00 00
>
> 00 00 00 00 00 88 00 00 00 00 00 00 00 00 00 00
>
> Mar 20 14:15:11 732447 [45007960] -> PortInfo dump:
> port number.............0x17
> node_guid...............0x0005ad00000281a7
> port_guid...............0x0005ad00000281a7
> m_key...................0x0000000000000000
> subnet_prefix...........0x0000000000000000
> base_lid................0x0
> master_sm_base_lid......0x0
> capability_mask.........0x0
> diag_code...............0x0
> m_key_lease_period......0x0
> local_port_num..........0x1
> link_width_enabled......0x3
> link_width_supported....0x3
> link_width_active.......0x2
> link_speed_supported....0x1
> port_state..............ACTIVE
> state_info2.............0x52
> m_key_protect_bits......0x0
> lmc.....................0x0
> link_speed..............0x11
> mtu_smsl................0x40
> vl_cap_init_type........0x40
> vl_high_limit...........0x0
> vl_arb_high_cap.........0x8
> vl_arb_low_cap..........0x8
> init_rep_mtu_cap........0x4
> vl_stall_life...........0xF2
> vl_enforce..............0x40
> m_key_violations........0x0
> p_key_violations........0x0
> q_key_violations........0x0
> guid_cap................0x0
> client_reregister.......0x0
> subnet_timeout..........0x0
> resp_time_value.........0x0
> error_threshold.........0x88
> Mar 20 14:15:11 732471 [45007960] -> Capabilities Mask:
> Mar 20 14:15:11 732511 [45007960] -> osm_pi_rcv_process_set: Received error
> status 0x1c for SetResp() during ACTIVE transition
> Mar 20 14:15:11 732516 [45007960] -> PortInfo dump:
> port number.............0x18
> node_guid...............0x0005ad00000281a7
> port_guid...............0x0005ad00000281a7
> m_key...................0x0000000000000000
> subnet_prefix...........0x0000000000000000
> base_lid................0x0
> master_sm_base_lid......0x0
> capability_mask.........0x0
> diag_code...............0x0
> m_key_lease_period......0x0
> local_port_num..........0x1
> link_width_enabled......0x3
> link_width_supported....0x3
> link_width_active.......0x2
> link_speed_supported....0x1
> port_state..............ACTIVE
> state_info2.............0x52
> m_key_protect_bits......0x0
> lmc.....................0x0
> link_speed..............0x11
> mtu_smsl................0x40
> vl_cap_init_type........0x40
> vl_high_limit...........0x0
> vl_arb_high_cap.........0x8
> vl_arb_low_cap..........0x8
> init_rep_mtu_cap........0x4
> vl_stall_life...........0xF2
> vl_enforce..............0x40
> m_key_violations........0x0
> p_key_violations........0x0
> q_key_violations........0x0
> guid_cap................0x0
> client_reregister.......0x0
> subnet_timeout..........0x0
> resp_time_value.........0x0
> error_threshold.........0x88
> Mar 20 14:15:11 732529 [45007960] -> Capabilities Mask:
> Mar 20 14:15:11 732556 [4780B960] -> __osm_sm_mad_ctrl_rcv_callback: ERR
> 3111: Error status = 0x1C00
> Mar 20 14:15:11 732591 [4780B960] -> SMP dump:
> base_ver................0x1
> mgmt_class..............0x81
> class_ver...............0x1
> method..................0x81 (SubnGetResp)
> D bit...................0x1
> status..................0x1C00
> hop_ptr.................0x0
> hop_count...............0x4
> trans_id................0x6b50b
> attr_id.................0x15 (PortInfo)
> resv....................0x0
> attr_mod................0x16
> m_key...................0x0000000000000000
> dr_slid.................0xFFFF
> dr_dlid.................0xFFFF
>
> Initial path: [0][1][11][2][5]
> Return path: [0][9][18][E][2]
> Reserved: [0][0][0][0][0][0][0]
>
> 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
>
> 00 00 00 00 00 00 00 00 00 00 00 00 02 03 03 02
>
> 14 52 00 11 40 40 00 08 08 04 F2 40 00 00 00 00
>
> 00 00 00 00 00 88 00 00 00 00 00 00 00 00 00 00
>
> Mar 20 14:15:11 732653 [43204960] -> osm_pi_rcv_process_set: Received error
> status 0x1c for SetResp() during ACTIVE transition
> Mar 20 14:15:11 732662 [43204960] -> PortInfo dump:
> port number.............0x16
> node_guid...............0x0005ad00000281b3
> port_guid...............0x0005ad00000281b3
> m_key...................0x0000000000000000
> subnet_prefix...........0x0000000000000000
> base_lid................0x0
> master_sm_base_lid......0x0
> capability_mask.........0x0
> diag_code...............0x0
> m_key_lease_period......0x0
> local_port_num..........0x2
> link_width_enabled......0x3
> link_width_supported....0x3
> link_width_active.......0x2
> link_speed_supported....0x1
> port_state..............ACTIVE
> state_info2.............0x52
> m_key_protect_bits......0x0
> lmc.....................0x0
> link_speed..............0x11
> mtu_smsl................0x40
> vl_cap_init_type........0x40
> vl_high_limit...........0x0
> vl_arb_high_cap.........0x8
> vl_arb_low_cap..........0x8
> init_rep_mtu_cap........0x4
> vl_stall_life...........0xF2
> vl_enforce..............0x40
> m_key_violations........0x0
> p_key_violations........0x0
> q_key_violations........0x0
> guid_cap................0x0
> client_reregister.......0x0
> subnet_timeout..........0x0
> resp_time_value.........0x0
> error_threshold.........0x88
> Mar 20 14:15:11 732673 [43204960] -> Capabilities Mask:
> Mar 20 14:15:11 732705 [4780B960] -> __osm_sm_mad_ctrl_rcv_callback: ERR
> 3111: Error status = 0x1C00
> Mar 20 14:15:11 732739 [4780B960] -> SMP dump:
> base_ver................0x1
> mgmt_class..............0x81
> class_ver...............0x1
> method..................0x81 (SubnGetResp)
> D bit...................0x1
> status..................0x1C00
> hop_ptr.................0x0
> hop_count...............0x4
> trans_id................0x6b50c
> attr_id.................0x15 (PortInfo)
> resv....................0x0
> attr_mod................0x17
> m_key...................0x0000000000000000
> dr_slid.................0xFFFF
> dr_dlid.................0xFFFF
>
> Initial path: [0][1][11][2][5]
> Return path: [0][9][18][E][2]
> Reserved: [0][0][0][0][0][0][0]
>
> 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
>
> 00 00 00 00 00 00 00 00 00 00 00 00 02 03 03 02
>
> 14 52 00 11 40 40 00 08 08 04 F2 40 00 00 00 00
>
> 00 00 00 00 00 88 00 00 00 00 00 00 00 00 00 00
>
> Mar 20 14:15:11 732809 [4780B960] -> __osm_sm_mad_ctrl_rcv_callback: ERR
> 3111: Error status = 0x1C00
> Mar 20 14:15:11 732805 [41E02960] -> osm_pi_rcv_process_set: Received error
> status 0x1c for SetResp() during ACTIVE transition
> Mar 20 14:15:11 732839 [4780B960] -> SMP dump:
> base_ver................0x1
> mgmt_class..............0x81
> class_ver...............0x1
> method..................0x81 (SubnGetResp)
> D bit...................0x1
> status..................0x1C00
> hop_ptr.................0x0
> hop_count...............0x4
> trans_id................0x6b50d
> attr_id.................0x15 (PortInfo)
> resv....................0x0
> attr_mod................0x18
> m_key...................0x0000000000000000
> dr_slid.................0xFFFF
> dr_dlid.................0xFFFF
>
> Initial path: [0][1][11][2][5]
> Return path: [0][9][18][E][2]
> Reserved: [0][0][0][0][0][0][0]
>
> 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
>
> 00 00 00 00 00 00 00 00 00 00 00 00 02 03 03 02
>
> 14 52 00 11 40 40 00 08 08 04 F2 40 00 00 00 00
>
> 00 00 00 00 00 88 00 00 00 00 00 00 00 00 00 00
>
> Mar 20 14:15:11 732837 [41E02960] -> PortInfo dump:
> port number.............0x17
> node_guid...............0x0005ad00000281b3
> port_guid...............0x0005ad00000281b3
> m_key...................0x0000000000000000
> subnet_prefix...........0x0000000000000000
> base_lid................0x0
> master_sm_base_lid......0x0
> capability_mask.........0x0
> diag_code...............0x0
> m_key_lease_period......0x0
> local_port_num..........0x2
> link_width_enabled......0x3
> link_width_supported....0x3
> link_width_active.......0x2
> link_speed_supported....0x1
> port_state..............ACTIVE
> state_info2.............0x52
> m_key_protect_bits......0x0
> lmc.....................0x0
> link_speed..............0x11
> mtu_smsl................0x40
> vl_cap_init_type........0x40
> vl_high_limit...........0x0
> vl_arb_high_cap.........0x8
> vl_arb_low_cap..........0x8
> init_rep_mtu_cap........0x4
> vl_stall_life...........0xF2
> vl_enforce..............0x40
> m_key_violations........0x0
> p_key_violations........0x0
> q_key_violations........0x0
> guid_cap................0x0
> client_reregister.......0x0
> subnet_timeout..........0x0
> resp_time_value.........0x0
> error_threshold.........0x88
> Mar 20 14:15:11 732856 [41E02960] -> Capabilities Mask:
> Mar 20 14:15:11 732898 [41E02960] -> osm_pi_rcv_process_set: Received error
> status 0x1c for SetResp() during ACTIVE transition
> Mar 20 14:15:11 732911 [41E02960] -> PortInfo dump:
> port number.............0x18
> node_guid...............0x0005ad00000281b3
> port_guid...............0x0005ad00000281b3
> m_key...................0x0000000000000000
> subnet_prefix...........0x0000000000000000
> base_lid................0x0
> master_sm_base_lid......0x0
> capability_mask.........0x0
> diag_code...............0x0
> m_key_lease_period......0x0
> local_port_num..........0x2
> link_width_enabled......0x3
> link_width_supported....0x3
> link_width_active.......0x2
> link_speed_supported....0x1
> port_state..............ACTIVE
> state_info2.............0x52
> m_key_protect_bits......0x0
> lmc.....................0x0
> link_speed..............0x11
> mtu_smsl................0x40
> vl_cap_init_type........0x40
> vl_high_limit...........0x0
> vl_arb_high_cap.........0x8
> vl_arb_low_cap..........0x8
> init_rep_mtu_cap........0x4
> vl_stall_life...........0xF2
> vl_enforce..............0x40
> m_key_violations........0x0
> p_key_violations........0x0
> q_key_violations........0x0
> guid_cap................0x0
> client_reregister.......0x0
> subnet_timeout..........0x0
> resp_time_value.........0x0
> error_threshold.........0x88
> Mar 20 14:15:11 732925 [41E02960] -> Capabilities Mask:
> Mar 20 14:15:11 738354 [45A08960] -> SUBNET UP
> Mar 20 14:15:12 115658 [44606960] -> osm_ucast_mgr_process: Min Hop Tables
> configured on all switches
> Mar 20 14:15:12 172029 [44606960] -> SUBNET UP
> Mar 20 14:15:27 277617 [41401960] -> __osm_trap_rcv_process_request:
> Received Generic Notice type:0x01 num:128 Producer:2 from LID:0x0152
> TID:0x0000000000000000
> Mar 20 14:15:27 277863 [41401960] -> osm_report_notice: Reporting Generic
> Notice type:1 num:128 from LID:0x0152
> GID:0xfe80000000000000,0x0005ad0000027c84
> Mar 20 14:15:27 510410 [43C05960] -> __osm_trap_rcv_process_request:
> Received Generic Notice type:0x01 num:128 Producer:2 from LID:0x0152
> TID:0x0000000000000001
> Mar 20 14:15:27 510626 [43C05960] -> osm_report_notice: Reporting Generic
> Notice type:1 num:128 from LID:0x0152
> GID:0xfe80000000000000,0x0005ad0000027c84
> Mar 20 14:15:27 532239 [41E02960] -> __osm_trap_rcv_process_request:
> Received Generic Notice type:0x01 num:128 Producer:2 from LID:0x0152
> TID:0x0000000000000002
> Mar 20 14:15:27 532443 [41E02960] -> osm_report_notice: Reporting Generic
> Notice type:1 num:128 from LID:0x0152
> GID:0xfe80000000000000,0x0005ad0000027c84
> Mar 20 14:15:27 533517 [45A08960] -> __osm_trap_rcv_process_request:
> Received Generic Notice type:0x01 num:128 Producer:2 from LID:0x0152
> TID:0x0000000000000003
> Mar 20 14:15:27 533612 [45A08960] -> osm_report_notice: Reporting Generic
> Notice type:1 num:128 from LID:0x0152
> GID:0xfe80000000000000,0x0005ad0000027c84
> Mar 20 14:15:27 591171 [41401960] -> osm_report_notice: Reporting Generic
> Notice type:3 num:65 from LID:0x0092
> GID:0xfe80000000000000,0x0005ad0000024bbb
> Mar 20 14:15:27 591185 [41401960] -> Removed port with
> GUID:0x0005ad0000024da7 LID range [0xB0,0xB0] of node:saguaro-23-1 HCA-1
> Mar 20 14:15:27 591206 [41401960] -> osm_report_notice: Reporting Generic
> Notice type:3 num:65 from LID:0x0092
> GID:0xfe80000000000000,0x0005ad0000024bbb
> Mar 20 14:15:27 591211 [41401960] -> Removed port with
> GUID:0x0005ad00000249d3 LID range [0xB1,0xB1] of node:saguaro-23-2 HCA-1
> Mar 20 14:15:27 625811 [41401960] -> osm_ucast_mgr_process: Min Hop Tables
> configured on all switches
> Mar 20 14:15:27 668356 [41401960] -> __osm_trap_rcv_process_request:
> Received Generic Notice type:0x01 num:128 Producer:2 from LID:0x0152
> TID:0x0000000000000004
> Mar 20 14:15:27 668485 [41401960] -> osm_report_notice: Reporting Generic
> Notice type:1 num:128 from LID:0x0152
> GID:0xfe80000000000000,0x0005ad0000027c84
> Mar 20 14:15:27 682282 [43204960] -> SUBNET UP
> Mar 20 14:15:27 737313 [41E02960] -> __osm_trap_rcv_process_request:
> Received Generic Notice type:0x01 num:128 Producer:2 from LID:0x0152
> TID:0x0000000000000005
> Mar 20 14:15:27 737387 [41E02960] -> osm_report_notice: Reporting Generic
> Notice type:1 num:128 from LID:0x0152
> GID:0xfe80000000000000,0x0005ad0000027c84
> Mar 20 14:15:27 809341 [42803960] -> __osm_trap_rcv_process_request:
> Received Generic Notice type:0x01 num:128 Producer:2 from LID:0x0152
> TID:0x0000000000000006
> Mar 20 14:15:27 809813 [42803960] -> osm_report_notice: Reporting Generic
> Notice type:1 num:128 from LID:0x0152
> GID:0xfe80000000000000,0x0005ad0000027c84
> Mar 20 14:15:27 998181 [45007960] -> __osm_trap_rcv_process_request:
> Received Generic Notice type:0x01 num:128 Producer:2 from LID:0x0152
> TID:0x0000000000000007
> Mar 20 14:15:27 998331 [45007960] -> osm_report_notice: Reporting Generic
> Notice type:1 num:128 from LID:0x0152
> GID:0xfe80000000000000,0x0005ad0000027c84
> Mar 20 14:15:28 012193 [45007960] -> __osm_trap_rcv_process_request:
> Received Generic Notice type:0x01 num:128 Producer:2 from LID:0x0152
> TID:0x0000000000000008
> Mar 20 14:15:28 012277 [45007960] -> osm_report_notice: Reporting Generic
> Notice type:1 num:128 from LID:0x0152
> GID:0xfe80000000000000,0x0005ad0000027c84
> Mar 20 14:15:28 496329 [43204960] -> __osm_trap_rcv_process_request:
> Received Generic Notice type:0x01 num:128 Producer:2 from LID:0x0152
> TID:0x0000000000000009
> Mar 20 14:15:28 496422 [43204960] -> osm_report_notice: Reporting Generic
> Notice type:1 num:128 from LID:0x0152
> GID:0xfe80000000000000,0x0005ad0000027c84
> Mar 20 14:15:28 624912 [43C05960] -> osm_report_notice: Reporting Generic
> Notice type:3 num:65 from LID:0x0092
> GID:0xfe80000000000000,0x0005ad0000024bbb
> Mar 20 14:15:28 624940 [43C05960] -> Removed port with
> GUID:0x0005ad0000024e0b LID range [0xB3,0xB3] of node:saguaro-23-4 HCA-1
> Mar 20 14:15:28 624965 [43C05960] -> osm_report_notice: Reporting Generic
> Notice type:3 num:65 from LID:0x0092
> GID:0xfe80000000000000,0x0005ad0000024bbb
> Mar 20 14:15:28 624972 [43C05960] -> Removed port with
> GUID:0x0005ad0000025043 LID range [0xB4,0xB4] of node:saguaro-23-5 HCA-1
> Mar 20 14:15:28 625001 [43C05960] -> osm_report_notice: Reporting Generic
> Notice type:3 num:65 from LID:0x0092
> GID:0xfe80000000000000,0x0005ad0000024bbb
> Mar 20 14:15:28 625008 [43C05960] -> Removed port with
> GUID:0x0005ad0000024cbb LID range [0xB2,0xB2] of node:saguaro-23-3 HCA-1
> Mar 20 14:15:28 629507 [42803960] -> osm_report_notice: Reporting Generic
> Notice type:3 num:64 from LID:0x0092
> GID:0xfe80000000000000,0x0005ad0000024bbb
> Mar 20 14:15:28 629518 [42803960] -> Discovered new port with
> GUID:0x0005ad00000249d3 LID range [0xB1,0xB1] of node:saguaro-23-2 HCA-1
> Mar 20 14:15:28 649776 [43204960] -> __osm_trap_rcv_process_request:
> Received Generic Notice type:0x01 num:128 Producer:2 from LID:0x0152
> TID:0x000000000000000a
> Mar 20 14:15:28 660297 [42803960] -> osm_ucast_mgr_process: Min Hop Tables
> configured on all switches
> Mar 20 14:15:28 699777 [43204960] -> osm_report_notice: Reporting Generic
> Notice type:1 num:128 from LID:0x0152
> GID:0xfe80000000000000,0x0005ad0000027c84
> Mar 20 14:15:28 716354 [41E02960] -> SUBNET UP
> Mar 20 14:15:28 744686 [45007960] -> __osm_trap_rcv_process_request:
> Received Generic Notice type:0x01 num:128 Producer:2 from LID:0x0152
> TID:0x000000000000000b
> Mar 20 14:15:28 744857 [45007960] -> __osm_trap_rcv_process_request: ERR
> 3804: Received trap 11 times consecutively
> Mar 20 14:15:28 811329 [45A08960] -> __osm_trap_rcv_process_request:
> Received Generic Notice type:0x01 num:128 Producer:2 from LID:0x0152
> TID:0x000000000000000c
> Mar 20 14:15:28 811392 [45A08960] -> __osm_trap_rcv_process_request: ERR
> 3804: Received trap 12 times consecutively
> Mar 20 14:15:28 999808 [45007960] -> __osm_trap_rcv_process_request:
> Received Generic Notice type:0x01 num:128 Producer:2 from LID:0x0152
> TID:0x000000000000000d
> Mar 20 14:15:28 999881 [45007960] -> __osm_trap_rcv_process_request: ERR
> 3804: Received trap 13 times consecutively
> Mar 20 14:15:29 029918 [43C05960] -> __osm_trap_rcv_process_request:
> Received Generic Notice type:0x01 num:128 Producer:2 from LID:0x0152
> TID:0x000000000000000e
> Mar 20 14:15:29 029969 [43C05960] -> __osm_trap_rcv_process_request: ERR
> 3804: Received trap 14 times consecutively
> Mar 20 14:15:29 031783 [45A08960] -> __osm_trap_rcv_process_request:
> Received Generic Notice type:0x01 num:128 Producer:2 from LID:0x0148
> TID:0x0000000000000052
> Mar 20 14:15:29 031900 [45A08960] -> osm_report_notice: Reporting Generic
> Notice type:1 num:128 from LID:0x0148
> GID:0xfe80000000000000,0x0005ad00000281b3
> Mar 20 14:15:29 037646 [44606960] -> osm_report_notice: Reporting Generic
> Notice type:3 num:65 from LID:0x0092
> GID:0xfe80000000000000,0x0005ad0000024bbb
> Mar 20 14:15:29 037662 [44606960] -> Removed port with
> GUID:0x0005ad0000024d6b LID range [0xB8,0xB8] of node:saguaro-23-9 HCA-1
> Mar 20 14:15:29 037683 [44606960] -> osm_report_notice: Reporting Generic
> Notice type:3 num:65 from LID:0x0092
> GID:0xfe80000000000000,0x0005ad0000024bbb
> Mar 20 14:15:29 037690 [44606960] -> Removed port with
> GUID:0x0005ad0000024977 LID range [0xA9,0xA9] of node:saguaro-22-4 HCA-1
> Mar 20 14:15:29 037721 [44606960] -> osm_report_notice: Reporting Generic
> Notice type:3 num:65 from LID:0x0092
> GID:0xfe80000000000000,0x0005ad0000024bbb
> Mar 20 14:15:29 037726 [44606960] -> Removed port with
> GUID:0x0005ad000002498f LID range [0xA8,0xA8] of node:saguaro-22-3 HCA-1
> Mar 20 14:15:29 037741 [44606960] -> osm_report_notice: Reporting Generic
> Notice type:3 num:65 from LID:0x0092
> GID:0xfe80000000000000,0x0005ad0000024bbb
> Mar 20 14:15:29 037746 [44606960] -> Removed port with
> GUID:0x0005ad0000024c9b LID range [0xA7,0xA7] of node:saguaro-22-2 HCA-1
> Mar 20 14:15:29 037766 [44606960] -> osm_report_notice: Reporting Generic
> Notice type:3 num:65 from LID:0x0092
> GID:0xfe80000000000000,0x0005ad0000024bbb
> Mar 20 14:15:29 037771 [44606960] -> Removed port with
> GUID:0x0005ad0000024afb LID range [0xA5,0xA5] of node:saguaro-22-0 HCA-1
> Mar 20 14:15:29 361560 [42803960] -> __osm_trap_rcv_process_request:
> Received Generic Notice type:0x01 num:128 Producer:2 from LID:0x0148
> TID:0x0000000000000053
> Mar 20 14:15:29 361622 [42803960] -> osm_report_notice: Reporting Generic
> Notice type:1 num:128 from LID:0x0148
> GID:0xfe80000000000000,0x0005ad00000281b3
> Mar 20 14:15:29 433665 [43204960] -> osm_report_notice: Reporting Generic
> Notice type:3 num:64 from LID:0x0092
> GID:0xfe80000000000000,0x0005ad0000024bbb
> Mar 20 14:15:29 433674 [43204960] -> Discovered new port with
> GUID:0x0005ad0000024da7 LID range [0xB0,0xB0] of node:saguaro-23-1 HCA-1
> Mar 20 14:15:29 433680 [43204960] -> osm_report_notice: Reporting Generic
> Notice type:3 num:64 from LID:0x0092
> GID:0xfe80000000000000,0x0005ad0000024bbb
> Mar 20 14:15:29 433687 [43204960] -> Discovered new port with
> GUID:0x0005ad0000024cbb LID range [0xB2,0xB2] of node:saguaro-23-3 HCA-1
> Mar 20 14:15:29 433692 [43204960] -> osm_report_notice: Reporting Generic
> Notice type:3 num:64 from LID:0x0092
> GID:0xfe80000000000000,0x0005ad0000024bbb
> Mar 20 14:15:29 433698 [43204960] -> Discovered new port with
> GUID:0x0005ad0000024e0b LID range [0xB3,0xB3] of node:saguaro-23-4 HCA-1
> Mar 20 14:15:29 433703 [43204960] -> osm_report_notice: Reporting Generic
> Notice type:3 num:64 from LID:0x0092
> GID:0xfe80000000000000,0x0005ad0000024bbb
> Mar 20 14:15:29 433709 [43204960] -> Discovered new port with
> GUID:0x0005ad0000025043 LID range [0xB4,0xB4] of node:saguaro-23-5 HCA-1
> Mar 20 14:15:29 464434 [43204960] -> osm_ucast_mgr_process: Min Hop Tables
> configured on all switches
> Mar 20 14:15:29 522011 [42803960] -> SUBNET UP
> Mar 20 14:15:29 699605 [41E02960] -> __osm_trap_rcv_process_request:
> Received Generic Notice type:0x01 num:128 Producer:2 from LID:0x001B
> TID:0x000000000000006b
> Mar 20 14:15:29 699782 [41E02960] -> osm_report_notice: Reporting Generic
> Notice type:1 num:128 from LID:0x001B
> GID:0xfe80000000000000,0x0005ad00000281a7
> Mar 20 14:15:29 701115 [45A08960] -> __osm_trap_rcv_process_request:
> Received Generic Notice type:0x01 num:128 Producer:2 from LID:0x0148
> TID:0x0000000000000054
> Mar 20 14:15:29 701301 [45A08960] -> osm_report_notice: Reporting Generic
> Notice type:1 num:128 from LID:0x0148
> GID:0xfe80000000000000,0x0005ad00000281b3
> Mar 20 14:15:29 818974 [41E02960] -> __osm_trap_rcv_process_request:
> Received Generic Notice type:0x01 num:128 Producer:2 from LID:0x0148
> TID:0x0000000000000055
> Mar 20 14:15:29 819054 [41E02960] -> osm_report_notice: Reporting Generic
> Notice type:1 num:128 from LID:0x0148
> GID:0xfe80000000000000,0x0005ad00000281b3
> Mar 20 14:15:29 992006 [41E02960] -> __osm_trap_rcv_process_request:
> Received Generic Notice type:0x01 num:128 Producer:2 from LID:0x0148
> TID:0x0000000000000056
> Mar 20 14:15:29 992080 [41E02960] -> osm_report_notice: Reporting Generic
> Notice type:1 num:128 from LID:0x0148
> GID:0xfe80000000000000,0x0005ad00000281b3
> Mar 20 14:15:30 184132 [44606960] -> __osm_trap_rcv_process_request:
> Received Generic Notice type:0x01 num:128 Producer:2 from LID:0x001B
> TID:0x000000000000006c
> Mar 20 14:15:30 184205 [44606960] -> osm_report_notice: Reporting Generic
> Notice type:1 num:128 from LID:0x001B
> GID:0xfe80000000000000,0x0005ad00000281a7
> Mar 20 14:15:30 207030 [43204960] -> __osm_trap_rcv_process_request:
> Received Generic Notice type:0x01 num:128 Producer:2 from LID:0x0148
> TID:0x0000000000000057
> Mar 20 14:15:30 207101 [43204960] -> osm_report_notice: Reporting Generic
> Notice type:1 num:128 from LID:0x0148
> GID:0xfe80000000000000,0x0005ad00000281b3
> Mar 20 14:15:30 250541 [43C05960] -> __osm_trap_rcv_process_request:
> Received Generic Notice type:0x01 num:128 Producer:2 from LID:0x001B
> TID:0x000000000000006d
> Mar 20 14:15:30 250635 [43C05960] -> osm_report_notice: Reporting Generic
> Notice type:1 num:128 from LID:0x001B
> GID:0xfe80000000000000,0x0005ad00000281a7
> Mar 20 14:15:30 317366 [45A08960] -> osm_drop_mgr_process: ERR 0108: Unknown
> remote side for node 0x0005ad00000281a7 port 22. Adding to light sweep
> sampling list
> Mar 20 14:15:30 317409 [45A08960] -> Directed Path Dump of 4 hop path:
> Path = [0][1][17][1][4]
> Mar 20 14:15:30 494183 [41401960] -> __osm_trap_rcv_process_request:
> Received Generic Notice type:0x01 num:128 Producer:2 from LID:0x001B
> TID:0x000000000000006e
> Mar 20 14:15:30 494247 [41401960] -> osm_report_notice: Reporting Generic
> Notice type:1 num:128 from LID:0x001B
> GID:0xfe80000000000000,0x0005ad00000281a7
> Mar 20 14:15:30 521869 [43C05960] -> osm_report_notice: Reporting Generic
> Notice type:3 num:64 from LID:0x0092
> GID:0xfe80000000000000,0x0005ad0000024bbb
> Mar 20 14:15:30 521879 [43C05960] -> Discovered new port with
> GUID:0x0005ad0000024d6b LID range [0xB8,0xB8] of node:saguaro-23-9 HCA-1
> Mar 20 14:15:30 521885 [43C05960] -> osm_report_notice: Reporting Generic
> Notice type:3 num:64 from LID:0x0092
> GID:0xfe80000000000000,0x0005ad0000024bbb
> Mar 20 14:15:30 521891 [43C05960] -> Discovered new port with
> GUID:0x0005ad0000024afb LID range [0xA5,0xA5] of node:saguaro-22-0 HCA-1
> Mar 20 14:15:30 521896 [43C05960] -> osm_report_notice: Reporting Generic
> Notice type:3 num:64 from LID:0x0092
> GID:0xfe80000000000000,0x0005ad0000024bbb
> Mar 20 14:15:30 521903 [43C05960] -> Discovered new port with
> GUID:0x0005ad0000024c9b LID range [0xA7,0xA7] of node:saguaro-22-2 HCA-1
> Mar 20 14:15:30 521908 [43C05960] -> osm_report_notice: Reporting Generic
> Notice type:3 num:64 from LID:0x0092
> GID:0xfe80000000000000,0x0005ad0000024bbb
> Mar 20 14:15:30 521914 [43C05960] -> Discovered new port with
> GUID:0x0005ad000002498f LID range [0xA8,0xA8] of node:saguaro-22-3 HCA-1
> Mar 20 14:15:30 521919 [43C05960] -> osm_report_notice: Reporting Generic
> Notice type:3 num:64 from LID:0x0092
> GID:0xfe80000000000000,0x0005ad0000024bbb
> Mar 20 14:15:30 521926 [43C05960] -> Discovered new port with
> GUID:0x0005ad0000024977 LID range [0xA9,0xA9] of node:saguaro-22-4 HCA-1
> Mar 20 14:15:30 552581 [43C05960] -> osm_ucast_mgr_process: Min Hop Tables
> configured on all switches
> Mar 20 14:15:30 553014 [45A08960] -> __osm_trap_rcv_process_request:
> Received Generic Notice type:0x01 num:128 Producer:2 from LID:0x001B
> TID:0x000000000000006f
> Mar 20 14:15:30 592863 [45A08960] -> osm_report_notice: Reporting Generic
> Notice type:1 num:128 from LID:0x001B
> GID:0xfe80000000000000,0x0005ad00000281a7
> Mar 20 14:15:30 607595 [4780B960] -> __osm_sm_mad_ctrl_rcv_callback: ERR
> 3111: Error status = 0x1C00
> Mar 20 14:15:30 607666 [4780B960] -> SMP dump:
> base_ver................0x1
> mgmt_class..............0x81
> class_ver...............0x1
> method..................0x81 (SubnGetResp)
> D bit...................0x1
> status..................0x1C00
> hop_ptr.................0x0
> hop_count...............0x4
> trans_id................0x6f744
> attr_id.................0x15 (PortInfo)
> resv....................0x0
> attr_mod................0x16
> m_key...................0x0000000000000000
> dr_slid.................0xFFFF
> dr_dlid.................0xFFFF
>
> Initial path: [0][1][14][1][6]
> Return path: [0][9][15][D][3]
> Reserved: [0][0][0][0][0][0][0]
>
> 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
>
> 00 00 00 00 00 00 00 00 00 00 00 00 03 03 03 02
>
> 14 52 00 11 40 40 00 08 08 04 F2 40 00 00 00 00
>
> 00 00 00 00 00 88 00 00 00 00 00 00 00 00 00 00
>
> Mar 20 14:15:30 607770 [44606960] -> osm_pi_rcv_process_set: Received error
> status 0x1c for SetResp() during ACTIVE transition
> Mar 20 14:15:30 607777 [44606960] -> PortInfo dump:
> port number.............0x16
> node_guid...............0x0005ad00000281b3
> port_guid...............0x0005ad00000281b3
> m_key...................0x0000000000000000
> subnet_prefix...........0x0000000000000000
> base_lid................0x0
> master_sm_base_lid......0x0
> capability_mask.........0x0
> diag_code...............0x0
> m_key_lease_period......0x0
> local_port_num..........0x3
> link_width_enabled......0x3
> link_width_supported....0x3
> link_width_active.......0x2
> link_speed_supported....0x1
> port_state..............ACTIVE
> state_info2.............0x52
> m_key_protect_bits......0x0
> lmc.....................0x0
> link_speed..............0x11
> mtu_smsl................0x40
> vl_cap_init_type........0x40
> vl_high_limit...........0x0
> vl_arb_high_cap.........0x8
> vl_arb_low_cap..........0x8
> init_rep_mtu_cap........0x4
> vl_stall_life...........0xF2
> vl_enforce..............0x40
> m_key_violations........0x0
> p_key_violations........0x0
> q_key_violations........0x0
> guid_cap................0x0
> client_reregister.......0x0
> subnet_timeout..........0x0
> resp_time_value.........0x0
> error_threshold.........0x88
> Mar 20 14:15:30 607794 [44606960] -> Capabilities Mask:
> Mar 20 14:15:30 607914 [4780B960] -> __osm_sm_mad_ctrl_rcv_callback: ERR
> 3111: Error status = 0x1C00
> Mar 20 14:15:30 607958 [4780B960] -> SMP dump:
> base_ver................0x1
> mgmt_class..............0x81
> class_ver...............0x1
> method..................0x81 (SubnGetResp)
> D bit...................0x1
> status..................0x1C00
> hop_ptr.................0x0
> hop_count...............0x4
> trans_id................0x6f745
> attr_id.................0x15 (PortInfo)
> resv....................0x0
> attr_mod................0x17
> m_key...................0x0000000000000000
> dr_slid.................0xFFFF
> dr_dlid.................0xFFFF
>
> Initial path: [0][1][14][1][6]
> Return path: [0][9][15][D][3]
> Reserved: [0][0][0][0][0][0][0]
>
> 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
>
> 00 00 00 00 00 00 00 00 00 00 00 00 03 03 03 02
>
> 14 52 00 11 40 40 00 08 08 04 F2 40 00 00 00 00
>
> 00 00 00 00 00 88 00 00 00 00 00 00 00 00 00 00
>
> Mar 20 14:15:30 608014 [43204960] -> osm_pi_rcv_process_set: Received error
> status 0x1c for SetResp() during ACTIVE transition
> Mar 20 14:15:30 608018 [43204960] -> PortInfo dump:
> port number.............0x17
> node_guid...............0x0005ad00000281b3
> port_guid...............0x0005ad00000281b3
> m_key...................0x0000000000000000
> subnet_prefix...........0x0000000000000000
> base_lid................0x0
> master_sm_base_lid......0x0
> capability_mask.........0x0
> diag_code...............0x0
> m_key_lease_period......0x0
> local_port_num..........0x3
> link_width_enabled......0x3
> link_width_supported....0x3
> link_width_active.......0x2
> link_speed_supported....0x1
> port_state..............ACTIVE
> state_info2.............0x52
> m_key_protect_bits......0x0
> lmc.....................0x0
> link_speed..............0x11
> mtu_smsl................0x40
> vl_cap_init_type........0x40
> vl_high_limit...........0x0
> vl_arb_high_cap.........0x8
> vl_arb_low_cap..........0x8
> init_rep_mtu_cap........0x4
> vl_stall_life...........0xF2
> vl_enforce..............0x40
> m_key_violations........0x0
> p_key_violations........0x0
> q_key_violations........0x0
> guid_cap................0x0
> client_reregister.......0x0
> subnet_timeout..........0x0
> resp_time_value.........0x0
> error_threshold.........0x88
> Mar 20 14:15:30 608031 [43204960] -> Capabilities Mask:
> Mar 20 14:15:30 613309 [41E02960] -> SUBNET UP
> Mar 20 14:15:30 995108 [41401960] -> osm_ucast_mgr_process: Min Hop Tables
> configured on all switches
> Mar 20 14:15:31 050102 [4780B960] -> __osm_sm_mad_ctrl_rcv_callback: ERR
> 3111: Error status = 0x1C00
> Mar 20 14:15:31 050180 [4780B960] -> SMP dump:
> base_ver................0x1
> mgmt_class..............0x81
> class_ver...............0x1
> method..................0x81 (SubnGetResp)
> D bit...................0x1
> status..................0x1C00
> hop_ptr.................0x0
> hop_count...............0x4
> trans_id................0x70486
> attr_id.................0x15 (PortInfo)
> resv....................0x0
> attr_mod................0x18
> m_key...................0x0000000000000000
> dr_slid.................0xFFFF
> dr_dlid.................0xFFFF
>
> Initial path: [0][1][11][1][4]
> Return path: [0][9][18][D][1]
> Reserved: [0][0][0][0][0][0][0]
>
> 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
>
> 00 00 00 00 00 00 00 00 00 00 00 00 01 03 03 02
>
> 14 52 00 11 40 40 00 08 08 04 F2 40 00 00 00 00
>
> 00 00 00 00 00 88 00 00 00 00 00 00 00 00 00 00
>
> Mar 20 14:15:31 050233 [45A08960] -> osm_pi_rcv_process_set: Received error
> status 0x1c for SetResp() during ACTIVE transition
> Mar 20 14:15:31 050238 [45A08960] -> PortInfo dump:
> port number.............0x18
> node_guid...............0x0005ad00000281b3
> port_guid...............0x0005ad00000281b3
> m_key...................0x0000000000000000
> subnet_prefix...........0x0000000000000000
> base_lid................0x0
> master_sm_base_lid......0x0
> capability_mask.........0x0
> diag_code...............0x0
> m_key_lease_period......0x0
> local_port_num..........0x1
> link_width_enabled......0x3
> link_width_supported....0x3
> link_width_active.......0x2
> link_speed_supported....0x1
> port_state..............ACTIVE
> state_info2.............0x52
> m_key_protect_bits......0x0
> lmc.....................0x0
> link_speed..............0x11
> mtu_smsl................0x40
> vl_cap_init_type........0x40
> vl_high_limit...........0x0
> vl_arb_high_cap.........0x8
> vl_arb_low_cap..........0x8
> init_rep_mtu_cap........0x4
> vl_stall_life...........0xF2
> vl_enforce..............0x40
> m_key_violations........0x0
> p_key_violations........0x0
> q_key_violations........0x0
> guid_cap................0x0
> client_reregister.......0x0
> subnet_timeout..........0x0
> resp_time_value.........0x0
> error_threshold.........0x88
> Mar 20 14:15:31 050251 [45A08960] -> Capabilities Mask:
> Mar 20 14:15:31 055273 [42803960] -> SUBNET UP
> Mar 20 14:15:31 106129 [41401960] -> __osm_trap_rcv_process_request:
> Received Generic Notice type:0x01 num:128 Producer:2 from LID:0x0152
> TID:0x000000000000000e
> Mar 20 14:15:31 106193 [41401960] -> __osm_trap_rcv_process_request: ERR
> 3804: Received trap 15 times consecutively
> Mar 20 14:17:18 456260 [43204960] -> __osm_trap_rcv_process_request:
> Received Generic Notice type:0x01 num:128 Producer:2 from LID:0x0148
> TID:0x0000000000000058
> Mar 20 14:17:18 456512 [43204960] -> osm_report_notice: Reporting Generic
> Notice type:1 num:128 from LID:0x0148
> GID:0xfe80000000000000,0x0005ad00000281b3
> Mar 20 14:17:18 456649 [41E02960] -> __osm_trap_rcv_process_request:
> Received Generic Notice type:0x01 num:128 Producer:2 from LID:0x001B
> TID:0x0000000000000070
> Mar 20 14:17:18 456761 [41E02960] -> osm_report_notice: Reporting Generic
> Notice type:1 num:128 from LID:0x001B
> GID:0xfe80000000000000,0x0005ad00000281a7
> Mar 20 14:17:18 769730 [45007960] -> osm_report_notice: Reporting Generic
> Notice type:3 num:65 from LID:0x0092
> GID:0xfe80000000000000,0x0005ad0000024bbb
> Mar 20 14:17:18 769751 [45007960] -> Removed port with
> GUID:0x0005ad0000024e0b LID range [0xB3,0xB3] of node:saguaro-23-4 HCA-1
> Mar 20 14:17:18 769773 [45007960] -> osm_report_notice: Reporting Generic
> Notice type:3 num:65 from LID:0x0092
> GID:0xfe80000000000000,0x0005ad0000024bbb
> Mar 20 14:17:18 769780 [45007960] -> Removed port with
> GUID:0x0005ad000002510b LID range [0xB5,0xB5] of node:saguaro-23-6 HCA-1
> Mar 20 14:17:18 769803 [45007960] -> osm_report_notice: Reporting Generic
> Notice type:3 num:65 from LID:0x0092
> GID:0xfe80000000000000,0x0005ad0000024bbb
> Mar 20 14:17:18 769809 [45007960] -> Removed port with
> GUID:0x0005ad000002511b LID range [0xA6,0xA6] of node:saguaro-22-1 HCA-1
> Mar 20 14:17:18 769832 [45007960] -> osm_report_notice: Reporting Generic
> Notice type:3 num:65 from LID:0x0092
> GID:0xfe80000000000000,0x0005ad0000024bbb
> Mar 20 14:17:18 769838 [45007960] -> Removed port with
> GUID:0x0005ad0000025043 LID range [0xB4,0xB4] of node:saguaro-23-5 HCA-1
> Mar 20 14:17:18 769858 [45007960] -> osm_report_notice: Reporting Generic
> Notice type:3 num:65 from LID:0x0092
> GID:0xfe80000000000000,0x0005ad0000024bbb
> Mar 20 14:17:18 769865 [45007960] -> Removed port with
> GUID:0x0005ad0000024d47 LID range [0xB6,0xB6] of node:saguaro-23-7 HCA-1
> Mar 20 14:17:18 769888 [45007960] -> osm_report_notice: Reporting Generic
> Notice type:3 num:65 from LID:0x0092
> GID:0xfe80000000000000,0x0005ad0000024bbb
> Mar 20 14:17:18 769895 [45007960] -> Removed port with
> GUID:0x0005ad0000024d6b LID range [0xB8,0xB8] of node:saguaro-23-9 HCA-1
> Mar 20 14:17:18 769927 [45007960] -> osm_report_notice: Reporting Generic
> Notice type:3 num:65 from LID:0x0092
> GID:0xfe80000000000000,0x0005ad0000024bbb
> Mar 20 14:17:18 769932 [45007960] -> Removed port with
> GUID:0x0005ad0000024977 LID range [0xA9,0xA9] of node:saguaro-22-4 HCA-1
> Mar 20 14:17:18 770075 [45007960] -> osm_report_notice: Reporting Generic
> Notice type:3 num:65 from LID:0x0092
> GID:0xfe80000000000000,0x0005ad0000024bbb
> Mar 20 14:17:18 770081 [45007960] -> Removed port with
> GUID:0x0005ad0000027c84 LID range [0x152,0x152] of node:Topspin Switch TS120
> Mar 20 14:17:18 770109 [45007960] -> osm_report_notice: Reporting Generic
> Notice type:3 num:65 from LID:0x0092
> GID:0xfe80000000000000,0x0005ad0000024bbb
> Mar 20 14:17:18 770114 [45007960] -> Removed port with
> GUID:0x0005ad0000024d8b LID range [0xB7,0xB7] of node:saguaro-23-8 HCA-1
> Mar 20 14:17:18 770130 [45007960] -> osm_report_notice: Reporting Generic
> Notice type:3 num:65 from LID:0x0092
> GID:0xfe80000000000000,0x0005ad0000024bbb
> Mar 20 14:17:18 770135 [45007960] -> Removed port with
> GUID:0x0005ad000002498f LID range [0xA8,0xA8] of node:saguaro-22-3 HCA-1
> Mar 20 14:17:18 770150 [45007960] -> osm_report_notice: Reporting Generic
> Notice type:3 num:65 from LID:0x0092
> GID:0xfe80000000000000,0x0005ad0000024bbb
> Mar 20 14:17:18 770155 [45007960] -> Removed port with
> GUID:0x0005ad0000024c9b LID range [0xA7,0xA7] of node:saguaro-22-2 HCA-1
> Mar 20 14:17:18 770171 [45007960] -> osm_report_notice: Reporting Generic
> Notice type:3 num:65 from LID:0x0092
> GID:0xfe80000000000000,0x0005ad0000024bbb
> Mar 20 14:17:18 770176 [45007960] -> Removed port with
> GUID:0x0005ad0000024da7 LID range [0xB0,0xB0] of node:saguaro-23-1 HCA-1
> Mar 20 14:17:18 770193 [45007960] -> osm_report_notice: Reporting Generic
> Notice type:3 num:65 from LID:0x0092
> GID:0xfe80000000000000,0x0005ad0000024bbb
> Mar 20 14:17:18 770198 [45007960] -> Removed port with
> GUID:0x0005ad0000024cbb LID range [0xB2,0xB2] of node:saguaro-23-3 HCA-1
> Mar 20 14:17:18 770216 [45007960] -> osm_report_notice: Reporting Generic
> Notice type:3 num:65 from LID:0x0092
> GID:0xfe80000000000000,0x0005ad0000024bbb
> Mar 20 14:17:18 770221 [45007960] -> Removed port with
> GUID:0x0005ad00000249d3 LID range [0xB1,0xB1] of node:saguaro-23-2 HCA-1
> Mar 20 14:17:18 770238 [45007960] -> osm_report_notice: Reporting Generic
> Notice type:3 num:65 from LID:0x0092
> GID:0xfe80000000000000,0x0005ad0000024bbb
> Mar 20 14:17:18 770301 [45007960] -> Removed port with
> GUID:0x0005ad0000024feb LID range [0x153,0x153] of node:saguaro-22-5 HCA-1
> Mar 20 14:17:18 770318 [45007960] -> osm_report_notice: Reporting Generic
> Notice type:3 num:65 from LID:0x0092
> GID:0xfe80000000000000,0x0005ad0000024bbb
> Mar 20 14:17:18 770323 [45007960] -> Removed port with
> GUID:0x0005ad0000024afb LID range [0xA5,0xA5] of node:saguaro-22-0 HCA-1
> Mar 20 14:17:18 803377 [45007960] -> osm_ucast_mgr_process: Min Hop Tables
> configured on all switches
> Mar 20 14:17:18 855545 [44606960] -> SUBNET UP
> Mar 20 14:17:19 249722 [43204960] -> osm_ucast_mgr_process: Min Hop Tables
> configured on all switches
> Mar 20 14:17:19 300999 [45A08960] -> SUBNET UP
> Mar 20 14:18:11 663850 [43C05960] -> __osm_trap_rcv_process_request:
> Received Generic Notice type:0x01 num:128 Producer:2 from LID:0x0148
> TID:0x0000000000000059
> Mar 20 14:18:11 664195 [43C05960] -> osm_report_notice: Reporting Generic
> Notice type:1 num:128 from LID:0x0148
> GID:0xfe80000000000000,0x0005ad00000281b3
> Mar 20 14:18:11 670836 [41E02960] -> __osm_trap_rcv_process_request:
> Received Generic Notice type:0x01 num:128 Producer:2 from LID:0x001B
> TID:0x0000000000000071
> Mar 20 14:18:11 670964 [41401960] -> __osm_trap_rcv_process_request:
> Received Generic Notice type:0x01 num:128 Producer:2 from LID:0x0148
> TID:0x000000000000005a
> Mar 20 14:18:11 671199 [41E02960] -> osm_report_notice: Reporting Generic
> Notice type:1 num:128 from LID:0x001B
> GID:0xfe80000000000000,0x0005ad00000281a7
> Mar 20 14:18:11 672933 [41401960] -> osm_report_notice: Reporting Generic
> Notice type:1 num:128 from LID:0x0148
> GID:0xfe80000000000000,0x0005ad00000281b3
> Mar 20 14:18:11 677654 [44606960] -> __osm_trap_rcv_process_request:
> Received Generic Notice type:0x01 num:128 Producer:2 from LID:0x001B
> TID:0x0000000000000072
> Mar 20 14:18:11 677826 [44606960] -> osm_report_notice: Reporting Generic
> Notice type:1 num:128 from LID:0x001B
> GID:0xfe80000000000000,0x0005ad00000281a7
> Mar 20 14:18:12 026661 [44606960] -> osm_report_notice: Reporting Generic
> Notice type:3 num:64 from LID:0x0092
> GID:0xfe80000000000000,0x0005ad0000024bbb
> Mar 20 14:18:12 026675 [44606960] -> Discovered new port with
> GUID:0x0005ad0000027c84 LID range [0x152,0x152] of node:Topspin Switch TS120
> Mar 20 14:18:12 026681 [44606960] -> osm_report_notice: Reporting Generic
> Notice type:3 num:64 from LID:0x0092
> GID:0xfe80000000000000,0x0005ad0000024bbb
> Mar 20 14:18:12 026688 [44606960] -> Discovered new port with
> GUID:0x0005ad0000024da7 LID range [0xB0,0xB0] of node:saguaro-23-1 HCA-1
> Mar 20 14:18:12 026693 [44606960] -> osm_report_notice: Reporting Generic
> Notice type:3 num:64 from LID:0x0092
> GID:0xfe80000000000000,0x0005ad0000024bbb
> Mar 20 14:18:12 026700 [44606960] -> Discovered new port with
> GUID:0x0005ad00000249d3 LID range [0xB1,0xB1] of node:saguaro-23-2 HCA-1
> Mar 20 14:18:12 026705 [44606960] -> osm_report_notice: Reporting Generic
> Notice type:3 num:64 from LID:0x0092
> GID:0xfe80000000000000,0x0005ad0000024bbb
> Mar 20 14:18:12 026711 [44606960] -> Discovered new port with
> GUID:0x0005ad0000024cbb LID range [0xB2,0xB2] of node:saguaro-23-3 HCA-1
> Mar 20 14:18:12 026716 [44606960] -> osm_report_notice: Reporting Generic
> Notice type:3 num:64 from LID:0x0092
> GID:0xfe80000000000000,0x0005ad0000024bbb
> Mar 20 14:18:12 026723 [44606960] -> Discovered new port with
> GUID:0x0005ad0000024e0b LID range [0xB3,0xB3] of node:saguaro-23-4 HCA-1
> Mar 20 14:18:12 026727 [44606960] -> osm_report_notice: Reporting Generic
> Notice type:3 num:64 from LID:0x0092
> GID:0xfe80000000000000,0x0005ad0000024bbb
> Mar 20 14:18:12 026740 [44606960] -> Discovered new port with
> GUID:0x0005ad0000025043 LID range [0xB4,0xB4] of node:saguaro-23-5 HCA-1
> Mar 20 14:18:12 026745 [44606960] -> osm_report_notice: Reporting Generic
> Notice type:3 num:64 from LID:0x0092
> GID:0xfe80000000000000,0x0005ad0000024bbb
> Mar 20 14:18:12 026751 [44606960] -> Discovered new port with
> GUID:0x0005ad000002510b LID range [0xB5,0xB5] of node:saguaro-23-6 HCA-1
> Mar 20 14:18:12 026758 [44606960] -> osm_report_notice: Reporting Generic
> Notice type:3 num:64 from LID:0x0092
> GID:0xfe80000000000000,0x0005ad0000024bbb
> Mar 20 14:18:12 026764 [44606960] -> Discovered new port with
> GUID:0x0005ad0000024d47 LID range [0xB6,0xB6] of node:saguaro-23-7 HCA-1
> Mar 20 14:18:12 026769 [44606960] -> osm_report_notice: Reporting Generic
> Notice type:3 num:64 from LID:0x0092
> GID:0xfe80000000000000,0x0005ad0000024bbb
> Mar 20 14:18:12 026776 [44606960] -> Discovered new port with
> GUID:0x0005ad0000024d8b LID range [0xB7,0xB7] of node:saguaro-23-8 HCA-1
> Mar 20 14:18:12 026781 [44606960] -> osm_report_notice: Reporting Generic
> Notice type:3 num:64 from LID:0x0092
> GID:0xfe80000000000000,0x0005ad0000024bbb
> Mar 20 14:18:12 026787 [44606960] -> Discovered new port with
> GUID:0x0005ad0000024d6b LID range [0xB8,0xB8] of node:saguaro-23-9 HCA-1
> Mar 20 14:18:12 026792 [44606960] -> osm_report_notice: Reporting Generic
> Notice type:3 num:64 from LID:0x0092
> GID:0xfe80000000000000,0x0005ad0000024bbb
> Mar 20 14:18:12 026798 [44606960] -> Discovered new port with
> GUID:0x0005ad0000024afb LID range [0xA5,0xA5] of node:saguaro-22-0 HCA-1
> Mar 20 14:18:12 026803 [44606960] -> osm_report_notice: Reporting Generic
> Notice type:3 num:64 from LID:0x0092
> GID:0xfe80000000000000,0x0005ad0000024bbb
> Mar 20 14:18:12 026809 [44606960] -> Discovered new port with
> GUID:0x0005ad000002511b LID range [0xA6,0xA6] of node:saguaro-22-1 HCA-1
> Mar 20 14:18:12 026814 [44606960] -> osm_report_notice: Reporting Generic
> Notice type:3 num:64 from LID:0x0092
> GID:0xfe80000000000000,0x0005ad0000024bbb
> Mar 20 14:18:12 026821 [44606960] -> Discovered new port with
> GUID:0x0005ad0000024c9b LID range [0xA7,0xA7] of node:saguaro-22-2 HCA-1
> Mar 20 14:18:12 026826 [44606960] -> osm_report_notice: Reporting Generic
> Notice type:3 num:64 from LID:0x0092
> GID:0xfe80000000000000,0x0005ad0000024bbb
> Mar 20 14:18:12 026832 [44606960] -> Discovered new port with
> GUID:0x0005ad000002498f LID range [0xA8,0xA8] of node:saguaro-22-3 HCA-1
> Mar 20 14:18:12 026869 [44606960] -> osm_report_notice: Reporting Generic
> Notice type:3 num:64 from LID:0x0092
> GID:0xfe80000000000000,0x0005ad0000024bbb
> Mar 20 14:18:12 026877 [44606960] -> Discovered new port with
> GUID:0x0005ad0000024977 LID range [0xA9,0xA9] of node:saguaro-22-4 HCA-1
> Mar 20 14:18:12 026882 [44606960] -> osm_report_notice: Reporting Generic
> Notice type:3 num:64 from LID:0x0092
> GID:0xfe80000000000000,0x0005ad0000024bbb
> Mar 20 14:18:12 026888 [44606960] -> Discovered new port with
> GUID:0x0005ad0000024feb LID range [0x153,0x153] of node:saguaro-22-5 HCA-1
> Mar 20 14:18:12 057534 [44606960] -> osm_ucast_mgr_process: Min Hop Tables
> configured on all switches
> Mar 20 14:18:12 133316 [4780B960] -> __osm_sm_mad_ctrl_rcv_callback: ERR
> 3111: Error status = 0x1C00
> Mar 20 14:18:12 133419 [4780B960] -> SMP dump:
> base_ver................0x1
> mgmt_class..............0x81
> class_ver...............0x1
> method..................0x81 (SubnGetResp)
> D bit...................0x1
> status..................0x1C00
> hop_ptr.................0x0
> hop_count...............0x4
> trans_id................0x72d97
> attr_id.................0x15 (PortInfo)
> resv....................0x0
> attr_mod................0x16
> m_key...................0x0000000000000000
> dr_slid.................0xFFFF
> dr_dlid.................0xFFFF
>
> Initial path: [0][1][14][3][6]
> Return path: [0][9][15][F][3]
> Reserved: [0][0][0][0][0][0][0]
>
> 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
>
> 00 00 00 00 00 00 00 00 00 00 00 00 03 03 03 02
>
> 14 52 00 11 40 40 00 08 08 04 F2 40 00 00 00 00
>
> 00 00 00 00 00 88 00 00 00 00 00 00 00 00 00 00
>
> Mar 20 14:18:12 133466 [4780B960] -> __osm_sm_mad_ctrl_rcv_callback: ERR
> 3111: Error status = 0x1C00
> Mar 20 14:18:12 133467 [43204960] -> osm_pi_rcv_process_set: Received error
> status 0x1c for SetResp() during ACTIVE transition
> Mar 20 14:18:12 133478 [43204960] -> PortInfo dump:
> port number.............0x16
> node_guid...............0x0005ad00000281b3
> port_guid...............0x0005ad00000281b3
> m_key...................0x0000000000000000
> subnet_prefix...........0x0000000000000000
> base_lid................0x0
> master_sm_base_lid......0x0
> capability_mask.........0x0
> diag_code...............0x0
> m_key_lease_period......0x0
> local_port_num..........0x3
> link_width_enabled......0x3
> link_width_supported....0x3
> link_width_active.......0x2
> link_speed_supported....0x1
> port_state..............ACTIVE
> state_info2.............0x52
> m_key_protect_bits......0x0
> lmc.....................0x0
> link_speed..............0x11
> mtu_smsl................0x40
> vl_cap_init_type........0x40
> vl_high_limit...........0x0
> vl_arb_high_cap.........0x8
> vl_arb_low_cap..........0x8
> init_rep_mtu_cap........0x4
> vl_stall_life...........0xF2
> vl_enforce..............0x40
> m_key_violations........0x0
> p_key_violations........0x0
> q_key_violations........0x0
> guid_cap................0x0
> client_reregister.......0x0
> subnet_timeout..........0x0
> resp_time_value.........0x0
> error_threshold.........0x88
> Mar 20 14:18:12 133490 [4780B960] -> SMP dump:
> base_ver................0x1
> mgmt_class..............0x81
> class_ver...............0x1
> method..................0x81 (SubnGetResp)
> D bit...................0x1
> status..................0x1C00
> hop_ptr.................0x0
> hop_count...............0x4
> trans_id................0x72d98
> attr_id.................0x15 (PortInfo)
> resv....................0x0
> attr_mod................0x17
> m_key...................0x0000000000000000
> dr_slid.................0xFFFF
> dr_dlid.................0xFFFF
>
> Initial path: [0][1][14][3][6]
> Return path: [0][9][15][F][3]
> Reserved: [0][0][0][0][0][0][0]
>
> 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
>
> 00 00 00 00 00 00 00 00 00 00 00 00 03 03 03 02
>
> 14 52 00 11 40 40 00 08 08 04 F2 40 00 00 00 00
>
> 00 00 00 00 00 88 00 00 00 00 00 00 00 00 00 00
>
> Mar 20 14:18:12 133493 [43204960] -> Capabilities Mask:
> Mar 20 14:18:12 133566 [45A08960] -> osm_pi_rcv_process_set: Received error
> status 0x1c for SetResp() during ACTIVE transition
> Mar 20 14:18:12 133595 [45A08960] -> PortInfo dump:
> port number.............0x17
> node_guid...............0x0005ad00000281b3
> port_guid...............0x0005ad00000281b3
> m_key...................0x0000000000000000
> subnet_prefix...........0x0000000000000000
> base_lid................0x0
> master_sm_base_lid......0x0
> capability_mask.........0x0
> diag_code...............0x0
> m_key_lease_period......0x0
> local_port_num..........0x3
> link_width_enabled......0x3
> link_width_supported....0x3
> link_width_active.......0x2
> link_speed_supported....0x1
> port_state..............ACTIVE
> state_info2.............0x52
> m_key_protect_bits......0x0
> lmc.....................0x0
> link_speed..............0x11
> mtu_smsl................0x40
> vl_cap_init_type........0x40
> vl_high_limit...........0x0
> vl_arb_high_cap.........0x8
> vl_arb_low_cap..........0x8
> init_rep_mtu_cap........0x4
> vl_stall_life...........0xF2
> vl_enforce..............0x40
> m_key_violations........0x0
> p_key_violations........0x0
> q_key_violations........0x0
> guid_cap................0x0
> client_reregister.......0x0
> subnet_timeout..........0x0
> resp_time_value.........0x0
> error_threshold.........0x88
> Mar 20 14:18:12 133614 [45A08960] -> Capabilities Mask:
> Mar 20 14:18:12 133583 [4780B960] -> __osm_sm_mad_ctrl_rcv_callback: ERR
> 3111: Error status = 0x1C00
> Mar 20 14:18:12 133671 [4780B960] -> SMP dump:
> base_ver................0x1
> mgmt_class..............0x81
> class_ver...............0x1
> method..................0x81 (SubnGetResp)
> D bit...................0x1
> status..................0x1C00
> hop_ptr.................0x0
> hop_count...............0x4
> trans_id................0x72d99
> attr_id.................0x15 (PortInfo)
> resv....................0x0
> attr_mod................0x18
> m_key...................0x0000000000000000
> dr_slid.................0xFFFF
> dr_dlid.................0xFFFF
>
> Initial path: [0][1][14][3][6]
> Return path: [0][9][15][F][3]
> Reserved: [0][0][0][0][0][0][0]
>
> 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
>
> 00 00 00 00 00 00 00 00 00 00 00 00 03 03 03 02
>
> 14 52 00 11 40 40 00 08 08 04 F2 40 00 00 00 00
>
> 00 00 00 00 00 88 00 00 00 00 00 00 00 00 00 00
>
> Mar 20 14:18:12 133760 [43C05960] -> osm_pi_rcv_process_set: Received error
> status 0x1c for SetResp() during ACTIVE transition
> Mar 20 14:18:12 133788 [43C05960] -> PortInfo dump:
> port number.............0x18
> node_guid...............0x0005ad00000281b3
> port_guid...............0x0005ad00000281b3
> m_key...................0x0000000000000000
> subnet_prefix...........0x0000000000000000
> base_lid................0x0
> master_sm_base_lid......0x0
> capability_mask.........0x0
> diag_code...............0x0
> m_key_lease_period......0x0
> local_port_num..........0x3
> link_width_enabled......0x3
> link_width_supported....0x3
> link_width_active.......0x2
> link_speed_supported....0x1
> port_state..............ACTIVE
> state_info2.............0x52
> m_key_protect_bits......0x0
> lmc.....................0x0
> link_speed..............0x11
> mtu_smsl................0x40
> vl_cap_init_type........0x40
> vl_high_limit...........0x0
> vl_arb_high_cap.........0x8
> vl_arb_low_cap..........0x8
> init_rep_mtu_cap........0x4
> vl_stall_life...........0xF2
> vl_enforce..............0x40
> m_key_violations........0x0
> p_key_violations........0x0
> q_key_violations........0x0
> guid_cap................0x0
> client_reregister.......0x0
> subnet_timeout..........0x0
> resp_time_value.........0x0
> error_threshold.........0x88
> Mar 20 14:18:12 133807 [43C05960] -> Capabilities Mask:
> Mar 20 14:18:12 139330 [41401960] -> SUBNET UP
> Mar 20 14:18:12 496444 [45A08960] -> osm_ucast_mgr_process: Min Hop Tables
> configured on all switches
> Mar 20 14:18:12 558965 [41401960] -> SUBNET UP
> Mar 20 14:18:27 748551 [43C05960] -> __osm_trap_rcv_process_request:
> Received Generic Notice type:0x01 num:128 Producer:2 from LID:0x0152
> TID:0x0000000000000000
> Mar 20 14:18:27 748795 [43C05960] -> osm_report_notice: Reporting Generic
> Notice type:1 num:128 from LID:0x0152
> GID:0xfe80000000000000,0x0005ad0000027c84
> Mar 20 14:18:27 888669 [44606960] -> __osm_trap_rcv_process_request:
> Received Generic Notice type:0x01 num:128 Producer:2 from LID:0x0152
> TID:0x0000000000000001
> Mar 20 14:18:27 888902 [44606960] -> osm_report_notice: Reporting Generic
> Notice type:1 num:128 from LID:0x0152
> GID:0xfe80000000000000,0x0005ad0000027c84
> Mar 20 14:18:27 910605 [44606960] -> __osm_trap_rcv_process_request:
> Received Generic Notice type:0x01 num:128 Producer:2 from LID:0x0152
> TID:0x0000000000000002
> Mar 20 14:18:27 910710 [44606960] -> osm_report_notice: Reporting Generic
> Notice type:1 num:128 from LID:0x0152
> GID:0xfe80000000000000,0x0005ad0000027c84
> Mar 20 14:18:27 911951 [41E02960] -> __osm_trap_rcv_process_request:
> Received Generic Notice type:0x01 num:128 Producer:2 from LID:0x0152
> TID:0x0000000000000003
> Mar 20 14:18:27 912119 [41E02960] -> osm_report_notice: Reporting Generic
> Notice type:1 num:128 from LID:0x0152
> GID:0xfe80000000000000,0x0005ad0000027c84
> Mar 20 14:18:28 012957 [45A08960] -> __osm_trap_rcv_process_request:
> Received Generic Notice type:0x01 num:128 Producer:2 from LID:0x0152
> TID:0x0000000000000004
> Mar 20 14:18:28 013058 [45A08960] -> osm_report_notice: Reporting Generic
> Notice type:1 num:128 from LID:0x0152
> GID:0xfe80000000000000,0x0005ad0000027c84
> Mar 20 14:18:28 075266 [43C05960] -> __osm_trap_rcv_process_request:
> Received Generic Notice type:0x01 num:128 Producer:2 from LID:0x0152
> TID:0x0000000000000005
> Mar 20 14:18:28 075397 [43C05960] -> osm_report_notice: Reporting Generic
> Notice type:1 num:128 from LID:0x0152
> GID:0xfe80000000000000,0x0005ad0000027c84
> Mar 20 14:18:28 259000 [41E02960] -> __osm_trap_rcv_process_request:
> Received Generic Notice type:0x01 num:128 Producer:2 from LID:0x0152
> TID:0x0000000000000006
> Mar 20 14:18:28 259121 [41E02960] -> osm_report_notice: Reporting Generic
> Notice type:1 num:128 from LID:0x0152
> GID:0xfe80000000000000,0x0005ad0000027c84
> Mar 20 14:18:28 308865 [42803960] -> __osm_trap_rcv_process_request:
> Received Generic Notice type:0x01 num:128 Producer:2 from LID:0x0152
> TID:0x0000000000000007
> Mar 20 14:18:28 309000 [42803960] -> osm_report_notice: Reporting Generic
> Notice type:1 num:128 from LID:0x0152
> GID:0xfe80000000000000,0x0005ad0000027c84
> Mar 20 14:18:28 330606 [45007960] -> __osm_trap_rcv_process_request:
> Received Generic Notice type:0x01 num:128 Producer:2 from LID:0x0152
> TID:0x0000000000000008
> Mar 20 14:18:28 330714 [45007960] -> osm_report_notice: Reporting Generic
> Notice type:1 num:128 from LID:0x0152
> GID:0xfe80000000000000,0x0005ad0000027c84
> Mar 20 14:18:28 444107 [45A08960] -> __osm_trap_rcv_process_request:
> Received Generic Notice type:0x01 num:128 Producer:2 from LID:0x0152
> TID:0x0000000000000009
> Mar 20 14:18:28 444191 [45A08960] -> osm_report_notice: Reporting Generic
> Notice type:1 num:128 from LID:0x0152
> GID:0xfe80000000000000,0x0005ad0000027c84
> Mar 20 14:18:28 466156 [44606960] -> __osm_trap_rcv_process_request:
> Received Generic Notice type:0x01 num:128 Producer:2 from LID:0x0152
> TID:0x000000000000000a
> Mar 20 14:18:28 466234 [44606960] -> osm_report_notice: Reporting Generic
> Notice type:1 num:128 from LID:0x0152
> GID:0xfe80000000000000,0x0005ad0000027c84
> Mar 20 14:18:28 478021 [43C05960] -> __osm_trap_rcv_process_request:
> Received Generic Notice type:0x01 num:128 Producer:2 from LID:0x0152
> TID:0x000000000000000b
> Mar 20 14:18:28 478070 [43C05960] -> __osm_trap_rcv_process_request: ERR
> 3804: Received trap 11 times consecutively
> Mar 20 14:18:28 489091 [43204960] -> osm_report_notice: Reporting Generic
> Notice type:3 num:65 from LID:0x0092
> GID:0xfe80000000000000,0x0005ad0000024bbb
> Mar 20 14:18:28 489106 [43204960] -> Removed port with
> GUID:0x0005ad00000249d3 LID range [0xB1,0xB1] of node:saguaro-23-2 HCA-1
> Mar 20 14:18:28 521430 [42803960] -> __osm_trap_rcv_process_request:
> Received Generic Notice type:0x01 num:128 Producer:2 from LID:0x0152
> TID:0x000000000000000c
> Mar 20 14:18:28 521499 [42803960] -> __osm_trap_rcv_process_request: ERR
> 3804: Received trap 12 times consecutively
> Mar 20 14:18:28 523658 [43204960] -> osm_ucast_mgr_process: Min Hop Tables
> configured on all switches
> Mar 20 14:18:28 580295 [43204960] -> SUBNET UP
> Mar 20 14:18:28 611805 [43204960] -> __osm_trap_rcv_process_request:
> Received Generic Notice type:0x01 num:128 Producer:2 from LID:0x0152
> TID:0x000000000000000d
> Mar 20 14:18:28 611893 [43204960] -> __osm_trap_rcv_process_request: ERR
> 3804: Received trap 13 times consecutively
> Mar 20 14:18:28 661292 [45A08960] -> __osm_trap_rcv_process_request:
> Received Generic Notice type:0x01 num:128 Producer:2 from LID:0x0152
> TID:0x000000000000000e
> Mar 20 14:18:28 661351 [45A08960] -> __osm_trap_rcv_process_request: ERR
> 3804: Received trap 14 times consecutively
> Mar 20 14:18:28 871670 [44606960] -> __osm_trap_rcv_process_request:
> Received Generic Notice type:0x01 num:128 Producer:2 from LID:0x0152
> TID:0x000000000000000f
> Mar 20 14:18:28 871732 [44606960] -> __osm_trap_rcv_process_request: ERR
> 3804: Received trap 15 times consecutively
> Mar 20 14:18:28 934440 [43204960] -> __osm_trap_rcv_process_request:
> Received Generic Notice type:0x01 num:128 Producer:2 from LID:0x0152
> TID:0x0000000000000010
> Mar 20 14:18:28 934505 [43204960] -> __osm_trap_rcv_process_request: ERR
> 3804: Received trap 16 times consecutively
> Mar 20 14:18:28 941281 [45A08960] -> osm_report_notice: Reporting Generic
> Notice type:3 num:65 from LID:0x0092
> GID:0xfe80000000000000,0x0005ad0000024bbb
> Mar 20 14:18:28 941303 [45A08960] -> Removed port with
> GUID:0x0005ad000002510b LID range [0xB5,0xB5] of node:saguaro-23-6 HCA-1
> Mar 20 14:18:28 941329 [45A08960] -> osm_report_notice: Reporting Generic
> Notice type:3 num:65 from LID:0x0092
> GID:0xfe80000000000000,0x0005ad0000024bbb
> Mar 20 14:18:28 941336 [45A08960] -> Removed port with
> GUID:0x0005ad0000025043 LID range [0xB4,0xB4] of node:saguaro-23-5 HCA-1
> Mar 20 14:18:28 941356 [45A08960] -> osm_report_notice: Reporting Generic
> Notice type:3 num:65 from LID:0x0092
> GID:0xfe80000000000000,0x0005ad0000024bbb
> Mar 20 14:18:28 941363 [45A08960] -> Removed port with
> GUID:0x0005ad0000024d47 LID range [0xB6,0xB6] of node:saguaro-23-7 HCA-1
> Mar 20 14:18:28 941388 [45A08960] -> osm_report_notice: Reporting Generic
> Notice type:3 num:65 from LID:0x0092
> GID:0xfe80000000000000,0x0005ad0000024bbb
> Mar 20 14:18:28 941395 [45A08960] -> Removed port with
> GUID:0x0005ad0000024d8b LID range [0xB7,0xB7] of node:saguaro-23-8 HCA-1
> Mar 20 14:18:28 941420 [45A08960] -> osm_report_notice: Reporting Generic
> Notice type:3 num:65 from LID:0x0092
> GID:0xfe80000000000000,0x0005ad0000024bbb
> Mar 20 14:18:28 941426 [45A08960] -> Removed port with
> GUID:0x0005ad0000024cbb LID range [0xB2,0xB2] of node:saguaro-23-3 HCA-1
> Mar 20 14:18:28 945507 [45A08960] -> osm_report_notice: Reporting Generic
> Notice type:3 num:64 from LID:0x0092
> GID:0xfe80000000000000,0x0005ad0000024bbb
> Mar 20 14:18:28 945515 [45A08960] -> Discovered new port with
> GUID:0x0005ad00000249d3 LID range [0xB1,0xB1] of node:saguaro-23-2 HCA-1
> Mar 20 14:18:28 956576 [44606960] -> __osm_trap_rcv_process_request:
> Received Generic Notice type:0x01 num:128 Producer:2 from LID:0x0152
> TID:0x0000000000000011
> Mar 20 14:18:28 956665 [44606960] -> __osm_trap_rcv_process_request: ERR
> 3804: Received trap 17 times consecutively
> Mar 20 14:18:28 976211 [45A08960] -> osm_ucast_mgr_process: Min Hop Tables
> configured on all switches
> Mar 20 14:18:29 033513 [42803960] -> SUBNET UP
> Mar 20 14:18:29 071283 [41401960] -> __osm_trap_rcv_process_request:
> Received Generic Notice type:0x01 num:128 Producer:2 from LID:0x0152
> TID:0x0000000000000012
> Mar 20 14:18:29 071345 [41401960] -> __osm_trap_rcv_process_request: ERR
> 3804: Received trap 18 times consecutively
> Mar 20 14:18:29 352103 [44606960] -> __osm_trap_rcv_process_request:
> Received Generic Notice type:0x01 num:128 Producer:2 from LID:0x0152
> TID:0x0000000000000013
> Mar 20 14:18:29 352155 [44606960] -> __osm_trap_rcv_process_request: ERR
> 3804: Received trap 19 times consecutively
> Mar 20 14:18:29 376386 [41E02960] -> __osm_trap_rcv_process_request:
> Received Generic Notice type:0x01 num:128 Producer:2 from LID:0x0152
> TID:0x0000000000000014
> Mar 20 14:18:29 376461 [41E02960] -> __osm_trap_rcv_process_request: ERR
> 3804: Received trap 20 times consecutively
> Mar 20 14:18:29 420228 [43204960] -> __osm_trap_rcv_process_request:
> Received Generic Notice type:0x01 num:128 Producer:2 from LID:0x0152
> TID:0x0000000000000015
> Mar 20 14:18:29 420282 [43204960] -> __osm_trap_rcv_process_request: ERR
> 3804: Received trap 21 times consecutively
> Mar 20 14:18:29 421294 [43C05960] -> __osm_trap_rcv_process_request:
> Received Generic Notice type:0x01 num:128 Producer:2 from LID:0x0152
> TID:0x0000000000000016
> Mar 20 14:18:29 421345 [43C05960] -> __osm_trap_rcv_process_request: ERR
> 3804: Received trap 22 times consecutively
> Mar 20 14:18:29 461135 [45A08960] -> __osm_trap_rcv_process_request:
> Received Generic Notice type:0x01 num:128 Producer:2 from LID:0x0152
> TID:0x0000000000000017
> Mar 20 14:18:29 461179 [45A08960] -> __osm_trap_rcv_process_request: ERR
> 3804: Received trap 23 times consecutively
> Mar 20 14:18:29 633008 [45007960] -> __osm_trap_rcv_process_request:
> Received Generic Notice type:0x01 num:128 Producer:2 from LID:0x0152
> TID:0x0000000000000018
> Mar 20 14:18:29 633050 [42803960] -> __osm_trap_rcv_process_request:
> Received Generic Notice type:0x01 num:128 Producer:2 from LID:0x0148
> TID:0x000000000000005b
> Mar 20 14:18:29 633062 [45007960] -> __osm_trap_rcv_process_request: ERR
> 3804: Received trap 24 times consecutively
> Mar 20 14:18:29 633350 [42803960] -> osm_report_notice: Reporting Generic
> Notice type:1 num:128 from LID:0x0148
> GID:0xfe80000000000000,0x0005ad00000281b3
> Mar 20 14:18:29 733039 [45A08960] -> __osm_trap_rcv_process_request:
> Received Generic Notice type:0x01 num:128 Producer:2 from LID:0x0148
> TID:0x000000000000005c
> Mar 20 14:18:29 733238 [45A08960] -> osm_report_notice: Reporting Generic
> Notice type:1 num:128 from LID:0x0148
> GID:0xfe80000000000000,0x0005ad00000281b3
> Mar 20 14:18:29 947440 [44606960] -> osm_report_notice: Reporting Generic
> Notice type:3 num:64 from LID:0x0092
> GID:0xfe80000000000000,0x0005ad0000024bbb
> Mar 20 14:18:29 947452 [44606960] -> Discovered new port with
> GUID:0x0005ad0000024cbb LID range [0xB2,0xB2] of node:saguaro-23-3 HCA-1
> Mar 20 14:18:29 947457 [44606960] -> osm_report_notice: Reporting Generic
> Notice type:3 num:64 from LID:0x0092
> GID:0xfe80000000000000,0x0005ad0000024bbb
> Mar 20 14:18:29 947462 [44606960] -> Discovered new port with
> GUID:0x0005ad0000025043 LID range [0xB4,0xB4] of node:saguaro-23-5 HCA-1
> Mar 20 14:18:29 947465 [44606960] -> osm_report_notice: Reporting Generic
> Notice type:3 num:64 from LID:0x0092
> GID:0xfe80000000000000,0x0005ad0000024bbb
> Mar 20 14:18:29 947470 [44606960] -> Discovered new port with
> GUID:0x0005ad000002510b LID range [0xB5,0xB5] of node:saguaro-23-6 HCA-1
> Mar 20 14:18:29 947474 [44606960] -> osm_report_notice: Reporting Generic
> Notice type:3 num:64 from LID:0x0092
> GID:0xfe80000000000000,0x0005ad0000024bbb
> Mar 20 14:18:29 947479 [44606960] -> Discovered new port with
> GUID:0x0005ad0000024d47 LID range [0xB6,0xB6] of node:saguaro-23-7 HCA-1
> Mar 20 14:18:29 947482 [44606960] -> osm_report_notice: Reporting Generic
> Notice type:3 num:64 from LID:0x0092
> GID:0xfe80000000000000,0x0005ad0000024bbb
> Mar 20 14:18:29 947487 [44606960] -> Discovered new port with
> GUID:0x0005ad0000024d8b LID range [0xB7,0xB7] of node:saguaro-23-8 HCA-1
> Mar 20 14:18:29 978182 [44606960] -> osm_ucast_mgr_process: Min Hop Tables
> configured on all switches
> Mar 20 14:18:30 027730 [4780B960] -> __osm_sm_mad_ctrl_rcv_callback: ERR
> 3111: Error status = 0x1C00
> Mar 20 14:18:30 027819 [4780B960] -> SMP dump:
> base_ver................0x1
> mgmt_class..............0x81
> class_ver...............0x1
> method..................0x81 (SubnGetResp)
> D bit...................0x1
> status..................0x1C00
> hop_ptr.................0x0
> hop_count...............0x4
> trans_id................0x762b8
> attr_id.................0x15 (PortInfo)
> resv....................0x0
> attr_mod................0x16
> m_key...................0x0000000000000000
> dr_slid.................0xFFFF
> dr_dlid.................0xFFFF
>
> Initial path: [0][1][11][1][4]
> Return path: [0][9][18][D][1]
> Reserved: [0][0][0][0][0][0][0]
>
> 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
>
> 00 00 00 00 00 00 00 00 00 00 00 00 01 03 03 02
>
> 14 52 00 11 40 40 00 08 08 04 F2 40 00 00 00 00
>
> 00 00 00 00 00 88 00 00 00 00 00 00 00 00 00 00
>
> Mar 20 14:18:30 027897 [41401960] -> osm_pi_rcv_process_set: Received error
> status 0x1c for SetResp() during ACTIVE transition
> Mar 20 14:18:30 027901 [41401960] -> PortInfo dump:
> port number.............0x16
> node_guid...............0x0005ad00000281b3
> port_guid...............0x0005ad00000281b3
> m_key...................0x0000000000000000
> subnet_prefix...........0x0000000000000000
> base_lid................0x0
> master_sm_base_lid......0x0
> capability_mask.........0x0
> diag_code...............0x0
> m_key_lease_period......0x0
> local_port_num..........0x1
> link_width_enabled......0x3
> link_width_supported....0x3
> link_width_active.......0x2
> link_speed_supported....0x1
> port_state..............ACTIVE
> state_info2.............0x52
> m_key_protect_bits......0x0
> lmc.....................0x0
> link_speed..............0x11
> mtu_smsl................0x40
> vl_cap_init_type........0x40
> vl_high_limit...........0x0
> vl_arb_high_cap.........0x8
> vl_arb_low_cap..........0x8
> init_rep_mtu_cap........0x4
> vl_stall_life...........0xF2
> vl_enforce..............0x40
> m_key_violations........0x0
> p_key_violations........0x0
> q_key_violations........0x0
> guid_cap................0x0
> client_reregister.......0x0
> subnet_timeout..........0x0
> resp_time_value.........0x0
> error_threshold.........0x88
> Mar 20 14:18:30 027914 [41401960] -> Capabilities Mask:
> Mar 20 14:18:30 027993 [4780B960] -> __osm_sm_mad_ctrl_rcv_callback: ERR
> 3111: Error status = 0x1C00
> Mar 20 14:18:30 028043 [4780B960] -> SMP dump:
> base_ver................0x1
> mgmt_class..............0x81
> class_ver...............0x1
> method..................0x81 (SubnGetResp)
> D bit...................0x1
> status..................0x1C00
> hop_ptr.................0x0
> hop_count...............0x4
> trans_id................0x762b9
> attr_id.................0x15 (PortInfo)
> resv....................0x0
> attr_mod................0x17
> m_key...................0x0000000000000000
> dr_slid.................0xFFFF
> dr_dlid.................0xFFFF
>
> Initial path: [0][1][11][1][4]
> Return path: [0][9][18][D][1]
> Reserved: [0][0][0][0][0][0][0]
>
> 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
>
> 00 00 00 00 00 00 00 00 00 00 00 00 01 03 03 02
>
> 14 52 00 11 40 40 00 08 08 04 F2 40 00 00 00 00
>
> 00 00 00 00 00 88 00 00 00 00 00 00 00 00 00 00
>
> Mar 20 14:18:30 028098 [45A08960] -> osm_pi_rcv_process_set: Received error
> status 0x1c for SetResp() during ACTIVE transition
> Mar 20 14:18:30 028109 [45A08960] -> PortInfo dump:
> port number.............0x17
> node_guid...............0x0005ad00000281b3
> port_guid...............0x0005ad00000281b3
> m_key...................0x0000000000000000
> subnet_prefix...........0x0000000000000000
> base_lid................0x0
> master_sm_base_lid......0x0
> capability_mask.........0x0
> diag_code...............0x0
> m_key_lease_period......0x0
> local_port_num..........0x1
> link_width_enabled......0x3
> link_width_supported....0x3
> link_width_active.......0x2
> link_speed_supported....0x1
> port_state..............ACTIVE
> state_info2.............0x52
> m_key_protect_bits......0x0
> lmc.....................0x0
> link_speed..............0x11
> mtu_smsl................0x40
> vl_cap_init_type........0x40
> vl_high_limit...........0x0
> vl_arb_high_cap.........0x8
> vl_arb_low_cap..........0x8
> init_rep_mtu_cap........0x4
> vl_stall_life...........0xF2
> vl_enforce..............0x40
> m_key_violations........0x0
> p_key_violations........0x0
> q_key_violations........0x0
> guid_cap................0x0
> client_reregister.......0x0
> subnet_timeout..........0x0
> resp_time_value.........0x0
> error_threshold.........0x88
> Mar 20 14:18:30 028124 [45A08960] -> Capabilities Mask:
> Mar 20 14:18:30 033824 [44606960] -> SUBNET UP
> Mar 20 14:18:30 418497 [43C05960] -> osm_report_notice: Reporting Generic
> Notice type:3 num:65 from LID:0x0092
> GID:0xfe80000000000000,0x0005ad0000024bbb
> Mar 20 14:18:30 418522 [43C05960] -> Removed port with
> GUID:0x0005ad0000024feb LID range [0x153,0x153] of node:saguaro-22-5 HCA-1
> Mar 20 14:18:30 453167 [43C05960] -> osm_ucast_mgr_process: Min Hop Tables
> configured on all switches
> Mar 20 14:18:30 494719 [41E02960] -> __osm_trap_rcv_process_request:
> Received Generic Notice type:0x01 num:128 Producer:2 from LID:0x0148
> TID:0x000000000000005d
> Mar 20 14:18:30 494877 [41E02960] -> osm_report_notice: Reporting Generic
> Notice type:1 num:128 from LID:0x0148
> GID:0xfe80000000000000,0x0005ad00000281b3
> Mar 20 14:18:30 662496 [44606960] -> __osm_trap_rcv_process_request:
> Received Generic Notice type:0x01 num:128 Producer:2 from LID:0x001B
> TID:0x0000000000000073
> Mar 20 14:18:30 662564 [44606960] -> osm_report_notice: Reporting Generic
> Notice type:1 num:128 from LID:0x001B
> GID:0xfe80000000000000,0x0005ad00000281a7
> Mar 20 14:18:30 662645 [43C05960] -> __osm_trap_rcv_process_request:
> Received Generic Notice type:0x01 num:128 Producer:2 from LID:0x0148
> TID:0x000000000000005e
> Mar 20 14:18:30 662759 [43C05960] -> osm_report_notice: Reporting Generic
> Notice type:1 num:128 from LID:0x0148
> GID:0xfe80000000000000,0x0005ad00000281b3
> Mar 20 14:18:30 707085 [42803960] -> __osm_trap_rcv_process_request:
> Received Generic Notice type:0x01 num:128 Producer:2 from LID:0x0148
> TID:0x000000000000005f
> Mar 20 14:18:30 707179 [42803960] -> osm_report_notice: Reporting Generic
> Notice type:1 num:128 from LID:0x0148
> GID:0xfe80000000000000,0x0005ad00000281b3
> Mar 20 14:18:30 728948 [41E02960] -> __osm_trap_rcv_process_request:
> Received Generic Notice type:0x01 num:128 Producer:2 from LID:0x0148
> TID:0x0000000000000060
> Mar 20 14:18:30 729041 [41E02960] -> osm_report_notice: Reporting Generic
> Notice type:1 num:128 from LID:0x0148
> GID:0xfe80000000000000,0x0005ad00000281b3
> Mar 20 14:18:30 872332 [45A08960] -> __osm_trap_rcv_process_request:
> Received Generic Notice type:0x01 num:128 Producer:2 from LID:0x0148
> TID:0x0000000000000061
> Mar 20 14:18:30 872412 [45A08960] -> osm_report_notice: Reporting Generic
> Notice type:1 num:128 from LID:0x0148
> GID:0xfe80000000000000,0x0005ad00000281b3
> Mar 20 14:18:30 899764 [45A08960] -> SUBNET UP
> Mar 20 14:18:31 047423 [43C05960] -> __osm_trap_rcv_process_request:
> Received Generic Notice type:0x01 num:128 Producer:2 from LID:0x0148
> TID:0x0000000000000062
> Mar 20 14:18:31 047611 [43C05960] -> osm_report_notice: Reporting Generic
> Notice type:1 num:128 from LID:0x0148
> GID:0xfe80000000000000,0x0005ad00000281b3
> Mar 20 14:18:31 165201 [45A08960] -> __osm_trap_rcv_process_request:
> Received Generic Notice type:0x01 num:128 Producer:2 from LID:0x0148
> TID:0x0000000000000063
> Mar 20 14:18:31 165272 [45A08960] -> osm_report_notice: Reporting Generic
> Notice type:1 num:128 from LID:0x0148
> GID:0xfe80000000000000,0x0005ad00000281b3
> Mar 20 14:18:31 182461 [44606960] -> __osm_trap_rcv_process_request:
> Received Generic Notice type:0x01 num:128 Producer:2 from LID:0x001B
> TID:0x0000000000000074
> Mar 20 14:18:31 182653 [44606960] -> osm_report_notice: Reporting Generic
> Notice type:1 num:128 from LID:0x001B
> GID:0xfe80000000000000,0x0005ad00000281a7
> Mar 20 14:18:31 248834 [44606960] -> __osm_trap_rcv_process_request:
> Received Generic Notice type:0x01 num:128 Producer:2 from LID:0x001B
> TID:0x0000000000000075
> Mar 20 14:18:31 248893 [44606960] -> osm_report_notice: Reporting Generic
> Notice type:1 num:128 from LID:0x001B
> GID:0xfe80000000000000,0x0005ad00000281a7
> Mar 20 14:18:31 499830 [45A08960] -> __osm_trap_rcv_process_request:
> Received Generic Notice type:0x01 num:128 Producer:2 from LID:0x001B
> TID:0x0000000000000076
> Mar 20 14:18:31 499908 [45A08960] -> osm_report_notice: Reporting Generic
> Notice type:1 num:128 from LID:0x001B
> GID:0xfe80000000000000,0x0005ad00000281a7
> Mar 20 14:18:31 521824 [41401960] -> __osm_trap_rcv_process_request:
> Received Generic Notice type:0x01 num:128 Producer:2 from LID:0x001B
> TID:0x0000000000000077
> Mar 20 14:18:31 521891 [41401960] -> osm_report_notice: Reporting Generic
> Notice type:1 num:128 from LID:0x001B
> GID:0xfe80000000000000,0x0005ad00000281a7
> Mar 20 14:18:31 543713 [44606960] -> __osm_trap_rcv_process_request:
> Received Generic Notice type:0x01 num:128 Producer:2 from LID:0x001B
> TID:0x0000000000000078
> Mar 20 14:18:31 543784 [44606960] -> osm_report_notice: Reporting Generic
> Notice type:1 num:128 from LID:0x001B
> GID:0xfe80000000000000,0x0005ad00000281a7
> Mar 20 14:18:31 589490 [43C05960] -> osm_report_notice: Reporting Generic
> Notice type:3 num:64 from LID:0x0092
> GID:0xfe80000000000000,0x0005ad0000024bbb
> Mar 20 14:18:31 589499 [43C05960] -> Discovered new port with
> GUID:0x0005ad0000024feb LID range [0x153,0x153] of node:saguaro-22-5 HCA-1
> Mar 20 14:18:31 620166 [43C05960] -> osm_ucast_mgr_process: Min Hop Tables
> configured on all switches
> Mar 20 14:18:31 672647 [4780B960] -> __osm_sm_mad_ctrl_rcv_callback: ERR
> 3111: Error status = 0x1C00
> Mar 20 14:18:31 672739 [4780B960] -> SMP dump:
> base_ver................0x1
> mgmt_class..............0x81
> class_ver...............0x1
> method..................0x81 (SubnGetResp)
> D bit...................0x1
> status..................0x1C00
> hop_ptr.................0x0
> hop_count...............0x4
> trans_id................0x77d11
> attr_id.................0x15 (PortInfo)
> resv....................0x0
> attr_mod................0x16
> m_key...................0x0000000000000000
> dr_slid.................0xFFFF
> dr_dlid.................0xFFFF
>
> Initial path: [0][1][11][1][4]
> Return path: [0][9][18][D][1]
> Reserved: [0][0][0][0][0][0][0]
>
> 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
>
> 00 00 00 00 00 00 00 00 00 00 00 00 01 03 03 02
>
> 14 52 00 11 40 40 00 08 08 04 F2 40 00 00 00 00
>
> 00 00 00 00 00 88 00 00 00 00 00 00 00 00 00 00
>
> Mar 20 14:18:31 672817 [43C05960] -> osm_pi_rcv_process_set: Received error
> status 0x1c for SetResp() during ACTIVE transition
> Mar 20 14:18:31 672823 [4780B960] -> __osm_sm_mad_ctrl_rcv_callback: ERR
> 3111: Error status = 0x1C00
> Mar 20 14:18:31 672833 [43C05960] -> PortInfo dump:
> port number.............0x16
> node_guid...............0x0005ad00000281b3
> port_guid...............0x0005ad00000281b3
> m_key...................0x0000000000000000
> subnet_prefix...........0x0000000000000000
> base_lid................0x0
> master_sm_base_lid......0x0
> capability_mask.........0x0
> diag_code...............0x0
> m_key_lease_period......0x0
> local_port_num..........0x1
> link_width_enabled......0x3
> link_width_supported....0x3
> link_width_active.......0x2
> link_speed_supported....0x1
> port_state..............ACTIVE
> state_info2.............0x52
> m_key_protect_bits......0x0
> lmc.....................0x0
> link_speed..............0x11
> mtu_smsl................0x40
> vl_cap_init_type........0x40
> vl_high_limit...........0x0
> vl_arb_high_cap.........0x8
> vl_arb_low_cap..........0x8
> init_rep_mtu_cap........0x4
> vl_stall_life...........0xF2
> vl_enforce..............0x40
> m_key_violations........0x0
> p_key_violations........0x0
> q_key_violations........0x0
> guid_cap................0x0
> client_reregister.......0x0
> subnet_timeout..........0x0
> resp_time_value.........0x0
> error_threshold.........0x88
> Mar 20 14:18:31 672852 [43C05960] -> Capabilities Mask:
> Mar 20 14:18:31 672861 [4780B960] -> SMP dump:
> base_ver................0x1
> mgmt_class..............0x81
> class_ver...............0x1
> method..................0x81 (SubnGetResp)
> D bit...................0x1
> status..................0x1C00
> hop_ptr.................0x0
> hop_count...............0x4
> trans_id................0x77d12
> attr_id.................0x15 (PortInfo)
> resv....................0x0
> attr_mod................0x17
> m_key...................0x0000000000000000
> dr_slid.................0xFFFF
> dr_dlid.................0xFFFF
>
> Initial path: [0][1][11][1][4]
> Return path: [0][9][18][D][1]
> Reserved: [0][0][0][0][0][0][0]
>
> 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
>
> 00 00 00 00 00 00 00 00 00 00 00 00 01 03 03 02
>
> 14 52 00 11 40 40 00 08 08 04 F2 40 00 00 00 00
>
> 00 00 00 00 00 88 00 00 00 00 00 00 00 00 00 00
>
> Mar 20 14:18:31 672918 [45007960] -> osm_pi_rcv_process_set: Received error
> status 0x1c for SetResp() during ACTIVE transition
> Mar 20 14:18:31 672922 [45007960] -> PortInfo dump:
> port number.............0x17
> node_guid...............0x0005ad00000281b3
> port_guid...............0x0005ad00000281b3
> m_key...................0x0000000000000000
> subnet_prefix...........0x0000000000000000
> base_lid................0x0
> master_sm_base_lid......0x0
> capability_mask.........0x0
> diag_code...............0x0
> m_key_lease_period......0x0
> local_port_num..........0x1
> link_width_enabled......0x3
> link_width_supported....0x3
> link_width_active.......0x2
> link_speed_supported....0x1
> port_state..............ACTIVE
> state_info2.............0x52
> m_key_protect_bits......0x0
> lmc.....................0x0
> link_speed..............0x11
> mtu_smsl................0x40
> vl_cap_init_type........0x40
> vl_high_limit...........0x0
> vl_arb_high_cap.........0x8
> vl_arb_low_cap..........0x8
> init_rep_mtu_cap........0x4
> vl_stall_life...........0xF2
> vl_enforce..............0x40
> m_key_violations........0x0
> p_key_violations........0x0
> q_key_violations........0x0
> guid_cap................0x0
> client_reregister.......0x0
> subnet_timeout..........0x0
> resp_time_value.........0x0
> error_threshold.........0x88
> Mar 20 14:18:31 672936 [45007960] -> Capabilities Mask:
> Mar 20 14:18:31 678085 [45A08960] -> SUBNET UP
> Mar 20 14:18:31 723715 [41E02960] -> __osm_trap_rcv_process_request:
> Received Generic Notice type:0x01 num:128 Producer:2 from LID:0x0152
> TID:0x0000000000000018
> Mar 20 14:18:31 723815 [41E02960] -> __osm_trap_rcv_process_request: ERR
> 3804: Received trap 25 times consecutively
> Mar 20 14:18:32 061932 [41401960] -> osm_ucast_mgr_process: Min Hop Tables
> configured on all switches
> Mar 20 14:18:32 113545 [4780B960] -> __osm_sm_mad_ctrl_rcv_callback: ERR
> 3111: Error status = 0x1C00
> Mar 20 14:18:32 113610 [4780B960] -> SMP dump:
> base_ver................0x1
> mgmt_class..............0x81
> class_ver...............0x1
> method..................0x81 (SubnGetResp)
> D bit...................0x1
> status..................0x1C00
> hop_ptr.................0x0
> hop_count...............0x4
> trans_id................0x78a4d
> attr_id.................0x15 (PortInfo)
> resv....................0x0
> attr_mod................0x13
> m_key...................0x0000000000000000
> dr_slid.................0xFFFF
> dr_dlid.................0xFFFF
>
> Initial path: [0][1][15][4][4]
> Return path: [0][9][18][D][4]
> Reserved: [0][0][0][0][0][0][0]
>
> 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
>
> 00 00 00 00 00 00 00 00 00 00 00 00 04 03 03 02
>
> 14 52 00 11 40 40 00 08 08 04 F2 40 00 00 00 00
>
> 00 00 00 00 00 88 00 00 00 00 00 00 00 00 00 00
>
> Mar 20 14:18:32 113712 [42803960] -> osm_pi_rcv_process_set: Received error
> status 0x1c for SetResp() during ACTIVE transition
> Mar 20 14:18:32 113725 [4780B960] -> __osm_sm_mad_ctrl_rcv_callback: ERR
> 3111: Error status = 0x1C00
> Mar 20 14:18:32 113730 [42803960] -> PortInfo dump:
> port number.............0x13
> node_guid...............0x0005ad00000281a7
> port_guid...............0x0005ad00000281a7
> m_key...................0x0000000000000000
> subnet_prefix...........0x0000000000000000
> base_lid................0x0
> master_sm_base_lid......0x0
> capability_mask.........0x0
> diag_code...............0x0
> m_key_lease_period......0x0
> local_port_num..........0x4
> link_width_enabled......0x3
> link_width_supported....0x3
> link_width_active.......0x2
> link_speed_supported....0x1
> port_state..............ACTIVE
> state_info2.............0x52
> m_key_protect_bits......0x0
> lmc.....................0x0
> link_speed..............0x11
> mtu_smsl................0x40
> vl_cap_init_type........0x40
> vl_high_limit...........0x0
> vl_arb_high_cap.........0x8
> vl_arb_low_cap..........0x8
> init_rep_mtu_cap........0x4
> vl_stall_life...........0xF2
> vl_enforce..............0x40
> m_key_violations........0x0
> p_key_violations........0x0
> q_key_violations........0x0
> guid_cap................0x0
> client_reregister.......0x0
> subnet_timeout..........0x0
> resp_time_value.........0x0
> error_threshold.........0x88
> Mar 20 14:18:32 113745 [42803960] -> Capabilities Mask:
> Mar 20 14:18:32 113751 [4780B960] -> SMP dump:
> base_ver................0x1
> mgmt_class..............0x81
> class_ver...............0x1
> method..................0x81 (SubnGetResp)
> D bit...................0x1
> status..................0x1C00
> hop_ptr.................0x0
> hop_count...............0x4
> trans_id................0x78a4e
> attr_id.................0x15 (PortInfo)
> resv....................0x0
> attr_mod................0x16
> m_key...................0x0000000000000000
> dr_slid.................0xFFFF
> dr_dlid.................0xFFFF
>
> Initial path: [0][1][15][4][4]
> Return path: [0][9][18][D][4]
> Reserved: [0][0][0][0][0][0][0]
>
> 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
>
> 00 00 00 00 00 00 00 00 00 00 00 00 04 03 03 02
>
> 14 52 00 11 40 40 00 08 08 04 F2 40 00 00 00 00
>
> 00 00 00 00 00 88 00 00 00 00 00 00 00 00 00 00
>
> Mar 20 14:18:32 113803 [44606960] -> osm_pi_rcv_process_set: Received error
> status 0x1c for SetResp() during ACTIVE transition
> Mar 20 14:18:32 113807 [44606960] -> PortInfo dump:
> port number.............0x16
> node_guid...............0x0005ad00000281a7
> port_guid...............0x0005ad00000281a7
> m_key...................0x0000000000000000
> subnet_prefix...........0x0000000000000000
> base_lid................0x0
> master_sm_base_lid......0x0
> capability_mask.........0x0
> diag_code...............0x0
> m_key_lease_period......0x0
> local_port_num..........0x4
> link_width_enabled......0x3
> link_width_supported....0x3
> link_width_active.......0x2
> link_speed_supported....0x1
> port_state..............ACTIVE
> state_info2.............0x52
> m_key_protect_bits......0x0
> lmc.....................0x0
> link_speed..............0x11
> mtu_smsl................0x40
> vl_cap_init_type........0x40
> vl_high_limit...........0x0
> vl_arb_high_cap.........0x8
> vl_arb_low_cap..........0x8
> init_rep_mtu_cap........0x4
> vl_stall_life...........0xF2
> vl_enforce..............0x40
> m_key_violations........0x0
> p_key_violations........0x0
> q_key_violations........0x0
> guid_cap................0x0
> client_reregister.......0x0
> subnet_timeout..........0x0
> resp_time_value.........0x0
> error_threshold.........0x88
> Mar 20 14:18:32 113820 [44606960] -> Capabilities Mask:
> Mar 20 14:18:32 113845 [4780B960] -> __osm_sm_mad_ctrl_rcv_callback: ERR
> 3111: Error status = 0x1C00
> Mar 20 14:18:32 113907 [4780B960] -> SMP dump:
> base_ver................0x1
> mgmt_class..............0x81
> class_ver...............0x1
> method..................0x81 (SubnGetResp)
> D bit...................0x1
> status..................0x1C00
> hop_ptr.................0x0
> hop_count...............0x4
> trans_id................0x78a4f
> attr_id.................0x15 (PortInfo)
> resv....................0x0
> attr_mod................0x17
> m_key...................0x0000000000000000
> dr_slid.................0xFFFF
> dr_dlid.................0xFFFF
>
> Initial path: [0][1][15][4][4]
> Return path: [0][9][18][D][4]
> Reserved: [0][0][0][0][0][0][0]
>
> 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
>
> 00 00 00 00 00 00 00 00 00 00 00 00 04 03 03 02
>
> 14 52 00 11 40 40 00 08 08 04 F2 40 00 00 00 00
>
> 00 00 00 00 00 88 00 00 00 00 00 00 00 00 00 00
>
> Mar 20 14:18:32 113958 [41E02960] -> osm_pi_rcv_process_set: Received error
> status 0x1c for SetResp() during ACTIVE transition
> Mar 20 14:18:32 113963 [4780B960] -> __osm_sm_mad_ctrl_rcv_callback: ERR
> 3111: Error status = 0x1C00
> Mar 20 14:18:32 113969 [41E02960] -> PortInfo dump:
> port number.............0x17
> node_guid...............0x0005ad00000281a7
> port_guid...............0x0005ad00000281a7
> m_key...................0x0000000000000000
> subnet_prefix...........0x0000000000000000
> base_lid................0x0
> master_sm_base_lid......0x0
> capability_mask.........0x0
> diag_code...............0x0
> m_key_lease_period......0x0
> local_port_num..........0x4
> link_width_enabled......0x3
> link_width_supported....0x3
> link_width_active.......0x2
> link_speed_supported....0x1
> port_state..............ACTIVE
> state_info2.............0x52
> m_key_protect_bits......0x0
> lmc.....................0x0
> link_speed..............0x11
> mtu_smsl................0x40
> vl_cap_init_type........0x40
> vl_high_limit...........0x0
> vl_arb_high_cap.........0x8
> vl_arb_low_cap..........0x8
> init_rep_mtu_cap........0x4
> vl_stall_life...........0xF2
> vl_enforce..............0x40
> m_key_violations........0x0
> p_key_violations........0x0
> q_key_violations........0x0
> guid_cap................0x0
> client_reregister.......0x0
> subnet_timeout..........0x0
> resp_time_value.........0x0
> error_threshold.........0x88
> Mar 20 14:18:32 113986 [41E02960] -> Capabilities Mask:
> Mar 20 14:18:32 113992 [4780B960] -> SMP dump:
> base_ver................0x1
> mgmt_class..............0x81
> class_ver...............0x1
> method..................0x81 (SubnGetResp)
> D bit...................0x1
> status..................0x1C00
> hop_ptr.................0x0
> hop_count...............0x4
> trans_id................0x78a50
> attr_id.................0x15 (PortInfo)
> resv....................0x0
> attr_mod................0x18
> m_key...................0x0000000000000000
> dr_slid.................0xFFFF
> dr_dlid.................0xFFFF
>
> Initial path: [0][1][15][4][4]
> Return path: [0][9][18][D][4]
> Reserved: [0][0][0][0][0][0][0]
>
> 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
>
> 00 00 00 00 00 00 00 00 00 00 00 00 04 03 03 02
>
> 14 52 00 11 40 40 00 08 08 04 F2 40 00 00 00 00
>
> 00 00 00 00 00 88 00 00 00 00 00 00 00 00 00 00
>
> Mar 20 14:18:32 114052 [45A08960] -> osm_pi_rcv_process_set: Received error
> status 0x1c for SetResp() during ACTIVE transition
> Mar 20 14:18:32 114066 [45A08960] -> PortInfo dump:
> port number.............0x18
> node_guid...............0x0005ad00000281a7
> port_guid...............0x0005ad00000281a7
> m_key...................0x0000000000000000
> subnet_prefix...........0x0000000000000000
> base_lid................0x0
> master_sm_base_lid......0x0
> capability_mask.........0x0
> diag_code...............0x0
> m_key_lease_period......0x0
> local_port_num..........0x4
> link_width_enabled......0x3
> link_width_supported....0x3
> link_width_active.......0x2
> link_speed_supported....0x1
> port_state..............ACTIVE
> state_info2.............0x52
> m_key_protect_bits......0x0
> lmc.....................0x0
> link_speed..............0x11
> mtu_smsl................0x40
> vl_cap_init_type........0x40
> vl_high_limit...........0x0
> vl_arb_high_cap.........0x8
> vl_arb_low_cap..........0x8
> init_rep_mtu_cap........0x4
> vl_stall_life...........0xF2
> vl_enforce..............0x40
> m_key_violations........0x0
> p_key_violations........0x0
> q_key_violations........0x0
> guid_cap................0x0
> client_reregister.......0x0
> subnet_timeout..........0x0
> resp_time_value.........0x0
> error_threshold.........0x88
> Mar 20 14:18:32 114089 [45A08960] -> Capabilities Mask:
> Mar 20 14:18:32 114052 [4780B960] -> __osm_sm_mad_ctrl_rcv_callback: ERR
> 3111: Error status = 0x1C00
> Mar 20 14:18:32 114171 [4780B960] -> SMP dump:
> base_ver................0x1
> mgmt_class..............0x81
> class_ver...............0x1
> method..................0x81 (SubnGetResp)
> D bit...................0x1
> status..................0x1C00
> hop_ptr.................0x0
> hop_count...............0x4
> trans_id................0x78a51
> attr_id.................0x15 (PortInfo)
> resv....................0x0
> attr_mod................0x18
> m_key...................0x0000000000000000
> dr_slid.................0xFFFF
> dr_dlid.................0xFFFF
>
> Initial path: [0][1][13][1][6]
> Return path: [0][9][13][D][3]
> Reserved: [0][0][0][0][0][0][0]
>
> 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
>
> 00 00 00 00 00 00 00 00 00 00 00 00 03 03 03 02
>
> 14 52 00 11 40 40 00 08 08 04 F2 40 00 00 00 00
>
> 00 00 00 00 00 88 00 00 00 00 00 00 00 00 00 00
>
> Mar 20 14:18:32 114224 [42803960] -> osm_pi_rcv_process_set: Received error
> status 0x1c for SetResp() during ACTIVE transition
> Mar 20 14:18:32 114228 [42803960] -> PortInfo dump:
> port number.............0x18
> node_guid...............0x0005ad00000281b3
> port_guid...............0x0005ad00000281b3
> m_key...................0x0000000000000000
> subnet_prefix...........0x0000000000000000
> base_lid................0x0
> master_sm_base_lid......0x0
> capability_mask.........0x0
> diag_code...............0x0
> m_key_lease_period......0x0
> local_port_num..........0x3
> link_width_enabled......0x3
> link_width_supported....0x3
> link_width_active.......0x2
> link_speed_supported....0x1
> port_state..............ACTIVE
> state_info2.............0x52
> m_key_protect_bits......0x0
> lmc.....................0x0
> link_speed..............0x11
> mtu_smsl................0x40
> vl_cap_init_type........0x40
> vl_high_limit...........0x0
> vl_arb_high_cap.........0x8
> vl_arb_low_cap..........0x8
> init_rep_mtu_cap........0x4
> vl_stall_life...........0xF2
> vl_enforce..............0x40
> m_key_violations........0x0
> p_key_violations........0x0
> q_key_violations........0x0
> guid_cap................0x0
> client_reregister.......0x0
> subnet_timeout..........0x0
> resp_time_value.........0x0
> error_threshold.........0x88
> Mar 20 14:18:32 114242 [42803960] -> Capabilities Mask:
> Mar 20 14:18:32 119326 [42803960] -> SUBNET UP
> Mar 20 14:23:02 506774 [41E02960] -> __osm_trap_rcv_process_request:
> Received Generic Notice type:0x01 num:128 Producer:2 from LID:0x0152
> TID:0x0000000000000019
> Mar 20 14:23:02 507064 [41E02960] -> osm_report_notice: Reporting Generic
> Notice type:1 num:128 from LID:0x0152
> GID:0xfe80000000000000,0x0005ad0000027c84
> Mar 20 14:23:02 861642 [43204960] -> osm_report_notice: Reporting Generic
> Notice type:3 num:64 from LID:0x0092
> GID:0xfe80000000000000,0x0005ad0000024bbb
> Mar 20 14:23:02 861653 [43204960] -> Discovered new port with
> GUID:0x0005ad0000024b27 LID range [0xAF,0xAF] of node:Topspin IB-DC
> Mar 20 14:23:02 893030 [43204960] -> osm_ucast_mgr_process: Min Hop Tables
> configured on all switches
> Mar 20 14:23:02 943693 [4780B960] -> __osm_sm_mad_ctrl_rcv_callback: ERR
> 3111: Error status = 0x1C00
> Mar 20 14:23:02 943765 [4780B960] -> SMP dump:
> base_ver................0x1
> mgmt_class..............0x81
> class_ver...............0x1
> method..................0x81 (SubnGetResp)
> D bit...................0x1
> status..................0x1C00
> hop_ptr.................0x0
> hop_count...............0x5
> trans_id................0x79aff
> attr_id.................0x15 (PortInfo)
> resv....................0x0
> attr_mod................0x1
> m_key...................0x0000000000000000
> dr_slid.................0xFFFF
> dr_dlid.................0xFFFF
>
> Initial path: [0][1][11][1][5][18]
> Return path: [0][9][18][D][2][13]
> Reserved: [0][0][0][0][0][0][0]
>
> 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
>
> 00 00 00 00 00 00 00 00 00 00 00 00 13 03 03 02
>
> 14 52 00 11 40 40 00 08 08 04 2C 4C 00 00 00 00
>
> 00 00 00 00 00 88 00 00 00 00 00 00 00 00 00 00
>
> Mar 20 14:23:02 943854 [43C05960] -> osm_pi_rcv_process_set: Received error
> status 0x1c for SetResp() during ACTIVE transition
> Mar 20 14:23:02 943870 [43C05960] -> PortInfo dump:
> port number.............0x1
> node_guid...............0x0005ad0000027c84
> port_guid...............0x0005ad0000027c84
> m_key...................0x0000000000000000
> subnet_prefix...........0x0000000000000000
> base_lid................0x0
> master_sm_base_lid......0x0
> capability_mask.........0x0
> diag_code...............0x0
> m_key_lease_period......0x0
> local_port_num..........0x13
> link_width_enabled......0x3
> link_width_supported....0x3
> link_width_active.......0x2
> link_speed_supported....0x1
> port_state..............ACTIVE
> state_info2.............0x52
> m_key_protect_bits......0x0
> lmc.....................0x0
> link_speed..............0x11
> mtu_smsl................0x40
> vl_cap_init_type........0x40
> vl_high_limit...........0x0
> vl_arb_high_cap.........0x8
> vl_arb_low_cap..........0x8
> init_rep_mtu_cap........0x4
> vl_stall_life...........0x2C
> vl_enforce..............0x4C
> m_key_violations........0x0
> p_key_violations........0x0
> q_key_violations........0x0
> guid_cap................0x0
> client_reregister.......0x0
> subnet_timeout..........0x0
> resp_time_value.........0x0
> error_threshold.........0x88
> Mar 20 14:23:02 943886 [43C05960] -> Capabilities Mask:
> Mar 20 14:23:02 948898 [43C05960] -> SUBNET UP
> Mar 20 14:23:03 237496 [42803960] -> __osm_trap_rcv_process_request:
> Received Generic Notice type:0x04 num:144 Producer:1 from LID:0x00AF
> TID:0x0000000000000000
> Mar 20 14:23:03 237710 [42803960] -> osm_report_notice: Reporting Generic
> Notice type:4 num:144 from LID:0x00AF
> GID:0xfe80000000000000,0x0005ad0000024b27
> Mar 20 14:23:03 605548 [45007960] -> osm_ucast_mgr_process: Min Hop Tables
> configured on all switches
> Mar 20 14:23:03 662757 [41401960] -> SUBNET UP
> Mar 20 14:24:54 675782 [44606960] -> __osm_trap_rcv_process_request:
> Received Generic Notice type:0x01 num:128 Producer:2 from LID:0x001B
> TID:0x0000000000000079
> Mar 20 14:24:54 676077 [44606960] -> osm_report_notice: Reporting Generic
> Notice type:1 num:128 from LID:0x001B
> GID:0xfe80000000000000,0x0005ad00000281a7
> Mar 20 14:24:54 677026 [43204960] -> __osm_trap_rcv_process_request:
> Received Generic Notice type:0x01 num:128 Producer:2 from LID:0x0148
> TID:0x0000000000000064
> Mar 20 14:24:54 677118 [43204960] -> osm_report_notice: Reporting Generic
> Notice type:1 num:128 from LID:0x0148
> GID:0xfe80000000000000,0x0005ad00000281b3
> Mar 20 14:24:55 047478 [43204960] -> osm_report_notice: Reporting Generic
> Notice type:3 num:65 from LID:0x0092
> GID:0xfe80000000000000,0x0005ad0000024bbb
> Mar 20 14:24:55 047501 [43204960] -> Removed port with
> GUID:0x0005ad0000024e0b LID range [0xB3,0xB3] of node:saguaro-23-4 HCA-1
> Mar 20 14:24:55 047520 [43204960] -> osm_report_notice: Reporting Generic
> Notice type:3 num:65 from LID:0x0092
> GID:0xfe80000000000000,0x0005ad0000024bbb
> Mar 20 14:24:55 047525 [43204960] -> Removed port with
> GUID:0x0005ad000002510b LID range [0xB5,0xB5] of node:saguaro-23-6 HCA-1
> Mar 20 14:24:55 047541 [43204960] -> osm_report_notice: Reporting Generic
> Notice type:3 num:65 from LID:0x0092
> GID:0xfe80000000000000,0x0005ad0000024bbb
> Mar 20 14:24:55 047546 [43204960] -> Removed port with
> GUID:0x0005ad000002511b LID range [0xA6,0xA6] of node:saguaro-22-1 HCA-1
> Mar 20 14:24:55 047563 [43204960] -> osm_report_notice: Reporting Generic
> Notice type:3 num:65 from LID:0x0092
> GID:0xfe80000000000000,0x0005ad0000024bbb
> Mar 20 14:24:55 047569 [43204960] -> Removed port with
> GUID:0x0005ad0000024b27 LID range [0xAF,0xAF] of node:Topspin IB-DC
> Mar 20 14:24:55 047586 [43204960] -> osm_report_notice: Reporting Generic
> Notice type:3 num:65 from LID:0x0092
> GID:0xfe80000000000000,0x0005ad0000024bbb
> Mar 20 14:24:55 047591 [43204960] -> Removed port with
> GUID:0x0005ad0000025043 LID range [0xB4,0xB4] of node:saguaro-23-5 HCA-1
> Mar 20 14:24:55 047607 [43204960] -> osm_report_notice: Reporting Generic
> Notice type:3 num:65 from LID:0x0092
> GID:0xfe80000000000000,0x0005ad0000024bbb
> Mar 20 14:24:55 047612 [43204960] -> Removed port with
> GUID:0x0005ad0000024d47 LID range [0xB6,0xB6] of node:saguaro-23-7 HCA-1
> Mar 20 14:24:55 047630 [43204960] -> osm_report_notice: Reporting Generic
> Notice type:3 num:65 from LID:0x0092
> GID:0xfe80000000000000,0x0005ad0000024bbb
> Mar 20 14:24:55 047635 [43204960] -> Removed port with
> GUID:0x0005ad0000024d6b LID range [0xB8,0xB8] of node:saguaro-23-9 HCA-1
> Mar 20 14:24:55 047652 [43204960] -> osm_report_notice: Reporting Generic
> Notice type:3 num:65 from LID:0x0092
> GID:0xfe80000000000000,0x0005ad0000024bbb
> Mar 20 14:24:55 047657 [43204960] -> Removed port with
> GUID:0x0005ad0000024977 LID range [0xA9,0xA9] of node:saguaro-22-4 HCA-1
> Mar 20 14:24:55 047798 [43204960] -> osm_report_notice: Reporting Generic
> Notice type:3 num:65 from LID:0x0092
> GID:0xfe80000000000000,0x0005ad0000024bbb
> Mar 20 14:24:55 047803 [43204960] -> Removed port with
> GUID:0x0005ad0000027c84 LID range [0x152,0x152] of node:Topspin Switch TS120
> Mar 20 14:24:55 047836 [43204960] -> osm_report_notice: Reporting Generic
> Notice type:3 num:65 from LID:0x0092
> GID:0xfe80000000000000,0x0005ad0000024bbb
> Mar 20 14:24:55 047842 [43204960] -> Removed port with
> GUID:0x0005ad0000024d8b LID range [0xB7,0xB7] of node:saguaro-23-8 HCA-1
> Mar 20 14:24:55 047857 [43204960] -> osm_report_notice: Reporting Generic
> Notice type:3 num:65 from LID:0x0092
> GID:0xfe80000000000000,0x0005ad0000024bbb
> Mar 20 14:24:55 047862 [43204960] -> Removed port with
> GUID:0x0005ad000002498f LID range [0xA8,0xA8] of node:saguaro-22-3 HCA-1
> Mar 20 14:24:55 047877 [43204960] -> osm_report_notice: Reporting Generic
> Notice type:3 num:65 from LID:0x0092
> GID:0xfe80000000000000,0x0005ad0000024bbb
> Mar 20 14:24:55 047882 [43204960] -> Removed port with
> GUID:0x0005ad0000024c9b LID range [0xA7,0xA7] of node:saguaro-22-2 HCA-1
> Mar 20 14:24:55 047896 [43204960] -> osm_report_notice: Reporting Generic
> Notice type:3 num:65 from LID:0x0092
> GID:0xfe80000000000000,0x0005ad0000024bbb
> Mar 20 14:24:55 047902 [43204960] -> Removed port with
> GUID:0x0005ad0000024da7 LID range [0xB0,0xB0] of node:saguaro-23-1 HCA-1
> Mar 20 14:24:55 047918 [43204960] -> osm_report_notice: Reporting Generic
> Notice type:3 num:65 from LID:0x0092
> GID:0xfe80000000000000,0x0005ad0000024bbb
> Mar 20 14:24:55 047923 [43204960] -> Removed port with
> GUID:0x0005ad0000024cbb LID range [0xB2,0xB2] of node:saguaro-23-3 HCA-1
> Mar 20 14:24:55 047939 [43204960] -> osm_report_notice: Reporting Generic
> Notice type:3 num:65 from LID:0x0092
> GID:0xfe80000000000000,0x0005ad0000024bbb
> Mar 20 14:24:55 047988 [43204960] -> Removed port with
> GUID:0x0005ad00000249d3 LID range [0xB1,0xB1] of node:saguaro-23-2 HCA-1
> Mar 20 14:24:55 048005 [43204960] -> osm_report_notice: Reporting Generic
> Notice type:3 num:65 from LID:0x0092
> GID:0xfe80000000000000,0x0005ad0000024bbb
> Mar 20 14:24:55 048010 [43204960] -> Removed port with
> GUID:0x0005ad0000024feb LID range [0x153,0x153] of node:saguaro-22-5 HCA-1
> Mar 20 14:24:55 048025 [43204960] -> osm_report_notice: Reporting Generic
> Notice type:3 num:65 from LID:0x0092
> GID:0xfe80000000000000,0x0005ad0000024bbb
> Mar 20 14:24:55 048030 [43204960] -> Removed port with
> GUID:0x0005ad0000024afb LID range [0xA5,0xA5] of node:saguaro-22-0 HCA-1
> Mar 20 14:24:55 081006 [43204960] -> osm_ucast_mgr_process: Min Hop Tables
> configured on all switches
> Mar 20 14:24:55 130875 [45A08960] -> SUBNET UP
> Mar 20 14:24:55 484995 [42803960] -> osm_ucast_mgr_process: Min Hop Tables
> configured on all switches
> Mar 20 14:24:55 535902 [42803960] -> SUBNET UP
> Mar 20 14:25:48 653788 [43204960] -> __osm_trap_rcv_process_request:
> Received Generic Notice type:0x01 num:128 Producer:2 from LID:0x0148
> TID:0x0000000000000065
> Mar 20 14:25:48 654009 [43204960] -> osm_report_notice: Reporting Generic
> Notice type:1 num:128 from LID:0x0148
> GID:0xfe80000000000000,0x0005ad00000281b3
> Mar 20 14:25:48 659749 [45A08960] -> __osm_trap_rcv_process_request:
> Received Generic Notice type:0x01 num:128 Producer:2 from LID:0x001B
> TID:0x000000000000007a
> Mar 20 14:25:48 659790 [42803960] -> __osm_trap_rcv_process_request:
> Received Generic Notice type:0x01 num:128 Producer:2 from LID:0x0148
> TID:0x0000000000000066
> Mar 20 14:25:48 659814 [45A08960] -> osm_report_notice: Reporting Generic
> Notice type:1 num:128 from LID:0x001B
> GID:0xfe80000000000000,0x0005ad00000281a7
> Mar 20 14:25:48 659963 [42803960] -> osm_report_notice: Reporting Generic
> Notice type:1 num:128 from LID:0x0148
> GID:0xfe80000000000000,0x0005ad00000281b3
> Mar 20 14:25:48 665972 [41401960] -> __osm_trap_rcv_process_request:
> Received Generic Notice type:0x01 num:128 Producer:2 from LID:0x001B
> TID:0x000000000000007b
> Mar 20 14:25:48 666050 [41401960] -> osm_report_notice: Reporting Generic
> Notice type:1 num:128 from LID:0x001B
> GID:0xfe80000000000000,0x0005ad00000281a7
> Mar 20 14:25:49 025384 [41E02960] -> osm_report_notice: Reporting Generic
> Notice type:3 num:64 from LID:0x0092
> GID:0xfe80000000000000,0x0005ad0000024bbb
> Mar 20 14:25:49 025396 [41E02960] -> Discovered new port with
> GUID:0x0005ad0000027c84 LID range [0x152,0x152] of node:Topspin Switch TS120
> Mar 20 14:25:49 025401 [41E02960] -> osm_report_notice: Reporting Generic
> Notice type:3 num:64 from LID:0x0092
> GID:0xfe80000000000000,0x0005ad0000024bbb
> Mar 20 14:25:49 025406 [41E02960] -> Discovered new port with
> GUID:0x0005ad0000024b27 LID range [0xAF,0xAF] of node:saguaro-23-0 HCA-1
> Mar 20 14:25:49 025410 [41E02960] -> osm_report_notice: Reporting Generic
> Notice type:3 num:64 from LID:0x0092
> GID:0xfe80000000000000,0x0005ad0000024bbb
> Mar 20 14:25:49 025416 [41E02960] -> Discovered new port with
> GUID:0x0005ad0000024da7 LID range [0xB0,0xB0] of node:saguaro-23-1 HCA-1
> Mar 20 14:25:49 025420 [41E02960] -> osm_report_notice: Reporting Generic
> Notice type:3 num:64 from LID:0x0092
> GID:0xfe80000000000000,0x0005ad0000024bbb
> Mar 20 14:25:49 025425 [41E02960] -> Discovered new port with
> GUID:0x0005ad00000249d3 LID range [0xB1,0xB1] of node:saguaro-23-2 HCA-1
> Mar 20 14:25:49 025428 [41E02960] -> osm_report_notice: Reporting Generic
> Notice type:3 num:64 from LID:0x0092
> GID:0xfe80000000000000,0x0005ad0000024bbb
> Mar 20 14:25:49 025433 [41E02960] -> Discovered new port with
> GUID:0x0005ad0000024cbb LID range [0xB2,0xB2] of node:saguaro-23-3 HCA-1
> Mar 20 14:25:49 025437 [41E02960] -> osm_report_notice: Reporting Generic
> Notice type:3 num:64 from LID:0x0092
> GID:0xfe80000000000000,0x0005ad0000024bbb
> Mar 20 14:25:49 025442 [41E02960] -> Discovered new port with
> GUID:0x0005ad0000024e0b LID range [0xB3,0xB3] of node:saguaro-23-4 HCA-1
> Mar 20 14:25:49 025446 [41E02960] -> osm_report_notice: Reporting Generic
> Notice type:3 num:64 from LID:0x0092
> GID:0xfe80000000000000,0x0005ad0000024bbb
> Mar 20 14:25:49 025451 [41E02960] -> Discovered new port with
> GUID:0x0005ad0000025043 LID range [0xB4,0xB4] of node:saguaro-23-5 HCA-1
> Mar 20 14:25:49 025461 [41E02960] -> osm_report_notice: Reporting Generic
> Notice type:3 num:64 from LID:0x0092
> GID:0xfe80000000000000,0x0005ad0000024bbb
> Mar 20 14:25:49 025466 [41E02960] -> Discovered new port with
> GUID:0x0005ad000002510b LID range [0xB5,0xB5] of node:saguaro-23-6 HCA-1
> Mar 20 14:25:49 025470 [41E02960] -> osm_report_notice: Reporting Generic
> Notice type:3 num:64 from LID:0x0092
> GID:0xfe80000000000000,0x0005ad0000024bbb
> Mar 20 14:25:49 025475 [41E02960] -> Discovered new port with
> GUID:0x0005ad0000024d47 LID range [0xB6,0xB6] of node:saguaro-23-7 HCA-1
> Mar 20 14:25:49 025483 [41E02960] -> osm_report_notice: Reporting Generic
> Notice type:3 num:64 from LID:0x0092
> GID:0xfe80000000000000,0x0005ad0000024bbb
> Mar 20 14:25:49 025488 [41E02960] -> Discovered new port with
> GUID:0x0005ad0000024d6b LID range [0xB8,0xB8] of node:saguaro-23-9 HCA-1
> Mar 20 14:25:49 025491 [41E02960] -> osm_report_notice: Reporting Generic
> Notice type:3 num:64 from LID:0x0092
> GID:0xfe80000000000000,0x0005ad0000024bbb
> Mar 20 14:25:49 025496 [41E02960] -> Discovered new port with
> GUID:0x0005ad0000024afb LID range [0xA5,0xA5] of node:saguaro-22-0 HCA-1
> Mar 20 14:25:49 025500 [41E02960] -> osm_report_notice: Reporting Generic
> Notice type:3 num:64 from LID:0x0092
> GID:0xfe80000000000000,0x0005ad0000024bbb
> Mar 20 14:25:49 025505 [41E02960] -> Discovered new port with
> GUID:0x0005ad000002511b LID range [0xA6,0xA6] of node:saguaro-22-1 HCA-1
> Mar 20 14:25:49 025508 [41E02960] -> osm_report_notice: Reporting Generic
> Notice type:3 num:64 from LID:0x0092
> GID:0xfe80000000000000,0x0005ad0000024bbb
> Mar 20 14:25:49 025513 [41E02960] -> Discovered new port with
> GUID:0x0005ad0000024c9b LID range [0xA7,0xA7] of node:saguaro-22-2 HCA-1
> Mar 20 14:25:49 025517 [41E02960] -> osm_report_notice: Reporting Generic
> Notice type:3 num:64 from LID:0x0092
> GID:0xfe80000000000000,0x0005ad0000024bbb
> Mar 20 14:25:49 025522 [41E02960] -> Discovered new port with
> GUID:0x0005ad000002498f LID range [0xA8,0xA8] of node:saguaro-22-3 HCA-1
> Mar 20 14:25:49 025556 [41E02960] -> osm_report_notice: Reporting Generic
> Notice type:3 num:64 from LID:0x0092
> GID:0xfe80000000000000,0x0005ad0000024bbb
> Mar 20 14:25:49 025562 [41E02960] -> Discovered new port with
> GUID:0x0005ad0000024977 LID range [0xA9,0xA9] of node:saguaro-22-4 HCA-1
> Mar 20 14:25:49 025565 [41E02960] -> osm_report_notice: Reporting Generic
> Notice type:3 num:64 from LID:0x0092
> GID:0xfe80000000000000,0x0005ad0000024bbb
> Mar 20 14:25:49 025570 [41E02960] -> Discovered new port with
> GUID:0x0005ad0000024feb LID range [0x153,0x153] of node:saguaro-22-5 HCA-1
> Mar 20 14:25:49 025574 [41E02960] -> osm_report_notice: Reporting Generic
> Notice type:3 num:64 from LID:0x0092
> GID:0xfe80000000000000,0x0005ad0000024bbb
> Mar 20 14:25:49 025579 [41E02960] -> Discovered new port with
> GUID:0x0005ad0000024d8b LID range [0xB7,0xB7] of node:saguaro-23-8 HCA-1
> Mar 20 14:25:49 056324 [41E02960] -> osm_ucast_mgr_process: Min Hop Tables
> configured on all switches
> Mar 20 14:25:49 126247 [4780B960] -> __osm_sm_mad_ctrl_rcv_callback: ERR
> 3111: Error status = 0x1C00
> Mar 20 14:25:49 126356 [4780B960] -> SMP dump:
> base_ver................0x1
> mgmt_class..............0x81
> class_ver...............0x1
> method..................0x81 (SubnGetResp)
> D bit...................0x1
> status..................0x1C00
> hop_ptr.................0x0
> hop_count...............0x4
> trans_id................0x7d165
> attr_id.................0x15 (PortInfo)
> resv....................0x0
> attr_mod................0x13
> m_key...................0x0000000000000000
> dr_slid.................0xFFFF
> dr_dlid.................0xFFFF
>
> Initial path: [0][1][15][1][6]
> Return path: [0][9][18][D][3]
> Reserved: [0][0][0][0][0][0][0]
>
> 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
>
> 00 00 00 00 00 00 00 00 00 00 00 00 03 03 03 02
>
> 14 52 00 11 40 40 00 08 08 04 F2 40 00 00 00 00
>
> 00 00 00 00 00 88 00 00 00 00 00 00 00 00 00 00
>
> Mar 20 14:25:49 126409 [4780B960] -> __osm_sm_mad_ctrl_rcv_callback: ERR
> 3111: Error status = 0x1C00
> Mar 20 14:25:49 126442 [4780B960] -> SMP dump:
> base_ver................0x1
> mgmt_class..............0x81
> class_ver...............0x1
> method..................0x81 (SubnGetResp)
> D bit...................0x1
> status..................0x1C00
> hop_ptr.................0x0
> hop_count...............0x4
> trans_id................0x7d166
> attr_id.................0x15 (PortInfo)
> resv....................0x0
> attr_mod................0x16
> m_key...................0x0000000000000000
> dr_slid.................0xFFFF
> dr_dlid.................0xFFFF
>
> Initial path: [0][1][15][1][6]
> Return path: [0][9][18][D][3]
> Reserved: [0][0][0][0][0][0][0]
>
> 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
>
> 00 00 00 00 00 00 00 00 00 00 00 00 03 03 03 02
>
> 14 52 00 11 40 40 00 08 08 04 F2 40 00 00 00 00
>
> 00 00 00 00 00 88 00 00 00 00 00 00 00 00 00 00
>
> Mar 20 14:25:49 126496 [4780B960] -> __osm_sm_mad_ctrl_rcv_callback: ERR
> 3111: Error status = 0x1C00
> Mar 20 14:25:49 126489 [42803960] -> osm_pi_rcv_process_set: Received error
> status 0x1c for SetResp() during ACTIVE transition
> Mar 20 14:25:49 126535 [4780B960] -> SMP dump:
> base_ver................0x1
> mgmt_class..............0x81
> class_ver...............0x1
> method..................0x81 (SubnGetResp)
> D bit...................0x1
> status..................0x1C00
> hop_ptr.................0x0
> hop_count...............0x4
> trans_id................0x7d167
> attr_id.................0x15 (PortInfo)
> resv....................0x0
> attr_mod................0x17
> m_key...................0x0000000000000000
> dr_slid.................0xFFFF
> dr_dlid.................0xFFFF
>
> Initial path: [0][1][15][1][6]
> Return path: [0][9][18][D][3]
> Reserved: [0][0][0][0][0][0][0]
>
> 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
>
> 00 00 00 00 00 00 00 00 00 00 00 00 03 03 03 02
>
> 14 52 00 11 40 40 00 08 08 04 F2 40 00 00 00 00
>
> 00 00 00 00 00 88 00 00 00 00 00 00 00 00 00 00
>
> Mar 20 14:25:49 126526 [42803960] -> PortInfo dump:
> port number.............0x13
> node_guid...............0x0005ad00000281a7
> port_guid...............0x0005ad00000281a7
> m_key...................0x0000000000000000
> subnet_prefix...........0x0000000000000000
> base_lid................0x0
> master_sm_base_lid......0x0
> capability_mask.........0x0
> diag_code...............0x0
> m_key_lease_period......0x0
> local_port_num..........0x3
> link_width_enabled......0x3
> link_width_supported....0x3
> link_width_active.......0x2
> link_speed_supported....0x1
> port_state..............ACTIVE
> state_info2.............0x52
> m_key_protect_bits......0x0
> lmc.....................0x0
> link_speed..............0x11
> mtu_smsl................0x40
> vl_cap_init_type........0x40
> vl_high_limit...........0x0
> vl_arb_high_cap.........0x8
> vl_arb_low_cap..........0x8
> init_rep_mtu_cap........0x4
> vl_stall_life...........0xF2
> vl_enforce..............0x40
> m_key_violations........0x0
> p_key_violations........0x0
> q_key_violations........0x0
> guid_cap................0x0
> client_reregister.......0x0
> subnet_timeout..........0x0
> resp_time_value.........0x0
> error_threshold.........0x88
> Mar 20 14:25:49 126567 [42803960] -> Capabilities Mask:
> Mar 20 14:25:49 126613 [4780B960] -> __osm_sm_mad_ctrl_rcv_callback: ERR
> 3111: Error status = 0x1C00
> Mar 20 14:25:49 126617 [42803960] -> osm_pi_rcv_process_set: Received error
> status 0x1c for SetResp() during ACTIVE transition
> Mar 20 14:25:49 126658 [4780B960] -> SMP dump:
> base_ver................0x1
> mgmt_class..............0x81
> class_ver...............0x1
> method..................0x81 (SubnGetResp)
> D bit...................0x1
> status..................0x1C00
> hop_ptr.................0x0
> hop_count...............0x4
> trans_id................0x7d168
> attr_id.................0x15 (PortInfo)
> resv....................0x0
> attr_mod................0x18
> m_key...................0x0000000000000000
> dr_slid.................0xFFFF
> dr_dlid.................0xFFFF
>
> Initial path: [0][1][15][1][6]
> Return path: [0][9][18][D][3]
> Reserved: [0][0][0][0][0][0][0]
>
> 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
>
> 00 00 00 00 00 00 00 00 00 00 00 00 03 03 03 02
>
> 14 52 00 11 40 40 00 08 08 04 F2 40 00 00 00 00
>
> 00 00 00 00 00 88 00 00 00 00 00 00 00 00 00 00
>
> Mar 20 14:25:49 126653 [42803960] -> PortInfo dump:
> port number.............0x16
> node_guid...............0x0005ad00000281a7
> port_guid...............0x0005ad00000281a7
> m_key...................0x0000000000000000
> subnet_prefix...........0x0000000000000000
> base_lid................0x0
> master_sm_base_lid......0x0
> capability_mask.........0x0
> diag_code...............0x0
> m_key_lease_period......0x0
> local_port_num..........0x3
> link_width_enabled......0x3
> link_width_supported....0x3
> link_width_active.......0x2
> link_speed_supported....0x1
> port_state..............ACTIVE
> state_info2.............0x52
> m_key_protect_bits......0x0
> lmc.....................0x0
> link_speed..............0x11
> mtu_smsl................0x40
> vl_cap_init_type........0x40
> vl_high_limit...........0x0
> vl_arb_high_cap.........0x8
> vl_arb_low_cap..........0x8
> init_rep_mtu_cap........0x4
> vl_stall_life...........0xF2
> vl_enforce..............0x40
> m_key_violations........0x0
> p_key_violations........0x0
> q_key_violations........0x0
> guid_cap................0x0
> client_reregister.......0x0
> subnet_timeout..........0x0
> resp_time_value.........0x0
> error_threshold.........0x88
> Mar 20 14:25:49 126687 [42803960] -> Capabilities Mask:
> Mar 20 14:25:49 126703 [43204960] -> osm_pi_rcv_process_set: Received error
> status 0x1c for SetResp() during ACTIVE transition
> Mar 20 14:25:49 126709 [43204960] -> PortInfo dump:
> port number.............0x18
> node_guid...............0x0005ad00000281a7
> port_guid...............0x0005ad00000281a7
> m_key...................0x0000000000000000
> subnet_prefix...........0x0000000000000000
> base_lid................0x0
> master_sm_base_lid......0x0
> capability_mask.........0x0
> diag_code...............0x0
> m_key_lease_period......0x0
> local_port_num..........0x3
> link_width_enabled......0x3
> link_width_supported....0x3
> link_width_active.......0x2
> link_speed_supported....0x1
> port_state..............ACTIVE
> state_info2.............0x52
> m_key_protect_bits......0x0
> lmc.....................0x0
> link_speed..............0x11
> mtu_smsl................0x40
> vl_cap_init_type........0x40
> vl_high_limit...........0x0
> vl_arb_high_cap.........0x8
> vl_arb_low_cap..........0x8
> init_rep_mtu_cap........0x4
> vl_stall_life...........0xF2
> vl_enforce..............0x40
> m_key_violations........0x0
> p_key_violations........0x0
> q_key_violations........0x0
> guid_cap................0x0
> client_reregister.......0x0
> subnet_timeout..........0x0
> resp_time_value.........0x0
> error_threshold.........0x88
> Mar 20 14:25:49 126744 [43204960] -> Capabilities Mask:
> Mar 20 14:25:49 126765 [43C05960] -> osm_pi_rcv_process_set: Received error
> status 0x1c for SetResp() during ACTIVE transition
> Mar 20 14:25:49 126770 [43C05960] -> PortInfo dump:
> port number.............0x17
> node_guid...............0x0005ad00000281a7
> port_guid...............0x0005ad00000281a7
> m_key...................0x0000000000000000
> subnet_prefix...........0x0000000000000000
> base_lid................0x0
> master_sm_base_lid......0x0
> capability_mask.........0x0
> diag_code...............0x0
> m_key_lease_period......0x0
> local_port_num..........0x3
> link_width_enabled......0x3
> link_width_supported....0x3
> link_width_active.......0x2
> link_speed_supported....0x1
> port_state..............ACTIVE
> state_info2.............0x52
> m_key_protect_bits......0x0
> lmc.....................0x0
> link_speed..............0x11
> mtu_smsl................0x40
> vl_cap_init_type........0x40
> vl_high_limit...........0x0
> vl_arb_high_cap.........0x8
> vl_arb_low_cap..........0x8
> init_rep_mtu_cap........0x4
> vl_stall_life...........0xF2
> vl_enforce..............0x40
> m_key_violations........0x0
> p_key_violations........0x0
> q_key_violations........0x0
> guid_cap................0x0
> client_reregister.......0x0
> subnet_timeout..........0x0
> resp_time_value.........0x0
> error_threshold.........0x88
> Mar 20 14:25:49 126874 [43C05960] -> Capabilities Mask:
> Mar 20 14:25:49 126975 [4780B960] -> __osm_sm_mad_ctrl_rcv_callback: ERR
> 3111: Error status = 0x1C00
> Mar 20 14:25:49 127015 [4780B960] -> SMP dump:
> base_ver................0x1
> mgmt_class..............0x81
> class_ver...............0x1
> method..................0x81 (SubnGetResp)
> D bit...................0x1
> status..................0x1C00
> hop_ptr.................0x0
> hop_count...............0x4
> trans_id................0x7d169
> attr_id.................0x15 (PortInfo)
> resv....................0x0
> attr_mod................0x16
> m_key...................0x0000000000000000
> dr_slid.................0xFFFF
> dr_dlid.................0xFFFF
>
> Initial path: [0][1][13][1][6]
> Return path: [0][9][13][D][3]
> Reserved: [0][0][0][0][0][0][0]
>
> 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
>
> 00 00 00 00 00 00 00 00 00 00 00 00 03 03 03 02
>
> 14 52 00 11 40 40 00 08 08 04 F2 40 00 00 00 00
>
> 00 00 00 00 00 88 00 00 00 00 00 00 00 00 00 00
>
> Mar 20 14:25:49 127066 [45A08960] -> osm_pi_rcv_process_set: Received error
> status 0x1c for SetResp() during ACTIVE transition
> Mar 20 14:25:49 127072 [4780B960] -> __osm_sm_mad_ctrl_rcv_callback: ERR
> 3111: Error status = 0x1C00
> Mar 20 14:25:49 127084 [45A08960] -> PortInfo dump:
> port number.............0x16
> node_guid...............0x0005ad00000281b3
> port_guid...............0x0005ad00000281b3
> m_key...................0x0000000000000000
> subnet_prefix...........0x0000000000000000
> base_lid................0x0
> master_sm_base_lid......0x0
> capability_mask.........0x0
> diag_code...............0x0
> m_key_lease_period......0x0
> local_port_num..........0x3
> link_width_enabled......0x3
> link_width_supported....0x3
> link_width_active.......0x2
> link_speed_supported....0x1
> port_state..............ACTIVE
> state_info2.............0x52
> m_key_protect_bits......0x0
> lmc.....................0x0
> link_speed..............0x11
> mtu_smsl................0x40
> vl_cap_init_type........0x40
> vl_high_limit...........0x0
> vl_arb_high_cap.........0x8
> vl_arb_low_cap..........0x8
> init_rep_mtu_cap........0x4
> vl_stall_life...........0xF2
> vl_enforce..............0x40
> m_key_violations........0x0
> p_key_violations........0x0
> q_key_violations........0x0
> guid_cap................0x0
> client_reregister.......0x0
> subnet_timeout..........0x0
> resp_time_value.........0x0
> error_threshold.........0x88
> Mar 20 14:25:49 127103 [45A08960] -> Capabilities Mask:
> Mar 20 14:25:49 127121 [4780B960] -> SMP dump:
> base_ver................0x1
> mgmt_class..............0x81
> class_ver...............0x1
> method..................0x81 (SubnGetResp)
> D bit...................0x1
> status..................0x1C00
> hop_ptr.................0x0
> hop_count...............0x4
> trans_id................0x7d16a
> attr_id.................0x15 (PortInfo)
> resv....................0x0
> attr_mod................0x17
> m_key...................0x0000000000000000
> dr_slid.................0xFFFF
> dr_dlid.................0xFFFF
>
> Initial path: [0][1][13][1][6]
> Return path: [0][9][13][D][3]
> Reserved: [0][0][0][0][0][0][0]
>
> 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
>
> 00 00 00 00 00 00 00 00 00 00 00 00 03 03 03 02
>
> 14 52 00 11 40 40 00 08 08 04 F2 40 00 00 00 00
>
> 00 00 00 00 00 88 00 00 00 00 00 00 00 00 00 00
>
> Mar 20 14:25:49 127188 [4780B960] -> __osm_sm_mad_ctrl_rcv_callback: ERR
> 3111: Error status = 0x1C00
> Mar 20 14:25:49 127220 [4780B960] -> SMP dump:
> base_ver................0x1
> mgmt_class..............0x81
> class_ver...............0x1
> method..................0x81 (SubnGetResp)
> D bit...................0x1
> status..................0x1C00
> hop_ptr.................0x0
> hop_count...............0x4
> trans_id................0x7d16b
> attr_id.................0x15 (PortInfo)
> resv....................0x0
> attr_mod................0x18
> m_key...................0x0000000000000000
> dr_slid.................0xFFFF
> dr_dlid.................0xFFFF
>
> Initial path: [0][1][13][1][6]
> Return path: [0][9][13][D][3]
> Reserved: [0][0][0][0][0][0][0]
>
> 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
>
> 00 00 00 00 00 00 00 00 00 00 00 00 03 03 03 02
>
> 14 52 00 11 40 40 00 08 08 04 F2 40 00 00 00 00
>
> 00 00 00 00 00 88 00 00 00 00 00 00 00 00 00 00
>
> Mar 20 14:25:49 127326 [44606960] -> osm_pi_rcv_process_set: Received error
> status 0x1c for SetResp() during ACTIVE transition
> Mar 20 14:25:49 127339 [44606960] -> PortInfo dump:
> port number.............0x17
> node_guid...............0x0005ad00000281b3
> port_guid...............0x0005ad00000281b3
> m_key...................0x0000000000000000
> subnet_prefix...........0x0000000000000000
> base_lid................0x0
> master_sm_base_lid......0x0
> capability_mask.........0x0
> diag_code...............0x0
> m_key_lease_period......0x0
> local_port_num..........0x3
> link_width_enabled......0x3
> link_width_supported....0x3
> link_width_active.......0x2
> link_speed_supported....0x1
> port_state..............ACTIVE
> state_info2.............0x52
> m_key_protect_bits......0x0
> lmc.....................0x0
> link_speed..............0x11
> mtu_smsl................0x40
> vl_cap_init_type........0x40
> vl_high_limit...........0x0
> vl_arb_high_cap.........0x8
> vl_arb_low_cap..........0x8
> init_rep_mtu_cap........0x4
> vl_stall_life...........0xF2
> vl_enforce..............0x40
> m_key_violations........0x0
> p_key_violations........0x0
> q_key_violations........0x0
> guid_cap................0x0
> client_reregister.......0x0
> subnet_timeout..........0x0
> resp_time_value.........0x0
> error_threshold.........0x88
> Mar 20 14:25:49 127357 [44606960] -> Capabilities Mask:
> Mar 20 14:25:49 127378 [45007960] -> osm_pi_rcv_process_set: Received error
> status 0x1c for SetResp() during ACTIVE transition
> Mar 20 14:25:49 127397 [45007960] -> PortInfo dump:
> port number.............0x18
> node_guid...............0x0005ad00000281b3
> port_guid...............0x0005ad00000281b3
> m_key...................0x0000000000000000
> subnet_prefix...........0x0000000000000000
> base_lid................0x0
> master_sm_base_lid......0x0
> capability_mask.........0x0
> diag_code...............0x0
> m_key_lease_period......0x0
> local_port_num..........0x3
> link_width_enabled......0x3
> link_width_supported....0x3
> link_width_active.......0x2
> link_speed_supported....0x1
> port_state..............ACTIVE
> state_info2.............0x52
> m_key_protect_bits......0x0
> lmc.....................0x0
> link_speed..............0x11
> mtu_smsl................0x40
> vl_cap_init_type........0x40
> vl_high_limit...........0x0
> vl_arb_high_cap.........0x8
> vl_arb_low_cap..........0x8
> init_rep_mtu_cap........0x4
> vl_stall_life...........0xF2
> vl_enforce..............0x40
> m_key_violations........0x0
> p_key_violations........0x0
> q_key_violations........0x0
> guid_cap................0x0
> client_reregister.......0x0
> subnet_timeout..........0x0
> resp_time_value.........0x0
> error_threshold.........0x88
> Mar 20 14:25:49 127410 [45007960] -> Capabilities Mask:
> Mar 20 14:25:49 132961 [43204960] -> SUBNET UP
> Mar 20 14:25:49 523879 [44606960] -> osm_ucast_mgr_process: Min Hop Tables
> configured on all switches
> Mar 20 14:25:49 580522 [42803960] -> SUBNET UP
> Mar 20 14:26:04 718574 [43C05960] -> __osm_trap_rcv_process_request:
> Received Generic Notice type:0x01 num:128 Producer:2 from LID:0x0152
> TID:0x0000000000000000
> Mar 20 14:26:04 718819 [43C05960] -> osm_report_notice: Reporting Generic
> Notice type:1 num:128 from LID:0x0152
> GID:0xfe80000000000000,0x0005ad0000027c84
> Mar 20 14:26:04 836781 [45A08960] -> __osm_trap_rcv_process_request:
> Received Generic Notice type:0x01 num:128 Producer:2 from LID:0x0152
> TID:0x0000000000000001
> Mar 20 14:26:04 836881 [45A08960] -> osm_report_notice: Reporting Generic
> Notice type:1 num:128 from LID:0x0152
> GID:0xfe80000000000000,0x0005ad0000027c84
> Mar 20 14:26:04 858762 [45A08960] -> __osm_trap_rcv_process_request:
> Received Generic Notice type:0x01 num:128 Producer:2 from LID:0x0152
> TID:0x0000000000000002
> Mar 20 14:26:04 860242 [45A08960] -> osm_report_notice: Reporting Generic
> Notice type:1 num:128 from LID:0x0152
> GID:0xfe80000000000000,0x0005ad0000027c84
> Mar 20 14:26:04 997451 [45007960] -> __osm_trap_rcv_process_request:
> Received Generic Notice type:0x01 num:128 Producer:2 from LID:0x0152
> TID:0x0000000000000003
> Mar 20 14:26:04 997647 [45007960] -> osm_report_notice: Reporting Generic
> Notice type:1 num:128 from LID:0x0152
> GID:0xfe80000000000000,0x0005ad0000027c84
> Mar 20 14:26:05 180722 [43204960] -> __osm_trap_rcv_process_request:
> Received Generic Notice type:0x01 num:128 Producer:2 from LID:0x0152
> TID:0x0000000000000004
> Mar 20 14:26:05 180855 [43204960] -> osm_report_notice: Reporting Generic
> Notice type:1 num:128 from LID:0x0152
> GID:0xfe80000000000000,0x0005ad0000027c84
> Mar 20 14:26:05 209122 [41401960] -> __osm_trap_rcv_process_request:
> Received Generic Notice type:0x01 num:128 Producer:2 from LID:0x0152
> TID:0x0000000000000005
> Mar 20 14:26:05 209200 [41401960] -> osm_report_notice: Reporting Generic
> Notice type:1 num:128 from LID:0x0152
> GID:0xfe80000000000000,0x0005ad0000027c84
> Mar 20 14:26:05 347419 [45A08960] -> __osm_trap_rcv_process_request:
> Received Generic Notice type:0x01 num:128 Producer:2 from LID:0x0152
> TID:0x0000000000000006
> Mar 20 14:26:05 347488 [45A08960] -> osm_report_notice: Reporting Generic
> Notice type:1 num:128 from LID:0x0152
> GID:0xfe80000000000000,0x0005ad0000027c84
> Mar 20 14:26:05 378670 [42803960] -> __osm_trap_rcv_process_request:
> Received Generic Notice type:0x01 num:128 Producer:2 from LID:0x0152
> TID:0x0000000000000007
> Mar 20 14:26:05 378739 [42803960] -> osm_report_notice: Reporting Generic
> Notice type:1 num:128 from LID:0x0152
> GID:0xfe80000000000000,0x0005ad0000027c84
> Mar 20 14:26:05 409112 [41401960] -> osm_report_notice: Reporting Generic
> Notice type:3 num:65 from LID:0x0092
> GID:0xfe80000000000000,0x0005ad0000024bbb
> Mar 20 14:26:05 409121 [41401960] -> Removed port with
> GUID:0x0005ad00000249d3 LID range [0xB1,0xB1] of node:saguaro-23-2 HCA-1
> Mar 20 14:26:05 443639 [41401960] -> osm_ucast_mgr_process: Min Hop Tables
> configured on all switches
> Mar 20 14:26:05 483503 [45007960] -> __osm_trap_rcv_process_request:
> Received Generic Notice type:0x01 num:128 Producer:2 from LID:0x0152
> TID:0x0000000000000008
> Mar 20 14:26:05 486002 [45007960] -> osm_report_notice: Reporting Generic
> Notice type:1 num:128 from LID:0x0152
> GID:0xfe80000000000000,0x0005ad0000027c84
> Mar 20 14:26:05 499183 [44606960] -> SUBNET UP
> Mar 20 14:26:05 499856 [43C05960] -> __osm_trap_rcv_process_request:
> Received Generic Notice type:0x01 num:128 Producer:2 from LID:0x0152
> TID:0x0000000000000009
> Mar 20 14:26:05 499941 [43C05960] -> osm_report_notice: Reporting Generic
> Notice type:1 num:128 from LID:0x0152
> GID:0xfe80000000000000,0x0005ad0000027c84
> Mar 20 14:26:05 521857 [43204960] -> __osm_trap_rcv_process_request:
> Received Generic Notice type:0x01 num:128 Producer:2 from LID:0x0152
> TID:0x000000000000000a
> Mar 20 14:26:05 521971 [43204960] -> osm_report_notice: Reporting Generic
> Notice type:1 num:128 from LID:0x0152
> GID:0xfe80000000000000,0x0005ad0000027c84
> Mar 20 14:26:05 532569 [41401960] -> __osm_trap_rcv_process_request:
> Received Generic Notice type:0x01 num:128 Producer:2 from LID:0x0152
> TID:0x000000000000000b
> Mar 20 14:26:05 532624 [41401960] -> __osm_trap_rcv_process_request: ERR
> 3804: Received trap 11 times consecutively
> Mar 20 14:26:05 633813 [43204960] -> __osm_trap_rcv_process_request:
> Received Generic Notice type:0x01 num:128 Producer:2 from LID:0x0152
> TID:0x000000000000000c
> Mar 20 14:26:05 633869 [43204960] -> __osm_trap_rcv_process_request: ERR
> 3804: Received trap 12 times consecutively
> Mar 20 14:26:05 655421 [41401960] -> __osm_trap_rcv_process_request:
> Received Generic Notice type:0x01 num:128 Producer:2 from LID:0x0152
> TID:0x000000000000000d
> Mar 20 14:26:05 655501 [41401960] -> __osm_trap_rcv_process_request: ERR
> 3804: Received trap 13 times consecutively
> Mar 20 14:26:05 702652 [42803960] -> __osm_trap_rcv_process_request:
> Received Generic Notice type:0x01 num:128 Producer:2 from LID:0x0152
> TID:0x000000000000000e
> Mar 20 14:26:05 702745 [42803960] -> __osm_trap_rcv_process_request: ERR
> 3804: Received trap 14 times consecutively
> Mar 20 14:26:05 817201 [43204960] -> osm_report_notice: Reporting Generic
> Notice type:3 num:65 from LID:0x0092
> GID:0xfe80000000000000,0x0005ad0000024bbb
> Mar 20 14:26:05 817216 [43204960] -> Removed port with
> GUID:0x0005ad000002510b LID range [0xB5,0xB5] of node:saguaro-23-6 HCA-1
> Mar 20 14:26:05 817235 [43204960] -> osm_report_notice: Reporting Generic
> Notice type:3 num:65 from LID:0x0092
> GID:0xfe80000000000000,0x0005ad0000024bbb
> Mar 20 14:26:05 817241 [43204960] -> Removed port with
> GUID:0x0005ad0000024d47 LID range [0xB6,0xB6] of node:saguaro-23-7 HCA-1
> Mar 20 14:26:05 817259 [43204960] -> osm_report_notice: Reporting Generic
> Notice type:3 num:65 from LID:0x0092
> GID:0xfe80000000000000,0x0005ad0000024bbb
> Mar 20 14:26:05 817264 [43204960] -> Removed port with
> GUID:0x0005ad0000024d8b LID range [0xB7,0xB7] of node:saguaro-23-8 HCA-1
> Mar 20 14:26:05 821322 [41E02960] -> osm_report_notice: Reporting Generic
> Notice type:3 num:64 from LID:0x0092
> GID:0xfe80000000000000,0x0005ad0000024bbb
> Mar 20 14:26:05 821330 [41E02960] -> Discovered new port with
> GUID:0x0005ad00000249d3 LID range [0xB1,0xB1] of node:saguaro-23-2 HCA-1
> Mar 20 14:26:05 847950 [45007960] -> __osm_trap_rcv_process_request:
> Received Generic Notice type:0x01 num:128 Producer:2 from LID:0x0152
> TID:0x000000000000000f
> Mar 20 14:26:05 848031 [45007960] -> __osm_trap_rcv_process_request: ERR
> 3804: Received trap 15 times consecutively
> Mar 20 14:26:05 852036 [41E02960] -> osm_ucast_mgr_process: Min Hop Tables
> configured on all switches
> Mar 20 14:26:05 893954 [45007960] -> __osm_trap_rcv_process_request:
> Received Generic Notice type:0x01 num:128 Producer:2 from LID:0x0152
> TID:0x0000000000000010
> Mar 20 14:26:05 894021 [45007960] -> __osm_trap_rcv_process_request: ERR
> 3804: Received trap 16 times consecutively
> Mar 20 14:26:05 910489 [44606960] -> SUBNET UP
> Mar 20 14:26:05 999993 [43C05960] -> __osm_trap_rcv_process_request:
> Received Generic Notice type:0x01 num:128 Producer:2 from LID:0x0152
> TID:0x0000000000000011
> Mar 20 14:26:06 000039 [43C05960] -> __osm_trap_rcv_process_request: ERR
> 3804: Received trap 17 times consecutively
> Mar 20 14:26:06 021880 [45A08960] -> __osm_trap_rcv_process_request:
> Received Generic Notice type:0x01 num:128 Producer:2 from LID:0x0152
> TID:0x0000000000000012
> Mar 20 14:26:06 021970 [45A08960] -> __osm_trap_rcv_process_request: ERR
> 3804: Received trap 18 times consecutively
> Mar 20 14:26:06 043912 [44606960] -> __osm_trap_rcv_process_request:
> Received Generic Notice type:0x01 num:128 Producer:2 from LID:0x0152
> TID:0x0000000000000013
> Mar 20 14:26:06 044001 [44606960] -> __osm_trap_rcv_process_request: ERR
> 3804: Received trap 19 times consecutively
> Mar 20 14:26:06 052878 [44606960] -> __osm_trap_rcv_process_request:
> Received Generic Notice type:0x01 num:128 Producer:2 from LID:0x0152
> TID:0x0000000000000014
> Mar 20 14:26:06 052975 [44606960] -> __osm_trap_rcv_process_request: ERR
> 3804: Received trap 20 times consecutively
> Mar 20 14:26:06 147560 [42803960] -> __osm_trap_rcv_process_request:
> Received Generic Notice type:0x01 num:128 Producer:2 from LID:0x0152
> TID:0x0000000000000015
> Mar 20 14:26:06 147616 [42803960] -> __osm_trap_rcv_process_request: ERR
> 3804: Received trap 21 times consecutively
> Mar 20 14:26:06 158945 [41401960] -> __osm_trap_rcv_process_request:
> Received Generic Notice type:0x01 num:128 Producer:2 from LID:0x0152
> TID:0x0000000000000016
> Mar 20 14:26:06 158978 [41401960] -> __osm_trap_rcv_process_request: ERR
> 3804: Received trap 22 times consecutively
> Mar 20 14:26:06 346046 [44606960] -> __osm_trap_rcv_process_request:
> Received Generic Notice type:0x01 num:128 Producer:2 from LID:0x0152
> TID:0x0000000000000017
> Mar 20 14:26:06 346106 [44606960] -> __osm_trap_rcv_process_request: ERR
> 3804: Received trap 23 times consecutively
> Mar 20 14:26:06 405311 [43204960] -> __osm_trap_rcv_process_request:
> Received Generic Notice type:0x01 num:128 Producer:2 from LID:0x0152
> TID:0x0000000000000018
> Mar 20 14:26:06 405349 [43204960] -> __osm_trap_rcv_process_request: ERR
> 3804: Received trap 24 times consecutively
> Mar 20 14:26:06 632882 [45007960] -> __osm_trap_rcv_process_request:
> Received Generic Notice type:0x01 num:128 Producer:2 from LID:0x0152
> TID:0x0000000000000019
> Mar 20 14:26:06 632923 [45007960] -> __osm_trap_rcv_process_request: ERR
> 3804: Received trap 25 times consecutively
> Mar 20 14:26:06 634031 [43C05960] -> __osm_trap_rcv_process_request:
> Received Generic Notice type:0x01 num:128 Producer:2 from LID:0x0148
> TID:0x0000000000000067
> Mar 20 14:26:06 634110 [43C05960] -> osm_report_notice: Reporting Generic
> Notice type:1 num:128 from LID:0x0148
> GID:0xfe80000000000000,0x0005ad00000281b3
> Mar 20 14:26:06 883831 [45007960] -> __osm_trap_rcv_process_request:
> Received Generic Notice type:0x01 num:128 Producer:2 from LID:0x0152
> TID:0x000000000000001a
> Mar 20 14:26:06 883879 [45007960] -> __osm_trap_rcv_process_request: ERR
> 3804: Received trap 26 times consecutively
> Mar 20 14:26:06 885475 [43C05960] -> __osm_trap_rcv_process_request:
> Received Generic Notice type:0x01 num:128 Producer:2 from LID:0x0148
> TID:0x0000000000000068
> Mar 20 14:26:06 885560 [43C05960] -> osm_report_notice: Reporting Generic
> Notice type:1 num:128 from LID:0x0148
> GID:0xfe80000000000000,0x0005ad00000281b3
> Mar 20 14:26:06 982877 [43204960] -> __osm_trap_rcv_process_request:
> Received Generic Notice type:0x01 num:128 Producer:2 from LID:0x0152
> TID:0x000000000000001b
> Mar 20 14:26:06 982926 [43204960] -> __osm_trap_rcv_process_request: ERR
> 3804: Received trap 27 times consecutively
> Mar 20 14:26:06 992809 [41E02960] -> __osm_trap_rcv_process_request:
> Received Generic Notice type:0x01 num:128 Producer:2 from LID:0x0148
> TID:0x0000000000000069
> Mar 20 14:26:06 992871 [41E02960] -> osm_report_notice: Reporting Generic
> Notice type:1 num:128 from LID:0x0148
> GID:0xfe80000000000000,0x0005ad00000281b3
> Mar 20 14:26:06 992909 [41E02960] -> __osm_trap_rcv_process_request:
> Received Generic Notice type:0x01 num:128 Producer:2 from LID:0x0152
> TID:0x000000000000001c
> Mar 20 14:26:06 992943 [41E02960] -> __osm_trap_rcv_process_request: ERR
> 3804: Received trap 28 times consecutively
> Mar 20 14:26:06 993058 [41E02960] -> osm_report_notice: Reporting Generic
> Notice type:3 num:64 from LID:0x0092
> GID:0xfe80000000000000,0x0005ad0000024bbb
> Mar 20 14:26:06 993065 [41E02960] -> Discovered new port with
> GUID:0x0005ad000002510b LID range [0xB5,0xB5] of node:saguaro-23-6 HCA-1
> Mar 20 14:26:06 993069 [41E02960] -> osm_report_notice: Reporting Generic
> Notice type:3 num:64 from LID:0x0092
> GID:0xfe80000000000000,0x0005ad0000024bbb
> Mar 20 14:26:06 993074 [41E02960] -> Discovered new port with
> GUID:0x0005ad0000024d47 LID range [0xB6,0xB6] of node:saguaro-23-7 HCA-1
> Mar 20 14:26:07 023890 [41E02960] -> osm_ucast_mgr_process: Min Hop Tables
> configured on all switches
> Mar 20 14:26:07 085081 [41E02960] -> SUBNET UP
> Mar 20 14:26:07 348105 [45A08960] -> __osm_trap_rcv_process_request:
> Received Generic Notice type:0x01 num:128 Producer:2 from LID:0x0152
> TID:0x000000000000001d
> Mar 20 14:26:07 348218 [45A08960] -> __osm_trap_rcv_process_request: ERR
> 3804: Received trap 29 times consecutively
> Mar 20 14:26:07 348958 [45A08960] -> __osm_trap_rcv_process_request:
> Received Generic Notice type:0x01 num:128 Producer:2 from LID:0x0148
> TID:0x000000000000006a
> Mar 20 14:26:07 349041 [45A08960] -> osm_report_notice: Reporting Generic
> Notice type:1 num:128 from LID:0x0148
> GID:0xfe80000000000000,0x0005ad00000281b3
> Mar 20 14:26:07 540871 [41401960] -> __osm_trap_rcv_process_request:
> Received Generic Notice type:0x01 num:128 Producer:2 from LID:0x0148
> TID:0x000000000000006b
> Mar 20 14:26:07 540983 [41401960] -> osm_report_notice: Reporting Generic
> Notice type:1 num:128 from LID:0x0148
> GID:0xfe80000000000000,0x0005ad00000281b3
> Mar 20 14:26:07 541063 [43204960] -> __osm_trap_rcv_process_request:
> Received Generic Notice type:0x01 num:128 Producer:2 from LID:0x001B
> TID:0x000000000000007c
> Mar 20 14:26:07 541131 [43204960] -> osm_report_notice: Reporting Generic
> Notice type:1 num:128 from LID:0x001B
> GID:0xfe80000000000000,0x0005ad00000281a7
> Mar 20 14:26:07 585394 [43C05960] -> __osm_trap_rcv_process_request:
> Received Generic Notice type:0x01 num:128 Producer:2 from LID:0x0148
> TID:0x000000000000006c
> Mar 20 14:26:07 585464 [43C05960] -> osm_report_notice: Reporting Generic
> Notice type:1 num:128 from LID:0x0148
> GID:0xfe80000000000000,0x0005ad00000281b3
> Mar 20 14:26:07 607406 [45A08960] -> __osm_trap_rcv_process_request:
> Received Generic Notice type:0x01 num:128 Producer:2 from LID:0x0148
> TID:0x000000000000006d
> Mar 20 14:26:07 607486 [45A08960] -> osm_report_notice: Reporting Generic
> Notice type:1 num:128 from LID:0x0148
> GID:0xfe80000000000000,0x0005ad00000281b3
> Mar 20 14:26:07 850410 [42803960] -> __osm_trap_rcv_process_request:
> Received Generic Notice type:0x01 num:128 Producer:2 from LID:0x0148
> TID:0x000000000000006e
> Mar 20 14:26:07 850483 [42803960] -> osm_report_notice: Reporting Generic
> Notice type:1 num:128 from LID:0x0148
> GID:0xfe80000000000000,0x0005ad00000281b3
> Mar 20 14:26:07 956365 [41E02960] -> __osm_trap_rcv_process_request:
> Received Generic Notice type:0x01 num:128 Producer:2 from LID:0x001B
> TID:0x000000000000007d
> Mar 20 14:26:07 956404 [42803960] -> osm_report_notice: Reporting Generic
> Notice type:3 num:64 from LID:0x0092
> GID:0xfe80000000000000,0x0005ad0000024bbb
> Mar 20 14:26:07 956413 [42803960] -> Discovered new port with
> GUID:0x0005ad0000024d8b LID range [0xB7,0xB7] of node:saguaro-23-8 HCA-1
> Mar 20 14:26:07 987136 [42803960] -> osm_ucast_mgr_process: Min Hop Tables
> configured on all switches
> Mar 20 14:26:08 018887 [41E02960] -> osm_report_notice: Reporting Generic
> Notice type:1 num:128 from LID:0x001B
> GID:0xfe80000000000000,0x0005ad00000281a7
> Mar 20 14:26:08 032634 [4780B960] -> __osm_sm_mad_ctrl_rcv_callback: ERR
> 3111: Error status = 0x1C00
> Mar 20 14:26:08 032679 [4780B960] -> SMP dump:
> base_ver................0x1
> mgmt_class..............0x81
> class_ver...............0x1
> method..................0x81 (SubnGetResp)
> D bit...................0x1
> status..................0x1C00
> hop_ptr.................0x0
> hop_count...............0x4
> trans_id................0x813ce
> attr_id.................0x15 (PortInfo)
> resv....................0x0
> attr_mod................0x16
> m_key...................0x0000000000000000
> dr_slid.................0xFFFF
> dr_dlid.................0xFFFF
>
> Initial path: [0][1][12][4][5]
> Return path: [0][9][14][D][5]
> Reserved: [0][0][0][0][0][0][0]
>
> 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
>
> 00 00 00 00 00 00 00 00 00 00 00 00 05 03 03 02
>
> 14 52 00 11 40 40 00 08 08 04 F2 40 00 00 00 00
>
> 00 00 00 00 00 88 00 00 00 00 00 00 00 00 00 00
>
> Mar 20 14:26:08 032749 [41E02960] -> osm_pi_rcv_process_set: Received error
> status 0x1c for SetResp() during ACTIVE transition
> Mar 20 14:26:08 032757 [41E02960] -> PortInfo dump:
> port number.............0x16
> node_guid...............0x0005ad00000281b3
> port_guid...............0x0005ad00000281b3
> m_key...................0x0000000000000000
> subnet_prefix...........0x0000000000000000
> base_lid................0x0
> master_sm_base_lid......0x0
> capability_mask.........0x0
> diag_code...............0x0
> m_key_lease_period......0x0
> local_port_num..........0x5
> link_width_enabled......0x3
> link_width_supported....0x3
> link_width_active.......0x2
> link_speed_supported....0x1
> port_state..............ACTIVE
> state_info2.............0x52
> m_key_protect_bits......0x0
> lmc.....................0x0
> link_speed..............0x11
> mtu_smsl................0x40
> vl_cap_init_type........0x40
> vl_high_limit...........0x0
> vl_arb_high_cap.........0x8
> vl_arb_low_cap..........0x8
> init_rep_mtu_cap........0x4
> vl_stall_life...........0xF2
> vl_enforce..............0x40
> m_key_violations........0x0
> p_key_violations........0x0
> q_key_violations........0x0
> guid_cap................0x0
> client_reregister.......0x0
> subnet_timeout..........0x0
> resp_time_value.........0x0
> error_threshold.........0x88
> Mar 20 14:26:08 032774 [41E02960] -> Capabilities Mask:
> Mar 20 14:26:08 033119 [4780B960] -> __osm_sm_mad_ctrl_rcv_callback: ERR
> 3111: Error status = 0x1C00
> Mar 20 14:26:08 033154 [4780B960] -> SMP dump:
> base_ver................0x1
> mgmt_class..............0x81
> class_ver...............0x1
> method..................0x81 (SubnGetResp)
> D bit...................0x1
> status..................0x1C00
> hop_ptr.................0x0
> hop_count...............0x4
> trans_id................0x813cf
> attr_id.................0x15 (PortInfo)
> resv....................0x0
> attr_mod................0x17
> m_key...................0x0000000000000000
> dr_slid.................0xFFFF
> dr_dlid.................0xFFFF
>
> Initial path: [0][1][12][4][5]
> Return path: [0][9][14][D][5]
> Reserved: [0][0][0][0][0][0][0]
>
> 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
>
> 00 00 00 00 00 00 00 00 00 00 00 00 05 03 03 02
>
> 14 52 00 11 40 40 00 08 08 04 F2 40 00 00 00 00
>
> 00 00 00 00 00 88 00 00 00 00 00 00 00 00 00 00
>
> Mar 20 14:26:08 033202 [43C05960] -> osm_pi_rcv_process_set: Received error
> status 0x1c for SetResp() during ACTIVE transition
> Mar 20 14:26:08 033213 [43C05960] -> PortInfo dump:
> port number.............0x17
> node_guid...............0x0005ad00000281b3
> port_guid...............0x0005ad00000281b3
> m_key...................0x0000000000000000
> subnet_prefix...........0x0000000000000000
> base_lid................0x0
> master_sm_base_lid......0x0
> capability_mask.........0x0
> diag_code...............0x0
> m_key_lease_period......0x0
> local_port_num..........0x5
> link_width_enabled......0x3
> link_width_supported....0x3
> link_width_active.......0x2
> link_speed_supported....0x1
> port_state..............ACTIVE
> state_info2.............0x52
> m_key_protect_bits......0x0
> lmc.....................0x0
> link_speed..............0x11
> mtu_smsl................0x40
> vl_cap_init_type........0x40
> vl_high_limit...........0x0
> vl_arb_high_cap.........0x8
> vl_arb_low_cap..........0x8
> init_rep_mtu_cap........0x4
> vl_stall_life...........0xF2
> vl_enforce..............0x40
> m_key_violations........0x0
> p_key_violations........0x0
> q_key_violations........0x0
> guid_cap................0x0
> client_reregister.......0x0
> subnet_timeout..........0x0
> resp_time_value.........0x0
> error_threshold.........0x88
> Mar 20 14:26:08 033231 [43C05960] -> Capabilities Mask:
> Mar 20 14:26:08 038497 [45A08960] -> SUBNET UP
> Mar 20 14:26:08 055480 [43C05960] -> __osm_trap_rcv_process_request:
> Received Generic Notice type:0x01 num:128 Producer:2 from LID:0x001B
> TID:0x000000000000007e
> Mar 20 14:26:08 055587 [43C05960] -> osm_report_notice: Reporting Generic
> Notice type:1 num:128 from LID:0x001B
> GID:0xfe80000000000000,0x0005ad00000281a7
> Mar 20 14:26:08 372288 [43204960] -> __osm_trap_rcv_process_request:
> Received Generic Notice type:0x01 num:128 Producer:2 from LID:0x001B
> TID:0x000000000000007f
> Mar 20 14:26:08 376158 [42803960] -> osm_ucast_mgr_process: Min Hop Tables
> configured on all switches
> Mar 20 14:26:08 418607 [44606960] -> __osm_trap_rcv_process_request:
> Received Generic Notice type:0x01 num:128 Producer:2 from LID:0x001B
> TID:0x0000000000000080
> Mar 20 14:26:08 420668 [43204960] -> osm_report_notice: Reporting Generic
> Notice type:1 num:128 from LID:0x001B
> GID:0xfe80000000000000,0x0005ad00000281a7
> Mar 20 14:26:08 420714 [44606960] -> osm_report_notice: Reporting Generic
> Notice type:1 num:128 from LID:0x001B
> GID:0xfe80000000000000,0x0005ad00000281a7
> Mar 20 14:26:08 430046 [4780B960] -> __osm_sm_mad_ctrl_rcv_callback: ERR
> 3111: Error status = 0x1C00
> Mar 20 14:26:08 430147 [4780B960] -> SMP dump:
> base_ver................0x1
> mgmt_class..............0x81
> class_ver...............0x1
> method..................0x81 (SubnGetResp)
> D bit...................0x1
> status..................0x1C00
> hop_ptr.................0x0
> hop_count...............0x4
> trans_id................0x820fa
> attr_id.................0x15 (PortInfo)
> resv....................0x0
> attr_mod................0x16
> m_key...................0x0000000000000000
> dr_slid.................0xFFFF
> dr_dlid.................0xFFFF
>
> Initial path: [0][1][15][1][4]
> Return path: [0][9][18][D][1]
> Reserved: [0][0][0][0][0][0][0]
>
> 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
>
> 00 00 00 00 00 00 00 00 00 00 00 00 01 03 03 02
>
> 14 52 00 11 40 40 00 08 08 04 F2 40 00 00 00 00
>
> 00 00 00 00 00 88 00 00 00 00 00 00 00 00 00 00
>
> Mar 20 14:26:08 430236 [4780B960] -> __osm_sm_mad_ctrl_rcv_callback: ERR
> 3111: Error status = 0x1C00
> Mar 20 14:26:08 430236 [43C05960] -> osm_pi_rcv_process_set: Received error
> status 0x1c for SetResp() during ACTIVE transition
> Mar 20 14:26:08 430267 [4780B960] -> SMP dump:
> base_ver................0x1
> mgmt_class..............0x81
> class_ver...............0x1
> method..................0x81 (SubnGetResp)
> D bit...................0x1
> status..................0x1C00
> hop_ptr.................0x0
> hop_count...............0x4
> trans_id................0x820fb
> attr_id.................0x15 (PortInfo)
> resv....................0x0
> attr_mod................0x18
> m_key...................0x0000000000000000
> dr_slid.................0xFFFF
> dr_dlid.................0xFFFF
>
> Initial path: [0][1][12][1][6]
> Return path: [0][9][14][D][3]
> Reserved: [0][0][0][0][0][0][0]
>
> 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
>
> 00 00 00 00 00 00 00 00 00 00 00 00 03 03 03 02
>
> 14 52 00 11 40 40 00 08 08 04 F2 40 00 00 00 00
>
> 00 00 00 00 00 88 00 00 00 00 00 00 00 00 00 00
>
> Mar 20 14:26:08 430262 [43C05960] -> PortInfo dump:
> port number.............0x16
> node_guid...............0x0005ad00000281a7
> port_guid...............0x0005ad00000281a7
> m_key...................0x0000000000000000
> subnet_prefix...........0x0000000000000000
> base_lid................0x0
> master_sm_base_lid......0x0
> capability_mask.........0x0
> diag_code...............0x0
> m_key_lease_period......0x0
> local_port_num..........0x1
> link_width_enabled......0x3
> link_width_supported....0x3
> link_width_active.......0x2
> link_speed_supported....0x1
> port_state..............ACTIVE
> state_info2.............0x52
> m_key_protect_bits......0x0
> lmc.....................0x0
> link_speed..............0x11
> mtu_smsl................0x40
> vl_cap_init_type........0x40
> vl_high_limit...........0x0
> vl_arb_high_cap.........0x8
> vl_arb_low_cap..........0x8
> init_rep_mtu_cap........0x4
> vl_stall_life...........0xF2
> vl_enforce..............0x40
> m_key_violations........0x0
> p_key_violations........0x0
> q_key_violations........0x0
> guid_cap................0x0
> client_reregister.......0x0
> subnet_timeout..........0x0
> resp_time_value.........0x0
> error_threshold.........0x88
> Mar 20 14:26:08 430286 [43C05960] -> Capabilities Mask:
> Mar 20 14:26:08 430350 [43C05960] -> osm_pi_rcv_process_set: Received error
> status 0x1c for SetResp() during ACTIVE transition
> Mar 20 14:26:08 430362 [43C05960] -> PortInfo dump:
> port number.............0x18
> node_guid...............0x0005ad00000281b3
> port_guid...............0x0005ad00000281b3
> m_key...................0x0000000000000000
> subnet_prefix...........0x0000000000000000
> base_lid................0x0
> master_sm_base_lid......0x0
> capability_mask.........0x0
> diag_code...............0x0
> m_key_lease_period......0x0
> local_port_num..........0x3
> link_width_enabled......0x3
> link_width_supported....0x3
> link_width_active.......0x2
> link_speed_supported....0x1
> port_state..............ACTIVE
> state_info2.............0x52
> m_key_protect_bits......0x0
> lmc.....................0x0
> link_speed..............0x11
> mtu_smsl................0x40
> vl_cap_init_type........0x40
> vl_high_limit...........0x0
> vl_arb_high_cap.........0x8
> vl_arb_low_cap..........0x8
> init_rep_mtu_cap........0x4
> vl_stall_life...........0xF2
> vl_enforce..............0x40
> m_key_violations........0x0
> p_key_violations........0x0
> q_key_violations........0x0
> guid_cap................0x0
> client_reregister.......0x0
> subnet_timeout..........0x0
> resp_time_value.........0x0
> error_threshold.........0x88
> Mar 20 14:26:08 430375 [43C05960] -> Capabilities Mask:
> Mar 20 14:26:08 435317 [43C05960] -> SUBNET UP
> Mar 20 14:26:08 583769 [41401960] -> __osm_trap_rcv_process_request:
> Received Generic Notice type:0x01 num:128 Producer:2 from LID:0x0152
> TID:0x000000000000001e
> Mar 20 14:26:08 583903 [41401960] -> __osm_trap_rcv_process_request: ERR
> 3804: Received trap 30 times consecutively
> Mar 20 14:26:08 854841 [41E02960] -> osm_ucast_mgr_process: Min Hop Tables
> configured on all switches
> Mar 20 14:26:08 913273 [4780B960] -> __osm_sm_mad_ctrl_rcv_callback: ERR
> 3111: Error status = 0x1C00
> Mar 20 14:26:08 913349 [4780B960] -> SMP dump:
> base_ver................0x1
> mgmt_class..............0x81
> class_ver...............0x1
> method..................0x81 (SubnGetResp)
> D bit...................0x1
> status..................0x1C00
> hop_ptr.................0x0
> hop_count...............0x4
> trans_id................0x82e32
> attr_id.................0x15 (PortInfo)
> resv....................0x0
> attr_mod................0x13
> m_key...................0x0000000000000000
> dr_slid.................0xFFFF
> dr_dlid.................0xFFFF
>
> Initial path: [0][1][17][2][5]
> Return path: [0][9][14][E][2]
> Reserved: [0][0][0][0][0][0][0]
>
> 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
>
> 00 00 00 00 00 00 00 00 00 00 00 00 02 03 03 02
>
> 14 52 00 11 40 40 00 08 08 04 F2 40 00 00 00 00
>
> 00 00 00 00 00 88 00 00 00 00 00 00 00 00 00 00
>
> Mar 20 14:26:08 913415 [45A08960] -> osm_pi_rcv_process_set: Received error
> status 0x1c for SetResp() during ACTIVE transition
> Mar 20 14:26:08 913432 [45A08960] -> PortInfo dump:
> port number.............0x13
> node_guid...............0x0005ad00000281a7
> port_guid...............0x0005ad00000281a7
> m_key...................0x0000000000000000
> subnet_prefix...........0x0000000000000000
> base_lid................0x0
> master_sm_base_lid......0x0
> capability_mask.........0x0
> diag_code...............0x0
> m_key_lease_period......0x0
> local_port_num..........0x2
> link_width_enabled......0x3
> link_width_supported....0x3
> link_width_active.......0x2
> link_speed_supported....0x1
> port_state..............ACTIVE
> state_info2.............0x52
> m_key_protect_bits......0x0
> lmc.....................0x0
> link_speed..............0x11
> mtu_smsl................0x40
> vl_cap_init_type........0x40
> vl_high_limit...........0x0
> vl_arb_high_cap.........0x8
> vl_arb_low_cap..........0x8
> init_rep_mtu_cap........0x4
> vl_stall_life...........0xF2
> vl_enforce..............0x40
> m_key_violations........0x0
> p_key_violations........0x0
> q_key_violations........0x0
> guid_cap................0x0
> client_reregister.......0x0
> subnet_timeout..........0x0
> resp_time_value.........0x0
> error_threshold.........0x88
> Mar 20 14:26:08 913449 [45A08960] -> Capabilities Mask:
> Mar 20 14:26:08 913598 [4780B960] -> __osm_sm_mad_ctrl_rcv_callback: ERR
> 3111: Error status = 0x1C00
> Mar 20 14:26:08 913676 [4780B960] -> SMP dump:
> base_ver................0x1
> mgmt_class..............0x81
> class_ver...............0x1
> method..................0x81 (SubnGetResp)
> D bit...................0x1
> status..................0x1C00
> hop_ptr.................0x0
> hop_count...............0x4
> trans_id................0x82e33
> attr_id.................0x15 (PortInfo)
> resv....................0x0
> attr_mod................0x17
> m_key...................0x0000000000000000
> dr_slid.................0xFFFF
> dr_dlid.................0xFFFF
>
> Initial path: [0][1][17][2][5]
> Return path: [0][9][14][E][2]
> Reserved: [0][0][0][0][0][0][0]
>
> 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
>
> 00 00 00 00 00 00 00 00 00 00 00 00 02 03 03 02
>
> 14 52 00 11 40 40 00 08 08 04 F2 40 00 00 00 00
>
> 00 00 00 00 00 88 00 00 00 00 00 00 00 00 00 00
>
> Mar 20 14:26:08 913727 [43C05960] -> osm_pi_rcv_process_set: Received error
> status 0x1c for SetResp() during ACTIVE transition
> Mar 20 14:26:08 913732 [4780B960] -> __osm_sm_mad_ctrl_rcv_callback: ERR
> 3111: Error status = 0x1C00
> Mar 20 14:26:08 913734 [43C05960] -> PortInfo dump:
> port number.............0x17
> node_guid...............0x0005ad00000281a7
> port_guid...............0x0005ad00000281a7
> m_key...................0x0000000000000000
> subnet_prefix...........0x0000000000000000
> base_lid................0x0
> master_sm_base_lid......0x0
> capability_mask.........0x0
> diag_code...............0x0
> m_key_lease_period......0x0
> local_port_num..........0x2
> link_width_enabled......0x3
> link_width_supported....0x3
> link_width_active.......0x2
> link_speed_supported....0x1
> port_state..............ACTIVE
> state_info2.............0x52
> m_key_protect_bits......0x0
> lmc.....................0x0
> link_speed..............0x11
> mtu_smsl................0x40
> vl_cap_init_type........0x40
> vl_high_limit...........0x0
> vl_arb_high_cap.........0x8
> vl_arb_low_cap..........0x8
> init_rep_mtu_cap........0x4
> vl_stall_life...........0xF2
> vl_enforce..............0x40
> m_key_violations........0x0
> p_key_violations........0x0
> q_key_violations........0x0
> guid_cap................0x0
> client_reregister.......0x0
> subnet_timeout..........0x0
> resp_time_value.........0x0
> error_threshold.........0x88
> Mar 20 14:26:08 913752 [43C05960] -> Capabilities Mask:
> Mar 20 14:26:08 913766 [4780B960] -> SMP dump:
> base_ver................0x1
> mgmt_class..............0x81
> class_ver...............0x1
> method..................0x81 (SubnGetResp)
> D bit...................0x1
> status..................0x1C00
> hop_ptr.................0x0
> hop_count...............0x4
> trans_id................0x82e34
> attr_id.................0x15 (PortInfo)
> resv....................0x0
> attr_mod................0x18
> m_key...................0x0000000000000000
> dr_slid.................0xFFFF
> dr_dlid.................0xFFFF
>
> Initial path: [0][1][17][2][5]
> Return path: [0][9][14][E][2]
> Reserved: [0][0][0][0][0][0][0]
>
> 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
>
> 00 00 00 00 00 00 00 00 00 00 00 00 02 03 03 02
>
> 14 52 00 11 40 40 00 08 08 04 F2 40 00 00 00 00
>
> 00 00 00 00 00 88 00 00 00 00 00 00 00 00 00 00
>
> Mar 20 14:26:08 913828 [41E02960] -> osm_pi_rcv_process_set: Received error
> status 0x1c for SetResp() during ACTIVE transition
> Mar 20 14:26:08 913833 [41E02960] -> PortInfo dump:
> port number.............0x18
> node_guid...............0x0005ad00000281a7
> port_guid...............0x0005ad00000281a7
> m_key...................0x0000000000000000
> subnet_prefix...........0x0000000000000000
> base_lid................0x0
> master_sm_base_lid......0x0
> capability_mask.........0x0
> diag_code...............0x0
> m_key_lease_period......0x0
> local_port_num..........0x2
> link_width_enabled......0x3
> link_width_supported....0x3
> link_width_active.......0x2
> link_speed_supported....0x1
> port_state..............ACTIVE
> state_info2.............0x52
> m_key_protect_bits......0x0
> lmc.....................0x0
> link_speed..............0x11
> mtu_smsl................0x40
> vl_cap_init_type........0x40
> vl_high_limit...........0x0
> vl_arb_high_cap.........0x8
> vl_arb_low_cap..........0x8
> init_rep_mtu_cap........0x4
> vl_stall_life...........0xF2
> vl_enforce..............0x40
> m_key_violations........0x0
> p_key_violations........0x0
> q_key_violations........0x0
> guid_cap................0x0
> client_reregister.......0x0
> subnet_timeout..........0x0
> resp_time_value.........0x0
> error_threshold.........0x88
> Mar 20 14:26:08 913848 [41E02960] -> Capabilities Mask:
> Mar 20 14:26:08 918887 [41E02960] -> SUBNET UP
> Mar 20 14:26:48 657517 [41E02960] -> __osm_trap_rcv_process_request:
> Received Generic Notice type:0x01 num:128 Producer:2 from LID:0x0148
> TID:0x000000000000006f
> Mar 20 14:26:48 657779 [41E02960] -> osm_report_notice: Reporting Generic
> Notice type:1 num:128 from LID:0x0148
> GID:0xfe80000000000000,0x0005ad00000281b3
> Mar 20 14:26:48 658393 [43204960] -> __osm_trap_rcv_process_request:
> Received Generic Notice type:0x01 num:128 Producer:2 from LID:0x001B
> TID:0x0000000000000081
> Mar 20 14:26:48 658465 [43204960] -> osm_report_notice: Reporting Generic
> Notice type:1 num:128 from LID:0x001B
> GID:0xfe80000000000000,0x0005ad00000281a7
> Mar 20 14:26:48 979610 [41401960] -> osm_report_notice: Reporting Generic
> Notice type:3 num:65 from LID:0x0092
> GID:0xfe80000000000000,0x0005ad0000024bbb
> Mar 20 14:26:48 979629 [41401960] -> Removed port with
> GUID:0x0005ad0000024e0b LID range [0xB3,0xB3] of node:saguaro-23-4 HCA-1
> Mar 20 14:26:48 979652 [41401960] -> osm_report_notice: Reporting Generic
> Notice type:3 num:65 from LID:0x0092
> GID:0xfe80000000000000,0x0005ad0000024bbb
> Mar 20 14:26:48 979660 [41401960] -> Removed port with
> GUID:0x0005ad000002510b LID range [0xB5,0xB5] of node:saguaro-23-6 HCA-1
> Mar 20 14:26:48 979682 [41401960] -> osm_report_notice: Reporting Generic
> Notice type:3 num:65 from LID:0x0092
> GID:0xfe80000000000000,0x0005ad0000024bbb
> Mar 20 14:26:48 979688 [41401960] -> Removed port with
> GUID:0x0005ad000002511b LID range [0xA6,0xA6] of node:saguaro-22-1 HCA-1
> Mar 20 14:26:48 979721 [41401960] -> osm_report_notice: Reporting Generic
> Notice type:3 num:65 from LID:0x0092
> GID:0xfe80000000000000,0x0005ad0000024bbb
> Mar 20 14:26:48 979727 [41401960] -> Removed port with
> GUID:0x0005ad0000024b27 LID range [0xAF,0xAF] of node:saguaro-23-0 HCA-1
> Mar 20 14:26:48 979770 [41401960] -> osm_report_notice: Reporting Generic
> Notice type:3 num:65 from LID:0x0092
> GID:0xfe80000000000000,0x0005ad0000024bbb
> Mar 20 14:26:48 979782 [41401960] -> Removed port with
> GUID:0x0005ad0000025043 LID range [0xB4,0xB4] of node:saguaro-23-5 HCA-1
> Mar 20 14:26:48 979799 [41401960] -> osm_report_notice: Reporting Generic
> Notice type:3 num:65 from LID:0x0092
> GID:0xfe80000000000000,0x0005ad0000024bbb
> Mar 20 14:26:48 979804 [41401960] -> Removed port with
> GUID:0x0005ad0000024d47 LID range [0xB6,0xB6] of node:saguaro-23-7 HCA-1
> Mar 20 14:26:48 979822 [41401960] -> osm_report_notice: Reporting Generic
> Notice type:3 num:65 from LID:0x0092
> GID:0xfe80000000000000,0x0005ad0000024bbb
> Mar 20 14:26:48 979827 [41401960] -> Removed port with
> GUID:0x0005ad0000024d6b LID range [0xB8,0xB8] of node:saguaro-23-9 HCA-1
> Mar 20 14:26:48 979845 [41401960] -> osm_report_notice: Reporting Generic
> Notice type:3 num:65 from LID:0x0092
> GID:0xfe80000000000000,0x0005ad0000024bbb
> Mar 20 14:26:48 979849 [41401960] -> Removed port with
> GUID:0x0005ad0000024977 LID range [0xA9,0xA9] of node:saguaro-22-4 HCA-1
> Mar 20 14:26:48 980028 [41401960] -> osm_report_notice: Reporting Generic
> Notice type:3 num:65 from LID:0x0092
> GID:0xfe80000000000000,0x0005ad0000024bbb
> Mar 20 14:26:48 980033 [41401960] -> Removed port with
> GUID:0x0005ad0000027c84 LID range [0x152,0x152] of node:Topspin Switch TS120
> Mar 20 14:26:48 980061 [41401960] -> osm_report_notice: Reporting Generic
> Notice type:3 num:65 from LID:0x0092
> GID:0xfe80000000000000,0x0005ad0000024bbb
> Mar 20 14:26:48 980066 [41401960] -> Removed port with
> GUID:0x0005ad0000024d8b LID range [0xB7,0xB7] of node:saguaro-23-8 HCA-1
> Mar 20 14:26:48 980081 [41401960] -> osm_report_notice: Reporting Generic
> Notice type:3 num:65 from LID:0x0092
> GID:0xfe80000000000000,0x0005ad0000024bbb
> Mar 20 14:26:48 980087 [41401960] -> Removed port with
> GUID:0x0005ad000002498f LID range [0xA8,0xA8] of node:saguaro-22-3 HCA-1
> Mar 20 14:26:48 980102 [41401960] -> osm_report_notice: Reporting Generic
> Notice type:3 num:65 from LID:0x0092
> GID:0xfe80000000000000,0x0005ad0000024bbb
> Mar 20 14:26:48 980107 [41401960] -> Removed port with
> GUID:0x0005ad0000024c9b LID range [0xA7,0xA7] of node:saguaro-22-2 HCA-1
> Mar 20 14:26:48 980122 [41401960] -> osm_report_notice: Reporting Generic
> Notice type:3 num:65 from LID:0x0092
> GID:0xfe80000000000000,0x0005ad0000024bbb
> Mar 20 14:26:48 980127 [41401960] -> Removed port with
> GUID:0x0005ad0000024da7 LID range [0xB0,0xB0] of node:saguaro-23-1 HCA-1
> Mar 20 14:26:48 980143 [41401960] -> osm_report_notice: Reporting Generic
> Notice type:3 num:65 from LID:0x0092
> GID:0xfe80000000000000,0x0005ad0000024bbb
> Mar 20 14:26:48 980148 [41401960] -> Removed port with
> GUID:0x0005ad0000024cbb LID range [0xB2,0xB2] of node:saguaro-23-3 HCA-1
> Mar 20 14:26:48 980163 [41401960] -> osm_report_notice: Reporting Generic
> Notice type:3 num:65 from LID:0x0092
> GID:0xfe80000000000000,0x0005ad0000024bbb
> Mar 20 14:26:48 980239 [41401960] -> Removed port with
> GUID:0x0005ad00000249d3 LID range [0xB1,0xB1] of node:saguaro-23-2 HCA-1
> Mar 20 14:26:48 980256 [41401960] -> osm_report_notice: Reporting Generic
> Notice type:3 num:65 from LID:0x0092
> GID:0xfe80000000000000,0x0005ad0000024bbb
> Mar 20 14:26:48 980261 [41401960] -> Removed port with
> GUID:0x0005ad0000024feb LID range [0x153,0x153] of node:saguaro-22-5 HCA-1
> Mar 20 14:26:48 980365 [41401960] -> osm_report_notice: Reporting Generic
> Notice type:3 num:65 from LID:0x0092
> GID:0xfe80000000000000,0x0005ad0000024bbb
> Mar 20 14:26:48 980371 [41401960] -> Removed port with
> GUID:0x0005ad0000024afb LID range [0xA5,0xA5] of node:saguaro-22-0 HCA-1
> Mar 20 14:26:49 013365 [41401960] -> osm_ucast_mgr_process: Min Hop Tables
> configured on all switches
> Mar 20 14:26:49 065887 [43C05960] -> SUBNET UP
> Mar 20 14:26:49 407010 [44606960] -> osm_ucast_mgr_process: Min Hop Tables
> configured on all switches
> Mar 20 14:26:49 459477 [44606960] -> SUBNET UP
> Mar 20 14:27:42 754098 [45007960] -> __osm_trap_rcv_process_request:
> Received Generic Notice type:0x01 num:128 Producer:2 from LID:0x0148
> TID:0x0000000000000070
> Mar 20 14:27:42 754349 [45007960] -> osm_report_notice: Reporting Generic
> Notice type:1 num:128 from LID:0x0148
> GID:0xfe80000000000000,0x0005ad00000281b3
> Mar 20 14:27:42 760115 [43C05960] -> __osm_trap_rcv_process_request:
> Received Generic Notice type:0x01 num:128 Producer:2 from LID:0x0148
> TID:0x0000000000000071
> Mar 20 14:27:42 760178 [44606960] -> __osm_trap_rcv_process_request:
> Received Generic Notice type:0x01 num:128 Producer:2 from LID:0x001B
> TID:0x0000000000000082
> Mar 20 14:27:42 760236 [43C05960] -> osm_report_notice: Reporting Generic
> Notice type:1 num:128 from LID:0x0148
> GID:0xfe80000000000000,0x0005ad00000281b3
> Mar 20 14:27:42 760406 [44606960] -> osm_report_notice: Reporting Generic
> Notice type:1 num:128 from LID:0x001B
> GID:0xfe80000000000000,0x0005ad00000281a7
> Mar 20 14:27:42 766931 [41E02960] -> __osm_trap_rcv_process_request:
> Received Generic Notice type:0x01 num:128 Producer:2 from LID:0x001B
> TID:0x0000000000000083
> Mar 20 14:27:42 767049 [41E02960] -> osm_report_notice: Reporting Generic
> Notice type:1 num:128 from LID:0x001B
> GID:0xfe80000000000000,0x0005ad00000281a7
> Mar 20 14:27:43 085327 [43C05960] -> osm_report_notice: Reporting Generic
> Notice type:3 num:64 from LID:0x0092
> GID:0xfe80000000000000,0x0005ad0000024bbb
> Mar 20 14:27:43 085345 [43C05960] -> Discovered new port with
> GUID:0x0005ad0000027c84 LID range [0x152,0x152] of node:Topspin Switch TS120
> Mar 20 14:27:43 085349 [43C05960] -> osm_report_notice: Reporting Generic
> Notice type:3 num:64 from LID:0x0092
> GID:0xfe80000000000000,0x0005ad0000024bbb
> Mar 20 14:27:43 085355 [43C05960] -> Discovered new port with
> GUID:0x0005ad00000249d3 LID range [0xB1,0xB1] of node:saguaro-23-2 HCA-1
> Mar 20 14:27:43 085359 [43C05960] -> osm_report_notice: Reporting Generic
> Notice type:3 num:64 from LID:0x0092
> GID:0xfe80000000000000,0x0005ad0000024bbb
> Mar 20 14:27:43 085364 [43C05960] -> Discovered new port with
> GUID:0x0005ad0000025043 LID range [0xB4,0xB4] of node:saguaro-23-5 HCA-1
> Mar 20 14:27:43 085368 [43C05960] -> osm_report_notice: Reporting Generic
> Notice type:3 num:64 from LID:0x0092
> GID:0xfe80000000000000,0x0005ad0000024bbb
> Mar 20 14:27:43 085373 [43C05960] -> Discovered new port with
> GUID:0x0005ad000002510b LID range [0xB5,0xB5] of node:saguaro-23-6 HCA-1
> Mar 20 14:27:43 085377 [43C05960] -> osm_report_notice: Reporting Generic
> Notice type:3 num:64 from LID:0x0092
> GID:0xfe80000000000000,0x0005ad0000024bbb
> Mar 20 14:27:43 085382 [43C05960] -> Discovered new port with
> GUID:0x0005ad0000024b27 LID range [0xAF,0xAF] of node:saguaro-23-0 HCA-1
> Mar 20 14:27:43 085386 [43C05960] -> osm_report_notice: Reporting Generic
> Notice type:3 num:64 from LID:0x0092
> GID:0xfe80000000000000,0x0005ad0000024bbb
> Mar 20 14:27:43 085390 [43C05960] -> Discovered new port with
> GUID:0x0005ad0000024da7 LID range [0xB0,0xB0] of node:saguaro-23-1 HCA-1
> Mar 20 14:27:43 085394 [43C05960] -> osm_report_notice: Reporting Generic
> Notice type:3 num:64 from LID:0x0092
> GID:0xfe80000000000000,0x0005ad0000024bbb
> Mar 20 14:27:43 085399 [43C05960] -> Discovered new port with
> GUID:0x0005ad0000024cbb LID range [0xB2,0xB2] of node:saguaro-23-3 HCA-1
> Mar 20 14:27:43 085403 [43C05960] -> osm_report_notice: Reporting Generic
> Notice type:3 num:64 from LID:0x0092
> GID:0xfe80000000000000,0x0005ad0000024bbb
> Mar 20 14:27:43 085407 [43C05960] -> Discovered new port with
> GUID:0x0005ad0000024afb LID range [0xA5,0xA5] of node:saguaro-22-0 HCA-1
> Mar 20 14:27:43 085411 [43C05960] -> osm_report_notice: Reporting Generic
> Notice type:3 num:64 from LID:0x0092
> GID:0xfe80000000000000,0x0005ad0000024bbb
> Mar 20 14:27:43 085416 [43C05960] -> Discovered new port with
> GUID:0x0005ad0000024e0b LID range [0xB3,0xB3] of node:saguaro-23-4 HCA-1
> Mar 20 14:27:43 085420 [43C05960] -> osm_report_notice: Reporting Generic
> Notice type:3 num:64 from LID:0x0092
> GID:0xfe80000000000000,0x0005ad0000024bbb
> Mar 20 14:27:43 085425 [43C05960] -> Discovered new port with
> GUID:0x0005ad0000024d47 LID range [0xB6,0xB6] of node:saguaro-23-7 HCA-1
> Mar 20 14:27:43 085428 [43C05960] -> osm_report_notice: Reporting Generic
> Notice type:3 num:64 from LID:0x0092
> GID:0xfe80000000000000,0x0005ad0000024bbb
> Mar 20 14:27:43 085433 [43C05960] -> Discovered new port with
> GUID:0x0005ad0000024d8b LID range [0xB7,0xB7] of node:saguaro-23-8 HCA-1
> Mar 20 14:27:43 085437 [43C05960] -> osm_report_notice: Reporting Generic
> Notice type:3 num:64 from LID:0x0092
> GID:0xfe80000000000000,0x0005ad0000024bbb
> Mar 20 14:27:43 085442 [43C05960] -> Discovered new port with
> GUID:0x0005ad0000024d6b LID range [0xB8,0xB8] of node:saguaro-23-9 HCA-1
> Mar 20 14:27:43 085446 [43C05960] -> osm_report_notice: Reporting Generic
> Notice type:3 num:64 from LID:0x0092
> GID:0xfe80000000000000,0x0005ad0000024bbb
> Mar 20 14:27:43 085450 [43C05960] -> Discovered new port with
> GUID:0x0005ad000002511b LID range [0xA6,0xA6] of node:saguaro-22-1 HCA-1
> Mar 20 14:27:43 085454 [43C05960] -> osm_report_notice: Reporting Generic
> Notice type:3 num:64 from LID:0x0092
> GID:0xfe80000000000000,0x0005ad0000024bbb
> Mar 20 14:27:43 085459 [43C05960] -> Discovered new port with
> GUID:0x0005ad0000024c9b LID range [0xA7,0xA7] of node:saguaro-22-2 HCA-1
> Mar 20 14:27:43 085511 [43C05960] -> osm_report_notice: Reporting Generic
> Notice type:3 num:64 from LID:0x0092
> GID:0xfe80000000000000,0x0005ad0000024bbb
> Mar 20 14:27:43 085517 [43C05960] -> Discovered new port with
> GUID:0x0005ad000002498f LID range [0xA8,0xA8] of node:saguaro-22-3 HCA-1
> Mar 20 14:27:43 085521 [43C05960] -> osm_report_notice: Reporting Generic
> Notice type:3 num:64 from LID:0x0092
> GID:0xfe80000000000000,0x0005ad0000024bbb
> Mar 20 14:27:43 085526 [43C05960] -> Discovered new port with
> GUID:0x0005ad0000024feb LID range [0x153,0x153] of node:saguaro-22-5 HCA-1
> Mar 20 14:27:43 085530 [43C05960] -> osm_report_notice: Reporting Generic
> Notice type:3 num:64 from LID:0x0092
> GID:0xfe80000000000000,0x0005ad0000024bbb
> Mar 20 14:27:43 085534 [43C05960] -> Discovered new port with
> GUID:0x0005ad0000024977 LID range [0xA9,0xA9] of node:saguaro-22-4 HCA-1
> Mar 20 14:27:43 116308 [43C05960] -> osm_ucast_mgr_process: Min Hop Tables
> configured on all switches
> Mar 20 14:27:43 179935 [4780B960] -> __osm_sm_mad_ctrl_rcv_callback: ERR
> 3111: Error status = 0x1C00
> Mar 20 14:27:43 179980 [4780B960] -> SMP dump:
> base_ver................0x1
> mgmt_class..............0x81
> class_ver...............0x1
> method..................0x81 (SubnGetResp)
> D bit...................0x1
> status..................0x1C00
> hop_ptr.................0x0
> hop_count...............0x4
> trans_id................0x85669
> attr_id.................0x15 (PortInfo)
> resv....................0x0
> attr_mod................0x13
> m_key...................0x0000000000000000
> dr_slid.................0xFFFF
> dr_dlid.................0xFFFF
>
> Initial path: [0][1][16][1][4]
> Return path: [0][9][13][D][1]
> Reserved: [0][0][0][0][0][0][0]
>
> 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
>
> 00 00 00 00 00 00 00 00 00 00 00 00 01 03 03 02
>
> 14 52 00 11 40 40 00 08 08 04 F2 40 00 00 00 00
>
> 00 00 00 00 00 88 00 00 00 00 00 00 00 00 00 00
>
> Mar 20 14:27:43 180019 [41401960] -> osm_pi_rcv_process_set: Received error
> status 0x1c for SetResp() during ACTIVE transition
> Mar 20 14:27:43 180037 [41401960] -> PortInfo dump:
> port number.............0x13
> node_guid...............0x0005ad00000281a7
> port_guid...............0x0005ad00000281a7
> m_key...................0x0000000000000000
> subnet_prefix...........0x0000000000000000
> base_lid................0x0
> master_sm_base_lid......0x0
> capability_mask.........0x0
> diag_code...............0x0
> m_key_lease_period......0x0
> local_port_num..........0x1
> link_width_enabled......0x3
> link_width_supported....0x3
> link_width_active.......0x2
> link_speed_supported....0x1
> port_state..............ACTIVE
> state_info2.............0x52
> m_key_protect_bits......0x0
> lmc.....................0x0
> link_speed..............0x11
> mtu_smsl................0x40
> vl_cap_init_type........0x40
> vl_high_limit...........0x0
> vl_arb_high_cap.........0x8
> vl_arb_low_cap..........0x8
> init_rep_mtu_cap........0x4
> vl_stall_life...........0xF2
> vl_enforce..............0x40
> m_key_violations........0x0
> p_key_violations........0x0
> q_key_violations........0x0
> guid_cap................0x0
> client_reregister.......0x0
> subnet_timeout..........0x0
> resp_time_value.........0x0
> error_threshold.........0x88
> Mar 20 14:27:43 180050 [41401960] -> Capabilities Mask:
> Mar 20 14:27:43 180092 [4780B960] -> __osm_sm_mad_ctrl_rcv_callback: ERR
> 3111: Error status = 0x1C00
> Mar 20 14:27:43 180137 [4780B960] -> SMP dump:
> base_ver................0x1
> mgmt_class..............0x81
> class_ver...............0x1
> method..................0x81 (SubnGetResp)
> D bit...................0x1
> status..................0x1C00
> hop_ptr.................0x0
> hop_count...............0x4
> trans_id................0x8566a
> attr_id.................0x15 (PortInfo)
> resv....................0x0
> attr_mod................0x16
> m_key...................0x0000000000000000
> dr_slid.................0xFFFF
> dr_dlid.................0xFFFF
>
> Initial path: [0][1][16][1][4]
> Return path: [0][9][13][D][1]
> Reserved: [0][0][0][0][0][0][0]
>
> 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
>
> 00 00 00 00 00 00 00 00 00 00 00 00 01 03 03 02
>
> 14 52 00 11 40 40 00 08 08 04 F2 40 00 00 00 00
>
> 00 00 00 00 00 88 00 00 00 00 00 00 00 00 00 00
>
> Mar 20 14:27:43 180185 [44606960] -> osm_pi_rcv_process_set: Received error
> status 0x1c for SetResp() during ACTIVE transition
> Mar 20 14:27:43 180189 [44606960] -> PortInfo dump:
> port number.............0x16
> node_guid...............0x0005ad00000281a7
> port_guid...............0x0005ad00000281a7
> m_key...................0x0000000000000000
> subnet_prefix...........0x0000000000000000
> base_lid................0x0
> master_sm_base_lid......0x0
> capability_mask.........0x0
> diag_code...............0x0
> m_key_lease_period......0x0
> local_port_num..........0x1
> link_width_enabled......0x3
> link_width_supported....0x3
> link_width_active.......0x2
> link_speed_supported....0x1
> port_state..............ACTIVE
> state_info2.............0x52
> m_key_protect_bits......0x0
> lmc.....................0x0
> link_speed..............0x11
> mtu_smsl................0x40
> vl_cap_init_type........0x40
> vl_high_limit...........0x0
> vl_arb_high_cap.........0x8
> vl_arb_low_cap..........0x8
> init_rep_mtu_cap........0x4
> vl_stall_life...........0xF2
> vl_enforce..............0x40
> m_key_violations........0x0
> p_key_violations........0x0
> q_key_violations........0x0
> guid_cap................0x0
> client_reregister.......0x0
> subnet_timeout..........0x0
> resp_time_value.........0x0
> error_threshold.........0x88
> Mar 20 14:27:43 180199 [44606960] -> Capabilities Mask:
> Mar 20 14:27:43 180239 [4780B960] -> __osm_sm_mad_ctrl_rcv_callback: ERR
> 3111: Error status = 0x1C00
> Mar 20 14:27:43 180263 [4780B960] -> SMP dump:
> base_ver................0x1
> mgmt_class..............0x81
> class_ver...............0x1
> method..................0x81 (SubnGetResp)
> D bit...................0x1
> status..................0x1C00
> hop_ptr.................0x0
> hop_count...............0x4
> trans_id................0x8566b
> attr_id.................0x15 (PortInfo)
> resv....................0x0
> attr_mod................0x17
> m_key...................0x0000000000000000
> dr_slid.................0xFFFF
> dr_dlid.................0xFFFF
>
> Initial path: [0][1][16][1][4]
> Return path: [0][9][13][D][1]
> Reserved: [0][0][0][0][0][0][0]
>
> 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
>
> 00 00 00 00 00 00 00 00 00 00 00 00 01 03 03 02
>
> 14 52 00 11 40 40 00 08 08 04 F2 40 00 00 00 00
>
> 00 00 00 00 00 88 00 00 00 00 00 00 00 00 00 00
>
> Mar 20 14:27:43 180307 [4780B960] -> __osm_sm_mad_ctrl_rcv_callback: ERR
> 3111: Error status = 0x1C00
> Mar 20 14:27:43 180319 [42803960] -> osm_pi_rcv_process_set: Received error
> status 0x1c for SetResp() during ACTIVE transition
> Mar 20 14:27:43 180332 [4780B960] -> SMP dump:
> base_ver................0x1
> mgmt_class..............0x81
> class_ver...............0x1
> method..................0x81 (SubnGetResp)
> D bit...................0x1
> status..................0x1C00
> hop_ptr.................0x0
> hop_count...............0x4
> trans_id................0x8566c
> attr_id.................0x15 (PortInfo)
> resv....................0x0
> attr_mod................0x18
> m_key...................0x0000000000000000
> dr_slid.................0xFFFF
> dr_dlid.................0xFFFF
>
> Initial path: [0][1][16][1][4]
> Return path: [0][9][13][D][1]
> Reserved: [0][0][0][0][0][0][0]
>
> 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
>
> 00 00 00 00 00 00 00 00 00 00 00 00 01 03 03 02
>
> 14 52 00 11 40 40 00 08 08 04 F2 40 00 00 00 00
>
> 00 00 00 00 00 88 00 00 00 00 00 00 00 00 00 00
>
> Mar 20 14:27:43 180336 [42803960] -> PortInfo dump:
> port number.............0x17
> node_guid...............0x0005ad00000281a7
> port_guid...............0x0005ad00000281a7
> m_key...................0x0000000000000000
> subnet_prefix...........0x0000000000000000
> base_lid................0x0
> master_sm_base_lid......0x0
> capability_mask.........0x0
> diag_code...............0x0
> m_key_lease_period......0x0
> local_port_num..........0x1
> link_width_enabled......0x3
> link_width_supported....0x3
> link_width_active.......0x2
> link_speed_supported....0x1
> port_state..............ACTIVE
> state_info2.............0x52
> m_key_protect_bits......0x0
> lmc.....................0x0
> link_speed..............0x11
> mtu_smsl................0x40
> vl_cap_init_type........0x40
> vl_high_limit...........0x0
> vl_arb_high_cap.........0x8
> vl_arb_low_cap..........0x8
> init_rep_mtu_cap........0x4
> vl_stall_life...........0xF2
> vl_enforce..............0x40
> m_key_violations........0x0
> p_key_violations........0x0
> q_key_violations........0x0
> guid_cap................0x0
> client_reregister.......0x0
> subnet_timeout..........0x0
> resp_time_value.........0x0
> error_threshold.........0x88
> Mar 20 14:27:43 180364 [42803960] -> Capabilities Mask:
> Mar 20 14:27:43 180389 [42803960] -> osm_pi_rcv_process_set: Received error
> status 0x1c for SetResp() during ACTIVE transition
> Mar 20 14:27:43 180410 [4780B960] -> __osm_sm_mad_ctrl_rcv_callback: ERR
> 3111: Error status = 0x1C00
> Mar 20 14:27:43 180392 [42803960] -> PortInfo dump:
> port number.............0x18
> node_guid...............0x0005ad00000281a7
> port_guid...............0x0005ad00000281a7
> m_key...................0x0000000000000000
> subnet_prefix...........0x0000000000000000
> base_lid................0x0
> master_sm_base_lid......0x0
> capability_mask.........0x0
> diag_code...............0x0
> m_key_lease_period......0x0
> local_port_num..........0x1
> link_width_enabled......0x3
> link_width_supported....0x3
> link_width_active.......0x2
> link_speed_supported....0x1
> port_state..............ACTIVE
> state_info2.............0x52
> m_key_protect_bits......0x0
> lmc.....................0x0
> link_speed..............0x11
> mtu_smsl................0x40
> vl_cap_init_type........0x40
> vl_high_limit...........0x0
> vl_arb_high_cap.........0x8
> vl_arb_low_cap..........0x8
> init_rep_mtu_cap........0x4
> vl_stall_life...........0xF2
> vl_enforce..............0x40
> m_key_violations........0x0
> p_key_violations........0x0
> q_key_violations........0x0
> guid_cap................0x0
> client_reregister.......0x0
> subnet_timeout..........0x0
> resp_time_value.........0x0
> error_threshold.........0x88
> Mar 20 14:27:43 180415 [42803960] -> Capabilities Mask:
> Mar 20 14:27:43 180436 [4780B960] -> SMP dump:
> base_ver................0x1
> mgmt_class..............0x81
> class_ver...............0x1
> method..................0x81 (SubnGetResp)
> D bit...................0x1
> status..................0x1C00
> hop_ptr.................0x0
> hop_count...............0x4
> trans_id................0x8566d
> attr_id.................0x15 (PortInfo)
> resv....................0x0
> attr_mod................0x16
> m_key...................0x0000000000000000
> dr_slid.................0xFFFF
> dr_dlid.................0xFFFF
>
> Initial path: [0][1][11][2][5]
> Return path: [0][9][18][E][2]
> Reserved: [0][0][0][0][0][0][0]
>
> 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
>
> 00 00 00 00 00 00 00 00 00 00 00 00 02 03 03 02
>
> 14 52 00 11 40 40 00 08 08 04 F2 40 00 00 00 00
>
> 00 00 00 00 00 88 00 00 00 00 00 00 00 00 00 00
>
> Mar 20 14:27:43 180490 [41E02960] -> osm_pi_rcv_process_set: Received error
> status 0x1c for SetResp() during ACTIVE transition
> Mar 20 14:27:43 180494 [41E02960] -> PortInfo dump:
> port number.............0x16
> node_guid...............0x0005ad00000281b3
> port_guid...............0x0005ad00000281b3
> m_key...................0x0000000000000000
> subnet_prefix...........0x0000000000000000
> base_lid................0x0
> master_sm_base_lid......0x0
> capability_mask.........0x0
> diag_code...............0x0
> m_key_lease_period......0x0
> local_port_num..........0x2
> link_width_enabled......0x3
> link_width_supported....0x3
> link_width_active.......0x2
> link_speed_supported....0x1
> port_state..............ACTIVE
> state_info2.............0x52
> m_key_protect_bits......0x0
> lmc.....................0x0
> link_speed..............0x11
> mtu_smsl................0x40
> vl_cap_init_type........0x40
> vl_high_limit...........0x0
> vl_arb_high_cap.........0x8
> vl_arb_low_cap..........0x8
> init_rep_mtu_cap........0x4
> vl_stall_life...........0xF2
> vl_enforce..............0x40
> m_key_violations........0x0
> p_key_violations........0x0
> q_key_violations........0x0
> guid_cap................0x0
> client_reregister.......0x0
> subnet_timeout..........0x0
> resp_time_value.........0x0
> error_threshold.........0x88
> Mar 20 14:27:43 180504 [41E02960] -> Capabilities Mask:
> Mar 20 14:27:43 180536 [4780B960] -> __osm_sm_mad_ctrl_rcv_callback: ERR
> 3111: Error status = 0x1C00
> Mar 20 14:27:43 180560 [4780B960] -> SMP dump:
> base_ver................0x1
> mgmt_class..............0x81
> class_ver...............0x1
> method..................0x81 (SubnGetResp)
> D bit...................0x1
> status..................0x1C00
> hop_ptr.................0x0
> hop_count...............0x4
> trans_id................0x8566e
> attr_id.................0x15 (PortInfo)
> resv....................0x0
> attr_mod................0x17
> m_key...................0x0000000000000000
> dr_slid.................0xFFFF
> dr_dlid.................0xFFFF
>
> Initial path: [0][1][11][2][5]
> Return path: [0][9][18][E][2]
> Reserved: [0][0][0][0][0][0][0]
>
> 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
>
> 00 00 00 00 00 00 00 00 00 00 00 00 02 03 03 02
>
> 14 52 00 11 40 40 00 08 08 04 F2 40 00 00 00 00
>
> 00 00 00 00 00 88 00 00 00 00 00 00 00 00 00 00
>
> Mar 20 14:27:43 180606 [45007960] -> osm_pi_rcv_process_set: Received error
> status 0x1c for SetResp() during ACTIVE transition
> Mar 20 14:27:43 180615 [4780B960] -> __osm_sm_mad_ctrl_rcv_callback: ERR
> 3111: Error status = 0x1C00
> Mar 20 14:27:43 180634 [45007960] -> PortInfo dump:
> port number.............0x17
> node_guid...............0x0005ad00000281b3
> port_guid...............0x0005ad00000281b3
> m_key...................0x0000000000000000
> subnet_prefix...........0x0000000000000000
> base_lid................0x0
> master_sm_base_lid......0x0
> capability_mask.........0x0
> diag_code...............0x0
> m_key_lease_period......0x0
> local_port_num..........0x2
> link_width_enabled......0x3
> link_width_supported....0x3
> link_width_active.......0x2
> link_speed_supported....0x1
> port_state..............ACTIVE
> state_info2.............0x52
> m_key_protect_bits......0x0
> lmc.....................0x0
> link_speed..............0x11
> mtu_smsl................0x40
> vl_cap_init_type........0x40
> vl_high_limit...........0x0
> vl_arb_high_cap.........0x8
> vl_arb_low_cap..........0x8
> init_rep_mtu_cap........0x4
> vl_stall_life...........0xF2
> vl_enforce..............0x40
> m_key_violations........0x0
> p_key_violations........0x0
> q_key_violations........0x0
> guid_cap................0x0
> client_reregister.......0x0
> subnet_timeout..........0x0
> resp_time_value.........0x0
> error_threshold.........0x88
> Mar 20 14:27:43 180657 [45007960] -> Capabilities Mask:
> Mar 20 14:27:43 180678 [4780B960] -> SMP dump:
> base_ver................0x1
> mgmt_class..............0x81
> class_ver...............0x1
> method..................0x81 (SubnGetResp)
> D bit...................0x1
> status..................0x1C00
> hop_ptr.................0x0
> hop_count...............0x4
> trans_id................0x8566f
> attr_id.................0x15 (PortInfo)
> resv....................0x0
> attr_mod................0x18
> m_key...................0x0000000000000000
> dr_slid.................0xFFFF
> dr_dlid.................0xFFFF
>
> Initial path: [0][1][11][2][5]
> Return path: [0][9][18][E][2]
> Reserved: [0][0][0][0][0][0][0]
>
> 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
>
> 00 00 00 00 00 00 00 00 00 00 00 00 02 03 03 02
>
> 14 52 00 11 40 40 00 08 08 04 F2 40 00 00 00 00
>
> 00 00 00 00 00 88 00 00 00 00 00 00 00 00 00 00
>
> Mar 20 14:27:43 180769 [43C05960] -> osm_pi_rcv_process_set: Received error
> status 0x1c for SetResp() during ACTIVE transition
> Mar 20 14:27:43 180775 [43C05960] -> PortInfo dump:
> port number.............0x18
> node_guid...............0x0005ad00000281b3
> port_guid...............0x0005ad00000281b3
> m_key...................0x0000000000000000
> subnet_prefix...........0x0000000000000000
> base_lid................0x0
> master_sm_base_lid......0x0
> capability_mask.........0x0
> diag_code...............0x0
> m_key_lease_period......0x0
> local_port_num..........0x2
> link_width_enabled......0x3
> link_width_supported....0x3
> link_width_active.......0x2
> link_speed_supported....0x1
> port_state..............ACTIVE
> state_info2.............0x52
> m_key_protect_bits......0x0
> lmc.....................0x0
> link_speed..............0x11
> mtu_smsl................0x40
> vl_cap_init_type........0x40
> vl_high_limit...........0x0
> vl_arb_high_cap.........0x8
> vl_arb_low_cap..........0x8
> init_rep_mtu_cap........0x4
> vl_stall_life...........0xF2
> vl_enforce..............0x40
> m_key_violations........0x0
> p_key_violations........0x0
> q_key_violations........0x0
> guid_cap................0x0
> client_reregister.......0x0
> subnet_timeout..........0x0
> resp_time_value.........0x0
> error_threshold.........0x88
> Mar 20 14:27:43 180789 [43C05960] -> Capabilities Mask:
> Mar 20 14:27:43 186228 [43204960] -> SUBNET UP
> Mar 20 14:27:43 557268 [45A08960] -> osm_ucast_mgr_process: Min Hop Tables
> configured on all switches
> Mar 20 14:27:43 611082 [45A08960] -> SUBNET UP
> Mar 20 14:27:58 852744 [45007960] -> __osm_trap_rcv_process_request:
> Received Generic Notice type:0x01 num:128 Producer:2 from LID:0x0152
> TID:0x0000000000000000
> Mar 20 14:27:58 852982 [45007960] -> osm_report_notice: Reporting Generic
> Notice type:1 num:128 from LID:0x0152
> GID:0xfe80000000000000,0x0005ad0000027c84
> Mar 20 14:27:58 970772 [43204960] -> __osm_trap_rcv_process_request:
> Received Generic Notice type:0x01 num:128 Producer:2 from LID:0x0152
> TID:0x0000000000000001
> Mar 20 14:27:58 970864 [43204960] -> osm_report_notice: Reporting Generic
> Notice type:1 num:128 from LID:0x0152
> GID:0xfe80000000000000,0x0005ad0000027c84
> Mar 20 14:27:58 992628 [41E02960] -> __osm_trap_rcv_process_request:
> Received Generic Notice type:0x01 num:128 Producer:2 from LID:0x0152
> TID:0x0000000000000002
> Mar 20 14:27:58 992712 [41E02960] -> osm_report_notice: Reporting Generic
> Notice type:1 num:128 from LID:0x0152
> GID:0xfe80000000000000,0x0005ad0000027c84
> Mar 20 14:27:59 132331 [42803960] -> __osm_trap_rcv_process_request:
> Received Generic Notice type:0x01 num:128 Producer:2 from LID:0x0152
> TID:0x0000000000000003
> Mar 20 14:27:59 132484 [42803960] -> osm_report_notice: Reporting Generic
> Notice type:1 num:128 from LID:0x0152
> GID:0xfe80000000000000,0x0005ad0000027c84
> Mar 20 14:27:59 314893 [41E02960] -> __osm_trap_rcv_process_request:
> Received Generic Notice type:0x01 num:128 Producer:2 from LID:0x0152
> TID:0x0000000000000004
> Mar 20 14:27:59 315006 [41E02960] -> osm_report_notice: Reporting Generic
> Notice type:1 num:128 from LID:0x0152
> GID:0xfe80000000000000,0x0005ad0000027c84
> Mar 20 14:27:59 343241 [42803960] -> __osm_trap_rcv_process_request:
> Received Generic Notice type:0x01 num:128 Producer:2 from LID:0x0152
> TID:0x0000000000000005
> Mar 20 14:27:59 343320 [42803960] -> osm_report_notice: Reporting Generic
> Notice type:1 num:128 from LID:0x0152
> GID:0xfe80000000000000,0x0005ad0000027c84
> Mar 20 14:27:59 481698 [45007960] -> __osm_trap_rcv_process_request:
> Received Generic Notice type:0x01 num:128 Producer:2 from LID:0x0152
> TID:0x0000000000000006
> Mar 20 14:27:59 481775 [45007960] -> osm_report_notice: Reporting Generic
> Notice type:1 num:128 from LID:0x0152
> GID:0xfe80000000000000,0x0005ad0000027c84
> Mar 20 14:27:59 512746 [45A08960] -> __osm_trap_rcv_process_request:
> Received Generic Notice type:0x01 num:128 Producer:2 from LID:0x0152
> TID:0x0000000000000007
> Mar 20 14:27:59 512853 [45A08960] -> osm_report_notice: Reporting Generic
> Notice type:1 num:128 from LID:0x0152
> GID:0xfe80000000000000,0x0005ad0000027c84
> Mar 20 14:27:59 548851 [41E02960] -> osm_report_notice: Reporting Generic
> Notice type:3 num:65 from LID:0x0092
> GID:0xfe80000000000000,0x0005ad0000024bbb
> Mar 20 14:27:59 548861 [41E02960] -> Removed port with
> GUID:0x0005ad00000249d3 LID range [0xB1,0xB1] of node:saguaro-23-2 HCA-1
> Mar 20 14:27:59 583414 [41E02960] -> osm_ucast_mgr_process: Min Hop Tables
> configured on all switches
> Mar 20 14:27:59 583817 [43C05960] -> __osm_trap_rcv_process_request:
> Received Generic Notice type:0x01 num:128 Producer:2 from LID:0x0152
> TID:0x0000000000000008
> Mar 20 14:27:59 623971 [43C05960] -> osm_report_notice: Reporting Generic
> Notice type:1 num:128 from LID:0x0152
> GID:0xfe80000000000000,0x0005ad0000027c84
> Mar 20 14:27:59 626182 [42803960] -> __osm_trap_rcv_process_request:
> Received Generic Notice type:0x01 num:128 Producer:2 from LID:0x0152
> TID:0x0000000000000009
> Mar 20 14:27:59 626329 [42803960] -> osm_report_notice: Reporting Generic
> Notice type:1 num:128 from LID:0x0152
> GID:0xfe80000000000000,0x0005ad0000027c84
> Mar 20 14:27:59 634080 [41E02960] -> __osm_trap_rcv_process_request:
> Received Generic Notice type:0x01 num:128 Producer:2 from LID:0x0152
> TID:0x000000000000000a
> Mar 20 14:27:59 634442 [41E02960] -> osm_report_notice: Reporting Generic
> Notice type:1 num:128 from LID:0x0152
> GID:0xfe80000000000000,0x0005ad0000027c84
> Mar 20 14:27:59 641962 [45A08960] -> SUBNET UP
> Mar 20 14:27:59 656231 [41401960] -> __osm_trap_rcv_process_request:
> Received Generic Notice type:0x01 num:128 Producer:2 from LID:0x0152
> TID:0x000000000000000b
> Mar 20 14:27:59 656307 [41401960] -> __osm_trap_rcv_process_request: ERR
> 3804: Received trap 11 times consecutively
> Mar 20 14:27:59 689788 [41E02960] -> __osm_trap_rcv_process_request:
> Received Generic Notice type:0x01 num:128 Producer:2 from LID:0x0152
> TID:0x000000000000000c
> Mar 20 14:27:59 690249 [41E02960] -> __osm_trap_rcv_process_request: ERR
> 3804: Received trap 12 times consecutively
> Mar 20 14:27:59 758521 [42803960] -> __osm_trap_rcv_process_request:
> Received Generic Notice type:0x01 num:128 Producer:2 from LID:0x0152
> TID:0x000000000000000d
> Mar 20 14:27:59 758646 [42803960] -> __osm_trap_rcv_process_request: ERR
> 3804: Received trap 13 times consecutively
> Mar 20 14:27:59 970740 [43204960] -> __osm_trap_rcv_process_request:
> Received Generic Notice type:0x01 num:128 Producer:2 from LID:0x0152
> TID:0x000000000000000e
> Mar 20 14:27:59 970812 [43204960] -> __osm_trap_rcv_process_request: ERR
> 3804: Received trap 14 times consecutively
> Mar 20 14:27:59 985557 [41E02960] -> osm_report_notice: Reporting Generic
> Notice type:3 num:65 from LID:0x0092
> GID:0xfe80000000000000,0x0005ad0000024bbb
> Mar 20 14:27:59 985577 [41E02960] -> Removed port with
> GUID:0x0005ad000002510b LID range [0xB5,0xB5] of node:saguaro-23-6 HCA-1
> Mar 20 14:27:59 985601 [41E02960] -> osm_report_notice: Reporting Generic
> Notice type:3 num:65 from LID:0x0092
> GID:0xfe80000000000000,0x0005ad0000024bbb
> Mar 20 14:27:59 985615 [41E02960] -> Removed port with
> GUID:0x0005ad0000024d47 LID range [0xB6,0xB6] of node:saguaro-23-7 HCA-1
> Mar 20 14:27:59 985649 [41E02960] -> osm_report_notice: Reporting Generic
> Notice type:3 num:65 from LID:0x0092
> GID:0xfe80000000000000,0x0005ad0000024bbb
> Mar 20 14:27:59 985656 [41E02960] -> Removed port with
> GUID:0x0005ad0000024d8b LID range [0xB7,0xB7] of node:saguaro-23-8 HCA-1
> Mar 20 14:27:59 989767 [42803960] -> osm_report_notice: Reporting Generic
> Notice type:3 num:64 from LID:0x0092
> GID:0xfe80000000000000,0x0005ad0000024bbb
> Mar 20 14:27:59 989787 [42803960] -> Discovered new port with
> GUID:0x0005ad00000249d3 LID range [0xB1,0xB1] of node:saguaro-23-2 HCA-1
> Mar 20 14:28:00 014445 [43C05960] -> __osm_trap_rcv_process_request:
> Received Generic Notice type:0x01 num:128 Producer:2 from LID:0x0152
> TID:0x000000000000000f
> Mar 20 14:28:00 014524 [43C05960] -> __osm_trap_rcv_process_request: ERR
> 3804: Received trap 15 times consecutively
> Mar 20 14:28:00 020896 [42803960] -> osm_ucast_mgr_process: Min Hop Tables
> configured on all switches
> Mar 20 14:28:00 086824 [43204960] -> SUBNET UP
> Mar 20 14:28:00 124057 [45007960] -> __osm_trap_rcv_process_request:
> Received Generic Notice type:0x01 num:128 Producer:2 from LID:0x0152
> TID:0x0000000000000010
> Mar 20 14:28:00 124108 [45007960] -> __osm_trap_rcv_process_request: ERR
> 3804: Received trap 16 times consecutively
> Mar 20 14:28:00 131596 [41401960] -> __osm_trap_rcv_process_request:
> Received Generic Notice type:0x01 num:128 Producer:2 from LID:0x0152
> TID:0x0000000000000011
> Mar 20 14:28:00 131643 [41401960] -> __osm_trap_rcv_process_request: ERR
> 3804: Received trap 17 times consecutively
> Mar 20 14:28:00 412484 [43C05960] -> __osm_trap_rcv_process_request:
> Received Generic Notice type:0x01 num:128 Producer:2 from LID:0x0152
> TID:0x0000000000000012
> Mar 20 14:28:00 412528 [43C05960] -> __osm_trap_rcv_process_request: ERR
> 3804: Received trap 18 times consecutively
> Mar 20 14:28:00 436877 [44606960] -> __osm_trap_rcv_process_request:
> Received Generic Notice type:0x01 num:128 Producer:2 from LID:0x0152
> TID:0x0000000000000013
> Mar 20 14:28:00 436921 [44606960] -> __osm_trap_rcv_process_request: ERR
> 3804: Received trap 19 times consecutively
> Mar 20 14:28:00 458745 [42803960] -> __osm_trap_rcv_process_request:
> Received Generic Notice type:0x01 num:128 Producer:2 from LID:0x0152
> TID:0x0000000000000014
> Mar 20 14:28:00 458816 [42803960] -> __osm_trap_rcv_process_request: ERR
> 3804: Received trap 20 times consecutively
> Mar 20 14:28:00 480551 [41E02960] -> __osm_trap_rcv_process_request:
> Received Generic Notice type:0x01 num:128 Producer:2 from LID:0x0152
> TID:0x0000000000000015
> Mar 20 14:28:00 480599 [41E02960] -> __osm_trap_rcv_process_request: ERR
> 3804: Received trap 21 times consecutively
> Mar 20 14:28:00 695340 [45A08960] -> __osm_trap_rcv_process_request:
> Received Generic Notice type:0x01 num:128 Producer:2 from LID:0x0152
> TID:0x0000000000000016
> Mar 20 14:28:00 695386 [45A08960] -> __osm_trap_rcv_process_request: ERR
> 3804: Received trap 22 times consecutively
> Mar 20 14:28:00 695726 [43204960] -> __osm_trap_rcv_process_request:
> Received Generic Notice type:0x01 num:128 Producer:2 from LID:0x0148
> TID:0x0000000000000072
> Mar 20 14:28:00 695886 [43204960] -> osm_report_notice: Reporting Generic
> Notice type:1 num:128 from LID:0x0148
> GID:0xfe80000000000000,0x0005ad00000281b3
> Mar 20 14:28:00 719764 [41401960] -> __osm_trap_rcv_process_request:
> Received Generic Notice type:0x01 num:128 Producer:2 from LID:0x0152
> TID:0x0000000000000017
> Mar 20 14:28:00 719825 [41401960] -> __osm_trap_rcv_process_request: ERR
> 3804: Received trap 23 times consecutively
> Mar 20 14:28:00 743680 [43204960] -> __osm_trap_rcv_process_request:
> Received Generic Notice type:0x01 num:128 Producer:2 from LID:0x0152
> TID:0x0000000000000018
> Mar 20 14:28:00 743775 [43204960] -> __osm_trap_rcv_process_request: ERR
> 3804: Received trap 24 times consecutively
> Mar 20 14:28:00 763599 [45007960] -> __osm_trap_rcv_process_request:
> Received Generic Notice type:0x01 num:128 Producer:2 from LID:0x0152
> TID:0x0000000000000019
> Mar 20 14:28:00 763654 [45007960] -> __osm_trap_rcv_process_request: ERR
> 3804: Received trap 25 times consecutively
> Mar 20 14:28:00 813393 [43C05960] -> __osm_trap_rcv_process_request:
> Received Generic Notice type:0x01 num:128 Producer:2 from LID:0x0152
> TID:0x000000000000001a
> Mar 20 14:28:00 813473 [43C05960] -> __osm_trap_rcv_process_request: ERR
> 3804: Received trap 26 times consecutively
> Mar 20 14:28:00 831287 [45A08960] -> __osm_trap_rcv_process_request:
> Received Generic Notice type:0x01 num:128 Producer:2 from LID:0x0152
> TID:0x000000000000001b
> Mar 20 14:28:00 831302 [44606960] -> __osm_trap_rcv_process_request:
> Received Generic Notice type:0x01 num:128 Producer:2 from LID:0x0148
> TID:0x0000000000000073
> Mar 20 14:28:00 831383 [45A08960] -> __osm_trap_rcv_process_request: ERR
> 3804: Received trap 27 times consecutively
> Mar 20 14:28:00 831424 [44606960] -> osm_report_notice: Reporting Generic
> Notice type:1 num:128 from LID:0x0148
> GID:0xfe80000000000000,0x0005ad00000281b3
> Mar 20 14:28:00 841593 [41E02960] -> __osm_trap_rcv_process_request:
> Received Generic Notice type:0x01 num:128 Producer:2 from LID:0x0152
> TID:0x000000000000001c
> Mar 20 14:28:00 841644 [41E02960] -> __osm_trap_rcv_process_request: ERR
> 3804: Received trap 28 times consecutively
> Mar 20 14:28:01 050511 [41E02960] -> osm_report_notice: Reporting Generic
> Notice type:3 num:64 from LID:0x0092
> GID:0xfe80000000000000,0x0005ad0000024bbb
> Mar 20 14:28:01 050529 [41E02960] -> Discovered new port with
> GUID:0x0005ad000002510b LID range [0xB5,0xB5] of node:saguaro-23-6 HCA-1
> Mar 20 14:28:01 050535 [41E02960] -> osm_report_notice: Reporting Generic
> Notice type:3 num:64 from LID:0x0092
> GID:0xfe80000000000000,0x0005ad0000024bbb
> Mar 20 14:28:01 050542 [41E02960] -> Discovered new port with
> GUID:0x0005ad0000024d47 LID range [0xB6,0xB6] of node:saguaro-23-7 HCA-1
> Mar 20 14:28:01 050547 [41E02960] -> osm_report_notice: Reporting Generic
> Notice type:3 num:64 from LID:0x0092
> GID:0xfe80000000000000,0x0005ad0000024bbb
> Mar 20 14:28:01 050554 [41E02960] -> Discovered new port with
> GUID:0x0005ad0000024d8b LID range [0xB7,0xB7] of node:saguaro-23-8 HCA-1
> Mar 20 14:28:01 081322 [41E02960] -> osm_ucast_mgr_process: Min Hop Tables
> configured on all switches
> Mar 20 14:28:01 142873 [43204960] -> SUBNET UP
> Mar 20 14:28:01 460275 [44606960] -> __osm_trap_rcv_process_request:
> Received Generic Notice type:0x01 num:128 Producer:2 from LID:0x0152
> TID:0x000000000000001d
> Mar 20 14:28:01 460358 [44606960] -> __osm_trap_rcv_process_request: ERR
> 3804: Received trap 29 times consecutively
> Mar 20 14:28:01 488474 [45007960] -> osm_ucast_mgr_process: Min Hop Tables
> configured on all switches
> Mar 20 14:28:01 538634 [4780B960] -> __osm_sm_mad_ctrl_rcv_callback: ERR
> 3111: Error status = 0x1C00
> Mar 20 14:28:01 538712 [4780B960] -> SMP dump:
> base_ver................0x1
> mgmt_class..............0x81
> class_ver...............0x1
> method..................0x81 (SubnGetResp)
> D bit...................0x1
> status..................0x1C00
> hop_ptr.................0x0
> hop_count...............0x4
> trans_id................0x898d1
> attr_id.................0x15 (PortInfo)
> resv....................0x0
> attr_mod................0x16
> m_key...................0x0000000000000000
> dr_slid.................0xFFFF
> dr_dlid.................0xFFFF
>
> Initial path: [0][1][11][1][6]
> Return path: [0][9][18][D][3]
> Reserved: [0][0][0][0][0][0][0]
>
> 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
>
> 00 00 00 00 00 00 00 00 00 00 00 00 03 03 03 02
>
> 11 42 00 11 40 40 00 08 08 04 F2 40 00 00 00 00
>
> 00 00 00 00 00 88 00 00 00 00 00 00 00 00 00 00
>
> Mar 20 14:28:01 538752 [42803960] -> osm_pi_rcv_process_set: ERR 0F10:
> Received error status for SetResp()
> Mar 20 14:28:01 538758 [4780B960] -> __osm_sm_mad_ctrl_rcv_callback: ERR
> 3111: Error status = 0x1C00
> Mar 20 14:28:01 538767 [42803960] -> PortInfo dump:
> port number.............0x16
> node_guid...............0x0005ad00000281b3
> port_guid...............0x0005ad00000281b3
> m_key...................0x0000000000000000
> subnet_prefix...........0x0000000000000000
> base_lid................0x0
> master_sm_base_lid......0x0
> capability_mask.........0x0
> diag_code...............0x0
> m_key_lease_period......0x0
> local_port_num..........0x3
> link_width_enabled......0x3
> link_width_supported....0x3
> link_width_active.......0x2
> link_speed_supported....0x1
> port_state..............DOWN
> state_info2.............0x42
> m_key_protect_bits......0x0
> lmc.....................0x0
> link_speed..............0x11
> mtu_smsl................0x40
> vl_cap_init_type........0x40
> vl_high_limit...........0x0
> vl_arb_high_cap.........0x8
> vl_arb_low_cap..........0x8
> init_rep_mtu_cap........0x4
> vl_stall_life...........0xF2
> vl_enforce..............0x40
> m_key_violations........0x0
> p_key_violations........0x0
> q_key_violations........0x0
> guid_cap................0x0
> client_reregister.......0x0
> subnet_timeout..........0x0
> resp_time_value.........0x0
> error_threshold.........0x88
> Mar 20 14:28:01 538795 [4780B960] -> SMP dump:
> base_ver................0x1
> mgmt_class..............0x81
> class_ver...............0x1
> method..................0x81 (SubnGetResp)
> D bit...................0x1
> status..................0x1C00
> hop_ptr.................0x0
> hop_count...............0x4
> trans_id................0x898d2
> attr_id.................0x15 (PortInfo)
> resv....................0x0
> attr_mod................0x17
> m_key...................0x0000000000000000
> dr_slid.................0xFFFF
> dr_dlid.................0xFFFF
>
> Initial path: [0][1][11][1][6]
> Return path: [0][9][18][D][3]
> Reserved: [0][0][0][0][0][0][0]
>
> 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
>
> 00 00 00 00 00 00 00 00 00 00 00 00 03 03 03 02
>
> 11 42 00 11 40 40 00 08 08 04 F2 40 00 00 00 00
>
> 00 00 00 00 00 88 00 00 00 00 00 00 00 00 00 00
>
> Mar 20 14:28:01 538810 [42803960] -> Capabilities Mask:
> Mar 20 14:28:01 538849 [42803960] -> osm_pi_rcv_process_set: ERR 0F10:
> Received error status for SetResp()
> Mar 20 14:28:01 538856 [42803960] -> PortInfo dump:
> port number.............0x17
> node_guid...............0x0005ad00000281b3
> port_guid...............0x0005ad00000281b3
> m_key...................0x0000000000000000
> subnet_prefix...........0x0000000000000000
> base_lid................0x0
> master_sm_base_lid......0x0
> capability_mask.........0x0
> diag_code...............0x0
> m_key_lease_period......0x0
> local_port_num..........0x3
> link_width_enabled......0x3
> link_width_supported....0x3
> link_width_active.......0x2
> link_speed_supported....0x1
> port_state..............DOWN
> state_info2.............0x42
> m_key_protect_bits......0x0
> lmc.....................0x0
> link_speed..............0x11
> mtu_smsl................0x40
> vl_cap_init_type........0x40
> vl_high_limit...........0x0
> vl_arb_high_cap.........0x8
> vl_arb_low_cap..........0x8
> init_rep_mtu_cap........0x4
> vl_stall_life...........0xF2
> vl_enforce..............0x40
> m_key_violations........0x0
> p_key_violations........0x0
> q_key_violations........0x0
> guid_cap................0x0
> client_reregister.......0x0
> subnet_timeout..........0x0
> resp_time_value.........0x0
> error_threshold.........0x88
> Mar 20 14:28:01 538871 [42803960] -> Capabilities Mask:
> Mar 20 14:28:01 539658 [4780B960] -> __osm_sm_mad_ctrl_rcv_callback: ERR
> 3111: Error status = 0x1C00
> Mar 20 14:28:01 539696 [4780B960] -> SMP dump:
> base_ver................0x1
> mgmt_class..............0x81
> class_ver...............0x1
> method..................0x81 (SubnGetResp)
> D bit...................0x1
> status..................0x1C00
> hop_ptr.................0x0
> hop_count...............0x5
> trans_id................0x898d3
> attr_id.................0x15 (PortInfo)
> resv....................0x0
> attr_mod................0x11
> m_key...................0x0000000000000000
> dr_slid.................0xFFFF
> dr_dlid.................0xFFFF
>
> Initial path: [0][1][15][1][4][17]
> Return path: [0][9][18][D][1][16]
> Reserved: [0][0][0][0][0][0][0]
>
> 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
>
> 00 00 00 00 00 00 00 00 00 00 00 00 16 02 03 02
>
> 11 42 00 11 40 40 00 08 08 04 F2 40 00 00 00 00
>
> 00 00 00 00 00 88 00 00 00 00 00 00 00 00 00 00
>
> Mar 20 14:28:01 539778 [45A08960] -> osm_pi_rcv_process_set: Received error
> status 0x1c for SetResp() during ACTIVE transition
> Mar 20 14:28:01 539784 [4780B960] -> __osm_sm_mad_ctrl_rcv_callback: ERR
> 3111: Error status = 0x1C00
> Mar 20 14:28:01 539798 [45A08960] -> PortInfo dump:
> port number.............0x11
> node_guid...............0x0005ad0000027c84
> port_guid...............0x0005ad0000027c84
> m_key...................0x0000000000000000
> subnet_prefix...........0x0000000000000000
> base_lid................0x0
> master_sm_base_lid......0x0
> capability_mask.........0x0
> diag_code...............0x0
> m_key_lease_period......0x0
> local_port_num..........0x16
> link_width_enabled......0x2
> link_width_supported....0x3
> link_width_active.......0x2
> link_speed_supported....0x1
> port_state..............DOWN
> state_info2.............0x42
> m_key_protect_bits......0x0
> lmc.....................0x0
> link_speed..............0x11
> mtu_smsl................0x40
> vl_cap_init_type........0x40
> vl_high_limit...........0x0
> vl_arb_high_cap.........0x8
> vl_arb_low_cap..........0x8
> init_rep_mtu_cap........0x4
> vl_stall_life...........0xF2
> vl_enforce..............0x40
> m_key_violations........0x0
> p_key_violations........0x0
> q_key_violations........0x0
> guid_cap................0x0
> client_reregister.......0x0
> subnet_timeout..........0x0
> resp_time_value.........0x0
> error_threshold.........0x88
> Mar 20 14:28:01 539834 [45A08960] -> Capabilities Mask:
> Mar 20 14:28:01 539844 [4780B960] -> SMP dump:
> base_ver................0x1
> mgmt_class..............0x81
> class_ver...............0x1
> method..................0x81 (SubnGetResp)
> D bit...................0x1
> status..................0x1C00
> hop_ptr.................0x0
> hop_count...............0x5
> trans_id................0x898d4
> attr_id.................0x15 (PortInfo)
> resv....................0x0
> attr_mod................0x12
> m_key...................0x0000000000000000
> dr_slid.................0xFFFF
> dr_dlid.................0xFFFF
>
> Initial path: [0][1][15][1][4][17]
> Return path: [0][9][18][D][1][16]
> Reserved: [0][0][0][0][0][0][0]
>
> 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
>
> 00 00 00 00 00 00 00 00 00 00 00 00 16 02 03 02
>
> 11 42 00 11 40 40 00 08 08 04 F2 40 00 00 00 00
>
> 00 00 00 00 00 88 00 00 00 00 00 00 00 00 00 00
>
> Mar 20 14:28:01 539903 [45007960] -> osm_pi_rcv_process_set: Received error
> status 0x1c for SetResp() during ACTIVE transition
> Mar 20 14:28:01 539908 [45007960] -> PortInfo dump:
> port number.............0x12
> node_guid...............0x0005ad0000027c84
> port_guid...............0x0005ad0000027c84
> m_key...................0x0000000000000000
> subnet_prefix...........0x0000000000000000
> base_lid................0x0
> master_sm_base_lid......0x0
> capability_mask.........0x0
> diag_code...............0x0
> m_key_lease_period......0x0
> local_port_num..........0x16
> link_width_enabled......0x2
> link_width_supported....0x3
> link_width_active.......0x2
> link_speed_supported....0x1
> port_state..............DOWN
> state_info2.............0x42
> m_key_protect_bits......0x0
> lmc.....................0x0
> link_speed..............0x11
> mtu_smsl................0x40
> vl_cap_init_type........0x40
> vl_high_limit...........0x0
> vl_arb_high_cap.........0x8
> vl_arb_low_cap..........0x8
> init_rep_mtu_cap........0x4
> vl_stall_life...........0xF2
> vl_enforce..............0x40
> m_key_violations........0x0
> p_key_violations........0x0
> q_key_violations........0x0
> guid_cap................0x0
> client_reregister.......0x0
> subnet_timeout..........0x0
> resp_time_value.........0x0
> error_threshold.........0x88
> Mar 20 14:28:01 539924 [45007960] -> Capabilities Mask:
> Mar 20 14:28:01 545091 [45007960] -> SUBNET UP
> Mar 20 14:28:01 652647 [43204960] -> __osm_trap_rcv_process_request:
> Received Generic Notice type:0x01 num:128 Producer:2 from LID:0x001B
> TID:0x0000000000000084
> Mar 20 14:28:01 652864 [43204960] -> osm_report_notice: Reporting Generic
> Notice type:1 num:128 from LID:0x001B
> GID:0xfe80000000000000,0x0005ad00000281a7
> Mar 20 14:28:01 879631 [44606960] -> __osm_trap_rcv_process_request:
> Received Generic Notice type:0x01 num:128 Producer:2 from LID:0x0148
> TID:0x0000000000000074
> Mar 20 14:28:01 880104 [44606960] -> osm_report_notice: Reporting Generic
> Notice type:1 num:128 from LID:0x0148
> GID:0xfe80000000000000,0x0005ad00000281b3
> Mar 20 14:28:01 962839 [44606960] -> __osm_trap_rcv_process_request:
> Received Generic Notice type:0x01 num:128 Producer:2 from LID:0x0148
> TID:0x0000000000000075
> Mar 20 14:28:01 965155 [44606960] -> osm_report_notice: Reporting Generic
> Notice type:1 num:128 from LID:0x0148
> GID:0xfe80000000000000,0x0005ad00000281b3
> Mar 20 14:28:02 006432 [41401960] -> __osm_trap_rcv_process_request:
> Received Generic Notice type:0x01 num:128 Producer:2 from LID:0x001B
> TID:0x0000000000000085
> Mar 20 14:28:02 030610 [42803960] -> osm_ucast_mgr_process: Min Hop Tables
> configured on all switches
> Mar 20 14:28:02 068999 [41401960] -> osm_report_notice: Reporting Generic
> Notice type:1 num:128 from LID:0x001B
> GID:0xfe80000000000000,0x0005ad00000281a7
> Mar 20 14:28:02 081130 [4780B960] -> __osm_sm_mad_ctrl_rcv_callback: ERR
> 3111: Error status = 0x1C00
> Mar 20 14:28:02 081198 [4780B960] -> SMP dump:
> base_ver................0x1
> mgmt_class..............0x81
> class_ver...............0x1
> method..................0x81 (SubnGetResp)
> D bit...................0x1
> status..................0x1C00
> hop_ptr.................0x0
> hop_count...............0x4
> trans_id................0x8a604
> attr_id.................0x15 (PortInfo)
> resv....................0x0
> attr_mod................0x16
> m_key...................0x0000000000000000
> dr_slid.................0xFFFF
> dr_dlid.................0xFFFF
>
> Initial path: [0][1][11][4][4]
> Return path: [0][9][18][D][4]
> Reserved: [0][0][0][0][0][0][0]
>
> 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
>
> 00 00 00 00 00 00 00 00 00 00 00 00 04 03 03 02
>
> 14 52 00 11 40 40 00 08 08 04 F2 40 00 00 00 00
>
> 00 00 00 00 00 88 00 00 00 00 00 00 00 00 00 00
>
> Mar 20 14:28:02 081249 [43204960] -> osm_pi_rcv_process_set: Received error
> status 0x1c for SetResp() during ACTIVE transition
> Mar 20 14:28:02 081257 [43204960] -> PortInfo dump:
> port number.............0x16
> node_guid...............0x0005ad00000281b3
> port_guid...............0x0005ad00000281b3
> m_key...................0x0000000000000000
> subnet_prefix...........0x0000000000000000
> base_lid................0x0
> master_sm_base_lid......0x0
> capability_mask.........0x0
> diag_code...............0x0
> m_key_lease_period......0x0
> local_port_num..........0x4
> link_width_enabled......0x3
> link_width_supported....0x3
> link_width_active.......0x2
> link_speed_supported....0x1
> port_state..............ACTIVE
> state_info2.............0x52
> m_key_protect_bits......0x0
> lmc.....................0x0
> link_speed..............0x11
> mtu_smsl................0x40
> vl_cap_init_type........0x40
> vl_high_limit...........0x0
> vl_arb_high_cap.........0x8
> vl_arb_low_cap..........0x8
> init_rep_mtu_cap........0x4
> vl_stall_life...........0xF2
> vl_enforce..............0x40
> m_key_violations........0x0
> p_key_violations........0x0
> q_key_violations........0x0
> guid_cap................0x0
> client_reregister.......0x0
> subnet_timeout..........0x0
> resp_time_value.........0x0
> error_threshold.........0x88
> Mar 20 14:28:02 081275 [43204960] -> Capabilities Mask:
> Mar 20 14:28:02 081650 [4780B960] -> __osm_sm_mad_ctrl_rcv_callback: ERR
> 3111: Error status = 0x1C00
> Mar 20 14:28:02 081713 [4780B960] -> SMP dump:
> base_ver................0x1
> mgmt_class..............0x81
> class_ver...............0x1
> method..................0x81 (SubnGetResp)
> D bit...................0x1
> status..................0x1C00
> hop_ptr.................0x0
> hop_count...............0x4
> trans_id................0x8a605
> attr_id.................0x15 (PortInfo)
> resv....................0x0
> attr_mod................0x17
> m_key...................0x0000000000000000
> dr_slid.................0xFFFF
> dr_dlid.................0xFFFF
>
> Initial path: [0][1][11][4][4]
> Return path: [0][9][18][D][4]
> Reserved: [0][0][0][0][0][0][0]
>
> 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
>
> 00 00 00 00 00 00 00 00 00 00 00 00 04 03 03 02
>
> 14 52 00 11 40 40 00 08 08 04 F2 40 00 00 00 00
>
> 00 00 00 00 00 88 00 00 00 00 00 00 00 00 00 00
>
> Mar 20 14:28:02 081782 [43C05960] -> osm_pi_rcv_process_set: Received error
> status 0x1c for SetResp() during ACTIVE transition
> Mar 20 14:28:02 081787 [43C05960] -> PortInfo dump:
> port number.............0x17
> node_guid...............0x0005ad00000281b3
> port_guid...............0x0005ad00000281b3
> m_key...................0x0000000000000000
> subnet_prefix...........0x0000000000000000
> base_lid................0x0
> master_sm_base_lid......0x0
> capability_mask.........0x0
> diag_code...............0x0
> m_key_lease_period......0x0
> local_port_num..........0x4
> link_width_enabled......0x3
> link_width_supported....0x3
> link_width_active.......0x2
> link_speed_supported....0x1
> port_state..............ACTIVE
> state_info2.............0x52
> m_key_protect_bits......0x0
> lmc.....................0x0
> link_speed..............0x11
> mtu_smsl................0x40
> vl_cap_init_type........0x40
> vl_high_limit...........0x0
> vl_arb_high_cap.........0x8
> vl_arb_low_cap..........0x8
> init_rep_mtu_cap........0x4
> vl_stall_life...........0xF2
> vl_enforce..............0x40
> m_key_violations........0x0
> p_key_violations........0x0
> q_key_violations........0x0
> guid_cap................0x0
> client_reregister.......0x0
> subnet_timeout..........0x0
> resp_time_value.........0x0
> error_threshold.........0x88
> Mar 20 14:28:02 081802 [43C05960] -> Capabilities Mask:
> Mar 20 14:28:02 087435 [45A08960] -> SUBNET UP
> Mar 20 14:28:02 170696 [41401960] -> __osm_trap_rcv_process_request:
> Received Generic Notice type:0x01 num:128 Producer:2 from LID:0x001B
> TID:0x0000000000000086
> Mar 20 14:28:02 170819 [41401960] -> osm_report_notice: Reporting Generic
> Notice type:1 num:128 from LID:0x001B
> GID:0xfe80000000000000,0x0005ad00000281a7
> Mar 20 14:28:02 459228 [41E02960] -> osm_ucast_mgr_process: Min Hop Tables
> configured on all switches
> Mar 20 14:28:02 500761 [43C05960] -> __osm_trap_rcv_process_request:
> Received Generic Notice type:0x01 num:128 Producer:2 from LID:0x001B
> TID:0x0000000000000087
> Mar 20 14:28:02 500979 [43C05960] -> osm_report_notice: Reporting Generic
> Notice type:1 num:128 from LID:0x001B
> GID:0xfe80000000000000,0x0005ad00000281a7
> Mar 20 14:28:02 510190 [4780B960] -> __osm_sm_mad_ctrl_rcv_callback: ERR
> 3111: Error status = 0x1C00
> Mar 20 14:28:02 510258 [4780B960] -> SMP dump:
> base_ver................0x1
> mgmt_class..............0x81
> class_ver...............0x1
> method..................0x81 (SubnGetResp)
> D bit...................0x1
> status..................0x1C00
> hop_ptr.................0x0
> hop_count...............0x4
> trans_id................0x8b330
> attr_id.................0x15 (PortInfo)
> resv....................0x0
> attr_mod................0x16
> m_key...................0x0000000000000000
> dr_slid.................0xFFFF
> dr_dlid.................0xFFFF
>
> Initial path: [0][1][17][1][5]
> Return path: [0][9][14][D][2]
> Reserved: [0][0][0][0][0][0][0]
>
> 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
>
> 00 00 00 00 00 00 00 00 00 00 00 00 02 03 03 02
>
> 14 52 00 11 40 40 00 08 08 04 F2 40 00 00 00 00
>
> 00 00 00 00 00 88 00 00 00 00 00 00 00 00 00 00
>
> Mar 20 14:28:02 510366 [4780B960] -> __osm_sm_mad_ctrl_rcv_callback: ERR
> 3111: Error status = 0x1C00
> Mar 20 14:28:02 510370 [45007960] -> osm_pi_rcv_process_set: Received error
> status 0x1c for SetResp() during ACTIVE transition
> Mar 20 14:28:02 510384 [45007960] -> PortInfo dump:
> port number.............0x16
> node_guid...............0x0005ad00000281a7
> port_guid...............0x0005ad00000281a7
> m_key...................0x0000000000000000
> subnet_prefix...........0x0000000000000000
> base_lid................0x0
> master_sm_base_lid......0x0
> capability_mask.........0x0
> diag_code...............0x0
> m_key_lease_period......0x0
> local_port_num..........0x2
> link_width_enabled......0x3
> link_width_supported....0x3
> link_width_active.......0x2
> link_speed_supported....0x1
> port_state..............ACTIVE
> state_info2.............0x52
> m_key_protect_bits......0x0
> lmc.....................0x0
> link_speed..............0x11
> mtu_smsl................0x40
> vl_cap_init_type........0x40
> vl_high_limit...........0x0
> vl_arb_high_cap.........0x8
> vl_arb_low_cap..........0x8
> init_rep_mtu_cap........0x4
> vl_stall_life...........0xF2
> vl_enforce..............0x40
> m_key_violations........0x0
> p_key_violations........0x0
> q_key_violations........0x0
> guid_cap................0x0
> client_reregister.......0x0
> subnet_timeout..........0x0
> resp_time_value.........0x0
> error_threshold.........0x88
> Mar 20 14:28:02 510394 [4780B960] -> SMP dump:
> base_ver................0x1
> mgmt_class..............0x81
> class_ver...............0x1
> method..................0x81 (SubnGetResp)
> D bit...................0x1
> status..................0x1C00
> hop_ptr.................0x0
> hop_count...............0x4
> trans_id................0x8b331
> attr_id.................0x15 (PortInfo)
> resv....................0x0
> attr_mod................0x18
> m_key...................0x0000000000000000
> dr_slid.................0xFFFF
> dr_dlid.................0xFFFF
>
> Initial path: [0][1][11][3][6]
> Return path: [0][9][18][F][3]
> Reserved: [0][0][0][0][0][0][0]
>
> 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
>
> 00 00 00 00 00 00 00 00 00 00 00 00 03 03 03 02
>
> 14 52 00 11 40 40 00 08 08 04 F2 40 00 00 00 00
>
> 00 00 00 00 00 88 00 00 00 00 00 00 00 00 00 00
>
> Mar 20 14:28:02 510398 [45007960] -> Capabilities Mask:
> Mar 20 14:28:02 510481 [41401960] -> osm_pi_rcv_process_set: Received error
> status 0x1c for SetResp() during ACTIVE transition
> Mar 20 14:28:02 510491 [41401960] -> PortInfo dump:
> port number.............0x18
> node_guid...............0x0005ad00000281b3
> port_guid...............0x0005ad00000281b3
> m_key...................0x0000000000000000
> subnet_prefix...........0x0000000000000000
> base_lid................0x0
> master_sm_base_lid......0x0
> capability_mask.........0x0
> diag_code...............0x0
> m_key_lease_period......0x0
> local_port_num..........0x3
> link_width_enabled......0x3
> link_width_supported....0x3
> link_width_active.......0x2
> link_speed_supported....0x1
> port_state..............ACTIVE
> state_info2.............0x52
> m_key_protect_bits......0x0
> lmc.....................0x0
> link_speed..............0x11
> mtu_smsl................0x40
> vl_cap_init_type........0x40
> vl_high_limit...........0x0
> vl_arb_high_cap.........0x8
> vl_arb_low_cap..........0x8
> init_rep_mtu_cap........0x4
> vl_stall_life...........0xF2
> vl_enforce..............0x40
> m_key_violations........0x0
> p_key_violations........0x0
> q_key_violations........0x0
> guid_cap................0x0
> client_reregister.......0x0
> subnet_timeout..........0x0
> resp_time_value.........0x0
> error_threshold.........0x88
> Mar 20 14:28:02 510509 [41401960] -> Capabilities Mask:
> Mar 20 14:28:02 510511 [42803960] -> __osm_trap_rcv_process_request:
> Received Generic Notice type:0x01 num:128 Producer:2 from LID:0x001B
> TID:0x0000000000000088
> Mar 20 14:28:02 515576 [41401960] -> SUBNET UP
> Mar 20 14:28:02 515695 [42803960] -> osm_report_notice: Reporting Generic
> Notice type:1 num:128 from LID:0x001B
> GID:0xfe80000000000000,0x0005ad00000281a7
> Mar 20 14:28:02 532552 [45A08960] -> __osm_trap_rcv_process_request:
> Received Generic Notice type:0x01 num:128 Producer:2 from LID:0x001B
> TID:0x0000000000000089
> Mar 20 14:28:02 538569 [45A08960] -> osm_report_notice: Reporting Generic
> Notice type:1 num:128 from LID:0x001B
> GID:0xfe80000000000000,0x0005ad00000281a7
> Mar 20 14:28:02 695997 [43204960] -> __osm_trap_rcv_process_request:
> Received Generic Notice type:0x01 num:128 Producer:2 from LID:0x0152
> TID:0x000000000000001e
> Mar 20 14:28:02 696096 [43204960] -> __osm_trap_rcv_process_request: ERR
> 3804: Received trap 30 times consecutively
> Mar 20 14:28:02 918226 [45007960] -> osm_ucast_mgr_process: Min Hop Tables
> configured on all switches
> Mar 20 14:28:02 975244 [43204960] -> SUBNET UP
> Mar 20 14:28:03 325494 [43204960] -> osm_ucast_mgr_process: Min Hop Tables
> configured on all switches
> Mar 20 14:28:03 379145 [41401960] -> SUBNET UP
> Mar 20 14:29:12 561841 [41401960] -> __osm_trap_rcv_process_request:
> Received Generic Notice type:0x01 num:128 Producer:2 from LID:0x001F
> TID:0x0000000000000019
> Mar 20 14:29:12 562033 [41401960] -> osm_report_notice: Reporting Generic
> Notice type:1 num:128 from LID:0x001F
> GID:0xfe80000000000000,0x0005ad0000027c56
> Mar 20 14:29:12 562751 [42803960] -> __osm_trap_rcv_process_request:
> Received Generic Notice type:0x01 num:128 Producer:2 from LID:0x001F
> TID:0x000000000000001a
> Mar 20 14:29:12 562902 [42803960] -> osm_report_notice: Reporting Generic
> Notice type:1 num:128 from LID:0x001F
> GID:0xfe80000000000000,0x0005ad0000027c56
> Mar 20 14:29:12 571346 [42803960] -> __osm_trap_rcv_process_request:
> Received Generic Notice type:0x01 num:128 Producer:2 from LID:0x0084
> TID:0x0000000000000018
> Mar 20 14:29:12 571569 [42803960] -> osm_report_notice: Reporting Generic
> Notice type:1 num:128 from LID:0x0084
> GID:0xfe80000000000000,0x0005ad0000027c70
> Mar 20 14:29:12 914371 [42803960] -> __osm_trap_rcv_process_request:
> Received Generic Notice type:0x01 num:128 Producer:2 from LID:0x001F
> TID:0x000000000000001b
> Mar 20 14:29:12 916287 [44606960] -> osm_report_notice: Reporting Generic
> Notice type:3 num:64 from LID:0x0092
> GID:0xfe80000000000000,0x0005ad0000024bbb
> Mar 20 14:29:12 916297 [44606960] -> Discovered new port with
> GUID:0x0005ad000002502f LID range [0x2,0x2] of node:Topspin IB-DC
> Mar 20 14:29:12 946985 [44606960] -> osm_ucast_mgr_process: Min Hop Tables
> configured on all switches
> Mar 20 14:29:12 976839 [42803960] -> osm_report_notice: Reporting Generic
> Notice type:1 num:128 from LID:0x001F
> GID:0xfe80000000000000,0x0005ad0000027c56
> Mar 20 14:29:12 987963 [4780B960] -> __osm_sm_mad_ctrl_rcv_callback: ERR
> 3111: Error status = 0x1C00
> Mar 20 14:29:12 988004 [4780B960] -> SMP dump:
> base_ver................0x1
> mgmt_class..............0x81
> class_ver...............0x1
> method..................0x81 (SubnGetResp)
> D bit...................0x1
> status..................0x1C00
> hop_ptr.................0x0
> hop_count...............0x5
> trans_id................0x8dbb2
> attr_id.................0x15 (PortInfo)
> resv....................0x0
> attr_mod................0xD
> m_key...................0x0000000000000000
> dr_slid.................0xFFFF
> dr_dlid.................0xFFFF
>
> Initial path: [0][1][11][2][4][D]
> Return path: [0][9][18][E][1][12]
> Reserved: [0][0][0][0][0][0][0]
>
> 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
>
> 00 00 00 00 00 00 00 00 00 00 00 00 12 03 03 02
>
> 14 52 00 11 40 40 00 08 08 04 2C 4C 00 00 00 00
>
> 00 00 00 00 00 88 00 00 00 00 00 00 00 00 00 00
>
> Mar 20 14:29:12 988078 [43C05960] -> osm_pi_rcv_process_set: Received error
> status 0x1c for SetResp() during ACTIVE transition
> Mar 20 14:29:12 988089 [43C05960] -> PortInfo dump:
> port number.............0xD
> node_guid...............0x0005ad0000027c70
> port_guid...............0x0005ad0000027c70
> m_key...................0x0000000000000000
> subnet_prefix...........0x0000000000000000
> base_lid................0x0
> master_sm_base_lid......0x0
> capability_mask.........0x0
> diag_code...............0x0
> m_key_lease_period......0x0
> local_port_num..........0x12
> link_width_enabled......0x3
> link_width_supported....0x3
> link_width_active.......0x2
> link_speed_supported....0x1
> port_state..............ACTIVE
> state_info2.............0x52
> m_key_protect_bits......0x0
> lmc.....................0x0
> link_speed..............0x11
> mtu_smsl................0x40
> vl_cap_init_type........0x40
> vl_high_limit...........0x0
> vl_arb_high_cap.........0x8
> vl_arb_low_cap..........0x8
> init_rep_mtu_cap........0x4
> vl_stall_life...........0x2C
> vl_enforce..............0x4C
> m_key_violations........0x0
> p_key_violations........0x0
> q_key_violations........0x0
> guid_cap................0x0
> client_reregister.......0x0
> subnet_timeout..........0x0
> resp_time_value.........0x0
> error_threshold.........0x88
> Mar 20 14:29:12 988105 [43C05960] -> Capabilities Mask:
> Mar 20 14:29:12 993136 [44606960] -> SUBNET UP
> Mar 20 14:29:13 300755 [41E02960] -> __osm_trap_rcv_process_request:
> Received Generic Notice type:0x04 num:144 Producer:1 from LID:0x0002
> TID:0x0000000000000000
> Mar 20 14:29:13 300874 [41E02960] -> osm_report_notice: Reporting Generic
> Notice type:4 num:144 from LID:0x0002
> GID:0xfe80000000000000,0x0005ad000002502f
> Mar 20 14:29:13 338077 [41E02960] -> osm_report_notice: Reporting Generic
> Notice type:3 num:64 from LID:0x0092
> GID:0xfe80000000000000,0x0005ad0000024bbb
> Mar 20 14:29:13 338099 [41E02960] -> Discovered new port with
> GUID:0x0005ad000002516f LID range [0xBA,0xBA] of node:Topspin IB-DC
> Mar 20 14:29:13 368879 [41E02960] -> osm_ucast_mgr_process: Min Hop Tables
> configured on all switches
> Mar 20 14:29:13 431763 [4780B960] -> __osm_sm_mad_ctrl_rcv_callback: ERR
> 3111: Error status = 0x1C00
> Mar 20 14:29:13 431806 [4780B960] -> SMP dump:
> base_ver................0x1
> mgmt_class..............0x81
> class_ver...............0x1
> method..................0x81 (SubnGetResp)
> D bit...................0x1
> status..................0x1C00
> hop_ptr.................0x0
> hop_count...............0x5
> trans_id................0x8e8e9
> attr_id.................0x15 (PortInfo)
> resv....................0x0
> attr_mod................0xA
> m_key...................0x0000000000000000
> dr_slid.................0xFFFF
> dr_dlid.................0xFFFF
>
> Initial path: [0][1][14][1][6][12]
> Return path: [0][9][15][D][3][11]
> Reserved: [0][0][0][0][0][0][0]
>
> 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
>
> 00 00 00 00 00 00 00 00 00 00 00 00 11 03 03 02
>
> 14 52 00 11 40 40 00 08 08 04 2C 4C 00 00 00 00
>
> 00 00 00 00 00 88 00 00 00 00 00 00 00 00 00 00
>
> Mar 20 14:29:13 432093 [43204960] -> osm_pi_rcv_process_set: Received error
> status 0x1c for SetResp() during ACTIVE transition
> Mar 20 14:29:13 432116 [43204960] -> PortInfo dump:
> port number.............0xA
> node_guid...............0x0005ad0000027c56
> port_guid...............0x0005ad0000027c56
> m_key...................0x0000000000000000
> subnet_prefix...........0x0000000000000000
> base_lid................0x0
> master_sm_base_lid......0x0
> capability_mask.........0x0
> diag_code...............0x0
> m_key_lease_period......0x0
> local_port_num..........0x11
> link_width_enabled......0x3
> link_width_supported....0x3
> link_width_active.......0x2
> link_speed_supported....0x1
> port_state..............ACTIVE
> state_info2.............0x52
> m_key_protect_bits......0x0
> lmc.....................0x0
> link_speed..............0x11
> mtu_smsl................0x40
> vl_cap_init_type........0x40
> vl_high_limit...........0x0
> vl_arb_high_cap.........0x8
> vl_arb_low_cap..........0x8
> init_rep_mtu_cap........0x4
> vl_stall_life...........0x2C
> vl_enforce..............0x4C
> m_key_violations........0x0
> p_key_violations........0x0
> q_key_violations........0x0
> guid_cap................0x0
> client_reregister.......0x0
> subnet_timeout..........0x0
> resp_time_value.........0x0
> error_threshold.........0x88
> Mar 20 14:29:13 432135 [43204960] -> Capabilities Mask:
> Mar 20 14:29:13 437219 [45007960] -> SUBNET UP
> Mar 20 14:29:13 690992 [42803960] -> __osm_trap_rcv_process_request:
> Received Generic Notice type:0x04 num:144 Producer:1 from LID:0x00BA
> TID:0x0000000000000000
> Mar 20 14:29:13 691128 [42803960] -> osm_report_notice: Reporting Generic
> Notice type:4 num:144 from LID:0x00BA
> GID:0xfe80000000000000,0x0005ad000002516f
> Mar 20 14:29:13 835017 [44606960] -> osm_ucast_mgr_process: Min Hop Tables
> configured on all switches
> Mar 20 14:29:13 891082 [42803960] -> SUBNET UP
> Mar 20 14:29:14 235714 [42803960] -> osm_ucast_mgr_process: Min Hop Tables
> configured on all switches
> Mar 20 14:29:14 289127 [41E02960] -> SUBNET UP
> Mar 20 14:29:17 689267 [43204960] -> __osm_trap_rcv_process_request:
> Received Generic Notice type:0x01 num:128 Producer:2 from LID:0x001B
> TID:0x000000000000008a
> Mar 20 14:29:17 689511 [43204960] -> osm_report_notice: Reporting Generic
> Notice type:1 num:128 from LID:0x001B
> GID:0xfe80000000000000,0x0005ad00000281a7
> Mar 20 14:29:17 689975 [42803960] -> __osm_trap_rcv_process_request:
> Received Generic Notice type:0x01 num:128 Producer:2 from LID:0x0148
> TID:0x0000000000000076
> Mar 20 14:29:17 690097 [42803960] -> osm_report_notice: Reporting Generic
> Notice type:1 num:128 from LID:0x0148
> GID:0xfe80000000000000,0x0005ad00000281b3
> Mar 20 14:29:18 025237 [44606960] -> osm_report_notice: Reporting Generic
> Notice type:3 num:65 from LID:0x0092
> GID:0xfe80000000000000,0x0005ad0000024bbb
> Mar 20 14:29:18 025255 [44606960] -> Removed port with
> GUID:0x0005ad0000024e0b LID range [0xB3,0xB3] of node:saguaro-23-4 HCA-1
> Mar 20 14:29:18 025273 [44606960] -> osm_report_notice: Reporting Generic
> Notice type:3 num:65 from LID:0x0092
> GID:0xfe80000000000000,0x0005ad0000024bbb
> Mar 20 14:29:18 025279 [44606960] -> Removed port with
> GUID:0x0005ad000002510b LID range [0xB5,0xB5] of node:saguaro-23-6 HCA-1
> Mar 20 14:29:18 025296 [44606960] -> osm_report_notice: Reporting Generic
> Notice type:3 num:65 from LID:0x0092
> GID:0xfe80000000000000,0x0005ad0000024bbb
> Mar 20 14:29:18 025300 [44606960] -> Removed port with
> GUID:0x0005ad000002511b LID range [0xA6,0xA6] of node:saguaro-22-1 HCA-1
> Mar 20 14:29:18 025317 [44606960] -> osm_report_notice: Reporting Generic
> Notice type:3 num:65 from LID:0x0092
> GID:0xfe80000000000000,0x0005ad0000024bbb
> Mar 20 14:29:18 025323 [44606960] -> Removed port with
> GUID:0x0005ad0000024b27 LID range [0xAF,0xAF] of node:saguaro-23-0 HCA-1
> Mar 20 14:29:18 025340 [44606960] -> osm_report_notice: Reporting Generic
> Notice type:3 num:65 from LID:0x0092
> GID:0xfe80000000000000,0x0005ad0000024bbb
> Mar 20 14:29:18 025345 [44606960] -> Removed port with
> GUID:0x0005ad0000025043 LID range [0xB4,0xB4] of node:saguaro-23-5 HCA-1
> Mar 20 14:29:18 025362 [44606960] -> osm_report_notice: Reporting Generic
> Notice type:3 num:65 from LID:0x0092
> GID:0xfe80000000000000,0x0005ad0000024bbb
> Mar 20 14:29:18 025367 [44606960] -> Removed port with
> GUID:0x0005ad0000024d47 LID range [0xB6,0xB6] of node:saguaro-23-7 HCA-1
> Mar 20 14:29:18 025385 [44606960] -> osm_report_notice: Reporting Generic
> Notice type:3 num:65 from LID:0x0092
> GID:0xfe80000000000000,0x0005ad0000024bbb
> Mar 20 14:29:18 025390 [44606960] -> Removed port with
> GUID:0x0005ad0000024d6b LID range [0xB8,0xB8] of node:saguaro-23-9 HCA-1
> Mar 20 14:29:18 025406 [44606960] -> osm_report_notice: Reporting Generic
> Notice type:3 num:65 from LID:0x0092
> GID:0xfe80000000000000,0x0005ad0000024bbb
> Mar 20 14:29:18 025411 [44606960] -> Removed port with
> GUID:0x0005ad0000024977 LID range [0xA9,0xA9] of node:saguaro-22-4 HCA-1
> Mar 20 14:29:18 025571 [44606960] -> osm_report_notice: Reporting Generic
> Notice type:3 num:65 from LID:0x0092
> GID:0xfe80000000000000,0x0005ad0000024bbb
> Mar 20 14:29:18 025576 [44606960] -> Removed port with
> GUID:0x0005ad0000027c84 LID range [0x152,0x152] of node:Topspin Switch TS120
> Mar 20 14:29:18 025612 [44606960] -> osm_report_notice: Reporting Generic
> Notice type:3 num:65 from LID:0x0092
> GID:0xfe80000000000000,0x0005ad0000024bbb
> Mar 20 14:29:18 025619 [44606960] -> Removed port with
> GUID:0x0005ad0000024d8b LID range [0xB7,0xB7] of node:saguaro-23-8 HCA-1
> Mar 20 14:29:18 025634 [44606960] -> osm_report_notice: Reporting Generic
> Notice type:3 num:65 from LID:0x0092
> GID:0xfe80000000000000,0x0005ad0000024bbb
> Mar 20 14:29:18 025639 [44606960] -> Removed port with
> GUID:0x0005ad000002498f LID range [0xA8,0xA8] of node:saguaro-22-3 HCA-1
> Mar 20 14:29:18 025655 [44606960] -> osm_report_notice: Reporting Generic
> Notice type:3 num:65 from LID:0x0092
> GID:0xfe80000000000000,0x0005ad0000024bbb
> Mar 20 14:29:18 025660 [44606960] -> Removed port with
> GUID:0x0005ad0000024c9b LID range [0xA7,0xA7] of node:saguaro-22-2 HCA-1
> Mar 20 14:29:18 025678 [44606960] -> osm_report_notice: Reporting Generic
> Notice type:3 num:65 from LID:0x0092
> GID:0xfe80000000000000,0x0005ad0000024bbb
> Mar 20 14:29:18 025683 [44606960] -> Removed port with
> GUID:0x0005ad0000024da7 LID range [0xB0,0xB0] of node:saguaro-23-1 HCA-1
> Mar 20 14:29:18 025700 [44606960] -> osm_report_notice: Reporting Generic
> Notice type:3 num:65 from LID:0x0092
> GID:0xfe80000000000000,0x0005ad0000024bbb
> Mar 20 14:29:18 025705 [44606960] -> Removed port with
> GUID:0x0005ad0000024cbb LID range [0xB2,0xB2] of node:saguaro-23-3 HCA-1
> Mar 20 14:29:18 025721 [44606960] -> osm_report_notice: Reporting Generic
> Notice type:3 num:65 from LID:0x0092
> GID:0xfe80000000000000,0x0005ad0000024bbb
> Mar 20 14:29:18 025777 [44606960] -> Removed port with
> GUID:0x0005ad00000249d3 LID range [0xB1,0xB1] of node:saguaro-23-2 HCA-1
> Mar 20 14:29:18 025794 [44606960] -> osm_report_notice: Reporting Generic
> Notice type:3 num:65 from LID:0x0092
> GID:0xfe80000000000000,0x0005ad0000024bbb
> Mar 20 14:29:18 025800 [44606960] -> Removed port with
> GUID:0x0005ad0000024feb LID range [0x153,0x153] of node:saguaro-22-5 HCA-1
> Mar 20 14:29:18 025816 [44606960] -> osm_report_notice: Reporting Generic
> Notice type:3 num:65 from LID:0x0092
> GID:0xfe80000000000000,0x0005ad0000024bbb
> Mar 20 14:29:18 025821 [44606960] -> Removed port with
> GUID:0x0005ad0000024afb LID range [0xA5,0xA5] of node:saguaro-22-0 HCA-1
> Mar 20 14:29:18 058968 [44606960] -> osm_ucast_mgr_process: Min Hop Tables
> configured on all switches
> Mar 20 14:29:18 112970 [43C05960] -> SUBNET UP
> Mar 20 14:29:18 511156 [45007960] -> osm_ucast_mgr_process: Min Hop Tables
> configured on all switches
> Mar 20 14:29:18 573846 [41E02960] -> SUBNET UP
> Mar 20 14:30:11 599965 [45A08960] -> __osm_trap_rcv_process_request:
> Received Generic Notice type:0x01 num:128 Producer:2 from LID:0x0148
> TID:0x0000000000000077
> Mar 20 14:30:11 600182 [45A08960] -> osm_report_notice: Reporting Generic
> Notice type:1 num:128 from LID:0x0148
> GID:0xfe80000000000000,0x0005ad00000281b3
> Mar 20 14:30:11 606044 [44606960] -> __osm_trap_rcv_process_request:
> Received Generic Notice type:0x01 num:128 Producer:2 from LID:0x0148
> TID:0x0000000000000078
> Mar 20 14:30:11 606078 [43C05960] -> __osm_trap_rcv_process_request:
> Received Generic Notice type:0x01 num:128 Producer:2 from LID:0x001B
> TID:0x000000000000008b
> Mar 20 14:30:11 606178 [44606960] -> osm_report_notice: Reporting Generic
> Notice type:1 num:128 from LID:0x0148
> GID:0xfe80000000000000,0x0005ad00000281b3
> Mar 20 14:30:11 606207 [43C05960] -> osm_report_notice: Reporting Generic
> Notice type:1 num:128 from LID:0x001B
> GID:0xfe80000000000000,0x0005ad00000281a7
> Mar 20 14:30:11 612375 [42803960] -> __osm_trap_rcv_process_request:
> Received Generic Notice type:0x01 num:128 Producer:2 from LID:0x001B
> TID:0x000000000000008c
> Mar 20 14:30:11 612499 [42803960] -> osm_report_notice: Reporting Generic
> Notice type:1 num:128 from LID:0x001B
> GID:0xfe80000000000000,0x0005ad00000281a7
> Mar 20 14:30:11 947057 [45007960] -> osm_report_notice: Reporting Generic
> Notice type:3 num:64 from LID:0x0092
> GID:0xfe80000000000000,0x0005ad0000024bbb
> Mar 20 14:30:11 947074 [45007960] -> Discovered new port with
> GUID:0x0005ad0000027c84 LID range [0x152,0x152] of node:Topspin Switch TS120
> Mar 20 14:30:11 947079 [45007960] -> osm_report_notice: Reporting Generic
> Notice type:3 num:64 from LID:0x0092
> GID:0xfe80000000000000,0x0005ad0000024bbb
> Mar 20 14:30:11 947084 [45007960] -> Discovered new port with
> GUID:0x0005ad0000024b27 LID range [0xAF,0xAF] of node:saguaro-23-0 HCA-1
> Mar 20 14:30:11 947088 [45007960] -> osm_report_notice: Reporting Generic
> Notice type:3 num:64 from LID:0x0092
> GID:0xfe80000000000000,0x0005ad0000024bbb
> Mar 20 14:30:11 947093 [45007960] -> Discovered new port with
> GUID:0x0005ad0000024da7 LID range [0xB0,0xB0] of node:saguaro-23-1 HCA-1
> Mar 20 14:30:11 947097 [45007960] -> osm_report_notice: Reporting Generic
> Notice type:3 num:64 from LID:0x0092
> GID:0xfe80000000000000,0x0005ad0000024bbb
> Mar 20 14:30:11 947102 [45007960] -> Discovered new port with
> GUID:0x0005ad00000249d3 LID range [0xB1,0xB1] of node:saguaro-23-2 HCA-1
> Mar 20 14:30:11 947106 [45007960] -> osm_report_notice: Reporting Generic
> Notice type:3 num:64 from LID:0x0092
> GID:0xfe80000000000000,0x0005ad0000024bbb
> Mar 20 14:30:11 947118 [45007960] -> Discovered new port with
> GUID:0x0005ad0000024cbb LID range [0xB2,0xB2] of node:saguaro-23-3 HCA-1
> Mar 20 14:30:11 947138 [45007960] -> osm_report_notice: Reporting Generic
> Notice type:3 num:64 from LID:0x0092
> GID:0xfe80000000000000,0x0005ad0000024bbb
> Mar 20 14:30:11 947143 [45007960] -> Discovered new port with
> GUID:0x0005ad0000024e0b LID range [0xB3,0xB3] of node:saguaro-23-4 HCA-1
> Mar 20 14:30:11 947148 [45007960] -> osm_report_notice: Reporting Generic
> Notice type:3 num:64 from LID:0x0092
> GID:0xfe80000000000000,0x0005ad0000024bbb
> Mar 20 14:30:11 947153 [45007960] -> Discovered new port with
> GUID:0x0005ad0000025043 LID range [0xB4,0xB4] of node:saguaro-23-5 HCA-1
> Mar 20 14:30:11 947157 [45007960] -> osm_report_notice: Reporting Generic
> Notice type:3 num:64 from LID:0x0092
> GID:0xfe80000000000000,0x0005ad0000024bbb
> Mar 20 14:30:11 947162 [45007960] -> Discovered new port with
> GUID:0x0005ad000002510b LID range [0xB5,0xB5] of node:saguaro-23-6 HCA-1
> Mar 20 14:30:11 947166 [45007960] -> osm_report_notice: Reporting Generic
> Notice type:3 num:64 from LID:0x0092
> GID:0xfe80000000000000,0x0005ad0000024bbb
> Mar 20 14:30:11 947170 [45007960] -> Discovered new port with
> GUID:0x0005ad0000024d47 LID range [0xB6,0xB6] of node:saguaro-23-7 HCA-1
> Mar 20 14:30:11 947174 [45007960] -> osm_report_notice: Reporting Generic
> Notice type:3 num:64 from LID:0x0092
> GID:0xfe80000000000000,0x0005ad0000024bbb
> Mar 20 14:30:11 947179 [45007960] -> Discovered new port with
> GUID:0x0005ad0000024d8b LID range [0xB7,0xB7] of node:saguaro-23-8 HCA-1
> Mar 20 14:30:11 947183 [45007960] -> osm_report_notice: Reporting Generic
> Notice type:3 num:64 from LID:0x0092
> GID:0xfe80000000000000,0x0005ad0000024bbb
> Mar 20 14:30:11 947188 [45007960] -> Discovered new port with
> GUID:0x0005ad0000024d6b LID range [0xB8,0xB8] of node:saguaro-23-9 HCA-1
> Mar 20 14:30:11 947191 [45007960] -> osm_report_notice: Reporting Generic
> Notice type:3 num:64 from LID:0x0092
> GID:0xfe80000000000000,0x0005ad0000024bbb
> Mar 20 14:30:11 947196 [45007960] -> Discovered new port with
> GUID:0x0005ad0000024afb LID range [0xA5,0xA5] of node:saguaro-22-0 HCA-1
> Mar 20 14:30:11 947200 [45007960] -> osm_report_notice: Reporting Generic
> Notice type:3 num:64 from LID:0x0092
> GID:0xfe80000000000000,0x0005ad0000024bbb
> Mar 20 14:30:11 947205 [45007960] -> Discovered new port with
> GUID:0x0005ad000002511b LID range [0xA6,0xA6] of node:saguaro-22-1 HCA-1
> Mar 20 14:30:11 947209 [45007960] -> osm_report_notice: Reporting Generic
> Notice type:3 num:64 from LID:0x0092
> GID:0xfe80000000000000,0x0005ad0000024bbb
> Mar 20 14:30:11 947214 [45007960] -> Discovered new port with
> GUID:0x0005ad0000024c9b LID range [0xA7,0xA7] of node:saguaro-22-2 HCA-1
> Mar 20 14:30:11 947282 [45007960] -> osm_report_notice: Reporting Generic
> Notice type:3 num:64 from LID:0x0092
> GID:0xfe80000000000000,0x0005ad0000024bbb
> Mar 20 14:30:11 947288 [45007960] -> Discovered new port with
> GUID:0x0005ad000002498f LID range [0xA8,0xA8] of node:saguaro-22-3 HCA-1
> Mar 20 14:30:11 947291 [45007960] -> osm_report_notice: Reporting Generic
> Notice type:3 num:64 from LID:0x0092
> GID:0xfe80000000000000,0x0005ad0000024bbb
> Mar 20 14:30:11 947296 [45007960] -> Discovered new port with
> GUID:0x0005ad0000024977 LID range [0xA9,0xA9] of node:saguaro-22-4 HCA-1
> Mar 20 14:30:11 947300 [45007960] -> osm_report_notice: Reporting Generic
> Notice type:3 num:64 from LID:0x0092
> GID:0xfe80000000000000,0x0005ad0000024bbb
> Mar 20 14:30:11 947305 [45007960] -> Discovered new port with
> GUID:0x0005ad0000024feb LID range [0x153,0x153] of node:saguaro-22-5 HCA-1
> Mar 20 14:30:11 978149 [45007960] -> osm_ucast_mgr_process: Min Hop Tables
> configured on all switches
> Mar 20 14:30:12 042474 [4780B960] -> __osm_sm_mad_ctrl_rcv_callback: ERR
> 3111: Error status = 0x1C00
> Mar 20 14:30:12 042577 [4780B960] -> SMP dump:
> base_ver................0x1
> mgmt_class..............0x81
> class_ver...............0x1
> method..................0x81 (SubnGetResp)
> D bit...................0x1
> status..................0x1C00
> hop_ptr.................0x0
> hop_count...............0x4
> trans_id................0x92b38
> attr_id.................0x15 (PortInfo)
> resv....................0x0
> attr_mod................0x13
> m_key...................0x0000000000000000
> dr_slid.................0xFFFF
> dr_dlid.................0xFFFF
>
> Initial path: [0][1][16][1][5]
> Return path: [0][9][13][D][2]
> Reserved: [0][0][0][0][0][0][0]
>
> 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
>
> 00 00 00 00 00 00 00 00 00 00 00 00 02 03 03 02
>
> 14 52 00 11 40 40 00 08 08 04 F2 40 00 00 00 00
>
> 00 00 00 00 00 88 00 00 00 00 00 00 00 00 00 00
>
> Mar 20 14:30:12 042668 [45007960] -> osm_pi_rcv_process_set: Received error
> status 0x1c for SetResp() during ACTIVE transition
> Mar 20 14:30:12 042676 [4780B960] -> __osm_sm_mad_ctrl_rcv_callback: ERR
> 3111: Error status = 0x1C00
> Mar 20 14:30:12 042682 [45007960] -> PortInfo dump:
> port number.............0x13
> node_guid...............0x0005ad00000281a7
> port_guid...............0x0005ad00000281a7
> m_key...................0x0000000000000000
> subnet_prefix...........0x0000000000000000
> base_lid................0x0
> master_sm_base_lid......0x0
> capability_mask.........0x0
> diag_code...............0x0
> m_key_lease_period......0x0
> local_port_num..........0x2
> link_width_enabled......0x3
> link_width_supported....0x3
> link_width_active.......0x2
> link_speed_supported....0x1
> port_state..............ACTIVE
> state_info2.............0x52
> m_key_protect_bits......0x0
> lmc.....................0x0
> link_speed..............0x11
> mtu_smsl................0x40
> vl_cap_init_type........0x40
> vl_high_limit...........0x0
> vl_arb_high_cap.........0x8
> vl_arb_low_cap..........0x8
> init_rep_mtu_cap........0x4
> vl_stall_life...........0xF2
> vl_enforce..............0x40
> m_key_violations........0x0
> p_key_violations........0x0
> q_key_violations........0x0
> guid_cap................0x0
> client_reregister.......0x0
> subnet_timeout..........0x0
> resp_time_value.........0x0
> error_threshold.........0x88
> Mar 20 14:30:12 042701 [45007960] -> Capabilities Mask:
> Mar 20 14:30:12 042714 [4780B960] -> SMP dump:
> base_ver................0x1
> mgmt_class..............0x81
> class_ver...............0x1
> method..................0x81 (SubnGetResp)
> D bit...................0x1
> status..................0x1C00
> hop_ptr.................0x0
> hop_count...............0x4
> trans_id................0x92b39
> attr_id.................0x15 (PortInfo)
> resv....................0x0
> attr_mod................0x16
> m_key...................0x0000000000000000
> dr_slid.................0xFFFF
> dr_dlid.................0xFFFF
>
> Initial path: [0][1][16][1][5]
> Return path: [0][9][13][D][2]
> Reserved: [0][0][0][0][0][0][0]
>
> 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
>
> 00 00 00 00 00 00 00 00 00 00 00 00 02 03 03 02
>
> 14 52 00 11 40 40 00 08 08 04 F2 40 00 00 00 00
>
> 00 00 00 00 00 88 00 00 00 00 00 00 00 00 00 00
>
> Mar 20 14:30:12 042845 [41401960] -> osm_pi_rcv_process_set: Received error
> status 0x1c for SetResp() during ACTIVE transition
> Mar 20 14:30:12 042856 [4780B960] -> __osm_sm_mad_ctrl_rcv_callback: ERR
> 3111: Error status = 0x1C00
> Mar 20 14:30:12 042851 [41401960] -> PortInfo dump:
> port number.............0x16
> node_guid...............0x0005ad00000281a7
> port_guid...............0x0005ad00000281a7
> m_key...................0x0000000000000000
> subnet_prefix...........0x0000000000000000
> base_lid................0x0
> master_sm_base_lid......0x0
> capability_mask.........0x0
> diag_code...............0x0
> m_key_lease_period......0x0
> local_port_num..........0x2
> link_width_enabled......0x3
> link_width_supported....0x3
> link_width_active.......0x2
> link_speed_supported....0x1
> port_state..............ACTIVE
> state_info2.............0x52
> m_key_protect_bits......0x0
> lmc.....................0x0
> link_speed..............0x11
> mtu_smsl................0x40
> vl_cap_init_type........0x40
> vl_high_limit...........0x0
> vl_arb_high_cap.........0x8
> vl_arb_low_cap..........0x8
> init_rep_mtu_cap........0x4
> vl_stall_life...........0xF2
> vl_enforce..............0x40
> m_key_violations........0x0
> p_key_violations........0x0
> q_key_violations........0x0
> guid_cap................0x0
> client_reregister.......0x0
> subnet_timeout..........0x0
> resp_time_value.........0x0
> error_threshold.........0x88
> Mar 20 14:30:12 042897 [41401960] -> Capabilities Mask:
> Mar 20 14:30:12 042907 [4780B960] -> SMP dump:
> base_ver................0x1
> mgmt_class..............0x81
> class_ver...............0x1
> method..................0x81 (SubnGetResp)
> D bit...................0x1
> status..................0x1C00
> hop_ptr.................0x0
> hop_count...............0x4
> trans_id................0x92b3a
> attr_id.................0x15 (PortInfo)
> resv....................0x0
> attr_mod................0x17
> m_key...................0x0000000000000000
> dr_slid.................0xFFFF
> dr_dlid.................0xFFFF
>
> Initial path: [0][1][16][1][5]
> Return path: [0][9][13][D][2]
> Reserved: [0][0][0][0][0][0][0]
>
> 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
>
> 00 00 00 00 00 00 00 00 00 00 00 00 02 03 03 02
>
> 14 52 00 11 40 40 00 08 08 04 F2 40 00 00 00 00
>
> 00 00 00 00 00 88 00 00 00 00 00 00 00 00 00 00
>
> Mar 20 14:30:12 043013 [43204960] -> osm_pi_rcv_process_set: Received error
> status 0x1c for SetResp() during ACTIVE transition
> Mar 20 14:30:12 043015 [4780B960] -> __osm_sm_mad_ctrl_rcv_callback: ERR
> 3111: Error status = 0x1C00
> Mar 20 14:30:12 043038 [43204960] -> PortInfo dump:
> port number.............0x17
> node_guid...............0x0005ad00000281a7
> port_guid...............0x0005ad00000281a7
> m_key...................0x0000000000000000
> subnet_prefix...........0x0000000000000000
> base_lid................0x0
> master_sm_base_lid......0x0
> capability_mask.........0x0
> diag_code...............0x0
> m_key_lease_period......0x0
> local_port_num..........0x2
> link_width_enabled......0x3
> link_width_supported....0x3
> link_width_active.......0x2
> link_speed_supported....0x1
> port_state..............ACTIVE
> state_info2.............0x52
> m_key_protect_bits......0x0
> lmc.....................0x0
> link_speed..............0x11
> mtu_smsl................0x40
> vl_cap_init_type........0x40
> vl_high_limit...........0x0
> vl_arb_high_cap.........0x8
> vl_arb_low_cap..........0x8
> init_rep_mtu_cap........0x4
> vl_stall_life...........0xF2
> vl_enforce..............0x40
> m_key_violations........0x0
> p_key_violations........0x0
> q_key_violations........0x0
> guid_cap................0x0
> client_reregister.......0x0
> subnet_timeout..........0x0
> resp_time_value.........0x0
> error_threshold.........0x88
> Mar 20 14:30:12 043090 [43204960] -> Capabilities Mask:
> Mar 20 14:30:12 043094 [4780B960] -> SMP dump:
> base_ver................0x1
> mgmt_class..............0x81
> class_ver...............0x1
> method..................0x81 (SubnGetResp)
> D bit...................0x1
> status..................0x1C00
> hop_ptr.................0x0
> hop_count...............0x4
> trans_id................0x92b3b
> attr_id.................0x15 (PortInfo)
> resv....................0x0
> attr_mod................0x18
> m_key...................0x0000000000000000
> dr_slid.................0xFFFF
> dr_dlid.................0xFFFF
>
> Initial path: [0][1][16][1][5]
> Return path: [0][9][13][D][2]
> Reserved: [0][0][0][0][0][0][0]
>
> 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
>
> 00 00 00 00 00 00 00 00 00 00 00 00 02 03 03 02
>
> 14 52 00 11 40 40 00 08 08 04 F2 40 00 00 00 00
>
> 00 00 00 00 00 88 00 00 00 00 00 00 00 00 00 00
>
> Mar 20 14:30:12 043173 [44606960] -> osm_pi_rcv_process_set: Received error
> status 0x1c for SetResp() during ACTIVE transition
> Mar 20 14:30:12 043178 [44606960] -> PortInfo dump:
> port number.............0x18
> node_guid...............0x0005ad00000281a7
> port_guid...............0x0005ad00000281a7
> m_key...................0x0000000000000000
> subnet_prefix...........0x0000000000000000
> base_lid................0x0
> master_sm_base_lid......0x0
> capability_mask.........0x0
> diag_code...............0x0
> m_key_lease_period......0x0
> local_port_num..........0x2
> link_width_enabled......0x3
> link_width_supported....0x3
> link_width_active.......0x2
> link_speed_supported....0x1
> port_state..............ACTIVE
> state_info2.............0x52
> m_key_protect_bits......0x0
> lmc.....................0x0
> link_speed..............0x11
> mtu_smsl................0x40
> vl_cap_init_type........0x40
> vl_high_limit...........0x0
> vl_arb_high_cap.........0x8
> vl_arb_low_cap..........0x8
> init_rep_mtu_cap........0x4
> vl_stall_life...........0xF2
> vl_enforce..............0x40
> m_key_violations........0x0
> p_key_violations........0x0
> q_key_violations........0x0
> guid_cap................0x0
> client_reregister.......0x0
> subnet_timeout..........0x0
> resp_time_value.........0x0
> error_threshold.........0x88
> Mar 20 14:30:12 043191 [44606960] -> Capabilities Mask:
> Mar 20 14:30:12 043222 [4780B960] -> __osm_sm_mad_ctrl_rcv_callback: ERR
> 3111: Error status = 0x1C00
> Mar 20 14:30:12 043247 [4780B960] -> SMP dump:
> base_ver................0x1
> mgmt_class..............0x81
> class_ver...............0x1
> method..................0x81 (SubnGetResp)
> D bit...................0x1
> status..................0x1C00
> hop_ptr.................0x0
> hop_count...............0x4
> trans_id................0x92b3c
> attr_id.................0x15 (PortInfo)
> resv....................0x0
> attr_mod................0x16
> m_key...................0x0000000000000000
> dr_slid.................0xFFFF
> dr_dlid.................0xFFFF
>
> Initial path: [0][1][12][1][4]
> Return path: [0][9][14][D][1]
> Reserved: [0][0][0][0][0][0][0]
>
> 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
>
> 00 00 00 00 00 00 00 00 00 00 00 00 01 03 03 02
>
> 14 52 00 11 40 40 00 08 08 04 F2 40 00 00 00 00
>
> 00 00 00 00 00 88 00 00 00 00 00 00 00 00 00 00
>
> Mar 20 14:30:12 043318 [4780B960] -> __osm_sm_mad_ctrl_rcv_callback: ERR
> 3111: Error status = 0x1C00
> Mar 20 14:30:12 043314 [41E02960] -> osm_pi_rcv_process_set: Received error
> status 0x1c for SetResp() during ACTIVE transition
> Mar 20 14:30:12 043357 [4780B960] -> SMP dump:
> base_ver................0x1
> mgmt_class..............0x81
> class_ver...............0x1
> method..................0x81 (SubnGetResp)
> D bit...................0x1
> status..................0x1C00
> hop_ptr.................0x0
> hop_count...............0x4
> trans_id................0x92b3d
> attr_id.................0x15 (PortInfo)
> resv....................0x0
> attr_mod................0x17
> m_key...................0x0000000000000000
> dr_slid.................0xFFFF
> dr_dlid.................0xFFFF
>
> Initial path: [0][1][12][1][4]
> Return path: [0][9][14][D][1]
> Reserved: [0][0][0][0][0][0][0]
>
> 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
>
> 00 00 00 00 00 00 00 00 00 00 00 00 01 03 03 02
>
> 14 52 00 11 40 40 00 08 08 04 F2 40 00 00 00 00
>
> 00 00 00 00 00 88 00 00 00 00 00 00 00 00 00 00
>
> Mar 20 14:30:12 043367 [41E02960] -> PortInfo dump:
> port number.............0x16
> node_guid...............0x0005ad00000281b3
> port_guid...............0x0005ad00000281b3
> m_key...................0x0000000000000000
> subnet_prefix...........0x0000000000000000
> base_lid................0x0
> master_sm_base_lid......0x0
> capability_mask.........0x0
> diag_code...............0x0
> m_key_lease_period......0x0
> local_port_num..........0x1
> link_width_enabled......0x3
> link_width_supported....0x3
> link_width_active.......0x2
> link_speed_supported....0x1
> port_state..............ACTIVE
> state_info2.............0x52
> m_key_protect_bits......0x0
> lmc.....................0x0
> link_speed..............0x11
> mtu_smsl................0x40
> vl_cap_init_type........0x40
> vl_high_limit...........0x0
> vl_arb_high_cap.........0x8
> vl_arb_low_cap..........0x8
> init_rep_mtu_cap........0x4
> vl_stall_life...........0xF2
> vl_enforce..............0x40
> m_key_violations........0x0
> p_key_violations........0x0
> q_key_violations........0x0
> guid_cap................0x0
> client_reregister.......0x0
> subnet_timeout..........0x0
> resp_time_value.........0x0
> error_threshold.........0x88
> Mar 20 14:30:12 043422 [41E02960] -> Capabilities Mask:
> Mar 20 14:30:12 043513 [43C05960] -> osm_pi_rcv_process_set: Received error
> status 0x1c for SetResp() during ACTIVE transition
> Mar 20 14:30:12 043518 [4780B960] -> __osm_sm_mad_ctrl_rcv_callback: ERR
> 3111: Error status = 0x1C00
> Mar 20 14:30:12 043519 [43C05960] -> PortInfo dump:
> port number.............0x17
> node_guid...............0x0005ad00000281b3
> port_guid...............0x0005ad00000281b3
> m_key...................0x0000000000000000
> subnet_prefix...........0x0000000000000000
> base_lid................0x0
> master_sm_base_lid......0x0
> capability_mask.........0x0
> diag_code...............0x0
> m_key_lease_period......0x0
> local_port_num..........0x1
> link_width_enabled......0x3
> link_width_supported....0x3
> link_width_active.......0x2
> link_speed_supported....0x1
> port_state..............ACTIVE
> state_info2.............0x52
> m_key_protect_bits......0x0
> lmc.....................0x0
> link_speed..............0x11
> mtu_smsl................0x40
> vl_cap_init_type........0x40
> vl_high_limit...........0x0
> vl_arb_high_cap.........0x8
> vl_arb_low_cap..........0x8
> init_rep_mtu_cap........0x4
> vl_stall_life...........0xF2
> vl_enforce..............0x40
> m_key_violations........0x0
> p_key_violations........0x0
> q_key_violations........0x0
> guid_cap................0x0
> client_reregister.......0x0
> subnet_timeout..........0x0
> resp_time_value.........0x0
> error_threshold.........0x88
> Mar 20 14:30:12 043535 [43C05960] -> Capabilities Mask:
> Mar 20 14:30:12 043553 [4780B960] -> SMP dump:
> base_ver................0x1
> mgmt_class..............0x81
> class_ver...............0x1
> method..................0x81 (SubnGetResp)
> D bit...................0x1
> status..................0x1C00
> hop_ptr.................0x0
> hop_count...............0x4
> trans_id................0x92b3e
> attr_id.................0x15 (PortInfo)
> resv....................0x0
> attr_mod................0x18
> m_key...................0x0000000000000000
> dr_slid.................0xFFFF
> dr_dlid.................0xFFFF
>
> Initial path: [0][1][12][1][4]
> Return path: [0][9][14][D][1]
> Reserved: [0][0][0][0][0][0][0]
>
> 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
>
> 00 00 00 00 00 00 00 00 00 00 00 00 01 03 03 02
>
> 14 52 00 11 40 40 00 08 08 04 F2 40 00 00 00 00
>
> 00 00 00 00 00 88 00 00 00 00 00 00 00 00 00 00
>
> Mar 20 14:30:12 043658 [42803960] -> osm_pi_rcv_process_set: Received error
> status 0x1c for SetResp() during ACTIVE transition
> Mar 20 14:30:12 043663 [42803960] -> PortInfo dump:
> port number.............0x18
> node_guid...............0x0005ad00000281b3
> port_guid...............0x0005ad00000281b3
> m_key...................0x0000000000000000
> subnet_prefix...........0x0000000000000000
> base_lid................0x0
> master_sm_base_lid......0x0
> capability_mask.........0x0
> diag_code...............0x0
> m_key_lease_period......0x0
> local_port_num..........0x1
> link_width_enabled......0x3
> link_width_supported....0x3
> link_width_active.......0x2
> link_speed_supported....0x1
> port_state..............ACTIVE
> state_info2.............0x52
> m_key_protect_bits......0x0
> lmc.....................0x0
> link_speed..............0x11
> mtu_smsl................0x40
> vl_cap_init_type........0x40
> vl_high_limit...........0x0
> vl_arb_high_cap.........0x8
> vl_arb_low_cap..........0x8
> init_rep_mtu_cap........0x4
> vl_stall_life...........0xF2
> vl_enforce..............0x40
> m_key_violations........0x0
> p_key_violations........0x0
> q_key_violations........0x0
> guid_cap................0x0
> client_reregister.......0x0
> subnet_timeout..........0x0
> resp_time_value.........0x0
> error_threshold.........0x88
> Mar 20 14:30:12 043678 [42803960] -> Capabilities Mask:
> Mar 20 14:30:12 049088 [43204960] -> SUBNET UP
> Mar 20 14:30:12 442903 [43C05960] -> osm_ucast_mgr_process: Min Hop Tables
> configured on all switches
> Mar 20 14:30:12 497312 [45007960] -> SUBNET UP
> Mar 20 14:30:27 571421 [43C05960] -> __osm_trap_rcv_process_request:
> Received Generic Notice type:0x01 num:128 Producer:2 from LID:0x0152
> TID:0x0000000000000000
> Mar 20 14:30:27 571674 [43C05960] -> osm_report_notice: Reporting Generic
> Notice type:1 num:128 from LID:0x0152
> GID:0xfe80000000000000,0x0005ad0000027c84
> Mar 20 14:30:27 782498 [45A08960] -> __osm_trap_rcv_process_request:
> Received Generic Notice type:0x01 num:128 Producer:2 from LID:0x0152
> TID:0x0000000000000001
> Mar 20 14:30:27 782616 [45A08960] -> osm_report_notice: Reporting Generic
> Notice type:1 num:128 from LID:0x0152
> GID:0xfe80000000000000,0x0005ad0000027c84
> Mar 20 14:30:27 804302 [42803960] -> __osm_trap_rcv_process_request:
> Received Generic Notice type:0x01 num:128 Producer:2 from LID:0x0152
> TID:0x0000000000000002
> Mar 20 14:30:27 805103 [42803960] -> osm_report_notice: Reporting Generic
> Notice type:1 num:128 from LID:0x0152
> GID:0xfe80000000000000,0x0005ad0000027c84
> Mar 20 14:30:27 924983 [41E02960] -> __osm_trap_rcv_process_request:
> Received Generic Notice type:0x01 num:128 Producer:2 from LID:0x0152
> TID:0x0000000000000003
> Mar 20 14:30:27 925088 [41E02960] -> osm_report_notice: Reporting Generic
> Notice type:1 num:128 from LID:0x0152
> GID:0xfe80000000000000,0x0005ad0000027c84
> Mar 20 14:30:27 934314 [43204960] -> osm_report_notice: Reporting Generic
> Notice type:3 num:65 from LID:0x0092
> GID:0xfe80000000000000,0x0005ad0000024bbb
> Mar 20 14:30:27 934327 [43204960] -> Removed port with
> GUID:0x0005ad00000249d3 LID range [0xB1,0xB1] of node:saguaro-23-2 HCA-1
> Mar 20 14:30:27 969077 [43204960] -> osm_ucast_mgr_process: Min Hop Tables
> configured on all switches
> Mar 20 14:30:28 017451 [41E02960] -> SUBNET UP
> Mar 20 14:30:28 030947 [45007960] -> __osm_trap_rcv_process_request:
> Received Generic Notice type:0x01 num:128 Producer:2 from LID:0x0152
> TID:0x0000000000000004
> Mar 20 14:30:28 031177 [45007960] -> osm_report_notice: Reporting Generic
> Notice type:1 num:128 from LID:0x0152
> GID:0xfe80000000000000,0x0005ad0000027c84
> Mar 20 14:30:28 120040 [42803960] -> __osm_trap_rcv_process_request:
> Received Generic Notice type:0x01 num:128 Producer:2 from LID:0x0152
> TID:0x0000000000000005
> Mar 20 14:30:28 120190 [42803960] -> osm_report_notice: Reporting Generic
> Notice type:1 num:128 from LID:0x0152
> GID:0xfe80000000000000,0x0005ad0000027c84
> Mar 20 14:30:28 148805 [41E02960] -> __osm_trap_rcv_process_request:
> Received Generic Notice type:0x01 num:128 Producer:2 from LID:0x0152
> TID:0x0000000000000006
> Mar 20 14:30:28 149108 [41E02960] -> osm_report_notice: Reporting Generic
> Notice type:1 num:128 from LID:0x0152
> GID:0xfe80000000000000,0x0005ad0000027c84
> Mar 20 14:30:28 170453 [41401960] -> __osm_trap_rcv_process_request:
> Received Generic Notice type:0x01 num:128 Producer:2 from LID:0x0152
> TID:0x0000000000000007
> Mar 20 14:30:28 170971 [41401960] -> osm_report_notice: Reporting Generic
> Notice type:1 num:128 from LID:0x0152
> GID:0xfe80000000000000,0x0005ad0000027c84
> Mar 20 14:30:28 336861 [43C05960] -> osm_report_notice: Reporting Generic
> Notice type:3 num:65 from LID:0x0092
> GID:0xfe80000000000000,0x0005ad0000024bbb
> Mar 20 14:30:28 336884 [43C05960] -> Removed port with
> GUID:0x0005ad0000024e0b LID range [0xB3,0xB3] of node:saguaro-23-4 HCA-1
> Mar 20 14:30:28 336910 [43C05960] -> osm_report_notice: Reporting Generic
> Notice type:3 num:65 from LID:0x0092
> GID:0xfe80000000000000,0x0005ad0000024bbb
> Mar 20 14:30:28 336916 [43C05960] -> Removed port with
> GUID:0x0005ad0000025043 LID range [0xB4,0xB4] of node:saguaro-23-5 HCA-1
> Mar 20 14:30:28 336945 [43C05960] -> osm_report_notice: Reporting Generic
> Notice type:3 num:65 from LID:0x0092
> GID:0xfe80000000000000,0x0005ad0000024bbb
> Mar 20 14:30:28 336951 [43C05960] -> Removed port with
> GUID:0x0005ad0000024cbb LID range [0xB2,0xB2] of node:saguaro-23-3 HCA-1
> Mar 20 14:30:28 371497 [43C05960] -> osm_ucast_mgr_process: Min Hop Tables
> configured on all switches
> Mar 20 14:30:28 410709 [41401960] -> __osm_trap_rcv_process_request:
> Received Generic Notice type:0x01 num:128 Producer:2 from LID:0x0152
> TID:0x0000000000000008
> Mar 20 14:30:28 410894 [41401960] -> osm_report_notice: Reporting Generic
> Notice type:1 num:128 from LID:0x0152
> GID:0xfe80000000000000,0x0005ad0000027c84
> Mar 20 14:30:28 415926 [43C05960] -> __osm_trap_rcv_process_request:
> Received Generic Notice type:0x01 num:128 Producer:2 from LID:0x0152
> TID:0x0000000000000009
> Mar 20 14:30:28 419624 [43C05960] -> osm_report_notice: Reporting Generic
> Notice type:1 num:128 from LID:0x0152
> GID:0xfe80000000000000,0x0005ad0000027c84
> Mar 20 14:30:28 426978 [45A08960] -> SUBNET UP
> Mar 20 14:30:28 438003 [41E02960] -> __osm_trap_rcv_process_request:
> Received Generic Notice type:0x01 num:128 Producer:2 from LID:0x0152
> TID:0x000000000000000a
> Mar 20 14:30:28 438182 [41E02960] -> osm_report_notice: Reporting Generic
> Notice type:1 num:128 from LID:0x0152
> GID:0xfe80000000000000,0x0005ad0000027c84
> Mar 20 14:30:28 470141 [41E02960] -> __osm_trap_rcv_process_request:
> Received Generic Notice type:0x01 num:128 Producer:2 from LID:0x0152
> TID:0x000000000000000b
> Mar 20 14:30:28 470197 [41E02960] -> __osm_trap_rcv_process_request: ERR
> 3804: Received trap 11 times consecutively
> Mar 20 14:30:28 652535 [42803960] -> __osm_trap_rcv_process_request:
> Received Generic Notice type:0x01 num:128 Producer:2 from LID:0x0152
> TID:0x000000000000000c
> Mar 20 14:30:28 652623 [42803960] -> __osm_trap_rcv_process_request: ERR
> 3804: Received trap 12 times consecutively
> Mar 20 14:30:28 681514 [43C05960] -> __osm_trap_rcv_process_request:
> Received Generic Notice type:0x01 num:128 Producer:2 from LID:0x0152
> TID:0x000000000000000d
> Mar 20 14:30:28 681636 [43C05960] -> __osm_trap_rcv_process_request: ERR
> 3804: Received trap 13 times consecutively
> Mar 20 14:30:28 703052 [44606960] -> __osm_trap_rcv_process_request:
> Received Generic Notice type:0x01 num:128 Producer:2 from LID:0x0152
> TID:0x000000000000000e
> Mar 20 14:30:28 703092 [44606960] -> __osm_trap_rcv_process_request: ERR
> 3804: Received trap 14 times consecutively
> Mar 20 14:30:28 724753 [43C05960] -> __osm_trap_rcv_process_request:
> Received Generic Notice type:0x01 num:128 Producer:2 from LID:0x0152
> TID:0x000000000000000f
> Mar 20 14:30:28 724809 [43C05960] -> __osm_trap_rcv_process_request: ERR
> 3804: Received trap 15 times consecutively
> Mar 20 14:30:28 855519 [42803960] -> __osm_trap_rcv_process_request:
> Received Generic Notice type:0x01 num:128 Producer:2 from LID:0x0152
> TID:0x0000000000000010
> Mar 20 14:30:28 855671 [42803960] -> __osm_trap_rcv_process_request: ERR
> 3804: Received trap 16 times consecutively
> Mar 20 14:30:28 877289 [45A08960] -> __osm_trap_rcv_process_request:
> Received Generic Notice type:0x01 num:128 Producer:2 from LID:0x0152
> TID:0x0000000000000011
> Mar 20 14:30:28 877354 [45A08960] -> __osm_trap_rcv_process_request: ERR
> 3804: Received trap 17 times consecutively
> Mar 20 14:30:28 899021 [45A08960] -> __osm_trap_rcv_process_request:
> Received Generic Notice type:0x01 num:128 Producer:2 from LID:0x0152
> TID:0x0000000000000012
> Mar 20 14:30:28 899062 [45A08960] -> __osm_trap_rcv_process_request: ERR
> 3804: Received trap 18 times consecutively
> Mar 20 14:30:29 006886 [45007960] -> __osm_trap_rcv_process_request:
> Received Generic Notice type:0x01 num:128 Producer:2 from LID:0x0152
> TID:0x0000000000000013
> Mar 20 14:30:29 006950 [45007960] -> __osm_trap_rcv_process_request: ERR
> 3804: Received trap 19 times consecutively
> Mar 20 14:30:29 099965 [44606960] -> __osm_trap_rcv_process_request:
> Received Generic Notice type:0x01 num:128 Producer:2 from LID:0x0152
> TID:0x0000000000000014
> Mar 20 14:30:29 100020 [44606960] -> __osm_trap_rcv_process_request: ERR
> 3804: Received trap 20 times consecutively
> Mar 20 14:30:29 146532 [41E02960] -> __osm_trap_rcv_process_request:
> Received Generic Notice type:0x01 num:128 Producer:2 from LID:0x0152
> TID:0x0000000000000015
> Mar 20 14:30:29 146578 [41E02960] -> __osm_trap_rcv_process_request: ERR
> 3804: Received trap 21 times consecutively
> Mar 20 14:30:29 356891 [43C05960] -> __osm_trap_rcv_process_request:
> Received Generic Notice type:0x01 num:128 Producer:2 from LID:0x0152
> TID:0x0000000000000016
> Mar 20 14:30:29 356938 [43C05960] -> __osm_trap_rcv_process_request: ERR
> 3804: Received trap 22 times consecutively
> Mar 20 14:30:29 383112 [43204960] -> __osm_trap_rcv_process_request:
> Received Generic Notice type:0x01 num:128 Producer:2 from LID:0x0152
> TID:0x0000000000000017
> Mar 20 14:30:29 383157 [43204960] -> __osm_trap_rcv_process_request: ERR
> 3804: Received trap 23 times consecutively
> Mar 20 14:30:29 383710 [41401960] -> __osm_trap_rcv_process_request:
> Received Generic Notice type:0x01 num:128 Producer:2 from LID:0x0148
> TID:0x0000000000000079
> Mar 20 14:30:29 383790 [41401960] -> osm_report_notice: Reporting Generic
> Notice type:1 num:128 from LID:0x0148
> GID:0xfe80000000000000,0x0005ad00000281b3
> Mar 20 14:30:29 407890 [42803960] -> __osm_trap_rcv_process_request:
> Received Generic Notice type:0x01 num:128 Producer:2 from LID:0x0152
> TID:0x0000000000000018
> Mar 20 14:30:29 407935 [42803960] -> __osm_trap_rcv_process_request: ERR
> 3804: Received trap 24 times consecutively
> Mar 20 14:30:29 429653 [45A08960] -> __osm_trap_rcv_process_request:
> Received Generic Notice type:0x01 num:128 Producer:2 from LID:0x0152
> TID:0x0000000000000019
> Mar 20 14:30:29 429700 [45A08960] -> __osm_trap_rcv_process_request: ERR
> 3804: Received trap 25 times consecutively
> Mar 20 14:30:29 451352 [45007960] -> __osm_trap_rcv_process_request:
> Received Generic Notice type:0x01 num:128 Producer:2 from LID:0x0152
> TID:0x000000000000001a
> Mar 20 14:30:29 451401 [45007960] -> __osm_trap_rcv_process_request: ERR
> 3804: Received trap 26 times consecutively
> Mar 20 14:30:29 479843 [4780B960] -> umad_receiver: ERR 5409: send completed
> with error (method=0x1 attr=0x11 trans_id=0x124ef00095cbf) -- dropping
> Mar 20 14:30:29 479855 [4780B960] -> umad_receiver: ERR 5411: DR SMP
> Mar 20 14:30:29 479865 [4780B960] -> __osm_sm_mad_ctrl_send_err_cb: ERR
> 3113: MAD completed in error (IB_TIMEOUT)
> Mar 20 14:30:29 479901 [4780B960] -> SMP dump:
> base_ver................0x1
> mgmt_class..............0x81
> class_ver...............0x1
> method..................0x1 (SubnGet)
> D bit...................0x0
> status..................0x0
> hop_ptr.................0x0
> hop_count...............0x6
> trans_id................0x95cbf
> attr_id.................0x11 (NodeInfo)
> resv....................0x0
> attr_mod................0x0
> m_key...................0x0000000000000000
> dr_slid.................0xFFFF
> dr_dlid.................0xFFFF
>
> Initial path: [0][1][11][1][5][17][C]
> Return path: [0][0][0][0][0][0][0]
> Reserved: [0][0][0][0][0][0][0]
>
> 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
>
> 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
>
> 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
>
> 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
>
> Mar 20 14:30:29 480017 [44606960] -> osm_report_notice: Reporting Generic
> Notice type:3 num:65 from LID:0x0092
> GID:0xfe80000000000000,0x0005ad0000024bbb
> Mar 20 14:30:29 480030 [44606960] -> Removed port with
> GUID:0x0005ad000002510b LID range [0xB5,0xB5] of node:saguaro-23-6 HCA-1
> Mar 20 14:30:29 480092 [44606960] -> osm_report_notice: Reporting Generic
> Notice type:3 num:65 from LID:0x0092
> GID:0xfe80000000000000,0x0005ad0000024bbb
> Mar 20 14:30:29 480099 [44606960] -> Removed port with
> GUID:0x0005ad000002511b LID range [0xA6,0xA6] of node:saguaro-22-1 HCA-1
> Mar 20 14:30:29 480121 [44606960] -> osm_report_notice: Reporting Generic
> Notice type:3 num:65 from LID:0x0092
> GID:0xfe80000000000000,0x0005ad0000024bbb
> Mar 20 14:30:29 480128 [44606960] -> Removed port with
> GUID:0x0005ad0000024d47 LID range [0xB6,0xB6] of node:saguaro-23-7 HCA-1
> Mar 20 14:30:29 480152 [44606960] -> osm_report_notice: Reporting Generic
> Notice type:3 num:65 from LID:0x0092
> GID:0xfe80000000000000,0x0005ad0000024bbb
> Mar 20 14:30:29 480160 [44606960] -> Removed port with
> GUID:0x0005ad0000024d8b LID range [0xB7,0xB7] of node:saguaro-23-8 HCA-1
> Mar 20 14:30:29 480325 [44606960] -> osm_drop_mgr_process: ERR 0108: Unknown
> remote side for node 0x0005ad0000027c84 port 12. Adding to light sweep
> sampling list
> Mar 20 14:30:29 480343 [44606960] -> Directed Path Dump of 5 hop path:
> Path = [0][1][11][1][5][17]
> Mar 20 14:30:29 665327 [41E02960] -> __osm_trap_rcv_process_request:
> Received Generic Notice type:0x01 num:128 Producer:2 from LID:0x0148
> TID:0x000000000000007a
> Mar 20 14:30:29 665355 [43C05960] -> __osm_trap_rcv_process_request:
> Received Generic Notice type:0x01 num:128 Producer:2 from LID:0x0152
> TID:0x000000000000001b
> Mar 20 14:30:29 665397 [41E02960] -> osm_report_notice: Reporting Generic
> Notice type:1 num:128 from LID:0x0148
> GID:0xfe80000000000000,0x0005ad00000281b3
> Mar 20 14:30:29 665404 [43C05960] -> __osm_trap_rcv_process_request: ERR
> 3804: Received trap 27 times consecutively
> Mar 20 14:30:29 680658 [45A08960] -> osm_report_notice: Reporting Generic
> Notice type:3 num:64 from LID:0x0092
> GID:0xfe80000000000000,0x0005ad0000024bbb
> Mar 20 14:30:29 680668 [45A08960] -> Discovered new port with
> GUID:0x0005ad00000249d3 LID range [0xB1,0xB1] of node:saguaro-23-2 HCA-1
> Mar 20 14:30:29 680672 [45A08960] -> osm_report_notice: Reporting Generic
> Notice type:3 num:64 from LID:0x0092
> GID:0xfe80000000000000,0x0005ad0000024bbb
> Mar 20 14:30:29 680678 [45A08960] -> Discovered new port with
> GUID:0x0005ad0000024cbb LID range [0xB2,0xB2] of node:saguaro-23-3 HCA-1
> Mar 20 14:30:29 680681 [45A08960] -> osm_report_notice: Reporting Generic
> Notice type:3 num:64 from LID:0x0092
> GID:0xfe80000000000000,0x0005ad0000024bbb
> Mar 20 14:30:29 680686 [45A08960] -> Discovered new port with
> GUID:0x0005ad0000024e0b LID range [0xB3,0xB3] of node:saguaro-23-4 HCA-1
> Mar 20 14:30:29 680690 [45A08960] -> osm_report_notice: Reporting Generic
> Notice type:3 num:64 from LID:0x0092
> GID:0xfe80000000000000,0x0005ad0000024bbb
> Mar 20 14:30:29 680695 [45A08960] -> Discovered new port with
> GUID:0x0005ad0000025043 LID range [0xB4,0xB4] of node:saguaro-23-5 HCA-1
> Mar 20 14:30:29 711542 [45A08960] -> osm_ucast_mgr_process: Min Hop Tables
> configured on all switches
> Mar 20 14:30:29 768280 [41401960] -> SUBNET UP
> Mar 20 14:30:30 113195 [45A08960] -> osm_report_notice: Reporting Generic
> Notice type:3 num:64 from LID:0x0092
> GID:0xfe80000000000000,0x0005ad0000024bbb
> Mar 20 14:30:30 113206 [45A08960] -> Discovered new port with
> GUID:0x0005ad000002510b LID range [0xB5,0xB5] of node:saguaro-23-6 HCA-1
> Mar 20 14:30:30 113211 [45A08960] -> osm_report_notice: Reporting Generic
> Notice type:3 num:64 from LID:0x0092
> GID:0xfe80000000000000,0x0005ad0000024bbb
> Mar 20 14:30:30 113216 [45A08960] -> Discovered new port with
> GUID:0x0005ad0000024d47 LID range [0xB6,0xB6] of node:saguaro-23-7 HCA-1
> Mar 20 14:30:30 113220 [45A08960] -> osm_report_notice: Reporting Generic
> Notice type:3 num:64 from LID:0x0092
> GID:0xfe80000000000000,0x0005ad0000024bbb
> Mar 20 14:30:30 113225 [45A08960] -> Discovered new port with
> GUID:0x0005ad0000024d8b LID range [0xB7,0xB7] of node:saguaro-23-8 HCA-1
> Mar 20 14:30:30 113228 [45A08960] -> osm_report_notice: Reporting Generic
> Notice type:3 num:64 from LID:0x0092
> GID:0xfe80000000000000,0x0005ad0000024bbb
> Mar 20 14:30:30 113233 [45A08960] -> Discovered new port with
> GUID:0x0005ad000002511b LID range [0xA6,0xA6] of node:saguaro-22-1 HCA-1
> Mar 20 14:30:30 144149 [45A08960] -> osm_ucast_mgr_process: Min Hop Tables
> configured on all switches
> Mar 20 14:30:30 195765 [4780B960] -> __osm_sm_mad_ctrl_rcv_callback: ERR
> 3111: Error status = 0x1C00
> Mar 20 14:30:30 195850 [4780B960] -> SMP dump:
> base_ver................0x1
> mgmt_class..............0x81
> class_ver...............0x1
> method..................0x81 (SubnGetResp)
> D bit...................0x1
> status..................0x1C00
> hop_ptr.................0x0
> hop_count...............0x4
> trans_id................0x96dcd
> attr_id.................0x15 (PortInfo)
> resv....................0x0
> attr_mod................0x16
> m_key...................0x0000000000000000
> dr_slid.................0xFFFF
> dr_dlid.................0xFFFF
>
> Initial path: [0][1][14][2][4]
> Return path: [0][9][15][E][1]
> Reserved: [0][0][0][0][0][0][0]
>
> 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
>
> 00 00 00 00 00 00 00 00 00 00 00 00 01 03 03 02
>
> 14 52 00 11 40 40 00 08 08 04 F2 40 00 00 00 00
>
> 00 00 00 00 00 88 00 00 00 00 00 00 00 00 00 00
>
> Mar 20 14:30:30 195929 [43C05960] -> osm_pi_rcv_process_set: Received error
> status 0x1c for SetResp() during ACTIVE transition
> Mar 20 14:30:30 195942 [43C05960] -> PortInfo dump:
> port number.............0x16
> node_guid...............0x0005ad00000281b3
> port_guid...............0x0005ad00000281b3
> m_key...................0x0000000000000000
> subnet_prefix...........0x0000000000000000
> base_lid................0x0
> master_sm_base_lid......0x0
> capability_mask.........0x0
> diag_code...............0x0
> m_key_lease_period......0x0
> local_port_num..........0x1
> link_width_enabled......0x3
> link_width_supported....0x3
> link_width_active.......0x2
> link_speed_supported....0x1
> port_state..............ACTIVE
> state_info2.............0x52
> m_key_protect_bits......0x0
> lmc.....................0x0
> link_speed..............0x11
> mtu_smsl................0x40
> vl_cap_init_type........0x40
> vl_high_limit...........0x0
> vl_arb_high_cap.........0x8
> vl_arb_low_cap..........0x8
> init_rep_mtu_cap........0x4
> vl_stall_life...........0xF2
> vl_enforce..............0x40
> m_key_violations........0x0
> p_key_violations........0x0
> q_key_violations........0x0
> guid_cap................0x0
> client_reregister.......0x0
> subnet_timeout..........0x0
> resp_time_value.........0x0
> error_threshold.........0x88
> Mar 20 14:30:30 195968 [43C05960] -> Capabilities Mask:
> Mar 20 14:30:30 196144 [4780B960] -> __osm_sm_mad_ctrl_rcv_callback: ERR
> 3111: Error status = 0x1C00
> Mar 20 14:30:30 196179 [4780B960] -> SMP dump:
> base_ver................0x1
> mgmt_class..............0x81
> class_ver...............0x1
> method..................0x81 (SubnGetResp)
> D bit...................0x1
> status..................0x1C00
> hop_ptr.................0x0
> hop_count...............0x4
> trans_id................0x96dce
> attr_id.................0x15 (PortInfo)
> resv....................0x0
> attr_mod................0x17
> m_key...................0x0000000000000000
> dr_slid.................0xFFFF
> dr_dlid.................0xFFFF
>
> Initial path: [0][1][14][2][4]
> Return path: [0][9][15][E][1]
> Reserved: [0][0][0][0][0][0][0]
>
> 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
>
> 00 00 00 00 00 00 00 00 00 00 00 00 01 03 03 02
>
> 14 52 00 11 40 40 00 08 08 04 F2 40 00 00 00 00
>
> 00 00 00 00 00 88 00 00 00 00 00 00 00 00 00 00
>
> Mar 20 14:30:30 196248 [43204960] -> osm_pi_rcv_process_set: Received error
> status 0x1c for SetResp() during ACTIVE transition
> Mar 20 14:30:30 196254 [43204960] -> PortInfo dump:
> port number.............0x17
> node_guid...............0x0005ad00000281b3
> port_guid...............0x0005ad00000281b3
> m_key...................0x0000000000000000
> subnet_prefix...........0x0000000000000000
> base_lid................0x0
> master_sm_base_lid......0x0
> capability_mask.........0x0
> diag_code...............0x0
> m_key_lease_period......0x0
> local_port_num..........0x1
> link_width_enabled......0x3
> link_width_supported....0x3
> link_width_active.......0x2
> link_speed_supported....0x1
> port_state..............ACTIVE
> state_info2.............0x52
> m_key_protect_bits......0x0
> lmc.....................0x0
> link_speed..............0x11
> mtu_smsl................0x40
> vl_cap_init_type........0x40
> vl_high_limit...........0x0
> vl_arb_high_cap.........0x8
> vl_arb_low_cap..........0x8
> init_rep_mtu_cap........0x4
> vl_stall_life...........0xF2
> vl_enforce..............0x40
> m_key_violations........0x0
> p_key_violations........0x0
> q_key_violations........0x0
> guid_cap................0x0
> client_reregister.......0x0
> subnet_timeout..........0x0
> resp_time_value.........0x0
> error_threshold.........0x88
> Mar 20 14:30:30 196269 [43204960] -> Capabilities Mask:
> Mar 20 14:30:30 201633 [45007960] -> SUBNET UP
> Mar 20 14:30:30 278051 [43C05960] -> __osm_trap_rcv_process_request:
> Received Generic Notice type:0x01 num:128 Producer:2 from LID:0x0152
> TID:0x000000000000001c
> Mar 20 14:30:30 278107 [43C05960] -> __osm_trap_rcv_process_request: ERR
> 3804: Received trap 28 times consecutively
> Mar 20 14:30:30 278656 [41E02960] -> __osm_trap_rcv_process_request:
> Received Generic Notice type:0x01 num:128 Producer:2 from LID:0x0148
> TID:0x000000000000007b
> Mar 20 14:30:30 278871 [41E02960] -> osm_report_notice: Reporting Generic
> Notice type:1 num:128 from LID:0x0148
> GID:0xfe80000000000000,0x0005ad00000281b3
> Mar 20 14:30:30 279653 [45007960] -> __osm_trap_rcv_process_request:
> Received Generic Notice type:0x01 num:128 Producer:2 from LID:0x001B
> TID:0x000000000000008d
> Mar 20 14:30:30 279765 [45007960] -> osm_report_notice: Reporting Generic
> Notice type:1 num:128 from LID:0x001B
> GID:0xfe80000000000000,0x0005ad00000281a7
> Mar 20 14:30:30 568539 [43C05960] -> __osm_trap_rcv_process_request:
> Received Generic Notice type:0x01 num:128 Producer:2 from LID:0x001B
> TID:0x000000000000008e
> Mar 20 14:30:30 568617 [43C05960] -> osm_report_notice: Reporting Generic
> Notice type:1 num:128 from LID:0x001B
> GID:0xfe80000000000000,0x0005ad00000281a7
> Mar 20 14:30:30 607916 [45A08960] -> __osm_trap_rcv_process_request:
> Received Generic Notice type:0x01 num:128 Producer:2 from LID:0x0148
> TID:0x000000000000007c
> Mar 20 14:30:30 625139 [44606960] -> osm_ucast_mgr_process: Min Hop Tables
> configured on all switches
> Mar 20 14:30:30 663838 [45A08960] -> osm_report_notice: Reporting Generic
> Notice type:1 num:128 from LID:0x0148
> GID:0xfe80000000000000,0x0005ad00000281b3
> Mar 20 14:30:30 664569 [44606960] -> __osm_trap_rcv_process_request:
> Received Generic Notice type:0x01 num:128 Producer:2 from LID:0x001B
> TID:0x000000000000008f
> Mar 20 14:30:30 664747 [44606960] -> osm_report_notice: Reporting Generic
> Notice type:1 num:128 from LID:0x001B
> GID:0xfe80000000000000,0x0005ad00000281a7
> Mar 20 14:30:30 679262 [45A08960] -> SUBNET UP
> Mar 20 14:30:30 784024 [43204960] -> __osm_trap_rcv_process_request:
> Received Generic Notice type:0x01 num:128 Producer:2 from LID:0x001B
> TID:0x0000000000000090
> Mar 20 14:30:30 784123 [43204960] -> osm_report_notice: Reporting Generic
> Notice type:1 num:128 from LID:0x001B
> GID:0xfe80000000000000,0x0005ad00000281a7
> Mar 20 14:30:30 804217 [41401960] -> __osm_trap_rcv_process_request:
> Received Generic Notice type:0x01 num:128 Producer:2 from LID:0x001B
> TID:0x0000000000000091
> Mar 20 14:30:30 807807 [41401960] -> osm_report_notice: Reporting Generic
> Notice type:1 num:128 from LID:0x001B
> GID:0xfe80000000000000,0x0005ad00000281a7
> Mar 20 14:30:30 825500 [45A08960] -> __osm_trap_rcv_process_request:
> Received Generic Notice type:0x01 num:128 Producer:2 from LID:0x001B
> TID:0x0000000000000092
> Mar 20 14:30:30 825600 [45A08960] -> osm_report_notice: Reporting Generic
> Notice type:1 num:128 from LID:0x001B
> GID:0xfe80000000000000,0x0005ad00000281a7
> Mar 20 14:30:30 988887 [43C05960] -> __osm_trap_rcv_process_request:
> Received Generic Notice type:0x01 num:128 Producer:2 from LID:0x001B
> TID:0x0000000000000093
> Mar 20 14:30:30 988978 [43C05960] -> osm_report_notice: Reporting Generic
> Notice type:1 num:128 from LID:0x001B
> GID:0xfe80000000000000,0x0005ad00000281a7
> Mar 20 14:30:31 059298 [41401960] -> osm_ucast_mgr_process: Min Hop Tables
> configured on all switches
> Mar 20 14:30:31 106840 [41E02960] -> __osm_trap_rcv_process_request:
> Received Generic Notice type:0x01 num:128 Producer:2 from LID:0x001B
> TID:0x0000000000000094
> Mar 20 14:30:31 111335 [41E02960] -> osm_report_notice: Reporting Generic
> Notice type:1 num:128 from LID:0x001B
> GID:0xfe80000000000000,0x0005ad00000281a7
> Mar 20 14:30:31 112465 [4780B960] -> __osm_sm_mad_ctrl_rcv_callback: ERR
> 3111: Error status = 0x1C00
> Mar 20 14:30:31 112497 [4780B960] -> SMP dump:
> base_ver................0x1
> mgmt_class..............0x81
> class_ver...............0x1
> method..................0x81 (SubnGetResp)
> D bit...................0x1
> status..................0x1C00
> hop_ptr.................0x0
> hop_count...............0x4
> trans_id................0x98837
> attr_id.................0x15 (PortInfo)
> resv....................0x0
> attr_mod................0x18
> m_key...................0x0000000000000000
> dr_slid.................0xFFFF
> dr_dlid.................0xFFFF
>
> Initial path: [0][1][16][1][5]
> Return path: [0][9][13][D][2]
> Reserved: [0][0][0][0][0][0][0]
>
> 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
>
> 00 00 00 00 00 00 00 00 00 00 00 00 02 03 03 02
>
> 11 42 00 11 40 40 00 08 08 04 F2 40 00 00 00 00
>
> 00 00 00 00 00 88 00 00 00 00 00 00 00 00 00 00
>
> Mar 20 14:30:31 112593 [44606960] -> osm_pi_rcv_process_set: ERR 0F10:
> Received error status for SetResp()
> Mar 20 14:30:31 112627 [44606960] -> PortInfo dump:
> port number.............0x18
> node_guid...............0x0005ad00000281a7
> port_guid...............0x0005ad00000281a7
> m_key...................0x0000000000000000
> subnet_prefix...........0x0000000000000000
> base_lid................0x0
> master_sm_base_lid......0x0
> capability_mask.........0x0
> diag_code...............0x0
> m_key_lease_period......0x0
> local_port_num..........0x2
> link_width_enabled......0x3
> link_width_supported....0x3
> link_width_active.......0x2
> link_speed_supported....0x1
> port_state..............DOWN
> state_info2.............0x42
> m_key_protect_bits......0x0
> lmc.....................0x0
> link_speed..............0x11
> mtu_smsl................0x40
> vl_cap_init_type........0x40
> vl_high_limit...........0x0
> vl_arb_high_cap.........0x8
> vl_arb_low_cap..........0x8
> init_rep_mtu_cap........0x4
> vl_stall_life...........0xF2
> vl_enforce..............0x40
> m_key_violations........0x0
> p_key_violations........0x0
> q_key_violations........0x0
> guid_cap................0x0
> client_reregister.......0x0
> subnet_timeout..........0x0
> resp_time_value.........0x0
> error_threshold.........0x88
> Mar 20 14:30:31 112673 [44606960] -> Capabilities Mask:
> Mar 20 14:30:31 113808 [4780B960] -> __osm_sm_mad_ctrl_rcv_callback: ERR
> 3111: Error status = 0x1C00
> Mar 20 14:30:31 113838 [4780B960] -> SMP dump:
> base_ver................0x1
> mgmt_class..............0x81
> class_ver...............0x1
> method..................0x81 (SubnGetResp)
> D bit...................0x1
> status..................0x1C00
> hop_ptr.................0x0
> hop_count...............0x4
> trans_id................0x9883e
> attr_id.................0x15 (PortInfo)
> resv....................0x0
> attr_mod................0x18
> m_key...................0x0000000000000000
> dr_slid.................0xFFFF
> dr_dlid.................0xFFFF
>
> Initial path: [0][1][11][1][4]
> Return path: [0][9][18][D][1]
> Reserved: [0][0][0][0][0][0][0]
>
> 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
>
> 00 00 00 00 00 00 00 00 00 00 00 00 01 03 03 02
>
> 14 52 00 11 40 40 00 08 08 04 F2 40 00 00 00 00
>
> 00 00 00 00 00 88 00 00 00 00 00 00 00 00 00 00
>
> Mar 20 14:30:31 113925 [43204960] -> osm_pi_rcv_process_set: Received error
> status 0x1c for SetResp() during ACTIVE transition
> Mar 20 14:30:31 113930 [43204960] -> PortInfo dump:
> port number.............0x18
> node_guid...............0x0005ad00000281b3
> port_guid...............0x0005ad00000281b3
> m_key...................0x0000000000000000
> subnet_prefix...........0x0000000000000000
> base_lid................0x0
> master_sm_base_lid......0x0
> capability_mask.........0x0
> diag_code...............0x0
> m_key_lease_period......0x0
> local_port_num..........0x1
> link_width_enabled......0x3
> link_width_supported....0x3
> link_width_active.......0x2
> link_speed_supported....0x1
> port_state..............ACTIVE
> state_info2.............0x52
> m_key_protect_bits......0x0
> lmc.....................0x0
> link_speed..............0x11
> mtu_smsl................0x40
> vl_cap_init_type........0x40
> vl_high_limit...........0x0
> vl_arb_high_cap.........0x8
> vl_arb_low_cap..........0x8
> init_rep_mtu_cap........0x4
> vl_stall_life...........0xF2
> vl_enforce..............0x40
> m_key_violations........0x0
> p_key_violations........0x0
> q_key_violations........0x0
> guid_cap................0x0
> client_reregister.......0x0
> subnet_timeout..........0x0
> resp_time_value.........0x0
> error_threshold.........0x88
> Mar 20 14:30:31 113946 [43204960] -> Capabilities Mask:
> Mar 20 14:30:31 119007 [43204960] -> SUBNET UP
> Mar 20 14:30:31 128758 [45007960] -> __osm_trap_rcv_process_request:
> Received Generic Notice type:0x01 num:128 Producer:2 from LID:0x001B
> TID:0x0000000000000095
> Mar 20 14:30:31 128851 [45007960] -> osm_report_notice: Reporting Generic
> Notice type:1 num:128 from LID:0x001B
> GID:0xfe80000000000000,0x0005ad00000281a7
> Mar 20 14:30:31 150370 [44606960] -> __osm_trap_rcv_process_request:
> Received Generic Notice type:0x01 num:128 Producer:2 from LID:0x001B
> TID:0x0000000000000096
> Mar 20 14:30:31 150468 [44606960] -> osm_report_notice: Reporting Generic
> Notice type:1 num:128 from LID:0x001B
> GID:0xfe80000000000000,0x0005ad00000281a7
> Mar 20 14:30:31 316422 [41401960] -> __osm_trap_rcv_process_request:
> Received Generic Notice type:0x01 num:128 Producer:2 from LID:0x0152
> TID:0x000000000000001c
> Mar 20 14:30:31 316498 [41401960] -> __osm_trap_rcv_process_request: ERR
> 3804: Received trap 29 times consecutively
>
> _______________________________________________
> general mailing list
> general at lists.openfabrics.org
> http://lists.openfabrics.org/cgi-bin/mailman/listinfo/general
>
> To unsubscribe, please visit http://openib.org/mailman/listinfo/openib-general
More information about the general
mailing list