[ofa-general] osm error messages
Douglas Fuller
Douglas.Fuller at asu.edu
Wed Mar 21 11:29:07 PDT 2007
I'm seeing some sporadic error activity from OpnSM (OFED 1.1; osm.log
below) that ay correlate with some ob failures -- I'm trying to get to the
bottom of this.
Before seeing this, I isolatedand disabled with ibportstate) what ppeared
to be a ba internal port in one of our core switches. That leads me to
suspectI have a switchmisbehaving somwhere.
Without any other intervention, things seem to check out (wth
ibdiagnet/ibchecknet). Any thought? Need any more nformatin?
Thanks,
--Doug Fuller
Mar 19 18:8:50 000354 [AB000160] -> OpenSM Rev:openib-2.0.5 OpnIBsvn
Exported revision
Mar 19 18:28:50 000466 [AB00160] -> OpenSM Rev:openib-2.0.5 OpenIB svn
Exported revision
Mar 19 18:28:50 007666 [AB000160] -> om_vendor_bind: Binding to port
0x5ad0000024bb
Mar 19 18:28:50 011279 [AB000160] ->osm_vendor_bind: Binding to port
0x5ad0000024bbb
Mar 19 18:2850 438326 [44606960] -> Entering MASTER stte
Mar 19 18:28:50 438628 [4606960] -> osm_report_notice: Reporting Generic
Notice type:3 num:66 from LID:0x0000
GID:0xfe8000000000000,0x0005ad000024bbb
Mar 19 1828:50 438661 [44606960] -> osm_report_notice: Reporting Generic
Notice type:3 num:6 from LID:0x0000
GID:0xf8000000000000,0x0005ad0000024bbb
Mar 1 18:28:50 504176 [41401960] -> osm_cast_mgr_process: Min Hop Tabes
onfigured on all switches
Mar 19 18:28:50 639453 [44606960] -> SUNET UP
Mar 19 18:28:50 853613 [1E02960 -> __osm_trap_rcv_process_reqest:
Received Generic Notice type:0x04 num:144 Producer:1 from LID:0x0092
TID:0x000000000000018
Mar 19 18:28:5 853813 [41E0960] -> osm_report_notice: Reporting Generic
Notice typ:4 num:144 from LID:0x0092
GID:0xfe8000000000000,0x0005ad0000024bbb
Mar 19 18:28:51 273470 [4460960] -> osm_ucast_mgr_process: Min HopTables
configured on all switches
Mar 19 18:28:51 33730 [43C05960] -> SUBNET UP
Mar 19 18:3033 565682 [4320490] -> __osm_trap_rcv_process_request:
Received Generic Notice type:0x1 num:128 Poducer:2 from LID:0x0001
TID:0x0000000000000019
Mar 19 18:30:33 565958 [43204960] -> sm_report_notice: Reporting Generic
Noicetype:1 num:128 from ID:0x0001
GID:0xfe80000000000000,0x005ad0000027c6
Mar 19 18:30:33 963901 [41401960] > osm_report_notice: Reporting Generic
Notice typ:3 num:64 fro LID:x0092
GID:0xfe80000000000000,0x005ad0000024bbb
Mar 19 18:30:33 963914 [41401960] -> Discovered nw port with
GUID:0x0005ad00000297b LI range [0x3,0x37] of node:saguro-14-9 HCA-1
Mar 19 18:30:33 994698 [4401960] > om_ucast_mgr_process: Min Hp Tables
configured n all switches
Mar 19 18:30:34 054763 [45A08960]-> UBNET UP
Mar 19 18:30:34 351397 [43C0560] -> __osm_trap_rcv_process_request:
Received Generi Ntice type:0x04 num:144 Producer:1 fomLID:0x0037
TID:0x000000000000000
Mar 19 18:30:34 351615 [43C05960] -> osm_report_notice Reportig Generic
otice type:4 num:144 from LID:0x0037
GID:0xfe80000000000000,0x0005ad00002497b
ar 19 18:30:34 777488 [45A08960] -> osm_ucast_mgr_process:Min Hop Tables
configured onall switces
Mar 19 18:30:34 832664 [4A08960] -> SUBNET UP
Mar 19 18:32:27 476136 [45A08960] -> __osm_trap_cv_process_request:
Received Generic Notice typ:0x01 num:128 Producer:2 from LID:0x0148
TID:0x000000000000002b
Mar 19 18:32:27 476340 [43204960] ->__osm_trap_rcv_process_request:
Reeivd Gneric Notice type:0x01 num:128 Producer:2 from LID:0x001B
TID:0x000000000000037
Mar 19 18:32:2 476389 [45A08960] -> osm_report_notice: Reporting Generic
Notice type: num:128 from LID:0x0148
GID:0xfe8000000000000,0x0005ad00000281b3
Mar 19 18:2:27 47485 [43204960] -> osm_report_ntice: Reporting Generic
Notice tye:1 num:128 from LID:0x001B
GID:0xfe80000000000000,0x0005ad0000081a7Mar 19 18:32:27 817617 [42803960] -> osm_report_notice: Reporting Generic
Notice type:3 nm:65 frm LID:0x0092
GID:0xfe80000000000000,0x005ad000024bbb
Mar 19 18:32:27 817637 [4280396] -> Remove port with
GUID:0x0005ad0000024e0b LID range [0xB3,xB3] of node:saguaro-23-4 HCA-1
Mar19 18:32:27 817655 [42803960] -> sm_report_notice: Reporting Generic
Notice type:3 num:65 from LID:0x0092
GID:0xfe80000000000000,00005ad0000024bbb
Mar 9 18:32:27 81766 [42803960] -> Remove port with
GUID:0x0005ad000002510b LID range [0xB5,0xB5] of node:saguaro-23-6 HCA-1
Mar 1 18:32:2 817694 [42803960] -> osm_report_notice: Reporting Generic
Ntice type:3 num:65 from LID:0x0092
GID:0xfe80000000000000,00005ad0000024bbb
Mar 19 18:3:7 81769 [42803960] -> Removed port wth
GUID0x0005ad000002511b LID range [0xA6,0xA6] of node:saguaro-22-1 HCA-1
Mar 19 18:3227 817716 42803960] -> osm_report_ntice: ReportingGeneric
Notice type:3 num:65 from LID:0x0092
GID:0xfe80000000000000,0x0005ad0000024bbb
Mar 19 18:32:27 81771 [42803960] -> Remved port with
GUID0x0005ad0000024b27 LID range [0xAF,0xAF] of node:sguaro-23-0 HCA-1
Mar 19 18:32:27 817738 [4280390] -> osm_report_notice: Reporting Generic
Notic type:3 num:65 from LID:0x0092
GID:0xfe80000000000000,0x0005ad0000024bbb
Mar 19 18:32:27 817743 [4203960] -> Removed ort with
GUID:0x000ad0000025043 LID range [0xB4,0xB4] of node:saguaro-23-5 HCA-1
Mar 19 18:32:27 817758 [42803960] - osm_report_notice: Reporing Generic
Notice type:3 num:65 from LID:0x0092
GID:0xfe8000000000000,0x0005ad0000024bbb
Mar 19 18:32:27 817763 [42803960 -> Removed port with
GUID:0x0005ad000024d7 LID range [0xB6,0xB6] of node:saguar-23-7 HCA-1
Mar 19 18:32:27 817780 [42803960] -> osm_rport_notice: Reporting Generic
Notice type:3 nu:65 fromLID:0x0092
GID:0xfe80000000000000,0x0005ad0000024bbb
Mar 19 18:32:27 17785 [42803960] -> Remved port with
GUID:0x0005ad0000024d6bLID range [0xB8,0xB8] of node:saguao-23-9 HCA-1
Mar 19 18:32:27 817803 [480396] -> osm_report_notce: Reporting Generic
Notice type:3 num:65 from LID:0x0092
GID:0xfe80000000000000,0x0005ad000024bbb
Mar 19 18:32:27 817808 [4283960] -> Removed por with
GUID:0x0005ad0000024977 LID rane [0xA9,0xA9] of node:saguaro-22-4 HCA1
Mar 19 18:32:27 817932 [42803960] -> osm_report_notice: Rporting Generic
Notice type:3 num:65 from LID:0x0092
GID:0xfe80000000000000,0x0005ad000024bbb
Mar 19 18:32:27 817938 [4280390] -> Removed port with
GID:0x0005ad0000027c84 LID range [0x15,0x152] of node:Topspin Switch TS20
Mar 19 18:32:27 817970 [42803960] -> osm_report_notice: Reporing Generic
Notice type:3 num:65 from LID:0x0092
GID:0xfe80000000000000,0x0005ad000024bbb
Mar 19 18:32:27 817977 [4280360] -> Removed port with
GUID0x0005ad0000024d8b LID range [0xB7,0xB7] of node:saguaro-23-8 HCA-1Mar 19 1:32:27 817992 [42803960] -> osm_report_notice: Reporting Generic
Notice tye:3 num:65 from LID:0x0092
GID:0xfe80000000000000,0x0005ad000004bbb
Mar 19 18:32:27 817997 [42803960 -> Removed port with
GUID:0x0005ad00000249f LID range [0xA8,0xA8] of node:saguaro-22-3 HCA-1
Ma 19 18:32:27 81811 [42803960] -> osm_report_notice: Reporting Generic
Notice type:3 num:65 from LID:0x0092
GID:0xe80000000000000,0x0005ad0000024bb
Mar 9 18:32:27 818016 [42803960] - Removed port with
UID:0x0005ad0000024c9b LID range [0xA7,0xA7] of node:saguaro-2-2 HCA-1
Mar 1 18:32:27 818032 [42803960] -> osm_report_notice: Reporing Generic
Notice type:3 num:65 from LID:0x0092
GID:0xfe80000000000000,0x0005ad0000024bbb
a 19 18:32:27 818037 [42803960] -> Rmoved port with
GUID:0x0005ad000004da7 LID range [0xB0,0xB0] of node:saguaro-23-1 HCA-1
Mar 19 1:32:27 818054 [4280360] -> osm_repot_notice: Reporting Generic
Notice typ:3 num:65 from LID:0x0092
GID:0xfe80000000000000,x0005ad0000024bbb
Mar19 18:32:27 818115 [42803960] -> Remoed port wth
GUID:0x0005ad0000024cbb LID range [xB2,0xB2] of node:saguaro-23-3 HCA-1
Mar 1 18:3:27 81812 [42803960] -> osm_report_notice: Reorting Generic
Notie type:3 num:65 from LID:0x0092
GID:0xfe80000000000000,0x0005ad000002bbb
Mar1918:32:27 818137 [42803960] -> Removedport with
GUID:0x0005ad00000249d3 LID range [0xB1,0B1] of node:saguaro-23-2 HCA-1
Mar 19 8:32:2 818153 [4280360] -> osm_report_notice: Reporting Generic
Notice type:3 num:65 from LID:0x0092
GID:0xfe800000000000,0x0005ad0000024bbb
Mar 19 1832:7 818158 [42803960] -> Removedpot with
GUID:0x0005ad0000024feb LID range [0x153,0x153] of node:saguaro-2-5 HCA-1
Mar 19 18:32:7818173 [42803960] -> osm_report_notic: Reporting Generic
Notice type:3 num:65 from LID:0x0092
GID:0xfe80000000000000,0x0005ad000024bbb
Mar 19 18:322 818178 [42803960 -> Removed port wih
GUID:0x0005ad0000024afb LID rnge [0xA5,0xA5] of node:saguaro-22-0 HCA-1
Mar 19 183227 851249[42803960] -> osm_ucast_mgr_procss: Min Hop Tables
configured on all sitches
Mar 19 18:32:27 898524 [43204960] -> SUBNET UP
Mar 19 8:32:2828664 [45007960] -> osm_ucast_mgr_proessMin Hop Tables
confiured on all switches
Mar 19 18:32:28 341659 [4466960] -> SUBNET UP
Mar 19 1:33:21814615 [41E0296] -> __osm_trap_rcv_process_request:
Rceived Gneric Notice type:0x01 num:128 Produce:2 from LID:0x0148
TID:0x00000000000002c
Mar 19 8:33:21 81484 [4E02960]-> osm_report_otice: Reporting Generi
Notice type:1 num:128 fom LID:0x0148
GID:0xfe8000000000000,0x0005ad0000281b3
Mar19 18:3321 820835 [41E02960] -> __osm_tap_rcv_process_request:
Received Genei Notce type:0x01 num:128 Producer:2 fro LI:0x001B
TID:0x000000000000008
Ma 19 18:33:21 82090 [41E02960] -> smreport_notice: Repoting Generic
Notice tye:1 num:128 fromLID:0x001B
GD:0xfe80000000000000,0x0005ad0000281a7
ar 19 18:33:21 82038 [41E02960] -> __osm_trap_rcv_processrequest:
Received Generic Notce tpe:0x num:128 Producer:2 from LID:00148
TID0x00000000000002d
Mar 19 18:33:21 820992 [1E060] -> osm_report_notice:Reporting Gnric
Notice type:1 num:128from LID:0x0148
GID:0xfe8000000000000,0x005ad00000281b3
Mar 19 18:3:21 826779 [4102960] -> __osm_trap_rcv_process_reqest:
Receivd Generic Notice type0x0 num:128 Prducr:2 from LID:0x001B
TID0x000000000000039
Mar 19 18:33:21 82742 [41E02960] - om_report_notice: Reporting Generc
Ntice type:1 num:128 from LID:0x001B
GID:0xfe8000000000000,0x0005a00000281a7Mar 19 18:33:22 164580 [45007960 > osm_drop_mgr_process: ERR 0108: Ukown
remote side fo node 0x0005a0000027c84 port 18. Addi to light sweep
sampling list
Mar 191:3:22 164654 [45007960] -> Direced PathDump of 5 hop path:
Path = [0][1][11][1][5][17]
Mar19 18:33:22 164712 [45007960] -> osm_op_mgrprocess: ERR 0108: Unknown
reote sde for node 0x0005ad00000281b3 port2. Adding to light swep
sampling lit
Mar 19 18:33:22164724 45007960] -> irected Path Dump of hop path:
Path = [0][1[11][1][5]
Mar 9 18:33:22 173634 [43C0960] ->osm_report_notic: Reporting Generic
Notice type:3 num:64 fromID0x0092
GID:0xfe80000000000000,0x005ad0000024bbb
Mar 19 18:33:22 17365 [43C05960] -> iscovered e port with
GUID:0x0005ad000027c4 LIDrange [0x152,0x152] of node:Toppin Switch TS120
Mar 19 18:33:22 17365 [43C05960] -> osm_reportnotice:RepotingGeneric
Notice type:3 num:64from LID:0x0092
GID:0xfe8000000000000,0x05ad0000024bbb
Mar 19 18:33:22 173662 [43C05960] -> icovered newpot with
GUID:0x005ad0000024b27 ID rage [0xAF,0xAF] o ode:saguaro-23- HCA-1
Mar 19 18:33:22 17366 [43C05960] -> osm_report_notice: Reprting Gneric
otice type: num:64 from LID:0x0092
GI0xfe80000000000000,0x0005ad0000024bbbMar19 18:33:22 173671[43C05960] -> Discovered new port with
GUID:0x005ad0000024da7 LID rage 0xB00xB0] ofnode:saguar-23-1 HCA-1
Mar 19 18:33:22 173675 [43C0596] -> osm_report_notice: Rerting Generic
Notice typ:3 num:64 fro LID:0x0092
GID:0xfe800000000000,0x0005ad0000024bbb
Mar 9 18:33:22173680 [43C05960] -> Discoveed new port withGUID:0x005ad00000249d3 LD range [0xB1,0xB1] of node:saguaro-2-2 HC1
Mar 1918:33:22 173684[43C05960] ->osm_report_notice: Reporting Generic
Notice type:3 num:6 fro LI:0x002
GID:0xfe80000000000000,0x005ad000002bb
Mar 19 18:33:22 173689 [43C05960] -> Dicovered new port with
GUI:0x0005ad000024cbb LID range [0xB,0xB] of node:saguaro-23-3 HCA-1
Mar 19 1:33:22 173693 [3C0960] ->osm_report_notice: Rporting Gneric
Notice type:3 num:64 fro LID:0x0092
GID:0xfe000000000000,0x005ad0000024bbb
Mar 19 18:33:22 173697 43C0596] -> Discoverednew port with
GUID:0x005ad000024e0b LI range [0xB3,0xB3] of nod:saguaro23-4 HCA-1
Mar 19 1833:22 173701 [43C5960] -> os_report_otice: Reporting Genric
Notice type:3 num:4 from LID:0x0092
GID:0xfe80000000000000,0x005d0000024bbb
Mr 19 1833:22 173706 [43C05960] -> Dscovered new port with
GUID:0x0005ad00025043 LID ange [0xB4,0xB4] ofnode:saguaro-23-5 HA-1
Mar 1 18:33:2 173710 43C05960] -> osm_repo_notice: Reporting Generic
Notice type:3 nm:64 from LID:0x0092
ID:0xfe800000000000,0x0005ad0000024bbb
Ma 1918:33:22 173715 [43C0596] -> Discverednew port with
GUID:0x0005ad00002510b LID range [0xB5,0xB5 of node:saguaro-23-6 HCA1
Mar 19 18:3:2 173719 [3C05960] -> osm_report_ntic: Reportin Generic
Notice type:3 num:64 from LID:0x0092
GID0xfe8000000000000,0x0005ad0000024bbb
Mar 19 18:33:22 1723 [43C05960]-> Disoveed new port wth
GUID:0x0005ad000002447 LID range [0xB6,0xB6] of node:saguaro-23-7 HCA-1Mar 1 18:33:22 173727 43C05960] -> osm_rert_notie: Reporting Generic
Notice type:3 num:64 from LID:0x0092
GID:0fe80000000000000,0x0005ad000004bbb
Mar 9 18:33:22 17373 [43C05960] - Discoverednew port wit
GUID:0x0005ad000024d8bLID range [0xB7,0xB7] of node:saguaro-23-8 HCA-1
Mr 19 18:33:22 173736 [4C05960] -> os_report_notice:Reporting Generic
Notic type:3 num:64 from LID:0x0092
GID:0xf80000000000000,0x0005ad0000024bbb
Mar 19 18:33:22 173741 43C05960] ->Dscovered ne port with
GID:0x0005ad0000024d6b LI range [0xB8,0xB8] ofnode:saguaro-23-9 HCA-1
Mar19 18:33:22 173744 [43C0960] -> osm_report_notice: Reorting Generic
otice type:3 num:64 from LID:0x0092
GI:0xfe8000000000000,0x0005ad0000024bbb
Mar 19 18:3:22 173749 [4305960] -> Discovered new prt with
GUI:0x0005d0000024afb LID rnge [0xA5,0xA5] of noe:saguaro-22-0 HCA-1
Mar 19 1833:22 13753 [43C05960] -> om_report_notice: Reporing Generic
Notice type:3 num:64 from LID:0x002
GID:0xfe8000000000000,0x0005ad00004bbb
Mar 19 18:33:22 17758 [43C0596] -> Discovered new portwith
GUID:00005ad000002511b LID rang [0xA6,0A6] f node:saguao-22-1 HCA-1
Mar 19 18:33:22 173762 [43C05960] -> osm_reort_notice: Reportin Geneic
Notice type:3 num:64 from LID:0x0092
GID:0xfe80000000000000,0x005ad000024bbb
Mar 19 18:33:22 17376 [3C0960] -> Discovered new port wihGUID:0x0005ad0000024c9b LID range [xA7,0xA7] of node:saguaro-222 HCA-1
Mar 19 8:33:22 173770 [43C05960] -> osm_report_notice Reporting Gneric
Notie type:3 nm:64 from LID:0x0092
GID:0xfe80000000000000,0x0005ad000024bbb
Mar 1918:33:22 173830 43C0590]-> Discovered new port with
ID:0x0005ad000002498f LID range [0xA,0xA8]of node:saguaro-22-3 HCA-1
Mar 9 18:33:22 173834 [43C05960] -> osm_eport_notice: Reporting Geneic
Notice type:3num:64 from LID:0x0092
GD:0xfe80000000000000,0x0005ad0000024bb
Mar 1 18:33:22 173839 [43C05960] -> Discovered new port ith
GUI:0x005ad0000024977 LID range [xA9,0A9 of node:saguaro-22-4 HCA-1
Mar 19 18:33:22 173843 [43C05960] ->osm_report_notice: Reporting GenericNotice ype: num:64 from LID:0x0092
GID:0xfe0000000000000,0x0005ad0000024bbb
Mar 9 8:33:22 173848 [43C05960] -> Discovered new port with
GUD:00005ad0000024feb LID range [0x153,x13 of node:saguaro-22-5 HCA-1
Mar 19 18:33:22 204620 [43C05960] -> osm_cast_mgr_process: Min Hop Tablesconfgued on all switches
Mar 19 18:33:22 278567 [45A08960] -> SUNET UP
Mar19 18:33:22 664286 [4141960] -> osm_ucast_mgr_process: Min Hop Table
configured on all switces
Mr 19 1833:22 734270 [45007960] -> SUBNET UP
Mar 19 1833:37 650358 [41401960] -> __osm_trap_cv_process_request
Rceived Geneic Noice type:0x01 num:128 Producer:2 from LID:0x0152
TID0x0000000000000000
Mar 19 18:33:37 65058 [41401960] -> os_report_notice Reporting Generic
Noticetype:1 num:28 from LID:0x0152GID:0xfe8000000000000,0x005ad0000027c84
Mar 19 18:33:37 927263 [45A08960] -> __osm_rap_rcv_procs_request:
Received Generic otice tye:0x01 num:128 Producer:2 fro LID:0x0152
TID:0x000000000000001
Mar 19 18:33:37 927420 [45A090] -> osm_report_notice: Reportig Geeric
Notice type:1 num:128from LID:0x0152
GD:0xfe80000000000000,0x0005ad0000027c84
Mar 19 18:3:37 95572 [4A0896] -> __osm_trap_rcv_process_rquest:
Received Generic Notice type001 num:128Producer:2 from LID:x0152
TID:0x000000000000002
Mar 1918:33:37 955657 [45A08960] -> osreprt_notice: Reporting Generic
Noticetype:1num:128 from LD:0x0152
GID:0xfe80000000000000,0x0005ad000002c84
Mar 1 18:33:37 97718 [44606960] -> _osm_tap_rcv_process_request:
Receivd Generic Notice type:0x01 nu:128 Produr:2 from LID:0x0152
TID:00000000000000003
Mar 19 18:33:3 97740 [44606960] -> osm_report_notice: Rporing Geneic
Notice type:1 num:128 frm LID:0x0152
GID:0xfe800000000000,0x0005d0000027c84
Mar 19 18:3337 999319 [41E02960] -> __osm_trap_rc_process_rquest:
Received Gneri Notice type:0x01 num:128 Producer:2 rom LID:0x052
TID:0x000000000000004
Mar 19 18:33:37 999447 [41E02960] > sm_report_notice: Reporting Generic
otice type:1 num:128 from LID:x152GID:xfe800000000000000x000ad0000027c84
Mar 19 18:33:38 045171 [4606960] -> __osm_trap_rcvprcess_request:
Received Gneric Notice type:0x0 num:128 Producer:2 frm LID:0x0152
TID:0x000000000000005
Mar 9 183:38 045271 [44606960] -> osm_report_notice: Reporting Generic
Ntice ype:1 num:18 from LID:0x05
GID:0xfe800000000000000x0005ad000027c84
Mar 19 18:33:38 06305 [432060] -> __osm_trap_rcv_process_request:
Received Generic Notice typ:0x01 nu:128 Producer:2 from ID:0x052
TID:0x000000000000006
Mar 1918:3:3 063102 [43204960] -> osm_reprt_notice: Rporting Generic
Notice type:1 num:128 from LID:0x0152ID:0xfe8000000000000,0x0005a0000027c4
ar 9 18:3338 182624 [2803960] -> __osm_trap_rcv_process_request:
Received Generic Notice type:0x01num:12 Produer2from LID:0x0152
TID:0x000000000000007
Mar 19 18:3338 18720 [4280360 -> osm_report_notice: Reporting Generic
Notice type:1 num:128 from LID:0x05
GID:xfe800000000000,0x0005ad000007c84
Mr 19 18:33:38 19435 [44606960] -> __osm_trap_rcv_process_request:
Reeived Generic Notice tpe:0x0 num:128 Prducer:2 from LID:0x012
TID:x0000000000000008
Mar 1918:33:38 194209[44606960] -> osm_reportnotice: Reporting Generic
Notic type:1 num:28 from LID:0x152
GID:0xfe000000000000,0x0005ad000007c4
Mar 1 18:33:38 379421 [43C05960] -> _om_trap_rcv_process_request:
Received Generi Notice type:0x01 num:12Producr:2 from LID:0x0152
TID:0x000000000000009
Mar 19 18:33:38 37959 [4305960] -> osm_report_otice: Reporting eneric
Ntice type:1 num:128 from LID:0x0152
GD:0xfe800000000000,0x0005ad0000027c84
Mar 19 1833:3 07685 [41401960] -> __osm_trap_rcv_rocss_request:Received GenericNotice type:0x01 num:128 Producer:2from LID:0x0152
TID:0x0000000000000a
Mar 19 18:33:38 47758 [4140190] -> osm_report_notice: eprting Generic
Notice type:1 num:128 rom LID:0x0152
GID:0xfe8000000000000,0000ad0000027c84
Mar 1 18:33:8 429658 [4A08960] -> __osm_trap_rcv_pocess_request:
Received Generic Ntice type:001 num:128 Producer:2 fm LID0x0152
TID:0x000000000000000bMa 19 8:33:8 429700 [45A08960] -> __osm_trap_rcv_process_reqest: ERR
3804: Received trap 11 ties oecutiveyMar 19 18:33:38 544177 [45007960] - __osm_trap_rcv_process_reuest:
Received Generic Notice type:0x0 num128 Prodcer:2 from LID:0x152
ID:0x000000000000000c
Mar 9 18:3338 544221 [4507960] -> __osm_trp_rv_process_request: ERR
304: Received trap 12 times consecutiely
Mar 1918:33:8 545235 [4280960] ->osm_repot_ntice:Reporting Generic
Notice type:3 num:65 from LID:0x0092
GID:0xfe800000000000,0x0005ad0000024b
Mar 19 18:3338 54247 [42803960] -> Removed port with
GUID:0x0005ad00024b27 LID range 0xAF,0xAF] of node:sauaro-23-0 HCA-1
Mar 19 18:33:3 545278 [42803960] -> osmreort_notice: Reporing Generic
Noticetype3 num:65 from LD:0x0092
GI:0xfe80000000000000,0x0005ad0000024bbb
Mar 19 18:33:38 54586 [428030] -> Removd port with
GUI:0x0005ad000024da7 LID range [0xB0,0x0] of node:saguao-23-1 HC-1
Mar 19 18:33:38 545312 42803960] ->osm_report_noice: Reporting Generic
Notice ype: num:65 from LID:0x0092
GID:0xfe800000000000,0x0005ad0000024bb
Mar 19 18:33:38 545318 [42803960] -> Reoved portwth
GUID:0x0005ad00000249d3 LID rang [xB1,0B1] of node:saguaro-23-2 HA-1
Mar 19 8:3:38 580005 [42803960 -> osm_ucast_mgr_process: Min Hop Tabes
configured on all swiches
Mar 19 18:33:38 66849[43C0590] -> SUBNET UP
Mar 19 18:33:38 648520[45A08960] -> __om_tra_rcv_process_reques:
ReceivedGeneric Notice type:0x01 num:128 Producer:2 from LID:0x015
D:0x00000000000000d
Mar 19 18:33:38 48616 [45A08960] -> __osm_trap_cv_process_requet: ERR
3804: eceived trap 13 tmes consecutiely
Mar 19 18:3338 676891[41E02960 -> __osm_trap_rcv_rocess_request:
Recived Genei Notice type:0x01 num:128 Producer:2 fo LID:0x0152
TID:0x0000000000000e
Mar 19 18:33:38 67670 [4102960] -> __osm_trap_rcv_proces_request: ERR
3804: Rceived trap 14 tes cosecutively
Mar 19 18:33:38 698797 [4460960] ->__osm_trap_rcv_prcessrequest:
Received Geneic Notice type:0x1 num:128 Producer:2frm LID:0x0152
TD:0x00000000000000f
Mr 19 18:33:8 69860 [44606960] -> __osm_trap_rcv_process_equest: ERR
3804: Receved trap 15 times consecutivey
Mar 19 18:33:38 20538 [43C05960] -> __s_trap_rcv_proces_request:
Received Generic Notce type:0x01 num:128 Poducer:2from ID:0x0152
TID:0x0000000000000010Mar 19 18:33:38 720612 [43C0960] -> __osm_trp_rcv_process_reqest: RR
3804: eceived trap 16 times onsecutively
ar 19 18:33:38 921253 [42803960] > __osm_trap_rcv_process_equest:
eceived Generic Notice type:x01 num:128 Producer:2 from LID0x012
TIDx0000000000000011
Mar 19 18:33:8 92130 [42803960] -> __osm_trap_rcv_procss_reques: ERR
3804: Recived trap 17 imes consecutively
Mar 198:33:38 97418 [43C05960] -> __osm_trap_rcv_proess_reqest:
Received Generic Notice ype:0x01 num:128 Producer:2 rom LID:0x152
TID:0x0000000000000012
a19 18:33:38 967479 [43C05960] > __os_trap_rcv_process_request: RR
3804: Received trap 18 times onsecutively
Mar 19 18:33:38 98519 [483960] -> _osm_trap_rcv_processreques:
ReceivedGeneric Notice type:0x01 num:128 Producer:2 from LID:0x015
TID:0x000000000000013
Mar 19 18:33:3 98955[2803960] -> __osm_trap_rcv_process_rquest: ERR3804: Receivedtrap 19 times consecutively
Mar 9 18:33:38 998342 [43204960] -> __osm_trap_rcv_poces_request:
ecived Generic Notice type0x01 num:128 Poducer:2 from LID:0x0152
TD:0x000000000000014
Mar 19 18:33:38 998380 [4320496] -> __osm_ap_rcv_process_request ERR
384: Received trap 20 times consecutively
ar 19 18:33:3 039293 43204960] -> __osm_trap_rcv_process_request:
Recived Gneric Notice type:0x0 num:128 Producr:2 frm LID:0x0152
TID:x0000000000000015
Mar9 18:33:39 039334 [43204960] -> __os_trap_rcv_process_requs: ERR
3804:Received trap 21 times consecutively
Mar19 18:33:39 061060 [3204960] -> __osm_trap_rcv_process_request:
Receid Generic Notice type:01 num:128 Producer:2 from LID:0x0152TID:0x000000000000016
Mar 19 18:3:3906108 [43204960] -> __osm_trap_rcv_prcess_request: ERR
3804: Reeied tra 22 times consecutivel
Mar 19 18:33:39 079032 [41E02960] -> __osm_tra_rcv_process_request:
Received eneric Notice type:0x01 num:128 Prducer2 from LID:0x0152
TID:0x000000000000017
Mar 19 18:33:39 07972 [4E02960] -> _osm_trap_rcv_proces_request: ERR
3804: Receied trap23 times consecutively
Mar 19 18:3:9 146006 [41E0960] -> osm_report_notice: Reporting Gneric
Notice ype:3 num:65 from LID:0x0092
GD:0xfe8000000000000,0x0005ad0000024bbMar 19 18:33:39 146018 [4E02960] - Removed portwith
GUID:0x005ad000002511b LID range [0xA6,xA6] of node:saguaro-2-1 HCA-1
Mar 19 18:33:39 14604 [41E02960] -> osm_eport_notice: Reporting Generic
Noticetype:3 num:65 from LID:0x0092
GID:0xfe80000000000000,0x005ad0000024bb
Mar 19 18:33:39 146050 [41E02960] -> Rmoved port with
UID:0x0005d0000024db LID range [0xB80xB8] of node:saguaro-23-9 HCA-1
Mar 19 18:33:39 146082 [41E2960] -> osm_report_notice Reporting Generic
Notic type:3 num:65 from LID:0x0092
GID:0xfe80000000000000,0x0005ad000024bb
Mar 19 18:33:39 146089 [41E02960] -> Removed port wth
GUID:00005ad0000024afb LID range 0xA,0xA5] of node:saguaro-22-0 HCA-1
Mar 19 18:33:39 150720 [4140190 -> osm_report_notice: Reporting Gneric
Notie type:3 num:64 from LID:0x0092
GID:0xfe80000000000000,0x0005ad00000bbb
Mar 19 18:33:39 150732 [41401960] -> Discovered new port with
GUI0x0005ad0000024b27 LID rage [0xAF,0xAF] of node:saguaro-23-0 HCA-1
Mar 19 18:33:39 150736 [4140160] -> osm_report_notice: Reporting Geneic
Notice ype:3 num:64 from LID:0x0092
GID:0xfe0000000000000,0x0005d000024bbb
Mar 19 18:33:39 150742 [41401960] -> Discovered new port with
GID:0x000ad0000024da LID range [0xB0,0xB0] of node:saguaro-23-1 HCA-1
Mar 19 18:33:39 150745 [4141960] -> osm_report_notice: Reporting Geneic
Notice tpe:3num:64 from LID:0x0092
GID:0xfe8000000000000,0x0005ad000024bbb
Mar 19 18:3:39 150750 [41401960] -> Discovered new port with
UID:0x0005ad0000249d3 LID range [0xB1,0xB1] of node:saguaro-23-2 HA-1
Mar 19 18:33:39 181553 [4141960] -> osm_ucast_mgr_process: Min Ho Tables
configured on al switches
Mar 19 18:33:39 218130 [43C5960] -> __osm_trap_rcv_process_request:
Received eneric Notice type:0x01 num:128 Producer:2 from LID:00152
TID:0x000000000000018
Mar 19 18:33:39 218197 [43C05960] -> __osm_trap_rcv_process_request: ERR
3804: Receivd trap 24 times consecutively
Mar 1918:33:39 375407 [42803960] -> __osm_trap_rcv_process_request:
Received Generc Notice type:0x01 um:128 Producer:2 from LID:0x0152
TID:0x000000000000019
Mar 19 18:3339 375456 [42803960] -> __osm_trap_rcv_process_request: ERR
3804: Received tra 25 times consecutvely
Mar 19 18:33:39 375588 [43C05960]-> __osm_trap_rcv_proces_request:
Received Generic Notic type:0x01 num:128 Producer:2 from LID:0x0148
TID:0x00000000000002e
Mr 19 18:33:39 375630 [43C05960] -> osm_report_notice Reporting Generic
Notice type:1 num:128 fro LID:0x0148
GID:0xfe80000000000000,0x0005ad00000281b
Mar 19 18:33:39 637844 [41401960] -> UBNET UP
Mar 19 18:33:9 664805 [45A08960] -> __osm_trap_rcv_process_request:
Received Generc Notce type:0x01 num128 Producer:2 from LID:0x0148
TID:0x000000000000002f
Mar 19 18:33:39 66490 [45A08960] -> osm_report_notice: Reporting Generic
Notice type:1 num:128 from LID:0x0148
GID:xfe800000000000000x0005ad00000281b3
ar 19 18:33:39 666276 [45A08960] -> __osm_trap_rcvprocess_request:
Received Generic Notice type:x01 num:128 Producer:2 from LID:0x001B
TID:0x00000000000003a
Mar 19 18:33:39 666364 [45A08960] -> osm_report_notice Reporting Generic
Notice type1 num:128 from LID:0x001B
GID:0xfe8000000000000,0x0005ad00000281a7Mar 19 18:33:39 710546 [41E02960] -> __osm_trap_rcv_proces_request
Received Generic Notice type:0x01 num:128 Producer2 from LID:0x014
TID:0x000000000000003
Mar 19 18:33:39 710642 [41E02960] -> osm_report_notice Reporting Generic
Notice type:1 num:28 from LID:0x0148
ID:0xfe80000000000000,0x0005ad00000281b3
Mar 19 18:33:39 732425 [41E0960] ->__osm_trap_rcvprocess_request:
Received Generic Notice type:0x01 num:128 Producer:2 from ID:0x0148
TID:0x0000000000000031
Mar 19 18:33:39732514 41E02960] -> osm_report_notice: Reporing Generic
Notice type:1num:128 from LID:0x0148
GID:0xfe8000000000000,0x0005ad00000281b3
Mar 1 18:33:39 784151 [43204960] -> __osm_trap_rcv_process_request:
ReceivedGeneric Notice type:0x01 num:128 Producer:2 from LID:0x0148
TID:000000000000032
Mar 19 18:33:39 784269 [43204960] -> osm_report_notice: Reporting neric
Notice type:1 nu:128 from LID:x0148
GID:0xfe80000000000000,0x0005ad00000281b3
Mar 19 18:33:39 824170 [42803960] -> __osm_trap_rcv_procss_request:
Received Generic Notice type:0x01 num:128 Produer:2 from LID:0x001B
TID:0x00000000000003b
Mar 19 18:33:39 824443 [42803960] -> osm_repot_notice: Reporting Generic
Notice tye:1 num:128 frm LID:0x001B
GID:0xfe80000000000000,0x0005ad00000281a7
Mar 19 18:33:40 015052 [44606960] - osm_report_notice: Reporting Generic
Notice type:3num:64 from ID:0x0092
GID:0xfe80000000000000,0x0005ad0000024bbb
Mar 9 18:33:40 015070[44606960] - Discovered new port with
GUID:0x0005ad0000024d6b LID range [0xB80xB8] of node:saguaro-23-9 HCA-1
Mar 19 18:33:40 015074 [44606960] -> osm_repot_notice: Reporting Generic
Notice type3 num:64 from LID:0x0092
GID:0fe80000000000000,0x0005ad0000024bbb
Mar 19 18:33:40 015080 [44606960] -> Discovered new port wit
GUID:0x0005ad000024afb LID range [0xA5,0xA5] of node:saguaro-22-0 HCA-1
Mar 19 18:3:40 015083 [44606960] -> osm_repor_notice: Reporting Generic
Notice type:3 num:64 from LID:0x002
GID:0xfe80000000000000,0x0005ad000002bbb
Mar 19 18:33:40 015088 [44606960] -> Discovered new port with
GUID:00005ad000002511b LID range [0xA6,0xA6] of node:saguaro-22-1 HCA-1
Mar19 18:33:40 046164 [44606960] -> osm_ucast_mgr_process: Min Hop Tables
configured on all switchesMar 19 18:33:40 106627 [42803960] -> SBNET UP
Mar 19 18:33:40 145952 [45007960] -> __osm_trap_rcv_process_request:
Received Generic Notice type:0x01 um:128 Producer:2 from LID:0x0148
TID:0x0000000000000033
Mar 19 18:3340 146076 [45007960] -> osm_report_notice: Reporting Generic
Notice type:1 num:128 from LID:0x018
GID:0xfe80000000000000,0x0005ad00000281b3
Mar 19 18:33:40 146486 [44606960] -> __osm_trap_rcv_process_request:Received Generic Notice ype:0x01 num:128 Producer:2 from LID:0x001B
ID:0x000000000000003c
Mar 19 18:33:40 146611 [44606960] -> osm_report_notice: Reporting Generic
otice type:1 num:128 from LID:0x001B
GID:0xfe80000000000000,0x0005ad00000281a7
Mar 19 18:33:40 306176 [41401960]-> __osm_trap_rcv_process_request:
Received Generic Notice type:0x01 um:128 Producer:2 from LID:0x001B
TID:0x000000000000003d
Mar 19 18:33:40 306270 [41401960] -> os_report_notice: Reporting Generic
Notice type:1 num:128 from LID:0x001B
GID:0xfe80000000000000,0x0005ad00000281a
Mar 19 18:33:40 420009 [43C05960] -> __osm_trap_rcv_process_rquest:Received Generic Notice typ:0x01 num:128 Producer:2 from LID:0x0152
TID:0x0000000000000019
Mar 918:33:4420071 [43C05960] -> __osm_trap_rcv_process_request: ERR
3804: Received trap 26 times conecutively
r 19 18:33:40 433566 [4280390] -> __osm_trap_rcv_process_request:
Reeived Geneic Notice type:0x01 num:128 Producer:2 from LID:0x0152
TID:0x00000000000001a
Mar 19 1:33:40 433596 [42803960] -> __osm_traprcv_proess_request: ERR
3804: Received trap 27 times consecutively
Mar 19 18:33:40 434996 [45007960] -> _osm_trap_cv_process_request:
Received Generic otice type:0x01 num:128 Producer:2 from LID:0x001BTID:0x000000000000003e
Mar 19 18:33:40 435041 [4500790] -> osm_reportotice: Reporting Generic
Notice ype:1 num:128 from LID:0x001B
GID:0xfe80000000000000,0x0005ad0000281a7
Mar 19 18:33:40 485454 [3204960 -> osm_ucast_mgr_procss: Mi Hop Tables
configured on all switches
Mar 19 18:33:40 528816 [43C05960] -> __osm_trap_cv_process_requet:
Received Generic Noice type:0x01 num:128 Producer:2 from LID:0x001B
TID:0x000000000000003f
Mar 19 18:33:40 528960 [43C05960] -> osm_reort_notie: Reporting Generic
Notice type:1 nu:128 from LID:0x001B
GID:0xfe80000000000000,0x0005ad00000281a7
Mar 19 18:33:40 546019 [42803960] -> SUBNT UP
Mar 19 18:33:40 551048 [42803960] -> __osm_trap_rcv_process_request:
Received Generic Notice type:0x01 num:128 Producer:2 from LID:0x0148
TID:x0000000000000034
Mar 19 18:33:40 55119 [42803960] -> osm_report_notice: Reporting Generic
Notice typ:1 num:128 from LID:0x0148
GID:0xfe8000000000000,0x0005ad00000281b3
Mar 19 18:33:40 594994 [44606960] -> __osm_trap_rcv_process_request:
Received Generic Notice type:0x01 num:128 Producer2 from LID:0x001B
TID:0x0000000000000040
Mar 19 18:33:40 595074 [44606960] -> osm_report_notice: Reporting Generic
Notice type:1 num:128 from LD:0x001B
GID:0xfe80000000000000,0x0005ad00000281a7
Mar 19 18:33:40 83973 [43204960] -> __osm_trap_rcv_process_request:
Received Generic Notice type:0x01 num:128 Prodcer:2 from LID:0x001B
TID:0x0000000000000041
Mar 19 18:33:40 840064 [43204960] -> osm_report_notice: Reporting Gneric
Notice type:1 num:128 from LID:0x001B
GID:0xfe80000000000000,0x005ad0000281a7
Mar 19 18:33:40 861973 [43204960] -> __osm_trap_rcv_process_request:
Received Genric Notice type:0x01 num:128 Producer: from LID:0x001B
TID:0x0000000000000042
Mar 19 18:33:40 862075 [43204960]-> osm_report_notice: Reporting Generic
Ntice type:1 num:128 from LID:0x001B
GID:0xfe80000000000000,0x005ad00000281a7
Mar 19 18:33:40 83777 [43204960] -> __osm_trap_rcv_process_request:
Received Generic otice type:0x01 num:128 Producr:2 from LID:0x001B
TID:0x0000000000000043
Mar 19 18:33:40 907658 [4803960] -> osm_ucast_mgr_process: Min Hop Tables
configured on all switches
Mar 19 18:33:40 947974 [43204960] -> osm_report_notice: Reporting Generic
Notice type:1 num:128 from LID:0x001B
GID:0xfe80000000000000,0x0005ad00000281a7
Mar 19 18:33:40 965203 [45007960] -> SUBNET UP
Mar 19 18:33:41 350582 [45007960] -> osm_ucast_mgr_process: Min Hop Tables
configured on all switches
Mar 19 18:33:41 417662 [43204960] -> SUBNET UP
Mar 19 18:33:41 571156 [45A08960] -> __osm_trap_rc_process_request:
Received Generic Notice type:0x01 num:128 Producer:2 from LID:0x0152
TID:0x000000000000001b
Mar 19 18:33:41 571256 [45A08960] -> __osm_trap_rcv_process_request: ERR
3804: Received trap 28 times consecutively
Mar 19 18:35:50 971684 [43C05960] -> __osm_trap_rcv_process_request:
Received Generic Notice type:0x01 num:128 Producer:2 from LID:0x0148
TID:0x0000000000000035
Mar 19 18:35:50 971926 [43C05960] -> osm_report_notice: Reporting Generic
Notice type:1 num:128 from LID:0x0148
GID:0xfe80000000000000,0x0005ad00000281b3
Mar 19 18:35:50 972301 [45007960] -> __osm_trap_rcv_process_request:
Received Generic Notice type:0x01 num:128 Producer:2 from LID:0x001B
TID:0x0000000000000044
Mar 19 18:35:50 972378 [45007960] -> osm_report_notice: Reporting Generic
Notice type:1 num:128 from LID:0x001B
GID:0xfe80000000000000,0x0005ad00000281a7
Mar 19 18:35:51 342826 [43204960] -> osm_report_notice: Reporting Generic
Notice type:3 num:65 from LID:0x0092
GID:0xfe80000000000000,0x0005ad0000024bbb
Mar 19 18:35:51 342845 [43204960] -> Removed port with
GUID:0x0005ad0000024e0b LID range [0xB3,0xB3] of node:saguaro-23-4 HCA-1
Mar 19 18:35:51 342866 [43204960] -> osm_report_notice: Reporting Generic
Notice type:3 num:65 from LID:0x0092
GID:0xfe80000000000000,0x0005ad0000024bbb
Mar 19 18:35:51 342873 [43204960] -> Removed port with
GUID:0x0005ad000002510b LID range [0xB5,0xB5] of node:saguaro-23-6 HCA-1
Mar 19 18:35:51 342895 [43204960] -> osm_report_notice: Reporting Generic
Notice type:3 num:65 from LID:0x0092
GID:0xfe80000000000000,0x0005ad0000024bbb
Mar 19 18:35:51 342901 [43204960] -> Removed port with
GUID:0x0005ad000002511b LID range [0xA6,0xA6] of node:saguaro-22-1 HCA-1
Mar 19 18:35:51 342923 [43204960] -> osm_report_notice: Reporting Generic
Notice type:3 num:65 from LID:0x0092
GID:0xfe80000000000000,0x0005ad0000024bbb
Mar 19 18:35:51 342930 [43204960] -> Removed port with
GUID:0x0005ad0000024b27 LID range [0xAF,0xAF] of node:saguaro-23-0 HCA-1
Mar 19 18:35:51 342968 [43204960] -> osm_report_notice: Reporting Generic
Notice type:3 num:65 from LID:0x0092
GID:0xfe80000000000000,0x0005ad0000024bbb
Mar 19 18:35:51 342972 [43204960] -> Removed port with
GUID:0x0005ad0000025043 LID range [0xB4,0xB4] of node:saguaro-23-5 HCA-1
Mar 19 18:35:51 342989 [43204960] -> osm_report_notice: Reporting Generic
Notice type:3 num:65 from LID:0x0092
GID:0xfe80000000000000,0x0005ad0000024bbb
Mar 19 18:35:51 342994 [43204960] -> Removed port with
GUID:0x0005ad0000024d47 LID range [0xB6,0xB6] of node:saguaro-23-7 HCA-1
Mar 19 18:35:51 343011 [43204960] -> osm_report_notice: Reporting Generic
Notice type:3 num:65 from LID:0x0092
GID:0xfe80000000000000,0x0005ad0000024bbb
Mar 19 18:35:51 343016 [43204960] -> Removed port with
GUID:0x0005ad0000024d6b LID range [0xB8,0xB8] of node:saguaro-23-9 HCA-1
Mar 19 18:35:51 343033 [43204960] -> osm_report_notice: Reporting Generic
Notice type:3 num:65 from LID:0x0092
GID:0xfe80000000000000,0x0005ad0000024bbb
Mar 19 18:35:51 343038 [43204960] -> Removed port with
GUID:0x0005ad0000024977 LID range [0xA9,0xA9] of node:saguaro-22-4 HCA-1
Mar 19 18:35:51 343189 [43204960] -> osm_report_notice: Reporting Generic
Notice type:3 num:65 from LID:0x0092
GID:0xfe80000000000000,0x0005ad0000024bbb
Mar 19 18:35:51 343194 [43204960] -> Removed port with
GUID:0x0005ad0000027c84 LID range [0x152,0x152] of node:Topspin Switch TS120
Mar 19 18:35:51 343234 [43204960] -> osm_report_notice: Reporting Generic
Notice type:3 num:65 from LID:0x0092
GID:0xfe80000000000000,0x0005ad0000024bbb
Mar 19 18:35:51 343239 [43204960] -> Removed port with
GUID:0x0005ad0000024d8b LID range [0xB7,0xB7] of node:saguaro-23-8 HCA-1
Mar 19 18:35:51 343253 [43204960] -> osm_report_notice: Reporting Generic
Notice type:3 num:65 from LID:0x0092
GID:0xfe80000000000000,0x0005ad0000024bbb
Mar 19 18:35:51 343258 [43204960] -> Removed port with
GUID:0x0005ad000002498f LID range [0xA8,0xA8] of node:saguaro-22-3 HCA-1
Mar 19 18:35:51 343273 [43204960] -> osm_report_notice: Reporting Generic
Notice type:3 num:65 from LID:0x0092
GID:0xfe80000000000000,0x0005ad0000024bbb
Mar 19 18:35:51 343278 [43204960] -> Removed port with
GUID:0x0005ad0000024c9b LID range [0xA7,0xA7] of node:saguaro-22-2 HCA-1
Mar 19 18:35:51 343293 [43204960] -> osm_report_notice: Reporting Generic
Notice type:3 num:65 from LID:0x0092
GID:0xfe80000000000000,0x0005ad0000024bbb
Mar 19 18:35:51 343298 [43204960] -> Removed port with
GUID:0x0005ad0000024da7 LID range [0xB0,0xB0] of node:saguaro-23-1 HCA-1
Mar 19 18:35:51 343314 [43204960] -> osm_report_notice: Reporting Generic
Notice type:3 num:65 from LID:0x0092
GID:0xfe80000000000000,0x0005ad0000024bbb
Mar 19 18:35:51 343319 [43204960] -> Removed port with
GUID:0x0005ad0000024cbb LID range [0xB2,0xB2] of node:saguaro-23-3 HCA-1
Mar 19 18:35:51 343334 [43204960] -> osm_report_notice: Reporting Generic
Notice type:3 num:65 from LID:0x0092
GID:0xfe80000000000000,0x0005ad0000024bbb
Mar 19 18:35:51 343393 [43204960] -> Removed port with
GUID:0x0005ad00000249d3 LID range [0xB1,0xB1] of node:saguaro-23-2 HCA-1
Mar 19 18:35:51 343410 [43204960] -> osm_report_notice: Reporting Generic
Notice type:3 num:65 from LID:0x0092
GID:0xfe80000000000000,0x0005ad0000024bbb
Mar 19 18:35:51 343415 [43204960] -> Removed port with
GUID:0x0005ad0000024feb LID range [0x153,0x153] of node:saguaro-22-5 HCA-1
Mar 19 18:35:51 343430 [43204960] -> osm_report_notice: Reporting Generic
Notice type:3 num:65 from LID:0x0092
GID:0xfe80000000000000,0x0005ad0000024bbb
Mar 19 18:35:51 343435 [43204960] -> Removed port with
GUID:0x0005ad0000024afb LID range [0xA5,0xA5] of node:saguaro-22-0 HCA-1
Mar 19 18:35:51 376525 [43204960] -> osm_ucast_mgr_process: Min Hop Tables
configured on all switches
Mar 19 18:35:51 433087 [43204960] -> SUBNET UP
Mar 19 18:35:51 849193 [44606960] -> osm_ucast_mgr_process: Min Hop Tables
configured on all switches
Mar 19 18:35:51 901399 [42803960] -> SUBNET UP
Mar 19 18:36:44 359407 [42803960] -> __osm_trap_rcv_process_request:
Received Generic Notice type:0x01 num:128 Producer:2 from LID:0x0148
TID:0x0000000000000036
Mar 19 18:36:44 359652 [42803960] -> osm_report_notice: Reporting Generic
Notice type:1 num:128 from LID:0x0148
GID:0xfe80000000000000,0x0005ad00000281b3
Mar 19 18:36:44 365352 [42803960] -> __osm_trap_rcv_process_request:
Received Generic Notice type:0x01 num:128 Producer:2 from LID:0x0148
TID:0x0000000000000037
Mar 19 18:36:44 365427 [42803960] -> osm_report_notice: Reporting Generic
Notice type:1 num:128 from LID:0x0148
GID:0xfe80000000000000,0x0005ad00000281b3
Mar 19 18:36:44 365432 [43204960] -> __osm_trap_rcv_process_request:
Received Generic Notice type:0x01 num:128 Producer:2 from LID:0x001B
TID:0x0000000000000045
Mar 19 18:36:44 365567 [43204960] -> osm_report_notice: Reporting Generic
Notice type:1 num:128 from LID:0x001B
GID:0xfe80000000000000,0x0005ad00000281a7
Mar 19 18:36:44 371481 [44606960] -> __osm_trap_rcv_process_request:
Received Generic Notice type:0x01 num:128 Producer:2 from LID:0x001B
TID:0x0000000000000046
Mar 19 18:36:44 371591 [44606960] -> osm_report_notice: Reporting Generic
Notice type:1 num:128 from LID:0x001B
GID:0xfe80000000000000,0x0005ad00000281a7
Mar 19 18:36:44 711649 [43204960] -> osm_drop_mgr_process: ERR 0108: Unknown
remote side for node 0x0005ad0000027c84 port 19. Adding to light sweep
sampling list
Mar 19 18:36:44 711691 [43204960] -> Directed Path Dump of 5 hop path:
Path = [0][1][11][1][6][18]
Mar 19 18:36:44 711738 [43204960] -> osm_drop_mgr_process: ERR 0108: Unknown
remote side for node 0x0005ad00000281b3 port 24. Adding to light sweep
sampling list
Mar 19 18:36:44 711748 [43204960] -> Directed Path Dump of 4 hop path:
Path = [0][1][11][1][6]
Mar 19 18:36:44 721719 [43204960] -> osm_report_notice: Reporting Generic
Notice type:3 num:64 from LID:0x0092
GID:0xfe80000000000000,0x0005ad0000024bbb
Mar 19 18:36:44 721730 [43204960] -> Discovered new port with
GUID:0x0005ad0000027c84 LID range [0x152,0x152] of node:Topspin Switch TS120
Mar 19 18:36:44 721736 [43204960] -> osm_report_notice: Reporting Generic
Notice type:3 num:64 from LID:0x0092
GID:0xfe80000000000000,0x0005ad0000024bbb
Mar 19 18:36:44 721744 [43204960] -> Discovered new port with
GUID:0x0005ad0000024da7 LID range [0xB0,0xB0] of node:saguaro-23-1 HCA-1
Mar 19 18:36:44 721749 [43204960] -> osm_report_notice: Reporting Generic
Notice type:3 num:64 from LID:0x0092
GID:0xfe80000000000000,0x0005ad0000024bbb
Mar 19 18:36:44 721756 [43204960] -> Discovered new port with
GUID:0x0005ad00000249d3 LID range [0xB1,0xB1] of node:saguaro-23-2 HCA-1
Mar 19 18:36:44 721761 [43204960] -> osm_report_notice: Reporting Generic
Notice type:3 num:64 from LID:0x0092
GID:0xfe80000000000000,0x0005ad0000024bbb
Mar 19 18:36:44 721767 [43204960] -> Discovered new port with
GUID:0x0005ad0000024b27 LID range [0xAF,0xAF] of node:saguaro-23-0 HCA-1
Mar 19 18:36:44 721772 [43204960] -> osm_report_notice: Reporting Generic
Notice type:3 num:64 from LID:0x0092
GID:0xfe80000000000000,0x0005ad0000024bbb
Mar 19 18:36:44 721779 [43204960] -> Discovered new port with
GUID:0x0005ad0000024cbb LID range [0xB2,0xB2] of node:saguaro-23-3 HCA-1
Mar 19 18:36:44 721784 [43204960] -> osm_report_notice: Reporting Generic
Notice type:3 num:64 from LID:0x0092
GID:0xfe80000000000000,0x0005ad0000024bbb
Mar 19 18:36:44 721790 [43204960] -> Discovered new port with
GUID:0x0005ad0000024e0b LID range [0xB3,0xB3] of node:saguaro-23-4 HCA-1
Mar 19 18:36:44 721795 [43204960] -> osm_report_notice: Reporting Generic
Notice type:3 num:64 from LID:0x0092
GID:0xfe80000000000000,0x0005ad0000024bbb
Mar 19 18:36:44 721802 [43204960] -> Discovered new port with
GUID:0x0005ad000002510b LID range [0xB5,0xB5] of node:saguaro-23-6 HCA-1
Mar 19 18:36:44 721826 [43204960] -> osm_report_notice: Reporting Generic
Notice type:3 num:64 from LID:0x0092
GID:0xfe80000000000000,0x0005ad0000024bbb
Mar 19 18:36:44 721831 [43204960] -> Discovered new port with
GUID:0x0005ad0000025043 LID range [0xB4,0xB4] of node:saguaro-23-5 HCA-1
Mar 19 18:36:44 721845 [43204960] -> osm_report_notice: Reporting Generic
Notice type:3 num:64 from LID:0x0092
GID:0xfe80000000000000,0x0005ad0000024bbb
Mar 19 18:36:44 721850 [43204960] -> Discovered new port with
GUID:0x0005ad0000024d47 LID range [0xB6,0xB6] of node:saguaro-23-7 HCA-1
Mar 19 18:36:44 721854 [43204960] -> osm_report_notice: Reporting Generic
Notice type:3 num:64 from LID:0x0092
GID:0xfe80000000000000,0x0005ad0000024bbb
Mar 19 18:36:44 721859 [43204960] -> Discovered new port with
GUID:0x0005ad0000024d8b LID range [0xB7,0xB7] of node:saguaro-23-8 HCA-1
Mar 19 18:36:44 721862 [43204960] -> osm_report_notice: Reporting Generic
Notice type:3 num:64 from LID:0x0092
GID:0xfe80000000000000,0x0005ad0000024bbb
Mar 19 18:36:44 721867 [43204960] -> Discovered new port with
GUID:0x0005ad0000024d6b LID range [0xB8,0xB8] of node:saguaro-23-9 HCA-1
Mar 19 18:36:44 721871 [43204960] -> osm_report_notice: Reporting Generic
Notice type:3 num:64 from LID:0x0092
GID:0xfe80000000000000,0x0005ad0000024bbb
Mar 19 18:36:44 721876 [43204960] -> Discovered new port with
GUID:0x0005ad0000024afb LID range [0xA5,0xA5] of node:saguaro-22-0 HCA-1
Mar 19 18:36:44 721880 [43204960] -> osm_report_notice: Reporting Generic
Notice type:3 num:64 from LID:0x0092
GID:0xfe80000000000000,0x0005ad0000024bbb
Mar 19 18:36:44 721884 [43204960] -> Discovered new port with
GUID:0x0005ad000002511b LID range [0xA6,0xA6] of node:saguaro-22-1 HCA-1
Mar 19 18:36:44 721888 [43204960] -> osm_report_notice: Reporting Generic
Notice type:3 num:64 from LID:0x0092
GID:0xfe80000000000000,0x0005ad0000024bbb
Mar 19 18:36:44 721893 [43204960] -> Discovered new port with
GUID:0x0005ad0000024c9b LID range [0xA7,0xA7] of node:saguaro-22-2 HCA-1
Mar 19 18:36:44 721897 [43204960] -> osm_report_notice: Reporting Generic
Notice type:3 num:64 from LID:0x0092
GID:0xfe80000000000000,0x0005ad0000024bbb
Mar 19 18:36:44 721923 [43204960] -> Discovered new port with
GUID:0x0005ad000002498f LID range [0xA8,0xA8] of node:saguaro-22-3 HCA-1
Mar 19 18:36:44 721927 [43204960] -> osm_report_notice: Reporting Generic
Notice type:3 num:64 from LID:0x0092
GID:0xfe80000000000000,0x0005ad0000024bbb
Mar 19 18:36:44 721932 [43204960] -> Discovered new port with
GUID:0x0005ad0000024977 LID range [0xA9,0xA9] of node:saguaro-22-4 HCA-1
Mar 19 18:36:44 721936 [43204960] -> osm_report_notice: Reporting Generic
Notice type:3 num:64 from LID:0x0092
GID:0xfe80000000000000,0x0005ad0000024bbb
Mar 19 18:36:44 721941 [43204960] -> Discovered new port with
GUID:0x0005ad0000024feb LID range [0x153,0x153] of node:saguaro-22-5 HCA-1
Mar 19 18:36:44 752683 [43204960] -> osm_ucast_mgr_process: Min Hop Tables
configured on all switches
Mar 19 18:36:44 820881 [43C05960] -> SUBNET UP
Mar 19 18:36:45 198990 [44606960] -> osm_ucast_mgr_process: Min Hop Tables
configured on all switches
Mar 19 18:36:45 258878 [44606960] -> SUBNET UP
Mar 19 18:37:00 446068 [45A08960] -> __osm_trap_rcv_process_request:
Received Generic Notice type:0x01 num:128 Producer:2 from LID:0x0152
TID:0x0000000000000000
Mar 19 18:37:00 446346 [45A08960] -> osm_report_notice: Reporting Generic
Notice type:1 num:128 from LID:0x0152
GID:0xfe80000000000000,0x0005ad0000027c84
Mar 19 18:37:00 564122 [41401960] -> __osm_trap_rcv_process_request:
Received Generic Notice type:0x01 num:128 Producer:2 from LID:0x0152
TID:0x0000000000000001
Mar 19 18:37:00 564810 [41401960] -> osm_report_notice: Reporting Generic
Notice type:1 num:128 from LID:0x0152
GID:0xfe80000000000000,0x0005ad0000027c84
Mar 19 18:37:00 589920 [45007960] -> __osm_trap_rcv_process_request:
Received Generic Notice type:0x01 num:128 Producer:2 from LID:0x0152
TID:0x0000000000000002
Mar 19 18:37:00 590067 [45007960] -> osm_report_notice: Reporting Generic
Notice type:1 num:128 from LID:0x0152
GID:0xfe80000000000000,0x0005ad0000027c84
Mar 19 18:37:00 611770 [41E02960] -> __osm_trap_rcv_process_request:
Received Generic Notice type:0x01 num:128 Producer:2 from LID:0x0152
TID:0x0000000000000003
Mar 19 18:37:00 611916 [41E02960] -> osm_report_notice: Reporting Generic
Notice type:1 num:128 from LID:0x0152
GID:0xfe80000000000000,0x0005ad0000027c84
Mar 19 18:37:00 800652 [42803960] -> __osm_trap_rcv_process_request:
Received Generic Notice type:0x01 num:128 Producer:2 from LID:0x0152
TID:0x0000000000000004
Mar 19 18:37:00 817995 [45007960] -> osm_ucast_mgr_process: Min Hop Tables
configured on all switches
Mar 19 18:37:00 861575 [42803960] -> osm_report_notice: Reporting Generic
Notice type:1 num:128 from LID:0x0152
GID:0xfe80000000000000,0x0005ad0000027c84
Mar 19 18:37:00 983908 [42803960] -> __osm_trap_rcv_process_request:
Received Generic Notice type:0x01 num:128 Producer:2 from LID:0x0152
TID:0x0000000000000005
Mar 19 18:37:00 984004 [42803960] -> osm_report_notice: Reporting Generic
Notice type:1 num:128 from LID:0x0152
GID:0xfe80000000000000,0x0005ad0000027c84
Mar 19 18:37:01 012195 [44606960] -> __osm_trap_rcv_process_request:
Received Generic Notice type:0x01 num:128 Producer:2 from LID:0x0152
TID:0x0000000000000006
Mar 19 18:37:01 012283 [44606960] -> osm_report_notice: Reporting Generic
Notice type:1 num:128 from LID:0x0152
GID:0xfe80000000000000,0x0005ad0000027c84
Mar 19 18:37:01 034177 [43204960] -> __osm_trap_rcv_process_request:
Received Generic Notice type:0x01 num:128 Producer:2 from LID:0x0152
TID:0x0000000000000007
Mar 19 18:37:01 034272 [43204960] -> osm_report_notice: Reporting Generic
Notice type:1 num:128 from LID:0x0152
GID:0xfe80000000000000,0x0005ad0000027c84
Mar 19 18:37:01 056001 [41401960] -> __osm_trap_rcv_process_request:
Received Generic Notice type:0x01 num:128 Producer:2 from LID:0x0152
TID:0x0000000000000008
Mar 19 18:37:01 056068 [41401960] -> osm_report_notice: Reporting Generic
Notice type:1 num:128 from LID:0x0152
GID:0xfe80000000000000,0x0005ad0000027c84
Mar 19 18:37:01 074341 [43204960] -> SUBNET UP
Mar 19 18:37:01 252871 [43204960] -> __osm_trap_rcv_process_request:
Received Generic Notice type:0x01 num:128 Producer:2 from LID:0x0152
TID:0x0000000000000009
Mar 19 18:37:01 253037 [43204960] -> osm_report_notice: Reporting Generic
Notice type:1 num:128 from LID:0x0152
GID:0xfe80000000000000,0x0005ad0000027c84
Mar 19 18:37:01 303407 [44606960] -> __osm_trap_rcv_process_request:
Received Generic Notice type:0x01 num:128 Producer:2 from LID:0x0152
TID:0x000000000000000a
Mar 19 18:37:01 303490 [44606960] -> osm_report_notice: Reporting Generic
Notice type:1 num:128 from LID:0x0152
GID:0xfe80000000000000,0x0005ad0000027c84
Mar 19 18:37:01 325057 [41E02960] -> __osm_trap_rcv_process_request:
Received Generic Notice type:0x01 num:128 Producer:2 from LID:0x0152
TID:0x000000000000000b
Mar 19 18:37:01 325160 [41E02960] -> __osm_trap_rcv_process_request: ERR
3804: Received trap 11 times consecutively
Mar 19 18:37:01 334059 [43204960] -> __osm_trap_rcv_process_request:
Received Generic Notice type:0x01 num:128 Producer:2 from LID:0x0152
TID:0x000000000000000c
Mar 19 18:37:01 334118 [43204960] -> __osm_trap_rcv_process_request: ERR
3804: Received trap 12 times consecutively
Mar 19 18:37:01 474293 [45007960] -> osm_report_notice: Reporting Generic
Notice type:3 num:65 from LID:0x0092
GID:0xfe80000000000000,0x0005ad0000024bbb
Mar 19 18:37:01 474317 [45007960] -> Removed port with
GUID:0x0005ad000002510b LID range [0xB5,0xB5] of node:saguaro-23-6 HCA-1
Mar 19 18:37:01 474341 [45007960] -> osm_report_notice: Reporting Generic
Notice type:3 num:65 from LID:0x0092
GID:0xfe80000000000000,0x0005ad0000024bbb
Mar 19 18:37:01 474348 [45007960] -> Removed port with
GUID:0x0005ad0000024d47 LID range [0xB6,0xB6] of node:saguaro-23-7 HCA-1
Mar 19 18:37:01 474371 [45007960] -> osm_report_notice: Reporting Generic
Notice type:3 num:65 from LID:0x0092
GID:0xfe80000000000000,0x0005ad0000024bbb
Mar 19 18:37:01 474378 [45007960] -> Removed port with
GUID:0x0005ad0000024d8b LID range [0xB7,0xB7] of node:saguaro-23-8 HCA-1
Mar 19 18:37:01 509205 [45007960] -> osm_ucast_mgr_process: Min Hop Tables
configured on all switches
Mar 19 18:37:01 557110 [45A08960] -> __osm_trap_rcv_process_request:
Received Generic Notice type:0x01 num:128 Producer:2 from LID:0x0152
TID:0x000000000000000d
Mar 19 18:37:01 557172 [45A08960] -> __osm_trap_rcv_process_request: ERR
3804: Received trap 13 times consecutively
Mar 19 18:37:01 565676 [43C05960] -> SUBNET UP
Mar 19 18:37:01 576199 [41401960] -> __osm_trap_rcv_process_request:
Received Generic Notice type:0x01 num:128 Producer:2 from LID:0x0152
TID:0x000000000000000e
Mar 19 18:37:01 576270 [41401960] -> __osm_trap_rcv_process_request: ERR
3804: Received trap 14 times consecutively
Mar 19 18:37:01 599713 [41E02960] -> __osm_trap_rcv_process_request:
Received Generic Notice type:0x01 num:128 Producer:2 from LID:0x0152
TID:0x000000000000000f
Mar 19 18:37:01 599779 [41E02960] -> __osm_trap_rcv_process_request: ERR
3804: Received trap 15 times consecutively
Mar 19 18:37:01 707096 [45007960] -> __osm_trap_rcv_process_request:
Received Generic Notice type:0x01 num:128 Producer:2 from LID:0x0152
TID:0x0000000000000010
Mar 19 18:37:01 707150 [45007960] -> __osm_trap_rcv_process_request: ERR
3804: Received trap 16 times consecutively
Mar 19 18:37:01 921406 [45A08960] -> osm_report_notice: Reporting Generic
Notice type:3 num:65 from LID:0x0092
GID:0xfe80000000000000,0x0005ad0000024bbb
Mar 19 18:37:01 921423 [45A08960] -> Removed port with
GUID:0x0005ad000002511b LID range [0xA6,0xA6] of node:saguaro-22-1 HCA-1
Mar 19 18:37:01 921448 [45A08960] -> osm_report_notice: Reporting Generic
Notice type:3 num:65 from LID:0x0092
GID:0xfe80000000000000,0x0005ad0000024bbb
Mar 19 18:37:01 921455 [45A08960] -> Removed port with
GUID:0x0005ad0000024d6b LID range [0xB8,0xB8] of node:saguaro-23-9 HCA-1
Mar 19 18:37:01 921495 [45A08960] -> osm_report_notice: Reporting Generic
Notice type:3 num:65 from LID:0x0092
GID:0xfe80000000000000,0x0005ad0000024bbb
Mar 19 18:37:01 921502 [45A08960] -> Removed port with
GUID:0x0005ad0000024afb LID range [0xA5,0xA5] of node:saguaro-22-0 HCA-1
Mar 19 18:37:01 925845 [41E02960] -> osm_report_notice: Reporting Generic
Notice type:3 num:64 from LID:0x0092
GID:0xfe80000000000000,0x0005ad0000024bbb
Mar 19 18:37:01 925855 [41E02960] -> Discovered new port with
GUID:0x0005ad000002510b LID range [0xB5,0xB5] of node:saguaro-23-6 HCA-1
Mar 19 18:37:01 925859 [41E02960] -> osm_report_notice: Reporting Generic
Notice type:3 num:64 from LID:0x0092
GID:0xfe80000000000000,0x0005ad0000024bbb
Mar 19 18:37:01 925864 [41E02960] -> Discovered new port with
GUID:0x0005ad0000024d47 LID range [0xB6,0xB6] of node:saguaro-23-7 HCA-1
Mar 19 18:37:01 925868 [41E02960] -> osm_report_notice: Reporting Generic
Notice type:3 num:64 from LID:0x0092
GID:0xfe80000000000000,0x0005ad0000024bbb
Mar 19 18:37:01 925873 [41E02960] -> Discovered new port with
GUID:0x0005ad0000024d8b LID range [0xB7,0xB7] of node:saguaro-23-8 HCA-1
Mar 19 18:37:01 956691 [41E02960] -> osm_ucast_mgr_process: Min Hop Tables
configured on all switches
Mar 19 18:37:01 999372 [43204960] -> __osm_trap_rcv_process_request:
Received Generic Notice type:0x01 num:128 Producer:2 from LID:0x0152
TID:0x0000000000000011
Mar 19 18:37:01 999470 [43204960] -> __osm_trap_rcv_process_request: ERR
3804: Received trap 17 times consecutively
Mar 19 18:37:02 012194 [41E02960] -> __osm_trap_rcv_process_request:
Received Generic Notice type:0x01 num:128 Producer:2 from LID:0x0152
TID:0x0000000000000012
Mar 19 18:37:02 012256 [41E02960] -> __osm_trap_rcv_process_request: ERR
3804: Received trap 18 times consecutively
Mar 19 18:37:02 014327 [41401960] -> SUBNET UP
Mar 19 18:37:02 034202 [44606960] -> __osm_trap_rcv_process_request:
Received Generic Notice type:0x01 num:128 Producer:2 from LID:0x0152
TID:0x0000000000000013
Mar 19 18:37:02 034250 [44606960] -> __osm_trap_rcv_process_request: ERR
3804: Received trap 19 times consecutively
Mar 19 18:37:02 056015 [45A08960] -> __osm_trap_rcv_process_request:
Received Generic Notice type:0x01 num:128 Producer:2 from LID:0x0152
TID:0x0000000000000014
Mar 19 18:37:02 056060 [45A08960] -> __osm_trap_rcv_process_request: ERR
3804: Received trap 20 times consecutively
Mar 19 18:37:02 270731 [43204960] -> __osm_trap_rcv_process_request:
Received Generic Notice type:0x01 num:128 Producer:2 from LID:0x0152
TID:0x0000000000000015
Mar 19 18:37:02 270777 [43204960] -> __osm_trap_rcv_process_request: ERR
3804: Received trap 21 times consecutively
Mar 19 18:37:02 271169 [43C05960] -> __osm_trap_rcv_process_request:
Received Generic Notice type:0x01 num:128 Producer:2 from LID:0x0148
TID:0x0000000000000038
Mar 19 18:37:02 271347 [43C05960] -> osm_report_notice: Reporting Generic
Notice type:1 num:128 from LID:0x0148
GID:0xfe80000000000000,0x0005ad00000281b3
Mar 19 18:37:02 462374 [41E02960] -> __osm_trap_rcv_process_request:
Received Generic Notice type:0x01 num:128 Producer:2 from LID:0x0148
TID:0x0000000000000039
Mar 19 18:37:02 462511 [41E02960] -> osm_report_notice: Reporting Generic
Notice type:1 num:128 from LID:0x0148
GID:0xfe80000000000000,0x0005ad00000281b3
Mar 19 18:37:02 496247 [45007960] -> __osm_trap_rcv_process_request:
Received Generic Notice type:0x01 num:128 Producer:2 from LID:0x0148
TID:0x000000000000003a
Mar 19 18:37:02 496310 [45007960] -> osm_report_notice: Reporting Generic
Notice type:1 num:128 from LID:0x0148
GID:0xfe80000000000000,0x0005ad00000281b3
Mar 19 18:37:02 624890 [45A08960] -> osm_report_notice: Reporting Generic
Notice type:3 num:64 from LID:0x0092
GID:0xfe80000000000000,0x0005ad0000024bbb
Mar 19 18:37:02 624902 [45A08960] -> Discovered new port with
GUID:0x0005ad0000024d6b LID range [0xB8,0xB8] of node:saguaro-23-9 HCA-1
Mar 19 18:37:02 624908 [45A08960] -> osm_report_notice: Reporting Generic
Notice type:3 num:64 from LID:0x0092
GID:0xfe80000000000000,0x0005ad0000024bbb
Mar 19 18:37:02 624914 [45A08960] -> Discovered new port with
GUID:0x0005ad0000024afb LID range [0xA5,0xA5] of node:saguaro-22-0 HCA-1
Mar 19 18:37:02 624919 [45A08960] -> osm_report_notice: Reporting Generic
Notice type:3 num:64 from LID:0x0092
GID:0xfe80000000000000,0x0005ad0000024bbb
Mar 19 18:37:02 624926 [45A08960] -> Discovered new port with
GUID:0x0005ad000002511b LID range [0xA6,0xA6] of node:saguaro-22-1 HCA-1
Mar 19 18:37:02 655848 [45A08960] -> osm_ucast_mgr_process: Min Hop Tables
configured on all switches
Mar 19 18:37:02 709115 [42803960] -> SUBNET UP
Mar 19 18:37:03 082995 [44606960] -> __osm_trap_rcv_process_request:
Received Generic Notice type:0x01 num:128 Producer:2 from LID:0x0148
TID:0x000000000000003b
Mar 19 18:37:03 106373 [43204960] -> osm_ucast_mgr_process: Min Hop Tables
configured on all switches
Mar 19 18:37:03 136757 [44606960] -> osm_report_notice: Reporting Generic
Notice type:1 num:128 from LID:0x0148
GID:0xfe80000000000000,0x0005ad00000281b3
Mar 19 18:37:03 178027 [41401960] -> __osm_trap_rcv_process_request:
Received Generic Notice type:0x01 num:128 Producer:2 from LID:0x001B
TID:0x0000000000000047
Mar 19 18:37:03 178064 [43C05960] -> __osm_trap_rcv_process_request:
Received Generic Notice type:0x01 num:128 Producer:2 from LID:0x0148
TID:0x000000000000003c
Mar 19 18:37:03 178139 [41401960] -> osm_report_notice: Reporting Generic
Notice type:1 num:128 from LID:0x001B
GID:0xfe80000000000000,0x0005ad00000281a7
Mar 19 18:37:03 178160 [43C05960] -> osm_report_notice: Reporting Generic
Notice type:1 num:128 from LID:0x0148
GID:0xfe80000000000000,0x0005ad00000281b3
Mar 19 18:37:03 315226 [41E02960] -> __osm_trap_rcv_process_request:
Received Generic Notice type:0x01 num:128 Producer:2 from LID:0x0152
TID:0x0000000000000015
Mar 19 18:37:03 315289 [41E02960] -> __osm_trap_rcv_process_request: ERR
3804: Received trap 22 times consecutively
Mar 19 18:37:03 341474 [41E02960] -> __osm_trap_rcv_process_request:
Received Generic Notice type:0x01 num:128 Producer:2 from LID:0x0152
TID:0x0000000000000016
Mar 19 18:37:03 341549 [41E02960] -> __osm_trap_rcv_process_request: ERR
3804: Received trap 23 times consecutively
Mar 19 18:37:03 341616 [41E02960] -> __osm_trap_rcv_process_request:
Received Generic Notice type:0x01 num:128 Producer:2 from LID:0x0148
TID:0x000000000000003d
Mar 19 18:37:03 342446 [41E02960] -> osm_report_notice: Reporting Generic
Notice type:1 num:128 from LID:0x0148
GID:0xfe80000000000000,0x0005ad00000281b3
Mar 19 18:37:03 343169 [4780B960] -> __osm_sm_mad_ctrl_rcv_callback: ERR
3111: Error status = 0x1C00
Mar 19 18:37:03 343262 [4780B960] -> SMP dump:
base_ver................0x1
mgmt_class..............0x81
class_ver...............0x1
method..................0x81 (SubnGetResp)
D bit...................0x1
status..................0x1C00
hop_ptr.................0x0
hop_count...............0x5
trans_id................0x14d08
attr_id.................0x15 (PortInfo)
resv....................0x0
attr_mod................0x11
m_key...................0x0000000000000000
dr_slid.................0xFFFF
dr_dlid.................0xFFFF
Initial path: [0][1][11][1][6][16]
Return path: [0][9][18][D][3][11]
Reserved: [0][0][0][0][0][0][0]
00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
00 00 00 00 00 00 00 00 00 00 00 00 11 02 03 02
12 52 00 11 40 40 00 08 08 04 F2 40 00 00 00 00
00 00 00 00 00 88 00 00 00 00 00 00 00 00 00 00
Mar 19 18:37:03 343371 [4780B960] -> __osm_sm_mad_ctrl_rcv_callback: ERR
3111: Error status = 0x1C00
Mar 19 18:37:03 343364 [45007960] -> osm_pi_rcv_process_set: Received error
status 0x1c for SetResp() during ACTIVE transition
Mar 19 18:37:03 343415 [4780B960] -> SMP dump:
base_ver................0x1
mgmt_class..............0x81
class_ver...............0x1
method..................0x81 (SubnGetResp)
D bit...................0x1
status..................0x1C00
hop_ptr.................0x0
hop_count...............0x5
trans_id................0x14d09
attr_id.................0x15 (PortInfo)
resv....................0x0
attr_mod................0x12
m_key...................0x0000000000000000
dr_slid.................0xFFFF
dr_dlid.................0xFFFF
Initial path: [0][1][11][1][6][16]
Return path: [0][9][18][D][3][11]
Reserved: [0][0][0][0][0][0][0]
00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
00 00 00 00 00 00 00 00 00 00 00 00 11 02 03 02
12 52 00 11 40 40 00 08 08 04 F2 40 00 00 00 00
00 00 00 00 00 88 00 00 00 00 00 00 00 00 00 00
Mar 19 18:37:03 343409 [45007960] -> PortInfo dump:
port number.............0x11
node_guid...............0x0005ad0000027c84
port_guid...............0x0005ad0000027c84
m_key...................0x0000000000000000
subnet_prefix...........0x0000000000000000
base_lid................0x0
master_sm_base_lid......0x0
capability_mask.........0x0
diag_code...............0x0
m_key_lease_period......0x0
local_port_num..........0x11
link_width_enabled......0x2
link_width_supported....0x3
link_width_active.......0x2
link_speed_supported....0x1
port_state..............INIT
state_info2.............0x52
m_key_protect_bits......0x0
lmc.....................0x0
link_speed..............0x11
mtu_smsl................0x40
vl_cap_init_type........0x40
vl_high_limit...........0x0
vl_arb_high_cap.........0x8
vl_arb_low_cap..........0x8
init_rep_mtu_cap........0x4
vl_stall_life...........0xF2
vl_enforce..............0x40
m_key_violations........0x0
p_key_violations........0x0
q_key_violations........0x0
guid_cap................0x0
client_reregister.......0x0
subnet_timeout..........0x0
resp_time_value.........0x0
error_threshold.........0x88
Mar 19 18:37:03 343481 [45007960] -> Capabilities Mask:
Mar 19 18:37:03 343532 [45007960] -> osm_pi_rcv_process_set: Received error
status 0x1c for SetResp() during ACTIVE transition
Mar 19 18:37:03 343537 [45007960] -> PortInfo dump:
port number.............0x12
node_guid...............0x0005ad0000027c84
port_guid...............0x0005ad0000027c84
m_key...................0x0000000000000000
subnet_prefix...........0x0000000000000000
base_lid................0x0
master_sm_base_lid......0x0
capability_mask.........0x0
diag_code...............0x0
m_key_lease_period......0x0
local_port_num..........0x11
link_width_enabled......0x2
link_width_supported....0x3
link_width_active.......0x2
link_speed_supported....0x1
port_state..............INIT
state_info2.............0x52
m_key_protect_bits......0x0
lmc.....................0x0
link_speed..............0x11
mtu_smsl................0x40
vl_cap_init_type........0x40
vl_high_limit...........0x0
vl_arb_high_cap.........0x8
vl_arb_low_cap..........0x8
init_rep_mtu_cap........0x4
vl_stall_life...........0xF2
vl_enforce..............0x40
m_key_violations........0x0
p_key_violations........0x0
q_key_violations........0x0
guid_cap................0x0
client_reregister.......0x0
subnet_timeout..........0x0
resp_time_value.........0x0
error_threshold.........0x88
Mar 19 18:37:03 343555 [45007960] -> Capabilities Mask:
Mar 19 18:37:03 348684 [45007960] -> SUBNET UP
Mar 19 18:37:03 461748 [44606960] -> __osm_trap_rcv_process_request:
Received Generic Notice type:0x01 num:128 Producer:2 from LID:0x001B
TID:0x0000000000000048
Mar 19 18:37:03 461958 [44606960] -> osm_report_notice: Reporting Generic
Notice type:1 num:128 from LID:0x001B
GID:0xfe80000000000000,0x0005ad00000281a7
Mar 19 18:37:03 484827 [43C05960] -> __osm_trap_rcv_process_request:
Received Generic Notice type:0x01 num:128 Producer:2 from LID:0x0148
TID:0x000000000000003e
Mar 19 18:37:03 486448 [43C05960] -> osm_report_notice: Reporting Generic
Notice type:1 num:128 from LID:0x0148
GID:0xfe80000000000000,0x0005ad00000281b3
Mar 19 18:37:03 528040 [43204960] -> __osm_trap_rcv_process_request:
Received Generic Notice type:0x01 num:128 Producer:2 from LID:0x001B
TID:0x0000000000000049
Mar 19 18:37:03 528154 [43204960] -> osm_report_notice: Reporting Generic
Notice type:1 num:128 from LID:0x001B
GID:0xfe80000000000000,0x0005ad00000281a7
Mar 19 18:37:03 580196 [43C05960] -> __osm_trap_rcv_process_request:
Received Generic Notice type:0x01 num:128 Producer:2 from LID:0x001B
TID:0x000000000000004a
Mar 19 18:37:03 580534 [43C05960] -> osm_report_notice: Reporting Generic
Notice type:1 num:128 from LID:0x001B
GID:0xfe80000000000000,0x0005ad00000281a7
Mar 19 18:37:03 599784 [44606960] -> __osm_trap_rcv_process_request:
Received Generic Notice type:0x01 num:128 Producer:2 from LID:0x001B
TID:0x000000000000004b
Mar 19 18:37:03 599879 [44606960] -> osm_report_notice: Reporting Generic
Notice type:1 num:128 from LID:0x001B
GID:0xfe80000000000000,0x0005ad00000281a7
Mar 19 18:37:03 621883 [45A08960] -> __osm_trap_rcv_process_request:
Received Generic Notice type:0x01 num:128 Producer:2 from LID:0x001B
TID:0x000000000000004c
Mar 19 18:37:03 621940 [45A08960] -> osm_report_notice: Reporting Generic
Notice type:1 num:128 from LID:0x001B
GID:0xfe80000000000000,0x0005ad00000281a7
Mar 19 18:37:03 707894 [43C05960] -> osm_ucast_mgr_process: Min Hop Tables
configured on all switches
Mar 19 18:37:03 764678 [43204960] -> SUBNET UP
Mar 19 18:37:03 783783 [41401960] -> __osm_trap_rcv_process_request:
Received Generic Notice type:0x01 num:128 Producer:2 from LID:0x001B
TID:0x000000000000004d
Mar 19 18:37:03 783844 [41401960] -> osm_report_notice: Reporting Generic
Notice type:1 num:128 from LID:0x001B
GID:0xfe80000000000000,0x0005ad00000281a7
Mar 19 18:37:04 000228 [43204960] -> __osm_trap_rcv_process_request:
Received Generic Notice type:0x01 num:128 Producer:2 from LID:0x001B
TID:0x000000000000004e
Mar 19 18:37:04 000628 [43204960] -> osm_report_notice: Reporting Generic
Notice type:1 num:128 from LID:0x001B
GID:0xfe80000000000000,0x0005ad00000281a7
Mar 19 18:37:04 022198 [43C05960] -> __osm_trap_rcv_process_request:
Received Generic Notice type:0x01 num:128 Producer:2 from LID:0x001B
TID:0x000000000000004f
Mar 19 18:37:04 022299 [43C05960] -> osm_report_notice: Reporting Generic
Notice type:1 num:128 from LID:0x001B
GID:0xfe80000000000000,0x0005ad00000281a7
Mar 19 18:37:04 043985 [42803960] -> __osm_trap_rcv_process_request:
Received Generic Notice type:0x01 num:128 Producer:2 from LID:0x001B
TID:0x0000000000000050
Mar 19 18:37:04 044052 [42803960] -> osm_report_notice: Reporting Generic
Notice type:1 num:128 from LID:0x001B
GID:0xfe80000000000000,0x0005ad00000281a7
Mar 19 18:37:04 155809 [45A08960] -> osm_ucast_mgr_process: Min Hop Tables
configured on all switches
Mar 19 18:37:04 210448 [41401960] -> SUBNET UP
Mar 19 18:37:04 504490 [43204960] -> __osm_trap_rcv_process_request:
Received Generic Notice type:0x01 num:128 Producer:2 from LID:0x0152
TID:0x0000000000000017
Mar 19 18:37:04 504569 [43204960] -> __osm_trap_rcv_process_request: ERR
3804: Received trap 24 times consecutively
Mar 19 18:37:04 570084 [42803960] -> osm_ucast_mgr_process: Min Hop Tables
configured on all switches
Mar 19 18:37:04 626298 [43C05960] -> SUBNET UP
Mar 19 18:37:54 424084 [41E02960] -> __osm_trap_rcv_process_request:
Received Generic Notice type:0x01 num:128 Producer:2 from LID:0x001B
TID:0x0000000000000051
Mar 19 18:37:54 424430 [41E02960] -> osm_report_notice: Reporting Generic
Notice type:1 num:128 from LID:0x001B
GID:0xfe80000000000000,0x0005ad00000281a7
Mar 19 18:37:54 424457 [41E02960] -> __osm_trap_rcv_process_request:
Received Generic Notice type:0x01 num:128 Producer:2 from LID:0x0148
TID:0x000000000000003f
Mar 19 18:37:54 424522 [41E02960] -> osm_report_notice: Reporting Generic
Notice type:1 num:128 from LID:0x0148
GID:0xfe80000000000000,0x0005ad00000281b3
Mar 19 18:37:54 722515 [44606960] -> osm_report_notice: Reporting Generic
Notice type:3 num:65 from LID:0x0092
GID:0xfe80000000000000,0x0005ad0000024bbb
Mar 19 18:37:54 722536 [44606960] -> Removed port with
GUID:0x0005ad0000024e0b LID range [0xB3,0xB3] of node:saguaro-23-4 HCA-1
Mar 19 18:37:54 722558 [44606960] -> osm_report_notice: Reporting Generic
Notice type:3 num:65 from LID:0x0092
GID:0xfe80000000000000,0x0005ad0000024bbb
Mar 19 18:37:54 722565 [44606960] -> Removed port with
GUID:0x0005ad000002510b LID range [0xB5,0xB5] of node:saguaro-23-6 HCA-1
Mar 19 18:37:54 722587 [44606960] -> osm_report_notice: Reporting Generic
Notice type:3 num:65 from LID:0x0092
GID:0xfe80000000000000,0x0005ad0000024bbb
Mar 19 18:37:54 722594 [44606960] -> Removed port with
GUID:0x0005ad000002511b LID range [0xA6,0xA6] of node:saguaro-22-1 HCA-1
Mar 19 18:37:54 722636 [44606960] -> osm_report_notice: Reporting Generic
Notice type:3 num:65 from LID:0x0092
GID:0xfe80000000000000,0x0005ad0000024bbb
Mar 19 18:37:54 722641 [44606960] -> Removed port with
GUID:0x0005ad0000024b27 LID range [0xAF,0xAF] of node:saguaro-23-0 HCA-1
Mar 19 18:37:54 722658 [44606960] -> osm_report_notice: Reporting Generic
Notice type:3 num:65 from LID:0x0092
GID:0xfe80000000000000,0x0005ad0000024bbb
Mar 19 18:37:54 722663 [44606960] -> Removed port with
GUID:0x0005ad0000025043 LID range [0xB4,0xB4] of node:saguaro-23-5 HCA-1
Mar 19 18:37:54 722679 [44606960] -> osm_report_notice: Reporting Generic
Notice type:3 num:65 from LID:0x0092
GID:0xfe80000000000000,0x0005ad0000024bbb
Mar 19 18:37:54 722684 [44606960] -> Removed port with
GUID:0x0005ad0000024d47 LID range [0xB6,0xB6] of node:saguaro-23-7 HCA-1
Mar 19 18:37:54 722701 [44606960] -> osm_report_notice: Reporting Generic
Notice type:3 num:65 from LID:0x0092
GID:0xfe80000000000000,0x0005ad0000024bbb
Mar 19 18:37:54 722706 [44606960] -> Removed port with
GUID:0x0005ad0000024d6b LID range [0xB8,0xB8] of node:saguaro-23-9 HCA-1
Mar 19 18:37:54 722723 [44606960] -> osm_report_notice: Reporting Generic
Notice type:3 num:65 from LID:0x0092
GID:0xfe80000000000000,0x0005ad0000024bbb
Mar 19 18:37:54 722728 [44606960] -> Removed port with
GUID:0x0005ad0000024977 LID range [0xA9,0xA9] of node:saguaro-22-4 HCA-1
Mar 19 18:37:54 722875 [44606960] -> osm_report_notice: Reporting Generic
Notice type:3 num:65 from LID:0x0092
GID:0xfe80000000000000,0x0005ad0000024bbb
Mar 19 18:37:54 722880 [44606960] -> Removed port with
GUID:0x0005ad0000027c84 LID range [0x152,0x152] of node:Topspin Switch TS120
Mar 19 18:37:54 722909 [44606960] -> osm_report_notice: Reporting Generic
Notice type:3 num:65 from LID:0x0092
GID:0xfe80000000000000,0x0005ad0000024bbb
Mar 19 18:37:54 722915 [44606960] -> Removed port with
GUID:0x0005ad0000024d8b LID range [0xB7,0xB7] of node:saguaro-23-8 HCA-1
Mar 19 18:37:54 722929 [44606960] -> osm_report_notice: Reporting Generic
Notice type:3 num:65 from LID:0x0092
GID:0xfe80000000000000,0x0005ad0000024bbb
Mar 19 18:37:54 722934 [44606960] -> Removed port with
GUID:0x0005ad000002498f LID range [0xA8,0xA8] of node:saguaro-22-3 HCA-1
Mar 19 18:37:54 722949 [44606960] -> osm_report_notice: Reporting Generic
Notice type:3 num:65 from LID:0x0092
GID:0xfe80000000000000,0x0005ad0000024bbb
Mar 19 18:37:54 722955 [44606960] -> Removed port with
GUID:0x0005ad0000024c9b LID range [0xA7,0xA7] of node:saguaro-22-2 HCA-1
Mar 19 18:37:54 722970 [44606960] -> osm_report_notice: Reporting Generic
Notice type:3 num:65 from LID:0x0092
GID:0xfe80000000000000,0x0005ad0000024bbb
Mar 19 18:37:54 722975 [44606960] -> Removed port with
GUID:0x0005ad0000024da7 LID range [0xB0,0xB0] of node:saguaro-23-1 HCA-1
Mar 19 18:37:54 722992 [44606960] -> osm_report_notice: Reporting Generic
Notice type:3 num:65 from LID:0x0092
GID:0xfe80000000000000,0x0005ad0000024bbb
Mar 19 18:37:54 722997 [44606960] -> Removed port with
GUID:0x0005ad0000024cbb LID range [0xB2,0xB2] of node:saguaro-23-3 HCA-1
Mar 19 18:37:54 723012 [44606960] -> osm_report_notice: Reporting Generic
Notice type:3 num:65 from LID:0x0092
GID:0xfe80000000000000,0x0005ad0000024bbb
Mar 19 18:37:54 723073 [44606960] -> Removed port with
GUID:0x0005ad00000249d3 LID range [0xB1,0xB1] of node:saguaro-23-2 HCA-1
Mar 19 18:37:54 723090 [44606960] -> osm_report_notice: Reporting Generic
Notice type:3 num:65 from LID:0x0092
GID:0xfe80000000000000,0x0005ad0000024bbb
Mar 19 18:37:54 723095 [44606960] -> Removed port with
GUID:0x0005ad0000024feb LID range [0x153,0x153] of node:saguaro-22-5 HCA-1
Mar 19 18:37:54 723111 [44606960] -> osm_report_notice: Reporting Generic
Notice type:3 num:65 from LID:0x0092
GID:0xfe80000000000000,0x0005ad0000024bbb
Mar 19 18:37:54 723116 [44606960] -> Removed port with
GUID:0x0005ad0000024afb LID range [0xA5,0xA5] of node:saguaro-22-0 HCA-1
Mar 19 18:37:54 756302 [44606960] -> osm_ucast_mgr_process: Min Hop Tables
configured on all switches
Mar 19 18:37:54 806787 [45A08960] -> SUBNET UP
Mar 19 18:37:55 149566 [41E02960] -> osm_ucast_mgr_process: Min Hop Tables
configured on all switches
Mar 19 18:37:55 198855 [41401960] -> SUBNET UP
Mar 19 18:38:48 131054 [41E02960] -> __osm_trap_rcv_process_request:
Received Generic Notice type:0x01 num:128 Producer:2 from LID:0x0148
TID:0x0000000000000040
Mar 19 18:38:48 131349 [41E02960] -> osm_report_notice: Reporting Generic
Notice type:1 num:128 from LID:0x0148
GID:0xfe80000000000000,0x0005ad00000281b3
Mar 19 18:38:48 137230 [44606960] -> __osm_trap_rcv_process_request:
Received Generic Notice type:0x01 num:128 Producer:2 from LID:0x001B
TID:0x0000000000000052
Mar 19 18:38:48 137268 [45007960] -> __osm_trap_rcv_process_request:
Received Generic Notice type:0x01 num:128 Producer:2 from LID:0x0148
TID:0x0000000000000041
Mar 19 18:38:48 137395 [44606960] -> osm_report_notice: Reporting Generic
Notice type:1 num:128 from LID:0x001B
GID:0xfe80000000000000,0x0005ad00000281a7
Mar 19 18:38:48 137432 [45007960] -> osm_report_notice: Reporting Generic
Notice type:1 num:128 from LID:0x0148
GID:0xfe80000000000000,0x0005ad00000281b3
Mar 19 18:38:48 143370 [45A08960] -> __osm_trap_rcv_process_request:
Received Generic Notice type:0x01 num:128 Producer:2 from LID:0x001B
TID:0x0000000000000053
Mar 19 18:38:48 144327 [45A08960] -> osm_report_notice: Reporting Generic
Notice type:1 num:128 from LID:0x001B
GID:0xfe80000000000000,0x0005ad00000281a7
Mar 19 18:38:48 529052 [41E02960] -> osm_report_notice: Reporting Generic
Notice type:3 num:64 from LID:0x0092
GID:0xfe80000000000000,0x0005ad0000024bbb
Mar 19 18:38:48 529065 [41E02960] -> Discovered new port with
GUID:0x0005ad0000027c84 LID range [0x152,0x152] of node:Topspin Switch TS120
Mar 19 18:38:48 529071 [41E02960] -> osm_report_notice: Reporting Generic
Notice type:3 num:64 from LID:0x0092
GID:0xfe80000000000000,0x0005ad0000024bbb
Mar 19 18:38:48 529078 [41E02960] -> Discovered new port with
GUID:0x0005ad0000024b27 LID range [0xAF,0xAF] of node:saguaro-23-0 HCA-1
Mar 19 18:38:48 529083 [41E02960] -> osm_report_notice: Reporting Generic
Notice type:3 num:64 from LID:0x0092
GID:0xfe80000000000000,0x0005ad0000024bbb
Mar 19 18:38:48 529090 [41E02960] -> Discovered new port with
GUID:0x0005ad00000249d3 LID range [0xB1,0xB1] of node:saguaro-23-2 HCA-1
Mar 19 18:38:48 529095 [41E02960] -> osm_report_notice: Reporting Generic
Notice type:3 num:64 from LID:0x0092
GID:0xfe80000000000000,0x0005ad0000024bbb
Mar 19 18:38:48 529101 [41E02960] -> Discovered new port with
GUID:0x0005ad0000024da7 LID range [0xB0,0xB0] of node:saguaro-23-1 HCA-1
Mar 19 18:38:48 529106 [41E02960] -> osm_report_notice: Reporting Generic
Notice type:3 num:64 from LID:0x0092
GID:0xfe80000000000000,0x0005ad0000024bbb
Mar 19 18:38:48 529113 [41E02960] -> Discovered new port with
GUID:0x0005ad0000024cbb LID range [0xB2,0xB2] of node:saguaro-23-3 HCA-1
Mar 19 18:38:48 529118 [41E02960] -> osm_report_notice: Reporting Generic
Notice type:3 num:64 from LID:0x0092
GID:0xfe80000000000000,0x0005ad0000024bbb
Mar 19 18:38:48 529124 [41E02960] -> Discovered new port with
GUID:0x0005ad0000024e0b LID range [0xB3,0xB3] of node:saguaro-23-4 HCA-1
Mar 19 18:38:48 529129 [41E02960] -> osm_report_notice: Reporting Generic
Notice type:3 num:64 from LID:0x0092
GID:0xfe80000000000000,0x0005ad0000024bbb
Mar 19 18:38:48 529136 [41E02960] -> Discovered new port with
GUID:0x0005ad0000025043 LID range [0xB4,0xB4] of node:saguaro-23-5 HCA-1
Mar 19 18:38:48 529141 [41E02960] -> osm_report_notice: Reporting Generic
Notice type:3 num:64 from LID:0x0092
GID:0xfe80000000000000,0x0005ad0000024bbb
Mar 19 18:38:48 529147 [41E02960] -> Discovered new port with
GUID:0x0005ad000002510b LID range [0xB5,0xB5] of node:saguaro-23-6 HCA-1
Mar 19 18:38:48 529152 [41E02960] -> osm_report_notice: Reporting Generic
Notice type:3 num:64 from LID:0x0092
GID:0xfe80000000000000,0x0005ad0000024bbb
Mar 19 18:38:48 529159 [41E02960] -> Discovered new port with
GUID:0x0005ad0000024d47 LID range [0xB6,0xB6] of node:saguaro-23-7 HCA-1
Mar 19 18:38:48 529164 [41E02960] -> osm_report_notice: Reporting Generic
Notice type:3 num:64 from LID:0x0092
GID:0xfe80000000000000,0x0005ad0000024bbb
Mar 19 18:38:48 529170 [41E02960] -> Discovered new port with
GUID:0x0005ad0000024d8b LID range [0xB7,0xB7] of node:saguaro-23-8 HCA-1
Mar 19 18:38:48 529175 [41E02960] -> osm_report_notice: Reporting Generic
Notice type:3 num:64 from LID:0x0092
GID:0xfe80000000000000,0x0005ad0000024bbb
Mar 19 18:38:48 529182 [41E02960] -> Discovered new port with
GUID:0x0005ad0000024d6b LID range [0xB8,0xB8] of node:saguaro-23-9 HCA-1
Mar 19 18:38:48 529186 [41E02960] -> osm_report_notice: Reporting Generic
Notice type:3 num:64 from LID:0x0092
GID:0xfe80000000000000,0x0005ad0000024bbb
Mar 19 18:38:48 529193 [41E02960] -> Discovered new port with
GUID:0x0005ad0000024afb LID range [0xA5,0xA5] of node:saguaro-22-0 HCA-1
Mar 19 18:38:48 529198 [41E02960] -> osm_report_notice: Reporting Generic
Notice type:3 num:64 from LID:0x0092
GID:0xfe80000000000000,0x0005ad0000024bbb
Mar 19 18:38:48 529204 [41E02960] -> Discovered new port with
GUID:0x0005ad000002511b LID range [0xA6,0xA6] of node:saguaro-22-1 HCA-1
Mar 19 18:38:48 529209 [41E02960] -> osm_report_notice: Reporting Generic
Notice type:3 num:64 from LID:0x0092
GID:0xfe80000000000000,0x0005ad0000024bbb
Mar 19 18:38:48 529216 [41E02960] -> Discovered new port with
GUID:0x0005ad0000024c9b LID range [0xA7,0xA7] of node:saguaro-22-2 HCA-1
Mar 19 18:38:48 529271 [41E02960] -> osm_report_notice: Reporting Generic
Notice type:3 num:64 from LID:0x0092
GID:0xfe80000000000000,0x0005ad0000024bbb
Mar 19 18:38:48 529277 [41E02960] -> Discovered new port with
GUID:0x0005ad000002498f LID range [0xA8,0xA8] of node:saguaro-22-3 HCA-1
Mar 19 18:38:48 529281 [41E02960] -> osm_report_notice: Reporting Generic
Notice type:3 num:64 from LID:0x0092
GID:0xfe80000000000000,0x0005ad0000024bbb
Mar 19 18:38:48 529286 [41E02960] -> Discovered new port with
GUID:0x0005ad0000024977 LID range [0xA9,0xA9] of node:saguaro-22-4 HCA-1
Mar 19 18:38:48 529290 [41E02960] -> osm_report_notice: Reporting Generic
Notice type:3 num:64 from LID:0x0092
GID:0xfe80000000000000,0x0005ad0000024bbb
Mar 19 18:38:48 529294 [41E02960] -> Discovered new port with
GUID:0x0005ad0000024feb LID range [0x153,0x153] of node:saguaro-22-5 HCA-1
Mar 19 18:38:48 560082 [41E02960] -> osm_ucast_mgr_process: Min Hop Tables
configured on all switches
Mar 19 18:38:48 630464 [43204960] -> SUBNET UP
Mar 19 18:38:49 018498 [44606960] -> osm_ucast_mgr_process: Min Hop Tables
configured on all switches
Mar 19 18:38:49 073355 [45007960] -> SUBNET UP
Mar 19 18:39:04 189829 [45007960] -> __osm_trap_rcv_process_request:
Received Generic Notice type:0x01 num:128 Producer:2 from LID:0x0152
TID:0x0000000000000000
Mar 19 18:39:04 190072 [45007960] -> osm_report_notice: Reporting Generic
Notice type:1 num:128 from LID:0x0152
GID:0xfe80000000000000,0x0005ad0000027c84
Mar 19 18:39:04 307827 [44606960] -> __osm_trap_rcv_process_request:
Received Generic Notice type:0x01 num:128 Producer:2 from LID:0x0152
TID:0x0000000000000001
Mar 19 18:39:04 307940 [44606960] -> osm_report_notice: Reporting Generic
Notice type:1 num:128 from LID:0x0152
GID:0xfe80000000000000,0x0005ad0000027c84
Mar 19 18:39:04 330104 [44606960] -> __osm_trap_rcv_process_request:
Received Generic Notice type:0x01 num:128 Producer:2 from LID:0x0152
TID:0x0000000000000002
Mar 19 18:39:04 330210 [44606960] -> osm_report_notice: Reporting Generic
Notice type:1 num:128 from LID:0x0152
GID:0xfe80000000000000,0x0005ad0000027c84
Mar 19 18:39:04 468676 [41401960] -> __osm_trap_rcv_process_request:
Received Generic Notice type:0x01 num:128 Producer:2 from LID:0x0152
TID:0x0000000000000003
Mar 19 18:39:04 468758 [41401960] -> osm_report_notice: Reporting Generic
Notice type:1 num:128 from LID:0x0152
GID:0xfe80000000000000,0x0005ad0000027c84
Mar 19 18:39:04 680305 [42803960] -> __osm_trap_rcv_process_request:
Received Generic Notice type:0x01 num:128 Producer:2 from LID:0x0152
TID:0x0000000000000004
Mar 19 18:39:04 680400 [42803960] -> osm_report_notice: Reporting Generic
Notice type:1 num:128 from LID:0x0152
GID:0xfe80000000000000,0x0005ad0000027c84
Mar 19 18:39:04 702144 [41E02960] -> __osm_trap_rcv_process_request:
Received Generic Notice type:0x01 num:128 Producer:2 from LID:0x0152
TID:0x0000000000000005
Mar 19 18:39:04 702286 [41E02960] -> osm_report_notice: Reporting Generic
Notice type:1 num:128 from LID:0x0152
GID:0xfe80000000000000,0x0005ad0000027c84
Mar 19 18:39:04 704346 [43204960] -> osm_report_notice: Reporting Generic
Notice type:3 num:65 from LID:0x0092
GID:0xfe80000000000000,0x0005ad0000024bbb
Mar 19 18:39:04 704354 [43204960] -> Removed port with
GUID:0x0005ad00000249d3 LID range [0xB1,0xB1] of node:saguaro-23-2 HCA-1
Mar 19 18:39:04 739059 [43204960] -> osm_ucast_mgr_process: Min Hop Tables
configured on all switches
Mar 19 18:39:04 739896 [41E02960] -> __osm_trap_rcv_process_request:
Received Generic Notice type:0x01 num:128 Producer:2 from LID:0x0152
TID:0x0000000000000006
Mar 19 18:39:04 783807 [41E02960] -> osm_report_notice: Reporting Generic
Notice type:1 num:128 from LID:0x0152
GID:0xfe80000000000000,0x0005ad0000027c84
Mar 19 18:39:04 797411 [44606960] -> SUBNET UP
Mar 19 18:39:04 849970 [42803960] -> __osm_trap_rcv_process_request:
Received Generic Notice type:0x01 num:128 Producer:2 from LID:0x0152
TID:0x0000000000000007
Mar 19 18:39:04 850195 [42803960] -> osm_report_notice: Reporting Generic
Notice type:1 num:128 from LID:0x0152
GID:0xfe80000000000000,0x0005ad0000027c84
Mar 19 18:39:04 853735 [43C05960] -> __osm_trap_rcv_process_request:
Received Generic Notice type:0x01 num:128 Producer:2 from LID:0x0152
TID:0x0000000000000008
Mar 19 18:39:04 853809 [43C05960] -> osm_report_notice: Reporting Generic
Notice type:1 num:128 from LID:0x0152
GID:0xfe80000000000000,0x0005ad0000027c84
Mar 19 18:39:04 897727 [43C05960] -> __osm_trap_rcv_process_request:
Received Generic Notice type:0x01 num:128 Producer:2 from LID:0x0152
TID:0x0000000000000009
Mar 19 18:39:04 897860 [43C05960] -> osm_report_notice: Reporting Generic
Notice type:1 num:128 from LID:0x0152
GID:0xfe80000000000000,0x0005ad0000027c84
Mar 19 18:39:04 901577 [41401960] -> __osm_trap_rcv_process_request:
Received Generic Notice type:0x01 num:128 Producer:2 from LID:0x0152
TID:0x000000000000000a
Mar 19 18:39:04 901719 [41401960] -> osm_report_notice: Reporting Generic
Notice type:1 num:128 from LID:0x0152
GID:0xfe80000000000000,0x0005ad0000027c84
Mar 19 18:39:04 923271 [45007960] -> __osm_trap_rcv_process_request:
Received Generic Notice type:0x01 num:128 Producer:2 from LID:0x0152
TID:0x000000000000000b
Mar 19 18:39:04 923377 [45007960] -> __osm_trap_rcv_process_request: ERR
3804: Received trap 11 times consecutively
Mar 19 18:39:05 106246 [45007960] -> __osm_trap_rcv_process_request:
Received Generic Notice type:0x01 num:128 Producer:2 from LID:0x0152
TID:0x000000000000000c
Mar 19 18:39:05 106314 [45007960] -> __osm_trap_rcv_process_request: ERR
3804: Received trap 12 times consecutively
Mar 19 18:39:05 178215 [44606960] -> __osm_trap_rcv_process_request:
Received Generic Notice type:0x01 num:128 Producer:2 from LID:0x0152
TID:0x000000000000000d
Mar 19 18:39:05 178258 [44606960] -> __osm_trap_rcv_process_request: ERR
3804: Received trap 13 times consecutively
Mar 19 18:39:05 272913 [42803960] -> __osm_trap_rcv_process_request:
Received Generic Notice type:0x01 num:128 Producer:2 from LID:0x0152
TID:0x000000000000000e
Mar 19 18:39:05 272983 [42803960] -> __osm_trap_rcv_process_request: ERR
3804: Received trap 14 times consecutively
Mar 19 18:39:05 339633 [43204960] -> __osm_trap_rcv_process_request:
Received Generic Notice type:0x01 num:128 Producer:2 from LID:0x0152
TID:0x000000000000000f
Mar 19 18:39:05 339679 [43204960] -> __osm_trap_rcv_process_request: ERR
3804: Received trap 15 times consecutively
Mar 19 18:39:05 469093 [41401960] -> __osm_trap_rcv_process_request:
Received Generic Notice type:0x01 num:128 Producer:2 from LID:0x0152
TID:0x0000000000000010
Mar 19 18:39:05 469145 [41401960] -> __osm_trap_rcv_process_request: ERR
3804: Received trap 16 times consecutively
Mar 19 18:39:05 484587 [44606960] -> __osm_trap_rcv_process_request:
Received Generic Notice type:0x01 num:128 Producer:2 from LID:0x0152
TID:0x0000000000000011
Mar 19 18:39:05 484633 [44606960] -> __osm_trap_rcv_process_request: ERR
3804: Received trap 17 times consecutively
Mar 19 18:39:05 574251 [43C05960] -> __osm_trap_rcv_process_request:
Received Generic Notice type:0x01 num:128 Producer:2 from LID:0x0152
TID:0x0000000000000012
Mar 19 18:39:05 574301 [43C05960] -> __osm_trap_rcv_process_request: ERR
3804: Received trap 18 times consecutively
Mar 19 18:39:05 602665 [41E02960] -> __osm_trap_rcv_process_request:
Received Generic Notice type:0x01 num:128 Producer:2 from LID:0x0152
TID:0x0000000000000013
Mar 19 18:39:05 602700 [41E02960] -> __osm_trap_rcv_process_request: ERR
3804: Received trap 19 times consecutively
Mar 19 18:39:05 646331 [45007960] -> __osm_trap_rcv_process_request:
Received Generic Notice type:0x01 num:128 Producer:2 from LID:0x0152
TID:0x0000000000000014
Mar 19 18:39:05 646369 [45007960] -> __osm_trap_rcv_process_request: ERR
3804: Received trap 20 times consecutively
Mar 19 18:39:05 834613 [41E02960] -> __osm_trap_rcv_process_request:
Received Generic Notice type:0x01 num:128 Producer:2 from LID:0x0152
TID:0x0000000000000015
Mar 19 18:39:05 834685 [41E02960] -> __osm_trap_rcv_process_request: ERR
3804: Received trap 21 times consecutively
Mar 19 18:39:05 851128 [45007960] -> __osm_trap_rcv_process_request:
Received Generic Notice type:0x01 num:128 Producer:2 from LID:0x0152
TID:0x0000000000000016
Mar 19 18:39:05 851166 [45007960] -> __osm_trap_rcv_process_request: ERR
3804: Received trap 22 times consecutively
Mar 19 18:39:05 875540 [45A08960] -> __osm_trap_rcv_process_request:
Received Generic Notice type:0x01 num:128 Producer:2 from LID:0x0152
TID:0x0000000000000017
Mar 19 18:39:05 875592 [45A08960] -> __osm_trap_rcv_process_request: ERR
3804: Received trap 23 times consecutively
Mar 19 18:39:05 897378 [42803960] -> __osm_trap_rcv_process_request:
Received Generic Notice type:0x01 num:128 Producer:2 from LID:0x0152
TID:0x0000000000000018
Mar 19 18:39:05 897424 [42803960] -> __osm_trap_rcv_process_request: ERR
3804: Received trap 24 times consecutively
Mar 19 18:39:05 907232 [4780B960] -> umad_receiver: ERR 5409: send completed
with error (method=0x1 attr=0x15 trans_id=0x124ef0001c2fe) -- dropping
Mar 19 18:39:05 907249 [4780B960] -> umad_receiver: ERR 5411: DR SMP
Mar 19 18:39:05 907259 [4780B960] -> __osm_sm_mad_ctrl_send_err_cb: ERR
3113: MAD completed in error (IB_TIMEOUT)
Mar 19 18:39:05 907295 [4780B960] -> SMP dump:
base_ver................0x1
mgmt_class..............0x81
class_ver...............0x1
method..................0x1 (SubnGet)
D bit...................0x0
status..................0x0
hop_ptr.................0x0
hop_count...............0x6
trans_id................0x1c2fe
attr_id.................0x15 (PortInfo)
resv....................0x0
attr_mod................0x1
m_key...................0x0000000000000000
dr_slid.................0xFFFF
dr_dlid.................0xFFFF
Initial path: [0][1][11][1][6][16][8]
Return path: [0][0][0][0][0][0][0]
Reserved: [0][0][0][0][0][0][0]
00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
Mar 19 18:39:05 907372 [41401960] -> osm_report_notice: Reporting Generic
Notice type:3 num:65 from LID:0x0092
GID:0xfe80000000000000,0x0005ad0000024bbb
Mar 19 18:39:05 907384 [41401960] -> Removed port with
GUID:0x0005ad0000024e0b LID range [0xB3,0xB3] of node:saguaro-23-4 HCA-1
Mar 19 18:39:05 907407 [41401960] -> osm_report_notice: Reporting Generic
Notice type:3 num:65 from LID:0x0092
GID:0xfe80000000000000,0x0005ad0000024bbb
Mar 19 18:39:05 907414 [41401960] -> Removed port with
GUID:0x0005ad0000025043 LID range [0xB4,0xB4] of node:saguaro-23-5 HCA-1
Mar 19 18:39:05 907480 [41401960] -> osm_report_notice: Reporting Generic
Notice type:3 num:65 from LID:0x0092
GID:0xfe80000000000000,0x0005ad0000024bbb
Mar 19 18:39:05 907485 [41401960] -> Removed port with
GUID:0x0005ad0000024cbb LID range [0xB2,0xB2] of node:saguaro-23-3 HCA-1
Mar 19 18:39:05 907577 [41401960] -> osm_report_notice: Reporting Generic
Notice type:3 num:65 from LID:0x0092
GID:0xfe80000000000000,0x0005ad0000024bbb
Mar 19 18:39:05 907582 [41401960] -> Removed port with
GUID:0x0005ad0000024d47 LID range [0xB6,0xB6] of node:saguaro-23-7 HCA-1
Mar 19 18:39:05 907618 [41401960] -> osm_drop_mgr_process: ERR 0108: Unknown
remote side for node 0x0005ad0000027c84 port 8. Adding to light sweep
sampling list
Mar 19 18:39:05 907657 [41401960] -> Directed Path Dump of 5 hop path:
Path = [0][1][11][1][6][16]
Mar 19 18:39:05 911559 [43204960] -> osm_report_notice: Reporting Generic
Notice type:3 num:64 from LID:0x0092
GID:0xfe80000000000000,0x0005ad0000024bbb
Mar 19 18:39:05 911572 [43204960] -> Discovered new port with
GUID:0x0005ad00000249d3 LID range [0xB1,0xB1] of node:saguaro-23-2 HCA-1
Mar 19 18:39:05 927229 [43C05960] -> __osm_trap_rcv_process_request:
Received Generic Notice type:0x01 num:128 Producer:2 from LID:0x0152
TID:0x0000000000000019
Mar 19 18:39:05 927285 [43C05960] -> __osm_trap_rcv_process_request: ERR
3804: Received trap 25 times consecutively
Mar 19 18:39:05 942538 [43204960] -> osm_ucast_mgr_process: Min Hop Tables
configured on all switches
Mar 19 18:39:06 000027 [41E02960] -> SUBNET UP
Mar 19 18:39:06 130255 [43204960] -> __osm_trap_rcv_process_request:
Received Generic Notice type:0x01 num:128 Producer:2 from LID:0x0152
TID:0x000000000000001a
Mar 19 18:39:06 130308 [43204960] -> __osm_trap_rcv_process_request: ERR
3804: Received trap 26 times consecutively
Mar 19 18:39:06 131922 [42803960] -> __osm_trap_rcv_process_request:
Received Generic Notice type:0x01 num:128 Producer:2 from LID:0x0148
TID:0x0000000000000042
Mar 19 18:39:06 132063 [42803960] -> osm_report_notice: Reporting Generic
Notice type:1 num:128 from LID:0x0148
GID:0xfe80000000000000,0x0005ad00000281b3
Mar 19 18:39:06 154579 [43C05960] -> __osm_trap_rcv_process_request:
Received Generic Notice type:0x01 num:128 Producer:2 from LID:0x0152
TID:0x000000000000001b
Mar 19 18:39:06 154681 [43C05960] -> __osm_trap_rcv_process_request: ERR
3804: Received trap 27 times consecutively
Mar 19 18:39:06 176248 [44606960] -> __osm_trap_rcv_process_request:
Received Generic Notice type:0x01 num:128 Producer:2 from LID:0x0152
TID:0x000000000000001c
Mar 19 18:39:06 176304 [44606960] -> __osm_trap_rcv_process_request: ERR
3804: Received trap 28 times consecutively
Mar 19 18:39:06 198132 [44606960] -> __osm_trap_rcv_process_request:
Received Generic Notice type:0x01 num:128 Producer:2 from LID:0x0152
TID:0x000000000000001d
Mar 19 18:39:06 198195 [44606960] -> __osm_trap_rcv_process_request: ERR
3804: Received trap 29 times consecutively
Mar 19 18:39:06 230022 [43C05960] -> __osm_trap_rcv_process_request:
Received Generic Notice type:0x01 num:128 Producer:2 from LID:0x0152
TID:0x000000000000001e
Mar 19 18:39:06 230108 [43C05960] -> __osm_trap_rcv_process_request: ERR
3804: Received trap 30 times consecutively
Mar 19 18:39:06 230229 [43204960] -> __osm_trap_rcv_process_request:
Received Generic Notice type:0x01 num:128 Producer:2 from LID:0x0148
TID:0x0000000000000043
Mar 19 18:39:06 230311 [43204960] -> osm_report_notice: Reporting Generic
Notice type:1 num:128 from LID:0x0148
GID:0xfe80000000000000,0x0005ad00000281b3
Mar 19 18:39:06 399543 [43C05960] -> osm_report_notice: Reporting Generic
Notice type:3 num:64 from LID:0x0092
GID:0xfe80000000000000,0x0005ad0000024bbb
Mar 19 18:39:06 399556 [43C05960] -> Discovered new port with
GUID:0x0005ad0000024cbb LID range [0xB2,0xB2] of node:saguaro-23-3 HCA-1
Mar 19 18:39:06 399562 [43C05960] -> osm_report_notice: Reporting Generic
Notice type:3 num:64 from LID:0x0092
GID:0xfe80000000000000,0x0005ad0000024bbb
Mar 19 18:39:06 399569 [43C05960] -> Discovered new port with
GUID:0x0005ad0000024e0b LID range [0xB3,0xB3] of node:saguaro-23-4 HCA-1
Mar 19 18:39:06 399574 [43C05960] -> osm_report_notice: Reporting Generic
Notice type:3 num:64 from LID:0x0092
GID:0xfe80000000000000,0x0005ad0000024bbb
Mar 19 18:39:06 399580 [43C05960] -> Discovered new port with
GUID:0x0005ad0000025043 LID range [0xB4,0xB4] of node:saguaro-23-5 HCA-1
Mar 19 18:39:06 399585 [43C05960] -> osm_report_notice: Reporting Generic
Notice type:3 num:64 from LID:0x0092
GID:0xfe80000000000000,0x0005ad0000024bbb
Mar 19 18:39:06 399592 [43C05960] -> Discovered new port with
GUID:0x0005ad0000024d47 LID range [0xB6,0xB6] of node:saguaro-23-7 HCA-1
Mar 19 18:39:06 430598 [43C05960] -> osm_ucast_mgr_process: Min Hop Tables
configured on all switches
Mar 19 18:39:06 494689 [44606960] -> SUBNET UP
Mar 19 18:39:06 837303 [43204960] -> __osm_trap_rcv_process_request:
Received Generic Notice type:0x01 num:128 Producer:2 from LID:0x0152
TID:0x000000000000001f
Mar 19 18:39:06 837446 [43204960] -> __osm_trap_rcv_process_request: ERR
3804: Received trap 31 times consecutively
Mar 19 18:39:06 838528 [43C05960] -> __osm_trap_rcv_process_request:
Received Generic Notice type:0x01 num:128 Producer:2 from LID:0x0148
TID:0x0000000000000044
Mar 19 18:39:06 838636 [43C05960] -> osm_report_notice: Reporting Generic
Notice type:1 num:128 from LID:0x0148
GID:0xfe80000000000000,0x0005ad00000281b3
Mar 19 18:39:06 876308 [43C05960] -> osm_ucast_mgr_process: Min Hop Tables
configured on all switches
Mar 19 18:39:07 028376 [45A08960] -> __osm_trap_rcv_process_request:
Received Generic Notice type:0x01 num:128 Producer:2 from LID:0x0152
TID:0x0000000000000020
Mar 19 18:39:07 028459 [45A08960] -> __osm_trap_rcv_process_request: ERR
3804: Received trap 32 times consecutively
Mar 19 18:39:07 028545 [43204960] -> __osm_trap_rcv_process_request:
Received Generic Notice type:0x01 num:128 Producer:2 from LID:0x0148
TID:0x0000000000000045
Mar 19 18:39:07 028652 [43204960] -> osm_report_notice: Reporting Generic
Notice type:1 num:128 from LID:0x0148
GID:0xfe80000000000000,0x0005ad00000281b3
Mar 19 18:39:07 030190 [45007960] -> __osm_trap_rcv_process_request:
Received Generic Notice type:0x01 num:128 Producer:2 from LID:0x001B
TID:0x0000000000000054
Mar 19 18:39:07 030277 [45007960] -> osm_report_notice: Reporting Generic
Notice type:1 num:128 from LID:0x001B
GID:0xfe80000000000000,0x0005ad00000281a7
Mar 19 18:39:07 096812 [41401960] -> __osm_trap_rcv_process_request:
Received Generic Notice type:0x01 num:128 Producer:2 from LID:0x0148
TID:0x0000000000000046
Mar 19 18:39:07 096959 [41401960] -> osm_report_notice: Reporting Generic
Notice type:1 num:128 from LID:0x0148
GID:0xfe80000000000000,0x0005ad00000281b3
Mar 19 18:39:07 111719 [4780B960] -> __osm_sm_mad_ctrl_rcv_callback: ERR
3111: Error status = 0x1C00
Mar 19 18:39:07 111759 [4780B960] -> SMP dump:
base_ver................0x1
mgmt_class..............0x81
class_ver...............0x1
method..................0x81 (SubnGetResp)
D bit...................0x1
status..................0x1C00
hop_ptr.................0x0
hop_count...............0x5
trans_id................0x1dfac
attr_id.................0x15 (PortInfo)
resv....................0x0
attr_mod................0x11
m_key...................0x0000000000000000
dr_slid.................0xFFFF
dr_dlid.................0xFFFF
Initial path: [0][1][11][1][4][16]
Return path: [0][9][18][D][1][11]
Reserved: [0][0][0][0][0][0][0]
00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
00 00 00 00 00 00 00 00 00 00 00 00 11 02 03 02
12 52 00 11 40 40 00 08 08 04 F2 40 00 00 00 00
00 00 00 00 00 88 00 00 00 00 00 00 00 00 00 00
Mar 19 18:39:07 111810 [41E02960] -> osm_pi_rcv_process_set: Received error
status 0x1c for SetResp() during ACTIVE transition
Mar 19 18:39:07 111814 [4780B960] -> __osm_sm_mad_ctrl_rcv_callback: ERR
3111: Error status = 0x1C00
Mar 19 18:39:07 111831 [41E02960] -> PortInfo dump:
port number.............0x11
node_guid...............0x0005ad0000027c84
port_guid...............0x0005ad0000027c84
m_key...................0x0000000000000000
subnet_prefix...........0x0000000000000000
base_lid................0x0
master_sm_base_lid......0x0
capability_mask.........0x0
diag_code...............0x0
m_key_lease_period......0x0
local_port_num..........0x11
link_width_enabled......0x2
link_width_supported....0x3
link_width_active.......0x2
link_speed_supported....0x1
port_state..............INIT
state_info2.............0x52
m_key_protect_bits......0x0
lmc.....................0x0
link_speed..............0x11
mtu_smsl................0x40
vl_cap_init_type........0x40
vl_high_limit...........0x0
vl_arb_high_cap.........0x8
vl_arb_low_cap..........0x8
init_rep_mtu_cap........0x4
vl_stall_life...........0xF2
vl_enforce..............0x40
m_key_violations........0x0
p_key_violations........0x0
q_key_violations........0x0
guid_cap................0x0
client_reregister.......0x0
subnet_timeout..........0x0
resp_time_value.........0x0
error_threshold.........0x88
Mar 19 18:39:07 111868 [41E02960] -> Capabilities Mask:
Mar 19 18:39:07 111844 [4780B960] -> SMP dump:
base_ver................0x1
mgmt_class..............0x81
class_ver...............0x1
method..................0x81 (SubnGetResp)
D bit...................0x1
status..................0x1C00
hop_ptr.................0x0
hop_count...............0x5
trans_id................0x1dfad
attr_id.................0x15 (PortInfo)
resv....................0x0
attr_mod................0x12
m_key...................0x0000000000000000
dr_slid.................0xFFFF
dr_dlid.................0xFFFF
Initial path: [0][1][11][1][4][16]
Return path: [0][9][18][D][1][11]
Reserved: [0][0][0][0][0][0][0]
00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
00 00 00 00 00 00 00 00 00 00 00 00 11 02 03 02
12 52 00 11 40 40 00 08 08 04 F2 40 00 00 00 00
00 00 00 00 00 88 00 00 00 00 00 00 00 00 00 00
Mar 19 18:39:07 112011 [41401960] -> osm_pi_rcv_process_set: Received error
status 0x1c for SetResp() during ACTIVE transition
Mar 19 18:39:07 112018 [41401960] -> PortInfo dump:
port number.............0x12
node_guid...............0x0005ad0000027c84
port_guid...............0x0005ad0000027c84
m_key...................0x0000000000000000
subnet_prefix...........0x0000000000000000
base_lid................0x0
master_sm_base_lid......0x0
capability_mask.........0x0
diag_code...............0x0
m_key_lease_period......0x0
local_port_num..........0x11
link_width_enabled......0x2
link_width_supported....0x3
link_width_active.......0x2
link_speed_supported....0x1
port_state..............INIT
state_info2.............0x52
m_key_protect_bits......0x0
lmc.....................0x0
link_speed..............0x11
mtu_smsl................0x40
vl_cap_init_type........0x40
vl_high_limit...........0x0
vl_arb_high_cap.........0x8
vl_arb_low_cap..........0x8
init_rep_mtu_cap........0x4
vl_stall_life...........0xF2
vl_enforce..............0x40
m_key_violations........0x0
p_key_violations........0x0
q_key_violations........0x0
guid_cap................0x0
client_reregister.......0x0
subnet_timeout..........0x0
resp_time_value.........0x0
error_threshold.........0x88
Mar 19 18:39:07 112034 [41401960] -> Capabilities Mask:
Mar 19 18:39:07 117211 [45A08960] -> SUBNET UP
Mar 19 18:39:07 354540 [41E02960] -> __osm_trap_rcv_process_request:
Received Generic Notice type:0x01 num:128 Producer:2 from LID:0x0148
TID:0x0000000000000047
Mar 19 18:39:07 354686 [41E02960] -> osm_report_notice: Reporting Generic
Notice type:1 num:128 from LID:0x0148
GID:0xfe80000000000000,0x0005ad00000281b3
Mar 19 18:39:07 383453 [42803960] -> __osm_trap_rcv_process_request:
Received Generic Notice type:0x01 num:128 Producer:2 from LID:0x001B
TID:0x0000000000000055
Mar 19 18:39:07 383530 [42803960] -> osm_report_notice: Reporting Generic
Notice type:1 num:128 from LID:0x001B
GID:0xfe80000000000000,0x0005ad00000281a7
Mar 19 18:39:07 497601 [42803960] -> osm_ucast_mgr_process: Min Hop Tables
configured on all switches
Mar 19 18:39:07 548184 [43204960] -> __osm_trap_rcv_process_request:
Received Generic Notice type:0x01 num:128 Producer:2 from LID:0x001B
TID:0x0000000000000056
Mar 19 18:39:07 548217 [43C05960] -> SUBNET UP
Mar 19 18:39:07 548427 [43204960] -> osm_report_notice: Reporting Generic
Notice type:1 num:128 from LID:0x001B
GID:0xfe80000000000000,0x0005ad00000281a7
Mar 19 18:39:07 878403 [45007960] -> __osm_trap_rcv_process_request:
Received Generic Notice type:0x01 num:128 Producer:2 from LID:0x001B
TID:0x0000000000000057
Mar 19 18:39:07 887312 [45A08960] -> __osm_trap_rcv_process_request:
Received Generic Notice type:0x01 num:128 Producer:2 from LID:0x001B
TID:0x0000000000000058
Mar 19 18:39:07 888156 [41E02960] -> osm_ucast_mgr_process: Min Hop Tables
configured on all switches
Mar 19 18:39:07 929819 [45007960] -> osm_report_notice: Reporting Generic
Notice type:1 num:128 from LID:0x001B
GID:0xfe80000000000000,0x0005ad00000281a7
Mar 19 18:39:07 929834 [45A08960] -> osm_report_notice: Reporting Generic
Notice type:1 num:128 from LID:0x001B
GID:0xfe80000000000000,0x0005ad00000281a7
Mar 19 18:39:07 931166 [45007960] -> __osm_trap_rcv_process_request:
Received Generic Notice type:0x01 num:128 Producer:2 from LID:0x001B
TID:0x0000000000000059
Mar 19 18:39:07 931288 [45007960] -> osm_report_notice: Reporting Generic
Notice type:1 num:128 from LID:0x001B
GID:0xfe80000000000000,0x0005ad00000281a7
Mar 19 18:39:07 946406 [42803960] -> SUBNET UP
Mar 19 18:39:08 073735 [41E02960] -> __osm_trap_rcv_process_request:
Received Generic Notice type:0x01 num:128 Producer:2 from LID:0x0152
TID:0x0000000000000020
Mar 19 18:39:08 073811 [41E02960] -> __osm_trap_rcv_process_request: ERR
3804: Received trap 33 times consecutively
Mar 19 18:39:08 400790 [43204960] -> osm_ucast_mgr_process: Min Hop Tables
configured on all switches
Mar 19 18:39:08 467925 [45A08960] -> SUBNET UP
Mar 19 20:24:07 009911 [42803960] -> __osm_trap_rcv_process_request:
Received Generic Notice type:0x01 num:128 Producer:2 from LID:0x0020
TID:0x0000000000000020
Mar 19 20:24:07 010153 [42803960] -> osm_report_notice: Reporting Generic
Notice type:1 num:128 from LID:0x0020
GID:0xfe80000000000000,0x0005ad00000281ad
Mar 19 20:24:07 010966 [41401960] -> __osm_trap_rcv_process_request:
Received Generic Notice type:0x01 num:128 Producer:2 from LID:0x0001
TID:0x000000000000001a
Mar 19 20:24:07 011064 [41401960] -> osm_report_notice: Reporting Generic
Notice type:1 num:128 from LID:0x0001
GID:0xfe80000000000000,0x0005ad0000027c6a
Mar 19 20:24:07 390927 [43204960] -> osm_ucast_mgr_process: Min Hop Tables
configured on all switches
Mar 19 20:24:07 453747 [43204960] -> SUBNET UP
Mar 19 20:24:07 839927 [45007960] -> osm_ucast_mgr_process: Min Hop Tables
configured on all switches
Mar 19 20:24:07 895694 [45A08960] -> SUBNET UP
Mar 19 20:24:08 049066 [41E02960] -> __osm_trap_rcv_process_request:
Received Generic Notice type:0x01 num:128 Producer:2 from LID:0x0001
TID:0x000000000000001a
Mar 19 20:24:08 049322 [41E02960] -> osm_report_notice: Reporting Generic
Notice type:1 num:128 from LID:0x0001
GID:0xfe80000000000000,0x0005ad0000027c6a
Mar 19 20:24:08 433979 [42803960] -> osm_ucast_mgr_process: Min Hop Tables
configured on all switches
Mar 19 20:24:08 487950 [43204960] -> SUBNET UP
Mar 19 20:26:28 608381 [42803960] -> __osm_trap_rcv_process_request:
Received Generic Notice type:0x01 num:128 Producer:2 from LID:0x0020
TID:0x0000000000000021
Mar 19 20:26:28 608406 [44606960] -> __osm_trap_rcv_process_request:
Received Generic Notice type:0x01 num:128 Producer:2 from LID:0x0001
TID:0x000000000000001b
Mar 19 20:26:28 608685 [42803960] -> osm_report_notice: Reporting Generic
Notice type:1 num:128 from LID:0x0020
GID:0xfe80000000000000,0x0005ad00000281ad
Mar 19 20:26:28 608693 [44606960] -> osm_report_notice: Reporting Generic
Notice type:1 num:128 from LID:0x0001
GID:0xfe80000000000000,0x0005ad0000027c6a
Mar 19 20:26:28 972140 [44606960] -> osm_ucast_mgr_process: Min Hop Tables
configured on all switches
Mar 19 20:26:29 028682 [43C05960] -> SUBNET UP
Mar 19 20:26:29 399649 [43204960] -> osm_ucast_mgr_process: Min Hop Tables
configured on all switches
Mar 19 20:26:29 465737 [45007960] -> SUBNET UP
Mar 19 21:30:38 775260 [45007960] -> __osm_trap_rcv_process_request:
Received Generic Notice type:0x01 num:128 Producer:2 from LID:0x0146
TID:0x000000000000002f
Mar 19 21:30:38 775533 [45007960] -> osm_report_notice: Reporting Generic
Notice type:1 num:128 from LID:0x0146
GID:0xfe80000000000000,0x0005ad00000281b6
Mar 19 21:30:38 777083 [45007960] -> __osm_trap_rcv_process_request:
Received Generic Notice type:0x01 num:128 Producer:2 from LID:0x0143
TID:0x0000000000000037
Mar 19 21:30:38 777242 [45007960] -> osm_report_notice: Reporting Generic
Notice type:1 num:128 from LID:0x0143
GID:0xfe80000000000000,0x0005ad00000281b9
Mar 19 21:30:39 144779 [43C05960] -> osm_ucast_mgr_process: Min Hop Tables
configured on all switches
Mar 19 21:30:39 200635 [43204960] -> SUBNET UP
Mar 19 21:30:39 536003 [43C05960] -> osm_ucast_mgr_process: Min Hop Tables
configured on all switches
Mar 19 21:30:39 591216 [42803960] -> SUBNET UP
Mar 20 14:06:48 971082 [41401960] -> __osm_trap_rcv_process_request:
Received Generic Notice type:0x01 num:128 Producer:2 from LID:0x0152
TID:0x0000000000000021
Mar 20 14:06:48 971376 [41401960] -> osm_report_notice: Reporting Generic
Notice type:1 num:128 from LID:0x0152
GID:0xfe80000000000000,0x0005ad0000027c84
Mar 20 14:06:49 346734 [42803960] -> osm_report_notice: Reporting Generic
Notice type:3 num:65 from LID:0x0092
GID:0xfe80000000000000,0x0005ad0000024bbb
Mar 20 14:06:49 346761 [42803960] -> Removed port with
GUID:0x0005ad0000024b27 LID range [0xAF,0xAF] of node:saguaro-23-0 HCA-1
Mar 20 14:06:49 381394 [42803960] -> osm_ucast_mgr_process: Min Hop Tables
configured on all switches
Mar 20 14:06:49 440803 [43204960] -> SUBNET UP
Mar 20 14:07:09 098449 [44606960] -> __osm_trap_rcv_process_request:
Received Generic Notice type:0x01 num:128 Producer:2 from LID:0x0148
TID:0x0000000000000048
Mar 20 14:07:09 098708 [44606960] -> osm_report_notice: Reporting Generic
Notice type:1 num:128 from LID:0x0148
GID:0xfe80000000000000,0x0005ad00000281b3
Mar 20 14:07:09 098733 [41E02960] -> __osm_trap_rcv_process_request:
Received Generic Notice type:0x01 num:128 Producer:2 from LID:0x001B
TID:0x000000000000005a
Mar 20 14:07:09 098777 [41E02960] -> osm_report_notice: Reporting Generic
Notice type:1 num:128 from LID:0x001B
GID:0xfe80000000000000,0x0005ad00000281a7
Mar 20 14:07:09 417844 [42803960] -> osm_report_notice: Reporting Generic
Notice type:3 num:65 from LID:0x0092
GID:0xfe80000000000000,0x0005ad0000024bbb
Mar 20 14:07:09 417862 [42803960] -> Removed port with
GUID:0x0005ad0000024e0b LID range [0xB3,0xB3] of node:saguaro-23-4 HCA-1
Mar 20 14:07:09 417879 [42803960] -> osm_report_notice: Reporting Generic
Notice type:3 num:65 from LID:0x0092
GID:0xfe80000000000000,0x0005ad0000024bbb
Mar 20 14:07:09 417885 [42803960] -> Removed port with
GUID:0x0005ad000002510b LID range [0xB5,0xB5] of node:saguaro-23-6 HCA-1
Mar 20 14:07:09 417902 [42803960] -> osm_report_notice: Reporting Generic
Notice type:3 num:65 from LID:0x0092
GID:0xfe80000000000000,0x0005ad0000024bbb
Mar 20 14:07:09 417907 [42803960] -> Removed port with
GUID:0x0005ad000002511b LID range [0xA6,0xA6] of node:saguaro-22-1 HCA-1
Mar 20 14:07:09 417924 [42803960] -> osm_report_notice: Reporting Generic
Notice type:3 num:65 from LID:0x0092
GID:0xfe80000000000000,0x0005ad0000024bbb
Mar 20 14:07:09 417929 [42803960] -> Removed port with
GUID:0x0005ad0000025043 LID range [0xB4,0xB4] of node:saguaro-23-5 HCA-1
Mar 20 14:07:09 417945 [42803960] -> osm_report_notice: Reporting Generic
Notice type:3 num:65 from LID:0x0092
GID:0xfe80000000000000,0x0005ad0000024bbb
Mar 20 14:07:09 417951 [42803960] -> Removed port with
GUID:0x0005ad0000024d47 LID range [0xB6,0xB6] of node:saguaro-23-7 HCA-1
Mar 20 14:07:09 417967 [42803960] -> osm_report_notice: Reporting Generic
Notice type:3 num:65 from LID:0x0092
GID:0xfe80000000000000,0x0005ad0000024bbb
Mar 20 14:07:09 417973 [42803960] -> Removed port with
GUID:0x0005ad0000024d6b LID range [0xB8,0xB8] of node:saguaro-23-9 HCA-1
Mar 20 14:07:09 417989 [42803960] -> osm_report_notice: Reporting Generic
Notice type:3 num:65 from LID:0x0092
GID:0xfe80000000000000,0x0005ad0000024bbb
Mar 20 14:07:09 417994 [42803960] -> Removed port with
GUID:0x0005ad0000024977 LID range [0xA9,0xA9] of node:saguaro-22-4 HCA-1
Mar 20 14:07:09 418131 [42803960] -> osm_report_notice: Reporting Generic
Notice type:3 num:65 from LID:0x0092
GID:0xfe80000000000000,0x0005ad0000024bbb
Mar 20 14:07:09 418137 [42803960] -> Removed port with
GUID:0x0005ad0000027c84 LID range [0x152,0x152] of node:Topspin Switch TS120
Mar 20 14:07:09 418168 [42803960] -> osm_report_notice: Reporting Generic
Notice type:3 num:65 from LID:0x0092
GID:0xfe80000000000000,0x0005ad0000024bbb
Mar 20 14:07:09 418173 [42803960] -> Removed port with
GUID:0x0005ad0000024d8b LID range [0xB7,0xB7] of node:saguaro-23-8 HCA-1
Mar 20 14:07:09 418188 [42803960] -> osm_report_notice: Reporting Generic
Notice type:3 num:65 from LID:0x0092
GID:0xfe80000000000000,0x0005ad0000024bbb
Mar 20 14:07:09 418193 [42803960] -> Removed port with
GUID:0x0005ad000002498f LID range [0xA8,0xA8] of node:saguaro-22-3 HCA-1
Mar 20 14:07:09 418207 [42803960] -> osm_report_notice: Reporting Generic
Notice type:3 num:65 from LID:0x0092
GID:0xfe80000000000000,0x0005ad0000024bbb
Mar 20 14:07:09 418212 [42803960] -> Removed port with
GUID:0x0005ad0000024c9b LID range [0xA7,0xA7] of node:saguaro-22-2 HCA-1
Mar 20 14:07:09 418227 [42803960] -> osm_report_notice: Reporting Generic
Notice type:3 num:65 from LID:0x0092
GID:0xfe80000000000000,0x0005ad0000024bbb
Mar 20 14:07:09 418232 [42803960] -> Removed port with
GUID:0x0005ad0000024da7 LID range [0xB0,0xB0] of node:saguaro-23-1 HCA-1
Mar 20 14:07:09 418248 [42803960] -> osm_report_notice: Reporting Generic
Notice type:3 num:65 from LID:0x0092
GID:0xfe80000000000000,0x0005ad0000024bbb
Mar 20 14:07:09 418253 [42803960] -> Removed port with
GUID:0x0005ad0000024cbb LID range [0xB2,0xB2] of node:saguaro-23-3 HCA-1
Mar 20 14:07:09 418285 [42803960] -> osm_report_notice: Reporting Generic
Notice type:3 num:65 from LID:0x0092
GID:0xfe80000000000000,0x0005ad0000024bbb
Mar 20 14:07:09 418290 [42803960] -> Removed port with
GUID:0x0005ad00000249d3 LID range [0xB1,0xB1] of node:saguaro-23-2 HCA-1
Mar 20 14:07:09 418306 [42803960] -> osm_report_notice: Reporting Generic
Notice type:3 num:65 from LID:0x0092
GID:0xfe80000000000000,0x0005ad0000024bbb
Mar 20 14:07:09 418362 [42803960] -> Removed port with
GUID:0x0005ad0000024feb LID range [0x153,0x153] of node:saguaro-22-5 HCA-1
Mar 20 14:07:09 418378 [42803960] -> osm_report_notice: Reporting Generic
Notice type:3 num:65 from LID:0x0092
GID:0xfe80000000000000,0x0005ad0000024bbb
Mar 20 14:07:09 418383 [42803960] -> Removed port with
GUID:0x0005ad0000024afb LID range [0xA5,0xA5] of node:saguaro-22-0 HCA-1
Mar 20 14:07:09 451317 [42803960] -> osm_ucast_mgr_process: Min Hop Tables
configured on all switches
Mar 20 14:07:09 502755 [41401960] -> SUBNET UP
Mar 20 14:07:09 902534 [41E02960] -> osm_ucast_mgr_process: Min Hop Tables
configured on all switches
Mar 20 14:07:09 955229 [45A08960] -> SUBNET UP
Mar 20 14:08:03 850926 [45A08960] -> __osm_trap_rcv_process_request:
Received Generic Notice type:0x01 num:128 Producer:2 from LID:0x0148
TID:0x0000000000000049
Mar 20 14:08:03 851134 [45A08960] -> osm_report_notice: Reporting Generic
Notice type:1 num:128 from LID:0x0148
GID:0xfe80000000000000,0x0005ad00000281b3
Mar 20 14:08:03 856880 [43204960] -> __osm_trap_rcv_process_request:
Received Generic Notice type:0x01 num:128 Producer:2 from LID:0x0148
TID:0x000000000000004a
Mar 20 14:08:03 856955 [43204960] -> osm_report_notice: Reporting Generic
Notice type:1 num:128 from LID:0x0148
GID:0xfe80000000000000,0x0005ad00000281b3
Mar 20 14:08:03 866819 [42803960] -> __osm_trap_rcv_process_request:
Received Generic Notice type:0x01 num:128 Producer:2 from LID:0x001B
TID:0x000000000000005b
Mar 20 14:08:03 866977 [42803960] -> osm_report_notice: Reporting Generic
Notice type:1 num:128 from LID:0x001B
GID:0xfe80000000000000,0x0005ad00000281a7
Mar 20 14:08:03 963024 [45A08960] -> __osm_trap_rcv_process_request:
Received Generic Notice type:0x01 num:128 Producer:2 from LID:0x001B
TID:0x000000000000005c
Mar 20 14:08:03 963130 [45A08960] -> osm_report_notice: Reporting Generic
Notice type:1 num:128 from LID:0x001B
GID:0xfe80000000000000,0x0005ad00000281a7
Mar 20 14:08:04 106856 [43C05960] -> __osm_trap_rcv_process_request:
Received Generic Notice type:0x01 num:128 Producer:2 from LID:0x001B
TID:0x000000000000005d
Mar 20 14:08:04 106995 [43C05960] -> osm_report_notice: Reporting Generic
Notice type:1 num:128 from LID:0x001B
GID:0xfe80000000000000,0x0005ad00000281a7
Mar 20 14:08:04 193747 [44606960] -> osm_report_notice: Reporting Generic
Notice type:3 num:64 from LID:0x0092
GID:0xfe80000000000000,0x0005ad0000024bbb
Mar 20 14:08:04 193766 [44606960] -> Discovered new port with
GUID:0x0005ad0000027c84 LID range [0x152,0x152] of node:Topspin Switch TS120
Mar 20 14:08:04 193771 [44606960] -> osm_report_notice: Reporting Generic
Notice type:3 num:64 from LID:0x0092
GID:0xfe80000000000000,0x0005ad0000024bbb
Mar 20 14:08:04 193777 [44606960] -> Discovered new port with
GUID:0x0005ad0000024da7 LID range [0xB0,0xB0] of node:saguaro-23-1 HCA-1
Mar 20 14:08:04 193781 [44606960] -> osm_report_notice: Reporting Generic
Notice type:3 num:64 from LID:0x0092
GID:0xfe80000000000000,0x0005ad0000024bbb
Mar 20 14:08:04 193786 [44606960] -> Discovered new port with
GUID:0x0005ad00000249d3 LID range [0xB1,0xB1] of node:saguaro-23-2 HCA-1
Mar 20 14:08:04 193790 [44606960] -> osm_report_notice: Reporting Generic
Notice type:3 num:64 from LID:0x0092
GID:0xfe80000000000000,0x0005ad0000024bbb
Mar 20 14:08:04 193795 [44606960] -> Discovered new port with
GUID:0x0005ad0000024cbb LID range [0xB2,0xB2] of node:saguaro-23-3 HCA-1
Mar 20 14:08:04 193799 [44606960] -> osm_report_notice: Reporting Generic
Notice type:3 num:64 from LID:0x0092
GID:0xfe80000000000000,0x0005ad0000024bbb
Mar 20 14:08:04 193804 [44606960] -> Discovered new port with
GUID:0x0005ad0000024e0b LID range [0xB3,0xB3] of node:saguaro-23-4 HCA-1
Mar 20 14:08:04 193808 [44606960] -> osm_report_notice: Reporting Generic
Notice type:3 num:64 from LID:0x0092
GID:0xfe80000000000000,0x0005ad0000024bbb
Mar 20 14:08:04 193813 [44606960] -> Discovered new port with
GUID:0x0005ad0000025043 LID range [0xB4,0xB4] of node:saguaro-23-5 HCA-1
Mar 20 14:08:04 193817 [44606960] -> osm_report_notice: Reporting Generic
Notice type:3 num:64 from LID:0x0092
GID:0xfe80000000000000,0x0005ad0000024bbb
Mar 20 14:08:04 193822 [44606960] -> Discovered new port with
GUID:0x0005ad000002510b LID range [0xB5,0xB5] of node:saguaro-23-6 HCA-1
Mar 20 14:08:04 193826 [44606960] -> osm_report_notice: Reporting Generic
Notice type:3 num:64 from LID:0x0092
GID:0xfe80000000000000,0x0005ad0000024bbb
Mar 20 14:08:04 193830 [44606960] -> Discovered new port with
GUID:0x0005ad0000024d8b LID range [0xB7,0xB7] of node:saguaro-23-8 HCA-1
Mar 20 14:08:04 193834 [44606960] -> osm_report_notice: Reporting Generic
Notice type:3 num:64 from LID:0x0092
GID:0xfe80000000000000,0x0005ad0000024bbb
Mar 20 14:08:04 193839 [44606960] -> Discovered new port with
GUID:0x0005ad0000024d47 LID range [0xB6,0xB6] of node:saguaro-23-7 HCA-1
Mar 20 14:08:04 193843 [44606960] -> osm_report_notice: Reporting Generic
Notice type:3 num:64 from LID:0x0092
GID:0xfe80000000000000,0x0005ad0000024bbb
Mar 20 14:08:04 193848 [44606960] -> Discovered new port with
GUID:0x0005ad0000024d6b LID range [0xB8,0xB8] of node:saguaro-23-9 HCA-1
Mar 20 14:08:04 193852 [44606960] -> osm_report_notice: Reporting Generic
Notice type:3 num:64 from LID:0x0092
GID:0xfe80000000000000,0x0005ad0000024bbb
Mar 20 14:08:04 193857 [44606960] -> Discovered new port with
GUID:0x0005ad0000024afb LID range [0xA5,0xA5] of node:saguaro-22-0 HCA-1
Mar 20 14:08:04 193861 [44606960] -> osm_report_notice: Reporting Generic
Notice type:3 num:64 from LID:0x0092
GID:0xfe80000000000000,0x0005ad0000024bbb
Mar 20 14:08:04 193866 [44606960] -> Discovered new port with
GUID:0x0005ad000002511b LID range [0xA6,0xA6] of node:saguaro-22-1 HCA-1
Mar 20 14:08:04 193870 [44606960] -> osm_report_notice: Reporting Generic
Notice type:3 num:64 from LID:0x0092
GID:0xfe80000000000000,0x0005ad0000024bbb
Mar 20 14:08:04 193874 [44606960] -> Discovered new port with
GUID:0x0005ad0000024c9b LID range [0xA7,0xA7] of node:saguaro-22-2 HCA-1
Mar 20 14:08:04 193878 [44606960] -> osm_report_notice: Reporting Generic
Notice type:3 num:64 from LID:0x0092
GID:0xfe80000000000000,0x0005ad0000024bbb
Mar 20 14:08:04 193883 [44606960] -> Discovered new port with
GUID:0x0005ad000002498f LID range [0xA8,0xA8] of node:saguaro-22-3 HCA-1
Mar 20 14:08:04 193938 [44606960] -> osm_report_notice: Reporting Generic
Notice type:3 num:64 from LID:0x0092
GID:0xfe80000000000000,0x0005ad0000024bbb
Mar 20 14:08:04 193944 [44606960] -> Discovered new port with
GUID:0x0005ad0000024977 LID range [0xA9,0xA9] of node:saguaro-22-4 HCA-1
Mar 20 14:08:04 193948 [44606960] -> osm_report_notice: Reporting Generic
Notice type:3 num:64 from LID:0x0092
GID:0xfe80000000000000,0x0005ad0000024bbb
Mar 20 14:08:04 193953 [44606960] -> Discovered new port with
GUID:0x0005ad0000024feb LID range [0x153,0x153] of node:saguaro-22-5 HCA-1
Mar 20 14:08:04 224695 [44606960] -> osm_ucast_mgr_process: Min Hop Tables
configured on all switches
Mar 20 14:08:04 281046 [4780B960] -> __osm_sm_mad_ctrl_rcv_callback: ERR
3111: Error status = 0x1C00
Mar 20 14:08:04 281106 [4780B960] -> SMP dump:
base_ver................0x1
mgmt_class..............0x81
class_ver...............0x1
method..................0x81 (SubnGetResp)
D bit...................0x1
status..................0x1C00
hop_ptr.................0x0
hop_count...............0x4
trans_id................0x61eec
attr_id.................0x15 (PortInfo)
resv....................0x0
attr_mod................0x13
m_key...................0x0000000000000000
dr_slid.................0xFFFF
dr_dlid.................0xFFFF
Initial path: [0][1][17][2][4]
Return path: [0][9][14][E][1]
Reserved: [0][0][0][0][0][0][0]
00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
00 00 00 00 00 00 00 00 00 00 00 00 01 03 03 02
14 52 00 11 40 40 00 08 08 04 F2 40 00 00 00 00
00 00 00 00 00 88 00 00 00 00 00 00 00 00 00 00
Mar 20 14:08:04 281154 [41401960] -> osm_pi_rcv_process_set: Received error
status 0x1c for SetResp() during ACTIVE transition
Mar 20 14:08:04 281159 [41401960] -> PortInfo dump:
port number.............0x13
node_guid...............0x0005ad00000281a7
port_guid...............0x0005ad00000281a7
m_key...................0x0000000000000000
subnet_prefix...........0x0000000000000000
base_lid................0x0
master_sm_base_lid......0x0
capability_mask.........0x0
diag_code...............0x0
m_key_lease_period......0x0
local_port_num..........0x1
link_width_enabled......0x3
link_width_supported....0x3
link_width_active.......0x2
link_speed_supported....0x1
port_state..............ACTIVE
state_info2.............0x52
m_key_protect_bits......0x0
lmc.....................0x0
link_speed..............0x11
mtu_smsl................0x40
vl_cap_init_type........0x40
vl_high_limit...........0x0
vl_arb_high_cap.........0x8
vl_arb_low_cap..........0x8
init_rep_mtu_cap........0x4
vl_stall_life...........0xF2
vl_enforce..............0x40
m_key_violations........0x0
p_key_violations........0x0
q_key_violations........0x0
guid_cap................0x0
client_reregister.......0x0
subnet_timeout..........0x0
resp_time_value.........0x0
error_threshold.........0x88
Mar 20 14:08:04 281172 [41401960] -> Capabilities Mask:
Mar 20 14:08:04 281187 [4780B960] -> __osm_sm_mad_ctrl_rcv_callback: ERR
3111: Error status = 0x1C00
Mar 20 14:08:04 281213 [4780B960] -> SMP dump:
base_ver................0x1
mgmt_class..............0x81
class_ver...............0x1
method..................0x81 (SubnGetResp)
D bit...................0x1
status..................0x1C00
hop_ptr.................0x0
hop_count...............0x4
trans_id................0x61eed
attr_id.................0x15 (PortInfo)
resv....................0x0
attr_mod................0x17
m_key...................0x0000000000000000
dr_slid.................0xFFFF
dr_dlid.................0xFFFF
Initial path: [0][1][17][2][4]
Return path: [0][9][14][E][1]
Reserved: [0][0][0][0][0][0][0]
00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
00 00 00 00 00 00 00 00 00 00 00 00 01 03 03 02
14 52 00 11 40 40 00 08 08 04 F2 40 00 00 00 00
00 00 00 00 00 88 00 00 00 00 00 00 00 00 00 00
Mar 20 14:08:04 281279 [4780B960] -> __osm_sm_mad_ctrl_rcv_callback: ERR
3111: Error status = 0x1C00
Mar 20 14:08:04 281316 [4780B960] -> SMP dump:
base_ver................0x1
mgmt_class..............0x81
class_ver...............0x1
method..................0x81 (SubnGetResp)
D bit...................0x1
status..................0x1C00
hop_ptr.................0x0
hop_count...............0x4
trans_id................0x61eee
attr_id.................0x15 (PortInfo)
resv....................0x0
attr_mod................0x18
m_key...................0x0000000000000000
dr_slid.................0xFFFF
dr_dlid.................0xFFFF
Initial path: [0][1][17][2][4]
Return path: [0][9][14][E][1]
Reserved: [0][0][0][0][0][0][0]
00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
00 00 00 00 00 00 00 00 00 00 00 00 01 03 03 02
14 52 00 11 40 40 00 08 08 04 F2 40 00 00 00 00
00 00 00 00 00 88 00 00 00 00 00 00 00 00 00 00
Mar 20 14:08:04 281392 [4780B960] -> __osm_sm_mad_ctrl_rcv_callback: ERR
3111: Error status = 0x1C00
Mar 20 14:08:04 281416 [4780B960] -> SMP dump:
base_ver................0x1
mgmt_class..............0x81
class_ver...............0x1
method..................0x81 (SubnGetResp)
D bit...................0x1
status..................0x1C00
hop_ptr.................0x0
hop_count...............0x4
trans_id................0x61eef
attr_id.................0x15 (PortInfo)
resv....................0x0
attr_mod................0x16
m_key...................0x0000000000000000
dr_slid.................0xFFFF
dr_dlid.................0xFFFF
Initial path: [0][1][11][1][6]
Return path: [0][9][18][D][3]
Reserved: [0][0][0][0][0][0][0]
00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
00 00 00 00 00 00 00 00 00 00 00 00 03 03 03 02
14 52 00 11 40 40 00 08 08 04 F2 40 00 00 00 00
00 00 00 00 00 88 00 00 00 00 00 00 00 00 00 00
Mar 20 14:08:04 281515 [44606960] -> osm_pi_rcv_process_set: Received error
status 0x1c for SetResp() during ACTIVE transition
Mar 20 14:08:04 281522 [4780B960] -> __osm_sm_mad_ctrl_rcv_callback: ERR
3111: Error status = 0x1C00
Mar 20 14:08:04 281542 [44606960] -> PortInfo dump:
port number.............0x17
node_guid...............0x0005ad00000281a7
port_guid...............0x0005ad00000281a7
m_key...................0x0000000000000000
subnet_prefix...........0x0000000000000000
base_lid................0x0
master_sm_base_lid......0x0
capability_mask.........0x0
diag_code...............0x0
m_key_lease_period......0x0
local_port_num..........0x1
link_width_enabled......0x3
link_width_supported....0x3
link_width_active.......0x2
link_speed_supported....0x1
port_state..............ACTIVE
state_info2.............0x52
m_key_protect_bits......0x0
lmc.....................0x0
link_speed..............0x11
mtu_smsl................0x40
vl_cap_init_type........0x40
vl_high_limit...........0x0
vl_arb_high_cap.........0x8
vl_arb_low_cap..........0x8
init_rep_mtu_cap........0x4
vl_stall_life...........0xF2
vl_enforce..............0x40
m_key_violations........0x0
p_key_violations........0x0
q_key_violations........0x0
guid_cap................0x0
client_reregister.......0x0
subnet_timeout..........0x0
resp_time_value.........0x0
error_threshold.........0x88
Mar 20 14:08:04 281553 [44606960] -> Capabilities Mask:
Mar 20 14:08:04 281561 [4780B960] -> SMP dump:
base_ver................0x1
mgmt_class..............0x81
class_ver...............0x1
method..................0x81 (SubnGetResp)
D bit...................0x1
status..................0x1C00
hop_ptr.................0x0
hop_count...............0x4
trans_id................0x61ef0
attr_id.................0x15 (PortInfo)
resv....................0x0
attr_mod................0x17
m_key...................0x0000000000000000
dr_slid.................0xFFFF
dr_dlid.................0xFFFF
Initial path: [0][1][11][1][6]
Return path: [0][9][18][D][3]
Reserved: [0][0][0][0][0][0][0]
00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
00 00 00 00 00 00 00 00 00 00 00 00 03 03 03 02
14 52 00 11 40 40 00 08 08 04 F2 40 00 00 00 00
00 00 00 00 00 88 00 00 00 00 00 00 00 00 00 00
Mar 20 14:08:04 281572 [44606960] -> osm_pi_rcv_process_set: Received error
status 0x1c for SetResp() during ACTIVE transition
Mar 20 14:08:04 281590 [44606960] -> PortInfo dump:
port number.............0x18
node_guid...............0x0005ad00000281a7
port_guid...............0x0005ad00000281a7
m_key...................0x0000000000000000
subnet_prefix...........0x0000000000000000
base_lid................0x0
master_sm_base_lid......0x0
capability_mask.........0x0
diag_code...............0x0
m_key_lease_period......0x0
local_port_num..........0x1
link_width_enabled......0x3
link_width_supported....0x3
link_width_active.......0x2
link_speed_supported....0x1
port_state..............ACTIVE
state_info2.............0x52
m_key_protect_bits......0x0
lmc.....................0x0
link_speed..............0x11
mtu_smsl................0x40
vl_cap_init_type........0x40
vl_high_limit...........0x0
vl_arb_high_cap.........0x8
vl_arb_low_cap..........0x8
init_rep_mtu_cap........0x4
vl_stall_life...........0xF2
vl_enforce..............0x40
m_key_violations........0x0
p_key_violations........0x0
q_key_violations........0x0
guid_cap................0x0
client_reregister.......0x0
subnet_timeout..........0x0
resp_time_value.........0x0
error_threshold.........0x88
Mar 20 14:08:04 281600 [44606960] -> Capabilities Mask:
Mar 20 14:08:04 281623 [44606960] -> osm_pi_rcv_process_set: Received error
status 0x1c for SetResp() during ACTIVE transition
Mar 20 14:08:04 281626 [44606960] -> PortInfo dump:
port number.............0x16
node_guid...............0x0005ad00000281b3
port_guid...............0x0005ad00000281b3
m_key...................0x0000000000000000
subnet_prefix...........0x0000000000000000
base_lid................0x0
master_sm_base_lid......0x0
capability_mask.........0x0
diag_code...............0x0
m_key_lease_period......0x0
local_port_num..........0x3
link_width_enabled......0x3
link_width_supported....0x3
link_width_active.......0x2
link_speed_supported....0x1
port_state..............ACTIVE
state_info2.............0x52
m_key_protect_bits......0x0
lmc.....................0x0
link_speed..............0x11
mtu_smsl................0x40
vl_cap_init_type........0x40
vl_high_limit...........0x0
vl_arb_high_cap.........0x8
vl_arb_low_cap..........0x8
init_rep_mtu_cap........0x4
vl_stall_life...........0xF2
vl_enforce..............0x40
m_key_violations........0x0
p_key_violations........0x0
q_key_violations........0x0
guid_cap................0x0
client_reregister.......0x0
subnet_timeout..........0x0
resp_time_value.........0x0
error_threshold.........0x88
Mar 20 14:08:04 281635 [44606960] -> Capabilities Mask:
Mar 20 14:08:04 281637 [4780B960] -> __osm_sm_mad_ctrl_rcv_callback: ERR
3111: Error status = 0x1C00
Mar 20 14:08:04 281652 [44606960] -> osm_pi_rcv_process_set: Received error
status 0x1c for SetResp() during ACTIVE transition
Mar 20 14:08:04 281663 [44606960] -> PortInfo dump:
port number.............0x17
node_guid...............0x0005ad00000281b3
port_guid...............0x0005ad00000281b3
m_key...................0x0000000000000000
subnet_prefix...........0x0000000000000000
base_lid................0x0
master_sm_base_lid......0x0
capability_mask.........0x0
diag_code...............0x0
m_key_lease_period......0x0
local_port_num..........0x3
link_width_enabled......0x3
link_width_supported....0x3
link_width_active.......0x2
link_speed_supported....0x1
port_state..............ACTIVE
state_info2.............0x52
m_key_protect_bits......0x0
lmc.....................0x0
link_speed..............0x11
mtu_smsl................0x40
vl_cap_init_type........0x40
vl_high_limit...........0x0
vl_arb_high_cap.........0x8
vl_arb_low_cap..........0x8
init_rep_mtu_cap........0x4
vl_stall_life...........0xF2
vl_enforce..............0x40
m_key_violations........0x0
p_key_violations........0x0
q_key_violations........0x0
guid_cap................0x0
client_reregister.......0x0
subnet_timeout..........0x0
resp_time_value.........0x0
error_threshold.........0x88
Mar 20 14:08:04 281673 [44606960] -> Capabilities Mask:
Mar 20 14:08:04 281675 [4780B960] -> SMP dump:
base_ver................0x1
mgmt_class..............0x81
class_ver...............0x1
method..................0x81 (SubnGetResp)
D bit...................0x1
status..................0x1C00
hop_ptr.................0x0
hop_count...............0x4
trans_id................0x61ef1
attr_id.................0x15 (PortInfo)
resv....................0x0
attr_mod................0x18
m_key...................0x0000000000000000
dr_slid.................0xFFFF
dr_dlid.................0xFFFF
Initial path: [0][1][11][1][6]
Return path: [0][9][18][D][3]
Reserved: [0][0][0][0][0][0][0]
00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
00 00 00 00 00 00 00 00 00 00 00 00 03 03 03 02
14 52 00 11 40 40 00 08 08 04 F2 40 00 00 00 00
00 00 00 00 00 88 00 00 00 00 00 00 00 00 00 00
Mar 20 14:08:04 281721 [41E02960] -> osm_pi_rcv_process_set: Received error
status 0x1c for SetResp() during ACTIVE transition
Mar 20 14:08:04 281726 [41E02960] -> PortInfo dump:
port number.............0x18
node_guid...............0x0005ad00000281b3
port_guid...............0x0005ad00000281b3
m_key...................0x0000000000000000
subnet_prefix...........0x0000000000000000
base_lid................0x0
master_sm_base_lid......0x0
capability_mask.........0x0
diag_code...............0x0
m_key_lease_period......0x0
local_port_num..........0x3
link_width_enabled......0x3
link_width_supported....0x3
link_width_active.......0x2
link_speed_supported....0x1
port_state..............ACTIVE
state_info2.............0x52
m_key_protect_bits......0x0
lmc.....................0x0
link_speed..............0x11
mtu_smsl................0x40
vl_cap_init_type........0x40
vl_high_limit...........0x0
vl_arb_high_cap.........0x8
vl_arb_low_cap..........0x8
init_rep_mtu_cap........0x4
vl_stall_life...........0xF2
vl_enforce..............0x40
m_key_violations........0x0
p_key_violations........0x0
q_key_violations........0x0
guid_cap................0x0
client_reregister.......0x0
subnet_timeout..........0x0
resp_time_value.........0x0
error_threshold.........0x88
Mar 20 14:08:04 281736 [41E02960] -> Capabilities Mask:
Mar 20 14:08:04 287136 [44606960] -> SUBNET UP
Mar 20 14:08:04 711595 [43C05960] -> osm_ucast_mgr_process: Min Hop Tables
configured on all switches
Mar 20 14:08:04 766488 [45A08960] -> SUBNET UP
Mar 20 14:08:19 947200 [43204960] -> __osm_trap_rcv_process_request:
Received Generic Notice type:0x01 num:128 Producer:2 from LID:0x0152
TID:0x0000000000000000
Mar 20 14:08:19 947479 [43204960] -> osm_report_notice: Reporting Generic
Notice type:1 num:128 from LID:0x0152
GID:0xfe80000000000000,0x0005ad0000027c84
Mar 20 14:08:20 086909 [41E02960] -> __osm_trap_rcv_process_request:
Received Generic Notice type:0x01 num:128 Producer:2 from LID:0x0152
TID:0x0000000000000001
Mar 20 14:08:20 087084 [41E02960] -> osm_report_notice: Reporting Generic
Notice type:1 num:128 from LID:0x0152
GID:0xfe80000000000000,0x0005ad0000027c84
Mar 20 14:08:20 108865 [41401960] -> __osm_trap_rcv_process_request:
Received Generic Notice type:0x01 num:128 Producer:2 from LID:0x0152
TID:0x0000000000000002
Mar 20 14:08:20 109210 [41401960] -> osm_report_notice: Reporting Generic
Notice type:1 num:128 from LID:0x0152
GID:0xfe80000000000000,0x0005ad0000027c84
Mar 20 14:08:20 109996 [41E02960] -> __osm_trap_rcv_process_request:
Received Generic Notice type:0x01 num:128 Producer:2 from LID:0x0152
TID:0x0000000000000003
Mar 20 14:08:20 110407 [41E02960] -> osm_report_notice: Reporting Generic
Notice type:1 num:128 from LID:0x0152
GID:0xfe80000000000000,0x0005ad0000027c84
Mar 20 14:08:20 222523 [45A08960] -> __osm_trap_rcv_process_request:
Received Generic Notice type:0x01 num:128 Producer:2 from LID:0x0152
TID:0x0000000000000004
Mar 20 14:08:20 222613 [45A08960] -> osm_report_notice: Reporting Generic
Notice type:1 num:128 from LID:0x0152
GID:0xfe80000000000000,0x0005ad0000027c84
Mar 20 14:08:20 404596 [41401960] -> __osm_trap_rcv_process_request:
Received Generic Notice type:0x01 num:128 Producer:2 from LID:0x0152
TID:0x0000000000000005
Mar 20 14:08:20 404698 [41401960] -> osm_report_notice: Reporting Generic
Notice type:1 num:128 from LID:0x0152
GID:0xfe80000000000000,0x0005ad0000027c84
Mar 20 14:08:20 476804 [45007960] -> __osm_trap_rcv_process_request:
Received Generic Notice type:0x01 num:128 Producer:2 from LID:0x0152
TID:0x0000000000000006
Mar 20 14:08:20 476897 [45007960] -> osm_report_notice: Reporting Generic
Notice type:1 num:128 from LID:0x0152
GID:0xfe80000000000000,0x0005ad0000027c84
Mar 20 14:08:20 572434 [44606960] -> __osm_trap_rcv_process_request:
Received Generic Notice type:0x01 num:128 Producer:2 from LID:0x0152
TID:0x0000000000000007
Mar 20 14:08:20 572520 [44606960] -> osm_report_notice: Reporting Generic
Notice type:1 num:128 from LID:0x0152
GID:0xfe80000000000000,0x0005ad0000027c84
Mar 20 14:08:20 621715 [42803960] -> osm_report_notice: Reporting Generic
Notice type:3 num:65 from LID:0x0092
GID:0xfe80000000000000,0x0005ad0000024bbb
Mar 20 14:08:20 621726 [42803960] -> Removed port with
GUID:0x0005ad00000249d3 LID range [0xB1,0xB1] of node:saguaro-23-2 HCA-1
Mar 20 14:08:20 656232 [42803960] -> osm_ucast_mgr_process: Min Hop Tables
configured on all switches
Mar 20 14:08:20 698700 [44606960] -> __osm_trap_rcv_process_request:
Received Generic Notice type:0x01 num:128 Producer:2 from LID:0x0152
TID:0x0000000000000008
Mar 20 14:08:20 698794 [44606960] -> osm_report_notice: Reporting Generic
Notice type:1 num:128 from LID:0x0152
GID:0xfe80000000000000,0x0005ad0000027c84
Mar 20 14:08:20 708598 [41401960] -> __osm_trap_rcv_process_request:
Received Generic Notice type:0x01 num:128 Producer:2 from LID:0x0152
TID:0x0000000000000009
Mar 20 14:08:20 708698 [41401960] -> osm_report_notice: Reporting Generic
Notice type:1 num:128 from LID:0x0152
GID:0xfe80000000000000,0x0005ad0000027c84
Mar 20 14:08:20 713653 [45007960] -> SUBNET UP
Mar 20 14:08:20 730554 [44606960] -> __osm_trap_rcv_process_request:
Received Generic Notice type:0x01 num:128 Producer:2 from LID:0x0152
TID:0x000000000000000a
Mar 20 14:08:20 730697 [44606960] -> osm_report_notice: Reporting Generic
Notice type:1 num:128 from LID:0x0152
GID:0xfe80000000000000,0x0005ad0000027c84
Mar 20 14:08:20 754139 [45007960] -> __osm_trap_rcv_process_request:
Received Generic Notice type:0x01 num:128 Producer:2 from LID:0x0152
TID:0x000000000000000b
Mar 20 14:08:20 754251 [45007960] -> __osm_trap_rcv_process_request: ERR
3804: Received trap 11 times consecutively
Mar 20 14:08:20 947339 [41401960] -> __osm_trap_rcv_process_request:
Received Generic Notice type:0x01 num:128 Producer:2 from LID:0x0152
TID:0x000000000000000c
Mar 20 14:08:20 947426 [41401960] -> __osm_trap_rcv_process_request: ERR
3804: Received trap 12 times consecutively
Mar 20 14:08:20 975965 [45A08960] -> __osm_trap_rcv_process_request:
Received Generic Notice type:0x01 num:128 Producer:2 from LID:0x0152
TID:0x000000000000000d
Mar 20 14:08:20 976024 [45A08960] -> __osm_trap_rcv_process_request: ERR
3804: Received trap 13 times consecutively
Mar 20 14:08:20 997569 [43C05960] -> __osm_trap_rcv_process_request:
Received Generic Notice type:0x01 num:128 Producer:2 from LID:0x0152
TID:0x000000000000000e
Mar 20 14:08:20 997648 [43C05960] -> __osm_trap_rcv_process_request: ERR
3804: Received trap 14 times consecutively
Mar 20 14:08:21 019465 [44606960] -> __osm_trap_rcv_process_request:
Received Generic Notice type:0x01 num:128 Producer:2 from LID:0x0152
TID:0x000000000000000f
Mar 20 14:08:21 019512 [44606960] -> __osm_trap_rcv_process_request: ERR
3804: Received trap 15 times consecutively
Mar 20 14:08:21 064967 [43204960] -> __osm_trap_rcv_process_request:
Received Generic Notice type:0x01 num:128 Producer:2 from LID:0x0152
TID:0x0000000000000010
Mar 20 14:08:21 065009 [43204960] -> __osm_trap_rcv_process_request: ERR
3804: Received trap 16 times consecutively
Mar 20 14:08:21 082838 [41401960] -> __osm_trap_rcv_process_request:
Received Generic Notice type:0x01 num:128 Producer:2 from LID:0x0152
TID:0x0000000000000011
Mar 20 14:08:21 082877 [41401960] -> __osm_trap_rcv_process_request: ERR
3804: Received trap 17 times consecutively
Mar 20 14:08:21 100567 [43204960] -> __osm_trap_rcv_process_request:
Received Generic Notice type:0x01 num:128 Producer:2 from LID:0x0152
TID:0x0000000000000012
Mar 20 14:08:21 100619 [43204960] -> __osm_trap_rcv_process_request: ERR
3804: Received trap 18 times consecutively
Mar 20 14:08:21 188128 [43C05960] -> osm_report_notice: Reporting Generic
Notice type:3 num:65 from LID:0x0092
GID:0xfe80000000000000,0x0005ad0000024bbb
Mar 20 14:08:21 188144 [43C05960] -> Removed port with
GUID:0x0005ad000002510b LID range [0xB5,0xB5] of node:saguaro-23-6 HCA-1
Mar 20 14:08:21 188166 [43C05960] -> osm_report_notice: Reporting Generic
Notice type:3 num:65 from LID:0x0092
GID:0xfe80000000000000,0x0005ad0000024bbb
Mar 20 14:08:21 188172 [43C05960] -> Removed port with
GUID:0x0005ad0000024d47 LID range [0xB6,0xB6] of node:saguaro-23-7 HCA-1
Mar 20 14:08:21 188194 [43C05960] -> osm_report_notice: Reporting Generic
Notice type:3 num:65 from LID:0x0092
GID:0xfe80000000000000,0x0005ad0000024bbb
Mar 20 14:08:21 188199 [43C05960] -> Removed port with
GUID:0x0005ad0000024d8b LID range [0xB7,0xB7] of node:saguaro-23-8 HCA-1
Mar 20 14:08:21 192421 [41E02960] -> osm_report_notice: Reporting Generic
Notice type:3 num:64 from LID:0x0092
GID:0xfe80000000000000,0x0005ad0000024bbb
Mar 20 14:08:21 192436 [41E02960] -> Discovered new port with
GUID:0x0005ad00000249d3 LID range [0xB1,0xB1] of node:saguaro-23-2 HCA-1
Mar 20 14:08:21 208455 [41401960] -> __osm_trap_rcv_process_request:
Received Generic Notice type:0x01 num:128 Producer:2 from LID:0x0152
TID:0x0000000000000013
Mar 20 14:08:21 208499 [41401960] -> __osm_trap_rcv_process_request: ERR
3804: Received trap 19 times consecutively
Mar 20 14:08:21 223240 [41E02960] -> osm_ucast_mgr_process: Min Hop Tables
configured on all switches
Mar 20 14:08:21 394585 [45007960] -> __osm_trap_rcv_process_request:
Received Generic Notice type:0x01 num:128 Producer:2 from LID:0x0152
TID:0x0000000000000014
Mar 20 14:08:21 394665 [45007960] -> __osm_trap_rcv_process_request: ERR
3804: Received trap 20 times consecutively
Mar 20 14:08:21 419333 [41E02960] -> __osm_trap_rcv_process_request:
Received Generic Notice type:0x01 num:128 Producer:2 from LID:0x0152
TID:0x0000000000000015
Mar 20 14:08:21 419393 [41E02960] -> __osm_trap_rcv_process_request: ERR
3804: Received trap 21 times consecutively
Mar 20 14:08:21 441228 [43204960] -> __osm_trap_rcv_process_request:
Received Generic Notice type:0x01 num:128 Producer:2 from LID:0x0152
TID:0x0000000000000016
Mar 20 14:08:21 441276 [43204960] -> __osm_trap_rcv_process_request: ERR
3804: Received trap 22 times consecutively
Mar 20 14:08:21 462915 [44606960] -> __osm_trap_rcv_process_request:
Received Generic Notice type:0x01 num:128 Producer:2 from LID:0x0152
TID:0x0000000000000017
Mar 20 14:08:21 462968 [44606960] -> __osm_trap_rcv_process_request: ERR
3804: Received trap 23 times consecutively
Mar 20 14:08:21 475440 [45007960] -> SUBNET UP
Mar 20 14:08:21 674045 [44606960] -> __osm_trap_rcv_process_request:
Received Generic Notice type:0x01 num:128 Producer:2 from LID:0x0152
TID:0x0000000000000018
Mar 20 14:08:21 674084 [43204960] -> __osm_trap_rcv_process_request:
Received Generic Notice type:0x01 num:128 Producer:2 from LID:0x0148
TID:0x000000000000004b
Mar 20 14:08:21 674137 [44606960] -> __osm_trap_rcv_process_request: ERR
3804: Received trap 24 times consecutively
Mar 20 14:08:21 674294 [43204960] -> osm_report_notice: Reporting Generic
Notice type:1 num:128 from LID:0x0148
GID:0xfe80000000000000,0x0005ad00000281b3
Mar 20 14:08:21 965885 [43204960] -> __osm_trap_rcv_process_request:
Received Generic Notice type:0x01 num:128 Producer:2 from LID:0x0148
TID:0x000000000000004c
Mar 20 14:08:21 965992 [43204960] -> osm_report_notice: Reporting Generic
Notice type:1 num:128 from LID:0x0148
GID:0xfe80000000000000,0x0005ad00000281b3
Mar 20 14:08:22 092378 [41401960] -> osm_report_notice: Reporting Generic
Notice type:3 num:65 from LID:0x0092
GID:0xfe80000000000000,0x0005ad0000024bbb
Mar 20 14:08:22 092395 [41401960] -> Removed port with
GUID:0x0005ad0000024977 LID range [0xA9,0xA9] of node:saguaro-22-4 HCA-1
Mar 20 14:08:22 092415 [41401960] -> osm_report_notice: Reporting Generic
Notice type:3 num:65 from LID:0x0092
GID:0xfe80000000000000,0x0005ad0000024bbb
Mar 20 14:08:22 092420 [41401960] -> Removed port with
GUID:0x0005ad0000024c9b LID range [0xA7,0xA7] of node:saguaro-22-2 HCA-1
Mar 20 14:08:22 092444 [41401960] -> osm_report_notice: Reporting Generic
Notice type:3 num:65 from LID:0x0092
GID:0xfe80000000000000,0x0005ad0000024bbb
Mar 20 14:08:22 092449 [41401960] -> Removed port with
GUID:0x0005ad0000024feb LID range [0x153,0x153] of node:saguaro-22-5 HCA-1
Mar 20 14:08:22 092625 [41401960] -> osm_drop_mgr_process: ERR 0108: Unknown
remote side for node 0x0005ad00000281b3 port 22. Adding to light sweep
sampling list
Mar 20 14:08:22 092655 [41401960] -> Directed Path Dump of 4 hop path:
Path = [0][1][11][1][4]
Mar 20 14:08:22 092663 [41401960] -> osm_drop_mgr_process: ERR 0108: Unknown
remote side for node 0x0005ad00000281b3 port 23. Adding to light sweep
sampling list
Mar 20 14:08:22 092672 [41401960] -> Directed Path Dump of 4 hop path:
Path = [0][1][11][1][4]
Mar 20 14:08:22 096789 [41E02960] -> osm_report_notice: Reporting Generic
Notice type:3 num:64 from LID:0x0092
GID:0xfe80000000000000,0x0005ad0000024bbb
Mar 20 14:08:22 096801 [41E02960] -> Discovered new port with
GUID:0x0005ad000002510b LID range [0xB5,0xB5] of node:saguaro-23-6 HCA-1
Mar 20 14:08:22 096805 [41E02960] -> osm_report_notice: Reporting Generic
Notice type:3 num:64 from LID:0x0092
GID:0xfe80000000000000,0x0005ad0000024bbb
Mar 20 14:08:22 096810 [41E02960] -> Discovered new port with
GUID:0x0005ad0000024d47 LID range [0xB6,0xB6] of node:saguaro-23-7 HCA-1
Mar 20 14:08:22 096814 [41E02960] -> osm_report_notice: Reporting Generic
Notice type:3 num:64 from LID:0x0092
GID:0xfe80000000000000,0x0005ad0000024bbb
Mar 20 14:08:22 096819 [41E02960] -> Discovered new port with
GUID:0x0005ad0000024d8b LID range [0xB7,0xB7] of node:saguaro-23-8 HCA-1
Mar 20 14:08:22 127266 [41E02960] -> osm_ucast_mgr_process: Min Hop Tables
configured on all switches
Mar 20 14:08:22 184734 [45007960] -> SUBNET UP
Mar 20 14:08:22 541974 [41401960] -> osm_report_notice: Reporting Generic
Notice type:3 num:64 from LID:0x0092
GID:0xfe80000000000000,0x0005ad0000024bbb
Mar 20 14:08:22 541985 [41401960] -> Discovered new port with
GUID:0x0005ad0000024c9b LID range [0xA7,0xA7] of node:saguaro-22-2 HCA-1
Mar 20 14:08:22 541989 [41401960] -> osm_report_notice: Reporting Generic
Notice type:3 num:64 from LID:0x0092
GID:0xfe80000000000000,0x0005ad0000024bbb
Mar 20 14:08:22 541995 [41401960] -> Discovered new port with
GUID:0x0005ad0000024977 LID range [0xA9,0xA9] of node:saguaro-22-4 HCA-1
Mar 20 14:08:22 541998 [41401960] -> osm_report_notice: Reporting Generic
Notice type:3 num:64 from LID:0x0092
GID:0xfe80000000000000,0x0005ad0000024bbb
Mar 20 14:08:22 542003 [41401960] -> Discovered new port with
GUID:0x0005ad0000024feb LID range [0x153,0x153] of node:saguaro-22-5 HCA-1
Mar 20 14:08:22 572711 [41401960] -> osm_ucast_mgr_process: Min Hop Tables
configured on all switches
Mar 20 14:08:22 611570 [41401960] -> __osm_trap_rcv_process_request:
Received Generic Notice type:0x01 num:128 Producer:2 from LID:0x0148
TID:0x000000000000004d
Mar 20 14:08:22 611751 [41401960] -> osm_report_notice: Reporting Generic
Notice type:1 num:128 from LID:0x0148
GID:0xfe80000000000000,0x0005ad00000281b3
Mar 20 14:08:22 611770 [44606960] -> __osm_trap_rcv_process_request:
Received Generic Notice type:0x01 num:128 Producer:2 from LID:0x001B
TID:0x000000000000005e
Mar 20 14:08:22 612060 [44606960] -> osm_report_notice: Reporting Generic
Notice type:1 num:128 from LID:0x001B
GID:0xfe80000000000000,0x0005ad00000281a7
Mar 20 14:08:22 623766 [4780B960] -> __osm_sm_mad_ctrl_rcv_callback: ERR
3111: Error status = 0x1C00
Mar 20 14:08:22 623814 [4780B960] -> SMP dump:
base_ver................0x1
mgmt_class..............0x81
class_ver...............0x1
method..................0x81 (SubnGetResp)
D bit...................0x1
status..................0x1C00
hop_ptr.................0x0
hop_count...............0x4
trans_id................0x66134
attr_id.................0x15 (PortInfo)
resv....................0x0
attr_mod................0x16
m_key...................0x0000000000000000
dr_slid.................0xFFFF
dr_dlid.................0xFFFF
Initial path: [0][1][11][1][5]
Return path: [0][9][18][D][2]
Reserved: [0][0][0][0][0][0][0]
00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
00 00 00 00 00 00 00 00 00 00 00 00 02 03 03 02
14 52 00 11 40 40 00 08 08 04 F2 40 00 00 00 00
00 00 00 00 00 88 00 00 00 00 00 00 00 00 00 00
Mar 20 14:08:22 623876 [45007960] -> osm_pi_rcv_process_set: Received error
status 0x1c for SetResp() during ACTIVE transition
Mar 20 14:08:22 623888 [45007960] -> PortInfo dump:
port number.............0x16
node_guid...............0x0005ad00000281b3
port_guid...............0x0005ad00000281b3
m_key...................0x0000000000000000
subnet_prefix...........0x0000000000000000
base_lid................0x0
master_sm_base_lid......0x0
capability_mask.........0x0
diag_code...............0x0
m_key_lease_period......0x0
local_port_num..........0x2
link_width_enabled......0x3
link_width_supported....0x3
link_width_active.......0x2
link_speed_supported....0x1
port_state..............ACTIVE
state_info2.............0x52
m_key_protect_bits......0x0
lmc.....................0x0
link_speed..............0x11
mtu_smsl................0x40
vl_cap_init_type........0x40
vl_high_limit...........0x0
vl_arb_high_cap.........0x8
vl_arb_low_cap..........0x8
init_rep_mtu_cap........0x4
vl_stall_life...........0xF2
vl_enforce..............0x40
m_key_violations........0x0
p_key_violations........0x0
q_key_violations........0x0
guid_cap................0x0
client_reregister.......0x0
subnet_timeout..........0x0
resp_time_value.........0x0
error_threshold.........0x88
Mar 20 14:08:22 623907 [45007960] -> Capabilities Mask:
Mar 20 14:08:22 623945 [4780B960] -> __osm_sm_mad_ctrl_rcv_callback: ERR
3111: Error status = 0x1C00
Mar 20 14:08:22 623973 [4780B960] -> SMP dump:
base_ver................0x1
mgmt_class..............0x81
class_ver...............0x1
method..................0x81 (SubnGetResp)
D bit...................0x1
status..................0x1C00
hop_ptr.................0x0
hop_count...............0x4
trans_id................0x66135
attr_id.................0x15 (PortInfo)
resv....................0x0
attr_mod................0x17
m_key...................0x0000000000000000
dr_slid.................0xFFFF
dr_dlid.................0xFFFF
Initial path: [0][1][11][1][5]
Return path: [0][9][18][D][2]
Reserved: [0][0][0][0][0][0][0]
00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
00 00 00 00 00 00 00 00 00 00 00 00 02 03 03 02
14 52 00 11 40 40 00 08 08 04 F2 40 00 00 00 00
00 00 00 00 00 88 00 00 00 00 00 00 00 00 00 00
Mar 20 14:08:22 624051 [44606960] -> osm_pi_rcv_process_set: Received error
status 0x1c for SetResp() during ACTIVE transition
Mar 20 14:08:22 624056 [44606960] -> PortInfo dump:
port number.............0x17
node_guid...............0x0005ad00000281b3
port_guid...............0x0005ad00000281b3
m_key...................0x0000000000000000
subnet_prefix...........0x0000000000000000
base_lid................0x0
master_sm_base_lid......0x0
capability_mask.........0x0
diag_code...............0x0
m_key_lease_period......0x0
local_port_num..........0x2
link_width_enabled......0x3
link_width_supported....0x3
link_width_active.......0x2
link_speed_supported....0x1
port_state..............ACTIVE
state_info2.............0x52
m_key_protect_bits......0x0
lmc.....................0x0
link_speed..............0x11
mtu_smsl................0x40
vl_cap_init_type........0x40
vl_high_limit...........0x0
vl_arb_high_cap.........0x8
vl_arb_low_cap..........0x8
init_rep_mtu_cap........0x4
vl_stall_life...........0xF2
vl_enforce..............0x40
m_key_violations........0x0
p_key_violations........0x0
q_key_violations........0x0
guid_cap................0x0
client_reregister.......0x0
subnet_timeout..........0x0
resp_time_value.........0x0
error_threshold.........0x88
Mar 20 14:08:22 624069 [44606960] -> Capabilities Mask:
Mar 20 14:08:22 629289 [45A08960] -> SUBNET UP
Mar 20 14:08:22 712180 [43204960] -> __osm_trap_rcv_process_request:
Received Generic Notice type:0x01 num:128 Producer:2 from LID:0x0152
TID:0x0000000000000018
Mar 20 14:08:22 712238 [43204960] -> __osm_trap_rcv_process_request: ERR
3804: Received trap 25 times consecutively
Mar 20 14:08:22 869303 [43C05960] -> __osm_trap_rcv_process_request:
Received Generic Notice type:0x01 num:128 Producer:2 from LID:0x001B
TID:0x000000000000005f
Mar 20 14:08:22 869527 [43C05960] -> osm_report_notice: Reporting Generic
Notice type:1 num:128 from LID:0x001B
GID:0xfe80000000000000,0x0005ad00000281a7
Mar 20 14:08:22 892522 [45A08960] -> __osm_trap_rcv_process_request:
Received Generic Notice type:0x01 num:128 Producer:2 from LID:0x0148
TID:0x000000000000004e
Mar 20 14:08:22 892707 [45A08960] -> osm_report_notice: Reporting Generic
Notice type:1 num:128 from LID:0x0148
GID:0xfe80000000000000,0x0005ad00000281b3
Mar 20 14:08:22 957086 [42803960] -> __osm_trap_rcv_process_request:
Received Generic Notice type:0x01 num:128 Producer:2 from LID:0x001B
TID:0x0000000000000060
Mar 20 14:08:22 957189 [42803960] -> osm_report_notice: Reporting Generic
Notice type:1 num:128 from LID:0x001B
GID:0xfe80000000000000,0x0005ad00000281a7
Mar 20 14:08:23 080551 [41E02960] -> __osm_trap_rcv_process_request:
Received Generic Notice type:0x01 num:128 Producer:2 from LID:0x001B
TID:0x0000000000000061
Mar 20 14:08:23 080621 [41E02960] -> osm_report_notice: Reporting Generic
Notice type:1 num:128 from LID:0x001B
GID:0xfe80000000000000,0x0005ad00000281a7
Mar 20 14:08:23 102292 [45007960] -> __osm_trap_rcv_process_request:
Received Generic Notice type:0x01 num:128 Producer:2 from LID:0x001B
TID:0x0000000000000062
Mar 20 14:08:23 102372 [45007960] -> osm_report_notice: Reporting Generic
Notice type:1 num:128 from LID:0x001B
GID:0xfe80000000000000,0x0005ad00000281a7
Mar 20 14:08:23 124176 [43C05960] -> __osm_trap_rcv_process_request:
Received Generic Notice type:0x01 num:128 Producer:2 from LID:0x001B
TID:0x0000000000000063
Mar 20 14:08:23 124278 [43C05960] -> osm_report_notice: Reporting Generic
Notice type:1 num:128 from LID:0x001B
GID:0xfe80000000000000,0x0005ad00000281a7
Mar 20 14:08:23 285320 [42803960] -> __osm_trap_rcv_process_request:
Received Generic Notice type:0x01 num:128 Producer:2 from LID:0x001B
TID:0x0000000000000064
Mar 20 14:08:23 285393 [42803960] -> osm_report_notice: Reporting Generic
Notice type:1 num:128 from LID:0x001B
GID:0xfe80000000000000,0x0005ad00000281a7
Mar 20 14:08:23 403309 [41401960] -> __osm_trap_rcv_process_request:
Received Generic Notice type:0x01 num:128 Producer:2 from LID:0x001B
TID:0x0000000000000065
Mar 20 14:08:23 403388 [41401960] -> osm_report_notice: Reporting Generic
Notice type:1 num:128 from LID:0x001B
GID:0xfe80000000000000,0x0005ad00000281a7
Mar 20 14:08:23 425052 [45A08960] -> __osm_trap_rcv_process_request:
Received Generic Notice type:0x01 num:128 Producer:2 from LID:0x001B
TID:0x0000000000000066
Mar 20 14:08:23 425117 [45A08960] -> osm_report_notice: Reporting Generic
Notice type:1 num:128 from LID:0x001B
GID:0xfe80000000000000,0x0005ad00000281a7
Mar 20 14:08:23 447189 [41E02960] -> __osm_trap_rcv_process_request:
Received Generic Notice type:0x01 num:128 Producer:2 from LID:0x001B
TID:0x0000000000000067
Mar 20 14:08:23 447266 [41E02960] -> osm_report_notice: Reporting Generic
Notice type:1 num:128 from LID:0x001B
GID:0xfe80000000000000,0x0005ad00000281a7
Mar 20 14:08:23 535175 [44606960] -> osm_ucast_mgr_process: Min Hop Tables
configured on all switches
Mar 20 14:08:23 595127 [41401960] -> SUBNET UP
Mar 20 14:08:23 750323 [41E02960] -> __osm_trap_rcv_process_request:
Received Generic Notice type:0x01 num:128 Producer:2 from LID:0x0152
TID:0x0000000000000018
Mar 20 14:08:23 750432 [41E02960] -> __osm_trap_rcv_process_request: ERR
3804: Received trap 26 times consecutively
Mar 20 14:08:23 960490 [42803960] -> osm_ucast_mgr_process: Min Hop Tables
configured on all switches
Mar 20 14:08:24 014256 [4780B960] -> __osm_sm_mad_ctrl_rcv_callback: ERR
3111: Error status = 0x1C00
Mar 20 14:08:24 014323 [4780B960] -> SMP dump:
base_ver................0x1
mgmt_class..............0x81
class_ver...............0x1
method..................0x81 (SubnGetResp)
D bit...................0x1
status..................0x1C00
hop_ptr.................0x0
hop_count...............0x4
trans_id................0x67b9d
attr_id.................0x15 (PortInfo)
resv....................0x0
attr_mod................0x18
m_key...................0x0000000000000000
dr_slid.................0xFFFF
dr_dlid.................0xFFFF
Initial path: [0][1][11][1][6]
Return path: [0][9][18][D][3]
Reserved: [0][0][0][0][0][0][0]
00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
00 00 00 00 00 00 00 00 00 00 00 00 03 03 03 02
14 52 00 11 40 40 00 08 08 04 F2 40 00 00 00 00
00 00 00 00 00 88 00 00 00 00 00 00 00 00 00 00
Mar 20 14:08:24 014398 [41401960] -> osm_pi_rcv_process_set: Received error
status 0x1c for SetResp() during ACTIVE transition
Mar 20 14:08:24 014408 [41401960] -> PortInfo dump:
port number.............0x18
node_guid...............0x0005ad00000281b3
port_guid...............0x0005ad00000281b3
m_key...................0x0000000000000000
subnet_prefix...........0x0000000000000000
base_lid................0x0
master_sm_base_lid......0x0
capability_mask.........0x0
diag_code...............0x0
m_key_lease_period......0x0
local_port_num..........0x3
link_width_enabled......0x3
link_width_supported....0x3
link_width_active.......0x2
link_speed_supported....0x1
port_state..............ACTIVE
state_info2.............0x52
m_key_protect_bits......0x0
lmc.....................0x0
link_speed..............0x11
mtu_smsl................0x40
vl_cap_init_type........0x40
vl_high_limit...........0x0
vl_arb_high_cap.........0x8
vl_arb_low_cap..........0x8
init_rep_mtu_cap........0x4
vl_stall_life...........0xF2
vl_enforce..............0x40
m_key_violations........0x0
p_key_violations........0x0
q_key_violations........0x0
guid_cap................0x0
client_reregister.......0x0
subnet_timeout..........0x0
resp_time_value.........0x0
error_threshold.........0x88
Mar 20 14:08:24 014422 [41401960] -> Capabilities Mask:
Mar 20 14:08:24 019479 [41401960] -> SUBNET UP
Mar 20 14:11:00 201308 [43204960] -> __osm_trap_rcv_process_request:
Received Generic Notice type:0x01 num:128 Producer:2 from LID:0x001F
TID:0x0000000000000018
Mar 20 14:11:00 201580 [43204960] -> osm_report_notice: Reporting Generic
Notice type:1 num:128 from LID:0x001F
GID:0xfe80000000000000,0x0005ad0000027c56
Mar 20 14:11:00 554517 [41E02960] -> osm_report_notice: Reporting Generic
Notice type:3 num:65 from LID:0x0092
GID:0xfe80000000000000,0x0005ad0000024bbb
Mar 20 14:11:00 554538 [41E02960] -> Removed port with
GUID:0x0005ad000002516f LID range [0xBA,0xBA] of node:saguaro-24-1 HCA-1
Mar 20 14:11:00 589140 [41E02960] -> osm_ucast_mgr_process: Min Hop Tables
configured on all switches
Mar 20 14:11:00 641315 [45A08960] -> SUBNET UP
Mar 20 14:14:16 904140 [41E02960] -> __osm_trap_rcv_process_request:
Received Generic Notice type:0x01 num:128 Producer:2 from LID:0x001B
TID:0x0000000000000068
Mar 20 14:14:16 904369 [41E02960] -> osm_report_notice: Reporting Generic
Notice type:1 num:128 from LID:0x001B
GID:0xfe80000000000000,0x0005ad00000281a7
Mar 20 14:14:16 904462 [45007960] -> __osm_trap_rcv_process_request:
Received Generic Notice type:0x01 num:128 Producer:2 from LID:0x0148
TID:0x000000000000004f
Mar 20 14:14:16 904600 [45007960] -> osm_report_notice: Reporting Generic
Notice type:1 num:128 from LID:0x0148
GID:0xfe80000000000000,0x0005ad00000281b3
Mar 20 14:14:17 210726 [41401960] -> osm_report_notice: Reporting Generic
Notice type:3 num:65 from LID:0x0092
GID:0xfe80000000000000,0x0005ad0000024bbb
Mar 20 14:14:17 210747 [41401960] -> Removed port with
GUID:0x0005ad0000024e0b LID range [0xB3,0xB3] of node:saguaro-23-4 HCA-1
Mar 20 14:14:17 210796 [41401960] -> osm_report_notice: Reporting Generic
Notice type:3 num:65 from LID:0x0092
GID:0xfe80000000000000,0x0005ad0000024bbb
Mar 20 14:14:17 210802 [41401960] -> Removed port with
GUID:0x0005ad000002510b LID range [0xB5,0xB5] of node:saguaro-23-6 HCA-1
Mar 20 14:14:17 210818 [41401960] -> osm_report_notice: Reporting Generic
Notice type:3 num:65 from LID:0x0092
GID:0xfe80000000000000,0x0005ad0000024bbb
Mar 20 14:14:17 210836 [41401960] -> Removed port with
GUID:0x0005ad000002511b LID range [0xA6,0xA6] of node:saguaro-22-1 HCA-1
Mar 20 14:14:17 210864 [41401960] -> osm_report_notice: Reporting Generic
Notice type:3 num:65 from LID:0x0092
GID:0xfe80000000000000,0x0005ad0000024bbb
Mar 20 14:14:17 210869 [41401960] -> Removed port with
GUID:0x0005ad0000025043 LID range [0xB4,0xB4] of node:saguaro-23-5 HCA-1
Mar 20 14:14:17 210885 [41401960] -> osm_report_notice: Reporting Generic
Notice type:3 num:65 from LID:0x0092
GID:0xfe80000000000000,0x0005ad0000024bbb
Mar 20 14:14:17 210890 [41401960] -> Removed port with
GUID:0x0005ad0000024d47 LID range [0xB6,0xB6] of node:saguaro-23-7 HCA-1
Mar 20 14:14:17 210908 [41401960] -> osm_report_notice: Reporting Generic
Notice type:3 num:65 from LID:0x0092
GID:0xfe80000000000000,0x0005ad0000024bbb
Mar 20 14:14:17 210913 [41401960] -> Removed port with
GUID:0x0005ad0000024d6b LID range [0xB8,0xB8] of node:saguaro-23-9 HCA-1
Mar 20 14:14:17 210931 [41401960] -> osm_report_notice: Reporting Generic
Notice type:3 num:65 from LID:0x0092
GID:0xfe80000000000000,0x0005ad0000024bbb
Mar 20 14:14:17 210936 [41401960] -> Removed port with
GUID:0x0005ad0000024977 LID range [0xA9,0xA9] of node:saguaro-22-4 HCA-1
Mar 20 14:14:17 211090 [41401960] -> osm_report_notice: Reporting Generic
Notice type:3 num:65 from LID:0x0092
GID:0xfe80000000000000,0x0005ad0000024bbb
Mar 20 14:14:17 211096 [41401960] -> Removed port with
GUID:0x0005ad0000027c84 LID range [0x152,0x152] of node:Topspin Switch TS120
Mar 20 14:14:17 211127 [41401960] -> osm_report_notice: Reporting Generic
Notice type:3 num:65 from LID:0x0092
GID:0xfe80000000000000,0x0005ad0000024bbb
Mar 20 14:14:17 211133 [41401960] -> Removed port with
GUID:0x0005ad0000024d8b LID range [0xB7,0xB7] of node:saguaro-23-8 HCA-1
Mar 20 14:14:17 211147 [41401960] -> osm_report_notice: Reporting Generic
Notice type:3 num:65 from LID:0x0092
GID:0xfe80000000000000,0x0005ad0000024bbb
Mar 20 14:14:17 211153 [41401960] -> Removed port with
GUID:0x0005ad000002498f LID range [0xA8,0xA8] of node:saguaro-22-3 HCA-1
Mar 20 14:14:17 211169 [41401960] -> osm_report_notice: Reporting Generic
Notice type:3 num:65 from LID:0x0092
GID:0xfe80000000000000,0x0005ad0000024bbb
Mar 20 14:14:17 211174 [41401960] -> Removed port with
GUID:0x0005ad0000024c9b LID range [0xA7,0xA7] of node:saguaro-22-2 HCA-1
Mar 20 14:14:17 211189 [41401960] -> osm_report_notice: Reporting Generic
Notice type:3 num:65 from LID:0x0092
GID:0xfe80000000000000,0x0005ad0000024bbb
Mar 20 14:14:17 211194 [41401960] -> Removed port with
GUID:0x0005ad0000024da7 LID range [0xB0,0xB0] of node:saguaro-23-1 HCA-1
Mar 20 14:14:17 211212 [41401960] -> osm_report_notice: Reporting Generic
Notice type:3 num:65 from LID:0x0092
GID:0xfe80000000000000,0x0005ad0000024bbb
Mar 20 14:14:17 211216 [41401960] -> Removed port with
GUID:0x0005ad0000024cbb LID range [0xB2,0xB2] of node:saguaro-23-3 HCA-1
Mar 20 14:14:17 211232 [41401960] -> osm_report_notice: Reporting Generic
Notice type:3 num:65 from LID:0x0092
GID:0xfe80000000000000,0x0005ad0000024bbb
Mar 20 14:14:17 211237 [41401960] -> Removed port with
GUID:0x0005ad00000249d3 LID range [0xB1,0xB1] of node:saguaro-23-2 HCA-1
Mar 20 14:14:17 211253 [41401960] -> osm_report_notice: Reporting Generic
Notice type:3 num:65 from LID:0x0092
GID:0xfe80000000000000,0x0005ad0000024bbb
Mar 20 14:14:17 211317 [41401960] -> Removed port with
GUID:0x0005ad0000024feb LID range [0x153,0x153] of node:saguaro-22-5 HCA-1
Mar 20 14:14:17 211333 [41401960] -> osm_report_notice: Reporting Generic
Notice type:3 num:65 from LID:0x0092
GID:0xfe80000000000000,0x0005ad0000024bbb
Mar 20 14:14:17 211338 [41401960] -> Removed port with
GUID:0x0005ad0000024afb LID range [0xA5,0xA5] of node:saguaro-22-0 HCA-1
Mar 20 14:14:17 244432 [41401960] -> osm_ucast_mgr_process: Min Hop Tables
configured on all switches
Mar 20 14:14:17 292747 [42803960] -> SUBNET UP
Mar 20 14:14:17 698554 [45A08960] -> osm_ucast_mgr_process: Min Hop Tables
configured on all switches
Mar 20 14:14:17 750419 [44606960] -> SUBNET UP
Mar 20 14:15:11 300343 [41401960] -> __osm_trap_rcv_process_request:
Received Generic Notice type:0x01 num:128 Producer:2 from LID:0x0148
TID:0x0000000000000050
Mar 20 14:15:11 300577 [41401960] -> osm_report_notice: Reporting Generic
Notice type:1 num:128 from LID:0x0148
GID:0xfe80000000000000,0x0005ad00000281b3
Mar 20 14:15:11 306375 [45A08960] -> __osm_trap_rcv_process_request:
Received Generic Notice type:0x01 num:128 Producer:2 from LID:0x001B
TID:0x0000000000000069
Mar 20 14:15:11 306439 [42803960] -> __osm_trap_rcv_process_request:
Received Generic Notice type:0x01 num:128 Producer:2 from LID:0x0148
TID:0x0000000000000051
Mar 20 14:15:11 306487 [45A08960] -> osm_report_notice: Reporting Generic
Notice type:1 num:128 from LID:0x001B
GID:0xfe80000000000000,0x0005ad00000281a7
Mar 20 14:15:11 306514 [42803960] -> osm_report_notice: Reporting Generic
Notice type:1 num:128 from LID:0x0148
GID:0xfe80000000000000,0x0005ad00000281b3
Mar 20 14:15:11 312487 [43204960] -> __osm_trap_rcv_process_request:
Received Generic Notice type:0x01 num:128 Producer:2 from LID:0x001B
TID:0x000000000000006a
Mar 20 14:15:11 312581 [43204960] -> osm_report_notice: Reporting Generic
Notice type:1 num:128 from LID:0x001B
GID:0xfe80000000000000,0x0005ad00000281a7
Mar 20 14:15:11 636546 [45007960] -> osm_report_notice: Reporting Generic
Notice type:3 num:64 from LID:0x0092
GID:0xfe80000000000000,0x0005ad0000024bbb
Mar 20 14:15:11 636559 [45007960] -> Discovered new port with
GUID:0x0005ad0000027c84 LID range [0x152,0x152] of node:Topspin Switch TS120
Mar 20 14:15:11 636565 [45007960] -> osm_report_notice: Reporting Generic
Notice type:3 num:64 from LID:0x0092
GID:0xfe80000000000000,0x0005ad0000024bbb
Mar 20 14:15:11 636572 [45007960] -> Discovered new port with
GUID:0x0005ad0000024da7 LID range [0xB0,0xB0] of node:saguaro-23-1 HCA-1
Mar 20 14:15:11 636577 [45007960] -> osm_report_notice: Reporting Generic
Notice type:3 num:64 from LID:0x0092
GID:0xfe80000000000000,0x0005ad0000024bbb
Mar 20 14:15:11 636584 [45007960] -> Discovered new port with
GUID:0x0005ad00000249d3 LID range [0xB1,0xB1] of node:saguaro-23-2 HCA-1
Mar 20 14:15:11 636589 [45007960] -> osm_report_notice: Reporting Generic
Notice type:3 num:64 from LID:0x0092
GID:0xfe80000000000000,0x0005ad0000024bbb
Mar 20 14:15:11 636595 [45007960] -> Discovered new port with
GUID:0x0005ad0000024cbb LID range [0xB2,0xB2] of node:saguaro-23-3 HCA-1
Mar 20 14:15:11 636600 [45007960] -> osm_report_notice: Reporting Generic
Notice type:3 num:64 from LID:0x0092
GID:0xfe80000000000000,0x0005ad0000024bbb
Mar 20 14:15:11 636606 [45007960] -> Discovered new port with
GUID:0x0005ad0000024e0b LID range [0xB3,0xB3] of node:saguaro-23-4 HCA-1
Mar 20 14:15:11 636612 [45007960] -> osm_report_notice: Reporting Generic
Notice type:3 num:64 from LID:0x0092
GID:0xfe80000000000000,0x0005ad0000024bbb
Mar 20 14:15:11 636618 [45007960] -> Discovered new port with
GUID:0x0005ad0000025043 LID range [0xB4,0xB4] of node:saguaro-23-5 HCA-1
Mar 20 14:15:11 636623 [45007960] -> osm_report_notice: Reporting Generic
Notice type:3 num:64 from LID:0x0092
GID:0xfe80000000000000,0x0005ad0000024bbb
Mar 20 14:15:11 636629 [45007960] -> Discovered new port with
GUID:0x0005ad000002510b LID range [0xB5,0xB5] of node:saguaro-23-6 HCA-1
Mar 20 14:15:11 636634 [45007960] -> osm_report_notice: Reporting Generic
Notice type:3 num:64 from LID:0x0092
GID:0xfe80000000000000,0x0005ad0000024bbb
Mar 20 14:15:11 636641 [45007960] -> Discovered new port with
GUID:0x0005ad0000024d47 LID range [0xB6,0xB6] of node:saguaro-23-7 HCA-1
Mar 20 14:15:11 636646 [45007960] -> osm_report_notice: Reporting Generic
Notice type:3 num:64 from LID:0x0092
GID:0xfe80000000000000,0x0005ad0000024bbb
Mar 20 14:15:11 636652 [45007960] -> Discovered new port with
GUID:0x0005ad0000024d8b LID range [0xB7,0xB7] of node:saguaro-23-8 HCA-1
Mar 20 14:15:11 636657 [45007960] -> osm_report_notice: Reporting Generic
Notice type:3 num:64 from LID:0x0092
GID:0xfe80000000000000,0x0005ad0000024bbb
Mar 20 14:15:11 636663 [45007960] -> Discovered new port with
GUID:0x0005ad0000024d6b LID range [0xB8,0xB8] of node:saguaro-23-9 HCA-1
Mar 20 14:15:11 636668 [45007960] -> osm_report_notice: Reporting Generic
Notice type:3 num:64 from LID:0x0092
GID:0xfe80000000000000,0x0005ad0000024bbb
Mar 20 14:15:11 636675 [45007960] -> Discovered new port with
GUID:0x0005ad0000024afb LID range [0xA5,0xA5] of node:saguaro-22-0 HCA-1
Mar 20 14:15:11 636680 [45007960] -> osm_report_notice: Reporting Generic
Notice type:3 num:64 from LID:0x0092
GID:0xfe80000000000000,0x0005ad0000024bbb
Mar 20 14:15:11 636686 [45007960] -> Discovered new port with
GUID:0x0005ad000002511b LID range [0xA6,0xA6] of node:saguaro-22-1 HCA-1
Mar 20 14:15:11 636691 [45007960] -> osm_report_notice: Reporting Generic
Notice type:3 num:64 from LID:0x0092
GID:0xfe80000000000000,0x0005ad0000024bbb
Mar 20 14:15:11 636698 [45007960] -> Discovered new port with
GUID:0x0005ad0000024c9b LID range [0xA7,0xA7] of node:saguaro-22-2 HCA-1
Mar 20 14:15:11 636703 [45007960] -> osm_report_notice: Reporting Generic
Notice type:3 num:64 from LID:0x0092
GID:0xfe80000000000000,0x0005ad0000024bbb
Mar 20 14:15:11 636709 [45007960] -> Discovered new port with
GUID:0x0005ad000002498f LID range [0xA8,0xA8] of node:saguaro-22-3 HCA-1
Mar 20 14:15:11 636742 [45007960] -> osm_report_notice: Reporting Generic
Notice type:3 num:64 from LID:0x0092
GID:0xfe80000000000000,0x0005ad0000024bbb
Mar 20 14:15:11 636750 [45007960] -> Discovered new port with
GUID:0x0005ad0000024977 LID range [0xA9,0xA9] of node:saguaro-22-4 HCA-1
Mar 20 14:15:11 636755 [45007960] -> osm_report_notice: Reporting Generic
Notice type:3 num:64 from LID:0x0092
GID:0xfe80000000000000,0x0005ad0000024bbb
Mar 20 14:15:11 636761 [45007960] -> Discovered new port with
GUID:0x0005ad0000024feb LID range [0x153,0x153] of node:saguaro-22-5 HCA-1
Mar 20 14:15:11 667436 [45007960] -> osm_ucast_mgr_process: Min Hop Tables
configured on all switches
Mar 20 14:15:11 731917 [4780B960] -> __osm_sm_mad_ctrl_rcv_callback: ERR
3111: Error status = 0x1C00
Mar 20 14:15:11 732017 [4780B960] -> SMP dump:
base_ver................0x1
mgmt_class..............0x81
class_ver...............0x1
method..................0x81 (SubnGetResp)
D bit...................0x1
status..................0x1C00
hop_ptr.................0x0
hop_count...............0x4
trans_id................0x6b507
attr_id.................0x15 (PortInfo)
resv....................0x0
attr_mod................0x13
m_key...................0x0000000000000000
dr_slid.................0xFFFF
dr_dlid.................0xFFFF
Initial path: [0][1][16][1][4]
Return path: [0][9][13][D][1]
Reserved: [0][0][0][0][0][0][0]
00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
00 00 00 00 00 00 00 00 00 00 00 00 01 03 03 02
14 52 00 11 40 40 00 08 08 04 F2 40 00 00 00 00
00 00 00 00 00 88 00 00 00 00 00 00 00 00 00 00
Mar 20 14:15:11 732102 [41401960] -> osm_pi_rcv_process_set: Received error
status 0x1c for SetResp() during ACTIVE transition
Mar 20 14:15:11 732106 [41401960] -> PortInfo dump:
port number.............0x13
node_guid...............0x0005ad00000281a7
port_guid...............0x0005ad00000281a7
m_key...................0x0000000000000000
subnet_prefix...........0x0000000000000000
base_lid................0x0
master_sm_base_lid......0x0
capability_mask.........0x0
diag_code...............0x0
m_key_lease_period......0x0
local_port_num..........0x1
link_width_enabled......0x3
link_width_supported....0x3
link_width_active.......0x2
link_speed_supported....0x1
port_state..............ACTIVE
state_info2.............0x52
m_key_protect_bits......0x0
lmc.....................0x0
link_speed..............0x11
mtu_smsl................0x40
vl_cap_init_type........0x40
vl_high_limit...........0x0
vl_arb_high_cap.........0x8
vl_arb_low_cap..........0x8
init_rep_mtu_cap........0x4
vl_stall_life...........0xF2
vl_enforce..............0x40
m_key_violations........0x0
p_key_violations........0x0
q_key_violations........0x0
guid_cap................0x0
client_reregister.......0x0
subnet_timeout..........0x0
resp_time_value.........0x0
error_threshold.........0x88
Mar 20 14:15:11 732128 [41401960] -> Capabilities Mask:
Mar 20 14:15:11 732160 [4780B960] -> __osm_sm_mad_ctrl_rcv_callback: ERR
3111: Error status = 0x1C00
Mar 20 14:15:11 732185 [4780B960] -> SMP dump:
base_ver................0x1
mgmt_class..............0x81
class_ver...............0x1
method..................0x81 (SubnGetResp)
D bit...................0x1
status..................0x1C00
hop_ptr.................0x0
hop_count...............0x4
trans_id................0x6b508
attr_id.................0x15 (PortInfo)
resv....................0x0
attr_mod................0x16
m_key...................0x0000000000000000
dr_slid.................0xFFFF
dr_dlid.................0xFFFF
Initial path: [0][1][16][1][4]
Return path: [0][9][13][D][1]
Reserved: [0][0][0][0][0][0][0]
00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
00 00 00 00 00 00 00 00 00 00 00 00 01 03 03 02
14 52 00 11 40 40 00 08 08 04 F2 40 00 00 00 00
00 00 00 00 00 88 00 00 00 00 00 00 00 00 00 00
Mar 20 14:15:11 732254 [44606960] -> osm_pi_rcv_process_set: Received error
status 0x1c for SetResp() during ACTIVE transition
Mar 20 14:15:11 732258 [44606960] -> PortInfo dump:
port number.............0x16
node_guid...............0x0005ad00000281a7
port_guid...............0x0005ad00000281a7
m_key...................0x0000000000000000
subnet_prefix...........0x0000000000000000
base_lid................0x0
master_sm_base_lid......0x0
capability_mask.........0x0
diag_code...............0x0
m_key_lease_period......0x0
local_port_num..........0x1
link_width_enabled......0x3
link_width_supported....0x3
link_width_active.......0x2
link_speed_supported....0x1
port_state..............ACTIVE
state_info2.............0x52
m_key_protect_bits......0x0
lmc.....................0x0
link_speed..............0x11
mtu_smsl................0x40
vl_cap_init_type........0x40
vl_high_limit...........0x0
vl_arb_high_cap.........0x8
vl_arb_low_cap..........0x8
init_rep_mtu_cap........0x4
vl_stall_life...........0xF2
vl_enforce..............0x40
m_key_violations........0x0
p_key_violations........0x0
q_key_violations........0x0
guid_cap................0x0
client_reregister.......0x0
subnet_timeout..........0x0
resp_time_value.........0x0
error_threshold.........0x88
Mar 20 14:15:11 732269 [44606960] -> Capabilities Mask:
Mar 20 14:15:11 732300 [4780B960] -> __osm_sm_mad_ctrl_rcv_callback: ERR
3111: Error status = 0x1C00
Mar 20 14:15:11 732334 [4780B960] -> SMP dump:
base_ver................0x1
mgmt_class..............0x81
class_ver...............0x1
method..................0x81 (SubnGetResp)
D bit...................0x1
status..................0x1C00
hop_ptr.................0x0
hop_count...............0x4
trans_id................0x6b509
attr_id.................0x15 (PortInfo)
resv....................0x0
attr_mod................0x17
m_key...................0x0000000000000000
dr_slid.................0xFFFF
dr_dlid.................0xFFFF
Initial path: [0][1][16][1][4]
Return path: [0][9][13][D][1]
Reserved: [0][0][0][0][0][0][0]
00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
00 00 00 00 00 00 00 00 00 00 00 00 01 03 03 02
14 52 00 11 40 40 00 08 08 04 F2 40 00 00 00 00
00 00 00 00 00 88 00 00 00 00 00 00 00 00 00 00
Mar 20 14:15:11 732420 [4780B960] -> __osm_sm_mad_ctrl_rcv_callback: ERR
3111: Error status = 0x1C00
Mar 20 14:15:11 732419 [45007960] -> osm_pi_rcv_process_set: Received error
status 0x1c for SetResp() during ACTIVE transition
Mar 20 14:15:11 732451 [4780B960] -> SMP dump:
base_ver................0x1
mgmt_class..............0x81
class_ver...............0x1
method..................0x81 (SubnGetResp)
D bit...................0x1
status..................0x1C00
hop_ptr.................0x0
hop_count...............0x4
trans_id................0x6b50a
attr_id.................0x15 (PortInfo)
resv....................0x0
attr_mod................0x18
m_key...................0x0000000000000000
dr_slid.................0xFFFF
dr_dlid.................0xFFFF
Initial path: [0][1][16][1][4]
Return path: [0][9][13][D][1]
Reserved: [0][0][0][0][0][0][0]
00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
00 00 00 00 00 00 00 00 00 00 00 00 01 03 03 02
14 52 00 11 40 40 00 08 08 04 F2 40 00 00 00 00
00 00 00 00 00 88 00 00 00 00 00 00 00 00 00 00
Mar 20 14:15:11 732447 [45007960] -> PortInfo dump:
port number.............0x17
node_guid...............0x0005ad00000281a7
port_guid...............0x0005ad00000281a7
m_key...................0x0000000000000000
subnet_prefix...........0x0000000000000000
base_lid................0x0
master_sm_base_lid......0x0
capability_mask.........0x0
diag_code...............0x0
m_key_lease_period......0x0
local_port_num..........0x1
link_width_enabled......0x3
link_width_supported....0x3
link_width_active.......0x2
link_speed_supported....0x1
port_state..............ACTIVE
state_info2.............0x52
m_key_protect_bits......0x0
lmc.....................0x0
link_speed..............0x11
mtu_smsl................0x40
vl_cap_init_type........0x40
vl_high_limit...........0x0
vl_arb_high_cap.........0x8
vl_arb_low_cap..........0x8
init_rep_mtu_cap........0x4
vl_stall_life...........0xF2
vl_enforce..............0x40
m_key_violations........0x0
p_key_violations........0x0
q_key_violations........0x0
guid_cap................0x0
client_reregister.......0x0
subnet_timeout..........0x0
resp_time_value.........0x0
error_threshold.........0x88
Mar 20 14:15:11 732471 [45007960] -> Capabilities Mask:
Mar 20 14:15:11 732511 [45007960] -> osm_pi_rcv_process_set: Received error
status 0x1c for SetResp() during ACTIVE transition
Mar 20 14:15:11 732516 [45007960] -> PortInfo dump:
port number.............0x18
node_guid...............0x0005ad00000281a7
port_guid...............0x0005ad00000281a7
m_key...................0x0000000000000000
subnet_prefix...........0x0000000000000000
base_lid................0x0
master_sm_base_lid......0x0
capability_mask.........0x0
diag_code...............0x0
m_key_lease_period......0x0
local_port_num..........0x1
link_width_enabled......0x3
link_width_supported....0x3
link_width_active.......0x2
link_speed_supported....0x1
port_state..............ACTIVE
state_info2.............0x52
m_key_protect_bits......0x0
lmc.....................0x0
link_speed..............0x11
mtu_smsl................0x40
vl_cap_init_type........0x40
vl_high_limit...........0x0
vl_arb_high_cap.........0x8
vl_arb_low_cap..........0x8
init_rep_mtu_cap........0x4
vl_stall_life...........0xF2
vl_enforce..............0x40
m_key_violations........0x0
p_key_violations........0x0
q_key_violations........0x0
guid_cap................0x0
client_reregister.......0x0
subnet_timeout..........0x0
resp_time_value.........0x0
error_threshold.........0x88
Mar 20 14:15:11 732529 [45007960] -> Capabilities Mask:
Mar 20 14:15:11 732556 [4780B960] -> __osm_sm_mad_ctrl_rcv_callback: ERR
3111: Error status = 0x1C00
Mar 20 14:15:11 732591 [4780B960] -> SMP dump:
base_ver................0x1
mgmt_class..............0x81
class_ver...............0x1
method..................0x81 (SubnGetResp)
D bit...................0x1
status..................0x1C00
hop_ptr.................0x0
hop_count...............0x4
trans_id................0x6b50b
attr_id.................0x15 (PortInfo)
resv....................0x0
attr_mod................0x16
m_key...................0x0000000000000000
dr_slid.................0xFFFF
dr_dlid.................0xFFFF
Initial path: [0][1][11][2][5]
Return path: [0][9][18][E][2]
Reserved: [0][0][0][0][0][0][0]
00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
00 00 00 00 00 00 00 00 00 00 00 00 02 03 03 02
14 52 00 11 40 40 00 08 08 04 F2 40 00 00 00 00
00 00 00 00 00 88 00 00 00 00 00 00 00 00 00 00
Mar 20 14:15:11 732653 [43204960] -> osm_pi_rcv_process_set: Received error
status 0x1c for SetResp() during ACTIVE transition
Mar 20 14:15:11 732662 [43204960] -> PortInfo dump:
port number.............0x16
node_guid...............0x0005ad00000281b3
port_guid...............0x0005ad00000281b3
m_key...................0x0000000000000000
subnet_prefix...........0x0000000000000000
base_lid................0x0
master_sm_base_lid......0x0
capability_mask.........0x0
diag_code...............0x0
m_key_lease_period......0x0
local_port_num..........0x2
link_width_enabled......0x3
link_width_supported....0x3
link_width_active.......0x2
link_speed_supported....0x1
port_state..............ACTIVE
state_info2.............0x52
m_key_protect_bits......0x0
lmc.....................0x0
link_speed..............0x11
mtu_smsl................0x40
vl_cap_init_type........0x40
vl_high_limit...........0x0
vl_arb_high_cap.........0x8
vl_arb_low_cap..........0x8
init_rep_mtu_cap........0x4
vl_stall_life...........0xF2
vl_enforce..............0x40
m_key_violations........0x0
p_key_violations........0x0
q_key_violations........0x0
guid_cap................0x0
client_reregister.......0x0
subnet_timeout..........0x0
resp_time_value.........0x0
error_threshold.........0x88
Mar 20 14:15:11 732673 [43204960] -> Capabilities Mask:
Mar 20 14:15:11 732705 [4780B960] -> __osm_sm_mad_ctrl_rcv_callback: ERR
3111: Error status = 0x1C00
Mar 20 14:15:11 732739 [4780B960] -> SMP dump:
base_ver................0x1
mgmt_class..............0x81
class_ver...............0x1
method..................0x81 (SubnGetResp)
D bit...................0x1
status..................0x1C00
hop_ptr.................0x0
hop_count...............0x4
trans_id................0x6b50c
attr_id.................0x15 (PortInfo)
resv....................0x0
attr_mod................0x17
m_key...................0x0000000000000000
dr_slid.................0xFFFF
dr_dlid.................0xFFFF
Initial path: [0][1][11][2][5]
Return path: [0][9][18][E][2]
Reserved: [0][0][0][0][0][0][0]
00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
00 00 00 00 00 00 00 00 00 00 00 00 02 03 03 02
14 52 00 11 40 40 00 08 08 04 F2 40 00 00 00 00
00 00 00 00 00 88 00 00 00 00 00 00 00 00 00 00
Mar 20 14:15:11 732809 [4780B960] -> __osm_sm_mad_ctrl_rcv_callback: ERR
3111: Error status = 0x1C00
Mar 20 14:15:11 732805 [41E02960] -> osm_pi_rcv_process_set: Received error
status 0x1c for SetResp() during ACTIVE transition
Mar 20 14:15:11 732839 [4780B960] -> SMP dump:
base_ver................0x1
mgmt_class..............0x81
class_ver...............0x1
method..................0x81 (SubnGetResp)
D bit...................0x1
status..................0x1C00
hop_ptr.................0x0
hop_count...............0x4
trans_id................0x6b50d
attr_id.................0x15 (PortInfo)
resv....................0x0
attr_mod................0x18
m_key...................0x0000000000000000
dr_slid.................0xFFFF
dr_dlid.................0xFFFF
Initial path: [0][1][11][2][5]
Return path: [0][9][18][E][2]
Reserved: [0][0][0][0][0][0][0]
00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
00 00 00 00 00 00 00 00 00 00 00 00 02 03 03 02
14 52 00 11 40 40 00 08 08 04 F2 40 00 00 00 00
00 00 00 00 00 88 00 00 00 00 00 00 00 00 00 00
Mar 20 14:15:11 732837 [41E02960] -> PortInfo dump:
port number.............0x17
node_guid...............0x0005ad00000281b3
port_guid...............0x0005ad00000281b3
m_key...................0x0000000000000000
subnet_prefix...........0x0000000000000000
base_lid................0x0
master_sm_base_lid......0x0
capability_mask.........0x0
diag_code...............0x0
m_key_lease_period......0x0
local_port_num..........0x2
link_width_enabled......0x3
link_width_supported....0x3
link_width_active.......0x2
link_speed_supported....0x1
port_state..............ACTIVE
state_info2.............0x52
m_key_protect_bits......0x0
lmc.....................0x0
link_speed..............0x11
mtu_smsl................0x40
vl_cap_init_type........0x40
vl_high_limit...........0x0
vl_arb_high_cap.........0x8
vl_arb_low_cap..........0x8
init_rep_mtu_cap........0x4
vl_stall_life...........0xF2
vl_enforce..............0x40
m_key_violations........0x0
p_key_violations........0x0
q_key_violations........0x0
guid_cap................0x0
client_reregister.......0x0
subnet_timeout..........0x0
resp_time_value.........0x0
error_threshold.........0x88
Mar 20 14:15:11 732856 [41E02960] -> Capabilities Mask:
Mar 20 14:15:11 732898 [41E02960] -> osm_pi_rcv_process_set: Received error
status 0x1c for SetResp() during ACTIVE transition
Mar 20 14:15:11 732911 [41E02960] -> PortInfo dump:
port number.............0x18
node_guid...............0x0005ad00000281b3
port_guid...............0x0005ad00000281b3
m_key...................0x0000000000000000
subnet_prefix...........0x0000000000000000
base_lid................0x0
master_sm_base_lid......0x0
capability_mask.........0x0
diag_code...............0x0
m_key_lease_period......0x0
local_port_num..........0x2
link_width_enabled......0x3
link_width_supported....0x3
link_width_active.......0x2
link_speed_supported....0x1
port_state..............ACTIVE
state_info2.............0x52
m_key_protect_bits......0x0
lmc.....................0x0
link_speed..............0x11
mtu_smsl................0x40
vl_cap_init_type........0x40
vl_high_limit...........0x0
vl_arb_high_cap.........0x8
vl_arb_low_cap..........0x8
init_rep_mtu_cap........0x4
vl_stall_life...........0xF2
vl_enforce..............0x40
m_key_violations........0x0
p_key_violations........0x0
q_key_violations........0x0
guid_cap................0x0
client_reregister.......0x0
subnet_timeout..........0x0
resp_time_value.........0x0
error_threshold.........0x88
Mar 20 14:15:11 732925 [41E02960] -> Capabilities Mask:
Mar 20 14:15:11 738354 [45A08960] -> SUBNET UP
Mar 20 14:15:12 115658 [44606960] -> osm_ucast_mgr_process: Min Hop Tables
configured on all switches
Mar 20 14:15:12 172029 [44606960] -> SUBNET UP
Mar 20 14:15:27 277617 [41401960] -> __osm_trap_rcv_process_request:
Received Generic Notice type:0x01 num:128 Producer:2 from LID:0x0152
TID:0x0000000000000000
Mar 20 14:15:27 277863 [41401960] -> osm_report_notice: Reporting Generic
Notice type:1 num:128 from LID:0x0152
GID:0xfe80000000000000,0x0005ad0000027c84
Mar 20 14:15:27 510410 [43C05960] -> __osm_trap_rcv_process_request:
Received Generic Notice type:0x01 num:128 Producer:2 from LID:0x0152
TID:0x0000000000000001
Mar 20 14:15:27 510626 [43C05960] -> osm_report_notice: Reporting Generic
Notice type:1 num:128 from LID:0x0152
GID:0xfe80000000000000,0x0005ad0000027c84
Mar 20 14:15:27 532239 [41E02960] -> __osm_trap_rcv_process_request:
Received Generic Notice type:0x01 num:128 Producer:2 from LID:0x0152
TID:0x0000000000000002
Mar 20 14:15:27 532443 [41E02960] -> osm_report_notice: Reporting Generic
Notice type:1 num:128 from LID:0x0152
GID:0xfe80000000000000,0x0005ad0000027c84
Mar 20 14:15:27 533517 [45A08960] -> __osm_trap_rcv_process_request:
Received Generic Notice type:0x01 num:128 Producer:2 from LID:0x0152
TID:0x0000000000000003
Mar 20 14:15:27 533612 [45A08960] -> osm_report_notice: Reporting Generic
Notice type:1 num:128 from LID:0x0152
GID:0xfe80000000000000,0x0005ad0000027c84
Mar 20 14:15:27 591171 [41401960] -> osm_report_notice: Reporting Generic
Notice type:3 num:65 from LID:0x0092
GID:0xfe80000000000000,0x0005ad0000024bbb
Mar 20 14:15:27 591185 [41401960] -> Removed port with
GUID:0x0005ad0000024da7 LID range [0xB0,0xB0] of node:saguaro-23-1 HCA-1
Mar 20 14:15:27 591206 [41401960] -> osm_report_notice: Reporting Generic
Notice type:3 num:65 from LID:0x0092
GID:0xfe80000000000000,0x0005ad0000024bbb
Mar 20 14:15:27 591211 [41401960] -> Removed port with
GUID:0x0005ad00000249d3 LID range [0xB1,0xB1] of node:saguaro-23-2 HCA-1
Mar 20 14:15:27 625811 [41401960] -> osm_ucast_mgr_process: Min Hop Tables
configured on all switches
Mar 20 14:15:27 668356 [41401960] -> __osm_trap_rcv_process_request:
Received Generic Notice type:0x01 num:128 Producer:2 from LID:0x0152
TID:0x0000000000000004
Mar 20 14:15:27 668485 [41401960] -> osm_report_notice: Reporting Generic
Notice type:1 num:128 from LID:0x0152
GID:0xfe80000000000000,0x0005ad0000027c84
Mar 20 14:15:27 682282 [43204960] -> SUBNET UP
Mar 20 14:15:27 737313 [41E02960] -> __osm_trap_rcv_process_request:
Received Generic Notice type:0x01 num:128 Producer:2 from LID:0x0152
TID:0x0000000000000005
Mar 20 14:15:27 737387 [41E02960] -> osm_report_notice: Reporting Generic
Notice type:1 num:128 from LID:0x0152
GID:0xfe80000000000000,0x0005ad0000027c84
Mar 20 14:15:27 809341 [42803960] -> __osm_trap_rcv_process_request:
Received Generic Notice type:0x01 num:128 Producer:2 from LID:0x0152
TID:0x0000000000000006
Mar 20 14:15:27 809813 [42803960] -> osm_report_notice: Reporting Generic
Notice type:1 num:128 from LID:0x0152
GID:0xfe80000000000000,0x0005ad0000027c84
Mar 20 14:15:27 998181 [45007960] -> __osm_trap_rcv_process_request:
Received Generic Notice type:0x01 num:128 Producer:2 from LID:0x0152
TID:0x0000000000000007
Mar 20 14:15:27 998331 [45007960] -> osm_report_notice: Reporting Generic
Notice type:1 num:128 from LID:0x0152
GID:0xfe80000000000000,0x0005ad0000027c84
Mar 20 14:15:28 012193 [45007960] -> __osm_trap_rcv_process_request:
Received Generic Notice type:0x01 num:128 Producer:2 from LID:0x0152
TID:0x0000000000000008
Mar 20 14:15:28 012277 [45007960] -> osm_report_notice: Reporting Generic
Notice type:1 num:128 from LID:0x0152
GID:0xfe80000000000000,0x0005ad0000027c84
Mar 20 14:15:28 496329 [43204960] -> __osm_trap_rcv_process_request:
Received Generic Notice type:0x01 num:128 Producer:2 from LID:0x0152
TID:0x0000000000000009
Mar 20 14:15:28 496422 [43204960] -> osm_report_notice: Reporting Generic
Notice type:1 num:128 from LID:0x0152
GID:0xfe80000000000000,0x0005ad0000027c84
Mar 20 14:15:28 624912 [43C05960] -> osm_report_notice: Reporting Generic
Notice type:3 num:65 from LID:0x0092
GID:0xfe80000000000000,0x0005ad0000024bbb
Mar 20 14:15:28 624940 [43C05960] -> Removed port with
GUID:0x0005ad0000024e0b LID range [0xB3,0xB3] of node:saguaro-23-4 HCA-1
Mar 20 14:15:28 624965 [43C05960] -> osm_report_notice: Reporting Generic
Notice type:3 num:65 from LID:0x0092
GID:0xfe80000000000000,0x0005ad0000024bbb
Mar 20 14:15:28 624972 [43C05960] -> Removed port with
GUID:0x0005ad0000025043 LID range [0xB4,0xB4] of node:saguaro-23-5 HCA-1
Mar 20 14:15:28 625001 [43C05960] -> osm_report_notice: Reporting Generic
Notice type:3 num:65 from LID:0x0092
GID:0xfe80000000000000,0x0005ad0000024bbb
Mar 20 14:15:28 625008 [43C05960] -> Removed port with
GUID:0x0005ad0000024cbb LID range [0xB2,0xB2] of node:saguaro-23-3 HCA-1
Mar 20 14:15:28 629507 [42803960] -> osm_report_notice: Reporting Generic
Notice type:3 num:64 from LID:0x0092
GID:0xfe80000000000000,0x0005ad0000024bbb
Mar 20 14:15:28 629518 [42803960] -> Discovered new port with
GUID:0x0005ad00000249d3 LID range [0xB1,0xB1] of node:saguaro-23-2 HCA-1
Mar 20 14:15:28 649776 [43204960] -> __osm_trap_rcv_process_request:
Received Generic Notice type:0x01 num:128 Producer:2 from LID:0x0152
TID:0x000000000000000a
Mar 20 14:15:28 660297 [42803960] -> osm_ucast_mgr_process: Min Hop Tables
configured on all switches
Mar 20 14:15:28 699777 [43204960] -> osm_report_notice: Reporting Generic
Notice type:1 num:128 from LID:0x0152
GID:0xfe80000000000000,0x0005ad0000027c84
Mar 20 14:15:28 716354 [41E02960] -> SUBNET UP
Mar 20 14:15:28 744686 [45007960] -> __osm_trap_rcv_process_request:
Received Generic Notice type:0x01 num:128 Producer:2 from LID:0x0152
TID:0x000000000000000b
Mar 20 14:15:28 744857 [45007960] -> __osm_trap_rcv_process_request: ERR
3804: Received trap 11 times consecutively
Mar 20 14:15:28 811329 [45A08960] -> __osm_trap_rcv_process_request:
Received Generic Notice type:0x01 num:128 Producer:2 from LID:0x0152
TID:0x000000000000000c
Mar 20 14:15:28 811392 [45A08960] -> __osm_trap_rcv_process_request: ERR
3804: Received trap 12 times consecutively
Mar 20 14:15:28 999808 [45007960] -> __osm_trap_rcv_process_request:
Received Generic Notice type:0x01 num:128 Producer:2 from LID:0x0152
TID:0x000000000000000d
Mar 20 14:15:28 999881 [45007960] -> __osm_trap_rcv_process_request: ERR
3804: Received trap 13 times consecutively
Mar 20 14:15:29 029918 [43C05960] -> __osm_trap_rcv_process_request:
Received Generic Notice type:0x01 num:128 Producer:2 from LID:0x0152
TID:0x000000000000000e
Mar 20 14:15:29 029969 [43C05960] -> __osm_trap_rcv_process_request: ERR
3804: Received trap 14 times consecutively
Mar 20 14:15:29 031783 [45A08960] -> __osm_trap_rcv_process_request:
Received Generic Notice type:0x01 num:128 Producer:2 from LID:0x0148
TID:0x0000000000000052
Mar 20 14:15:29 031900 [45A08960] -> osm_report_notice: Reporting Generic
Notice type:1 num:128 from LID:0x0148
GID:0xfe80000000000000,0x0005ad00000281b3
Mar 20 14:15:29 037646 [44606960] -> osm_report_notice: Reporting Generic
Notice type:3 num:65 from LID:0x0092
GID:0xfe80000000000000,0x0005ad0000024bbb
Mar 20 14:15:29 037662 [44606960] -> Removed port with
GUID:0x0005ad0000024d6b LID range [0xB8,0xB8] of node:saguaro-23-9 HCA-1
Mar 20 14:15:29 037683 [44606960] -> osm_report_notice: Reporting Generic
Notice type:3 num:65 from LID:0x0092
GID:0xfe80000000000000,0x0005ad0000024bbb
Mar 20 14:15:29 037690 [44606960] -> Removed port with
GUID:0x0005ad0000024977 LID range [0xA9,0xA9] of node:saguaro-22-4 HCA-1
Mar 20 14:15:29 037721 [44606960] -> osm_report_notice: Reporting Generic
Notice type:3 num:65 from LID:0x0092
GID:0xfe80000000000000,0x0005ad0000024bbb
Mar 20 14:15:29 037726 [44606960] -> Removed port with
GUID:0x0005ad000002498f LID range [0xA8,0xA8] of node:saguaro-22-3 HCA-1
Mar 20 14:15:29 037741 [44606960] -> osm_report_notice: Reporting Generic
Notice type:3 num:65 from LID:0x0092
GID:0xfe80000000000000,0x0005ad0000024bbb
Mar 20 14:15:29 037746 [44606960] -> Removed port with
GUID:0x0005ad0000024c9b LID range [0xA7,0xA7] of node:saguaro-22-2 HCA-1
Mar 20 14:15:29 037766 [44606960] -> osm_report_notice: Reporting Generic
Notice type:3 num:65 from LID:0x0092
GID:0xfe80000000000000,0x0005ad0000024bbb
Mar 20 14:15:29 037771 [44606960] -> Removed port with
GUID:0x0005ad0000024afb LID range [0xA5,0xA5] of node:saguaro-22-0 HCA-1
Mar 20 14:15:29 361560 [42803960] -> __osm_trap_rcv_process_request:
Received Generic Notice type:0x01 num:128 Producer:2 from LID:0x0148
TID:0x0000000000000053
Mar 20 14:15:29 361622 [42803960] -> osm_report_notice: Reporting Generic
Notice type:1 num:128 from LID:0x0148
GID:0xfe80000000000000,0x0005ad00000281b3
Mar 20 14:15:29 433665 [43204960] -> osm_report_notice: Reporting Generic
Notice type:3 num:64 from LID:0x0092
GID:0xfe80000000000000,0x0005ad0000024bbb
Mar 20 14:15:29 433674 [43204960] -> Discovered new port with
GUID:0x0005ad0000024da7 LID range [0xB0,0xB0] of node:saguaro-23-1 HCA-1
Mar 20 14:15:29 433680 [43204960] -> osm_report_notice: Reporting Generic
Notice type:3 num:64 from LID:0x0092
GID:0xfe80000000000000,0x0005ad0000024bbb
Mar 20 14:15:29 433687 [43204960] -> Discovered new port with
GUID:0x0005ad0000024cbb LID range [0xB2,0xB2] of node:saguaro-23-3 HCA-1
Mar 20 14:15:29 433692 [43204960] -> osm_report_notice: Reporting Generic
Notice type:3 num:64 from LID:0x0092
GID:0xfe80000000000000,0x0005ad0000024bbb
Mar 20 14:15:29 433698 [43204960] -> Discovered new port with
GUID:0x0005ad0000024e0b LID range [0xB3,0xB3] of node:saguaro-23-4 HCA-1
Mar 20 14:15:29 433703 [43204960] -> osm_report_notice: Reporting Generic
Notice type:3 num:64 from LID:0x0092
GID:0xfe80000000000000,0x0005ad0000024bbb
Mar 20 14:15:29 433709 [43204960] -> Discovered new port with
GUID:0x0005ad0000025043 LID range [0xB4,0xB4] of node:saguaro-23-5 HCA-1
Mar 20 14:15:29 464434 [43204960] -> osm_ucast_mgr_process: Min Hop Tables
configured on all switches
Mar 20 14:15:29 522011 [42803960] -> SUBNET UP
Mar 20 14:15:29 699605 [41E02960] -> __osm_trap_rcv_process_request:
Received Generic Notice type:0x01 num:128 Producer:2 from LID:0x001B
TID:0x000000000000006b
Mar 20 14:15:29 699782 [41E02960] -> osm_report_notice: Reporting Generic
Notice type:1 num:128 from LID:0x001B
GID:0xfe80000000000000,0x0005ad00000281a7
Mar 20 14:15:29 701115 [45A08960] -> __osm_trap_rcv_process_request:
Received Generic Notice type:0x01 num:128 Producer:2 from LID:0x0148
TID:0x0000000000000054
Mar 20 14:15:29 701301 [45A08960] -> osm_report_notice: Reporting Generic
Notice type:1 num:128 from LID:0x0148
GID:0xfe80000000000000,0x0005ad00000281b3
Mar 20 14:15:29 818974 [41E02960] -> __osm_trap_rcv_process_request:
Received Generic Notice type:0x01 num:128 Producer:2 from LID:0x0148
TID:0x0000000000000055
Mar 20 14:15:29 819054 [41E02960] -> osm_report_notice: Reporting Generic
Notice type:1 num:128 from LID:0x0148
GID:0xfe80000000000000,0x0005ad00000281b3
Mar 20 14:15:29 992006 [41E02960] -> __osm_trap_rcv_process_request:
Received Generic Notice type:0x01 num:128 Producer:2 from LID:0x0148
TID:0x0000000000000056
Mar 20 14:15:29 992080 [41E02960] -> osm_report_notice: Reporting Generic
Notice type:1 num:128 from LID:0x0148
GID:0xfe80000000000000,0x0005ad00000281b3
Mar 20 14:15:30 184132 [44606960] -> __osm_trap_rcv_process_request:
Received Generic Notice type:0x01 num:128 Producer:2 from LID:0x001B
TID:0x000000000000006c
Mar 20 14:15:30 184205 [44606960] -> osm_report_notice: Reporting Generic
Notice type:1 num:128 from LID:0x001B
GID:0xfe80000000000000,0x0005ad00000281a7
Mar 20 14:15:30 207030 [43204960] -> __osm_trap_rcv_process_request:
Received Generic Notice type:0x01 num:128 Producer:2 from LID:0x0148
TID:0x0000000000000057
Mar 20 14:15:30 207101 [43204960] -> osm_report_notice: Reporting Generic
Notice type:1 num:128 from LID:0x0148
GID:0xfe80000000000000,0x0005ad00000281b3
Mar 20 14:15:30 250541 [43C05960] -> __osm_trap_rcv_process_request:
Received Generic Notice type:0x01 num:128 Producer:2 from LID:0x001B
TID:0x000000000000006d
Mar 20 14:15:30 250635 [43C05960] -> osm_report_notice: Reporting Generic
Notice type:1 num:128 from LID:0x001B
GID:0xfe80000000000000,0x0005ad00000281a7
Mar 20 14:15:30 317366 [45A08960] -> osm_drop_mgr_process: ERR 0108: Unknown
remote side for node 0x0005ad00000281a7 port 22. Adding to light sweep
sampling list
Mar 20 14:15:30 317409 [45A08960] -> Directed Path Dump of 4 hop path:
Path = [0][1][17][1][4]
Mar 20 14:15:30 494183 [41401960] -> __osm_trap_rcv_process_request:
Received Generic Notice type:0x01 num:128 Producer:2 from LID:0x001B
TID:0x000000000000006e
Mar 20 14:15:30 494247 [41401960] -> osm_report_notice: Reporting Generic
Notice type:1 num:128 from LID:0x001B
GID:0xfe80000000000000,0x0005ad00000281a7
Mar 20 14:15:30 521869 [43C05960] -> osm_report_notice: Reporting Generic
Notice type:3 num:64 from LID:0x0092
GID:0xfe80000000000000,0x0005ad0000024bbb
Mar 20 14:15:30 521879 [43C05960] -> Discovered new port with
GUID:0x0005ad0000024d6b LID range [0xB8,0xB8] of node:saguaro-23-9 HCA-1
Mar 20 14:15:30 521885 [43C05960] -> osm_report_notice: Reporting Generic
Notice type:3 num:64 from LID:0x0092
GID:0xfe80000000000000,0x0005ad0000024bbb
Mar 20 14:15:30 521891 [43C05960] -> Discovered new port with
GUID:0x0005ad0000024afb LID range [0xA5,0xA5] of node:saguaro-22-0 HCA-1
Mar 20 14:15:30 521896 [43C05960] -> osm_report_notice: Reporting Generic
Notice type:3 num:64 from LID:0x0092
GID:0xfe80000000000000,0x0005ad0000024bbb
Mar 20 14:15:30 521903 [43C05960] -> Discovered new port with
GUID:0x0005ad0000024c9b LID range [0xA7,0xA7] of node:saguaro-22-2 HCA-1
Mar 20 14:15:30 521908 [43C05960] -> osm_report_notice: Reporting Generic
Notice type:3 num:64 from LID:0x0092
GID:0xfe80000000000000,0x0005ad0000024bbb
Mar 20 14:15:30 521914 [43C05960] -> Discovered new port with
GUID:0x0005ad000002498f LID range [0xA8,0xA8] of node:saguaro-22-3 HCA-1
Mar 20 14:15:30 521919 [43C05960] -> osm_report_notice: Reporting Generic
Notice type:3 num:64 from LID:0x0092
GID:0xfe80000000000000,0x0005ad0000024bbb
Mar 20 14:15:30 521926 [43C05960] -> Discovered new port with
GUID:0x0005ad0000024977 LID range [0xA9,0xA9] of node:saguaro-22-4 HCA-1
Mar 20 14:15:30 552581 [43C05960] -> osm_ucast_mgr_process: Min Hop Tables
configured on all switches
Mar 20 14:15:30 553014 [45A08960] -> __osm_trap_rcv_process_request:
Received Generic Notice type:0x01 num:128 Producer:2 from LID:0x001B
TID:0x000000000000006f
Mar 20 14:15:30 592863 [45A08960] -> osm_report_notice: Reporting Generic
Notice type:1 num:128 from LID:0x001B
GID:0xfe80000000000000,0x0005ad00000281a7
Mar 20 14:15:30 607595 [4780B960] -> __osm_sm_mad_ctrl_rcv_callback: ERR
3111: Error status = 0x1C00
Mar 20 14:15:30 607666 [4780B960] -> SMP dump:
base_ver................0x1
mgmt_class..............0x81
class_ver...............0x1
method..................0x81 (SubnGetResp)
D bit...................0x1
status..................0x1C00
hop_ptr.................0x0
hop_count...............0x4
trans_id................0x6f744
attr_id.................0x15 (PortInfo)
resv....................0x0
attr_mod................0x16
m_key...................0x0000000000000000
dr_slid.................0xFFFF
dr_dlid.................0xFFFF
Initial path: [0][1][14][1][6]
Return path: [0][9][15][D][3]
Reserved: [0][0][0][0][0][0][0]
00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
00 00 00 00 00 00 00 00 00 00 00 00 03 03 03 02
14 52 00 11 40 40 00 08 08 04 F2 40 00 00 00 00
00 00 00 00 00 88 00 00 00 00 00 00 00 00 00 00
Mar 20 14:15:30 607770 [44606960] -> osm_pi_rcv_process_set: Received error
status 0x1c for SetResp() during ACTIVE transition
Mar 20 14:15:30 607777 [44606960] -> PortInfo dump:
port number.............0x16
node_guid...............0x0005ad00000281b3
port_guid...............0x0005ad00000281b3
m_key...................0x0000000000000000
subnet_prefix...........0x0000000000000000
base_lid................0x0
master_sm_base_lid......0x0
capability_mask.........0x0
diag_code...............0x0
m_key_lease_period......0x0
local_port_num..........0x3
link_width_enabled......0x3
link_width_supported....0x3
link_width_active.......0x2
link_speed_supported....0x1
port_state..............ACTIVE
state_info2.............0x52
m_key_protect_bits......0x0
lmc.....................0x0
link_speed..............0x11
mtu_smsl................0x40
vl_cap_init_type........0x40
vl_high_limit...........0x0
vl_arb_high_cap.........0x8
vl_arb_low_cap..........0x8
init_rep_mtu_cap........0x4
vl_stall_life...........0xF2
vl_enforce..............0x40
m_key_violations........0x0
p_key_violations........0x0
q_key_violations........0x0
guid_cap................0x0
client_reregister.......0x0
subnet_timeout..........0x0
resp_time_value.........0x0
error_threshold.........0x88
Mar 20 14:15:30 607794 [44606960] -> Capabilities Mask:
Mar 20 14:15:30 607914 [4780B960] -> __osm_sm_mad_ctrl_rcv_callback: ERR
3111: Error status = 0x1C00
Mar 20 14:15:30 607958 [4780B960] -> SMP dump:
base_ver................0x1
mgmt_class..............0x81
class_ver...............0x1
method..................0x81 (SubnGetResp)
D bit...................0x1
status..................0x1C00
hop_ptr.................0x0
hop_count...............0x4
trans_id................0x6f745
attr_id.................0x15 (PortInfo)
resv....................0x0
attr_mod................0x17
m_key...................0x0000000000000000
dr_slid.................0xFFFF
dr_dlid.................0xFFFF
Initial path: [0][1][14][1][6]
Return path: [0][9][15][D][3]
Reserved: [0][0][0][0][0][0][0]
00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
00 00 00 00 00 00 00 00 00 00 00 00 03 03 03 02
14 52 00 11 40 40 00 08 08 04 F2 40 00 00 00 00
00 00 00 00 00 88 00 00 00 00 00 00 00 00 00 00
Mar 20 14:15:30 608014 [43204960] -> osm_pi_rcv_process_set: Received error
status 0x1c for SetResp() during ACTIVE transition
Mar 20 14:15:30 608018 [43204960] -> PortInfo dump:
port number.............0x17
node_guid...............0x0005ad00000281b3
port_guid...............0x0005ad00000281b3
m_key...................0x0000000000000000
subnet_prefix...........0x0000000000000000
base_lid................0x0
master_sm_base_lid......0x0
capability_mask.........0x0
diag_code...............0x0
m_key_lease_period......0x0
local_port_num..........0x3
link_width_enabled......0x3
link_width_supported....0x3
link_width_active.......0x2
link_speed_supported....0x1
port_state..............ACTIVE
state_info2.............0x52
m_key_protect_bits......0x0
lmc.....................0x0
link_speed..............0x11
mtu_smsl................0x40
vl_cap_init_type........0x40
vl_high_limit...........0x0
vl_arb_high_cap.........0x8
vl_arb_low_cap..........0x8
init_rep_mtu_cap........0x4
vl_stall_life...........0xF2
vl_enforce..............0x40
m_key_violations........0x0
p_key_violations........0x0
q_key_violations........0x0
guid_cap................0x0
client_reregister.......0x0
subnet_timeout..........0x0
resp_time_value.........0x0
error_threshold.........0x88
Mar 20 14:15:30 608031 [43204960] -> Capabilities Mask:
Mar 20 14:15:30 613309 [41E02960] -> SUBNET UP
Mar 20 14:15:30 995108 [41401960] -> osm_ucast_mgr_process: Min Hop Tables
configured on all switches
Mar 20 14:15:31 050102 [4780B960] -> __osm_sm_mad_ctrl_rcv_callback: ERR
3111: Error status = 0x1C00
Mar 20 14:15:31 050180 [4780B960] -> SMP dump:
base_ver................0x1
mgmt_class..............0x81
class_ver...............0x1
method..................0x81 (SubnGetResp)
D bit...................0x1
status..................0x1C00
hop_ptr.................0x0
hop_count...............0x4
trans_id................0x70486
attr_id.................0x15 (PortInfo)
resv....................0x0
attr_mod................0x18
m_key...................0x0000000000000000
dr_slid.................0xFFFF
dr_dlid.................0xFFFF
Initial path: [0][1][11][1][4]
Return path: [0][9][18][D][1]
Reserved: [0][0][0][0][0][0][0]
00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
00 00 00 00 00 00 00 00 00 00 00 00 01 03 03 02
14 52 00 11 40 40 00 08 08 04 F2 40 00 00 00 00
00 00 00 00 00 88 00 00 00 00 00 00 00 00 00 00
Mar 20 14:15:31 050233 [45A08960] -> osm_pi_rcv_process_set: Received error
status 0x1c for SetResp() during ACTIVE transition
Mar 20 14:15:31 050238 [45A08960] -> PortInfo dump:
port number.............0x18
node_guid...............0x0005ad00000281b3
port_guid...............0x0005ad00000281b3
m_key...................0x0000000000000000
subnet_prefix...........0x0000000000000000
base_lid................0x0
master_sm_base_lid......0x0
capability_mask.........0x0
diag_code...............0x0
m_key_lease_period......0x0
local_port_num..........0x1
link_width_enabled......0x3
link_width_supported....0x3
link_width_active.......0x2
link_speed_supported....0x1
port_state..............ACTIVE
state_info2.............0x52
m_key_protect_bits......0x0
lmc.....................0x0
link_speed..............0x11
mtu_smsl................0x40
vl_cap_init_type........0x40
vl_high_limit...........0x0
vl_arb_high_cap.........0x8
vl_arb_low_cap..........0x8
init_rep_mtu_cap........0x4
vl_stall_life...........0xF2
vl_enforce..............0x40
m_key_violations........0x0
p_key_violations........0x0
q_key_violations........0x0
guid_cap................0x0
client_reregister.......0x0
subnet_timeout..........0x0
resp_time_value.........0x0
error_threshold.........0x88
Mar 20 14:15:31 050251 [45A08960] -> Capabilities Mask:
Mar 20 14:15:31 055273 [42803960] -> SUBNET UP
Mar 20 14:15:31 106129 [41401960] -> __osm_trap_rcv_process_request:
Received Generic Notice type:0x01 num:128 Producer:2 from LID:0x0152
TID:0x000000000000000e
Mar 20 14:15:31 106193 [41401960] -> __osm_trap_rcv_process_request: ERR
3804: Received trap 15 times consecutively
Mar 20 14:17:18 456260 [43204960] -> __osm_trap_rcv_process_request:
Received Generic Notice type:0x01 num:128 Producer:2 from LID:0x0148
TID:0x0000000000000058
Mar 20 14:17:18 456512 [43204960] -> osm_report_notice: Reporting Generic
Notice type:1 num:128 from LID:0x0148
GID:0xfe80000000000000,0x0005ad00000281b3
Mar 20 14:17:18 456649 [41E02960] -> __osm_trap_rcv_process_request:
Received Generic Notice type:0x01 num:128 Producer:2 from LID:0x001B
TID:0x0000000000000070
Mar 20 14:17:18 456761 [41E02960] -> osm_report_notice: Reporting Generic
Notice type:1 num:128 from LID:0x001B
GID:0xfe80000000000000,0x0005ad00000281a7
Mar 20 14:17:18 769730 [45007960] -> osm_report_notice: Reporting Generic
Notice type:3 num:65 from LID:0x0092
GID:0xfe80000000000000,0x0005ad0000024bbb
Mar 20 14:17:18 769751 [45007960] -> Removed port with
GUID:0x0005ad0000024e0b LID range [0xB3,0xB3] of node:saguaro-23-4 HCA-1
Mar 20 14:17:18 769773 [45007960] -> osm_report_notice: Reporting Generic
Notice type:3 num:65 from LID:0x0092
GID:0xfe80000000000000,0x0005ad0000024bbb
Mar 20 14:17:18 769780 [45007960] -> Removed port with
GUID:0x0005ad000002510b LID range [0xB5,0xB5] of node:saguaro-23-6 HCA-1
Mar 20 14:17:18 769803 [45007960] -> osm_report_notice: Reporting Generic
Notice type:3 num:65 from LID:0x0092
GID:0xfe80000000000000,0x0005ad0000024bbb
Mar 20 14:17:18 769809 [45007960] -> Removed port with
GUID:0x0005ad000002511b LID range [0xA6,0xA6] of node:saguaro-22-1 HCA-1
Mar 20 14:17:18 769832 [45007960] -> osm_report_notice: Reporting Generic
Notice type:3 num:65 from LID:0x0092
GID:0xfe80000000000000,0x0005ad0000024bbb
Mar 20 14:17:18 769838 [45007960] -> Removed port with
GUID:0x0005ad0000025043 LID range [0xB4,0xB4] of node:saguaro-23-5 HCA-1
Mar 20 14:17:18 769858 [45007960] -> osm_report_notice: Reporting Generic
Notice type:3 num:65 from LID:0x0092
GID:0xfe80000000000000,0x0005ad0000024bbb
Mar 20 14:17:18 769865 [45007960] -> Removed port with
GUID:0x0005ad0000024d47 LID range [0xB6,0xB6] of node:saguaro-23-7 HCA-1
Mar 20 14:17:18 769888 [45007960] -> osm_report_notice: Reporting Generic
Notice type:3 num:65 from LID:0x0092
GID:0xfe80000000000000,0x0005ad0000024bbb
Mar 20 14:17:18 769895 [45007960] -> Removed port with
GUID:0x0005ad0000024d6b LID range [0xB8,0xB8] of node:saguaro-23-9 HCA-1
Mar 20 14:17:18 769927 [45007960] -> osm_report_notice: Reporting Generic
Notice type:3 num:65 from LID:0x0092
GID:0xfe80000000000000,0x0005ad0000024bbb
Mar 20 14:17:18 769932 [45007960] -> Removed port with
GUID:0x0005ad0000024977 LID range [0xA9,0xA9] of node:saguaro-22-4 HCA-1
Mar 20 14:17:18 770075 [45007960] -> osm_report_notice: Reporting Generic
Notice type:3 num:65 from LID:0x0092
GID:0xfe80000000000000,0x0005ad0000024bbb
Mar 20 14:17:18 770081 [45007960] -> Removed port with
GUID:0x0005ad0000027c84 LID range [0x152,0x152] of node:Topspin Switch TS120
Mar 20 14:17:18 770109 [45007960] -> osm_report_notice: Reporting Generic
Notice type:3 num:65 from LID:0x0092
GID:0xfe80000000000000,0x0005ad0000024bbb
Mar 20 14:17:18 770114 [45007960] -> Removed port with
GUID:0x0005ad0000024d8b LID range [0xB7,0xB7] of node:saguaro-23-8 HCA-1
Mar 20 14:17:18 770130 [45007960] -> osm_report_notice: Reporting Generic
Notice type:3 num:65 from LID:0x0092
GID:0xfe80000000000000,0x0005ad0000024bbb
Mar 20 14:17:18 770135 [45007960] -> Removed port with
GUID:0x0005ad000002498f LID range [0xA8,0xA8] of node:saguaro-22-3 HCA-1
Mar 20 14:17:18 770150 [45007960] -> osm_report_notice: Reporting Generic
Notice type:3 num:65 from LID:0x0092
GID:0xfe80000000000000,0x0005ad0000024bbb
Mar 20 14:17:18 770155 [45007960] -> Removed port with
GUID:0x0005ad0000024c9b LID range [0xA7,0xA7] of node:saguaro-22-2 HCA-1
Mar 20 14:17:18 770171 [45007960] -> osm_report_notice: Reporting Generic
Notice type:3 num:65 from LID:0x0092
GID:0xfe80000000000000,0x0005ad0000024bbb
Mar 20 14:17:18 770176 [45007960] -> Removed port with
GUID:0x0005ad0000024da7 LID range [0xB0,0xB0] of node:saguaro-23-1 HCA-1
Mar 20 14:17:18 770193 [45007960] -> osm_report_notice: Reporting Generic
Notice type:3 num:65 from LID:0x0092
GID:0xfe80000000000000,0x0005ad0000024bbb
Mar 20 14:17:18 770198 [45007960] -> Removed port with
GUID:0x0005ad0000024cbb LID range [0xB2,0xB2] of node:saguaro-23-3 HCA-1
Mar 20 14:17:18 770216 [45007960] -> osm_report_notice: Reporting Generic
Notice type:3 num:65 from LID:0x0092
GID:0xfe80000000000000,0x0005ad0000024bbb
Mar 20 14:17:18 770221 [45007960] -> Removed port with
GUID:0x0005ad00000249d3 LID range [0xB1,0xB1] of node:saguaro-23-2 HCA-1
Mar 20 14:17:18 770238 [45007960] -> osm_report_notice: Reporting Generic
Notice type:3 num:65 from LID:0x0092
GID:0xfe80000000000000,0x0005ad0000024bbb
Mar 20 14:17:18 770301 [45007960] -> Removed port with
GUID:0x0005ad0000024feb LID range [0x153,0x153] of node:saguaro-22-5 HCA-1
Mar 20 14:17:18 770318 [45007960] -> osm_report_notice: Reporting Generic
Notice type:3 num:65 from LID:0x0092
GID:0xfe80000000000000,0x0005ad0000024bbb
Mar 20 14:17:18 770323 [45007960] -> Removed port with
GUID:0x0005ad0000024afb LID range [0xA5,0xA5] of node:saguaro-22-0 HCA-1
Mar 20 14:17:18 803377 [45007960] -> osm_ucast_mgr_process: Min Hop Tables
configured on all switches
Mar 20 14:17:18 855545 [44606960] -> SUBNET UP
Mar 20 14:17:19 249722 [43204960] -> osm_ucast_mgr_process: Min Hop Tables
configured on all switches
Mar 20 14:17:19 300999 [45A08960] -> SUBNET UP
Mar 20 14:18:11 663850 [43C05960] -> __osm_trap_rcv_process_request:
Received Generic Notice type:0x01 num:128 Producer:2 from LID:0x0148
TID:0x0000000000000059
Mar 20 14:18:11 664195 [43C05960] -> osm_report_notice: Reporting Generic
Notice type:1 num:128 from LID:0x0148
GID:0xfe80000000000000,0x0005ad00000281b3
Mar 20 14:18:11 670836 [41E02960] -> __osm_trap_rcv_process_request:
Received Generic Notice type:0x01 num:128 Producer:2 from LID:0x001B
TID:0x0000000000000071
Mar 20 14:18:11 670964 [41401960] -> __osm_trap_rcv_process_request:
Received Generic Notice type:0x01 num:128 Producer:2 from LID:0x0148
TID:0x000000000000005a
Mar 20 14:18:11 671199 [41E02960] -> osm_report_notice: Reporting Generic
Notice type:1 num:128 from LID:0x001B
GID:0xfe80000000000000,0x0005ad00000281a7
Mar 20 14:18:11 672933 [41401960] -> osm_report_notice: Reporting Generic
Notice type:1 num:128 from LID:0x0148
GID:0xfe80000000000000,0x0005ad00000281b3
Mar 20 14:18:11 677654 [44606960] -> __osm_trap_rcv_process_request:
Received Generic Notice type:0x01 num:128 Producer:2 from LID:0x001B
TID:0x0000000000000072
Mar 20 14:18:11 677826 [44606960] -> osm_report_notice: Reporting Generic
Notice type:1 num:128 from LID:0x001B
GID:0xfe80000000000000,0x0005ad00000281a7
Mar 20 14:18:12 026661 [44606960] -> osm_report_notice: Reporting Generic
Notice type:3 num:64 from LID:0x0092
GID:0xfe80000000000000,0x0005ad0000024bbb
Mar 20 14:18:12 026675 [44606960] -> Discovered new port with
GUID:0x0005ad0000027c84 LID range [0x152,0x152] of node:Topspin Switch TS120
Mar 20 14:18:12 026681 [44606960] -> osm_report_notice: Reporting Generic
Notice type:3 num:64 from LID:0x0092
GID:0xfe80000000000000,0x0005ad0000024bbb
Mar 20 14:18:12 026688 [44606960] -> Discovered new port with
GUID:0x0005ad0000024da7 LID range [0xB0,0xB0] of node:saguaro-23-1 HCA-1
Mar 20 14:18:12 026693 [44606960] -> osm_report_notice: Reporting Generic
Notice type:3 num:64 from LID:0x0092
GID:0xfe80000000000000,0x0005ad0000024bbb
Mar 20 14:18:12 026700 [44606960] -> Discovered new port with
GUID:0x0005ad00000249d3 LID range [0xB1,0xB1] of node:saguaro-23-2 HCA-1
Mar 20 14:18:12 026705 [44606960] -> osm_report_notice: Reporting Generic
Notice type:3 num:64 from LID:0x0092
GID:0xfe80000000000000,0x0005ad0000024bbb
Mar 20 14:18:12 026711 [44606960] -> Discovered new port with
GUID:0x0005ad0000024cbb LID range [0xB2,0xB2] of node:saguaro-23-3 HCA-1
Mar 20 14:18:12 026716 [44606960] -> osm_report_notice: Reporting Generic
Notice type:3 num:64 from LID:0x0092
GID:0xfe80000000000000,0x0005ad0000024bbb
Mar 20 14:18:12 026723 [44606960] -> Discovered new port with
GUID:0x0005ad0000024e0b LID range [0xB3,0xB3] of node:saguaro-23-4 HCA-1
Mar 20 14:18:12 026727 [44606960] -> osm_report_notice: Reporting Generic
Notice type:3 num:64 from LID:0x0092
GID:0xfe80000000000000,0x0005ad0000024bbb
Mar 20 14:18:12 026740 [44606960] -> Discovered new port with
GUID:0x0005ad0000025043 LID range [0xB4,0xB4] of node:saguaro-23-5 HCA-1
Mar 20 14:18:12 026745 [44606960] -> osm_report_notice: Reporting Generic
Notice type:3 num:64 from LID:0x0092
GID:0xfe80000000000000,0x0005ad0000024bbb
Mar 20 14:18:12 026751 [44606960] -> Discovered new port with
GUID:0x0005ad000002510b LID range [0xB5,0xB5] of node:saguaro-23-6 HCA-1
Mar 20 14:18:12 026758 [44606960] -> osm_report_notice: Reporting Generic
Notice type:3 num:64 from LID:0x0092
GID:0xfe80000000000000,0x0005ad0000024bbb
Mar 20 14:18:12 026764 [44606960] -> Discovered new port with
GUID:0x0005ad0000024d47 LID range [0xB6,0xB6] of node:saguaro-23-7 HCA-1
Mar 20 14:18:12 026769 [44606960] -> osm_report_notice: Reporting Generic
Notice type:3 num:64 from LID:0x0092
GID:0xfe80000000000000,0x0005ad0000024bbb
Mar 20 14:18:12 026776 [44606960] -> Discovered new port with
GUID:0x0005ad0000024d8b LID range [0xB7,0xB7] of node:saguaro-23-8 HCA-1
Mar 20 14:18:12 026781 [44606960] -> osm_report_notice: Reporting Generic
Notice type:3 num:64 from LID:0x0092
GID:0xfe80000000000000,0x0005ad0000024bbb
Mar 20 14:18:12 026787 [44606960] -> Discovered new port with
GUID:0x0005ad0000024d6b LID range [0xB8,0xB8] of node:saguaro-23-9 HCA-1
Mar 20 14:18:12 026792 [44606960] -> osm_report_notice: Reporting Generic
Notice type:3 num:64 from LID:0x0092
GID:0xfe80000000000000,0x0005ad0000024bbb
Mar 20 14:18:12 026798 [44606960] -> Discovered new port with
GUID:0x0005ad0000024afb LID range [0xA5,0xA5] of node:saguaro-22-0 HCA-1
Mar 20 14:18:12 026803 [44606960] -> osm_report_notice: Reporting Generic
Notice type:3 num:64 from LID:0x0092
GID:0xfe80000000000000,0x0005ad0000024bbb
Mar 20 14:18:12 026809 [44606960] -> Discovered new port with
GUID:0x0005ad000002511b LID range [0xA6,0xA6] of node:saguaro-22-1 HCA-1
Mar 20 14:18:12 026814 [44606960] -> osm_report_notice: Reporting Generic
Notice type:3 num:64 from LID:0x0092
GID:0xfe80000000000000,0x0005ad0000024bbb
Mar 20 14:18:12 026821 [44606960] -> Discovered new port with
GUID:0x0005ad0000024c9b LID range [0xA7,0xA7] of node:saguaro-22-2 HCA-1
Mar 20 14:18:12 026826 [44606960] -> osm_report_notice: Reporting Generic
Notice type:3 num:64 from LID:0x0092
GID:0xfe80000000000000,0x0005ad0000024bbb
Mar 20 14:18:12 026832 [44606960] -> Discovered new port with
GUID:0x0005ad000002498f LID range [0xA8,0xA8] of node:saguaro-22-3 HCA-1
Mar 20 14:18:12 026869 [44606960] -> osm_report_notice: Reporting Generic
Notice type:3 num:64 from LID:0x0092
GID:0xfe80000000000000,0x0005ad0000024bbb
Mar 20 14:18:12 026877 [44606960] -> Discovered new port with
GUID:0x0005ad0000024977 LID range [0xA9,0xA9] of node:saguaro-22-4 HCA-1
Mar 20 14:18:12 026882 [44606960] -> osm_report_notice: Reporting Generic
Notice type:3 num:64 from LID:0x0092
GID:0xfe80000000000000,0x0005ad0000024bbb
Mar 20 14:18:12 026888 [44606960] -> Discovered new port with
GUID:0x0005ad0000024feb LID range [0x153,0x153] of node:saguaro-22-5 HCA-1
Mar 20 14:18:12 057534 [44606960] -> osm_ucast_mgr_process: Min Hop Tables
configured on all switches
Mar 20 14:18:12 133316 [4780B960] -> __osm_sm_mad_ctrl_rcv_callback: ERR
3111: Error status = 0x1C00
Mar 20 14:18:12 133419 [4780B960] -> SMP dump:
base_ver................0x1
mgmt_class..............0x81
class_ver...............0x1
method..................0x81 (SubnGetResp)
D bit...................0x1
status..................0x1C00
hop_ptr.................0x0
hop_count...............0x4
trans_id................0x72d97
attr_id.................0x15 (PortInfo)
resv....................0x0
attr_mod................0x16
m_key...................0x0000000000000000
dr_slid.................0xFFFF
dr_dlid.................0xFFFF
Initial path: [0][1][14][3][6]
Return path: [0][9][15][F][3]
Reserved: [0][0][0][0][0][0][0]
00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
00 00 00 00 00 00 00 00 00 00 00 00 03 03 03 02
14 52 00 11 40 40 00 08 08 04 F2 40 00 00 00 00
00 00 00 00 00 88 00 00 00 00 00 00 00 00 00 00
Mar 20 14:18:12 133466 [4780B960] -> __osm_sm_mad_ctrl_rcv_callback: ERR
3111: Error status = 0x1C00
Mar 20 14:18:12 133467 [43204960] -> osm_pi_rcv_process_set: Received error
status 0x1c for SetResp() during ACTIVE transition
Mar 20 14:18:12 133478 [43204960] -> PortInfo dump:
port number.............0x16
node_guid...............0x0005ad00000281b3
port_guid...............0x0005ad00000281b3
m_key...................0x0000000000000000
subnet_prefix...........0x0000000000000000
base_lid................0x0
master_sm_base_lid......0x0
capability_mask.........0x0
diag_code...............0x0
m_key_lease_period......0x0
local_port_num..........0x3
link_width_enabled......0x3
link_width_supported....0x3
link_width_active.......0x2
link_speed_supported....0x1
port_state..............ACTIVE
state_info2.............0x52
m_key_protect_bits......0x0
lmc.....................0x0
link_speed..............0x11
mtu_smsl................0x40
vl_cap_init_type........0x40
vl_high_limit...........0x0
vl_arb_high_cap.........0x8
vl_arb_low_cap..........0x8
init_rep_mtu_cap........0x4
vl_stall_life...........0xF2
vl_enforce..............0x40
m_key_violations........0x0
p_key_violations........0x0
q_key_violations........0x0
guid_cap................0x0
client_reregister.......0x0
subnet_timeout..........0x0
resp_time_value.........0x0
error_threshold.........0x88
Mar 20 14:18:12 133490 [4780B960] -> SMP dump:
base_ver................0x1
mgmt_class..............0x81
class_ver...............0x1
method..................0x81 (SubnGetResp)
D bit...................0x1
status..................0x1C00
hop_ptr.................0x0
hop_count...............0x4
trans_id................0x72d98
attr_id.................0x15 (PortInfo)
resv....................0x0
attr_mod................0x17
m_key...................0x0000000000000000
dr_slid.................0xFFFF
dr_dlid.................0xFFFF
Initial path: [0][1][14][3][6]
Return path: [0][9][15][F][3]
Reserved: [0][0][0][0][0][0][0]
00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
00 00 00 00 00 00 00 00 00 00 00 00 03 03 03 02
14 52 00 11 40 40 00 08 08 04 F2 40 00 00 00 00
00 00 00 00 00 88 00 00 00 00 00 00 00 00 00 00
Mar 20 14:18:12 133493 [43204960] -> Capabilities Mask:
Mar 20 14:18:12 133566 [45A08960] -> osm_pi_rcv_process_set: Received error
status 0x1c for SetResp() during ACTIVE transition
Mar 20 14:18:12 133595 [45A08960] -> PortInfo dump:
port number.............0x17
node_guid...............0x0005ad00000281b3
port_guid...............0x0005ad00000281b3
m_key...................0x0000000000000000
subnet_prefix...........0x0000000000000000
base_lid................0x0
master_sm_base_lid......0x0
capability_mask.........0x0
diag_code...............0x0
m_key_lease_period......0x0
local_port_num..........0x3
link_width_enabled......0x3
link_width_supported....0x3
link_width_active.......0x2
link_speed_supported....0x1
port_state..............ACTIVE
state_info2.............0x52
m_key_protect_bits......0x0
lmc.....................0x0
link_speed..............0x11
mtu_smsl................0x40
vl_cap_init_type........0x40
vl_high_limit...........0x0
vl_arb_high_cap.........0x8
vl_arb_low_cap..........0x8
init_rep_mtu_cap........0x4
vl_stall_life...........0xF2
vl_enforce..............0x40
m_key_violations........0x0
p_key_violations........0x0
q_key_violations........0x0
guid_cap................0x0
client_reregister.......0x0
subnet_timeout..........0x0
resp_time_value.........0x0
error_threshold.........0x88
Mar 20 14:18:12 133614 [45A08960] -> Capabilities Mask:
Mar 20 14:18:12 133583 [4780B960] -> __osm_sm_mad_ctrl_rcv_callback: ERR
3111: Error status = 0x1C00
Mar 20 14:18:12 133671 [4780B960] -> SMP dump:
base_ver................0x1
mgmt_class..............0x81
class_ver...............0x1
method..................0x81 (SubnGetResp)
D bit...................0x1
status..................0x1C00
hop_ptr.................0x0
hop_count...............0x4
trans_id................0x72d99
attr_id.................0x15 (PortInfo)
resv....................0x0
attr_mod................0x18
m_key...................0x0000000000000000
dr_slid.................0xFFFF
dr_dlid.................0xFFFF
Initial path: [0][1][14][3][6]
Return path: [0][9][15][F][3]
Reserved: [0][0][0][0][0][0][0]
00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
00 00 00 00 00 00 00 00 00 00 00 00 03 03 03 02
14 52 00 11 40 40 00 08 08 04 F2 40 00 00 00 00
00 00 00 00 00 88 00 00 00 00 00 00 00 00 00 00
Mar 20 14:18:12 133760 [43C05960] -> osm_pi_rcv_process_set: Received error
status 0x1c for SetResp() during ACTIVE transition
Mar 20 14:18:12 133788 [43C05960] -> PortInfo dump:
port number.............0x18
node_guid...............0x0005ad00000281b3
port_guid...............0x0005ad00000281b3
m_key...................0x0000000000000000
subnet_prefix...........0x0000000000000000
base_lid................0x0
master_sm_base_lid......0x0
capability_mask.........0x0
diag_code...............0x0
m_key_lease_period......0x0
local_port_num..........0x3
link_width_enabled......0x3
link_width_supported....0x3
link_width_active.......0x2
link_speed_supported....0x1
port_state..............ACTIVE
state_info2.............0x52
m_key_protect_bits......0x0
lmc.....................0x0
link_speed..............0x11
mtu_smsl................0x40
vl_cap_init_type........0x40
vl_high_limit...........0x0
vl_arb_high_cap.........0x8
vl_arb_low_cap..........0x8
init_rep_mtu_cap........0x4
vl_stall_life...........0xF2
vl_enforce..............0x40
m_key_violations........0x0
p_key_violations........0x0
q_key_violations........0x0
guid_cap................0x0
client_reregister.......0x0
subnet_timeout..........0x0
resp_time_value.........0x0
error_threshold.........0x88
Mar 20 14:18:12 133807 [43C05960] -> Capabilities Mask:
Mar 20 14:18:12 139330 [41401960] -> SUBNET UP
Mar 20 14:18:12 496444 [45A08960] -> osm_ucast_mgr_process: Min Hop Tables
configured on all switches
Mar 20 14:18:12 558965 [41401960] -> SUBNET UP
Mar 20 14:18:27 748551 [43C05960] -> __osm_trap_rcv_process_request:
Received Generic Notice type:0x01 num:128 Producer:2 from LID:0x0152
TID:0x0000000000000000
Mar 20 14:18:27 748795 [43C05960] -> osm_report_notice: Reporting Generic
Notice type:1 num:128 from LID:0x0152
GID:0xfe80000000000000,0x0005ad0000027c84
Mar 20 14:18:27 888669 [44606960] -> __osm_trap_rcv_process_request:
Received Generic Notice type:0x01 num:128 Producer:2 from LID:0x0152
TID:0x0000000000000001
Mar 20 14:18:27 888902 [44606960] -> osm_report_notice: Reporting Generic
Notice type:1 num:128 from LID:0x0152
GID:0xfe80000000000000,0x0005ad0000027c84
Mar 20 14:18:27 910605 [44606960] -> __osm_trap_rcv_process_request:
Received Generic Notice type:0x01 num:128 Producer:2 from LID:0x0152
TID:0x0000000000000002
Mar 20 14:18:27 910710 [44606960] -> osm_report_notice: Reporting Generic
Notice type:1 num:128 from LID:0x0152
GID:0xfe80000000000000,0x0005ad0000027c84
Mar 20 14:18:27 911951 [41E02960] -> __osm_trap_rcv_process_request:
Received Generic Notice type:0x01 num:128 Producer:2 from LID:0x0152
TID:0x0000000000000003
Mar 20 14:18:27 912119 [41E02960] -> osm_report_notice: Reporting Generic
Notice type:1 num:128 from LID:0x0152
GID:0xfe80000000000000,0x0005ad0000027c84
Mar 20 14:18:28 012957 [45A08960] -> __osm_trap_rcv_process_request:
Received Generic Notice type:0x01 num:128 Producer:2 from LID:0x0152
TID:0x0000000000000004
Mar 20 14:18:28 013058 [45A08960] -> osm_report_notice: Reporting Generic
Notice type:1 num:128 from LID:0x0152
GID:0xfe80000000000000,0x0005ad0000027c84
Mar 20 14:18:28 075266 [43C05960] -> __osm_trap_rcv_process_request:
Received Generic Notice type:0x01 num:128 Producer:2 from LID:0x0152
TID:0x0000000000000005
Mar 20 14:18:28 075397 [43C05960] -> osm_report_notice: Reporting Generic
Notice type:1 num:128 from LID:0x0152
GID:0xfe80000000000000,0x0005ad0000027c84
Mar 20 14:18:28 259000 [41E02960] -> __osm_trap_rcv_process_request:
Received Generic Notice type:0x01 num:128 Producer:2 from LID:0x0152
TID:0x0000000000000006
Mar 20 14:18:28 259121 [41E02960] -> osm_report_notice: Reporting Generic
Notice type:1 num:128 from LID:0x0152
GID:0xfe80000000000000,0x0005ad0000027c84
Mar 20 14:18:28 308865 [42803960] -> __osm_trap_rcv_process_request:
Received Generic Notice type:0x01 num:128 Producer:2 from LID:0x0152
TID:0x0000000000000007
Mar 20 14:18:28 309000 [42803960] -> osm_report_notice: Reporting Generic
Notice type:1 num:128 from LID:0x0152
GID:0xfe80000000000000,0x0005ad0000027c84
Mar 20 14:18:28 330606 [45007960] -> __osm_trap_rcv_process_request:
Received Generic Notice type:0x01 num:128 Producer:2 from LID:0x0152
TID:0x0000000000000008
Mar 20 14:18:28 330714 [45007960] -> osm_report_notice: Reporting Generic
Notice type:1 num:128 from LID:0x0152
GID:0xfe80000000000000,0x0005ad0000027c84
Mar 20 14:18:28 444107 [45A08960] -> __osm_trap_rcv_process_request:
Received Generic Notice type:0x01 num:128 Producer:2 from LID:0x0152
TID:0x0000000000000009
Mar 20 14:18:28 444191 [45A08960] -> osm_report_notice: Reporting Generic
Notice type:1 num:128 from LID:0x0152
GID:0xfe80000000000000,0x0005ad0000027c84
Mar 20 14:18:28 466156 [44606960] -> __osm_trap_rcv_process_request:
Received Generic Notice type:0x01 num:128 Producer:2 from LID:0x0152
TID:0x000000000000000a
Mar 20 14:18:28 466234 [44606960] -> osm_report_notice: Reporting Generic
Notice type:1 num:128 from LID:0x0152
GID:0xfe80000000000000,0x0005ad0000027c84
Mar 20 14:18:28 478021 [43C05960] -> __osm_trap_rcv_process_request:
Received Generic Notice type:0x01 num:128 Producer:2 from LID:0x0152
TID:0x000000000000000b
Mar 20 14:18:28 478070 [43C05960] -> __osm_trap_rcv_process_request: ERR
3804: Received trap 11 times consecutively
Mar 20 14:18:28 489091 [43204960] -> osm_report_notice: Reporting Generic
Notice type:3 num:65 from LID:0x0092
GID:0xfe80000000000000,0x0005ad0000024bbb
Mar 20 14:18:28 489106 [43204960] -> Removed port with
GUID:0x0005ad00000249d3 LID range [0xB1,0xB1] of node:saguaro-23-2 HCA-1
Mar 20 14:18:28 521430 [42803960] -> __osm_trap_rcv_process_request:
Received Generic Notice type:0x01 num:128 Producer:2 from LID:0x0152
TID:0x000000000000000c
Mar 20 14:18:28 521499 [42803960] -> __osm_trap_rcv_process_request: ERR
3804: Received trap 12 times consecutively
Mar 20 14:18:28 523658 [43204960] -> osm_ucast_mgr_process: Min Hop Tables
configured on all switches
Mar 20 14:18:28 580295 [43204960] -> SUBNET UP
Mar 20 14:18:28 611805 [43204960] -> __osm_trap_rcv_process_request:
Received Generic Notice type:0x01 num:128 Producer:2 from LID:0x0152
TID:0x000000000000000d
Mar 20 14:18:28 611893 [43204960] -> __osm_trap_rcv_process_request: ERR
3804: Received trap 13 times consecutively
Mar 20 14:18:28 661292 [45A08960] -> __osm_trap_rcv_process_request:
Received Generic Notice type:0x01 num:128 Producer:2 from LID:0x0152
TID:0x000000000000000e
Mar 20 14:18:28 661351 [45A08960] -> __osm_trap_rcv_process_request: ERR
3804: Received trap 14 times consecutively
Mar 20 14:18:28 871670 [44606960] -> __osm_trap_rcv_process_request:
Received Generic Notice type:0x01 num:128 Producer:2 from LID:0x0152
TID:0x000000000000000f
Mar 20 14:18:28 871732 [44606960] -> __osm_trap_rcv_process_request: ERR
3804: Received trap 15 times consecutively
Mar 20 14:18:28 934440 [43204960] -> __osm_trap_rcv_process_request:
Received Generic Notice type:0x01 num:128 Producer:2 from LID:0x0152
TID:0x0000000000000010
Mar 20 14:18:28 934505 [43204960] -> __osm_trap_rcv_process_request: ERR
3804: Received trap 16 times consecutively
Mar 20 14:18:28 941281 [45A08960] -> osm_report_notice: Reporting Generic
Notice type:3 num:65 from LID:0x0092
GID:0xfe80000000000000,0x0005ad0000024bbb
Mar 20 14:18:28 941303 [45A08960] -> Removed port with
GUID:0x0005ad000002510b LID range [0xB5,0xB5] of node:saguaro-23-6 HCA-1
Mar 20 14:18:28 941329 [45A08960] -> osm_report_notice: Reporting Generic
Notice type:3 num:65 from LID:0x0092
GID:0xfe80000000000000,0x0005ad0000024bbb
Mar 20 14:18:28 941336 [45A08960] -> Removed port with
GUID:0x0005ad0000025043 LID range [0xB4,0xB4] of node:saguaro-23-5 HCA-1
Mar 20 14:18:28 941356 [45A08960] -> osm_report_notice: Reporting Generic
Notice type:3 num:65 from LID:0x0092
GID:0xfe80000000000000,0x0005ad0000024bbb
Mar 20 14:18:28 941363 [45A08960] -> Removed port with
GUID:0x0005ad0000024d47 LID range [0xB6,0xB6] of node:saguaro-23-7 HCA-1
Mar 20 14:18:28 941388 [45A08960] -> osm_report_notice: Reporting Generic
Notice type:3 num:65 from LID:0x0092
GID:0xfe80000000000000,0x0005ad0000024bbb
Mar 20 14:18:28 941395 [45A08960] -> Removed port with
GUID:0x0005ad0000024d8b LID range [0xB7,0xB7] of node:saguaro-23-8 HCA-1
Mar 20 14:18:28 941420 [45A08960] -> osm_report_notice: Reporting Generic
Notice type:3 num:65 from LID:0x0092
GID:0xfe80000000000000,0x0005ad0000024bbb
Mar 20 14:18:28 941426 [45A08960] -> Removed port with
GUID:0x0005ad0000024cbb LID range [0xB2,0xB2] of node:saguaro-23-3 HCA-1
Mar 20 14:18:28 945507 [45A08960] -> osm_report_notice: Reporting Generic
Notice type:3 num:64 from LID:0x0092
GID:0xfe80000000000000,0x0005ad0000024bbb
Mar 20 14:18:28 945515 [45A08960] -> Discovered new port with
GUID:0x0005ad00000249d3 LID range [0xB1,0xB1] of node:saguaro-23-2 HCA-1
Mar 20 14:18:28 956576 [44606960] -> __osm_trap_rcv_process_request:
Received Generic Notice type:0x01 num:128 Producer:2 from LID:0x0152
TID:0x0000000000000011
Mar 20 14:18:28 956665 [44606960] -> __osm_trap_rcv_process_request: ERR
3804: Received trap 17 times consecutively
Mar 20 14:18:28 976211 [45A08960] -> osm_ucast_mgr_process: Min Hop Tables
configured on all switches
Mar 20 14:18:29 033513 [42803960] -> SUBNET UP
Mar 20 14:18:29 071283 [41401960] -> __osm_trap_rcv_process_request:
Received Generic Notice type:0x01 num:128 Producer:2 from LID:0x0152
TID:0x0000000000000012
Mar 20 14:18:29 071345 [41401960] -> __osm_trap_rcv_process_request: ERR
3804: Received trap 18 times consecutively
Mar 20 14:18:29 352103 [44606960] -> __osm_trap_rcv_process_request:
Received Generic Notice type:0x01 num:128 Producer:2 from LID:0x0152
TID:0x0000000000000013
Mar 20 14:18:29 352155 [44606960] -> __osm_trap_rcv_process_request: ERR
3804: Received trap 19 times consecutively
Mar 20 14:18:29 376386 [41E02960] -> __osm_trap_rcv_process_request:
Received Generic Notice type:0x01 num:128 Producer:2 from LID:0x0152
TID:0x0000000000000014
Mar 20 14:18:29 376461 [41E02960] -> __osm_trap_rcv_process_request: ERR
3804: Received trap 20 times consecutively
Mar 20 14:18:29 420228 [43204960] -> __osm_trap_rcv_process_request:
Received Generic Notice type:0x01 num:128 Producer:2 from LID:0x0152
TID:0x0000000000000015
Mar 20 14:18:29 420282 [43204960] -> __osm_trap_rcv_process_request: ERR
3804: Received trap 21 times consecutively
Mar 20 14:18:29 421294 [43C05960] -> __osm_trap_rcv_process_request:
Received Generic Notice type:0x01 num:128 Producer:2 from LID:0x0152
TID:0x0000000000000016
Mar 20 14:18:29 421345 [43C05960] -> __osm_trap_rcv_process_request: ERR
3804: Received trap 22 times consecutively
Mar 20 14:18:29 461135 [45A08960] -> __osm_trap_rcv_process_request:
Received Generic Notice type:0x01 num:128 Producer:2 from LID:0x0152
TID:0x0000000000000017
Mar 20 14:18:29 461179 [45A08960] -> __osm_trap_rcv_process_request: ERR
3804: Received trap 23 times consecutively
Mar 20 14:18:29 633008 [45007960] -> __osm_trap_rcv_process_request:
Received Generic Notice type:0x01 num:128 Producer:2 from LID:0x0152
TID:0x0000000000000018
Mar 20 14:18:29 633050 [42803960] -> __osm_trap_rcv_process_request:
Received Generic Notice type:0x01 num:128 Producer:2 from LID:0x0148
TID:0x000000000000005b
Mar 20 14:18:29 633062 [45007960] -> __osm_trap_rcv_process_request: ERR
3804: Received trap 24 times consecutively
Mar 20 14:18:29 633350 [42803960] -> osm_report_notice: Reporting Generic
Notice type:1 num:128 from LID:0x0148
GID:0xfe80000000000000,0x0005ad00000281b3
Mar 20 14:18:29 733039 [45A08960] -> __osm_trap_rcv_process_request:
Received Generic Notice type:0x01 num:128 Producer:2 from LID:0x0148
TID:0x000000000000005c
Mar 20 14:18:29 733238 [45A08960] -> osm_report_notice: Reporting Generic
Notice type:1 num:128 from LID:0x0148
GID:0xfe80000000000000,0x0005ad00000281b3
Mar 20 14:18:29 947440 [44606960] -> osm_report_notice: Reporting Generic
Notice type:3 num:64 from LID:0x0092
GID:0xfe80000000000000,0x0005ad0000024bbb
Mar 20 14:18:29 947452 [44606960] -> Discovered new port with
GUID:0x0005ad0000024cbb LID range [0xB2,0xB2] of node:saguaro-23-3 HCA-1
Mar 20 14:18:29 947457 [44606960] -> osm_report_notice: Reporting Generic
Notice type:3 num:64 from LID:0x0092
GID:0xfe80000000000000,0x0005ad0000024bbb
Mar 20 14:18:29 947462 [44606960] -> Discovered new port with
GUID:0x0005ad0000025043 LID range [0xB4,0xB4] of node:saguaro-23-5 HCA-1
Mar 20 14:18:29 947465 [44606960] -> osm_report_notice: Reporting Generic
Notice type:3 num:64 from LID:0x0092
GID:0xfe80000000000000,0x0005ad0000024bbb
Mar 20 14:18:29 947470 [44606960] -> Discovered new port with
GUID:0x0005ad000002510b LID range [0xB5,0xB5] of node:saguaro-23-6 HCA-1
Mar 20 14:18:29 947474 [44606960] -> osm_report_notice: Reporting Generic
Notice type:3 num:64 from LID:0x0092
GID:0xfe80000000000000,0x0005ad0000024bbb
Mar 20 14:18:29 947479 [44606960] -> Discovered new port with
GUID:0x0005ad0000024d47 LID range [0xB6,0xB6] of node:saguaro-23-7 HCA-1
Mar 20 14:18:29 947482 [44606960] -> osm_report_notice: Reporting Generic
Notice type:3 num:64 from LID:0x0092
GID:0xfe80000000000000,0x0005ad0000024bbb
Mar 20 14:18:29 947487 [44606960] -> Discovered new port with
GUID:0x0005ad0000024d8b LID range [0xB7,0xB7] of node:saguaro-23-8 HCA-1
Mar 20 14:18:29 978182 [44606960] -> osm_ucast_mgr_process: Min Hop Tables
configured on all switches
Mar 20 14:18:30 027730 [4780B960] -> __osm_sm_mad_ctrl_rcv_callback: ERR
3111: Error status = 0x1C00
Mar 20 14:18:30 027819 [4780B960] -> SMP dump:
base_ver................0x1
mgmt_class..............0x81
class_ver...............0x1
method..................0x81 (SubnGetResp)
D bit...................0x1
status..................0x1C00
hop_ptr.................0x0
hop_count...............0x4
trans_id................0x762b8
attr_id.................0x15 (PortInfo)
resv....................0x0
attr_mod................0x16
m_key...................0x0000000000000000
dr_slid.................0xFFFF
dr_dlid.................0xFFFF
Initial path: [0][1][11][1][4]
Return path: [0][9][18][D][1]
Reserved: [0][0][0][0][0][0][0]
00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
00 00 00 00 00 00 00 00 00 00 00 00 01 03 03 02
14 52 00 11 40 40 00 08 08 04 F2 40 00 00 00 00
00 00 00 00 00 88 00 00 00 00 00 00 00 00 00 00
Mar 20 14:18:30 027897 [41401960] -> osm_pi_rcv_process_set: Received error
status 0x1c for SetResp() during ACTIVE transition
Mar 20 14:18:30 027901 [41401960] -> PortInfo dump:
port number.............0x16
node_guid...............0x0005ad00000281b3
port_guid...............0x0005ad00000281b3
m_key...................0x0000000000000000
subnet_prefix...........0x0000000000000000
base_lid................0x0
master_sm_base_lid......0x0
capability_mask.........0x0
diag_code...............0x0
m_key_lease_period......0x0
local_port_num..........0x1
link_width_enabled......0x3
link_width_supported....0x3
link_width_active.......0x2
link_speed_supported....0x1
port_state..............ACTIVE
state_info2.............0x52
m_key_protect_bits......0x0
lmc.....................0x0
link_speed..............0x11
mtu_smsl................0x40
vl_cap_init_type........0x40
vl_high_limit...........0x0
vl_arb_high_cap.........0x8
vl_arb_low_cap..........0x8
init_rep_mtu_cap........0x4
vl_stall_life...........0xF2
vl_enforce..............0x40
m_key_violations........0x0
p_key_violations........0x0
q_key_violations........0x0
guid_cap................0x0
client_reregister.......0x0
subnet_timeout..........0x0
resp_time_value.........0x0
error_threshold.........0x88
Mar 20 14:18:30 027914 [41401960] -> Capabilities Mask:
Mar 20 14:18:30 027993 [4780B960] -> __osm_sm_mad_ctrl_rcv_callback: ERR
3111: Error status = 0x1C00
Mar 20 14:18:30 028043 [4780B960] -> SMP dump:
base_ver................0x1
mgmt_class..............0x81
class_ver...............0x1
method..................0x81 (SubnGetResp)
D bit...................0x1
status..................0x1C00
hop_ptr.................0x0
hop_count...............0x4
trans_id................0x762b9
attr_id.................0x15 (PortInfo)
resv....................0x0
attr_mod................0x17
m_key...................0x0000000000000000
dr_slid.................0xFFFF
dr_dlid.................0xFFFF
Initial path: [0][1][11][1][4]
Return path: [0][9][18][D][1]
Reserved: [0][0][0][0][0][0][0]
00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
00 00 00 00 00 00 00 00 00 00 00 00 01 03 03 02
14 52 00 11 40 40 00 08 08 04 F2 40 00 00 00 00
00 00 00 00 00 88 00 00 00 00 00 00 00 00 00 00
Mar 20 14:18:30 028098 [45A08960] -> osm_pi_rcv_process_set: Received error
status 0x1c for SetResp() during ACTIVE transition
Mar 20 14:18:30 028109 [45A08960] -> PortInfo dump:
port number.............0x17
node_guid...............0x0005ad00000281b3
port_guid...............0x0005ad00000281b3
m_key...................0x0000000000000000
subnet_prefix...........0x0000000000000000
base_lid................0x0
master_sm_base_lid......0x0
capability_mask.........0x0
diag_code...............0x0
m_key_lease_period......0x0
local_port_num..........0x1
link_width_enabled......0x3
link_width_supported....0x3
link_width_active.......0x2
link_speed_supported....0x1
port_state..............ACTIVE
state_info2.............0x52
m_key_protect_bits......0x0
lmc.....................0x0
link_speed..............0x11
mtu_smsl................0x40
vl_cap_init_type........0x40
vl_high_limit...........0x0
vl_arb_high_cap.........0x8
vl_arb_low_cap..........0x8
init_rep_mtu_cap........0x4
vl_stall_life...........0xF2
vl_enforce..............0x40
m_key_violations........0x0
p_key_violations........0x0
q_key_violations........0x0
guid_cap................0x0
client_reregister.......0x0
subnet_timeout..........0x0
resp_time_value.........0x0
error_threshold.........0x88
Mar 20 14:18:30 028124 [45A08960] -> Capabilities Mask:
Mar 20 14:18:30 033824 [44606960] -> SUBNET UP
Mar 20 14:18:30 418497 [43C05960] -> osm_report_notice: Reporting Generic
Notice type:3 num:65 from LID:0x0092
GID:0xfe80000000000000,0x0005ad0000024bbb
Mar 20 14:18:30 418522 [43C05960] -> Removed port with
GUID:0x0005ad0000024feb LID range [0x153,0x153] of node:saguaro-22-5 HCA-1
Mar 20 14:18:30 453167 [43C05960] -> osm_ucast_mgr_process: Min Hop Tables
configured on all switches
Mar 20 14:18:30 494719 [41E02960] -> __osm_trap_rcv_process_request:
Received Generic Notice type:0x01 num:128 Producer:2 from LID:0x0148
TID:0x000000000000005d
Mar 20 14:18:30 494877 [41E02960] -> osm_report_notice: Reporting Generic
Notice type:1 num:128 from LID:0x0148
GID:0xfe80000000000000,0x0005ad00000281b3
Mar 20 14:18:30 662496 [44606960] -> __osm_trap_rcv_process_request:
Received Generic Notice type:0x01 num:128 Producer:2 from LID:0x001B
TID:0x0000000000000073
Mar 20 14:18:30 662564 [44606960] -> osm_report_notice: Reporting Generic
Notice type:1 num:128 from LID:0x001B
GID:0xfe80000000000000,0x0005ad00000281a7
Mar 20 14:18:30 662645 [43C05960] -> __osm_trap_rcv_process_request:
Received Generic Notice type:0x01 num:128 Producer:2 from LID:0x0148
TID:0x000000000000005e
Mar 20 14:18:30 662759 [43C05960] -> osm_report_notice: Reporting Generic
Notice type:1 num:128 from LID:0x0148
GID:0xfe80000000000000,0x0005ad00000281b3
Mar 20 14:18:30 707085 [42803960] -> __osm_trap_rcv_process_request:
Received Generic Notice type:0x01 num:128 Producer:2 from LID:0x0148
TID:0x000000000000005f
Mar 20 14:18:30 707179 [42803960] -> osm_report_notice: Reporting Generic
Notice type:1 num:128 from LID:0x0148
GID:0xfe80000000000000,0x0005ad00000281b3
Mar 20 14:18:30 728948 [41E02960] -> __osm_trap_rcv_process_request:
Received Generic Notice type:0x01 num:128 Producer:2 from LID:0x0148
TID:0x0000000000000060
Mar 20 14:18:30 729041 [41E02960] -> osm_report_notice: Reporting Generic
Notice type:1 num:128 from LID:0x0148
GID:0xfe80000000000000,0x0005ad00000281b3
Mar 20 14:18:30 872332 [45A08960] -> __osm_trap_rcv_process_request:
Received Generic Notice type:0x01 num:128 Producer:2 from LID:0x0148
TID:0x0000000000000061
Mar 20 14:18:30 872412 [45A08960] -> osm_report_notice: Reporting Generic
Notice type:1 num:128 from LID:0x0148
GID:0xfe80000000000000,0x0005ad00000281b3
Mar 20 14:18:30 899764 [45A08960] -> SUBNET UP
Mar 20 14:18:31 047423 [43C05960] -> __osm_trap_rcv_process_request:
Received Generic Notice type:0x01 num:128 Producer:2 from LID:0x0148
TID:0x0000000000000062
Mar 20 14:18:31 047611 [43C05960] -> osm_report_notice: Reporting Generic
Notice type:1 num:128 from LID:0x0148
GID:0xfe80000000000000,0x0005ad00000281b3
Mar 20 14:18:31 165201 [45A08960] -> __osm_trap_rcv_process_request:
Received Generic Notice type:0x01 num:128 Producer:2 from LID:0x0148
TID:0x0000000000000063
Mar 20 14:18:31 165272 [45A08960] -> osm_report_notice: Reporting Generic
Notice type:1 num:128 from LID:0x0148
GID:0xfe80000000000000,0x0005ad00000281b3
Mar 20 14:18:31 182461 [44606960] -> __osm_trap_rcv_process_request:
Received Generic Notice type:0x01 num:128 Producer:2 from LID:0x001B
TID:0x0000000000000074
Mar 20 14:18:31 182653 [44606960] -> osm_report_notice: Reporting Generic
Notice type:1 num:128 from LID:0x001B
GID:0xfe80000000000000,0x0005ad00000281a7
Mar 20 14:18:31 248834 [44606960] -> __osm_trap_rcv_process_request:
Received Generic Notice type:0x01 num:128 Producer:2 from LID:0x001B
TID:0x0000000000000075
Mar 20 14:18:31 248893 [44606960] -> osm_report_notice: Reporting Generic
Notice type:1 num:128 from LID:0x001B
GID:0xfe80000000000000,0x0005ad00000281a7
Mar 20 14:18:31 499830 [45A08960] -> __osm_trap_rcv_process_request:
Received Generic Notice type:0x01 num:128 Producer:2 from LID:0x001B
TID:0x0000000000000076
Mar 20 14:18:31 499908 [45A08960] -> osm_report_notice: Reporting Generic
Notice type:1 num:128 from LID:0x001B
GID:0xfe80000000000000,0x0005ad00000281a7
Mar 20 14:18:31 521824 [41401960] -> __osm_trap_rcv_process_request:
Received Generic Notice type:0x01 num:128 Producer:2 from LID:0x001B
TID:0x0000000000000077
Mar 20 14:18:31 521891 [41401960] -> osm_report_notice: Reporting Generic
Notice type:1 num:128 from LID:0x001B
GID:0xfe80000000000000,0x0005ad00000281a7
Mar 20 14:18:31 543713 [44606960] -> __osm_trap_rcv_process_request:
Received Generic Notice type:0x01 num:128 Producer:2 from LID:0x001B
TID:0x0000000000000078
Mar 20 14:18:31 543784 [44606960] -> osm_report_notice: Reporting Generic
Notice type:1 num:128 from LID:0x001B
GID:0xfe80000000000000,0x0005ad00000281a7
Mar 20 14:18:31 589490 [43C05960] -> osm_report_notice: Reporting Generic
Notice type:3 num:64 from LID:0x0092
GID:0xfe80000000000000,0x0005ad0000024bbb
Mar 20 14:18:31 589499 [43C05960] -> Discovered new port with
GUID:0x0005ad0000024feb LID range [0x153,0x153] of node:saguaro-22-5 HCA-1
Mar 20 14:18:31 620166 [43C05960] -> osm_ucast_mgr_process: Min Hop Tables
configured on all switches
Mar 20 14:18:31 672647 [4780B960] -> __osm_sm_mad_ctrl_rcv_callback: ERR
3111: Error status = 0x1C00
Mar 20 14:18:31 672739 [4780B960] -> SMP dump:
base_ver................0x1
mgmt_class..............0x81
class_ver...............0x1
method..................0x81 (SubnGetResp)
D bit...................0x1
status..................0x1C00
hop_ptr.................0x0
hop_count...............0x4
trans_id................0x77d11
attr_id.................0x15 (PortInfo)
resv....................0x0
attr_mod................0x16
m_key...................0x0000000000000000
dr_slid.................0xFFFF
dr_dlid.................0xFFFF
Initial path: [0][1][11][1][4]
Return path: [0][9][18][D][1]
Reserved: [0][0][0][0][0][0][0]
00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
00 00 00 00 00 00 00 00 00 00 00 00 01 03 03 02
14 52 00 11 40 40 00 08 08 04 F2 40 00 00 00 00
00 00 00 00 00 88 00 00 00 00 00 00 00 00 00 00
Mar 20 14:18:31 672817 [43C05960] -> osm_pi_rcv_process_set: Received error
status 0x1c for SetResp() during ACTIVE transition
Mar 20 14:18:31 672823 [4780B960] -> __osm_sm_mad_ctrl_rcv_callback: ERR
3111: Error status = 0x1C00
Mar 20 14:18:31 672833 [43C05960] -> PortInfo dump:
port number.............0x16
node_guid...............0x0005ad00000281b3
port_guid...............0x0005ad00000281b3
m_key...................0x0000000000000000
subnet_prefix...........0x0000000000000000
base_lid................0x0
master_sm_base_lid......0x0
capability_mask.........0x0
diag_code...............0x0
m_key_lease_period......0x0
local_port_num..........0x1
link_width_enabled......0x3
link_width_supported....0x3
link_width_active.......0x2
link_speed_supported....0x1
port_state..............ACTIVE
state_info2.............0x52
m_key_protect_bits......0x0
lmc.....................0x0
link_speed..............0x11
mtu_smsl................0x40
vl_cap_init_type........0x40
vl_high_limit...........0x0
vl_arb_high_cap.........0x8
vl_arb_low_cap..........0x8
init_rep_mtu_cap........0x4
vl_stall_life...........0xF2
vl_enforce..............0x40
m_key_violations........0x0
p_key_violations........0x0
q_key_violations........0x0
guid_cap................0x0
client_reregister.......0x0
subnet_timeout..........0x0
resp_time_value.........0x0
error_threshold.........0x88
Mar 20 14:18:31 672852 [43C05960] -> Capabilities Mask:
Mar 20 14:18:31 672861 [4780B960] -> SMP dump:
base_ver................0x1
mgmt_class..............0x81
class_ver...............0x1
method..................0x81 (SubnGetResp)
D bit...................0x1
status..................0x1C00
hop_ptr.................0x0
hop_count...............0x4
trans_id................0x77d12
attr_id.................0x15 (PortInfo)
resv....................0x0
attr_mod................0x17
m_key...................0x0000000000000000
dr_slid.................0xFFFF
dr_dlid.................0xFFFF
Initial path: [0][1][11][1][4]
Return path: [0][9][18][D][1]
Reserved: [0][0][0][0][0][0][0]
00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
00 00 00 00 00 00 00 00 00 00 00 00 01 03 03 02
14 52 00 11 40 40 00 08 08 04 F2 40 00 00 00 00
00 00 00 00 00 88 00 00 00 00 00 00 00 00 00 00
Mar 20 14:18:31 672918 [45007960] -> osm_pi_rcv_process_set: Received error
status 0x1c for SetResp() during ACTIVE transition
Mar 20 14:18:31 672922 [45007960] -> PortInfo dump:
port number.............0x17
node_guid...............0x0005ad00000281b3
port_guid...............0x0005ad00000281b3
m_key...................0x0000000000000000
subnet_prefix...........0x0000000000000000
base_lid................0x0
master_sm_base_lid......0x0
capability_mask.........0x0
diag_code...............0x0
m_key_lease_period......0x0
local_port_num..........0x1
link_width_enabled......0x3
link_width_supported....0x3
link_width_active.......0x2
link_speed_supported....0x1
port_state..............ACTIVE
state_info2.............0x52
m_key_protect_bits......0x0
lmc.....................0x0
link_speed..............0x11
mtu_smsl................0x40
vl_cap_init_type........0x40
vl_high_limit...........0x0
vl_arb_high_cap.........0x8
vl_arb_low_cap..........0x8
init_rep_mtu_cap........0x4
vl_stall_life...........0xF2
vl_enforce..............0x40
m_key_violations........0x0
p_key_violations........0x0
q_key_violations........0x0
guid_cap................0x0
client_reregister.......0x0
subnet_timeout..........0x0
resp_time_value.........0x0
error_threshold.........0x88
Mar 20 14:18:31 672936 [45007960] -> Capabilities Mask:
Mar 20 14:18:31 678085 [45A08960] -> SUBNET UP
Mar 20 14:18:31 723715 [41E02960] -> __osm_trap_rcv_process_request:
Received Generic Notice type:0x01 num:128 Producer:2 from LID:0x0152
TID:0x0000000000000018
Mar 20 14:18:31 723815 [41E02960] -> __osm_trap_rcv_process_request: ERR
3804: Received trap 25 times consecutively
Mar 20 14:18:32 061932 [41401960] -> osm_ucast_mgr_process: Min Hop Tables
configured on all switches
Mar 20 14:18:32 113545 [4780B960] -> __osm_sm_mad_ctrl_rcv_callback: ERR
3111: Error status = 0x1C00
Mar 20 14:18:32 113610 [4780B960] -> SMP dump:
base_ver................0x1
mgmt_class..............0x81
class_ver...............0x1
method..................0x81 (SubnGetResp)
D bit...................0x1
status..................0x1C00
hop_ptr.................0x0
hop_count...............0x4
trans_id................0x78a4d
attr_id.................0x15 (PortInfo)
resv....................0x0
attr_mod................0x13
m_key...................0x0000000000000000
dr_slid.................0xFFFF
dr_dlid.................0xFFFF
Initial path: [0][1][15][4][4]
Return path: [0][9][18][D][4]
Reserved: [0][0][0][0][0][0][0]
00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
00 00 00 00 00 00 00 00 00 00 00 00 04 03 03 02
14 52 00 11 40 40 00 08 08 04 F2 40 00 00 00 00
00 00 00 00 00 88 00 00 00 00 00 00 00 00 00 00
Mar 20 14:18:32 113712 [42803960] -> osm_pi_rcv_process_set: Received error
status 0x1c for SetResp() during ACTIVE transition
Mar 20 14:18:32 113725 [4780B960] -> __osm_sm_mad_ctrl_rcv_callback: ERR
3111: Error status = 0x1C00
Mar 20 14:18:32 113730 [42803960] -> PortInfo dump:
port number.............0x13
node_guid...............0x0005ad00000281a7
port_guid...............0x0005ad00000281a7
m_key...................0x0000000000000000
subnet_prefix...........0x0000000000000000
base_lid................0x0
master_sm_base_lid......0x0
capability_mask.........0x0
diag_code...............0x0
m_key_lease_period......0x0
local_port_num..........0x4
link_width_enabled......0x3
link_width_supported....0x3
link_width_active.......0x2
link_speed_supported....0x1
port_state..............ACTIVE
state_info2.............0x52
m_key_protect_bits......0x0
lmc.....................0x0
link_speed..............0x11
mtu_smsl................0x40
vl_cap_init_type........0x40
vl_high_limit...........0x0
vl_arb_high_cap.........0x8
vl_arb_low_cap..........0x8
init_rep_mtu_cap........0x4
vl_stall_life...........0xF2
vl_enforce..............0x40
m_key_violations........0x0
p_key_violations........0x0
q_key_violations........0x0
guid_cap................0x0
client_reregister.......0x0
subnet_timeout..........0x0
resp_time_value.........0x0
error_threshold.........0x88
Mar 20 14:18:32 113745 [42803960] -> Capabilities Mask:
Mar 20 14:18:32 113751 [4780B960] -> SMP dump:
base_ver................0x1
mgmt_class..............0x81
class_ver...............0x1
method..................0x81 (SubnGetResp)
D bit...................0x1
status..................0x1C00
hop_ptr.................0x0
hop_count...............0x4
trans_id................0x78a4e
attr_id.................0x15 (PortInfo)
resv....................0x0
attr_mod................0x16
m_key...................0x0000000000000000
dr_slid.................0xFFFF
dr_dlid.................0xFFFF
Initial path: [0][1][15][4][4]
Return path: [0][9][18][D][4]
Reserved: [0][0][0][0][0][0][0]
00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
00 00 00 00 00 00 00 00 00 00 00 00 04 03 03 02
14 52 00 11 40 40 00 08 08 04 F2 40 00 00 00 00
00 00 00 00 00 88 00 00 00 00 00 00 00 00 00 00
Mar 20 14:18:32 113803 [44606960] -> osm_pi_rcv_process_set: Received error
status 0x1c for SetResp() during ACTIVE transition
Mar 20 14:18:32 113807 [44606960] -> PortInfo dump:
port number.............0x16
node_guid...............0x0005ad00000281a7
port_guid...............0x0005ad00000281a7
m_key...................0x0000000000000000
subnet_prefix...........0x0000000000000000
base_lid................0x0
master_sm_base_lid......0x0
capability_mask.........0x0
diag_code...............0x0
m_key_lease_period......0x0
local_port_num..........0x4
link_width_enabled......0x3
link_width_supported....0x3
link_width_active.......0x2
link_speed_supported....0x1
port_state..............ACTIVE
state_info2.............0x52
m_key_protect_bits......0x0
lmc.....................0x0
link_speed..............0x11
mtu_smsl................0x40
vl_cap_init_type........0x40
vl_high_limit...........0x0
vl_arb_high_cap.........0x8
vl_arb_low_cap..........0x8
init_rep_mtu_cap........0x4
vl_stall_life...........0xF2
vl_enforce..............0x40
m_key_violations........0x0
p_key_violations........0x0
q_key_violations........0x0
guid_cap................0x0
client_reregister.......0x0
subnet_timeout..........0x0
resp_time_value.........0x0
error_threshold.........0x88
Mar 20 14:18:32 113820 [44606960] -> Capabilities Mask:
Mar 20 14:18:32 113845 [4780B960] -> __osm_sm_mad_ctrl_rcv_callback: ERR
3111: Error status = 0x1C00
Mar 20 14:18:32 113907 [4780B960] -> SMP dump:
base_ver................0x1
mgmt_class..............0x81
class_ver...............0x1
method..................0x81 (SubnGetResp)
D bit...................0x1
status..................0x1C00
hop_ptr.................0x0
hop_count...............0x4
trans_id................0x78a4f
attr_id.................0x15 (PortInfo)
resv....................0x0
attr_mod................0x17
m_key...................0x0000000000000000
dr_slid.................0xFFFF
dr_dlid.................0xFFFF
Initial path: [0][1][15][4][4]
Return path: [0][9][18][D][4]
Reserved: [0][0][0][0][0][0][0]
00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
00 00 00 00 00 00 00 00 00 00 00 00 04 03 03 02
14 52 00 11 40 40 00 08 08 04 F2 40 00 00 00 00
00 00 00 00 00 88 00 00 00 00 00 00 00 00 00 00
Mar 20 14:18:32 113958 [41E02960] -> osm_pi_rcv_process_set: Received error
status 0x1c for SetResp() during ACTIVE transition
Mar 20 14:18:32 113963 [4780B960] -> __osm_sm_mad_ctrl_rcv_callback: ERR
3111: Error status = 0x1C00
Mar 20 14:18:32 113969 [41E02960] -> PortInfo dump:
port number.............0x17
node_guid...............0x0005ad00000281a7
port_guid...............0x0005ad00000281a7
m_key...................0x0000000000000000
subnet_prefix...........0x0000000000000000
base_lid................0x0
master_sm_base_lid......0x0
capability_mask.........0x0
diag_code...............0x0
m_key_lease_period......0x0
local_port_num..........0x4
link_width_enabled......0x3
link_width_supported....0x3
link_width_active.......0x2
link_speed_supported....0x1
port_state..............ACTIVE
state_info2.............0x52
m_key_protect_bits......0x0
lmc.....................0x0
link_speed..............0x11
mtu_smsl................0x40
vl_cap_init_type........0x40
vl_high_limit...........0x0
vl_arb_high_cap.........0x8
vl_arb_low_cap..........0x8
init_rep_mtu_cap........0x4
vl_stall_life...........0xF2
vl_enforce..............0x40
m_key_violations........0x0
p_key_violations........0x0
q_key_violations........0x0
guid_cap................0x0
client_reregister.......0x0
subnet_timeout..........0x0
resp_time_value.........0x0
error_threshold.........0x88
Mar 20 14:18:32 113986 [41E02960] -> Capabilities Mask:
Mar 20 14:18:32 113992 [4780B960] -> SMP dump:
base_ver................0x1
mgmt_class..............0x81
class_ver...............0x1
method..................0x81 (SubnGetResp)
D bit...................0x1
status..................0x1C00
hop_ptr.................0x0
hop_count...............0x4
trans_id................0x78a50
attr_id.................0x15 (PortInfo)
resv....................0x0
attr_mod................0x18
m_key...................0x0000000000000000
dr_slid.................0xFFFF
dr_dlid.................0xFFFF
Initial path: [0][1][15][4][4]
Return path: [0][9][18][D][4]
Reserved: [0][0][0][0][0][0][0]
00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
00 00 00 00 00 00 00 00 00 00 00 00 04 03 03 02
14 52 00 11 40 40 00 08 08 04 F2 40 00 00 00 00
00 00 00 00 00 88 00 00 00 00 00 00 00 00 00 00
Mar 20 14:18:32 114052 [45A08960] -> osm_pi_rcv_process_set: Received error
status 0x1c for SetResp() during ACTIVE transition
Mar 20 14:18:32 114066 [45A08960] -> PortInfo dump:
port number.............0x18
node_guid...............0x0005ad00000281a7
port_guid...............0x0005ad00000281a7
m_key...................0x0000000000000000
subnet_prefix...........0x0000000000000000
base_lid................0x0
master_sm_base_lid......0x0
capability_mask.........0x0
diag_code...............0x0
m_key_lease_period......0x0
local_port_num..........0x4
link_width_enabled......0x3
link_width_supported....0x3
link_width_active.......0x2
link_speed_supported....0x1
port_state..............ACTIVE
state_info2.............0x52
m_key_protect_bits......0x0
lmc.....................0x0
link_speed..............0x11
mtu_smsl................0x40
vl_cap_init_type........0x40
vl_high_limit...........0x0
vl_arb_high_cap.........0x8
vl_arb_low_cap..........0x8
init_rep_mtu_cap........0x4
vl_stall_life...........0xF2
vl_enforce..............0x40
m_key_violations........0x0
p_key_violations........0x0
q_key_violations........0x0
guid_cap................0x0
client_reregister.......0x0
subnet_timeout..........0x0
resp_time_value.........0x0
error_threshold.........0x88
Mar 20 14:18:32 114089 [45A08960] -> Capabilities Mask:
Mar 20 14:18:32 114052 [4780B960] -> __osm_sm_mad_ctrl_rcv_callback: ERR
3111: Error status = 0x1C00
Mar 20 14:18:32 114171 [4780B960] -> SMP dump:
base_ver................0x1
mgmt_class..............0x81
class_ver...............0x1
method..................0x81 (SubnGetResp)
D bit...................0x1
status..................0x1C00
hop_ptr.................0x0
hop_count...............0x4
trans_id................0x78a51
attr_id.................0x15 (PortInfo)
resv....................0x0
attr_mod................0x18
m_key...................0x0000000000000000
dr_slid.................0xFFFF
dr_dlid.................0xFFFF
Initial path: [0][1][13][1][6]
Return path: [0][9][13][D][3]
Reserved: [0][0][0][0][0][0][0]
00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
00 00 00 00 00 00 00 00 00 00 00 00 03 03 03 02
14 52 00 11 40 40 00 08 08 04 F2 40 00 00 00 00
00 00 00 00 00 88 00 00 00 00 00 00 00 00 00 00
Mar 20 14:18:32 114224 [42803960] -> osm_pi_rcv_process_set: Received error
status 0x1c for SetResp() during ACTIVE transition
Mar 20 14:18:32 114228 [42803960] -> PortInfo dump:
port number.............0x18
node_guid...............0x0005ad00000281b3
port_guid...............0x0005ad00000281b3
m_key...................0x0000000000000000
subnet_prefix...........0x0000000000000000
base_lid................0x0
master_sm_base_lid......0x0
capability_mask.........0x0
diag_code...............0x0
m_key_lease_period......0x0
local_port_num..........0x3
link_width_enabled......0x3
link_width_supported....0x3
link_width_active.......0x2
link_speed_supported....0x1
port_state..............ACTIVE
state_info2.............0x52
m_key_protect_bits......0x0
lmc.....................0x0
link_speed..............0x11
mtu_smsl................0x40
vl_cap_init_type........0x40
vl_high_limit...........0x0
vl_arb_high_cap.........0x8
vl_arb_low_cap..........0x8
init_rep_mtu_cap........0x4
vl_stall_life...........0xF2
vl_enforce..............0x40
m_key_violations........0x0
p_key_violations........0x0
q_key_violations........0x0
guid_cap................0x0
client_reregister.......0x0
subnet_timeout..........0x0
resp_time_value.........0x0
error_threshold.........0x88
Mar 20 14:18:32 114242 [42803960] -> Capabilities Mask:
Mar 20 14:18:32 119326 [42803960] -> SUBNET UP
Mar 20 14:23:02 506774 [41E02960] -> __osm_trap_rcv_process_request:
Received Generic Notice type:0x01 num:128 Producer:2 from LID:0x0152
TID:0x0000000000000019
Mar 20 14:23:02 507064 [41E02960] -> osm_report_notice: Reporting Generic
Notice type:1 num:128 from LID:0x0152
GID:0xfe80000000000000,0x0005ad0000027c84
Mar 20 14:23:02 861642 [43204960] -> osm_report_notice: Reporting Generic
Notice type:3 num:64 from LID:0x0092
GID:0xfe80000000000000,0x0005ad0000024bbb
Mar 20 14:23:02 861653 [43204960] -> Discovered new port with
GUID:0x0005ad0000024b27 LID range [0xAF,0xAF] of node:Topspin IB-DC
Mar 20 14:23:02 893030 [43204960] -> osm_ucast_mgr_process: Min Hop Tables
configured on all switches
Mar 20 14:23:02 943693 [4780B960] -> __osm_sm_mad_ctrl_rcv_callback: ERR
3111: Error status = 0x1C00
Mar 20 14:23:02 943765 [4780B960] -> SMP dump:
base_ver................0x1
mgmt_class..............0x81
class_ver...............0x1
method..................0x81 (SubnGetResp)
D bit...................0x1
status..................0x1C00
hop_ptr.................0x0
hop_count...............0x5
trans_id................0x79aff
attr_id.................0x15 (PortInfo)
resv....................0x0
attr_mod................0x1
m_key...................0x0000000000000000
dr_slid.................0xFFFF
dr_dlid.................0xFFFF
Initial path: [0][1][11][1][5][18]
Return path: [0][9][18][D][2][13]
Reserved: [0][0][0][0][0][0][0]
00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
00 00 00 00 00 00 00 00 00 00 00 00 13 03 03 02
14 52 00 11 40 40 00 08 08 04 2C 4C 00 00 00 00
00 00 00 00 00 88 00 00 00 00 00 00 00 00 00 00
Mar 20 14:23:02 943854 [43C05960] -> osm_pi_rcv_process_set: Received error
status 0x1c for SetResp() during ACTIVE transition
Mar 20 14:23:02 943870 [43C05960] -> PortInfo dump:
port number.............0x1
node_guid...............0x0005ad0000027c84
port_guid...............0x0005ad0000027c84
m_key...................0x0000000000000000
subnet_prefix...........0x0000000000000000
base_lid................0x0
master_sm_base_lid......0x0
capability_mask.........0x0
diag_code...............0x0
m_key_lease_period......0x0
local_port_num..........0x13
link_width_enabled......0x3
link_width_supported....0x3
link_width_active.......0x2
link_speed_supported....0x1
port_state..............ACTIVE
state_info2.............0x52
m_key_protect_bits......0x0
lmc.....................0x0
link_speed..............0x11
mtu_smsl................0x40
vl_cap_init_type........0x40
vl_high_limit...........0x0
vl_arb_high_cap.........0x8
vl_arb_low_cap..........0x8
init_rep_mtu_cap........0x4
vl_stall_life...........0x2C
vl_enforce..............0x4C
m_key_violations........0x0
p_key_violations........0x0
q_key_violations........0x0
guid_cap................0x0
client_reregister.......0x0
subnet_timeout..........0x0
resp_time_value.........0x0
error_threshold.........0x88
Mar 20 14:23:02 943886 [43C05960] -> Capabilities Mask:
Mar 20 14:23:02 948898 [43C05960] -> SUBNET UP
Mar 20 14:23:03 237496 [42803960] -> __osm_trap_rcv_process_request:
Received Generic Notice type:0x04 num:144 Producer:1 from LID:0x00AF
TID:0x0000000000000000
Mar 20 14:23:03 237710 [42803960] -> osm_report_notice: Reporting Generic
Notice type:4 num:144 from LID:0x00AF
GID:0xfe80000000000000,0x0005ad0000024b27
Mar 20 14:23:03 605548 [45007960] -> osm_ucast_mgr_process: Min Hop Tables
configured on all switches
Mar 20 14:23:03 662757 [41401960] -> SUBNET UP
Mar 20 14:24:54 675782 [44606960] -> __osm_trap_rcv_process_request:
Received Generic Notice type:0x01 num:128 Producer:2 from LID:0x001B
TID:0x0000000000000079
Mar 20 14:24:54 676077 [44606960] -> osm_report_notice: Reporting Generic
Notice type:1 num:128 from LID:0x001B
GID:0xfe80000000000000,0x0005ad00000281a7
Mar 20 14:24:54 677026 [43204960] -> __osm_trap_rcv_process_request:
Received Generic Notice type:0x01 num:128 Producer:2 from LID:0x0148
TID:0x0000000000000064
Mar 20 14:24:54 677118 [43204960] -> osm_report_notice: Reporting Generic
Notice type:1 num:128 from LID:0x0148
GID:0xfe80000000000000,0x0005ad00000281b3
Mar 20 14:24:55 047478 [43204960] -> osm_report_notice: Reporting Generic
Notice type:3 num:65 from LID:0x0092
GID:0xfe80000000000000,0x0005ad0000024bbb
Mar 20 14:24:55 047501 [43204960] -> Removed port with
GUID:0x0005ad0000024e0b LID range [0xB3,0xB3] of node:saguaro-23-4 HCA-1
Mar 20 14:24:55 047520 [43204960] -> osm_report_notice: Reporting Generic
Notice type:3 num:65 from LID:0x0092
GID:0xfe80000000000000,0x0005ad0000024bbb
Mar 20 14:24:55 047525 [43204960] -> Removed port with
GUID:0x0005ad000002510b LID range [0xB5,0xB5] of node:saguaro-23-6 HCA-1
Mar 20 14:24:55 047541 [43204960] -> osm_report_notice: Reporting Generic
Notice type:3 num:65 from LID:0x0092
GID:0xfe80000000000000,0x0005ad0000024bbb
Mar 20 14:24:55 047546 [43204960] -> Removed port with
GUID:0x0005ad000002511b LID range [0xA6,0xA6] of node:saguaro-22-1 HCA-1
Mar 20 14:24:55 047563 [43204960] -> osm_report_notice: Reporting Generic
Notice type:3 num:65 from LID:0x0092
GID:0xfe80000000000000,0x0005ad0000024bbb
Mar 20 14:24:55 047569 [43204960] -> Removed port with
GUID:0x0005ad0000024b27 LID range [0xAF,0xAF] of node:Topspin IB-DC
Mar 20 14:24:55 047586 [43204960] -> osm_report_notice: Reporting Generic
Notice type:3 num:65 from LID:0x0092
GID:0xfe80000000000000,0x0005ad0000024bbb
Mar 20 14:24:55 047591 [43204960] -> Removed port with
GUID:0x0005ad0000025043 LID range [0xB4,0xB4] of node:saguaro-23-5 HCA-1
Mar 20 14:24:55 047607 [43204960] -> osm_report_notice: Reporting Generic
Notice type:3 num:65 from LID:0x0092
GID:0xfe80000000000000,0x0005ad0000024bbb
Mar 20 14:24:55 047612 [43204960] -> Removed port with
GUID:0x0005ad0000024d47 LID range [0xB6,0xB6] of node:saguaro-23-7 HCA-1
Mar 20 14:24:55 047630 [43204960] -> osm_report_notice: Reporting Generic
Notice type:3 num:65 from LID:0x0092
GID:0xfe80000000000000,0x0005ad0000024bbb
Mar 20 14:24:55 047635 [43204960] -> Removed port with
GUID:0x0005ad0000024d6b LID range [0xB8,0xB8] of node:saguaro-23-9 HCA-1
Mar 20 14:24:55 047652 [43204960] -> osm_report_notice: Reporting Generic
Notice type:3 num:65 from LID:0x0092
GID:0xfe80000000000000,0x0005ad0000024bbb
Mar 20 14:24:55 047657 [43204960] -> Removed port with
GUID:0x0005ad0000024977 LID range [0xA9,0xA9] of node:saguaro-22-4 HCA-1
Mar 20 14:24:55 047798 [43204960] -> osm_report_notice: Reporting Generic
Notice type:3 num:65 from LID:0x0092
GID:0xfe80000000000000,0x0005ad0000024bbb
Mar 20 14:24:55 047803 [43204960] -> Removed port with
GUID:0x0005ad0000027c84 LID range [0x152,0x152] of node:Topspin Switch TS120
Mar 20 14:24:55 047836 [43204960] -> osm_report_notice: Reporting Generic
Notice type:3 num:65 from LID:0x0092
GID:0xfe80000000000000,0x0005ad0000024bbb
Mar 20 14:24:55 047842 [43204960] -> Removed port with
GUID:0x0005ad0000024d8b LID range [0xB7,0xB7] of node:saguaro-23-8 HCA-1
Mar 20 14:24:55 047857 [43204960] -> osm_report_notice: Reporting Generic
Notice type:3 num:65 from LID:0x0092
GID:0xfe80000000000000,0x0005ad0000024bbb
Mar 20 14:24:55 047862 [43204960] -> Removed port with
GUID:0x0005ad000002498f LID range [0xA8,0xA8] of node:saguaro-22-3 HCA-1
Mar 20 14:24:55 047877 [43204960] -> osm_report_notice: Reporting Generic
Notice type:3 num:65 from LID:0x0092
GID:0xfe80000000000000,0x0005ad0000024bbb
Mar 20 14:24:55 047882 [43204960] -> Removed port with
GUID:0x0005ad0000024c9b LID range [0xA7,0xA7] of node:saguaro-22-2 HCA-1
Mar 20 14:24:55 047896 [43204960] -> osm_report_notice: Reporting Generic
Notice type:3 num:65 from LID:0x0092
GID:0xfe80000000000000,0x0005ad0000024bbb
Mar 20 14:24:55 047902 [43204960] -> Removed port with
GUID:0x0005ad0000024da7 LID range [0xB0,0xB0] of node:saguaro-23-1 HCA-1
Mar 20 14:24:55 047918 [43204960] -> osm_report_notice: Reporting Generic
Notice type:3 num:65 from LID:0x0092
GID:0xfe80000000000000,0x0005ad0000024bbb
Mar 20 14:24:55 047923 [43204960] -> Removed port with
GUID:0x0005ad0000024cbb LID range [0xB2,0xB2] of node:saguaro-23-3 HCA-1
Mar 20 14:24:55 047939 [43204960] -> osm_report_notice: Reporting Generic
Notice type:3 num:65 from LID:0x0092
GID:0xfe80000000000000,0x0005ad0000024bbb
Mar 20 14:24:55 047988 [43204960] -> Removed port with
GUID:0x0005ad00000249d3 LID range [0xB1,0xB1] of node:saguaro-23-2 HCA-1
Mar 20 14:24:55 048005 [43204960] -> osm_report_notice: Reporting Generic
Notice type:3 num:65 from LID:0x0092
GID:0xfe80000000000000,0x0005ad0000024bbb
Mar 20 14:24:55 048010 [43204960] -> Removed port with
GUID:0x0005ad0000024feb LID range [0x153,0x153] of node:saguaro-22-5 HCA-1
Mar 20 14:24:55 048025 [43204960] -> osm_report_notice: Reporting Generic
Notice type:3 num:65 from LID:0x0092
GID:0xfe80000000000000,0x0005ad0000024bbb
Mar 20 14:24:55 048030 [43204960] -> Removed port with
GUID:0x0005ad0000024afb LID range [0xA5,0xA5] of node:saguaro-22-0 HCA-1
Mar 20 14:24:55 081006 [43204960] -> osm_ucast_mgr_process: Min Hop Tables
configured on all switches
Mar 20 14:24:55 130875 [45A08960] -> SUBNET UP
Mar 20 14:24:55 484995 [42803960] -> osm_ucast_mgr_process: Min Hop Tables
configured on all switches
Mar 20 14:24:55 535902 [42803960] -> SUBNET UP
Mar 20 14:25:48 653788 [43204960] -> __osm_trap_rcv_process_request:
Received Generic Notice type:0x01 num:128 Producer:2 from LID:0x0148
TID:0x0000000000000065
Mar 20 14:25:48 654009 [43204960] -> osm_report_notice: Reporting Generic
Notice type:1 num:128 from LID:0x0148
GID:0xfe80000000000000,0x0005ad00000281b3
Mar 20 14:25:48 659749 [45A08960] -> __osm_trap_rcv_process_request:
Received Generic Notice type:0x01 num:128 Producer:2 from LID:0x001B
TID:0x000000000000007a
Mar 20 14:25:48 659790 [42803960] -> __osm_trap_rcv_process_request:
Received Generic Notice type:0x01 num:128 Producer:2 from LID:0x0148
TID:0x0000000000000066
Mar 20 14:25:48 659814 [45A08960] -> osm_report_notice: Reporting Generic
Notice type:1 num:128 from LID:0x001B
GID:0xfe80000000000000,0x0005ad00000281a7
Mar 20 14:25:48 659963 [42803960] -> osm_report_notice: Reporting Generic
Notice type:1 num:128 from LID:0x0148
GID:0xfe80000000000000,0x0005ad00000281b3
Mar 20 14:25:48 665972 [41401960] -> __osm_trap_rcv_process_request:
Received Generic Notice type:0x01 num:128 Producer:2 from LID:0x001B
TID:0x000000000000007b
Mar 20 14:25:48 666050 [41401960] -> osm_report_notice: Reporting Generic
Notice type:1 num:128 from LID:0x001B
GID:0xfe80000000000000,0x0005ad00000281a7
Mar 20 14:25:49 025384 [41E02960] -> osm_report_notice: Reporting Generic
Notice type:3 num:64 from LID:0x0092
GID:0xfe80000000000000,0x0005ad0000024bbb
Mar 20 14:25:49 025396 [41E02960] -> Discovered new port with
GUID:0x0005ad0000027c84 LID range [0x152,0x152] of node:Topspin Switch TS120
Mar 20 14:25:49 025401 [41E02960] -> osm_report_notice: Reporting Generic
Notice type:3 num:64 from LID:0x0092
GID:0xfe80000000000000,0x0005ad0000024bbb
Mar 20 14:25:49 025406 [41E02960] -> Discovered new port with
GUID:0x0005ad0000024b27 LID range [0xAF,0xAF] of node:saguaro-23-0 HCA-1
Mar 20 14:25:49 025410 [41E02960] -> osm_report_notice: Reporting Generic
Notice type:3 num:64 from LID:0x0092
GID:0xfe80000000000000,0x0005ad0000024bbb
Mar 20 14:25:49 025416 [41E02960] -> Discovered new port with
GUID:0x0005ad0000024da7 LID range [0xB0,0xB0] of node:saguaro-23-1 HCA-1
Mar 20 14:25:49 025420 [41E02960] -> osm_report_notice: Reporting Generic
Notice type:3 num:64 from LID:0x0092
GID:0xfe80000000000000,0x0005ad0000024bbb
Mar 20 14:25:49 025425 [41E02960] -> Discovered new port with
GUID:0x0005ad00000249d3 LID range [0xB1,0xB1] of node:saguaro-23-2 HCA-1
Mar 20 14:25:49 025428 [41E02960] -> osm_report_notice: Reporting Generic
Notice type:3 num:64 from LID:0x0092
GID:0xfe80000000000000,0x0005ad0000024bbb
Mar 20 14:25:49 025433 [41E02960] -> Discovered new port with
GUID:0x0005ad0000024cbb LID range [0xB2,0xB2] of node:saguaro-23-3 HCA-1
Mar 20 14:25:49 025437 [41E02960] -> osm_report_notice: Reporting Generic
Notice type:3 num:64 from LID:0x0092
GID:0xfe80000000000000,0x0005ad0000024bbb
Mar 20 14:25:49 025442 [41E02960] -> Discovered new port with
GUID:0x0005ad0000024e0b LID range [0xB3,0xB3] of node:saguaro-23-4 HCA-1
Mar 20 14:25:49 025446 [41E02960] -> osm_report_notice: Reporting Generic
Notice type:3 num:64 from LID:0x0092
GID:0xfe80000000000000,0x0005ad0000024bbb
Mar 20 14:25:49 025451 [41E02960] -> Discovered new port with
GUID:0x0005ad0000025043 LID range [0xB4,0xB4] of node:saguaro-23-5 HCA-1
Mar 20 14:25:49 025461 [41E02960] -> osm_report_notice: Reporting Generic
Notice type:3 num:64 from LID:0x0092
GID:0xfe80000000000000,0x0005ad0000024bbb
Mar 20 14:25:49 025466 [41E02960] -> Discovered new port with
GUID:0x0005ad000002510b LID range [0xB5,0xB5] of node:saguaro-23-6 HCA-1
Mar 20 14:25:49 025470 [41E02960] -> osm_report_notice: Reporting Generic
Notice type:3 num:64 from LID:0x0092
GID:0xfe80000000000000,0x0005ad0000024bbb
Mar 20 14:25:49 025475 [41E02960] -> Discovered new port with
GUID:0x0005ad0000024d47 LID range [0xB6,0xB6] of node:saguaro-23-7 HCA-1
Mar 20 14:25:49 025483 [41E02960] -> osm_report_notice: Reporting Generic
Notice type:3 num:64 from LID:0x0092
GID:0xfe80000000000000,0x0005ad0000024bbb
Mar 20 14:25:49 025488 [41E02960] -> Discovered new port with
GUID:0x0005ad0000024d6b LID range [0xB8,0xB8] of node:saguaro-23-9 HCA-1
Mar 20 14:25:49 025491 [41E02960] -> osm_report_notice: Reporting Generic
Notice type:3 num:64 from LID:0x0092
GID:0xfe80000000000000,0x0005ad0000024bbb
Mar 20 14:25:49 025496 [41E02960] -> Discovered new port with
GUID:0x0005ad0000024afb LID range [0xA5,0xA5] of node:saguaro-22-0 HCA-1
Mar 20 14:25:49 025500 [41E02960] -> osm_report_notice: Reporting Generic
Notice type:3 num:64 from LID:0x0092
GID:0xfe80000000000000,0x0005ad0000024bbb
Mar 20 14:25:49 025505 [41E02960] -> Discovered new port with
GUID:0x0005ad000002511b LID range [0xA6,0xA6] of node:saguaro-22-1 HCA-1
Mar 20 14:25:49 025508 [41E02960] -> osm_report_notice: Reporting Generic
Notice type:3 num:64 from LID:0x0092
GID:0xfe80000000000000,0x0005ad0000024bbb
Mar 20 14:25:49 025513 [41E02960] -> Discovered new port with
GUID:0x0005ad0000024c9b LID range [0xA7,0xA7] of node:saguaro-22-2 HCA-1
Mar 20 14:25:49 025517 [41E02960] -> osm_report_notice: Reporting Generic
Notice type:3 num:64 from LID:0x0092
GID:0xfe80000000000000,0x0005ad0000024bbb
Mar 20 14:25:49 025522 [41E02960] -> Discovered new port with
GUID:0x0005ad000002498f LID range [0xA8,0xA8] of node:saguaro-22-3 HCA-1
Mar 20 14:25:49 025556 [41E02960] -> osm_report_notice: Reporting Generic
Notice type:3 num:64 from LID:0x0092
GID:0xfe80000000000000,0x0005ad0000024bbb
Mar 20 14:25:49 025562 [41E02960] -> Discovered new port with
GUID:0x0005ad0000024977 LID range [0xA9,0xA9] of node:saguaro-22-4 HCA-1
Mar 20 14:25:49 025565 [41E02960] -> osm_report_notice: Reporting Generic
Notice type:3 num:64 from LID:0x0092
GID:0xfe80000000000000,0x0005ad0000024bbb
Mar 20 14:25:49 025570 [41E02960] -> Discovered new port with
GUID:0x0005ad0000024feb LID range [0x153,0x153] of node:saguaro-22-5 HCA-1
Mar 20 14:25:49 025574 [41E02960] -> osm_report_notice: Reporting Generic
Notice type:3 num:64 from LID:0x0092
GID:0xfe80000000000000,0x0005ad0000024bbb
Mar 20 14:25:49 025579 [41E02960] -> Discovered new port with
GUID:0x0005ad0000024d8b LID range [0xB7,0xB7] of node:saguaro-23-8 HCA-1
Mar 20 14:25:49 056324 [41E02960] -> osm_ucast_mgr_process: Min Hop Tables
configured on all switches
Mar 20 14:25:49 126247 [4780B960] -> __osm_sm_mad_ctrl_rcv_callback: ERR
3111: Error status = 0x1C00
Mar 20 14:25:49 126356 [4780B960] -> SMP dump:
base_ver................0x1
mgmt_class..............0x81
class_ver...............0x1
method..................0x81 (SubnGetResp)
D bit...................0x1
status..................0x1C00
hop_ptr.................0x0
hop_count...............0x4
trans_id................0x7d165
attr_id.................0x15 (PortInfo)
resv....................0x0
attr_mod................0x13
m_key...................0x0000000000000000
dr_slid.................0xFFFF
dr_dlid.................0xFFFF
Initial path: [0][1][15][1][6]
Return path: [0][9][18][D][3]
Reserved: [0][0][0][0][0][0][0]
00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
00 00 00 00 00 00 00 00 00 00 00 00 03 03 03 02
14 52 00 11 40 40 00 08 08 04 F2 40 00 00 00 00
00 00 00 00 00 88 00 00 00 00 00 00 00 00 00 00
Mar 20 14:25:49 126409 [4780B960] -> __osm_sm_mad_ctrl_rcv_callback: ERR
3111: Error status = 0x1C00
Mar 20 14:25:49 126442 [4780B960] -> SMP dump:
base_ver................0x1
mgmt_class..............0x81
class_ver...............0x1
method..................0x81 (SubnGetResp)
D bit...................0x1
status..................0x1C00
hop_ptr.................0x0
hop_count...............0x4
trans_id................0x7d166
attr_id.................0x15 (PortInfo)
resv....................0x0
attr_mod................0x16
m_key...................0x0000000000000000
dr_slid.................0xFFFF
dr_dlid.................0xFFFF
Initial path: [0][1][15][1][6]
Return path: [0][9][18][D][3]
Reserved: [0][0][0][0][0][0][0]
00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
00 00 00 00 00 00 00 00 00 00 00 00 03 03 03 02
14 52 00 11 40 40 00 08 08 04 F2 40 00 00 00 00
00 00 00 00 00 88 00 00 00 00 00 00 00 00 00 00
Mar 20 14:25:49 126496 [4780B960] -> __osm_sm_mad_ctrl_rcv_callback: ERR
3111: Error status = 0x1C00
Mar 20 14:25:49 126489 [42803960] -> osm_pi_rcv_process_set: Received error
status 0x1c for SetResp() during ACTIVE transition
Mar 20 14:25:49 126535 [4780B960] -> SMP dump:
base_ver................0x1
mgmt_class..............0x81
class_ver...............0x1
method..................0x81 (SubnGetResp)
D bit...................0x1
status..................0x1C00
hop_ptr.................0x0
hop_count...............0x4
trans_id................0x7d167
attr_id.................0x15 (PortInfo)
resv....................0x0
attr_mod................0x17
m_key...................0x0000000000000000
dr_slid.................0xFFFF
dr_dlid.................0xFFFF
Initial path: [0][1][15][1][6]
Return path: [0][9][18][D][3]
Reserved: [0][0][0][0][0][0][0]
00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
00 00 00 00 00 00 00 00 00 00 00 00 03 03 03 02
14 52 00 11 40 40 00 08 08 04 F2 40 00 00 00 00
00 00 00 00 00 88 00 00 00 00 00 00 00 00 00 00
Mar 20 14:25:49 126526 [42803960] -> PortInfo dump:
port number.............0x13
node_guid...............0x0005ad00000281a7
port_guid...............0x0005ad00000281a7
m_key...................0x0000000000000000
subnet_prefix...........0x0000000000000000
base_lid................0x0
master_sm_base_lid......0x0
capability_mask.........0x0
diag_code...............0x0
m_key_lease_period......0x0
local_port_num..........0x3
link_width_enabled......0x3
link_width_supported....0x3
link_width_active.......0x2
link_speed_supported....0x1
port_state..............ACTIVE
state_info2.............0x52
m_key_protect_bits......0x0
lmc.....................0x0
link_speed..............0x11
mtu_smsl................0x40
vl_cap_init_type........0x40
vl_high_limit...........0x0
vl_arb_high_cap.........0x8
vl_arb_low_cap..........0x8
init_rep_mtu_cap........0x4
vl_stall_life...........0xF2
vl_enforce..............0x40
m_key_violations........0x0
p_key_violations........0x0
q_key_violations........0x0
guid_cap................0x0
client_reregister.......0x0
subnet_timeout..........0x0
resp_time_value.........0x0
error_threshold.........0x88
Mar 20 14:25:49 126567 [42803960] -> Capabilities Mask:
Mar 20 14:25:49 126613 [4780B960] -> __osm_sm_mad_ctrl_rcv_callback: ERR
3111: Error status = 0x1C00
Mar 20 14:25:49 126617 [42803960] -> osm_pi_rcv_process_set: Received error
status 0x1c for SetResp() during ACTIVE transition
Mar 20 14:25:49 126658 [4780B960] -> SMP dump:
base_ver................0x1
mgmt_class..............0x81
class_ver...............0x1
method..................0x81 (SubnGetResp)
D bit...................0x1
status..................0x1C00
hop_ptr.................0x0
hop_count...............0x4
trans_id................0x7d168
attr_id.................0x15 (PortInfo)
resv....................0x0
attr_mod................0x18
m_key...................0x0000000000000000
dr_slid.................0xFFFF
dr_dlid.................0xFFFF
Initial path: [0][1][15][1][6]
Return path: [0][9][18][D][3]
Reserved: [0][0][0][0][0][0][0]
00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
00 00 00 00 00 00 00 00 00 00 00 00 03 03 03 02
14 52 00 11 40 40 00 08 08 04 F2 40 00 00 00 00
00 00 00 00 00 88 00 00 00 00 00 00 00 00 00 00
Mar 20 14:25:49 126653 [42803960] -> PortInfo dump:
port number.............0x16
node_guid...............0x0005ad00000281a7
port_guid...............0x0005ad00000281a7
m_key...................0x0000000000000000
subnet_prefix...........0x0000000000000000
base_lid................0x0
master_sm_base_lid......0x0
capability_mask.........0x0
diag_code...............0x0
m_key_lease_period......0x0
local_port_num..........0x3
link_width_enabled......0x3
link_width_supported....0x3
link_width_active.......0x2
link_speed_supported....0x1
port_state..............ACTIVE
state_info2.............0x52
m_key_protect_bits......0x0
lmc.....................0x0
link_speed..............0x11
mtu_smsl................0x40
vl_cap_init_type........0x40
vl_high_limit...........0x0
vl_arb_high_cap.........0x8
vl_arb_low_cap..........0x8
init_rep_mtu_cap........0x4
vl_stall_life...........0xF2
vl_enforce..............0x40
m_key_violations........0x0
p_key_violations........0x0
q_key_violations........0x0
guid_cap................0x0
client_reregister.......0x0
subnet_timeout..........0x0
resp_time_value.........0x0
error_threshold.........0x88
Mar 20 14:25:49 126687 [42803960] -> Capabilities Mask:
Mar 20 14:25:49 126703 [43204960] -> osm_pi_rcv_process_set: Received error
status 0x1c for SetResp() during ACTIVE transition
Mar 20 14:25:49 126709 [43204960] -> PortInfo dump:
port number.............0x18
node_guid...............0x0005ad00000281a7
port_guid...............0x0005ad00000281a7
m_key...................0x0000000000000000
subnet_prefix...........0x0000000000000000
base_lid................0x0
master_sm_base_lid......0x0
capability_mask.........0x0
diag_code...............0x0
m_key_lease_period......0x0
local_port_num..........0x3
link_width_enabled......0x3
link_width_supported....0x3
link_width_active.......0x2
link_speed_supported....0x1
port_state..............ACTIVE
state_info2.............0x52
m_key_protect_bits......0x0
lmc.....................0x0
link_speed..............0x11
mtu_smsl................0x40
vl_cap_init_type........0x40
vl_high_limit...........0x0
vl_arb_high_cap.........0x8
vl_arb_low_cap..........0x8
init_rep_mtu_cap........0x4
vl_stall_life...........0xF2
vl_enforce..............0x40
m_key_violations........0x0
p_key_violations........0x0
q_key_violations........0x0
guid_cap................0x0
client_reregister.......0x0
subnet_timeout..........0x0
resp_time_value.........0x0
error_threshold.........0x88
Mar 20 14:25:49 126744 [43204960] -> Capabilities Mask:
Mar 20 14:25:49 126765 [43C05960] -> osm_pi_rcv_process_set: Received error
status 0x1c for SetResp() during ACTIVE transition
Mar 20 14:25:49 126770 [43C05960] -> PortInfo dump:
port number.............0x17
node_guid...............0x0005ad00000281a7
port_guid...............0x0005ad00000281a7
m_key...................0x0000000000000000
subnet_prefix...........0x0000000000000000
base_lid................0x0
master_sm_base_lid......0x0
capability_mask.........0x0
diag_code...............0x0
m_key_lease_period......0x0
local_port_num..........0x3
link_width_enabled......0x3
link_width_supported....0x3
link_width_active.......0x2
link_speed_supported....0x1
port_state..............ACTIVE
state_info2.............0x52
m_key_protect_bits......0x0
lmc.....................0x0
link_speed..............0x11
mtu_smsl................0x40
vl_cap_init_type........0x40
vl_high_limit...........0x0
vl_arb_high_cap.........0x8
vl_arb_low_cap..........0x8
init_rep_mtu_cap........0x4
vl_stall_life...........0xF2
vl_enforce..............0x40
m_key_violations........0x0
p_key_violations........0x0
q_key_violations........0x0
guid_cap................0x0
client_reregister.......0x0
subnet_timeout..........0x0
resp_time_value.........0x0
error_threshold.........0x88
Mar 20 14:25:49 126874 [43C05960] -> Capabilities Mask:
Mar 20 14:25:49 126975 [4780B960] -> __osm_sm_mad_ctrl_rcv_callback: ERR
3111: Error status = 0x1C00
Mar 20 14:25:49 127015 [4780B960] -> SMP dump:
base_ver................0x1
mgmt_class..............0x81
class_ver...............0x1
method..................0x81 (SubnGetResp)
D bit...................0x1
status..................0x1C00
hop_ptr.................0x0
hop_count...............0x4
trans_id................0x7d169
attr_id.................0x15 (PortInfo)
resv....................0x0
attr_mod................0x16
m_key...................0x0000000000000000
dr_slid.................0xFFFF
dr_dlid.................0xFFFF
Initial path: [0][1][13][1][6]
Return path: [0][9][13][D][3]
Reserved: [0][0][0][0][0][0][0]
00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
00 00 00 00 00 00 00 00 00 00 00 00 03 03 03 02
14 52 00 11 40 40 00 08 08 04 F2 40 00 00 00 00
00 00 00 00 00 88 00 00 00 00 00 00 00 00 00 00
Mar 20 14:25:49 127066 [45A08960] -> osm_pi_rcv_process_set: Received error
status 0x1c for SetResp() during ACTIVE transition
Mar 20 14:25:49 127072 [4780B960] -> __osm_sm_mad_ctrl_rcv_callback: ERR
3111: Error status = 0x1C00
Mar 20 14:25:49 127084 [45A08960] -> PortInfo dump:
port number.............0x16
node_guid...............0x0005ad00000281b3
port_guid...............0x0005ad00000281b3
m_key...................0x0000000000000000
subnet_prefix...........0x0000000000000000
base_lid................0x0
master_sm_base_lid......0x0
capability_mask.........0x0
diag_code...............0x0
m_key_lease_period......0x0
local_port_num..........0x3
link_width_enabled......0x3
link_width_supported....0x3
link_width_active.......0x2
link_speed_supported....0x1
port_state..............ACTIVE
state_info2.............0x52
m_key_protect_bits......0x0
lmc.....................0x0
link_speed..............0x11
mtu_smsl................0x40
vl_cap_init_type........0x40
vl_high_limit...........0x0
vl_arb_high_cap.........0x8
vl_arb_low_cap..........0x8
init_rep_mtu_cap........0x4
vl_stall_life...........0xF2
vl_enforce..............0x40
m_key_violations........0x0
p_key_violations........0x0
q_key_violations........0x0
guid_cap................0x0
client_reregister.......0x0
subnet_timeout..........0x0
resp_time_value.........0x0
error_threshold.........0x88
Mar 20 14:25:49 127103 [45A08960] -> Capabilities Mask:
Mar 20 14:25:49 127121 [4780B960] -> SMP dump:
base_ver................0x1
mgmt_class..............0x81
class_ver...............0x1
method..................0x81 (SubnGetResp)
D bit...................0x1
status..................0x1C00
hop_ptr.................0x0
hop_count...............0x4
trans_id................0x7d16a
attr_id.................0x15 (PortInfo)
resv....................0x0
attr_mod................0x17
m_key...................0x0000000000000000
dr_slid.................0xFFFF
dr_dlid.................0xFFFF
Initial path: [0][1][13][1][6]
Return path: [0][9][13][D][3]
Reserved: [0][0][0][0][0][0][0]
00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
00 00 00 00 00 00 00 00 00 00 00 00 03 03 03 02
14 52 00 11 40 40 00 08 08 04 F2 40 00 00 00 00
00 00 00 00 00 88 00 00 00 00 00 00 00 00 00 00
Mar 20 14:25:49 127188 [4780B960] -> __osm_sm_mad_ctrl_rcv_callback: ERR
3111: Error status = 0x1C00
Mar 20 14:25:49 127220 [4780B960] -> SMP dump:
base_ver................0x1
mgmt_class..............0x81
class_ver...............0x1
method..................0x81 (SubnGetResp)
D bit...................0x1
status..................0x1C00
hop_ptr.................0x0
hop_count...............0x4
trans_id................0x7d16b
attr_id.................0x15 (PortInfo)
resv....................0x0
attr_mod................0x18
m_key...................0x0000000000000000
dr_slid.................0xFFFF
dr_dlid.................0xFFFF
Initial path: [0][1][13][1][6]
Return path: [0][9][13][D][3]
Reserved: [0][0][0][0][0][0][0]
00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
00 00 00 00 00 00 00 00 00 00 00 00 03 03 03 02
14 52 00 11 40 40 00 08 08 04 F2 40 00 00 00 00
00 00 00 00 00 88 00 00 00 00 00 00 00 00 00 00
Mar 20 14:25:49 127326 [44606960] -> osm_pi_rcv_process_set: Received error
status 0x1c for SetResp() during ACTIVE transition
Mar 20 14:25:49 127339 [44606960] -> PortInfo dump:
port number.............0x17
node_guid...............0x0005ad00000281b3
port_guid...............0x0005ad00000281b3
m_key...................0x0000000000000000
subnet_prefix...........0x0000000000000000
base_lid................0x0
master_sm_base_lid......0x0
capability_mask.........0x0
diag_code...............0x0
m_key_lease_period......0x0
local_port_num..........0x3
link_width_enabled......0x3
link_width_supported....0x3
link_width_active.......0x2
link_speed_supported....0x1
port_state..............ACTIVE
state_info2.............0x52
m_key_protect_bits......0x0
lmc.....................0x0
link_speed..............0x11
mtu_smsl................0x40
vl_cap_init_type........0x40
vl_high_limit...........0x0
vl_arb_high_cap.........0x8
vl_arb_low_cap..........0x8
init_rep_mtu_cap........0x4
vl_stall_life...........0xF2
vl_enforce..............0x40
m_key_violations........0x0
p_key_violations........0x0
q_key_violations........0x0
guid_cap................0x0
client_reregister.......0x0
subnet_timeout..........0x0
resp_time_value.........0x0
error_threshold.........0x88
Mar 20 14:25:49 127357 [44606960] -> Capabilities Mask:
Mar 20 14:25:49 127378 [45007960] -> osm_pi_rcv_process_set: Received error
status 0x1c for SetResp() during ACTIVE transition
Mar 20 14:25:49 127397 [45007960] -> PortInfo dump:
port number.............0x18
node_guid...............0x0005ad00000281b3
port_guid...............0x0005ad00000281b3
m_key...................0x0000000000000000
subnet_prefix...........0x0000000000000000
base_lid................0x0
master_sm_base_lid......0x0
capability_mask.........0x0
diag_code...............0x0
m_key_lease_period......0x0
local_port_num..........0x3
link_width_enabled......0x3
link_width_supported....0x3
link_width_active.......0x2
link_speed_supported....0x1
port_state..............ACTIVE
state_info2.............0x52
m_key_protect_bits......0x0
lmc.....................0x0
link_speed..............0x11
mtu_smsl................0x40
vl_cap_init_type........0x40
vl_high_limit...........0x0
vl_arb_high_cap.........0x8
vl_arb_low_cap..........0x8
init_rep_mtu_cap........0x4
vl_stall_life...........0xF2
vl_enforce..............0x40
m_key_violations........0x0
p_key_violations........0x0
q_key_violations........0x0
guid_cap................0x0
client_reregister.......0x0
subnet_timeout..........0x0
resp_time_value.........0x0
error_threshold.........0x88
Mar 20 14:25:49 127410 [45007960] -> Capabilities Mask:
Mar 20 14:25:49 132961 [43204960] -> SUBNET UP
Mar 20 14:25:49 523879 [44606960] -> osm_ucast_mgr_process: Min Hop Tables
configured on all switches
Mar 20 14:25:49 580522 [42803960] -> SUBNET UP
Mar 20 14:26:04 718574 [43C05960] -> __osm_trap_rcv_process_request:
Received Generic Notice type:0x01 num:128 Producer:2 from LID:0x0152
TID:0x0000000000000000
Mar 20 14:26:04 718819 [43C05960] -> osm_report_notice: Reporting Generic
Notice type:1 num:128 from LID:0x0152
GID:0xfe80000000000000,0x0005ad0000027c84
Mar 20 14:26:04 836781 [45A08960] -> __osm_trap_rcv_process_request:
Received Generic Notice type:0x01 num:128 Producer:2 from LID:0x0152
TID:0x0000000000000001
Mar 20 14:26:04 836881 [45A08960] -> osm_report_notice: Reporting Generic
Notice type:1 num:128 from LID:0x0152
GID:0xfe80000000000000,0x0005ad0000027c84
Mar 20 14:26:04 858762 [45A08960] -> __osm_trap_rcv_process_request:
Received Generic Notice type:0x01 num:128 Producer:2 from LID:0x0152
TID:0x0000000000000002
Mar 20 14:26:04 860242 [45A08960] -> osm_report_notice: Reporting Generic
Notice type:1 num:128 from LID:0x0152
GID:0xfe80000000000000,0x0005ad0000027c84
Mar 20 14:26:04 997451 [45007960] -> __osm_trap_rcv_process_request:
Received Generic Notice type:0x01 num:128 Producer:2 from LID:0x0152
TID:0x0000000000000003
Mar 20 14:26:04 997647 [45007960] -> osm_report_notice: Reporting Generic
Notice type:1 num:128 from LID:0x0152
GID:0xfe80000000000000,0x0005ad0000027c84
Mar 20 14:26:05 180722 [43204960] -> __osm_trap_rcv_process_request:
Received Generic Notice type:0x01 num:128 Producer:2 from LID:0x0152
TID:0x0000000000000004
Mar 20 14:26:05 180855 [43204960] -> osm_report_notice: Reporting Generic
Notice type:1 num:128 from LID:0x0152
GID:0xfe80000000000000,0x0005ad0000027c84
Mar 20 14:26:05 209122 [41401960] -> __osm_trap_rcv_process_request:
Received Generic Notice type:0x01 num:128 Producer:2 from LID:0x0152
TID:0x0000000000000005
Mar 20 14:26:05 209200 [41401960] -> osm_report_notice: Reporting Generic
Notice type:1 num:128 from LID:0x0152
GID:0xfe80000000000000,0x0005ad0000027c84
Mar 20 14:26:05 347419 [45A08960] -> __osm_trap_rcv_process_request:
Received Generic Notice type:0x01 num:128 Producer:2 from LID:0x0152
TID:0x0000000000000006
Mar 20 14:26:05 347488 [45A08960] -> osm_report_notice: Reporting Generic
Notice type:1 num:128 from LID:0x0152
GID:0xfe80000000000000,0x0005ad0000027c84
Mar 20 14:26:05 378670 [42803960] -> __osm_trap_rcv_process_request:
Received Generic Notice type:0x01 num:128 Producer:2 from LID:0x0152
TID:0x0000000000000007
Mar 20 14:26:05 378739 [42803960] -> osm_report_notice: Reporting Generic
Notice type:1 num:128 from LID:0x0152
GID:0xfe80000000000000,0x0005ad0000027c84
Mar 20 14:26:05 409112 [41401960] -> osm_report_notice: Reporting Generic
Notice type:3 num:65 from LID:0x0092
GID:0xfe80000000000000,0x0005ad0000024bbb
Mar 20 14:26:05 409121 [41401960] -> Removed port with
GUID:0x0005ad00000249d3 LID range [0xB1,0xB1] of node:saguaro-23-2 HCA-1
Mar 20 14:26:05 443639 [41401960] -> osm_ucast_mgr_process: Min Hop Tables
configured on all switches
Mar 20 14:26:05 483503 [45007960] -> __osm_trap_rcv_process_request:
Received Generic Notice type:0x01 num:128 Producer:2 from LID:0x0152
TID:0x0000000000000008
Mar 20 14:26:05 486002 [45007960] -> osm_report_notice: Reporting Generic
Notice type:1 num:128 from LID:0x0152
GID:0xfe80000000000000,0x0005ad0000027c84
Mar 20 14:26:05 499183 [44606960] -> SUBNET UP
Mar 20 14:26:05 499856 [43C05960] -> __osm_trap_rcv_process_request:
Received Generic Notice type:0x01 num:128 Producer:2 from LID:0x0152
TID:0x0000000000000009
Mar 20 14:26:05 499941 [43C05960] -> osm_report_notice: Reporting Generic
Notice type:1 num:128 from LID:0x0152
GID:0xfe80000000000000,0x0005ad0000027c84
Mar 20 14:26:05 521857 [43204960] -> __osm_trap_rcv_process_request:
Received Generic Notice type:0x01 num:128 Producer:2 from LID:0x0152
TID:0x000000000000000a
Mar 20 14:26:05 521971 [43204960] -> osm_report_notice: Reporting Generic
Notice type:1 num:128 from LID:0x0152
GID:0xfe80000000000000,0x0005ad0000027c84
Mar 20 14:26:05 532569 [41401960] -> __osm_trap_rcv_process_request:
Received Generic Notice type:0x01 num:128 Producer:2 from LID:0x0152
TID:0x000000000000000b
Mar 20 14:26:05 532624 [41401960] -> __osm_trap_rcv_process_request: ERR
3804: Received trap 11 times consecutively
Mar 20 14:26:05 633813 [43204960] -> __osm_trap_rcv_process_request:
Received Generic Notice type:0x01 num:128 Producer:2 from LID:0x0152
TID:0x000000000000000c
Mar 20 14:26:05 633869 [43204960] -> __osm_trap_rcv_process_request: ERR
3804: Received trap 12 times consecutively
Mar 20 14:26:05 655421 [41401960] -> __osm_trap_rcv_process_request:
Received Generic Notice type:0x01 num:128 Producer:2 from LID:0x0152
TID:0x000000000000000d
Mar 20 14:26:05 655501 [41401960] -> __osm_trap_rcv_process_request: ERR
3804: Received trap 13 times consecutively
Mar 20 14:26:05 702652 [42803960] -> __osm_trap_rcv_process_request:
Received Generic Notice type:0x01 num:128 Producer:2 from LID:0x0152
TID:0x000000000000000e
Mar 20 14:26:05 702745 [42803960] -> __osm_trap_rcv_process_request: ERR
3804: Received trap 14 times consecutively
Mar 20 14:26:05 817201 [43204960] -> osm_report_notice: Reporting Generic
Notice type:3 num:65 from LID:0x0092
GID:0xfe80000000000000,0x0005ad0000024bbb
Mar 20 14:26:05 817216 [43204960] -> Removed port with
GUID:0x0005ad000002510b LID range [0xB5,0xB5] of node:saguaro-23-6 HCA-1
Mar 20 14:26:05 817235 [43204960] -> osm_report_notice: Reporting Generic
Notice type:3 num:65 from LID:0x0092
GID:0xfe80000000000000,0x0005ad0000024bbb
Mar 20 14:26:05 817241 [43204960] -> Removed port with
GUID:0x0005ad0000024d47 LID range [0xB6,0xB6] of node:saguaro-23-7 HCA-1
Mar 20 14:26:05 817259 [43204960] -> osm_report_notice: Reporting Generic
Notice type:3 num:65 from LID:0x0092
GID:0xfe80000000000000,0x0005ad0000024bbb
Mar 20 14:26:05 817264 [43204960] -> Removed port with
GUID:0x0005ad0000024d8b LID range [0xB7,0xB7] of node:saguaro-23-8 HCA-1
Mar 20 14:26:05 821322 [41E02960] -> osm_report_notice: Reporting Generic
Notice type:3 num:64 from LID:0x0092
GID:0xfe80000000000000,0x0005ad0000024bbb
Mar 20 14:26:05 821330 [41E02960] -> Discovered new port with
GUID:0x0005ad00000249d3 LID range [0xB1,0xB1] of node:saguaro-23-2 HCA-1
Mar 20 14:26:05 847950 [45007960] -> __osm_trap_rcv_process_request:
Received Generic Notice type:0x01 num:128 Producer:2 from LID:0x0152
TID:0x000000000000000f
Mar 20 14:26:05 848031 [45007960] -> __osm_trap_rcv_process_request: ERR
3804: Received trap 15 times consecutively
Mar 20 14:26:05 852036 [41E02960] -> osm_ucast_mgr_process: Min Hop Tables
configured on all switches
Mar 20 14:26:05 893954 [45007960] -> __osm_trap_rcv_process_request:
Received Generic Notice type:0x01 num:128 Producer:2 from LID:0x0152
TID:0x0000000000000010
Mar 20 14:26:05 894021 [45007960] -> __osm_trap_rcv_process_request: ERR
3804: Received trap 16 times consecutively
Mar 20 14:26:05 910489 [44606960] -> SUBNET UP
Mar 20 14:26:05 999993 [43C05960] -> __osm_trap_rcv_process_request:
Received Generic Notice type:0x01 num:128 Producer:2 from LID:0x0152
TID:0x0000000000000011
Mar 20 14:26:06 000039 [43C05960] -> __osm_trap_rcv_process_request: ERR
3804: Received trap 17 times consecutively
Mar 20 14:26:06 021880 [45A08960] -> __osm_trap_rcv_process_request:
Received Generic Notice type:0x01 num:128 Producer:2 from LID:0x0152
TID:0x0000000000000012
Mar 20 14:26:06 021970 [45A08960] -> __osm_trap_rcv_process_request: ERR
3804: Received trap 18 times consecutively
Mar 20 14:26:06 043912 [44606960] -> __osm_trap_rcv_process_request:
Received Generic Notice type:0x01 num:128 Producer:2 from LID:0x0152
TID:0x0000000000000013
Mar 20 14:26:06 044001 [44606960] -> __osm_trap_rcv_process_request: ERR
3804: Received trap 19 times consecutively
Mar 20 14:26:06 052878 [44606960] -> __osm_trap_rcv_process_request:
Received Generic Notice type:0x01 num:128 Producer:2 from LID:0x0152
TID:0x0000000000000014
Mar 20 14:26:06 052975 [44606960] -> __osm_trap_rcv_process_request: ERR
3804: Received trap 20 times consecutively
Mar 20 14:26:06 147560 [42803960] -> __osm_trap_rcv_process_request:
Received Generic Notice type:0x01 num:128 Producer:2 from LID:0x0152
TID:0x0000000000000015
Mar 20 14:26:06 147616 [42803960] -> __osm_trap_rcv_process_request: ERR
3804: Received trap 21 times consecutively
Mar 20 14:26:06 158945 [41401960] -> __osm_trap_rcv_process_request:
Received Generic Notice type:0x01 num:128 Producer:2 from LID:0x0152
TID:0x0000000000000016
Mar 20 14:26:06 158978 [41401960] -> __osm_trap_rcv_process_request: ERR
3804: Received trap 22 times consecutively
Mar 20 14:26:06 346046 [44606960] -> __osm_trap_rcv_process_request:
Received Generic Notice type:0x01 num:128 Producer:2 from LID:0x0152
TID:0x0000000000000017
Mar 20 14:26:06 346106 [44606960] -> __osm_trap_rcv_process_request: ERR
3804: Received trap 23 times consecutively
Mar 20 14:26:06 405311 [43204960] -> __osm_trap_rcv_process_request:
Received Generic Notice type:0x01 num:128 Producer:2 from LID:0x0152
TID:0x0000000000000018
Mar 20 14:26:06 405349 [43204960] -> __osm_trap_rcv_process_request: ERR
3804: Received trap 24 times consecutively
Mar 20 14:26:06 632882 [45007960] -> __osm_trap_rcv_process_request:
Received Generic Notice type:0x01 num:128 Producer:2 from LID:0x0152
TID:0x0000000000000019
Mar 20 14:26:06 632923 [45007960] -> __osm_trap_rcv_process_request: ERR
3804: Received trap 25 times consecutively
Mar 20 14:26:06 634031 [43C05960] -> __osm_trap_rcv_process_request:
Received Generic Notice type:0x01 num:128 Producer:2 from LID:0x0148
TID:0x0000000000000067
Mar 20 14:26:06 634110 [43C05960] -> osm_report_notice: Reporting Generic
Notice type:1 num:128 from LID:0x0148
GID:0xfe80000000000000,0x0005ad00000281b3
Mar 20 14:26:06 883831 [45007960] -> __osm_trap_rcv_process_request:
Received Generic Notice type:0x01 num:128 Producer:2 from LID:0x0152
TID:0x000000000000001a
Mar 20 14:26:06 883879 [45007960] -> __osm_trap_rcv_process_request: ERR
3804: Received trap 26 times consecutively
Mar 20 14:26:06 885475 [43C05960] -> __osm_trap_rcv_process_request:
Received Generic Notice type:0x01 num:128 Producer:2 from LID:0x0148
TID:0x0000000000000068
Mar 20 14:26:06 885560 [43C05960] -> osm_report_notice: Reporting Generic
Notice type:1 num:128 from LID:0x0148
GID:0xfe80000000000000,0x0005ad00000281b3
Mar 20 14:26:06 982877 [43204960] -> __osm_trap_rcv_process_request:
Received Generic Notice type:0x01 num:128 Producer:2 from LID:0x0152
TID:0x000000000000001b
Mar 20 14:26:06 982926 [43204960] -> __osm_trap_rcv_process_request: ERR
3804: Received trap 27 times consecutively
Mar 20 14:26:06 992809 [41E02960] -> __osm_trap_rcv_process_request:
Received Generic Notice type:0x01 num:128 Producer:2 from LID:0x0148
TID:0x0000000000000069
Mar 20 14:26:06 992871 [41E02960] -> osm_report_notice: Reporting Generic
Notice type:1 num:128 from LID:0x0148
GID:0xfe80000000000000,0x0005ad00000281b3
Mar 20 14:26:06 992909 [41E02960] -> __osm_trap_rcv_process_request:
Received Generic Notice type:0x01 num:128 Producer:2 from LID:0x0152
TID:0x000000000000001c
Mar 20 14:26:06 992943 [41E02960] -> __osm_trap_rcv_process_request: ERR
3804: Received trap 28 times consecutively
Mar 20 14:26:06 993058 [41E02960] -> osm_report_notice: Reporting Generic
Notice type:3 num:64 from LID:0x0092
GID:0xfe80000000000000,0x0005ad0000024bbb
Mar 20 14:26:06 993065 [41E02960] -> Discovered new port with
GUID:0x0005ad000002510b LID range [0xB5,0xB5] of node:saguaro-23-6 HCA-1
Mar 20 14:26:06 993069 [41E02960] -> osm_report_notice: Reporting Generic
Notice type:3 num:64 from LID:0x0092
GID:0xfe80000000000000,0x0005ad0000024bbb
Mar 20 14:26:06 993074 [41E02960] -> Discovered new port with
GUID:0x0005ad0000024d47 LID range [0xB6,0xB6] of node:saguaro-23-7 HCA-1
Mar 20 14:26:07 023890 [41E02960] -> osm_ucast_mgr_process: Min Hop Tables
configured on all switches
Mar 20 14:26:07 085081 [41E02960] -> SUBNET UP
Mar 20 14:26:07 348105 [45A08960] -> __osm_trap_rcv_process_request:
Received Generic Notice type:0x01 num:128 Producer:2 from LID:0x0152
TID:0x000000000000001d
Mar 20 14:26:07 348218 [45A08960] -> __osm_trap_rcv_process_request: ERR
3804: Received trap 29 times consecutively
Mar 20 14:26:07 348958 [45A08960] -> __osm_trap_rcv_process_request:
Received Generic Notice type:0x01 num:128 Producer:2 from LID:0x0148
TID:0x000000000000006a
Mar 20 14:26:07 349041 [45A08960] -> osm_report_notice: Reporting Generic
Notice type:1 num:128 from LID:0x0148
GID:0xfe80000000000000,0x0005ad00000281b3
Mar 20 14:26:07 540871 [41401960] -> __osm_trap_rcv_process_request:
Received Generic Notice type:0x01 num:128 Producer:2 from LID:0x0148
TID:0x000000000000006b
Mar 20 14:26:07 540983 [41401960] -> osm_report_notice: Reporting Generic
Notice type:1 num:128 from LID:0x0148
GID:0xfe80000000000000,0x0005ad00000281b3
Mar 20 14:26:07 541063 [43204960] -> __osm_trap_rcv_process_request:
Received Generic Notice type:0x01 num:128 Producer:2 from LID:0x001B
TID:0x000000000000007c
Mar 20 14:26:07 541131 [43204960] -> osm_report_notice: Reporting Generic
Notice type:1 num:128 from LID:0x001B
GID:0xfe80000000000000,0x0005ad00000281a7
Mar 20 14:26:07 585394 [43C05960] -> __osm_trap_rcv_process_request:
Received Generic Notice type:0x01 num:128 Producer:2 from LID:0x0148
TID:0x000000000000006c
Mar 20 14:26:07 585464 [43C05960] -> osm_report_notice: Reporting Generic
Notice type:1 num:128 from LID:0x0148
GID:0xfe80000000000000,0x0005ad00000281b3
Mar 20 14:26:07 607406 [45A08960] -> __osm_trap_rcv_process_request:
Received Generic Notice type:0x01 num:128 Producer:2 from LID:0x0148
TID:0x000000000000006d
Mar 20 14:26:07 607486 [45A08960] -> osm_report_notice: Reporting Generic
Notice type:1 num:128 from LID:0x0148
GID:0xfe80000000000000,0x0005ad00000281b3
Mar 20 14:26:07 850410 [42803960] -> __osm_trap_rcv_process_request:
Received Generic Notice type:0x01 num:128 Producer:2 from LID:0x0148
TID:0x000000000000006e
Mar 20 14:26:07 850483 [42803960] -> osm_report_notice: Reporting Generic
Notice type:1 num:128 from LID:0x0148
GID:0xfe80000000000000,0x0005ad00000281b3
Mar 20 14:26:07 956365 [41E02960] -> __osm_trap_rcv_process_request:
Received Generic Notice type:0x01 num:128 Producer:2 from LID:0x001B
TID:0x000000000000007d
Mar 20 14:26:07 956404 [42803960] -> osm_report_notice: Reporting Generic
Notice type:3 num:64 from LID:0x0092
GID:0xfe80000000000000,0x0005ad0000024bbb
Mar 20 14:26:07 956413 [42803960] -> Discovered new port with
GUID:0x0005ad0000024d8b LID range [0xB7,0xB7] of node:saguaro-23-8 HCA-1
Mar 20 14:26:07 987136 [42803960] -> osm_ucast_mgr_process: Min Hop Tables
configured on all switches
Mar 20 14:26:08 018887 [41E02960] -> osm_report_notice: Reporting Generic
Notice type:1 num:128 from LID:0x001B
GID:0xfe80000000000000,0x0005ad00000281a7
Mar 20 14:26:08 032634 [4780B960] -> __osm_sm_mad_ctrl_rcv_callback: ERR
3111: Error status = 0x1C00
Mar 20 14:26:08 032679 [4780B960] -> SMP dump:
base_ver................0x1
mgmt_class..............0x81
class_ver...............0x1
method..................0x81 (SubnGetResp)
D bit...................0x1
status..................0x1C00
hop_ptr.................0x0
hop_count...............0x4
trans_id................0x813ce
attr_id.................0x15 (PortInfo)
resv....................0x0
attr_mod................0x16
m_key...................0x0000000000000000
dr_slid.................0xFFFF
dr_dlid.................0xFFFF
Initial path: [0][1][12][4][5]
Return path: [0][9][14][D][5]
Reserved: [0][0][0][0][0][0][0]
00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
00 00 00 00 00 00 00 00 00 00 00 00 05 03 03 02
14 52 00 11 40 40 00 08 08 04 F2 40 00 00 00 00
00 00 00 00 00 88 00 00 00 00 00 00 00 00 00 00
Mar 20 14:26:08 032749 [41E02960] -> osm_pi_rcv_process_set: Received error
status 0x1c for SetResp() during ACTIVE transition
Mar 20 14:26:08 032757 [41E02960] -> PortInfo dump:
port number.............0x16
node_guid...............0x0005ad00000281b3
port_guid...............0x0005ad00000281b3
m_key...................0x0000000000000000
subnet_prefix...........0x0000000000000000
base_lid................0x0
master_sm_base_lid......0x0
capability_mask.........0x0
diag_code...............0x0
m_key_lease_period......0x0
local_port_num..........0x5
link_width_enabled......0x3
link_width_supported....0x3
link_width_active.......0x2
link_speed_supported....0x1
port_state..............ACTIVE
state_info2.............0x52
m_key_protect_bits......0x0
lmc.....................0x0
link_speed..............0x11
mtu_smsl................0x40
vl_cap_init_type........0x40
vl_high_limit...........0x0
vl_arb_high_cap.........0x8
vl_arb_low_cap..........0x8
init_rep_mtu_cap........0x4
vl_stall_life...........0xF2
vl_enforce..............0x40
m_key_violations........0x0
p_key_violations........0x0
q_key_violations........0x0
guid_cap................0x0
client_reregister.......0x0
subnet_timeout..........0x0
resp_time_value.........0x0
error_threshold.........0x88
Mar 20 14:26:08 032774 [41E02960] -> Capabilities Mask:
Mar 20 14:26:08 033119 [4780B960] -> __osm_sm_mad_ctrl_rcv_callback: ERR
3111: Error status = 0x1C00
Mar 20 14:26:08 033154 [4780B960] -> SMP dump:
base_ver................0x1
mgmt_class..............0x81
class_ver...............0x1
method..................0x81 (SubnGetResp)
D bit...................0x1
status..................0x1C00
hop_ptr.................0x0
hop_count...............0x4
trans_id................0x813cf
attr_id.................0x15 (PortInfo)
resv....................0x0
attr_mod................0x17
m_key...................0x0000000000000000
dr_slid.................0xFFFF
dr_dlid.................0xFFFF
Initial path: [0][1][12][4][5]
Return path: [0][9][14][D][5]
Reserved: [0][0][0][0][0][0][0]
00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
00 00 00 00 00 00 00 00 00 00 00 00 05 03 03 02
14 52 00 11 40 40 00 08 08 04 F2 40 00 00 00 00
00 00 00 00 00 88 00 00 00 00 00 00 00 00 00 00
Mar 20 14:26:08 033202 [43C05960] -> osm_pi_rcv_process_set: Received error
status 0x1c for SetResp() during ACTIVE transition
Mar 20 14:26:08 033213 [43C05960] -> PortInfo dump:
port number.............0x17
node_guid...............0x0005ad00000281b3
port_guid...............0x0005ad00000281b3
m_key...................0x0000000000000000
subnet_prefix...........0x0000000000000000
base_lid................0x0
master_sm_base_lid......0x0
capability_mask.........0x0
diag_code...............0x0
m_key_lease_period......0x0
local_port_num..........0x5
link_width_enabled......0x3
link_width_supported....0x3
link_width_active.......0x2
link_speed_supported....0x1
port_state..............ACTIVE
state_info2.............0x52
m_key_protect_bits......0x0
lmc.....................0x0
link_speed..............0x11
mtu_smsl................0x40
vl_cap_init_type........0x40
vl_high_limit...........0x0
vl_arb_high_cap.........0x8
vl_arb_low_cap..........0x8
init_rep_mtu_cap........0x4
vl_stall_life...........0xF2
vl_enforce..............0x40
m_key_violations........0x0
p_key_violations........0x0
q_key_violations........0x0
guid_cap................0x0
client_reregister.......0x0
subnet_timeout..........0x0
resp_time_value.........0x0
error_threshold.........0x88
Mar 20 14:26:08 033231 [43C05960] -> Capabilities Mask:
Mar 20 14:26:08 038497 [45A08960] -> SUBNET UP
Mar 20 14:26:08 055480 [43C05960] -> __osm_trap_rcv_process_request:
Received Generic Notice type:0x01 num:128 Producer:2 from LID:0x001B
TID:0x000000000000007e
Mar 20 14:26:08 055587 [43C05960] -> osm_report_notice: Reporting Generic
Notice type:1 num:128 from LID:0x001B
GID:0xfe80000000000000,0x0005ad00000281a7
Mar 20 14:26:08 372288 [43204960] -> __osm_trap_rcv_process_request:
Received Generic Notice type:0x01 num:128 Producer:2 from LID:0x001B
TID:0x000000000000007f
Mar 20 14:26:08 376158 [42803960] -> osm_ucast_mgr_process: Min Hop Tables
configured on all switches
Mar 20 14:26:08 418607 [44606960] -> __osm_trap_rcv_process_request:
Received Generic Notice type:0x01 num:128 Producer:2 from LID:0x001B
TID:0x0000000000000080
Mar 20 14:26:08 420668 [43204960] -> osm_report_notice: Reporting Generic
Notice type:1 num:128 from LID:0x001B
GID:0xfe80000000000000,0x0005ad00000281a7
Mar 20 14:26:08 420714 [44606960] -> osm_report_notice: Reporting Generic
Notice type:1 num:128 from LID:0x001B
GID:0xfe80000000000000,0x0005ad00000281a7
Mar 20 14:26:08 430046 [4780B960] -> __osm_sm_mad_ctrl_rcv_callback: ERR
3111: Error status = 0x1C00
Mar 20 14:26:08 430147 [4780B960] -> SMP dump:
base_ver................0x1
mgmt_class..............0x81
class_ver...............0x1
method..................0x81 (SubnGetResp)
D bit...................0x1
status..................0x1C00
hop_ptr.................0x0
hop_count...............0x4
trans_id................0x820fa
attr_id.................0x15 (PortInfo)
resv....................0x0
attr_mod................0x16
m_key...................0x0000000000000000
dr_slid.................0xFFFF
dr_dlid.................0xFFFF
Initial path: [0][1][15][1][4]
Return path: [0][9][18][D][1]
Reserved: [0][0][0][0][0][0][0]
00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
00 00 00 00 00 00 00 00 00 00 00 00 01 03 03 02
14 52 00 11 40 40 00 08 08 04 F2 40 00 00 00 00
00 00 00 00 00 88 00 00 00 00 00 00 00 00 00 00
Mar 20 14:26:08 430236 [4780B960] -> __osm_sm_mad_ctrl_rcv_callback: ERR
3111: Error status = 0x1C00
Mar 20 14:26:08 430236 [43C05960] -> osm_pi_rcv_process_set: Received error
status 0x1c for SetResp() during ACTIVE transition
Mar 20 14:26:08 430267 [4780B960] -> SMP dump:
base_ver................0x1
mgmt_class..............0x81
class_ver...............0x1
method..................0x81 (SubnGetResp)
D bit...................0x1
status..................0x1C00
hop_ptr.................0x0
hop_count...............0x4
trans_id................0x820fb
attr_id.................0x15 (PortInfo)
resv....................0x0
attr_mod................0x18
m_key...................0x0000000000000000
dr_slid.................0xFFFF
dr_dlid.................0xFFFF
Initial path: [0][1][12][1][6]
Return path: [0][9][14][D][3]
Reserved: [0][0][0][0][0][0][0]
00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
00 00 00 00 00 00 00 00 00 00 00 00 03 03 03 02
14 52 00 11 40 40 00 08 08 04 F2 40 00 00 00 00
00 00 00 00 00 88 00 00 00 00 00 00 00 00 00 00
Mar 20 14:26:08 430262 [43C05960] -> PortInfo dump:
port number.............0x16
node_guid...............0x0005ad00000281a7
port_guid...............0x0005ad00000281a7
m_key...................0x0000000000000000
subnet_prefix...........0x0000000000000000
base_lid................0x0
master_sm_base_lid......0x0
capability_mask.........0x0
diag_code...............0x0
m_key_lease_period......0x0
local_port_num..........0x1
link_width_enabled......0x3
link_width_supported....0x3
link_width_active.......0x2
link_speed_supported....0x1
port_state..............ACTIVE
state_info2.............0x52
m_key_protect_bits......0x0
lmc.....................0x0
link_speed..............0x11
mtu_smsl................0x40
vl_cap_init_type........0x40
vl_high_limit...........0x0
vl_arb_high_cap.........0x8
vl_arb_low_cap..........0x8
init_rep_mtu_cap........0x4
vl_stall_life...........0xF2
vl_enforce..............0x40
m_key_violations........0x0
p_key_violations........0x0
q_key_violations........0x0
guid_cap................0x0
client_reregister.......0x0
subnet_timeout..........0x0
resp_time_value.........0x0
error_threshold.........0x88
Mar 20 14:26:08 430286 [43C05960] -> Capabilities Mask:
Mar 20 14:26:08 430350 [43C05960] -> osm_pi_rcv_process_set: Received error
status 0x1c for SetResp() during ACTIVE transition
Mar 20 14:26:08 430362 [43C05960] -> PortInfo dump:
port number.............0x18
node_guid...............0x0005ad00000281b3
port_guid...............0x0005ad00000281b3
m_key...................0x0000000000000000
subnet_prefix...........0x0000000000000000
base_lid................0x0
master_sm_base_lid......0x0
capability_mask.........0x0
diag_code...............0x0
m_key_lease_period......0x0
local_port_num..........0x3
link_width_enabled......0x3
link_width_supported....0x3
link_width_active.......0x2
link_speed_supported....0x1
port_state..............ACTIVE
state_info2.............0x52
m_key_protect_bits......0x0
lmc.....................0x0
link_speed..............0x11
mtu_smsl................0x40
vl_cap_init_type........0x40
vl_high_limit...........0x0
vl_arb_high_cap.........0x8
vl_arb_low_cap..........0x8
init_rep_mtu_cap........0x4
vl_stall_life...........0xF2
vl_enforce..............0x40
m_key_violations........0x0
p_key_violations........0x0
q_key_violations........0x0
guid_cap................0x0
client_reregister.......0x0
subnet_timeout..........0x0
resp_time_value.........0x0
error_threshold.........0x88
Mar 20 14:26:08 430375 [43C05960] -> Capabilities Mask:
Mar 20 14:26:08 435317 [43C05960] -> SUBNET UP
Mar 20 14:26:08 583769 [41401960] -> __osm_trap_rcv_process_request:
Received Generic Notice type:0x01 num:128 Producer:2 from LID:0x0152
TID:0x000000000000001e
Mar 20 14:26:08 583903 [41401960] -> __osm_trap_rcv_process_request: ERR
3804: Received trap 30 times consecutively
Mar 20 14:26:08 854841 [41E02960] -> osm_ucast_mgr_process: Min Hop Tables
configured on all switches
Mar 20 14:26:08 913273 [4780B960] -> __osm_sm_mad_ctrl_rcv_callback: ERR
3111: Error status = 0x1C00
Mar 20 14:26:08 913349 [4780B960] -> SMP dump:
base_ver................0x1
mgmt_class..............0x81
class_ver...............0x1
method..................0x81 (SubnGetResp)
D bit...................0x1
status..................0x1C00
hop_ptr.................0x0
hop_count...............0x4
trans_id................0x82e32
attr_id.................0x15 (PortInfo)
resv....................0x0
attr_mod................0x13
m_key...................0x0000000000000000
dr_slid.................0xFFFF
dr_dlid.................0xFFFF
Initial path: [0][1][17][2][5]
Return path: [0][9][14][E][2]
Reserved: [0][0][0][0][0][0][0]
00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
00 00 00 00 00 00 00 00 00 00 00 00 02 03 03 02
14 52 00 11 40 40 00 08 08 04 F2 40 00 00 00 00
00 00 00 00 00 88 00 00 00 00 00 00 00 00 00 00
Mar 20 14:26:08 913415 [45A08960] -> osm_pi_rcv_process_set: Received error
status 0x1c for SetResp() during ACTIVE transition
Mar 20 14:26:08 913432 [45A08960] -> PortInfo dump:
port number.............0x13
node_guid...............0x0005ad00000281a7
port_guid...............0x0005ad00000281a7
m_key...................0x0000000000000000
subnet_prefix...........0x0000000000000000
base_lid................0x0
master_sm_base_lid......0x0
capability_mask.........0x0
diag_code...............0x0
m_key_lease_period......0x0
local_port_num..........0x2
link_width_enabled......0x3
link_width_supported....0x3
link_width_active.......0x2
link_speed_supported....0x1
port_state..............ACTIVE
state_info2.............0x52
m_key_protect_bits......0x0
lmc.....................0x0
link_speed..............0x11
mtu_smsl................0x40
vl_cap_init_type........0x40
vl_high_limit...........0x0
vl_arb_high_cap.........0x8
vl_arb_low_cap..........0x8
init_rep_mtu_cap........0x4
vl_stall_life...........0xF2
vl_enforce..............0x40
m_key_violations........0x0
p_key_violations........0x0
q_key_violations........0x0
guid_cap................0x0
client_reregister.......0x0
subnet_timeout..........0x0
resp_time_value.........0x0
error_threshold.........0x88
Mar 20 14:26:08 913449 [45A08960] -> Capabilities Mask:
Mar 20 14:26:08 913598 [4780B960] -> __osm_sm_mad_ctrl_rcv_callback: ERR
3111: Error status = 0x1C00
Mar 20 14:26:08 913676 [4780B960] -> SMP dump:
base_ver................0x1
mgmt_class..............0x81
class_ver...............0x1
method..................0x81 (SubnGetResp)
D bit...................0x1
status..................0x1C00
hop_ptr.................0x0
hop_count...............0x4
trans_id................0x82e33
attr_id.................0x15 (PortInfo)
resv....................0x0
attr_mod................0x17
m_key...................0x0000000000000000
dr_slid.................0xFFFF
dr_dlid.................0xFFFF
Initial path: [0][1][17][2][5]
Return path: [0][9][14][E][2]
Reserved: [0][0][0][0][0][0][0]
00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
00 00 00 00 00 00 00 00 00 00 00 00 02 03 03 02
14 52 00 11 40 40 00 08 08 04 F2 40 00 00 00 00
00 00 00 00 00 88 00 00 00 00 00 00 00 00 00 00
Mar 20 14:26:08 913727 [43C05960] -> osm_pi_rcv_process_set: Received error
status 0x1c for SetResp() during ACTIVE transition
Mar 20 14:26:08 913732 [4780B960] -> __osm_sm_mad_ctrl_rcv_callback: ERR
3111: Error status = 0x1C00
Mar 20 14:26:08 913734 [43C05960] -> PortInfo dump:
port number.............0x17
node_guid...............0x0005ad00000281a7
port_guid...............0x0005ad00000281a7
m_key...................0x0000000000000000
subnet_prefix...........0x0000000000000000
base_lid................0x0
master_sm_base_lid......0x0
capability_mask.........0x0
diag_code...............0x0
m_key_lease_period......0x0
local_port_num..........0x2
link_width_enabled......0x3
link_width_supported....0x3
link_width_active.......0x2
link_speed_supported....0x1
port_state..............ACTIVE
state_info2.............0x52
m_key_protect_bits......0x0
lmc.....................0x0
link_speed..............0x11
mtu_smsl................0x40
vl_cap_init_type........0x40
vl_high_limit...........0x0
vl_arb_high_cap.........0x8
vl_arb_low_cap..........0x8
init_rep_mtu_cap........0x4
vl_stall_life...........0xF2
vl_enforce..............0x40
m_key_violations........0x0
p_key_violations........0x0
q_key_violations........0x0
guid_cap................0x0
client_reregister.......0x0
subnet_timeout..........0x0
resp_time_value.........0x0
error_threshold.........0x88
Mar 20 14:26:08 913752 [43C05960] -> Capabilities Mask:
Mar 20 14:26:08 913766 [4780B960] -> SMP dump:
base_ver................0x1
mgmt_class..............0x81
class_ver...............0x1
method..................0x81 (SubnGetResp)
D bit...................0x1
status..................0x1C00
hop_ptr.................0x0
hop_count...............0x4
trans_id................0x82e34
attr_id.................0x15 (PortInfo)
resv....................0x0
attr_mod................0x18
m_key...................0x0000000000000000
dr_slid.................0xFFFF
dr_dlid.................0xFFFF
Initial path: [0][1][17][2][5]
Return path: [0][9][14][E][2]
Reserved: [0][0][0][0][0][0][0]
00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
00 00 00 00 00 00 00 00 00 00 00 00 02 03 03 02
14 52 00 11 40 40 00 08 08 04 F2 40 00 00 00 00
00 00 00 00 00 88 00 00 00 00 00 00 00 00 00 00
Mar 20 14:26:08 913828 [41E02960] -> osm_pi_rcv_process_set: Received error
status 0x1c for SetResp() during ACTIVE transition
Mar 20 14:26:08 913833 [41E02960] -> PortInfo dump:
port number.............0x18
node_guid...............0x0005ad00000281a7
port_guid...............0x0005ad00000281a7
m_key...................0x0000000000000000
subnet_prefix...........0x0000000000000000
base_lid................0x0
master_sm_base_lid......0x0
capability_mask.........0x0
diag_code...............0x0
m_key_lease_period......0x0
local_port_num..........0x2
link_width_enabled......0x3
link_width_supported....0x3
link_width_active.......0x2
link_speed_supported....0x1
port_state..............ACTIVE
state_info2.............0x52
m_key_protect_bits......0x0
lmc.....................0x0
link_speed..............0x11
mtu_smsl................0x40
vl_cap_init_type........0x40
vl_high_limit...........0x0
vl_arb_high_cap.........0x8
vl_arb_low_cap..........0x8
init_rep_mtu_cap........0x4
vl_stall_life...........0xF2
vl_enforce..............0x40
m_key_violations........0x0
p_key_violations........0x0
q_key_violations........0x0
guid_cap................0x0
client_reregister.......0x0
subnet_timeout..........0x0
resp_time_value.........0x0
error_threshold.........0x88
Mar 20 14:26:08 913848 [41E02960] -> Capabilities Mask:
Mar 20 14:26:08 918887 [41E02960] -> SUBNET UP
Mar 20 14:26:48 657517 [41E02960] -> __osm_trap_rcv_process_request:
Received Generic Notice type:0x01 num:128 Producer:2 from LID:0x0148
TID:0x000000000000006f
Mar 20 14:26:48 657779 [41E02960] -> osm_report_notice: Reporting Generic
Notice type:1 num:128 from LID:0x0148
GID:0xfe80000000000000,0x0005ad00000281b3
Mar 20 14:26:48 658393 [43204960] -> __osm_trap_rcv_process_request:
Received Generic Notice type:0x01 num:128 Producer:2 from LID:0x001B
TID:0x0000000000000081
Mar 20 14:26:48 658465 [43204960] -> osm_report_notice: Reporting Generic
Notice type:1 num:128 from LID:0x001B
GID:0xfe80000000000000,0x0005ad00000281a7
Mar 20 14:26:48 979610 [41401960] -> osm_report_notice: Reporting Generic
Notice type:3 num:65 from LID:0x0092
GID:0xfe80000000000000,0x0005ad0000024bbb
Mar 20 14:26:48 979629 [41401960] -> Removed port with
GUID:0x0005ad0000024e0b LID range [0xB3,0xB3] of node:saguaro-23-4 HCA-1
Mar 20 14:26:48 979652 [41401960] -> osm_report_notice: Reporting Generic
Notice type:3 num:65 from LID:0x0092
GID:0xfe80000000000000,0x0005ad0000024bbb
Mar 20 14:26:48 979660 [41401960] -> Removed port with
GUID:0x0005ad000002510b LID range [0xB5,0xB5] of node:saguaro-23-6 HCA-1
Mar 20 14:26:48 979682 [41401960] -> osm_report_notice: Reporting Generic
Notice type:3 num:65 from LID:0x0092
GID:0xfe80000000000000,0x0005ad0000024bbb
Mar 20 14:26:48 979688 [41401960] -> Removed port with
GUID:0x0005ad000002511b LID range [0xA6,0xA6] of node:saguaro-22-1 HCA-1
Mar 20 14:26:48 979721 [41401960] -> osm_report_notice: Reporting Generic
Notice type:3 num:65 from LID:0x0092
GID:0xfe80000000000000,0x0005ad0000024bbb
Mar 20 14:26:48 979727 [41401960] -> Removed port with
GUID:0x0005ad0000024b27 LID range [0xAF,0xAF] of node:saguaro-23-0 HCA-1
Mar 20 14:26:48 979770 [41401960] -> osm_report_notice: Reporting Generic
Notice type:3 num:65 from LID:0x0092
GID:0xfe80000000000000,0x0005ad0000024bbb
Mar 20 14:26:48 979782 [41401960] -> Removed port with
GUID:0x0005ad0000025043 LID range [0xB4,0xB4] of node:saguaro-23-5 HCA-1
Mar 20 14:26:48 979799 [41401960] -> osm_report_notice: Reporting Generic
Notice type:3 num:65 from LID:0x0092
GID:0xfe80000000000000,0x0005ad0000024bbb
Mar 20 14:26:48 979804 [41401960] -> Removed port with
GUID:0x0005ad0000024d47 LID range [0xB6,0xB6] of node:saguaro-23-7 HCA-1
Mar 20 14:26:48 979822 [41401960] -> osm_report_notice: Reporting Generic
Notice type:3 num:65 from LID:0x0092
GID:0xfe80000000000000,0x0005ad0000024bbb
Mar 20 14:26:48 979827 [41401960] -> Removed port with
GUID:0x0005ad0000024d6b LID range [0xB8,0xB8] of node:saguaro-23-9 HCA-1
Mar 20 14:26:48 979845 [41401960] -> osm_report_notice: Reporting Generic
Notice type:3 num:65 from LID:0x0092
GID:0xfe80000000000000,0x0005ad0000024bbb
Mar 20 14:26:48 979849 [41401960] -> Removed port with
GUID:0x0005ad0000024977 LID range [0xA9,0xA9] of node:saguaro-22-4 HCA-1
Mar 20 14:26:48 980028 [41401960] -> osm_report_notice: Reporting Generic
Notice type:3 num:65 from LID:0x0092
GID:0xfe80000000000000,0x0005ad0000024bbb
Mar 20 14:26:48 980033 [41401960] -> Removed port with
GUID:0x0005ad0000027c84 LID range [0x152,0x152] of node:Topspin Switch TS120
Mar 20 14:26:48 980061 [41401960] -> osm_report_notice: Reporting Generic
Notice type:3 num:65 from LID:0x0092
GID:0xfe80000000000000,0x0005ad0000024bbb
Mar 20 14:26:48 980066 [41401960] -> Removed port with
GUID:0x0005ad0000024d8b LID range [0xB7,0xB7] of node:saguaro-23-8 HCA-1
Mar 20 14:26:48 980081 [41401960] -> osm_report_notice: Reporting Generic
Notice type:3 num:65 from LID:0x0092
GID:0xfe80000000000000,0x0005ad0000024bbb
Mar 20 14:26:48 980087 [41401960] -> Removed port with
GUID:0x0005ad000002498f LID range [0xA8,0xA8] of node:saguaro-22-3 HCA-1
Mar 20 14:26:48 980102 [41401960] -> osm_report_notice: Reporting Generic
Notice type:3 num:65 from LID:0x0092
GID:0xfe80000000000000,0x0005ad0000024bbb
Mar 20 14:26:48 980107 [41401960] -> Removed port with
GUID:0x0005ad0000024c9b LID range [0xA7,0xA7] of node:saguaro-22-2 HCA-1
Mar 20 14:26:48 980122 [41401960] -> osm_report_notice: Reporting Generic
Notice type:3 num:65 from LID:0x0092
GID:0xfe80000000000000,0x0005ad0000024bbb
Mar 20 14:26:48 980127 [41401960] -> Removed port with
GUID:0x0005ad0000024da7 LID range [0xB0,0xB0] of node:saguaro-23-1 HCA-1
Mar 20 14:26:48 980143 [41401960] -> osm_report_notice: Reporting Generic
Notice type:3 num:65 from LID:0x0092
GID:0xfe80000000000000,0x0005ad0000024bbb
Mar 20 14:26:48 980148 [41401960] -> Removed port with
GUID:0x0005ad0000024cbb LID range [0xB2,0xB2] of node:saguaro-23-3 HCA-1
Mar 20 14:26:48 980163 [41401960] -> osm_report_notice: Reporting Generic
Notice type:3 num:65 from LID:0x0092
GID:0xfe80000000000000,0x0005ad0000024bbb
Mar 20 14:26:48 980239 [41401960] -> Removed port with
GUID:0x0005ad00000249d3 LID range [0xB1,0xB1] of node:saguaro-23-2 HCA-1
Mar 20 14:26:48 980256 [41401960] -> osm_report_notice: Reporting Generic
Notice type:3 num:65 from LID:0x0092
GID:0xfe80000000000000,0x0005ad0000024bbb
Mar 20 14:26:48 980261 [41401960] -> Removed port with
GUID:0x0005ad0000024feb LID range [0x153,0x153] of node:saguaro-22-5 HCA-1
Mar 20 14:26:48 980365 [41401960] -> osm_report_notice: Reporting Generic
Notice type:3 num:65 from LID:0x0092
GID:0xfe80000000000000,0x0005ad0000024bbb
Mar 20 14:26:48 980371 [41401960] -> Removed port with
GUID:0x0005ad0000024afb LID range [0xA5,0xA5] of node:saguaro-22-0 HCA-1
Mar 20 14:26:49 013365 [41401960] -> osm_ucast_mgr_process: Min Hop Tables
configured on all switches
Mar 20 14:26:49 065887 [43C05960] -> SUBNET UP
Mar 20 14:26:49 407010 [44606960] -> osm_ucast_mgr_process: Min Hop Tables
configured on all switches
Mar 20 14:26:49 459477 [44606960] -> SUBNET UP
Mar 20 14:27:42 754098 [45007960] -> __osm_trap_rcv_process_request:
Received Generic Notice type:0x01 num:128 Producer:2 from LID:0x0148
TID:0x0000000000000070
Mar 20 14:27:42 754349 [45007960] -> osm_report_notice: Reporting Generic
Notice type:1 num:128 from LID:0x0148
GID:0xfe80000000000000,0x0005ad00000281b3
Mar 20 14:27:42 760115 [43C05960] -> __osm_trap_rcv_process_request:
Received Generic Notice type:0x01 num:128 Producer:2 from LID:0x0148
TID:0x0000000000000071
Mar 20 14:27:42 760178 [44606960] -> __osm_trap_rcv_process_request:
Received Generic Notice type:0x01 num:128 Producer:2 from LID:0x001B
TID:0x0000000000000082
Mar 20 14:27:42 760236 [43C05960] -> osm_report_notice: Reporting Generic
Notice type:1 num:128 from LID:0x0148
GID:0xfe80000000000000,0x0005ad00000281b3
Mar 20 14:27:42 760406 [44606960] -> osm_report_notice: Reporting Generic
Notice type:1 num:128 from LID:0x001B
GID:0xfe80000000000000,0x0005ad00000281a7
Mar 20 14:27:42 766931 [41E02960] -> __osm_trap_rcv_process_request:
Received Generic Notice type:0x01 num:128 Producer:2 from LID:0x001B
TID:0x0000000000000083
Mar 20 14:27:42 767049 [41E02960] -> osm_report_notice: Reporting Generic
Notice type:1 num:128 from LID:0x001B
GID:0xfe80000000000000,0x0005ad00000281a7
Mar 20 14:27:43 085327 [43C05960] -> osm_report_notice: Reporting Generic
Notice type:3 num:64 from LID:0x0092
GID:0xfe80000000000000,0x0005ad0000024bbb
Mar 20 14:27:43 085345 [43C05960] -> Discovered new port with
GUID:0x0005ad0000027c84 LID range [0x152,0x152] of node:Topspin Switch TS120
Mar 20 14:27:43 085349 [43C05960] -> osm_report_notice: Reporting Generic
Notice type:3 num:64 from LID:0x0092
GID:0xfe80000000000000,0x0005ad0000024bbb
Mar 20 14:27:43 085355 [43C05960] -> Discovered new port with
GUID:0x0005ad00000249d3 LID range [0xB1,0xB1] of node:saguaro-23-2 HCA-1
Mar 20 14:27:43 085359 [43C05960] -> osm_report_notice: Reporting Generic
Notice type:3 num:64 from LID:0x0092
GID:0xfe80000000000000,0x0005ad0000024bbb
Mar 20 14:27:43 085364 [43C05960] -> Discovered new port with
GUID:0x0005ad0000025043 LID range [0xB4,0xB4] of node:saguaro-23-5 HCA-1
Mar 20 14:27:43 085368 [43C05960] -> osm_report_notice: Reporting Generic
Notice type:3 num:64 from LID:0x0092
GID:0xfe80000000000000,0x0005ad0000024bbb
Mar 20 14:27:43 085373 [43C05960] -> Discovered new port with
GUID:0x0005ad000002510b LID range [0xB5,0xB5] of node:saguaro-23-6 HCA-1
Mar 20 14:27:43 085377 [43C05960] -> osm_report_notice: Reporting Generic
Notice type:3 num:64 from LID:0x0092
GID:0xfe80000000000000,0x0005ad0000024bbb
Mar 20 14:27:43 085382 [43C05960] -> Discovered new port with
GUID:0x0005ad0000024b27 LID range [0xAF,0xAF] of node:saguaro-23-0 HCA-1
Mar 20 14:27:43 085386 [43C05960] -> osm_report_notice: Reporting Generic
Notice type:3 num:64 from LID:0x0092
GID:0xfe80000000000000,0x0005ad0000024bbb
Mar 20 14:27:43 085390 [43C05960] -> Discovered new port with
GUID:0x0005ad0000024da7 LID range [0xB0,0xB0] of node:saguaro-23-1 HCA-1
Mar 20 14:27:43 085394 [43C05960] -> osm_report_notice: Reporting Generic
Notice type:3 num:64 from LID:0x0092
GID:0xfe80000000000000,0x0005ad0000024bbb
Mar 20 14:27:43 085399 [43C05960] -> Discovered new port with
GUID:0x0005ad0000024cbb LID range [0xB2,0xB2] of node:saguaro-23-3 HCA-1
Mar 20 14:27:43 085403 [43C05960] -> osm_report_notice: Reporting Generic
Notice type:3 num:64 from LID:0x0092
GID:0xfe80000000000000,0x0005ad0000024bbb
Mar 20 14:27:43 085407 [43C05960] -> Discovered new port with
GUID:0x0005ad0000024afb LID range [0xA5,0xA5] of node:saguaro-22-0 HCA-1
Mar 20 14:27:43 085411 [43C05960] -> osm_report_notice: Reporting Generic
Notice type:3 num:64 from LID:0x0092
GID:0xfe80000000000000,0x0005ad0000024bbb
Mar 20 14:27:43 085416 [43C05960] -> Discovered new port with
GUID:0x0005ad0000024e0b LID range [0xB3,0xB3] of node:saguaro-23-4 HCA-1
Mar 20 14:27:43 085420 [43C05960] -> osm_report_notice: Reporting Generic
Notice type:3 num:64 from LID:0x0092
GID:0xfe80000000000000,0x0005ad0000024bbb
Mar 20 14:27:43 085425 [43C05960] -> Discovered new port with
GUID:0x0005ad0000024d47 LID range [0xB6,0xB6] of node:saguaro-23-7 HCA-1
Mar 20 14:27:43 085428 [43C05960] -> osm_report_notice: Reporting Generic
Notice type:3 num:64 from LID:0x0092
GID:0xfe80000000000000,0x0005ad0000024bbb
Mar 20 14:27:43 085433 [43C05960] -> Discovered new port with
GUID:0x0005ad0000024d8b LID range [0xB7,0xB7] of node:saguaro-23-8 HCA-1
Mar 20 14:27:43 085437 [43C05960] -> osm_report_notice: Reporting Generic
Notice type:3 num:64 from LID:0x0092
GID:0xfe80000000000000,0x0005ad0000024bbb
Mar 20 14:27:43 085442 [43C05960] -> Discovered new port with
GUID:0x0005ad0000024d6b LID range [0xB8,0xB8] of node:saguaro-23-9 HCA-1
Mar 20 14:27:43 085446 [43C05960] -> osm_report_notice: Reporting Generic
Notice type:3 num:64 from LID:0x0092
GID:0xfe80000000000000,0x0005ad0000024bbb
Mar 20 14:27:43 085450 [43C05960] -> Discovered new port with
GUID:0x0005ad000002511b LID range [0xA6,0xA6] of node:saguaro-22-1 HCA-1
Mar 20 14:27:43 085454 [43C05960] -> osm_report_notice: Reporting Generic
Notice type:3 num:64 from LID:0x0092
GID:0xfe80000000000000,0x0005ad0000024bbb
Mar 20 14:27:43 085459 [43C05960] -> Discovered new port with
GUID:0x0005ad0000024c9b LID range [0xA7,0xA7] of node:saguaro-22-2 HCA-1
Mar 20 14:27:43 085511 [43C05960] -> osm_report_notice: Reporting Generic
Notice type:3 num:64 from LID:0x0092
GID:0xfe80000000000000,0x0005ad0000024bbb
Mar 20 14:27:43 085517 [43C05960] -> Discovered new port with
GUID:0x0005ad000002498f LID range [0xA8,0xA8] of node:saguaro-22-3 HCA-1
Mar 20 14:27:43 085521 [43C05960] -> osm_report_notice: Reporting Generic
Notice type:3 num:64 from LID:0x0092
GID:0xfe80000000000000,0x0005ad0000024bbb
Mar 20 14:27:43 085526 [43C05960] -> Discovered new port with
GUID:0x0005ad0000024feb LID range [0x153,0x153] of node:saguaro-22-5 HCA-1
Mar 20 14:27:43 085530 [43C05960] -> osm_report_notice: Reporting Generic
Notice type:3 num:64 from LID:0x0092
GID:0xfe80000000000000,0x0005ad0000024bbb
Mar 20 14:27:43 085534 [43C05960] -> Discovered new port with
GUID:0x0005ad0000024977 LID range [0xA9,0xA9] of node:saguaro-22-4 HCA-1
Mar 20 14:27:43 116308 [43C05960] -> osm_ucast_mgr_process: Min Hop Tables
configured on all switches
Mar 20 14:27:43 179935 [4780B960] -> __osm_sm_mad_ctrl_rcv_callback: ERR
3111: Error status = 0x1C00
Mar 20 14:27:43 179980 [4780B960] -> SMP dump:
base_ver................0x1
mgmt_class..............0x81
class_ver...............0x1
method..................0x81 (SubnGetResp)
D bit...................0x1
status..................0x1C00
hop_ptr.................0x0
hop_count...............0x4
trans_id................0x85669
attr_id.................0x15 (PortInfo)
resv....................0x0
attr_mod................0x13
m_key...................0x0000000000000000
dr_slid.................0xFFFF
dr_dlid.................0xFFFF
Initial path: [0][1][16][1][4]
Return path: [0][9][13][D][1]
Reserved: [0][0][0][0][0][0][0]
00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
00 00 00 00 00 00 00 00 00 00 00 00 01 03 03 02
14 52 00 11 40 40 00 08 08 04 F2 40 00 00 00 00
00 00 00 00 00 88 00 00 00 00 00 00 00 00 00 00
Mar 20 14:27:43 180019 [41401960] -> osm_pi_rcv_process_set: Received error
status 0x1c for SetResp() during ACTIVE transition
Mar 20 14:27:43 180037 [41401960] -> PortInfo dump:
port number.............0x13
node_guid...............0x0005ad00000281a7
port_guid...............0x0005ad00000281a7
m_key...................0x0000000000000000
subnet_prefix...........0x0000000000000000
base_lid................0x0
master_sm_base_lid......0x0
capability_mask.........0x0
diag_code...............0x0
m_key_lease_period......0x0
local_port_num..........0x1
link_width_enabled......0x3
link_width_supported....0x3
link_width_active.......0x2
link_speed_supported....0x1
port_state..............ACTIVE
state_info2.............0x52
m_key_protect_bits......0x0
lmc.....................0x0
link_speed..............0x11
mtu_smsl................0x40
vl_cap_init_type........0x40
vl_high_limit...........0x0
vl_arb_high_cap.........0x8
vl_arb_low_cap..........0x8
init_rep_mtu_cap........0x4
vl_stall_life...........0xF2
vl_enforce..............0x40
m_key_violations........0x0
p_key_violations........0x0
q_key_violations........0x0
guid_cap................0x0
client_reregister.......0x0
subnet_timeout..........0x0
resp_time_value.........0x0
error_threshold.........0x88
Mar 20 14:27:43 180050 [41401960] -> Capabilities Mask:
Mar 20 14:27:43 180092 [4780B960] -> __osm_sm_mad_ctrl_rcv_callback: ERR
3111: Error status = 0x1C00
Mar 20 14:27:43 180137 [4780B960] -> SMP dump:
base_ver................0x1
mgmt_class..............0x81
class_ver...............0x1
method..................0x81 (SubnGetResp)
D bit...................0x1
status..................0x1C00
hop_ptr.................0x0
hop_count...............0x4
trans_id................0x8566a
attr_id.................0x15 (PortInfo)
resv....................0x0
attr_mod................0x16
m_key...................0x0000000000000000
dr_slid.................0xFFFF
dr_dlid.................0xFFFF
Initial path: [0][1][16][1][4]
Return path: [0][9][13][D][1]
Reserved: [0][0][0][0][0][0][0]
00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
00 00 00 00 00 00 00 00 00 00 00 00 01 03 03 02
14 52 00 11 40 40 00 08 08 04 F2 40 00 00 00 00
00 00 00 00 00 88 00 00 00 00 00 00 00 00 00 00
Mar 20 14:27:43 180185 [44606960] -> osm_pi_rcv_process_set: Received error
status 0x1c for SetResp() during ACTIVE transition
Mar 20 14:27:43 180189 [44606960] -> PortInfo dump:
port number.............0x16
node_guid...............0x0005ad00000281a7
port_guid...............0x0005ad00000281a7
m_key...................0x0000000000000000
subnet_prefix...........0x0000000000000000
base_lid................0x0
master_sm_base_lid......0x0
capability_mask.........0x0
diag_code...............0x0
m_key_lease_period......0x0
local_port_num..........0x1
link_width_enabled......0x3
link_width_supported....0x3
link_width_active.......0x2
link_speed_supported....0x1
port_state..............ACTIVE
state_info2.............0x52
m_key_protect_bits......0x0
lmc.....................0x0
link_speed..............0x11
mtu_smsl................0x40
vl_cap_init_type........0x40
vl_high_limit...........0x0
vl_arb_high_cap.........0x8
vl_arb_low_cap..........0x8
init_rep_mtu_cap........0x4
vl_stall_life...........0xF2
vl_enforce..............0x40
m_key_violations........0x0
p_key_violations........0x0
q_key_violations........0x0
guid_cap................0x0
client_reregister.......0x0
subnet_timeout..........0x0
resp_time_value.........0x0
error_threshold.........0x88
Mar 20 14:27:43 180199 [44606960] -> Capabilities Mask:
Mar 20 14:27:43 180239 [4780B960] -> __osm_sm_mad_ctrl_rcv_callback: ERR
3111: Error status = 0x1C00
Mar 20 14:27:43 180263 [4780B960] -> SMP dump:
base_ver................0x1
mgmt_class..............0x81
class_ver...............0x1
method..................0x81 (SubnGetResp)
D bit...................0x1
status..................0x1C00
hop_ptr.................0x0
hop_count...............0x4
trans_id................0x8566b
attr_id.................0x15 (PortInfo)
resv....................0x0
attr_mod................0x17
m_key...................0x0000000000000000
dr_slid.................0xFFFF
dr_dlid.................0xFFFF
Initial path: [0][1][16][1][4]
Return path: [0][9][13][D][1]
Reserved: [0][0][0][0][0][0][0]
00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
00 00 00 00 00 00 00 00 00 00 00 00 01 03 03 02
14 52 00 11 40 40 00 08 08 04 F2 40 00 00 00 00
00 00 00 00 00 88 00 00 00 00 00 00 00 00 00 00
Mar 20 14:27:43 180307 [4780B960] -> __osm_sm_mad_ctrl_rcv_callback: ERR
3111: Error status = 0x1C00
Mar 20 14:27:43 180319 [42803960] -> osm_pi_rcv_process_set: Received error
status 0x1c for SetResp() during ACTIVE transition
Mar 20 14:27:43 180332 [4780B960] -> SMP dump:
base_ver................0x1
mgmt_class..............0x81
class_ver...............0x1
method..................0x81 (SubnGetResp)
D bit...................0x1
status..................0x1C00
hop_ptr.................0x0
hop_count...............0x4
trans_id................0x8566c
attr_id.................0x15 (PortInfo)
resv....................0x0
attr_mod................0x18
m_key...................0x0000000000000000
dr_slid.................0xFFFF
dr_dlid.................0xFFFF
Initial path: [0][1][16][1][4]
Return path: [0][9][13][D][1]
Reserved: [0][0][0][0][0][0][0]
00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
00 00 00 00 00 00 00 00 00 00 00 00 01 03 03 02
14 52 00 11 40 40 00 08 08 04 F2 40 00 00 00 00
00 00 00 00 00 88 00 00 00 00 00 00 00 00 00 00
Mar 20 14:27:43 180336 [42803960] -> PortInfo dump:
port number.............0x17
node_guid...............0x0005ad00000281a7
port_guid...............0x0005ad00000281a7
m_key...................0x0000000000000000
subnet_prefix...........0x0000000000000000
base_lid................0x0
master_sm_base_lid......0x0
capability_mask.........0x0
diag_code...............0x0
m_key_lease_period......0x0
local_port_num..........0x1
link_width_enabled......0x3
link_width_supported....0x3
link_width_active.......0x2
link_speed_supported....0x1
port_state..............ACTIVE
state_info2.............0x52
m_key_protect_bits......0x0
lmc.....................0x0
link_speed..............0x11
mtu_smsl................0x40
vl_cap_init_type........0x40
vl_high_limit...........0x0
vl_arb_high_cap.........0x8
vl_arb_low_cap..........0x8
init_rep_mtu_cap........0x4
vl_stall_life...........0xF2
vl_enforce..............0x40
m_key_violations........0x0
p_key_violations........0x0
q_key_violations........0x0
guid_cap................0x0
client_reregister.......0x0
subnet_timeout..........0x0
resp_time_value.........0x0
error_threshold.........0x88
Mar 20 14:27:43 180364 [42803960] -> Capabilities Mask:
Mar 20 14:27:43 180389 [42803960] -> osm_pi_rcv_process_set: Received error
status 0x1c for SetResp() during ACTIVE transition
Mar 20 14:27:43 180410 [4780B960] -> __osm_sm_mad_ctrl_rcv_callback: ERR
3111: Error status = 0x1C00
Mar 20 14:27:43 180392 [42803960] -> PortInfo dump:
port number.............0x18
node_guid...............0x0005ad00000281a7
port_guid...............0x0005ad00000281a7
m_key...................0x0000000000000000
subnet_prefix...........0x0000000000000000
base_lid................0x0
master_sm_base_lid......0x0
capability_mask.........0x0
diag_code...............0x0
m_key_lease_period......0x0
local_port_num..........0x1
link_width_enabled......0x3
link_width_supported....0x3
link_width_active.......0x2
link_speed_supported....0x1
port_state..............ACTIVE
state_info2.............0x52
m_key_protect_bits......0x0
lmc.....................0x0
link_speed..............0x11
mtu_smsl................0x40
vl_cap_init_type........0x40
vl_high_limit...........0x0
vl_arb_high_cap.........0x8
vl_arb_low_cap..........0x8
init_rep_mtu_cap........0x4
vl_stall_life...........0xF2
vl_enforce..............0x40
m_key_violations........0x0
p_key_violations........0x0
q_key_violations........0x0
guid_cap................0x0
client_reregister.......0x0
subnet_timeout..........0x0
resp_time_value.........0x0
error_threshold.........0x88
Mar 20 14:27:43 180415 [42803960] -> Capabilities Mask:
Mar 20 14:27:43 180436 [4780B960] -> SMP dump:
base_ver................0x1
mgmt_class..............0x81
class_ver...............0x1
method..................0x81 (SubnGetResp)
D bit...................0x1
status..................0x1C00
hop_ptr.................0x0
hop_count...............0x4
trans_id................0x8566d
attr_id.................0x15 (PortInfo)
resv....................0x0
attr_mod................0x16
m_key...................0x0000000000000000
dr_slid.................0xFFFF
dr_dlid.................0xFFFF
Initial path: [0][1][11][2][5]
Return path: [0][9][18][E][2]
Reserved: [0][0][0][0][0][0][0]
00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
00 00 00 00 00 00 00 00 00 00 00 00 02 03 03 02
14 52 00 11 40 40 00 08 08 04 F2 40 00 00 00 00
00 00 00 00 00 88 00 00 00 00 00 00 00 00 00 00
Mar 20 14:27:43 180490 [41E02960] -> osm_pi_rcv_process_set: Received error
status 0x1c for SetResp() during ACTIVE transition
Mar 20 14:27:43 180494 [41E02960] -> PortInfo dump:
port number.............0x16
node_guid...............0x0005ad00000281b3
port_guid...............0x0005ad00000281b3
m_key...................0x0000000000000000
subnet_prefix...........0x0000000000000000
base_lid................0x0
master_sm_base_lid......0x0
capability_mask.........0x0
diag_code...............0x0
m_key_lease_period......0x0
local_port_num..........0x2
link_width_enabled......0x3
link_width_supported....0x3
link_width_active.......0x2
link_speed_supported....0x1
port_state..............ACTIVE
state_info2.............0x52
m_key_protect_bits......0x0
lmc.....................0x0
link_speed..............0x11
mtu_smsl................0x40
vl_cap_init_type........0x40
vl_high_limit...........0x0
vl_arb_high_cap.........0x8
vl_arb_low_cap..........0x8
init_rep_mtu_cap........0x4
vl_stall_life...........0xF2
vl_enforce..............0x40
m_key_violations........0x0
p_key_violations........0x0
q_key_violations........0x0
guid_cap................0x0
client_reregister.......0x0
subnet_timeout..........0x0
resp_time_value.........0x0
error_threshold.........0x88
Mar 20 14:27:43 180504 [41E02960] -> Capabilities Mask:
Mar 20 14:27:43 180536 [4780B960] -> __osm_sm_mad_ctrl_rcv_callback: ERR
3111: Error status = 0x1C00
Mar 20 14:27:43 180560 [4780B960] -> SMP dump:
base_ver................0x1
mgmt_class..............0x81
class_ver...............0x1
method..................0x81 (SubnGetResp)
D bit...................0x1
status..................0x1C00
hop_ptr.................0x0
hop_count...............0x4
trans_id................0x8566e
attr_id.................0x15 (PortInfo)
resv....................0x0
attr_mod................0x17
m_key...................0x0000000000000000
dr_slid.................0xFFFF
dr_dlid.................0xFFFF
Initial path: [0][1][11][2][5]
Return path: [0][9][18][E][2]
Reserved: [0][0][0][0][0][0][0]
00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
00 00 00 00 00 00 00 00 00 00 00 00 02 03 03 02
14 52 00 11 40 40 00 08 08 04 F2 40 00 00 00 00
00 00 00 00 00 88 00 00 00 00 00 00 00 00 00 00
Mar 20 14:27:43 180606 [45007960] -> osm_pi_rcv_process_set: Received error
status 0x1c for SetResp() during ACTIVE transition
Mar 20 14:27:43 180615 [4780B960] -> __osm_sm_mad_ctrl_rcv_callback: ERR
3111: Error status = 0x1C00
Mar 20 14:27:43 180634 [45007960] -> PortInfo dump:
port number.............0x17
node_guid...............0x0005ad00000281b3
port_guid...............0x0005ad00000281b3
m_key...................0x0000000000000000
subnet_prefix...........0x0000000000000000
base_lid................0x0
master_sm_base_lid......0x0
capability_mask.........0x0
diag_code...............0x0
m_key_lease_period......0x0
local_port_num..........0x2
link_width_enabled......0x3
link_width_supported....0x3
link_width_active.......0x2
link_speed_supported....0x1
port_state..............ACTIVE
state_info2.............0x52
m_key_protect_bits......0x0
lmc.....................0x0
link_speed..............0x11
mtu_smsl................0x40
vl_cap_init_type........0x40
vl_high_limit...........0x0
vl_arb_high_cap.........0x8
vl_arb_low_cap..........0x8
init_rep_mtu_cap........0x4
vl_stall_life...........0xF2
vl_enforce..............0x40
m_key_violations........0x0
p_key_violations........0x0
q_key_violations........0x0
guid_cap................0x0
client_reregister.......0x0
subnet_timeout..........0x0
resp_time_value.........0x0
error_threshold.........0x88
Mar 20 14:27:43 180657 [45007960] -> Capabilities Mask:
Mar 20 14:27:43 180678 [4780B960] -> SMP dump:
base_ver................0x1
mgmt_class..............0x81
class_ver...............0x1
method..................0x81 (SubnGetResp)
D bit...................0x1
status..................0x1C00
hop_ptr.................0x0
hop_count...............0x4
trans_id................0x8566f
attr_id.................0x15 (PortInfo)
resv....................0x0
attr_mod................0x18
m_key...................0x0000000000000000
dr_slid.................0xFFFF
dr_dlid.................0xFFFF
Initial path: [0][1][11][2][5]
Return path: [0][9][18][E][2]
Reserved: [0][0][0][0][0][0][0]
00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
00 00 00 00 00 00 00 00 00 00 00 00 02 03 03 02
14 52 00 11 40 40 00 08 08 04 F2 40 00 00 00 00
00 00 00 00 00 88 00 00 00 00 00 00 00 00 00 00
Mar 20 14:27:43 180769 [43C05960] -> osm_pi_rcv_process_set: Received error
status 0x1c for SetResp() during ACTIVE transition
Mar 20 14:27:43 180775 [43C05960] -> PortInfo dump:
port number.............0x18
node_guid...............0x0005ad00000281b3
port_guid...............0x0005ad00000281b3
m_key...................0x0000000000000000
subnet_prefix...........0x0000000000000000
base_lid................0x0
master_sm_base_lid......0x0
capability_mask.........0x0
diag_code...............0x0
m_key_lease_period......0x0
local_port_num..........0x2
link_width_enabled......0x3
link_width_supported....0x3
link_width_active.......0x2
link_speed_supported....0x1
port_state..............ACTIVE
state_info2.............0x52
m_key_protect_bits......0x0
lmc.....................0x0
link_speed..............0x11
mtu_smsl................0x40
vl_cap_init_type........0x40
vl_high_limit...........0x0
vl_arb_high_cap.........0x8
vl_arb_low_cap..........0x8
init_rep_mtu_cap........0x4
vl_stall_life...........0xF2
vl_enforce..............0x40
m_key_violations........0x0
p_key_violations........0x0
q_key_violations........0x0
guid_cap................0x0
client_reregister.......0x0
subnet_timeout..........0x0
resp_time_value.........0x0
error_threshold.........0x88
Mar 20 14:27:43 180789 [43C05960] -> Capabilities Mask:
Mar 20 14:27:43 186228 [43204960] -> SUBNET UP
Mar 20 14:27:43 557268 [45A08960] -> osm_ucast_mgr_process: Min Hop Tables
configured on all switches
Mar 20 14:27:43 611082 [45A08960] -> SUBNET UP
Mar 20 14:27:58 852744 [45007960] -> __osm_trap_rcv_process_request:
Received Generic Notice type:0x01 num:128 Producer:2 from LID:0x0152
TID:0x0000000000000000
Mar 20 14:27:58 852982 [45007960] -> osm_report_notice: Reporting Generic
Notice type:1 num:128 from LID:0x0152
GID:0xfe80000000000000,0x0005ad0000027c84
Mar 20 14:27:58 970772 [43204960] -> __osm_trap_rcv_process_request:
Received Generic Notice type:0x01 num:128 Producer:2 from LID:0x0152
TID:0x0000000000000001
Mar 20 14:27:58 970864 [43204960] -> osm_report_notice: Reporting Generic
Notice type:1 num:128 from LID:0x0152
GID:0xfe80000000000000,0x0005ad0000027c84
Mar 20 14:27:58 992628 [41E02960] -> __osm_trap_rcv_process_request:
Received Generic Notice type:0x01 num:128 Producer:2 from LID:0x0152
TID:0x0000000000000002
Mar 20 14:27:58 992712 [41E02960] -> osm_report_notice: Reporting Generic
Notice type:1 num:128 from LID:0x0152
GID:0xfe80000000000000,0x0005ad0000027c84
Mar 20 14:27:59 132331 [42803960] -> __osm_trap_rcv_process_request:
Received Generic Notice type:0x01 num:128 Producer:2 from LID:0x0152
TID:0x0000000000000003
Mar 20 14:27:59 132484 [42803960] -> osm_report_notice: Reporting Generic
Notice type:1 num:128 from LID:0x0152
GID:0xfe80000000000000,0x0005ad0000027c84
Mar 20 14:27:59 314893 [41E02960] -> __osm_trap_rcv_process_request:
Received Generic Notice type:0x01 num:128 Producer:2 from LID:0x0152
TID:0x0000000000000004
Mar 20 14:27:59 315006 [41E02960] -> osm_report_notice: Reporting Generic
Notice type:1 num:128 from LID:0x0152
GID:0xfe80000000000000,0x0005ad0000027c84
Mar 20 14:27:59 343241 [42803960] -> __osm_trap_rcv_process_request:
Received Generic Notice type:0x01 num:128 Producer:2 from LID:0x0152
TID:0x0000000000000005
Mar 20 14:27:59 343320 [42803960] -> osm_report_notice: Reporting Generic
Notice type:1 num:128 from LID:0x0152
GID:0xfe80000000000000,0x0005ad0000027c84
Mar 20 14:27:59 481698 [45007960] -> __osm_trap_rcv_process_request:
Received Generic Notice type:0x01 num:128 Producer:2 from LID:0x0152
TID:0x0000000000000006
Mar 20 14:27:59 481775 [45007960] -> osm_report_notice: Reporting Generic
Notice type:1 num:128 from LID:0x0152
GID:0xfe80000000000000,0x0005ad0000027c84
Mar 20 14:27:59 512746 [45A08960] -> __osm_trap_rcv_process_request:
Received Generic Notice type:0x01 num:128 Producer:2 from LID:0x0152
TID:0x0000000000000007
Mar 20 14:27:59 512853 [45A08960] -> osm_report_notice: Reporting Generic
Notice type:1 num:128 from LID:0x0152
GID:0xfe80000000000000,0x0005ad0000027c84
Mar 20 14:27:59 548851 [41E02960] -> osm_report_notice: Reporting Generic
Notice type:3 num:65 from LID:0x0092
GID:0xfe80000000000000,0x0005ad0000024bbb
Mar 20 14:27:59 548861 [41E02960] -> Removed port with
GUID:0x0005ad00000249d3 LID range [0xB1,0xB1] of node:saguaro-23-2 HCA-1
Mar 20 14:27:59 583414 [41E02960] -> osm_ucast_mgr_process: Min Hop Tables
configured on all switches
Mar 20 14:27:59 583817 [43C05960] -> __osm_trap_rcv_process_request:
Received Generic Notice type:0x01 num:128 Producer:2 from LID:0x0152
TID:0x0000000000000008
Mar 20 14:27:59 623971 [43C05960] -> osm_report_notice: Reporting Generic
Notice type:1 num:128 from LID:0x0152
GID:0xfe80000000000000,0x0005ad0000027c84
Mar 20 14:27:59 626182 [42803960] -> __osm_trap_rcv_process_request:
Received Generic Notice type:0x01 num:128 Producer:2 from LID:0x0152
TID:0x0000000000000009
Mar 20 14:27:59 626329 [42803960] -> osm_report_notice: Reporting Generic
Notice type:1 num:128 from LID:0x0152
GID:0xfe80000000000000,0x0005ad0000027c84
Mar 20 14:27:59 634080 [41E02960] -> __osm_trap_rcv_process_request:
Received Generic Notice type:0x01 num:128 Producer:2 from LID:0x0152
TID:0x000000000000000a
Mar 20 14:27:59 634442 [41E02960] -> osm_report_notice: Reporting Generic
Notice type:1 num:128 from LID:0x0152
GID:0xfe80000000000000,0x0005ad0000027c84
Mar 20 14:27:59 641962 [45A08960] -> SUBNET UP
Mar 20 14:27:59 656231 [41401960] -> __osm_trap_rcv_process_request:
Received Generic Notice type:0x01 num:128 Producer:2 from LID:0x0152
TID:0x000000000000000b
Mar 20 14:27:59 656307 [41401960] -> __osm_trap_rcv_process_request: ERR
3804: Received trap 11 times consecutively
Mar 20 14:27:59 689788 [41E02960] -> __osm_trap_rcv_process_request:
Received Generic Notice type:0x01 num:128 Producer:2 from LID:0x0152
TID:0x000000000000000c
Mar 20 14:27:59 690249 [41E02960] -> __osm_trap_rcv_process_request: ERR
3804: Received trap 12 times consecutively
Mar 20 14:27:59 758521 [42803960] -> __osm_trap_rcv_process_request:
Received Generic Notice type:0x01 num:128 Producer:2 from LID:0x0152
TID:0x000000000000000d
Mar 20 14:27:59 758646 [42803960] -> __osm_trap_rcv_process_request: ERR
3804: Received trap 13 times consecutively
Mar 20 14:27:59 970740 [43204960] -> __osm_trap_rcv_process_request:
Received Generic Notice type:0x01 num:128 Producer:2 from LID:0x0152
TID:0x000000000000000e
Mar 20 14:27:59 970812 [43204960] -> __osm_trap_rcv_process_request: ERR
3804: Received trap 14 times consecutively
Mar 20 14:27:59 985557 [41E02960] -> osm_report_notice: Reporting Generic
Notice type:3 num:65 from LID:0x0092
GID:0xfe80000000000000,0x0005ad0000024bbb
Mar 20 14:27:59 985577 [41E02960] -> Removed port with
GUID:0x0005ad000002510b LID range [0xB5,0xB5] of node:saguaro-23-6 HCA-1
Mar 20 14:27:59 985601 [41E02960] -> osm_report_notice: Reporting Generic
Notice type:3 num:65 from LID:0x0092
GID:0xfe80000000000000,0x0005ad0000024bbb
Mar 20 14:27:59 985615 [41E02960] -> Removed port with
GUID:0x0005ad0000024d47 LID range [0xB6,0xB6] of node:saguaro-23-7 HCA-1
Mar 20 14:27:59 985649 [41E02960] -> osm_report_notice: Reporting Generic
Notice type:3 num:65 from LID:0x0092
GID:0xfe80000000000000,0x0005ad0000024bbb
Mar 20 14:27:59 985656 [41E02960] -> Removed port with
GUID:0x0005ad0000024d8b LID range [0xB7,0xB7] of node:saguaro-23-8 HCA-1
Mar 20 14:27:59 989767 [42803960] -> osm_report_notice: Reporting Generic
Notice type:3 num:64 from LID:0x0092
GID:0xfe80000000000000,0x0005ad0000024bbb
Mar 20 14:27:59 989787 [42803960] -> Discovered new port with
GUID:0x0005ad00000249d3 LID range [0xB1,0xB1] of node:saguaro-23-2 HCA-1
Mar 20 14:28:00 014445 [43C05960] -> __osm_trap_rcv_process_request:
Received Generic Notice type:0x01 num:128 Producer:2 from LID:0x0152
TID:0x000000000000000f
Mar 20 14:28:00 014524 [43C05960] -> __osm_trap_rcv_process_request: ERR
3804: Received trap 15 times consecutively
Mar 20 14:28:00 020896 [42803960] -> osm_ucast_mgr_process: Min Hop Tables
configured on all switches
Mar 20 14:28:00 086824 [43204960] -> SUBNET UP
Mar 20 14:28:00 124057 [45007960] -> __osm_trap_rcv_process_request:
Received Generic Notice type:0x01 num:128 Producer:2 from LID:0x0152
TID:0x0000000000000010
Mar 20 14:28:00 124108 [45007960] -> __osm_trap_rcv_process_request: ERR
3804: Received trap 16 times consecutively
Mar 20 14:28:00 131596 [41401960] -> __osm_trap_rcv_process_request:
Received Generic Notice type:0x01 num:128 Producer:2 from LID:0x0152
TID:0x0000000000000011
Mar 20 14:28:00 131643 [41401960] -> __osm_trap_rcv_process_request: ERR
3804: Received trap 17 times consecutively
Mar 20 14:28:00 412484 [43C05960] -> __osm_trap_rcv_process_request:
Received Generic Notice type:0x01 num:128 Producer:2 from LID:0x0152
TID:0x0000000000000012
Mar 20 14:28:00 412528 [43C05960] -> __osm_trap_rcv_process_request: ERR
3804: Received trap 18 times consecutively
Mar 20 14:28:00 436877 [44606960] -> __osm_trap_rcv_process_request:
Received Generic Notice type:0x01 num:128 Producer:2 from LID:0x0152
TID:0x0000000000000013
Mar 20 14:28:00 436921 [44606960] -> __osm_trap_rcv_process_request: ERR
3804: Received trap 19 times consecutively
Mar 20 14:28:00 458745 [42803960] -> __osm_trap_rcv_process_request:
Received Generic Notice type:0x01 num:128 Producer:2 from LID:0x0152
TID:0x0000000000000014
Mar 20 14:28:00 458816 [42803960] -> __osm_trap_rcv_process_request: ERR
3804: Received trap 20 times consecutively
Mar 20 14:28:00 480551 [41E02960] -> __osm_trap_rcv_process_request:
Received Generic Notice type:0x01 num:128 Producer:2 from LID:0x0152
TID:0x0000000000000015
Mar 20 14:28:00 480599 [41E02960] -> __osm_trap_rcv_process_request: ERR
3804: Received trap 21 times consecutively
Mar 20 14:28:00 695340 [45A08960] -> __osm_trap_rcv_process_request:
Received Generic Notice type:0x01 num:128 Producer:2 from LID:0x0152
TID:0x0000000000000016
Mar 20 14:28:00 695386 [45A08960] -> __osm_trap_rcv_process_request: ERR
3804: Received trap 22 times consecutively
Mar 20 14:28:00 695726 [43204960] -> __osm_trap_rcv_process_request:
Received Generic Notice type:0x01 num:128 Producer:2 from LID:0x0148
TID:0x0000000000000072
Mar 20 14:28:00 695886 [43204960] -> osm_report_notice: Reporting Generic
Notice type:1 num:128 from LID:0x0148
GID:0xfe80000000000000,0x0005ad00000281b3
Mar 20 14:28:00 719764 [41401960] -> __osm_trap_rcv_process_request:
Received Generic Notice type:0x01 num:128 Producer:2 from LID:0x0152
TID:0x0000000000000017
Mar 20 14:28:00 719825 [41401960] -> __osm_trap_rcv_process_request: ERR
3804: Received trap 23 times consecutively
Mar 20 14:28:00 743680 [43204960] -> __osm_trap_rcv_process_request:
Received Generic Notice type:0x01 num:128 Producer:2 from LID:0x0152
TID:0x0000000000000018
Mar 20 14:28:00 743775 [43204960] -> __osm_trap_rcv_process_request: ERR
3804: Received trap 24 times consecutively
Mar 20 14:28:00 763599 [45007960] -> __osm_trap_rcv_process_request:
Received Generic Notice type:0x01 num:128 Producer:2 from LID:0x0152
TID:0x0000000000000019
Mar 20 14:28:00 763654 [45007960] -> __osm_trap_rcv_process_request: ERR
3804: Received trap 25 times consecutively
Mar 20 14:28:00 813393 [43C05960] -> __osm_trap_rcv_process_request:
Received Generic Notice type:0x01 num:128 Producer:2 from LID:0x0152
TID:0x000000000000001a
Mar 20 14:28:00 813473 [43C05960] -> __osm_trap_rcv_process_request: ERR
3804: Received trap 26 times consecutively
Mar 20 14:28:00 831287 [45A08960] -> __osm_trap_rcv_process_request:
Received Generic Notice type:0x01 num:128 Producer:2 from LID:0x0152
TID:0x000000000000001b
Mar 20 14:28:00 831302 [44606960] -> __osm_trap_rcv_process_request:
Received Generic Notice type:0x01 num:128 Producer:2 from LID:0x0148
TID:0x0000000000000073
Mar 20 14:28:00 831383 [45A08960] -> __osm_trap_rcv_process_request: ERR
3804: Received trap 27 times consecutively
Mar 20 14:28:00 831424 [44606960] -> osm_report_notice: Reporting Generic
Notice type:1 num:128 from LID:0x0148
GID:0xfe80000000000000,0x0005ad00000281b3
Mar 20 14:28:00 841593 [41E02960] -> __osm_trap_rcv_process_request:
Received Generic Notice type:0x01 num:128 Producer:2 from LID:0x0152
TID:0x000000000000001c
Mar 20 14:28:00 841644 [41E02960] -> __osm_trap_rcv_process_request: ERR
3804: Received trap 28 times consecutively
Mar 20 14:28:01 050511 [41E02960] -> osm_report_notice: Reporting Generic
Notice type:3 num:64 from LID:0x0092
GID:0xfe80000000000000,0x0005ad0000024bbb
Mar 20 14:28:01 050529 [41E02960] -> Discovered new port with
GUID:0x0005ad000002510b LID range [0xB5,0xB5] of node:saguaro-23-6 HCA-1
Mar 20 14:28:01 050535 [41E02960] -> osm_report_notice: Reporting Generic
Notice type:3 num:64 from LID:0x0092
GID:0xfe80000000000000,0x0005ad0000024bbb
Mar 20 14:28:01 050542 [41E02960] -> Discovered new port with
GUID:0x0005ad0000024d47 LID range [0xB6,0xB6] of node:saguaro-23-7 HCA-1
Mar 20 14:28:01 050547 [41E02960] -> osm_report_notice: Reporting Generic
Notice type:3 num:64 from LID:0x0092
GID:0xfe80000000000000,0x0005ad0000024bbb
Mar 20 14:28:01 050554 [41E02960] -> Discovered new port with
GUID:0x0005ad0000024d8b LID range [0xB7,0xB7] of node:saguaro-23-8 HCA-1
Mar 20 14:28:01 081322 [41E02960] -> osm_ucast_mgr_process: Min Hop Tables
configured on all switches
Mar 20 14:28:01 142873 [43204960] -> SUBNET UP
Mar 20 14:28:01 460275 [44606960] -> __osm_trap_rcv_process_request:
Received Generic Notice type:0x01 num:128 Producer:2 from LID:0x0152
TID:0x000000000000001d
Mar 20 14:28:01 460358 [44606960] -> __osm_trap_rcv_process_request: ERR
3804: Received trap 29 times consecutively
Mar 20 14:28:01 488474 [45007960] -> osm_ucast_mgr_process: Min Hop Tables
configured on all switches
Mar 20 14:28:01 538634 [4780B960] -> __osm_sm_mad_ctrl_rcv_callback: ERR
3111: Error status = 0x1C00
Mar 20 14:28:01 538712 [4780B960] -> SMP dump:
base_ver................0x1
mgmt_class..............0x81
class_ver...............0x1
method..................0x81 (SubnGetResp)
D bit...................0x1
status..................0x1C00
hop_ptr.................0x0
hop_count...............0x4
trans_id................0x898d1
attr_id.................0x15 (PortInfo)
resv....................0x0
attr_mod................0x16
m_key...................0x0000000000000000
dr_slid.................0xFFFF
dr_dlid.................0xFFFF
Initial path: [0][1][11][1][6]
Return path: [0][9][18][D][3]
Reserved: [0][0][0][0][0][0][0]
00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
00 00 00 00 00 00 00 00 00 00 00 00 03 03 03 02
11 42 00 11 40 40 00 08 08 04 F2 40 00 00 00 00
00 00 00 00 00 88 00 00 00 00 00 00 00 00 00 00
Mar 20 14:28:01 538752 [42803960] -> osm_pi_rcv_process_set: ERR 0F10:
Received error status for SetResp()
Mar 20 14:28:01 538758 [4780B960] -> __osm_sm_mad_ctrl_rcv_callback: ERR
3111: Error status = 0x1C00
Mar 20 14:28:01 538767 [42803960] -> PortInfo dump:
port number.............0x16
node_guid...............0x0005ad00000281b3
port_guid...............0x0005ad00000281b3
m_key...................0x0000000000000000
subnet_prefix...........0x0000000000000000
base_lid................0x0
master_sm_base_lid......0x0
capability_mask.........0x0
diag_code...............0x0
m_key_lease_period......0x0
local_port_num..........0x3
link_width_enabled......0x3
link_width_supported....0x3
link_width_active.......0x2
link_speed_supported....0x1
port_state..............DOWN
state_info2.............0x42
m_key_protect_bits......0x0
lmc.....................0x0
link_speed..............0x11
mtu_smsl................0x40
vl_cap_init_type........0x40
vl_high_limit...........0x0
vl_arb_high_cap.........0x8
vl_arb_low_cap..........0x8
init_rep_mtu_cap........0x4
vl_stall_life...........0xF2
vl_enforce..............0x40
m_key_violations........0x0
p_key_violations........0x0
q_key_violations........0x0
guid_cap................0x0
client_reregister.......0x0
subnet_timeout..........0x0
resp_time_value.........0x0
error_threshold.........0x88
Mar 20 14:28:01 538795 [4780B960] -> SMP dump:
base_ver................0x1
mgmt_class..............0x81
class_ver...............0x1
method..................0x81 (SubnGetResp)
D bit...................0x1
status..................0x1C00
hop_ptr.................0x0
hop_count...............0x4
trans_id................0x898d2
attr_id.................0x15 (PortInfo)
resv....................0x0
attr_mod................0x17
m_key...................0x0000000000000000
dr_slid.................0xFFFF
dr_dlid.................0xFFFF
Initial path: [0][1][11][1][6]
Return path: [0][9][18][D][3]
Reserved: [0][0][0][0][0][0][0]
00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
00 00 00 00 00 00 00 00 00 00 00 00 03 03 03 02
11 42 00 11 40 40 00 08 08 04 F2 40 00 00 00 00
00 00 00 00 00 88 00 00 00 00 00 00 00 00 00 00
Mar 20 14:28:01 538810 [42803960] -> Capabilities Mask:
Mar 20 14:28:01 538849 [42803960] -> osm_pi_rcv_process_set: ERR 0F10:
Received error status for SetResp()
Mar 20 14:28:01 538856 [42803960] -> PortInfo dump:
port number.............0x17
node_guid...............0x0005ad00000281b3
port_guid...............0x0005ad00000281b3
m_key...................0x0000000000000000
subnet_prefix...........0x0000000000000000
base_lid................0x0
master_sm_base_lid......0x0
capability_mask.........0x0
diag_code...............0x0
m_key_lease_period......0x0
local_port_num..........0x3
link_width_enabled......0x3
link_width_supported....0x3
link_width_active.......0x2
link_speed_supported....0x1
port_state..............DOWN
state_info2.............0x42
m_key_protect_bits......0x0
lmc.....................0x0
link_speed..............0x11
mtu_smsl................0x40
vl_cap_init_type........0x40
vl_high_limit...........0x0
vl_arb_high_cap.........0x8
vl_arb_low_cap..........0x8
init_rep_mtu_cap........0x4
vl_stall_life...........0xF2
vl_enforce..............0x40
m_key_violations........0x0
p_key_violations........0x0
q_key_violations........0x0
guid_cap................0x0
client_reregister.......0x0
subnet_timeout..........0x0
resp_time_value.........0x0
error_threshold.........0x88
Mar 20 14:28:01 538871 [42803960] -> Capabilities Mask:
Mar 20 14:28:01 539658 [4780B960] -> __osm_sm_mad_ctrl_rcv_callback: ERR
3111: Error status = 0x1C00
Mar 20 14:28:01 539696 [4780B960] -> SMP dump:
base_ver................0x1
mgmt_class..............0x81
class_ver...............0x1
method..................0x81 (SubnGetResp)
D bit...................0x1
status..................0x1C00
hop_ptr.................0x0
hop_count...............0x5
trans_id................0x898d3
attr_id.................0x15 (PortInfo)
resv....................0x0
attr_mod................0x11
m_key...................0x0000000000000000
dr_slid.................0xFFFF
dr_dlid.................0xFFFF
Initial path: [0][1][15][1][4][17]
Return path: [0][9][18][D][1][16]
Reserved: [0][0][0][0][0][0][0]
00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
00 00 00 00 00 00 00 00 00 00 00 00 16 02 03 02
11 42 00 11 40 40 00 08 08 04 F2 40 00 00 00 00
00 00 00 00 00 88 00 00 00 00 00 00 00 00 00 00
Mar 20 14:28:01 539778 [45A08960] -> osm_pi_rcv_process_set: Received error
status 0x1c for SetResp() during ACTIVE transition
Mar 20 14:28:01 539784 [4780B960] -> __osm_sm_mad_ctrl_rcv_callback: ERR
3111: Error status = 0x1C00
Mar 20 14:28:01 539798 [45A08960] -> PortInfo dump:
port number.............0x11
node_guid...............0x0005ad0000027c84
port_guid...............0x0005ad0000027c84
m_key...................0x0000000000000000
subnet_prefix...........0x0000000000000000
base_lid................0x0
master_sm_base_lid......0x0
capability_mask.........0x0
diag_code...............0x0
m_key_lease_period......0x0
local_port_num..........0x16
link_width_enabled......0x2
link_width_supported....0x3
link_width_active.......0x2
link_speed_supported....0x1
port_state..............DOWN
state_info2.............0x42
m_key_protect_bits......0x0
lmc.....................0x0
link_speed..............0x11
mtu_smsl................0x40
vl_cap_init_type........0x40
vl_high_limit...........0x0
vl_arb_high_cap.........0x8
vl_arb_low_cap..........0x8
init_rep_mtu_cap........0x4
vl_stall_life...........0xF2
vl_enforce..............0x40
m_key_violations........0x0
p_key_violations........0x0
q_key_violations........0x0
guid_cap................0x0
client_reregister.......0x0
subnet_timeout..........0x0
resp_time_value.........0x0
error_threshold.........0x88
Mar 20 14:28:01 539834 [45A08960] -> Capabilities Mask:
Mar 20 14:28:01 539844 [4780B960] -> SMP dump:
base_ver................0x1
mgmt_class..............0x81
class_ver...............0x1
method..................0x81 (SubnGetResp)
D bit...................0x1
status..................0x1C00
hop_ptr.................0x0
hop_count...............0x5
trans_id................0x898d4
attr_id.................0x15 (PortInfo)
resv....................0x0
attr_mod................0x12
m_key...................0x0000000000000000
dr_slid.................0xFFFF
dr_dlid.................0xFFFF
Initial path: [0][1][15][1][4][17]
Return path: [0][9][18][D][1][16]
Reserved: [0][0][0][0][0][0][0]
00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
00 00 00 00 00 00 00 00 00 00 00 00 16 02 03 02
11 42 00 11 40 40 00 08 08 04 F2 40 00 00 00 00
00 00 00 00 00 88 00 00 00 00 00 00 00 00 00 00
Mar 20 14:28:01 539903 [45007960] -> osm_pi_rcv_process_set: Received error
status 0x1c for SetResp() during ACTIVE transition
Mar 20 14:28:01 539908 [45007960] -> PortInfo dump:
port number.............0x12
node_guid...............0x0005ad0000027c84
port_guid...............0x0005ad0000027c84
m_key...................0x0000000000000000
subnet_prefix...........0x0000000000000000
base_lid................0x0
master_sm_base_lid......0x0
capability_mask.........0x0
diag_code...............0x0
m_key_lease_period......0x0
local_port_num..........0x16
link_width_enabled......0x2
link_width_supported....0x3
link_width_active.......0x2
link_speed_supported....0x1
port_state..............DOWN
state_info2.............0x42
m_key_protect_bits......0x0
lmc.....................0x0
link_speed..............0x11
mtu_smsl................0x40
vl_cap_init_type........0x40
vl_high_limit...........0x0
vl_arb_high_cap.........0x8
vl_arb_low_cap..........0x8
init_rep_mtu_cap........0x4
vl_stall_life...........0xF2
vl_enforce..............0x40
m_key_violations........0x0
p_key_violations........0x0
q_key_violations........0x0
guid_cap................0x0
client_reregister.......0x0
subnet_timeout..........0x0
resp_time_value.........0x0
error_threshold.........0x88
Mar 20 14:28:01 539924 [45007960] -> Capabilities Mask:
Mar 20 14:28:01 545091 [45007960] -> SUBNET UP
Mar 20 14:28:01 652647 [43204960] -> __osm_trap_rcv_process_request:
Received Generic Notice type:0x01 num:128 Producer:2 from LID:0x001B
TID:0x0000000000000084
Mar 20 14:28:01 652864 [43204960] -> osm_report_notice: Reporting Generic
Notice type:1 num:128 from LID:0x001B
GID:0xfe80000000000000,0x0005ad00000281a7
Mar 20 14:28:01 879631 [44606960] -> __osm_trap_rcv_process_request:
Received Generic Notice type:0x01 num:128 Producer:2 from LID:0x0148
TID:0x0000000000000074
Mar 20 14:28:01 880104 [44606960] -> osm_report_notice: Reporting Generic
Notice type:1 num:128 from LID:0x0148
GID:0xfe80000000000000,0x0005ad00000281b3
Mar 20 14:28:01 962839 [44606960] -> __osm_trap_rcv_process_request:
Received Generic Notice type:0x01 num:128 Producer:2 from LID:0x0148
TID:0x0000000000000075
Mar 20 14:28:01 965155 [44606960] -> osm_report_notice: Reporting Generic
Notice type:1 num:128 from LID:0x0148
GID:0xfe80000000000000,0x0005ad00000281b3
Mar 20 14:28:02 006432 [41401960] -> __osm_trap_rcv_process_request:
Received Generic Notice type:0x01 num:128 Producer:2 from LID:0x001B
TID:0x0000000000000085
Mar 20 14:28:02 030610 [42803960] -> osm_ucast_mgr_process: Min Hop Tables
configured on all switches
Mar 20 14:28:02 068999 [41401960] -> osm_report_notice: Reporting Generic
Notice type:1 num:128 from LID:0x001B
GID:0xfe80000000000000,0x0005ad00000281a7
Mar 20 14:28:02 081130 [4780B960] -> __osm_sm_mad_ctrl_rcv_callback: ERR
3111: Error status = 0x1C00
Mar 20 14:28:02 081198 [4780B960] -> SMP dump:
base_ver................0x1
mgmt_class..............0x81
class_ver...............0x1
method..................0x81 (SubnGetResp)
D bit...................0x1
status..................0x1C00
hop_ptr.................0x0
hop_count...............0x4
trans_id................0x8a604
attr_id.................0x15 (PortInfo)
resv....................0x0
attr_mod................0x16
m_key...................0x0000000000000000
dr_slid.................0xFFFF
dr_dlid.................0xFFFF
Initial path: [0][1][11][4][4]
Return path: [0][9][18][D][4]
Reserved: [0][0][0][0][0][0][0]
00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
00 00 00 00 00 00 00 00 00 00 00 00 04 03 03 02
14 52 00 11 40 40 00 08 08 04 F2 40 00 00 00 00
00 00 00 00 00 88 00 00 00 00 00 00 00 00 00 00
Mar 20 14:28:02 081249 [43204960] -> osm_pi_rcv_process_set: Received error
status 0x1c for SetResp() during ACTIVE transition
Mar 20 14:28:02 081257 [43204960] -> PortInfo dump:
port number.............0x16
node_guid...............0x0005ad00000281b3
port_guid...............0x0005ad00000281b3
m_key...................0x0000000000000000
subnet_prefix...........0x0000000000000000
base_lid................0x0
master_sm_base_lid......0x0
capability_mask.........0x0
diag_code...............0x0
m_key_lease_period......0x0
local_port_num..........0x4
link_width_enabled......0x3
link_width_supported....0x3
link_width_active.......0x2
link_speed_supported....0x1
port_state..............ACTIVE
state_info2.............0x52
m_key_protect_bits......0x0
lmc.....................0x0
link_speed..............0x11
mtu_smsl................0x40
vl_cap_init_type........0x40
vl_high_limit...........0x0
vl_arb_high_cap.........0x8
vl_arb_low_cap..........0x8
init_rep_mtu_cap........0x4
vl_stall_life...........0xF2
vl_enforce..............0x40
m_key_violations........0x0
p_key_violations........0x0
q_key_violations........0x0
guid_cap................0x0
client_reregister.......0x0
subnet_timeout..........0x0
resp_time_value.........0x0
error_threshold.........0x88
Mar 20 14:28:02 081275 [43204960] -> Capabilities Mask:
Mar 20 14:28:02 081650 [4780B960] -> __osm_sm_mad_ctrl_rcv_callback: ERR
3111: Error status = 0x1C00
Mar 20 14:28:02 081713 [4780B960] -> SMP dump:
base_ver................0x1
mgmt_class..............0x81
class_ver...............0x1
method..................0x81 (SubnGetResp)
D bit...................0x1
status..................0x1C00
hop_ptr.................0x0
hop_count...............0x4
trans_id................0x8a605
attr_id.................0x15 (PortInfo)
resv....................0x0
attr_mod................0x17
m_key...................0x0000000000000000
dr_slid.................0xFFFF
dr_dlid.................0xFFFF
Initial path: [0][1][11][4][4]
Return path: [0][9][18][D][4]
Reserved: [0][0][0][0][0][0][0]
00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
00 00 00 00 00 00 00 00 00 00 00 00 04 03 03 02
14 52 00 11 40 40 00 08 08 04 F2 40 00 00 00 00
00 00 00 00 00 88 00 00 00 00 00 00 00 00 00 00
Mar 20 14:28:02 081782 [43C05960] -> osm_pi_rcv_process_set: Received error
status 0x1c for SetResp() during ACTIVE transition
Mar 20 14:28:02 081787 [43C05960] -> PortInfo dump:
port number.............0x17
node_guid...............0x0005ad00000281b3
port_guid...............0x0005ad00000281b3
m_key...................0x0000000000000000
subnet_prefix...........0x0000000000000000
base_lid................0x0
master_sm_base_lid......0x0
capability_mask.........0x0
diag_code...............0x0
m_key_lease_period......0x0
local_port_num..........0x4
link_width_enabled......0x3
link_width_supported....0x3
link_width_active.......0x2
link_speed_supported....0x1
port_state..............ACTIVE
state_info2.............0x52
m_key_protect_bits......0x0
lmc.....................0x0
link_speed..............0x11
mtu_smsl................0x40
vl_cap_init_type........0x40
vl_high_limit...........0x0
vl_arb_high_cap.........0x8
vl_arb_low_cap..........0x8
init_rep_mtu_cap........0x4
vl_stall_life...........0xF2
vl_enforce..............0x40
m_key_violations........0x0
p_key_violations........0x0
q_key_violations........0x0
guid_cap................0x0
client_reregister.......0x0
subnet_timeout..........0x0
resp_time_value.........0x0
error_threshold.........0x88
Mar 20 14:28:02 081802 [43C05960] -> Capabilities Mask:
Mar 20 14:28:02 087435 [45A08960] -> SUBNET UP
Mar 20 14:28:02 170696 [41401960] -> __osm_trap_rcv_process_request:
Received Generic Notice type:0x01 num:128 Producer:2 from LID:0x001B
TID:0x0000000000000086
Mar 20 14:28:02 170819 [41401960] -> osm_report_notice: Reporting Generic
Notice type:1 num:128 from LID:0x001B
GID:0xfe80000000000000,0x0005ad00000281a7
Mar 20 14:28:02 459228 [41E02960] -> osm_ucast_mgr_process: Min Hop Tables
configured on all switches
Mar 20 14:28:02 500761 [43C05960] -> __osm_trap_rcv_process_request:
Received Generic Notice type:0x01 num:128 Producer:2 from LID:0x001B
TID:0x0000000000000087
Mar 20 14:28:02 500979 [43C05960] -> osm_report_notice: Reporting Generic
Notice type:1 num:128 from LID:0x001B
GID:0xfe80000000000000,0x0005ad00000281a7
Mar 20 14:28:02 510190 [4780B960] -> __osm_sm_mad_ctrl_rcv_callback: ERR
3111: Error status = 0x1C00
Mar 20 14:28:02 510258 [4780B960] -> SMP dump:
base_ver................0x1
mgmt_class..............0x81
class_ver...............0x1
method..................0x81 (SubnGetResp)
D bit...................0x1
status..................0x1C00
hop_ptr.................0x0
hop_count...............0x4
trans_id................0x8b330
attr_id.................0x15 (PortInfo)
resv....................0x0
attr_mod................0x16
m_key...................0x0000000000000000
dr_slid.................0xFFFF
dr_dlid.................0xFFFF
Initial path: [0][1][17][1][5]
Return path: [0][9][14][D][2]
Reserved: [0][0][0][0][0][0][0]
00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
00 00 00 00 00 00 00 00 00 00 00 00 02 03 03 02
14 52 00 11 40 40 00 08 08 04 F2 40 00 00 00 00
00 00 00 00 00 88 00 00 00 00 00 00 00 00 00 00
Mar 20 14:28:02 510366 [4780B960] -> __osm_sm_mad_ctrl_rcv_callback: ERR
3111: Error status = 0x1C00
Mar 20 14:28:02 510370 [45007960] -> osm_pi_rcv_process_set: Received error
status 0x1c for SetResp() during ACTIVE transition
Mar 20 14:28:02 510384 [45007960] -> PortInfo dump:
port number.............0x16
node_guid...............0x0005ad00000281a7
port_guid...............0x0005ad00000281a7
m_key...................0x0000000000000000
subnet_prefix...........0x0000000000000000
base_lid................0x0
master_sm_base_lid......0x0
capability_mask.........0x0
diag_code...............0x0
m_key_lease_period......0x0
local_port_num..........0x2
link_width_enabled......0x3
link_width_supported....0x3
link_width_active.......0x2
link_speed_supported....0x1
port_state..............ACTIVE
state_info2.............0x52
m_key_protect_bits......0x0
lmc.....................0x0
link_speed..............0x11
mtu_smsl................0x40
vl_cap_init_type........0x40
vl_high_limit...........0x0
vl_arb_high_cap.........0x8
vl_arb_low_cap..........0x8
init_rep_mtu_cap........0x4
vl_stall_life...........0xF2
vl_enforce..............0x40
m_key_violations........0x0
p_key_violations........0x0
q_key_violations........0x0
guid_cap................0x0
client_reregister.......0x0
subnet_timeout..........0x0
resp_time_value.........0x0
error_threshold.........0x88
Mar 20 14:28:02 510394 [4780B960] -> SMP dump:
base_ver................0x1
mgmt_class..............0x81
class_ver...............0x1
method..................0x81 (SubnGetResp)
D bit...................0x1
status..................0x1C00
hop_ptr.................0x0
hop_count...............0x4
trans_id................0x8b331
attr_id.................0x15 (PortInfo)
resv....................0x0
attr_mod................0x18
m_key...................0x0000000000000000
dr_slid.................0xFFFF
dr_dlid.................0xFFFF
Initial path: [0][1][11][3][6]
Return path: [0][9][18][F][3]
Reserved: [0][0][0][0][0][0][0]
00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
00 00 00 00 00 00 00 00 00 00 00 00 03 03 03 02
14 52 00 11 40 40 00 08 08 04 F2 40 00 00 00 00
00 00 00 00 00 88 00 00 00 00 00 00 00 00 00 00
Mar 20 14:28:02 510398 [45007960] -> Capabilities Mask:
Mar 20 14:28:02 510481 [41401960] -> osm_pi_rcv_process_set: Received error
status 0x1c for SetResp() during ACTIVE transition
Mar 20 14:28:02 510491 [41401960] -> PortInfo dump:
port number.............0x18
node_guid...............0x0005ad00000281b3
port_guid...............0x0005ad00000281b3
m_key...................0x0000000000000000
subnet_prefix...........0x0000000000000000
base_lid................0x0
master_sm_base_lid......0x0
capability_mask.........0x0
diag_code...............0x0
m_key_lease_period......0x0
local_port_num..........0x3
link_width_enabled......0x3
link_width_supported....0x3
link_width_active.......0x2
link_speed_supported....0x1
port_state..............ACTIVE
state_info2.............0x52
m_key_protect_bits......0x0
lmc.....................0x0
link_speed..............0x11
mtu_smsl................0x40
vl_cap_init_type........0x40
vl_high_limit...........0x0
vl_arb_high_cap.........0x8
vl_arb_low_cap..........0x8
init_rep_mtu_cap........0x4
vl_stall_life...........0xF2
vl_enforce..............0x40
m_key_violations........0x0
p_key_violations........0x0
q_key_violations........0x0
guid_cap................0x0
client_reregister.......0x0
subnet_timeout..........0x0
resp_time_value.........0x0
error_threshold.........0x88
Mar 20 14:28:02 510509 [41401960] -> Capabilities Mask:
Mar 20 14:28:02 510511 [42803960] -> __osm_trap_rcv_process_request:
Received Generic Notice type:0x01 num:128 Producer:2 from LID:0x001B
TID:0x0000000000000088
Mar 20 14:28:02 515576 [41401960] -> SUBNET UP
Mar 20 14:28:02 515695 [42803960] -> osm_report_notice: Reporting Generic
Notice type:1 num:128 from LID:0x001B
GID:0xfe80000000000000,0x0005ad00000281a7
Mar 20 14:28:02 532552 [45A08960] -> __osm_trap_rcv_process_request:
Received Generic Notice type:0x01 num:128 Producer:2 from LID:0x001B
TID:0x0000000000000089
Mar 20 14:28:02 538569 [45A08960] -> osm_report_notice: Reporting Generic
Notice type:1 num:128 from LID:0x001B
GID:0xfe80000000000000,0x0005ad00000281a7
Mar 20 14:28:02 695997 [43204960] -> __osm_trap_rcv_process_request:
Received Generic Notice type:0x01 num:128 Producer:2 from LID:0x0152
TID:0x000000000000001e
Mar 20 14:28:02 696096 [43204960] -> __osm_trap_rcv_process_request: ERR
3804: Received trap 30 times consecutively
Mar 20 14:28:02 918226 [45007960] -> osm_ucast_mgr_process: Min Hop Tables
configured on all switches
Mar 20 14:28:02 975244 [43204960] -> SUBNET UP
Mar 20 14:28:03 325494 [43204960] -> osm_ucast_mgr_process: Min Hop Tables
configured on all switches
Mar 20 14:28:03 379145 [41401960] -> SUBNET UP
Mar 20 14:29:12 561841 [41401960] -> __osm_trap_rcv_process_request:
Received Generic Notice type:0x01 num:128 Producer:2 from LID:0x001F
TID:0x0000000000000019
Mar 20 14:29:12 562033 [41401960] -> osm_report_notice: Reporting Generic
Notice type:1 num:128 from LID:0x001F
GID:0xfe80000000000000,0x0005ad0000027c56
Mar 20 14:29:12 562751 [42803960] -> __osm_trap_rcv_process_request:
Received Generic Notice type:0x01 num:128 Producer:2 from LID:0x001F
TID:0x000000000000001a
Mar 20 14:29:12 562902 [42803960] -> osm_report_notice: Reporting Generic
Notice type:1 num:128 from LID:0x001F
GID:0xfe80000000000000,0x0005ad0000027c56
Mar 20 14:29:12 571346 [42803960] -> __osm_trap_rcv_process_request:
Received Generic Notice type:0x01 num:128 Producer:2 from LID:0x0084
TID:0x0000000000000018
Mar 20 14:29:12 571569 [42803960] -> osm_report_notice: Reporting Generic
Notice type:1 num:128 from LID:0x0084
GID:0xfe80000000000000,0x0005ad0000027c70
Mar 20 14:29:12 914371 [42803960] -> __osm_trap_rcv_process_request:
Received Generic Notice type:0x01 num:128 Producer:2 from LID:0x001F
TID:0x000000000000001b
Mar 20 14:29:12 916287 [44606960] -> osm_report_notice: Reporting Generic
Notice type:3 num:64 from LID:0x0092
GID:0xfe80000000000000,0x0005ad0000024bbb
Mar 20 14:29:12 916297 [44606960] -> Discovered new port with
GUID:0x0005ad000002502f LID range [0x2,0x2] of node:Topspin IB-DC
Mar 20 14:29:12 946985 [44606960] -> osm_ucast_mgr_process: Min Hop Tables
configured on all switches
Mar 20 14:29:12 976839 [42803960] -> osm_report_notice: Reporting Generic
Notice type:1 num:128 from LID:0x001F
GID:0xfe80000000000000,0x0005ad0000027c56
Mar 20 14:29:12 987963 [4780B960] -> __osm_sm_mad_ctrl_rcv_callback: ERR
3111: Error status = 0x1C00
Mar 20 14:29:12 988004 [4780B960] -> SMP dump:
base_ver................0x1
mgmt_class..............0x81
class_ver...............0x1
method..................0x81 (SubnGetResp)
D bit...................0x1
status..................0x1C00
hop_ptr.................0x0
hop_count...............0x5
trans_id................0x8dbb2
attr_id.................0x15 (PortInfo)
resv....................0x0
attr_mod................0xD
m_key...................0x0000000000000000
dr_slid.................0xFFFF
dr_dlid.................0xFFFF
Initial path: [0][1][11][2][4][D]
Return path: [0][9][18][E][1][12]
Reserved: [0][0][0][0][0][0][0]
00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
00 00 00 00 00 00 00 00 00 00 00 00 12 03 03 02
14 52 00 11 40 40 00 08 08 04 2C 4C 00 00 00 00
00 00 00 00 00 88 00 00 00 00 00 00 00 00 00 00
Mar 20 14:29:12 988078 [43C05960] -> osm_pi_rcv_process_set: Received error
status 0x1c for SetResp() during ACTIVE transition
Mar 20 14:29:12 988089 [43C05960] -> PortInfo dump:
port number.............0xD
node_guid...............0x0005ad0000027c70
port_guid...............0x0005ad0000027c70
m_key...................0x0000000000000000
subnet_prefix...........0x0000000000000000
base_lid................0x0
master_sm_base_lid......0x0
capability_mask.........0x0
diag_code...............0x0
m_key_lease_period......0x0
local_port_num..........0x12
link_width_enabled......0x3
link_width_supported....0x3
link_width_active.......0x2
link_speed_supported....0x1
port_state..............ACTIVE
state_info2.............0x52
m_key_protect_bits......0x0
lmc.....................0x0
link_speed..............0x11
mtu_smsl................0x40
vl_cap_init_type........0x40
vl_high_limit...........0x0
vl_arb_high_cap.........0x8
vl_arb_low_cap..........0x8
init_rep_mtu_cap........0x4
vl_stall_life...........0x2C
vl_enforce..............0x4C
m_key_violations........0x0
p_key_violations........0x0
q_key_violations........0x0
guid_cap................0x0
client_reregister.......0x0
subnet_timeout..........0x0
resp_time_value.........0x0
error_threshold.........0x88
Mar 20 14:29:12 988105 [43C05960] -> Capabilities Mask:
Mar 20 14:29:12 993136 [44606960] -> SUBNET UP
Mar 20 14:29:13 300755 [41E02960] -> __osm_trap_rcv_process_request:
Received Generic Notice type:0x04 num:144 Producer:1 from LID:0x0002
TID:0x0000000000000000
Mar 20 14:29:13 300874 [41E02960] -> osm_report_notice: Reporting Generic
Notice type:4 num:144 from LID:0x0002
GID:0xfe80000000000000,0x0005ad000002502f
Mar 20 14:29:13 338077 [41E02960] -> osm_report_notice: Reporting Generic
Notice type:3 num:64 from LID:0x0092
GID:0xfe80000000000000,0x0005ad0000024bbb
Mar 20 14:29:13 338099 [41E02960] -> Discovered new port with
GUID:0x0005ad000002516f LID range [0xBA,0xBA] of node:Topspin IB-DC
Mar 20 14:29:13 368879 [41E02960] -> osm_ucast_mgr_process: Min Hop Tables
configured on all switches
Mar 20 14:29:13 431763 [4780B960] -> __osm_sm_mad_ctrl_rcv_callback: ERR
3111: Error status = 0x1C00
Mar 20 14:29:13 431806 [4780B960] -> SMP dump:
base_ver................0x1
mgmt_class..............0x81
class_ver...............0x1
method..................0x81 (SubnGetResp)
D bit...................0x1
status..................0x1C00
hop_ptr.................0x0
hop_count...............0x5
trans_id................0x8e8e9
attr_id.................0x15 (PortInfo)
resv....................0x0
attr_mod................0xA
m_key...................0x0000000000000000
dr_slid.................0xFFFF
dr_dlid.................0xFFFF
Initial path: [0][1][14][1][6][12]
Return path: [0][9][15][D][3][11]
Reserved: [0][0][0][0][0][0][0]
00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
00 00 00 00 00 00 00 00 00 00 00 00 11 03 03 02
14 52 00 11 40 40 00 08 08 04 2C 4C 00 00 00 00
00 00 00 00 00 88 00 00 00 00 00 00 00 00 00 00
Mar 20 14:29:13 432093 [43204960] -> osm_pi_rcv_process_set: Received error
status 0x1c for SetResp() during ACTIVE transition
Mar 20 14:29:13 432116 [43204960] -> PortInfo dump:
port number.............0xA
node_guid...............0x0005ad0000027c56
port_guid...............0x0005ad0000027c56
m_key...................0x0000000000000000
subnet_prefix...........0x0000000000000000
base_lid................0x0
master_sm_base_lid......0x0
capability_mask.........0x0
diag_code...............0x0
m_key_lease_period......0x0
local_port_num..........0x11
link_width_enabled......0x3
link_width_supported....0x3
link_width_active.......0x2
link_speed_supported....0x1
port_state..............ACTIVE
state_info2.............0x52
m_key_protect_bits......0x0
lmc.....................0x0
link_speed..............0x11
mtu_smsl................0x40
vl_cap_init_type........0x40
vl_high_limit...........0x0
vl_arb_high_cap.........0x8
vl_arb_low_cap..........0x8
init_rep_mtu_cap........0x4
vl_stall_life...........0x2C
vl_enforce..............0x4C
m_key_violations........0x0
p_key_violations........0x0
q_key_violations........0x0
guid_cap................0x0
client_reregister.......0x0
subnet_timeout..........0x0
resp_time_value.........0x0
error_threshold.........0x88
Mar 20 14:29:13 432135 [43204960] -> Capabilities Mask:
Mar 20 14:29:13 437219 [45007960] -> SUBNET UP
Mar 20 14:29:13 690992 [42803960] -> __osm_trap_rcv_process_request:
Received Generic Notice type:0x04 num:144 Producer:1 from LID:0x00BA
TID:0x0000000000000000
Mar 20 14:29:13 691128 [42803960] -> osm_report_notice: Reporting Generic
Notice type:4 num:144 from LID:0x00BA
GID:0xfe80000000000000,0x0005ad000002516f
Mar 20 14:29:13 835017 [44606960] -> osm_ucast_mgr_process: Min Hop Tables
configured on all switches
Mar 20 14:29:13 891082 [42803960] -> SUBNET UP
Mar 20 14:29:14 235714 [42803960] -> osm_ucast_mgr_process: Min Hop Tables
configured on all switches
Mar 20 14:29:14 289127 [41E02960] -> SUBNET UP
Mar 20 14:29:17 689267 [43204960] -> __osm_trap_rcv_process_request:
Received Generic Notice type:0x01 num:128 Producer:2 from LID:0x001B
TID:0x000000000000008a
Mar 20 14:29:17 689511 [43204960] -> osm_report_notice: Reporting Generic
Notice type:1 num:128 from LID:0x001B
GID:0xfe80000000000000,0x0005ad00000281a7
Mar 20 14:29:17 689975 [42803960] -> __osm_trap_rcv_process_request:
Received Generic Notice type:0x01 num:128 Producer:2 from LID:0x0148
TID:0x0000000000000076
Mar 20 14:29:17 690097 [42803960] -> osm_report_notice: Reporting Generic
Notice type:1 num:128 from LID:0x0148
GID:0xfe80000000000000,0x0005ad00000281b3
Mar 20 14:29:18 025237 [44606960] -> osm_report_notice: Reporting Generic
Notice type:3 num:65 from LID:0x0092
GID:0xfe80000000000000,0x0005ad0000024bbb
Mar 20 14:29:18 025255 [44606960] -> Removed port with
GUID:0x0005ad0000024e0b LID range [0xB3,0xB3] of node:saguaro-23-4 HCA-1
Mar 20 14:29:18 025273 [44606960] -> osm_report_notice: Reporting Generic
Notice type:3 num:65 from LID:0x0092
GID:0xfe80000000000000,0x0005ad0000024bbb
Mar 20 14:29:18 025279 [44606960] -> Removed port with
GUID:0x0005ad000002510b LID range [0xB5,0xB5] of node:saguaro-23-6 HCA-1
Mar 20 14:29:18 025296 [44606960] -> osm_report_notice: Reporting Generic
Notice type:3 num:65 from LID:0x0092
GID:0xfe80000000000000,0x0005ad0000024bbb
Mar 20 14:29:18 025300 [44606960] -> Removed port with
GUID:0x0005ad000002511b LID range [0xA6,0xA6] of node:saguaro-22-1 HCA-1
Mar 20 14:29:18 025317 [44606960] -> osm_report_notice: Reporting Generic
Notice type:3 num:65 from LID:0x0092
GID:0xfe80000000000000,0x0005ad0000024bbb
Mar 20 14:29:18 025323 [44606960] -> Removed port with
GUID:0x0005ad0000024b27 LID range [0xAF,0xAF] of node:saguaro-23-0 HCA-1
Mar 20 14:29:18 025340 [44606960] -> osm_report_notice: Reporting Generic
Notice type:3 num:65 from LID:0x0092
GID:0xfe80000000000000,0x0005ad0000024bbb
Mar 20 14:29:18 025345 [44606960] -> Removed port with
GUID:0x0005ad0000025043 LID range [0xB4,0xB4] of node:saguaro-23-5 HCA-1
Mar 20 14:29:18 025362 [44606960] -> osm_report_notice: Reporting Generic
Notice type:3 num:65 from LID:0x0092
GID:0xfe80000000000000,0x0005ad0000024bbb
Mar 20 14:29:18 025367 [44606960] -> Removed port with
GUID:0x0005ad0000024d47 LID range [0xB6,0xB6] of node:saguaro-23-7 HCA-1
Mar 20 14:29:18 025385 [44606960] -> osm_report_notice: Reporting Generic
Notice type:3 num:65 from LID:0x0092
GID:0xfe80000000000000,0x0005ad0000024bbb
Mar 20 14:29:18 025390 [44606960] -> Removed port with
GUID:0x0005ad0000024d6b LID range [0xB8,0xB8] of node:saguaro-23-9 HCA-1
Mar 20 14:29:18 025406 [44606960] -> osm_report_notice: Reporting Generic
Notice type:3 num:65 from LID:0x0092
GID:0xfe80000000000000,0x0005ad0000024bbb
Mar 20 14:29:18 025411 [44606960] -> Removed port with
GUID:0x0005ad0000024977 LID range [0xA9,0xA9] of node:saguaro-22-4 HCA-1
Mar 20 14:29:18 025571 [44606960] -> osm_report_notice: Reporting Generic
Notice type:3 num:65 from LID:0x0092
GID:0xfe80000000000000,0x0005ad0000024bbb
Mar 20 14:29:18 025576 [44606960] -> Removed port with
GUID:0x0005ad0000027c84 LID range [0x152,0x152] of node:Topspin Switch TS120
Mar 20 14:29:18 025612 [44606960] -> osm_report_notice: Reporting Generic
Notice type:3 num:65 from LID:0x0092
GID:0xfe80000000000000,0x0005ad0000024bbb
Mar 20 14:29:18 025619 [44606960] -> Removed port with
GUID:0x0005ad0000024d8b LID range [0xB7,0xB7] of node:saguaro-23-8 HCA-1
Mar 20 14:29:18 025634 [44606960] -> osm_report_notice: Reporting Generic
Notice type:3 num:65 from LID:0x0092
GID:0xfe80000000000000,0x0005ad0000024bbb
Mar 20 14:29:18 025639 [44606960] -> Removed port with
GUID:0x0005ad000002498f LID range [0xA8,0xA8] of node:saguaro-22-3 HCA-1
Mar 20 14:29:18 025655 [44606960] -> osm_report_notice: Reporting Generic
Notice type:3 num:65 from LID:0x0092
GID:0xfe80000000000000,0x0005ad0000024bbb
Mar 20 14:29:18 025660 [44606960] -> Removed port with
GUID:0x0005ad0000024c9b LID range [0xA7,0xA7] of node:saguaro-22-2 HCA-1
Mar 20 14:29:18 025678 [44606960] -> osm_report_notice: Reporting Generic
Notice type:3 num:65 from LID:0x0092
GID:0xfe80000000000000,0x0005ad0000024bbb
Mar 20 14:29:18 025683 [44606960] -> Removed port with
GUID:0x0005ad0000024da7 LID range [0xB0,0xB0] of node:saguaro-23-1 HCA-1
Mar 20 14:29:18 025700 [44606960] -> osm_report_notice: Reporting Generic
Notice type:3 num:65 from LID:0x0092
GID:0xfe80000000000000,0x0005ad0000024bbb
Mar 20 14:29:18 025705 [44606960] -> Removed port with
GUID:0x0005ad0000024cbb LID range [0xB2,0xB2] of node:saguaro-23-3 HCA-1
Mar 20 14:29:18 025721 [44606960] -> osm_report_notice: Reporting Generic
Notice type:3 num:65 from LID:0x0092
GID:0xfe80000000000000,0x0005ad0000024bbb
Mar 20 14:29:18 025777 [44606960] -> Removed port with
GUID:0x0005ad00000249d3 LID range [0xB1,0xB1] of node:saguaro-23-2 HCA-1
Mar 20 14:29:18 025794 [44606960] -> osm_report_notice: Reporting Generic
Notice type:3 num:65 from LID:0x0092
GID:0xfe80000000000000,0x0005ad0000024bbb
Mar 20 14:29:18 025800 [44606960] -> Removed port with
GUID:0x0005ad0000024feb LID range [0x153,0x153] of node:saguaro-22-5 HCA-1
Mar 20 14:29:18 025816 [44606960] -> osm_report_notice: Reporting Generic
Notice type:3 num:65 from LID:0x0092
GID:0xfe80000000000000,0x0005ad0000024bbb
Mar 20 14:29:18 025821 [44606960] -> Removed port with
GUID:0x0005ad0000024afb LID range [0xA5,0xA5] of node:saguaro-22-0 HCA-1
Mar 20 14:29:18 058968 [44606960] -> osm_ucast_mgr_process: Min Hop Tables
configured on all switches
Mar 20 14:29:18 112970 [43C05960] -> SUBNET UP
Mar 20 14:29:18 511156 [45007960] -> osm_ucast_mgr_process: Min Hop Tables
configured on all switches
Mar 20 14:29:18 573846 [41E02960] -> SUBNET UP
Mar 20 14:30:11 599965 [45A08960] -> __osm_trap_rcv_process_request:
Received Generic Notice type:0x01 num:128 Producer:2 from LID:0x0148
TID:0x0000000000000077
Mar 20 14:30:11 600182 [45A08960] -> osm_report_notice: Reporting Generic
Notice type:1 num:128 from LID:0x0148
GID:0xfe80000000000000,0x0005ad00000281b3
Mar 20 14:30:11 606044 [44606960] -> __osm_trap_rcv_process_request:
Received Generic Notice type:0x01 num:128 Producer:2 from LID:0x0148
TID:0x0000000000000078
Mar 20 14:30:11 606078 [43C05960] -> __osm_trap_rcv_process_request:
Received Generic Notice type:0x01 num:128 Producer:2 from LID:0x001B
TID:0x000000000000008b
Mar 20 14:30:11 606178 [44606960] -> osm_report_notice: Reporting Generic
Notice type:1 num:128 from LID:0x0148
GID:0xfe80000000000000,0x0005ad00000281b3
Mar 20 14:30:11 606207 [43C05960] -> osm_report_notice: Reporting Generic
Notice type:1 num:128 from LID:0x001B
GID:0xfe80000000000000,0x0005ad00000281a7
Mar 20 14:30:11 612375 [42803960] -> __osm_trap_rcv_process_request:
Received Generic Notice type:0x01 num:128 Producer:2 from LID:0x001B
TID:0x000000000000008c
Mar 20 14:30:11 612499 [42803960] -> osm_report_notice: Reporting Generic
Notice type:1 num:128 from LID:0x001B
GID:0xfe80000000000000,0x0005ad00000281a7
Mar 20 14:30:11 947057 [45007960] -> osm_report_notice: Reporting Generic
Notice type:3 num:64 from LID:0x0092
GID:0xfe80000000000000,0x0005ad0000024bbb
Mar 20 14:30:11 947074 [45007960] -> Discovered new port with
GUID:0x0005ad0000027c84 LID range [0x152,0x152] of node:Topspin Switch TS120
Mar 20 14:30:11 947079 [45007960] -> osm_report_notice: Reporting Generic
Notice type:3 num:64 from LID:0x0092
GID:0xfe80000000000000,0x0005ad0000024bbb
Mar 20 14:30:11 947084 [45007960] -> Discovered new port with
GUID:0x0005ad0000024b27 LID range [0xAF,0xAF] of node:saguaro-23-0 HCA-1
Mar 20 14:30:11 947088 [45007960] -> osm_report_notice: Reporting Generic
Notice type:3 num:64 from LID:0x0092
GID:0xfe80000000000000,0x0005ad0000024bbb
Mar 20 14:30:11 947093 [45007960] -> Discovered new port with
GUID:0x0005ad0000024da7 LID range [0xB0,0xB0] of node:saguaro-23-1 HCA-1
Mar 20 14:30:11 947097 [45007960] -> osm_report_notice: Reporting Generic
Notice type:3 num:64 from LID:0x0092
GID:0xfe80000000000000,0x0005ad0000024bbb
Mar 20 14:30:11 947102 [45007960] -> Discovered new port with
GUID:0x0005ad00000249d3 LID range [0xB1,0xB1] of node:saguaro-23-2 HCA-1
Mar 20 14:30:11 947106 [45007960] -> osm_report_notice: Reporting Generic
Notice type:3 num:64 from LID:0x0092
GID:0xfe80000000000000,0x0005ad0000024bbb
Mar 20 14:30:11 947118 [45007960] -> Discovered new port with
GUID:0x0005ad0000024cbb LID range [0xB2,0xB2] of node:saguaro-23-3 HCA-1
Mar 20 14:30:11 947138 [45007960] -> osm_report_notice: Reporting Generic
Notice type:3 num:64 from LID:0x0092
GID:0xfe80000000000000,0x0005ad0000024bbb
Mar 20 14:30:11 947143 [45007960] -> Discovered new port with
GUID:0x0005ad0000024e0b LID range [0xB3,0xB3] of node:saguaro-23-4 HCA-1
Mar 20 14:30:11 947148 [45007960] -> osm_report_notice: Reporting Generic
Notice type:3 num:64 from LID:0x0092
GID:0xfe80000000000000,0x0005ad0000024bbb
Mar 20 14:30:11 947153 [45007960] -> Discovered new port with
GUID:0x0005ad0000025043 LID range [0xB4,0xB4] of node:saguaro-23-5 HCA-1
Mar 20 14:30:11 947157 [45007960] -> osm_report_notice: Reporting Generic
Notice type:3 num:64 from LID:0x0092
GID:0xfe80000000000000,0x0005ad0000024bbb
Mar 20 14:30:11 947162 [45007960] -> Discovered new port with
GUID:0x0005ad000002510b LID range [0xB5,0xB5] of node:saguaro-23-6 HCA-1
Mar 20 14:30:11 947166 [45007960] -> osm_report_notice: Reporting Generic
Notice type:3 num:64 from LID:0x0092
GID:0xfe80000000000000,0x0005ad0000024bbb
Mar 20 14:30:11 947170 [45007960] -> Discovered new port with
GUID:0x0005ad0000024d47 LID range [0xB6,0xB6] of node:saguaro-23-7 HCA-1
Mar 20 14:30:11 947174 [45007960] -> osm_report_notice: Reporting Generic
Notice type:3 num:64 from LID:0x0092
GID:0xfe80000000000000,0x0005ad0000024bbb
Mar 20 14:30:11 947179 [45007960] -> Discovered new port with
GUID:0x0005ad0000024d8b LID range [0xB7,0xB7] of node:saguaro-23-8 HCA-1
Mar 20 14:30:11 947183 [45007960] -> osm_report_notice: Reporting Generic
Notice type:3 num:64 from LID:0x0092
GID:0xfe80000000000000,0x0005ad0000024bbb
Mar 20 14:30:11 947188 [45007960] -> Discovered new port with
GUID:0x0005ad0000024d6b LID range [0xB8,0xB8] of node:saguaro-23-9 HCA-1
Mar 20 14:30:11 947191 [45007960] -> osm_report_notice: Reporting Generic
Notice type:3 num:64 from LID:0x0092
GID:0xfe80000000000000,0x0005ad0000024bbb
Mar 20 14:30:11 947196 [45007960] -> Discovered new port with
GUID:0x0005ad0000024afb LID range [0xA5,0xA5] of node:saguaro-22-0 HCA-1
Mar 20 14:30:11 947200 [45007960] -> osm_report_notice: Reporting Generic
Notice type:3 num:64 from LID:0x0092
GID:0xfe80000000000000,0x0005ad0000024bbb
Mar 20 14:30:11 947205 [45007960] -> Discovered new port with
GUID:0x0005ad000002511b LID range [0xA6,0xA6] of node:saguaro-22-1 HCA-1
Mar 20 14:30:11 947209 [45007960] -> osm_report_notice: Reporting Generic
Notice type:3 num:64 from LID:0x0092
GID:0xfe80000000000000,0x0005ad0000024bbb
Mar 20 14:30:11 947214 [45007960] -> Discovered new port with
GUID:0x0005ad0000024c9b LID range [0xA7,0xA7] of node:saguaro-22-2 HCA-1
Mar 20 14:30:11 947282 [45007960] -> osm_report_notice: Reporting Generic
Notice type:3 num:64 from LID:0x0092
GID:0xfe80000000000000,0x0005ad0000024bbb
Mar 20 14:30:11 947288 [45007960] -> Discovered new port with
GUID:0x0005ad000002498f LID range [0xA8,0xA8] of node:saguaro-22-3 HCA-1
Mar 20 14:30:11 947291 [45007960] -> osm_report_notice: Reporting Generic
Notice type:3 num:64 from LID:0x0092
GID:0xfe80000000000000,0x0005ad0000024bbb
Mar 20 14:30:11 947296 [45007960] -> Discovered new port with
GUID:0x0005ad0000024977 LID range [0xA9,0xA9] of node:saguaro-22-4 HCA-1
Mar 20 14:30:11 947300 [45007960] -> osm_report_notice: Reporting Generic
Notice type:3 num:64 from LID:0x0092
GID:0xfe80000000000000,0x0005ad0000024bbb
Mar 20 14:30:11 947305 [45007960] -> Discovered new port with
GUID:0x0005ad0000024feb LID range [0x153,0x153] of node:saguaro-22-5 HCA-1
Mar 20 14:30:11 978149 [45007960] -> osm_ucast_mgr_process: Min Hop Tables
configured on all switches
Mar 20 14:30:12 042474 [4780B960] -> __osm_sm_mad_ctrl_rcv_callback: ERR
3111: Error status = 0x1C00
Mar 20 14:30:12 042577 [4780B960] -> SMP dump:
base_ver................0x1
mgmt_class..............0x81
class_ver...............0x1
method..................0x81 (SubnGetResp)
D bit...................0x1
status..................0x1C00
hop_ptr.................0x0
hop_count...............0x4
trans_id................0x92b38
attr_id.................0x15 (PortInfo)
resv....................0x0
attr_mod................0x13
m_key...................0x0000000000000000
dr_slid.................0xFFFF
dr_dlid.................0xFFFF
Initial path: [0][1][16][1][5]
Return path: [0][9][13][D][2]
Reserved: [0][0][0][0][0][0][0]
00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
00 00 00 00 00 00 00 00 00 00 00 00 02 03 03 02
14 52 00 11 40 40 00 08 08 04 F2 40 00 00 00 00
00 00 00 00 00 88 00 00 00 00 00 00 00 00 00 00
Mar 20 14:30:12 042668 [45007960] -> osm_pi_rcv_process_set: Received error
status 0x1c for SetResp() during ACTIVE transition
Mar 20 14:30:12 042676 [4780B960] -> __osm_sm_mad_ctrl_rcv_callback: ERR
3111: Error status = 0x1C00
Mar 20 14:30:12 042682 [45007960] -> PortInfo dump:
port number.............0x13
node_guid...............0x0005ad00000281a7
port_guid...............0x0005ad00000281a7
m_key...................0x0000000000000000
subnet_prefix...........0x0000000000000000
base_lid................0x0
master_sm_base_lid......0x0
capability_mask.........0x0
diag_code...............0x0
m_key_lease_period......0x0
local_port_num..........0x2
link_width_enabled......0x3
link_width_supported....0x3
link_width_active.......0x2
link_speed_supported....0x1
port_state..............ACTIVE
state_info2.............0x52
m_key_protect_bits......0x0
lmc.....................0x0
link_speed..............0x11
mtu_smsl................0x40
vl_cap_init_type........0x40
vl_high_limit...........0x0
vl_arb_high_cap.........0x8
vl_arb_low_cap..........0x8
init_rep_mtu_cap........0x4
vl_stall_life...........0xF2
vl_enforce..............0x40
m_key_violations........0x0
p_key_violations........0x0
q_key_violations........0x0
guid_cap................0x0
client_reregister.......0x0
subnet_timeout..........0x0
resp_time_value.........0x0
error_threshold.........0x88
Mar 20 14:30:12 042701 [45007960] -> Capabilities Mask:
Mar 20 14:30:12 042714 [4780B960] -> SMP dump:
base_ver................0x1
mgmt_class..............0x81
class_ver...............0x1
method..................0x81 (SubnGetResp)
D bit...................0x1
status..................0x1C00
hop_ptr.................0x0
hop_count...............0x4
trans_id................0x92b39
attr_id.................0x15 (PortInfo)
resv....................0x0
attr_mod................0x16
m_key...................0x0000000000000000
dr_slid.................0xFFFF
dr_dlid.................0xFFFF
Initial path: [0][1][16][1][5]
Return path: [0][9][13][D][2]
Reserved: [0][0][0][0][0][0][0]
00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
00 00 00 00 00 00 00 00 00 00 00 00 02 03 03 02
14 52 00 11 40 40 00 08 08 04 F2 40 00 00 00 00
00 00 00 00 00 88 00 00 00 00 00 00 00 00 00 00
Mar 20 14:30:12 042845 [41401960] -> osm_pi_rcv_process_set: Received error
status 0x1c for SetResp() during ACTIVE transition
Mar 20 14:30:12 042856 [4780B960] -> __osm_sm_mad_ctrl_rcv_callback: ERR
3111: Error status = 0x1C00
Mar 20 14:30:12 042851 [41401960] -> PortInfo dump:
port number.............0x16
node_guid...............0x0005ad00000281a7
port_guid...............0x0005ad00000281a7
m_key...................0x0000000000000000
subnet_prefix...........0x0000000000000000
base_lid................0x0
master_sm_base_lid......0x0
capability_mask.........0x0
diag_code...............0x0
m_key_lease_period......0x0
local_port_num..........0x2
link_width_enabled......0x3
link_width_supported....0x3
link_width_active.......0x2
link_speed_supported....0x1
port_state..............ACTIVE
state_info2.............0x52
m_key_protect_bits......0x0
lmc.....................0x0
link_speed..............0x11
mtu_smsl................0x40
vl_cap_init_type........0x40
vl_high_limit...........0x0
vl_arb_high_cap.........0x8
vl_arb_low_cap..........0x8
init_rep_mtu_cap........0x4
vl_stall_life...........0xF2
vl_enforce..............0x40
m_key_violations........0x0
p_key_violations........0x0
q_key_violations........0x0
guid_cap................0x0
client_reregister.......0x0
subnet_timeout..........0x0
resp_time_value.........0x0
error_threshold.........0x88
Mar 20 14:30:12 042897 [41401960] -> Capabilities Mask:
Mar 20 14:30:12 042907 [4780B960] -> SMP dump:
base_ver................0x1
mgmt_class..............0x81
class_ver...............0x1
method..................0x81 (SubnGetResp)
D bit...................0x1
status..................0x1C00
hop_ptr.................0x0
hop_count...............0x4
trans_id................0x92b3a
attr_id.................0x15 (PortInfo)
resv....................0x0
attr_mod................0x17
m_key...................0x0000000000000000
dr_slid.................0xFFFF
dr_dlid.................0xFFFF
Initial path: [0][1][16][1][5]
Return path: [0][9][13][D][2]
Reserved: [0][0][0][0][0][0][0]
00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
00 00 00 00 00 00 00 00 00 00 00 00 02 03 03 02
14 52 00 11 40 40 00 08 08 04 F2 40 00 00 00 00
00 00 00 00 00 88 00 00 00 00 00 00 00 00 00 00
Mar 20 14:30:12 043013 [43204960] -> osm_pi_rcv_process_set: Received error
status 0x1c for SetResp() during ACTIVE transition
Mar 20 14:30:12 043015 [4780B960] -> __osm_sm_mad_ctrl_rcv_callback: ERR
3111: Error status = 0x1C00
Mar 20 14:30:12 043038 [43204960] -> PortInfo dump:
port number.............0x17
node_guid...............0x0005ad00000281a7
port_guid...............0x0005ad00000281a7
m_key...................0x0000000000000000
subnet_prefix...........0x0000000000000000
base_lid................0x0
master_sm_base_lid......0x0
capability_mask.........0x0
diag_code...............0x0
m_key_lease_period......0x0
local_port_num..........0x2
link_width_enabled......0x3
link_width_supported....0x3
link_width_active.......0x2
link_speed_supported....0x1
port_state..............ACTIVE
state_info2.............0x52
m_key_protect_bits......0x0
lmc.....................0x0
link_speed..............0x11
mtu_smsl................0x40
vl_cap_init_type........0x40
vl_high_limit...........0x0
vl_arb_high_cap.........0x8
vl_arb_low_cap..........0x8
init_rep_mtu_cap........0x4
vl_stall_life...........0xF2
vl_enforce..............0x40
m_key_violations........0x0
p_key_violations........0x0
q_key_violations........0x0
guid_cap................0x0
client_reregister.......0x0
subnet_timeout..........0x0
resp_time_value.........0x0
error_threshold.........0x88
Mar 20 14:30:12 043090 [43204960] -> Capabilities Mask:
Mar 20 14:30:12 043094 [4780B960] -> SMP dump:
base_ver................0x1
mgmt_class..............0x81
class_ver...............0x1
method..................0x81 (SubnGetResp)
D bit...................0x1
status..................0x1C00
hop_ptr.................0x0
hop_count...............0x4
trans_id................0x92b3b
attr_id.................0x15 (PortInfo)
resv....................0x0
attr_mod................0x18
m_key...................0x0000000000000000
dr_slid.................0xFFFF
dr_dlid.................0xFFFF
Initial path: [0][1][16][1][5]
Return path: [0][9][13][D][2]
Reserved: [0][0][0][0][0][0][0]
00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
00 00 00 00 00 00 00 00 00 00 00 00 02 03 03 02
14 52 00 11 40 40 00 08 08 04 F2 40 00 00 00 00
00 00 00 00 00 88 00 00 00 00 00 00 00 00 00 00
Mar 20 14:30:12 043173 [44606960] -> osm_pi_rcv_process_set: Received error
status 0x1c for SetResp() during ACTIVE transition
Mar 20 14:30:12 043178 [44606960] -> PortInfo dump:
port number.............0x18
node_guid...............0x0005ad00000281a7
port_guid...............0x0005ad00000281a7
m_key...................0x0000000000000000
subnet_prefix...........0x0000000000000000
base_lid................0x0
master_sm_base_lid......0x0
capability_mask.........0x0
diag_code...............0x0
m_key_lease_period......0x0
local_port_num..........0x2
link_width_enabled......0x3
link_width_supported....0x3
link_width_active.......0x2
link_speed_supported....0x1
port_state..............ACTIVE
state_info2.............0x52
m_key_protect_bits......0x0
lmc.....................0x0
link_speed..............0x11
mtu_smsl................0x40
vl_cap_init_type........0x40
vl_high_limit...........0x0
vl_arb_high_cap.........0x8
vl_arb_low_cap..........0x8
init_rep_mtu_cap........0x4
vl_stall_life...........0xF2
vl_enforce..............0x40
m_key_violations........0x0
p_key_violations........0x0
q_key_violations........0x0
guid_cap................0x0
client_reregister.......0x0
subnet_timeout..........0x0
resp_time_value.........0x0
error_threshold.........0x88
Mar 20 14:30:12 043191 [44606960] -> Capabilities Mask:
Mar 20 14:30:12 043222 [4780B960] -> __osm_sm_mad_ctrl_rcv_callback: ERR
3111: Error status = 0x1C00
Mar 20 14:30:12 043247 [4780B960] -> SMP dump:
base_ver................0x1
mgmt_class..............0x81
class_ver...............0x1
method..................0x81 (SubnGetResp)
D bit...................0x1
status..................0x1C00
hop_ptr.................0x0
hop_count...............0x4
trans_id................0x92b3c
attr_id.................0x15 (PortInfo)
resv....................0x0
attr_mod................0x16
m_key...................0x0000000000000000
dr_slid.................0xFFFF
dr_dlid.................0xFFFF
Initial path: [0][1][12][1][4]
Return path: [0][9][14][D][1]
Reserved: [0][0][0][0][0][0][0]
00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
00 00 00 00 00 00 00 00 00 00 00 00 01 03 03 02
14 52 00 11 40 40 00 08 08 04 F2 40 00 00 00 00
00 00 00 00 00 88 00 00 00 00 00 00 00 00 00 00
Mar 20 14:30:12 043318 [4780B960] -> __osm_sm_mad_ctrl_rcv_callback: ERR
3111: Error status = 0x1C00
Mar 20 14:30:12 043314 [41E02960] -> osm_pi_rcv_process_set: Received error
status 0x1c for SetResp() during ACTIVE transition
Mar 20 14:30:12 043357 [4780B960] -> SMP dump:
base_ver................0x1
mgmt_class..............0x81
class_ver...............0x1
method..................0x81 (SubnGetResp)
D bit...................0x1
status..................0x1C00
hop_ptr.................0x0
hop_count...............0x4
trans_id................0x92b3d
attr_id.................0x15 (PortInfo)
resv....................0x0
attr_mod................0x17
m_key...................0x0000000000000000
dr_slid.................0xFFFF
dr_dlid.................0xFFFF
Initial path: [0][1][12][1][4]
Return path: [0][9][14][D][1]
Reserved: [0][0][0][0][0][0][0]
00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
00 00 00 00 00 00 00 00 00 00 00 00 01 03 03 02
14 52 00 11 40 40 00 08 08 04 F2 40 00 00 00 00
00 00 00 00 00 88 00 00 00 00 00 00 00 00 00 00
Mar 20 14:30:12 043367 [41E02960] -> PortInfo dump:
port number.............0x16
node_guid...............0x0005ad00000281b3
port_guid...............0x0005ad00000281b3
m_key...................0x0000000000000000
subnet_prefix...........0x0000000000000000
base_lid................0x0
master_sm_base_lid......0x0
capability_mask.........0x0
diag_code...............0x0
m_key_lease_period......0x0
local_port_num..........0x1
link_width_enabled......0x3
link_width_supported....0x3
link_width_active.......0x2
link_speed_supported....0x1
port_state..............ACTIVE
state_info2.............0x52
m_key_protect_bits......0x0
lmc.....................0x0
link_speed..............0x11
mtu_smsl................0x40
vl_cap_init_type........0x40
vl_high_limit...........0x0
vl_arb_high_cap.........0x8
vl_arb_low_cap..........0x8
init_rep_mtu_cap........0x4
vl_stall_life...........0xF2
vl_enforce..............0x40
m_key_violations........0x0
p_key_violations........0x0
q_key_violations........0x0
guid_cap................0x0
client_reregister.......0x0
subnet_timeout..........0x0
resp_time_value.........0x0
error_threshold.........0x88
Mar 20 14:30:12 043422 [41E02960] -> Capabilities Mask:
Mar 20 14:30:12 043513 [43C05960] -> osm_pi_rcv_process_set: Received error
status 0x1c for SetResp() during ACTIVE transition
Mar 20 14:30:12 043518 [4780B960] -> __osm_sm_mad_ctrl_rcv_callback: ERR
3111: Error status = 0x1C00
Mar 20 14:30:12 043519 [43C05960] -> PortInfo dump:
port number.............0x17
node_guid...............0x0005ad00000281b3
port_guid...............0x0005ad00000281b3
m_key...................0x0000000000000000
subnet_prefix...........0x0000000000000000
base_lid................0x0
master_sm_base_lid......0x0
capability_mask.........0x0
diag_code...............0x0
m_key_lease_period......0x0
local_port_num..........0x1
link_width_enabled......0x3
link_width_supported....0x3
link_width_active.......0x2
link_speed_supported....0x1
port_state..............ACTIVE
state_info2.............0x52
m_key_protect_bits......0x0
lmc.....................0x0
link_speed..............0x11
mtu_smsl................0x40
vl_cap_init_type........0x40
vl_high_limit...........0x0
vl_arb_high_cap.........0x8
vl_arb_low_cap..........0x8
init_rep_mtu_cap........0x4
vl_stall_life...........0xF2
vl_enforce..............0x40
m_key_violations........0x0
p_key_violations........0x0
q_key_violations........0x0
guid_cap................0x0
client_reregister.......0x0
subnet_timeout..........0x0
resp_time_value.........0x0
error_threshold.........0x88
Mar 20 14:30:12 043535 [43C05960] -> Capabilities Mask:
Mar 20 14:30:12 043553 [4780B960] -> SMP dump:
base_ver................0x1
mgmt_class..............0x81
class_ver...............0x1
method..................0x81 (SubnGetResp)
D bit...................0x1
status..................0x1C00
hop_ptr.................0x0
hop_count...............0x4
trans_id................0x92b3e
attr_id.................0x15 (PortInfo)
resv....................0x0
attr_mod................0x18
m_key...................0x0000000000000000
dr_slid.................0xFFFF
dr_dlid.................0xFFFF
Initial path: [0][1][12][1][4]
Return path: [0][9][14][D][1]
Reserved: [0][0][0][0][0][0][0]
00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
00 00 00 00 00 00 00 00 00 00 00 00 01 03 03 02
14 52 00 11 40 40 00 08 08 04 F2 40 00 00 00 00
00 00 00 00 00 88 00 00 00 00 00 00 00 00 00 00
Mar 20 14:30:12 043658 [42803960] -> osm_pi_rcv_process_set: Received error
status 0x1c for SetResp() during ACTIVE transition
Mar 20 14:30:12 043663 [42803960] -> PortInfo dump:
port number.............0x18
node_guid...............0x0005ad00000281b3
port_guid...............0x0005ad00000281b3
m_key...................0x0000000000000000
subnet_prefix...........0x0000000000000000
base_lid................0x0
master_sm_base_lid......0x0
capability_mask.........0x0
diag_code...............0x0
m_key_lease_period......0x0
local_port_num..........0x1
link_width_enabled......0x3
link_width_supported....0x3
link_width_active.......0x2
link_speed_supported....0x1
port_state..............ACTIVE
state_info2.............0x52
m_key_protect_bits......0x0
lmc.....................0x0
link_speed..............0x11
mtu_smsl................0x40
vl_cap_init_type........0x40
vl_high_limit...........0x0
vl_arb_high_cap.........0x8
vl_arb_low_cap..........0x8
init_rep_mtu_cap........0x4
vl_stall_life...........0xF2
vl_enforce..............0x40
m_key_violations........0x0
p_key_violations........0x0
q_key_violations........0x0
guid_cap................0x0
client_reregister.......0x0
subnet_timeout..........0x0
resp_time_value.........0x0
error_threshold.........0x88
Mar 20 14:30:12 043678 [42803960] -> Capabilities Mask:
Mar 20 14:30:12 049088 [43204960] -> SUBNET UP
Mar 20 14:30:12 442903 [43C05960] -> osm_ucast_mgr_process: Min Hop Tables
configured on all switches
Mar 20 14:30:12 497312 [45007960] -> SUBNET UP
Mar 20 14:30:27 571421 [43C05960] -> __osm_trap_rcv_process_request:
Received Generic Notice type:0x01 num:128 Producer:2 from LID:0x0152
TID:0x0000000000000000
Mar 20 14:30:27 571674 [43C05960] -> osm_report_notice: Reporting Generic
Notice type:1 num:128 from LID:0x0152
GID:0xfe80000000000000,0x0005ad0000027c84
Mar 20 14:30:27 782498 [45A08960] -> __osm_trap_rcv_process_request:
Received Generic Notice type:0x01 num:128 Producer:2 from LID:0x0152
TID:0x0000000000000001
Mar 20 14:30:27 782616 [45A08960] -> osm_report_notice: Reporting Generic
Notice type:1 num:128 from LID:0x0152
GID:0xfe80000000000000,0x0005ad0000027c84
Mar 20 14:30:27 804302 [42803960] -> __osm_trap_rcv_process_request:
Received Generic Notice type:0x01 num:128 Producer:2 from LID:0x0152
TID:0x0000000000000002
Mar 20 14:30:27 805103 [42803960] -> osm_report_notice: Reporting Generic
Notice type:1 num:128 from LID:0x0152
GID:0xfe80000000000000,0x0005ad0000027c84
Mar 20 14:30:27 924983 [41E02960] -> __osm_trap_rcv_process_request:
Received Generic Notice type:0x01 num:128 Producer:2 from LID:0x0152
TID:0x0000000000000003
Mar 20 14:30:27 925088 [41E02960] -> osm_report_notice: Reporting Generic
Notice type:1 num:128 from LID:0x0152
GID:0xfe80000000000000,0x0005ad0000027c84
Mar 20 14:30:27 934314 [43204960] -> osm_report_notice: Reporting Generic
Notice type:3 num:65 from LID:0x0092
GID:0xfe80000000000000,0x0005ad0000024bbb
Mar 20 14:30:27 934327 [43204960] -> Removed port with
GUID:0x0005ad00000249d3 LID range [0xB1,0xB1] of node:saguaro-23-2 HCA-1
Mar 20 14:30:27 969077 [43204960] -> osm_ucast_mgr_process: Min Hop Tables
configured on all switches
Mar 20 14:30:28 017451 [41E02960] -> SUBNET UP
Mar 20 14:30:28 030947 [45007960] -> __osm_trap_rcv_process_request:
Received Generic Notice type:0x01 num:128 Producer:2 from LID:0x0152
TID:0x0000000000000004
Mar 20 14:30:28 031177 [45007960] -> osm_report_notice: Reporting Generic
Notice type:1 num:128 from LID:0x0152
GID:0xfe80000000000000,0x0005ad0000027c84
Mar 20 14:30:28 120040 [42803960] -> __osm_trap_rcv_process_request:
Received Generic Notice type:0x01 num:128 Producer:2 from LID:0x0152
TID:0x0000000000000005
Mar 20 14:30:28 120190 [42803960] -> osm_report_notice: Reporting Generic
Notice type:1 num:128 from LID:0x0152
GID:0xfe80000000000000,0x0005ad0000027c84
Mar 20 14:30:28 148805 [41E02960] -> __osm_trap_rcv_process_request:
Received Generic Notice type:0x01 num:128 Producer:2 from LID:0x0152
TID:0x0000000000000006
Mar 20 14:30:28 149108 [41E02960] -> osm_report_notice: Reporting Generic
Notice type:1 num:128 from LID:0x0152
GID:0xfe80000000000000,0x0005ad0000027c84
Mar 20 14:30:28 170453 [41401960] -> __osm_trap_rcv_process_request:
Received Generic Notice type:0x01 num:128 Producer:2 from LID:0x0152
TID:0x0000000000000007
Mar 20 14:30:28 170971 [41401960] -> osm_report_notice: Reporting Generic
Notice type:1 num:128 from LID:0x0152
GID:0xfe80000000000000,0x0005ad0000027c84
Mar 20 14:30:28 336861 [43C05960] -> osm_report_notice: Reporting Generic
Notice type:3 num:65 from LID:0x0092
GID:0xfe80000000000000,0x0005ad0000024bbb
Mar 20 14:30:28 336884 [43C05960] -> Removed port with
GUID:0x0005ad0000024e0b LID range [0xB3,0xB3] of node:saguaro-23-4 HCA-1
Mar 20 14:30:28 336910 [43C05960] -> osm_report_notice: Reporting Generic
Notice type:3 num:65 from LID:0x0092
GID:0xfe80000000000000,0x0005ad0000024bbb
Mar 20 14:30:28 336916 [43C05960] -> Removed port with
GUID:0x0005ad0000025043 LID range [0xB4,0xB4] of node:saguaro-23-5 HCA-1
Mar 20 14:30:28 336945 [43C05960] -> osm_report_notice: Reporting Generic
Notice type:3 num:65 from LID:0x0092
GID:0xfe80000000000000,0x0005ad0000024bbb
Mar 20 14:30:28 336951 [43C05960] -> Removed port with
GUID:0x0005ad0000024cbb LID range [0xB2,0xB2] of node:saguaro-23-3 HCA-1
Mar 20 14:30:28 371497 [43C05960] -> osm_ucast_mgr_process: Min Hop Tables
configured on all switches
Mar 20 14:30:28 410709 [41401960] -> __osm_trap_rcv_process_request:
Received Generic Notice type:0x01 num:128 Producer:2 from LID:0x0152
TID:0x0000000000000008
Mar 20 14:30:28 410894 [41401960] -> osm_report_notice: Reporting Generic
Notice type:1 num:128 from LID:0x0152
GID:0xfe80000000000000,0x0005ad0000027c84
Mar 20 14:30:28 415926 [43C05960] -> __osm_trap_rcv_process_request:
Received Generic Notice type:0x01 num:128 Producer:2 from LID:0x0152
TID:0x0000000000000009
Mar 20 14:30:28 419624 [43C05960] -> osm_report_notice: Reporting Generic
Notice type:1 num:128 from LID:0x0152
GID:0xfe80000000000000,0x0005ad0000027c84
Mar 20 14:30:28 426978 [45A08960] -> SUBNET UP
Mar 20 14:30:28 438003 [41E02960] -> __osm_trap_rcv_process_request:
Received Generic Notice type:0x01 num:128 Producer:2 from LID:0x0152
TID:0x000000000000000a
Mar 20 14:30:28 438182 [41E02960] -> osm_report_notice: Reporting Generic
Notice type:1 num:128 from LID:0x0152
GID:0xfe80000000000000,0x0005ad0000027c84
Mar 20 14:30:28 470141 [41E02960] -> __osm_trap_rcv_process_request:
Received Generic Notice type:0x01 num:128 Producer:2 from LID:0x0152
TID:0x000000000000000b
Mar 20 14:30:28 470197 [41E02960] -> __osm_trap_rcv_process_request: ERR
3804: Received trap 11 times consecutively
Mar 20 14:30:28 652535 [42803960] -> __osm_trap_rcv_process_request:
Received Generic Notice type:0x01 num:128 Producer:2 from LID:0x0152
TID:0x000000000000000c
Mar 20 14:30:28 652623 [42803960] -> __osm_trap_rcv_process_request: ERR
3804: Received trap 12 times consecutively
Mar 20 14:30:28 681514 [43C05960] -> __osm_trap_rcv_process_request:
Received Generic Notice type:0x01 num:128 Producer:2 from LID:0x0152
TID:0x000000000000000d
Mar 20 14:30:28 681636 [43C05960] -> __osm_trap_rcv_process_request: ERR
3804: Received trap 13 times consecutively
Mar 20 14:30:28 703052 [44606960] -> __osm_trap_rcv_process_request:
Received Generic Notice type:0x01 num:128 Producer:2 from LID:0x0152
TID:0x000000000000000e
Mar 20 14:30:28 703092 [44606960] -> __osm_trap_rcv_process_request: ERR
3804: Received trap 14 times consecutively
Mar 20 14:30:28 724753 [43C05960] -> __osm_trap_rcv_process_request:
Received Generic Notice type:0x01 num:128 Producer:2 from LID:0x0152
TID:0x000000000000000f
Mar 20 14:30:28 724809 [43C05960] -> __osm_trap_rcv_process_request: ERR
3804: Received trap 15 times consecutively
Mar 20 14:30:28 855519 [42803960] -> __osm_trap_rcv_process_request:
Received Generic Notice type:0x01 num:128 Producer:2 from LID:0x0152
TID:0x0000000000000010
Mar 20 14:30:28 855671 [42803960] -> __osm_trap_rcv_process_request: ERR
3804: Received trap 16 times consecutively
Mar 20 14:30:28 877289 [45A08960] -> __osm_trap_rcv_process_request:
Received Generic Notice type:0x01 num:128 Producer:2 from LID:0x0152
TID:0x0000000000000011
Mar 20 14:30:28 877354 [45A08960] -> __osm_trap_rcv_process_request: ERR
3804: Received trap 17 times consecutively
Mar 20 14:30:28 899021 [45A08960] -> __osm_trap_rcv_process_request:
Received Generic Notice type:0x01 num:128 Producer:2 from LID:0x0152
TID:0x0000000000000012
Mar 20 14:30:28 899062 [45A08960] -> __osm_trap_rcv_process_request: ERR
3804: Received trap 18 times consecutively
Mar 20 14:30:29 006886 [45007960] -> __osm_trap_rcv_process_request:
Received Generic Notice type:0x01 num:128 Producer:2 from LID:0x0152
TID:0x0000000000000013
Mar 20 14:30:29 006950 [45007960] -> __osm_trap_rcv_process_request: ERR
3804: Received trap 19 times consecutively
Mar 20 14:30:29 099965 [44606960] -> __osm_trap_rcv_process_request:
Received Generic Notice type:0x01 num:128 Producer:2 from LID:0x0152
TID:0x0000000000000014
Mar 20 14:30:29 100020 [44606960] -> __osm_trap_rcv_process_request: ERR
3804: Received trap 20 times consecutively
Mar 20 14:30:29 146532 [41E02960] -> __osm_trap_rcv_process_request:
Received Generic Notice type:0x01 num:128 Producer:2 from LID:0x0152
TID:0x0000000000000015
Mar 20 14:30:29 146578 [41E02960] -> __osm_trap_rcv_process_request: ERR
3804: Received trap 21 times consecutively
Mar 20 14:30:29 356891 [43C05960] -> __osm_trap_rcv_process_request:
Received Generic Notice type:0x01 num:128 Producer:2 from LID:0x0152
TID:0x0000000000000016
Mar 20 14:30:29 356938 [43C05960] -> __osm_trap_rcv_process_request: ERR
3804: Received trap 22 times consecutively
Mar 20 14:30:29 383112 [43204960] -> __osm_trap_rcv_process_request:
Received Generic Notice type:0x01 num:128 Producer:2 from LID:0x0152
TID:0x0000000000000017
Mar 20 14:30:29 383157 [43204960] -> __osm_trap_rcv_process_request: ERR
3804: Received trap 23 times consecutively
Mar 20 14:30:29 383710 [41401960] -> __osm_trap_rcv_process_request:
Received Generic Notice type:0x01 num:128 Producer:2 from LID:0x0148
TID:0x0000000000000079
Mar 20 14:30:29 383790 [41401960] -> osm_report_notice: Reporting Generic
Notice type:1 num:128 from LID:0x0148
GID:0xfe80000000000000,0x0005ad00000281b3
Mar 20 14:30:29 407890 [42803960] -> __osm_trap_rcv_process_request:
Received Generic Notice type:0x01 num:128 Producer:2 from LID:0x0152
TID:0x0000000000000018
Mar 20 14:30:29 407935 [42803960] -> __osm_trap_rcv_process_request: ERR
3804: Received trap 24 times consecutively
Mar 20 14:30:29 429653 [45A08960] -> __osm_trap_rcv_process_request:
Received Generic Notice type:0x01 num:128 Producer:2 from LID:0x0152
TID:0x0000000000000019
Mar 20 14:30:29 429700 [45A08960] -> __osm_trap_rcv_process_request: ERR
3804: Received trap 25 times consecutively
Mar 20 14:30:29 451352 [45007960] -> __osm_trap_rcv_process_request:
Received Generic Notice type:0x01 num:128 Producer:2 from LID:0x0152
TID:0x000000000000001a
Mar 20 14:30:29 451401 [45007960] -> __osm_trap_rcv_process_request: ERR
3804: Received trap 26 times consecutively
Mar 20 14:30:29 479843 [4780B960] -> umad_receiver: ERR 5409: send completed
with error (method=0x1 attr=0x11 trans_id=0x124ef00095cbf) -- dropping
Mar 20 14:30:29 479855 [4780B960] -> umad_receiver: ERR 5411: DR SMP
Mar 20 14:30:29 479865 [4780B960] -> __osm_sm_mad_ctrl_send_err_cb: ERR
3113: MAD completed in error (IB_TIMEOUT)
Mar 20 14:30:29 479901 [4780B960] -> SMP dump:
base_ver................0x1
mgmt_class..............0x81
class_ver...............0x1
method..................0x1 (SubnGet)
D bit...................0x0
status..................0x0
hop_ptr.................0x0
hop_count...............0x6
trans_id................0x95cbf
attr_id.................0x11 (NodeInfo)
resv....................0x0
attr_mod................0x0
m_key...................0x0000000000000000
dr_slid.................0xFFFF
dr_dlid.................0xFFFF
Initial path: [0][1][11][1][5][17][C]
Return path: [0][0][0][0][0][0][0]
Reserved: [0][0][0][0][0][0][0]
00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
Mar 20 14:30:29 480017 [44606960] -> osm_report_notice: Reporting Generic
Notice type:3 num:65 from LID:0x0092
GID:0xfe80000000000000,0x0005ad0000024bbb
Mar 20 14:30:29 480030 [44606960] -> Removed port with
GUID:0x0005ad000002510b LID range [0xB5,0xB5] of node:saguaro-23-6 HCA-1
Mar 20 14:30:29 480092 [44606960] -> osm_report_notice: Reporting Generic
Notice type:3 num:65 from LID:0x0092
GID:0xfe80000000000000,0x0005ad0000024bbb
Mar 20 14:30:29 480099 [44606960] -> Removed port with
GUID:0x0005ad000002511b LID range [0xA6,0xA6] of node:saguaro-22-1 HCA-1
Mar 20 14:30:29 480121 [44606960] -> osm_report_notice: Reporting Generic
Notice type:3 num:65 from LID:0x0092
GID:0xfe80000000000000,0x0005ad0000024bbb
Mar 20 14:30:29 480128 [44606960] -> Removed port with
GUID:0x0005ad0000024d47 LID range [0xB6,0xB6] of node:saguaro-23-7 HCA-1
Mar 20 14:30:29 480152 [44606960] -> osm_report_notice: Reporting Generic
Notice type:3 num:65 from LID:0x0092
GID:0xfe80000000000000,0x0005ad0000024bbb
Mar 20 14:30:29 480160 [44606960] -> Removed port with
GUID:0x0005ad0000024d8b LID range [0xB7,0xB7] of node:saguaro-23-8 HCA-1
Mar 20 14:30:29 480325 [44606960] -> osm_drop_mgr_process: ERR 0108: Unknown
remote side for node 0x0005ad0000027c84 port 12. Adding to light sweep
sampling list
Mar 20 14:30:29 480343 [44606960] -> Directed Path Dump of 5 hop path:
Path = [0][1][11][1][5][17]
Mar 20 14:30:29 665327 [41E02960] -> __osm_trap_rcv_process_request:
Received Generic Notice type:0x01 num:128 Producer:2 from LID:0x0148
TID:0x000000000000007a
Mar 20 14:30:29 665355 [43C05960] -> __osm_trap_rcv_process_request:
Received Generic Notice type:0x01 num:128 Producer:2 from LID:0x0152
TID:0x000000000000001b
Mar 20 14:30:29 665397 [41E02960] -> osm_report_notice: Reporting Generic
Notice type:1 num:128 from LID:0x0148
GID:0xfe80000000000000,0x0005ad00000281b3
Mar 20 14:30:29 665404 [43C05960] -> __osm_trap_rcv_process_request: ERR
3804: Received trap 27 times consecutively
Mar 20 14:30:29 680658 [45A08960] -> osm_report_notice: Reporting Generic
Notice type:3 num:64 from LID:0x0092
GID:0xfe80000000000000,0x0005ad0000024bbb
Mar 20 14:30:29 680668 [45A08960] -> Discovered new port with
GUID:0x0005ad00000249d3 LID range [0xB1,0xB1] of node:saguaro-23-2 HCA-1
Mar 20 14:30:29 680672 [45A08960] -> osm_report_notice: Reporting Generic
Notice type:3 num:64 from LID:0x0092
GID:0xfe80000000000000,0x0005ad0000024bbb
Mar 20 14:30:29 680678 [45A08960] -> Discovered new port with
GUID:0x0005ad0000024cbb LID range [0xB2,0xB2] of node:saguaro-23-3 HCA-1
Mar 20 14:30:29 680681 [45A08960] -> osm_report_notice: Reporting Generic
Notice type:3 num:64 from LID:0x0092
GID:0xfe80000000000000,0x0005ad0000024bbb
Mar 20 14:30:29 680686 [45A08960] -> Discovered new port with
GUID:0x0005ad0000024e0b LID range [0xB3,0xB3] of node:saguaro-23-4 HCA-1
Mar 20 14:30:29 680690 [45A08960] -> osm_report_notice: Reporting Generic
Notice type:3 num:64 from LID:0x0092
GID:0xfe80000000000000,0x0005ad0000024bbb
Mar 20 14:30:29 680695 [45A08960] -> Discovered new port with
GUID:0x0005ad0000025043 LID range [0xB4,0xB4] of node:saguaro-23-5 HCA-1
Mar 20 14:30:29 711542 [45A08960] -> osm_ucast_mgr_process: Min Hop Tables
configured on all switches
Mar 20 14:30:29 768280 [41401960] -> SUBNET UP
Mar 20 14:30:30 113195 [45A08960] -> osm_report_notice: Reporting Generic
Notice type:3 num:64 from LID:0x0092
GID:0xfe80000000000000,0x0005ad0000024bbb
Mar 20 14:30:30 113206 [45A08960] -> Discovered new port with
GUID:0x0005ad000002510b LID range [0xB5,0xB5] of node:saguaro-23-6 HCA-1
Mar 20 14:30:30 113211 [45A08960] -> osm_report_notice: Reporting Generic
Notice type:3 num:64 from LID:0x0092
GID:0xfe80000000000000,0x0005ad0000024bbb
Mar 20 14:30:30 113216 [45A08960] -> Discovered new port with
GUID:0x0005ad0000024d47 LID range [0xB6,0xB6] of node:saguaro-23-7 HCA-1
Mar 20 14:30:30 113220 [45A08960] -> osm_report_notice: Reporting Generic
Notice type:3 num:64 from LID:0x0092
GID:0xfe80000000000000,0x0005ad0000024bbb
Mar 20 14:30:30 113225 [45A08960] -> Discovered new port with
GUID:0x0005ad0000024d8b LID range [0xB7,0xB7] of node:saguaro-23-8 HCA-1
Mar 20 14:30:30 113228 [45A08960] -> osm_report_notice: Reporting Generic
Notice type:3 num:64 from LID:0x0092
GID:0xfe80000000000000,0x0005ad0000024bbb
Mar 20 14:30:30 113233 [45A08960] -> Discovered new port with
GUID:0x0005ad000002511b LID range [0xA6,0xA6] of node:saguaro-22-1 HCA-1
Mar 20 14:30:30 144149 [45A08960] -> osm_ucast_mgr_process: Min Hop Tables
configured on all switches
Mar 20 14:30:30 195765 [4780B960] -> __osm_sm_mad_ctrl_rcv_callback: ERR
3111: Error status = 0x1C00
Mar 20 14:30:30 195850 [4780B960] -> SMP dump:
base_ver................0x1
mgmt_class..............0x81
class_ver...............0x1
method..................0x81 (SubnGetResp)
D bit...................0x1
status..................0x1C00
hop_ptr.................0x0
hop_count...............0x4
trans_id................0x96dcd
attr_id.................0x15 (PortInfo)
resv....................0x0
attr_mod................0x16
m_key...................0x0000000000000000
dr_slid.................0xFFFF
dr_dlid.................0xFFFF
Initial path: [0][1][14][2][4]
Return path: [0][9][15][E][1]
Reserved: [0][0][0][0][0][0][0]
00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
00 00 00 00 00 00 00 00 00 00 00 00 01 03 03 02
14 52 00 11 40 40 00 08 08 04 F2 40 00 00 00 00
00 00 00 00 00 88 00 00 00 00 00 00 00 00 00 00
Mar 20 14:30:30 195929 [43C05960] -> osm_pi_rcv_process_set: Received error
status 0x1c for SetResp() during ACTIVE transition
Mar 20 14:30:30 195942 [43C05960] -> PortInfo dump:
port number.............0x16
node_guid...............0x0005ad00000281b3
port_guid...............0x0005ad00000281b3
m_key...................0x0000000000000000
subnet_prefix...........0x0000000000000000
base_lid................0x0
master_sm_base_lid......0x0
capability_mask.........0x0
diag_code...............0x0
m_key_lease_period......0x0
local_port_num..........0x1
link_width_enabled......0x3
link_width_supported....0x3
link_width_active.......0x2
link_speed_supported....0x1
port_state..............ACTIVE
state_info2.............0x52
m_key_protect_bits......0x0
lmc.....................0x0
link_speed..............0x11
mtu_smsl................0x40
vl_cap_init_type........0x40
vl_high_limit...........0x0
vl_arb_high_cap.........0x8
vl_arb_low_cap..........0x8
init_rep_mtu_cap........0x4
vl_stall_life...........0xF2
vl_enforce..............0x40
m_key_violations........0x0
p_key_violations........0x0
q_key_violations........0x0
guid_cap................0x0
client_reregister.......0x0
subnet_timeout..........0x0
resp_time_value.........0x0
error_threshold.........0x88
Mar 20 14:30:30 195968 [43C05960] -> Capabilities Mask:
Mar 20 14:30:30 196144 [4780B960] -> __osm_sm_mad_ctrl_rcv_callback: ERR
3111: Error status = 0x1C00
Mar 20 14:30:30 196179 [4780B960] -> SMP dump:
base_ver................0x1
mgmt_class..............0x81
class_ver...............0x1
method..................0x81 (SubnGetResp)
D bit...................0x1
status..................0x1C00
hop_ptr.................0x0
hop_count...............0x4
trans_id................0x96dce
attr_id.................0x15 (PortInfo)
resv....................0x0
attr_mod................0x17
m_key...................0x0000000000000000
dr_slid.................0xFFFF
dr_dlid.................0xFFFF
Initial path: [0][1][14][2][4]
Return path: [0][9][15][E][1]
Reserved: [0][0][0][0][0][0][0]
00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
00 00 00 00 00 00 00 00 00 00 00 00 01 03 03 02
14 52 00 11 40 40 00 08 08 04 F2 40 00 00 00 00
00 00 00 00 00 88 00 00 00 00 00 00 00 00 00 00
Mar 20 14:30:30 196248 [43204960] -> osm_pi_rcv_process_set: Received error
status 0x1c for SetResp() during ACTIVE transition
Mar 20 14:30:30 196254 [43204960] -> PortInfo dump:
port number.............0x17
node_guid...............0x0005ad00000281b3
port_guid...............0x0005ad00000281b3
m_key...................0x0000000000000000
subnet_prefix...........0x0000000000000000
base_lid................0x0
master_sm_base_lid......0x0
capability_mask.........0x0
diag_code...............0x0
m_key_lease_period......0x0
local_port_num..........0x1
link_width_enabled......0x3
link_width_supported....0x3
link_width_active.......0x2
link_speed_supported....0x1
port_state..............ACTIVE
state_info2.............0x52
m_key_protect_bits......0x0
lmc.....................0x0
link_speed..............0x11
mtu_smsl................0x40
vl_cap_init_type........0x40
vl_high_limit...........0x0
vl_arb_high_cap.........0x8
vl_arb_low_cap..........0x8
init_rep_mtu_cap........0x4
vl_stall_life...........0xF2
vl_enforce..............0x40
m_key_violations........0x0
p_key_violations........0x0
q_key_violations........0x0
guid_cap................0x0
client_reregister.......0x0
subnet_timeout..........0x0
resp_time_value.........0x0
error_threshold.........0x88
Mar 20 14:30:30 196269 [43204960] -> Capabilities Mask:
Mar 20 14:30:30 201633 [45007960] -> SUBNET UP
Mar 20 14:30:30 278051 [43C05960] -> __osm_trap_rcv_process_request:
Received Generic Notice type:0x01 num:128 Producer:2 from LID:0x0152
TID:0x000000000000001c
Mar 20 14:30:30 278107 [43C05960] -> __osm_trap_rcv_process_request: ERR
3804: Received trap 28 times consecutively
Mar 20 14:30:30 278656 [41E02960] -> __osm_trap_rcv_process_request:
Received Generic Notice type:0x01 num:128 Producer:2 from LID:0x0148
TID:0x000000000000007b
Mar 20 14:30:30 278871 [41E02960] -> osm_report_notice: Reporting Generic
Notice type:1 num:128 from LID:0x0148
GID:0xfe80000000000000,0x0005ad00000281b3
Mar 20 14:30:30 279653 [45007960] -> __osm_trap_rcv_process_request:
Received Generic Notice type:0x01 num:128 Producer:2 from LID:0x001B
TID:0x000000000000008d
Mar 20 14:30:30 279765 [45007960] -> osm_report_notice: Reporting Generic
Notice type:1 num:128 from LID:0x001B
GID:0xfe80000000000000,0x0005ad00000281a7
Mar 20 14:30:30 568539 [43C05960] -> __osm_trap_rcv_process_request:
Received Generic Notice type:0x01 num:128 Producer:2 from LID:0x001B
TID:0x000000000000008e
Mar 20 14:30:30 568617 [43C05960] -> osm_report_notice: Reporting Generic
Notice type:1 num:128 from LID:0x001B
GID:0xfe80000000000000,0x0005ad00000281a7
Mar 20 14:30:30 607916 [45A08960] -> __osm_trap_rcv_process_request:
Received Generic Notice type:0x01 num:128 Producer:2 from LID:0x0148
TID:0x000000000000007c
Mar 20 14:30:30 625139 [44606960] -> osm_ucast_mgr_process: Min Hop Tables
configured on all switches
Mar 20 14:30:30 663838 [45A08960] -> osm_report_notice: Reporting Generic
Notice type:1 num:128 from LID:0x0148
GID:0xfe80000000000000,0x0005ad00000281b3
Mar 20 14:30:30 664569 [44606960] -> __osm_trap_rcv_process_request:
Received Generic Notice type:0x01 num:128 Producer:2 from LID:0x001B
TID:0x000000000000008f
Mar 20 14:30:30 664747 [44606960] -> osm_report_notice: Reporting Generic
Notice type:1 num:128 from LID:0x001B
GID:0xfe80000000000000,0x0005ad00000281a7
Mar 20 14:30:30 679262 [45A08960] -> SUBNET UP
Mar 20 14:30:30 784024 [43204960] -> __osm_trap_rcv_process_request:
Received Generic Notice type:0x01 num:128 Producer:2 from LID:0x001B
TID:0x0000000000000090
Mar 20 14:30:30 784123 [43204960] -> osm_report_notice: Reporting Generic
Notice type:1 num:128 from LID:0x001B
GID:0xfe80000000000000,0x0005ad00000281a7
Mar 20 14:30:30 804217 [41401960] -> __osm_trap_rcv_process_request:
Received Generic Notice type:0x01 num:128 Producer:2 from LID:0x001B
TID:0x0000000000000091
Mar 20 14:30:30 807807 [41401960] -> osm_report_notice: Reporting Generic
Notice type:1 num:128 from LID:0x001B
GID:0xfe80000000000000,0x0005ad00000281a7
Mar 20 14:30:30 825500 [45A08960] -> __osm_trap_rcv_process_request:
Received Generic Notice type:0x01 num:128 Producer:2 from LID:0x001B
TID:0x0000000000000092
Mar 20 14:30:30 825600 [45A08960] -> osm_report_notice: Reporting Generic
Notice type:1 num:128 from LID:0x001B
GID:0xfe80000000000000,0x0005ad00000281a7
Mar 20 14:30:30 988887 [43C05960] -> __osm_trap_rcv_process_request:
Received Generic Notice type:0x01 num:128 Producer:2 from LID:0x001B
TID:0x0000000000000093
Mar 20 14:30:30 988978 [43C05960] -> osm_report_notice: Reporting Generic
Notice type:1 num:128 from LID:0x001B
GID:0xfe80000000000000,0x0005ad00000281a7
Mar 20 14:30:31 059298 [41401960] -> osm_ucast_mgr_process: Min Hop Tables
configured on all switches
Mar 20 14:30:31 106840 [41E02960] -> __osm_trap_rcv_process_request:
Received Generic Notice type:0x01 num:128 Producer:2 from LID:0x001B
TID:0x0000000000000094
Mar 20 14:30:31 111335 [41E02960] -> osm_report_notice: Reporting Generic
Notice type:1 num:128 from LID:0x001B
GID:0xfe80000000000000,0x0005ad00000281a7
Mar 20 14:30:31 112465 [4780B960] -> __osm_sm_mad_ctrl_rcv_callback: ERR
3111: Error status = 0x1C00
Mar 20 14:30:31 112497 [4780B960] -> SMP dump:
base_ver................0x1
mgmt_class..............0x81
class_ver...............0x1
method..................0x81 (SubnGetResp)
D bit...................0x1
status..................0x1C00
hop_ptr.................0x0
hop_count...............0x4
trans_id................0x98837
attr_id.................0x15 (PortInfo)
resv....................0x0
attr_mod................0x18
m_key...................0x0000000000000000
dr_slid.................0xFFFF
dr_dlid.................0xFFFF
Initial path: [0][1][16][1][5]
Return path: [0][9][13][D][2]
Reserved: [0][0][0][0][0][0][0]
00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
00 00 00 00 00 00 00 00 00 00 00 00 02 03 03 02
11 42 00 11 40 40 00 08 08 04 F2 40 00 00 00 00
00 00 00 00 00 88 00 00 00 00 00 00 00 00 00 00
Mar 20 14:30:31 112593 [44606960] -> osm_pi_rcv_process_set: ERR 0F10:
Received error status for SetResp()
Mar 20 14:30:31 112627 [44606960] -> PortInfo dump:
port number.............0x18
node_guid...............0x0005ad00000281a7
port_guid...............0x0005ad00000281a7
m_key...................0x0000000000000000
subnet_prefix...........0x0000000000000000
base_lid................0x0
master_sm_base_lid......0x0
capability_mask.........0x0
diag_code...............0x0
m_key_lease_period......0x0
local_port_num..........0x2
link_width_enabled......0x3
link_width_supported....0x3
link_width_active.......0x2
link_speed_supported....0x1
port_state..............DOWN
state_info2.............0x42
m_key_protect_bits......0x0
lmc.....................0x0
link_speed..............0x11
mtu_smsl................0x40
vl_cap_init_type........0x40
vl_high_limit...........0x0
vl_arb_high_cap.........0x8
vl_arb_low_cap..........0x8
init_rep_mtu_cap........0x4
vl_stall_life...........0xF2
vl_enforce..............0x40
m_key_violations........0x0
p_key_violations........0x0
q_key_violations........0x0
guid_cap................0x0
client_reregister.......0x0
subnet_timeout..........0x0
resp_time_value.........0x0
error_threshold.........0x88
Mar 20 14:30:31 112673 [44606960] -> Capabilities Mask:
Mar 20 14:30:31 113808 [4780B960] -> __osm_sm_mad_ctrl_rcv_callback: ERR
3111: Error status = 0x1C00
Mar 20 14:30:31 113838 [4780B960] -> SMP dump:
base_ver................0x1
mgmt_class..............0x81
class_ver...............0x1
method..................0x81 (SubnGetResp)
D bit...................0x1
status..................0x1C00
hop_ptr.................0x0
hop_count...............0x4
trans_id................0x9883e
attr_id.................0x15 (PortInfo)
resv....................0x0
attr_mod................0x18
m_key...................0x0000000000000000
dr_slid.................0xFFFF
dr_dlid.................0xFFFF
Initial path: [0][1][11][1][4]
Return path: [0][9][18][D][1]
Reserved: [0][0][0][0][0][0][0]
00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
00 00 00 00 00 00 00 00 00 00 00 00 01 03 03 02
14 52 00 11 40 40 00 08 08 04 F2 40 00 00 00 00
00 00 00 00 00 88 00 00 00 00 00 00 00 00 00 00
Mar 20 14:30:31 113925 [43204960] -> osm_pi_rcv_process_set: Received error
status 0x1c for SetResp() during ACTIVE transition
Mar 20 14:30:31 113930 [43204960] -> PortInfo dump:
port number.............0x18
node_guid...............0x0005ad00000281b3
port_guid...............0x0005ad00000281b3
m_key...................0x0000000000000000
subnet_prefix...........0x0000000000000000
base_lid................0x0
master_sm_base_lid......0x0
capability_mask.........0x0
diag_code...............0x0
m_key_lease_period......0x0
local_port_num..........0x1
link_width_enabled......0x3
link_width_supported....0x3
link_width_active.......0x2
link_speed_supported....0x1
port_state..............ACTIVE
state_info2.............0x52
m_key_protect_bits......0x0
lmc.....................0x0
link_speed..............0x11
mtu_smsl................0x40
vl_cap_init_type........0x40
vl_high_limit...........0x0
vl_arb_high_cap.........0x8
vl_arb_low_cap..........0x8
init_rep_mtu_cap........0x4
vl_stall_life...........0xF2
vl_enforce..............0x40
m_key_violations........0x0
p_key_violations........0x0
q_key_violations........0x0
guid_cap................0x0
client_reregister.......0x0
subnet_timeout..........0x0
resp_time_value.........0x0
error_threshold.........0x88
Mar 20 14:30:31 113946 [43204960] -> Capabilities Mask:
Mar 20 14:30:31 119007 [43204960] -> SUBNET UP
Mar 20 14:30:31 128758 [45007960] -> __osm_trap_rcv_process_request:
Received Generic Notice type:0x01 num:128 Producer:2 from LID:0x001B
TID:0x0000000000000095
Mar 20 14:30:31 128851 [45007960] -> osm_report_notice: Reporting Generic
Notice type:1 num:128 from LID:0x001B
GID:0xfe80000000000000,0x0005ad00000281a7
Mar 20 14:30:31 150370 [44606960] -> __osm_trap_rcv_process_request:
Received Generic Notice type:0x01 num:128 Producer:2 from LID:0x001B
TID:0x0000000000000096
Mar 20 14:30:31 150468 [44606960] -> osm_report_notice: Reporting Generic
Notice type:1 num:128 from LID:0x001B
GID:0xfe80000000000000,0x0005ad00000281a7
Mar 20 14:30:31 316422 [41401960] -> __osm_trap_rcv_process_request:
Received Generic Notice type:0x01 num:128 Producer:2 from LID:0x0152
TID:0x000000000000001c
Mar 20 14:30:31 316498 [41401960] -> __osm_trap_rcv_process_request: ERR
3804: Received trap 29 times consecutively
More information about the general
mailing list