[openib-general] [ANNOUCEv2] OpenIB OpenSM 1.1.0: trunk now supports 1.8.0 features
Troy Benjegerdes
hozer at hozed.org
Tue Sep 13 17:12:36 PDT 2005
On Tue, Sep 13, 2005 at 12:19:30PM -0400, Hal Rosenstock wrote:
> On Tue, 2005-09-13 at 12:15, Troy Benjegerdes wrote:
> > We just had a node crash on our network, and it caused our OpenSM to
> > stop working.. we were running version openib-1.0.0..
>
> Can you define stop working (more details) ? Are there any logs ?
>
> > I suppose this means I should start beating up on 1.1.0 now, right?
>
> Yes but the same issue might still exist. Can you reproduce it on the
> OpenSM you are running on now and then move up and see if it still
> exists ?
Stop working as in IPoIB arp seems to stop.
I've got a log now of the latest opensm-1.1.0 attached.
The time (was) off on that machine, FYI.
At the log entry 'Sep 13 12:06:55', I plugged in the node that is hung/crashed
.. which caused a bunch of opensm errors.. I have since unplugged that
node, and can put it back in tommorow if you want more debug info.
-------------- next part --------------
Sep 13 12:03:12 584304 [AB448A30] -> OpenSM Rev:openib-1.1.0
Sep 13 12:03:12 584427 [0000] -> OpenSM Rev:openib-1.1.0
Sep 13 12:03:12 585421 [AB448A30] -> osm_report_notice: Reporting Generic Notice type:3 num:66 from LID:0x0000 GID:0xfe80000000000000,0x0000000000000000
Sep 13 12:03:12 585486 [AB448A30] -> osm_report_notice: Reporting Generic Notice type:3 num:66 from LID:0x0000 GID:0xfe80000000000000,0x0000000000000000
Sep 13 12:03:12 588442 [AB448A30] -> osm_vendor_get_all_port_attr: ERR 5420: assign CA mthca0 port 1 guid (0x2c90200402781) as the default port.
Sep 13 12:03:12 588531 [AB448A30] -> osm_vendor_bind: Binding to port 0x2c90200402781.
Sep 13 12:03:12 590652 [AB448A30] -> osm_vendor_bind: Binding to port 0x2c90200402781.
Sep 13 12:03:12 605601 [417FF970] -> osm_ucast_mgr_process: Min Hop Tables configured on all switches.
Sep 13 12:03:12 612854 [0000] -> SUBNET UP
Sep 13 12:03:12 632300 [417FF970] -> __osm_trap_rcv_process_request: Received Generic Notice type:0x04 num:144 Producer:1 from LID:0x0001 TID:0x0000000000000004
Sep 13 12:03:12 632467 [417FF970] -> osm_report_notice: Reporting Generic Notice type:4 num:144 from LID:0x0001 GID:0xfe80000000000000,0x0002c90200402781
Sep 13 12:03:12 644055 [417FF970] -> osm_ucast_mgr_process: Min Hop Tables configured on all switches.
Sep 13 12:03:12 872848 [417FF970] -> __osm_trap_rcv_process_request: Received Generic Notice type:0x04 num:144 Producer:1 from LID:0x0003 TID:0x0000000000000005
Sep 13 12:03:12 872966 [417FF970] -> osm_report_notice: Reporting Generic Notice type:4 num:144 from LID:0x0003 GID:0xfe80000000000000,0x0002c9020040272d
Sep 13 12:03:12 882910 [417FF970] -> osm_ucast_mgr_process: Min Hop Tables configured on all switches.
Sep 13 12:03:13 652369 [417FF970] -> __osm_trap_rcv_process_request: Received Generic Notice type:0x01 num:128 Producer:2 from LID:0x0002 TID:0x0000000000000019
Sep 13 12:03:13 652509 [417FF970] -> osm_report_notice: Reporting Generic Notice type:1 num:128 from LID:0x0000 GID:0xfe80000000000000,0x0002c90200402917
Sep 13 12:03:13 662539 [40FFF970] -> osm_ucast_mgr_process: Min Hop Tables configured on all switches.
Sep 13 12:03:51 924918 [40FFF970] -> __osm_trap_rcv_process_request: Received Generic Notice type:0x01 num:128 Producer:2 from LID:0x0002 TID:0x000000000000001a
Sep 13 12:03:51 925056 [40FFF970] -> osm_report_notice: Reporting Generic Notice type:1 num:128 from LID:0x0000 GID:0xfe80000000000000,0x0002c90200402917
Sep 13 12:03:51 959970 [417FF970] -> osm_report_notice: Reporting Generic Notice type:3 num:64 from LID:0x0001 GID:0xfe80000000000000,0x0002c90200402781
Sep 13 12:03:51 959986 [417FF970] -> Discovered new port with GUID:0x0002c90200402915 LID range [0xE,0xE] of node:MT47396 Infiniscale-III Mellanox Technologies
Sep 13 12:03:51 959996 [417FF970] -> osm_report_notice: Reporting Generic Notice type:3 num:64 from LID:0x0001 GID:0xfe80000000000000,0x0002c90200402781
Sep 13 12:03:51 960006 [417FF970] -> Discovered new port with GUID:0x0002c90108cd0b71 LID range [0x10,0x10] of node:MT23108 InfiniHost Mellanox Technologies
Sep 13 12:03:51 960015 [417FF970] -> osm_report_notice: Reporting Generic Notice type:3 num:64 from LID:0x0001 GID:0xfe80000000000000,0x0002c90200402781
Sep 13 12:03:51 960025 [417FF970] -> Discovered new port with GUID:0x00066a00a000044e LID range [0x11,0x11] of node:MT23108 InfiniHost Mellanox Technologies
Sep 13 12:03:51 960034 [417FF970] -> osm_report_notice: Reporting Generic Notice type:3 num:64 from LID:0x0001 GID:0xfe80000000000000,0x0002c90200402781
Sep 13 12:03:51 960043 [417FF970] -> Discovered new port with GUID:0x00066a00a0000444 LID range [0xF,0xF] of node:MT23108 InfiniHost Mellanox Technologies
Sep 13 12:03:51 960052 [417FF970] -> osm_report_notice: Reporting Generic Notice type:3 num:64 from LID:0x0001 GID:0xfe80000000000000,0x0002c90200402781
Sep 13 12:03:51 960062 [417FF970] -> Discovered new port with GUID:0x0002c90108cd85f1 LID range [0xC,0xC] of node:MT23108 InfiniHost Mellanox Technologies
Sep 13 12:03:51 960071 [417FF970] -> osm_report_notice: Reporting Generic Notice type:3 num:64 from LID:0x0001 GID:0xfe80000000000000,0x0002c90200402781
Sep 13 12:03:51 960080 [417FF970] -> Discovered new port with GUID:0x0002c90108cd84a1 LID range [0xB,0xB] of node:MT23108 InfiniHost Mellanox Technologies
Sep 13 12:03:51 960089 [417FF970] -> osm_report_notice: Reporting Generic Notice type:3 num:64 from LID:0x0001 GID:0xfe80000000000000,0x0002c90200402781
Sep 13 12:03:51 960099 [417FF970] -> Discovered new port with GUID:0x00066a00a000043c LID range [0x12,0x12] of node:MT23108 InfiniHost Mellanox Technologies
Sep 13 12:03:51 960108 [417FF970] -> osm_report_notice: Reporting Generic Notice type:3 num:64 from LID:0x0001 GID:0xfe80000000000000,0x0002c90200402781
Sep 13 12:03:51 960118 [417FF970] -> Discovered new port with GUID:0x0002c90108cd9bd1 LID range [0x9,0x9] of node:MT23108 InfiniHost Mellanox Technologies
Sep 13 12:03:51 960127 [417FF970] -> osm_report_notice: Reporting Generic Notice type:3 num:64 from LID:0x0001 GID:0xfe80000000000000,0x0002c90200402781
Sep 13 12:03:51 960136 [417FF970] -> Discovered new port with GUID:0x0002c90200007c91 LID range [0xA,0xA] of node:MT23108 InfiniHost Mellanox Technologies
Sep 13 12:03:51 960145 [417FF970] -> osm_report_notice: Reporting Generic Notice type:3 num:64 from LID:0x0001 GID:0xfe80000000000000,0x0002c90200402781
Sep 13 12:03:51 960155 [417FF970] -> Discovered new port with GUID:0x00066a00a0000441 LID range [0x8,0x8] of node:MT23108 InfiniHost Mellanox Technologies
Sep 13 12:03:51 960164 [417FF970] -> osm_report_notice: Reporting Generic Notice type:3 num:64 from LID:0x0001 GID:0xfe80000000000000,0x0002c90200402781
Sep 13 12:03:51 960173 [417FF970] -> Discovered new port with GUID:0x0002c90108ccc571 LID range [0xD,0xD] of node:MT23108 InfiniHost Mellanox Technologies
Sep 13 12:03:51 960298 [417FF970] -> osm_ucast_mgr_process: Min Hop Tables configured on all switches.
Sep 13 12:03:51 979975 [417FF970] -> osm_mcmr_rcv_leave_mgrp: ERR 1B25: Received an Invalid Delete Request on MGID: 0xff12401bffff0000 : 0x0000000000000001 for PortGID: 0xfe80000000000000 : 0x0002c90108ccc571
Sep 13 12:03:51 980910 [40FFF970] -> osm_mcmr_rcv_leave_mgrp: ERR 1B25: Received an Invalid Delete Request on MGID: 0xff12401bffff0000 : 0x00000000ffffffff for PortGID: 0xfe80000000000000 : 0x0002c90108ccc571
Sep 13 12:03:52 007394 [40FFF970] -> osm_report_notice: Reporting Generic Notice type:3 num:66 from LID:0x0001 GID:0xfe80000000000000,0x0002c90200402781
Sep 13 12:03:52 009667 [417FF970] -> osm_report_notice: Reporting Generic Notice type:3 num:66 from LID:0x0001 GID:0xfe80000000000000,0x0002c90200402781
Sep 13 12:03:52 862217 [417FF970] -> __osm_trap_rcv_process_request: Received Generic Notice type:0x01 num:128 Producer:2 from LID:0x000E TID:0x0000000000000012
Sep 13 12:03:52 862341 [417FF970] -> osm_report_notice: Reporting Generic Notice type:1 num:128 from LID:0x0000 GID:0xfe80000000000000,0x0002c90200402915
Sep 13 12:03:52 889912 [417FF970] -> osm_ucast_mgr_process: Min Hop Tables configured on all switches.
Sep 13 12:06:55 936933 [417FF970] -> __osm_trap_rcv_process_request: Received Generic Notice type:0x01 num:128 Producer:2 from LID:0x000E TID:0x0000000000000013
Sep 13 12:06:55 937087 [417FF970] -> osm_report_notice: Reporting Generic Notice type:1 num:128 from LID:0x0000 GID:0xfe80000000000000,0x0002c90200402915
Sep 13 12:06:56 354422 [42FFF970] -> umad_receiver: ERR 5409: send completed with error (method=1 attr=11) -- dropping.
Sep 13 12:06:56 354439 [42FFF970] -> umad_receiver: ERR 5411: DR SMP hop ptr 0 hop count 3 DR SLID 0x0 DR DLID 0x0
Sep 13 12:06:56 354449 [42FFF970] -> __osm_sm_mad_ctrl_send_err_cb: ERR 3113: MAD completed in error (IB_TIMEOUT).
Sep 13 12:06:56 354511 [42FFF970] -> SMP dump:
base_ver................0x1
mgmt_class..............0x81
class_ver...............0x1
method..................0x1 (SubnGet)
D bit...................0x0
status..................0x0
hop_ptr.................0x0
hop_count...............0x3
trans_id................0x1611
attr_id.................0x11 (NodeInfo)
resv....................0x0
attr_mod................0x0
m_key...................0x0000000000000000
dr_slid.................0xFFFF
dr_dlid.................0xFFFF
Initial path: [0][1][D][C]
Return path: [0][0][0][0]
Reserved: [0][0][0][0][0][0][0]
00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
Sep 13 12:06:56 363771 [40FFF970] -> osm_drop_mgr_process: ERR 0108: Unknown remote side for node 0x0002c90200402915 port 12. Adding to light sweep sampling list.
Sep 13 12:06:56 363815 [40FFF970] -> Directed Path Dump of 2 hop path:
Path = [0][1][D]
Sep 13 12:06:56 364602 [40FFF970] -> osm_ucast_mgr_process: Min Hop Tables configured on all switches.
Sep 13 12:06:56 370884 [417FF970] -> __osm_trap_rcv_process_request: Received Generic Notice type:0x01 num:128 Producer:2 from LID:0x000E TID:0x0000000000000014
Sep 13 12:06:56 371013 [417FF970] -> osm_report_notice: Reporting Generic Notice type:1 num:128 from LID:0x0000 GID:0xfe80000000000000,0x0002c90200402915
Sep 13 12:06:56 794437 [42FFF970] -> umad_receiver: ERR 5409: send completed with error (method=1 attr=11) -- dropping.
Sep 13 12:06:56 794445 [42FFF970] -> umad_receiver: ERR 5411: DR SMP hop ptr 0 hop count 3 DR SLID 0x0 DR DLID 0x0
Sep 13 12:06:56 794455 [42FFF970] -> __osm_sm_mad_ctrl_send_err_cb: ERR 3113: MAD completed in error (IB_TIMEOUT).
Sep 13 12:06:56 794489 [42FFF970] -> SMP dump:
base_ver................0x1
mgmt_class..............0x81
class_ver...............0x1
method..................0x1 (SubnGet)
D bit...................0x0
status..................0x0
hop_ptr.................0x0
hop_count...............0x3
trans_id................0x16c3
attr_id.................0x11 (NodeInfo)
resv....................0x0
attr_mod................0x0
m_key...................0x0000000000000000
dr_slid.................0xFFFF
dr_dlid.................0xFFFF
Initial path: [0][1][D][C]
Return path: [0][0][0][0]
Reserved: [0][0][0][0][0][0][0]
00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
Sep 13 12:06:56 803608 [417FF970] -> osm_drop_mgr_process: ERR 0108: Unknown remote side for node 0x0002c90200402915 port 12. Adding to light sweep sampling list.
Sep 13 12:06:56 803623 [417FF970] -> Directed Path Dump of 2 hop path:
Path = [0][1][D]
Sep 13 12:06:56 804375 [417FF970] -> osm_ucast_mgr_process: Min Hop Tables configured on all switches.
Sep 13 12:06:56 810409 [40FFF970] -> __osm_trap_rcv_process_request: Received Generic Notice type:0x01 num:128 Producer:2 from LID:0x000E TID:0x0000000000000015
Sep 13 12:06:56 810557 [40FFF970] -> osm_report_notice: Reporting Generic Notice type:1 num:128 from LID:0x0000 GID:0xfe80000000000000,0x0002c90200402915
Sep 13 12:06:57 230427 [42FFF970] -> umad_receiver: ERR 5409: send completed with error (method=1 attr=11) -- dropping.
Sep 13 12:06:57 230437 [42FFF970] -> umad_receiver: ERR 5411: DR SMP hop ptr 0 hop count 3 DR SLID 0x0 DR DLID 0x0
Sep 13 12:06:57 230455 [42FFF970] -> __osm_sm_mad_ctrl_send_err_cb: ERR 3113: MAD completed in error (IB_TIMEOUT).
Sep 13 12:06:57 230490 [42FFF970] -> SMP dump:
base_ver................0x1
mgmt_class..............0x81
class_ver...............0x1
method..................0x1 (SubnGet)
D bit...................0x0
status..................0x0
hop_ptr.................0x0
hop_count...............0x3
trans_id................0x1774
attr_id.................0x11 (NodeInfo)
resv....................0x0
attr_mod................0x0
m_key...................0x0000000000000000
dr_slid.................0xFFFF
dr_dlid.................0xFFFF
Initial path: [0][1][D][C]
Return path: [0][0][0][0]
Reserved: [0][0][0][0][0][0][0]
00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
Sep 13 12:06:57 239625 [40FFF970] -> osm_drop_mgr_process: ERR 0108: Unknown remote side for node 0x0002c90200402915 port 12. Adding to light sweep sampling list.
Sep 13 12:06:57 239639 [40FFF970] -> Directed Path Dump of 2 hop path:
Path = [0][1][D]
Sep 13 12:06:57 240336 [40FFF970] -> osm_ucast_mgr_process: Min Hop Tables configured on all switches.
Sep 13 12:06:57 246604 [417FF970] -> __osm_trap_rcv_process_request: Received Generic Notice type:0x01 num:128 Producer:2 from LID:0x000E TID:0x0000000000000016
Sep 13 12:06:57 246733 [417FF970] -> osm_report_notice: Reporting Generic Notice type:1 num:128 from LID:0x0000 GID:0xfe80000000000000,0x0002c90200402915
Sep 13 12:06:57 666427 [42FFF970] -> umad_receiver: ERR 5409: send completed with error (method=1 attr=11) -- dropping.
Sep 13 12:06:57 666435 [42FFF970] -> umad_receiver: ERR 5411: DR SMP hop ptr 0 hop count 3 DR SLID 0x0 DR DLID 0x0
Sep 13 12:06:57 666445 [42FFF970] -> __osm_sm_mad_ctrl_send_err_cb: ERR 3113: MAD completed in error (IB_TIMEOUT).
Sep 13 12:06:57 666480 [42FFF970] -> SMP dump:
base_ver................0x1
mgmt_class..............0x81
class_ver...............0x1
method..................0x1 (SubnGet)
D bit...................0x0
status..................0x0
hop_ptr.................0x0
hop_count...............0x3
trans_id................0x1825
attr_id.................0x11 (NodeInfo)
resv....................0x0
attr_mod................0x0
m_key...................0x0000000000000000
dr_slid.................0xFFFF
dr_dlid.................0xFFFF
Initial path: [0][1][D][C]
Return path: [0][0][0][0]
Reserved: [0][0][0][0][0][0][0]
00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
Sep 13 12:06:57 675698 [417FF970] -> osm_drop_mgr_process: ERR 0108: Unknown remote side for node 0x0002c90200402915 port 12. Adding to light sweep sampling list.
Sep 13 12:06:57 675709 [417FF970] -> Directed Path Dump of 2 hop path:
Path = [0][1][D]
Sep 13 12:06:57 676374 [417FF970] -> osm_ucast_mgr_process: Min Hop Tables configured on all switches.
Sep 13 12:06:57 682388 [40FFF970] -> __osm_trap_rcv_process_request: Received Generic Notice type:0x01 num:128 Producer:2 from LID:0x000E TID:0x0000000000000017
Sep 13 12:06:57 682529 [40FFF970] -> osm_report_notice: Reporting Generic Notice type:1 num:128 from LID:0x0000 GID:0xfe80000000000000,0x0002c90200402915
Sep 13 12:06:58 102430 [42FFF970] -> umad_receiver: ERR 5409: send completed with error (method=1 attr=11) -- dropping.
Sep 13 12:06:58 102440 [42FFF970] -> umad_receiver: ERR 5411: DR SMP hop ptr 0 hop count 3 DR SLID 0x0 DR DLID 0x0
Sep 13 12:06:58 102450 [42FFF970] -> __osm_sm_mad_ctrl_send_err_cb: ERR 3113: MAD completed in error (IB_TIMEOUT).
Sep 13 12:06:58 102484 [42FFF970] -> SMP dump:
base_ver................0x1
mgmt_class..............0x81
class_ver...............0x1
method..................0x1 (SubnGet)
D bit...................0x0
status..................0x0
hop_ptr.................0x0
hop_count...............0x3
trans_id................0x18d6
attr_id.................0x11 (NodeInfo)
resv....................0x0
attr_mod................0x0
m_key...................0x0000000000000000
dr_slid.................0xFFFF
dr_dlid.................0xFFFF
Initial path: [0][1][D][C]
Return path: [0][0][0][0]
Reserved: [0][0][0][0][0][0][0]
00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
Sep 13 12:06:58 111732 [40FFF970] -> osm_drop_mgr_process: ERR 0108: Unknown remote side for node 0x0002c90200402915 port 12. Adding to light sweep sampling list.
Sep 13 12:06:58 111748 [40FFF970] -> Directed Path Dump of 2 hop path:
Path = [0][1][D]
Sep 13 12:06:58 112437 [40FFF970] -> osm_ucast_mgr_process: Min Hop Tables configured on all switches.
Sep 13 12:06:58 118714 [417FF970] -> __osm_trap_rcv_process_request: Received Generic Notice type:0x01 num:128 Producer:2 from LID:0x000E TID:0x0000000000000018
Sep 13 12:06:58 118852 [417FF970] -> osm_report_notice: Reporting Generic Notice type:1 num:128 from LID:0x0000 GID:0xfe80000000000000,0x0002c90200402915
Sep 13 12:06:58 538435 [42FFF970] -> umad_receiver: ERR 5409: send completed with error (method=1 attr=11) -- dropping.
Sep 13 12:06:58 538445 [42FFF970] -> umad_receiver: ERR 5411: DR SMP hop ptr 0 hop count 3 DR SLID 0x0 DR DLID 0x0
Sep 13 12:06:58 538454 [42FFF970] -> __osm_sm_mad_ctrl_send_err_cb: ERR 3113: MAD completed in error (IB_TIMEOUT).
Sep 13 12:06:58 538488 [42FFF970] -> SMP dump:
base_ver................0x1
mgmt_class..............0x81
class_ver...............0x1
method..................0x1 (SubnGet)
D bit...................0x0
status..................0x0
hop_ptr.................0x0
hop_count...............0x3
trans_id................0x1987
attr_id.................0x11 (NodeInfo)
resv....................0x0
attr_mod................0x0
m_key...................0x0000000000000000
dr_slid.................0xFFFF
dr_dlid.................0xFFFF
Initial path: [0][1][D][C]
Return path: [0][0][0][0]
Reserved: [0][0][0][0][0][0][0]
00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
Sep 13 12:06:58 547707 [417FF970] -> osm_drop_mgr_process: ERR 0108: Unknown remote side for node 0x0002c90200402915 port 12. Adding to light sweep sampling list.
Sep 13 12:06:58 547718 [417FF970] -> Directed Path Dump of 2 hop path:
Path = [0][1][D]
Sep 13 12:06:58 548381 [417FF970] -> osm_ucast_mgr_process: Min Hop Tables configured on all switches.
Sep 13 12:06:58 554395 [40FFF970] -> __osm_trap_rcv_process_request: Received Generic Notice type:0x01 num:128 Producer:2 from LID:0x000E TID:0x0000000000000019
Sep 13 12:06:58 554540 [40FFF970] -> osm_report_notice: Reporting Generic Notice type:1 num:128 from LID:0x0000 GID:0xfe80000000000000,0x0002c90200402915
Sep 13 12:06:58 974436 [42FFF970] -> umad_receiver: ERR 5409: send completed with error (method=1 attr=11) -- dropping.
Sep 13 12:06:58 974446 [42FFF970] -> umad_receiver: ERR 5411: DR SMP hop ptr 0 hop count 3 DR SLID 0x0 DR DLID 0x0
Sep 13 12:06:58 974456 [42FFF970] -> __osm_sm_mad_ctrl_send_err_cb: ERR 3113: MAD completed in error (IB_TIMEOUT).
Sep 13 12:06:58 974490 [42FFF970] -> SMP dump:
base_ver................0x1
mgmt_class..............0x81
class_ver...............0x1
method..................0x1 (SubnGet)
D bit...................0x0
status..................0x0
hop_ptr.................0x0
hop_count...............0x3
trans_id................0x1a38
attr_id.................0x11 (NodeInfo)
resv....................0x0
attr_mod................0x0
m_key...................0x0000000000000000
dr_slid.................0xFFFF
dr_dlid.................0xFFFF
Initial path: [0][1][D][C]
Return path: [0][0][0][0]
Reserved: [0][0][0][0][0][0][0]
00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
Sep 13 12:06:58 983728 [417FF970] -> osm_drop_mgr_process: ERR 0108: Unknown remote side for node 0x0002c90200402915 port 12. Adding to light sweep sampling list.
Sep 13 12:06:58 983738 [417FF970] -> Directed Path Dump of 2 hop path:
Path = [0][1][D]
Sep 13 12:06:58 984394 [417FF970] -> osm_ucast_mgr_process: Min Hop Tables configured on all switches.
Sep 13 12:06:58 990412 [40FFF970] -> __osm_trap_rcv_process_request: Received Generic Notice type:0x01 num:128 Producer:2 from LID:0x000E TID:0x000000000000001a
Sep 13 12:06:58 990544 [40FFF970] -> osm_report_notice: Reporting Generic Notice type:1 num:128 from LID:0x0000 GID:0xfe80000000000000,0x0002c90200402915
Sep 13 12:06:59 410441 [42FFF970] -> umad_receiver: ERR 5409: send completed with error (method=1 attr=11) -- dropping.
Sep 13 12:06:59 410451 [42FFF970] -> umad_receiver: ERR 5411: DR SMP hop ptr 0 hop count 3 DR SLID 0x0 DR DLID 0x0
Sep 13 12:06:59 410461 [42FFF970] -> __osm_sm_mad_ctrl_send_err_cb: ERR 3113: MAD completed in error (IB_TIMEOUT).
Sep 13 12:06:59 410496 [42FFF970] -> SMP dump:
base_ver................0x1
mgmt_class..............0x81
class_ver...............0x1
method..................0x1 (SubnGet)
D bit...................0x0
status..................0x0
hop_ptr.................0x0
hop_count...............0x3
trans_id................0x1ae9
attr_id.................0x11 (NodeInfo)
resv....................0x0
attr_mod................0x0
m_key...................0x0000000000000000
dr_slid.................0xFFFF
dr_dlid.................0xFFFF
Initial path: [0][1][D][C]
Return path: [0][0][0][0]
Reserved: [0][0][0][0][0][0][0]
00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
Sep 13 12:06:59 419699 [40FFF970] -> osm_drop_mgr_process: ERR 0108: Unknown remote side for node 0x0002c90200402915 port 12. Adding to light sweep sampling list.
Sep 13 12:06:59 419713 [40FFF970] -> Directed Path Dump of 2 hop path:
Path = [0][1][D]
Sep 13 12:06:59 420420 [40FFF970] -> osm_ucast_mgr_process: Min Hop Tables configured on all switches.
Sep 13 12:06:59 426687 [417FF970] -> __osm_trap_rcv_process_request: Received Generic Notice type:0x01 num:128 Producer:2 from LID:0x000E TID:0x000000000000001b
Sep 13 12:06:59 426817 [417FF970] -> osm_report_notice: Reporting Generic Notice type:1 num:128 from LID:0x0000 GID:0xfe80000000000000,0x0002c90200402915
Sep 13 12:06:59 846442 [42FFF970] -> umad_receiver: ERR 5409: send completed with error (method=1 attr=11) -- dropping.
Sep 13 12:06:59 846451 [42FFF970] -> umad_receiver: ERR 5411: DR SMP hop ptr 0 hop count 3 DR SLID 0x0 DR DLID 0x0
Sep 13 12:06:59 846460 [42FFF970] -> __osm_sm_mad_ctrl_send_err_cb: ERR 3113: MAD completed in error (IB_TIMEOUT).
Sep 13 12:06:59 846494 [42FFF970] -> SMP dump:
base_ver................0x1
mgmt_class..............0x81
class_ver...............0x1
method..................0x1 (SubnGet)
D bit...................0x0
status..................0x0
hop_ptr.................0x0
hop_count...............0x3
trans_id................0x1b9a
attr_id.................0x11 (NodeInfo)
resv....................0x0
attr_mod................0x0
m_key...................0x0000000000000000
dr_slid.................0xFFFF
dr_dlid.................0xFFFF
Initial path: [0][1][D][C]
Return path: [0][0][0][0]
Reserved: [0][0][0][0][0][0][0]
00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
Sep 13 12:06:59 855582 [40FFF970] -> osm_drop_mgr_process: ERR 0108: Unknown remote side for node 0x0002c90200402915 port 12. Adding to light sweep sampling list.
Sep 13 12:06:59 855596 [40FFF970] -> Directed Path Dump of 2 hop path:
Path = [0][1][D]
Sep 13 12:06:59 856268 [40FFF970] -> osm_ucast_mgr_process: Min Hop Tables configured on all switches.
Sep 13 12:06:59 862523 [417FF970] -> __osm_trap_rcv_process_request: Received Generic Notice type:0x01 num:128 Producer:2 from LID:0x000E TID:0x000000000000001c
Sep 13 12:06:59 862670 [417FF970] -> osm_report_notice: Reporting Generic Notice type:1 num:128 from LID:0x0000 GID:0xfe80000000000000,0x0002c90200402915
Sep 13 12:07:00 282446 [42FFF970] -> umad_receiver: ERR 5409: send completed with error (method=1 attr=11) -- dropping.
Sep 13 12:07:00 282458 [42FFF970] -> umad_receiver: ERR 5411: DR SMP hop ptr 0 hop count 3 DR SLID 0x0 DR DLID 0x0
Sep 13 12:07:00 282467 [42FFF970] -> __osm_sm_mad_ctrl_send_err_cb: ERR 3113: MAD completed in error (IB_TIMEOUT).
Sep 13 12:07:00 282501 [42FFF970] -> SMP dump:
base_ver................0x1
mgmt_class..............0x81
class_ver...............0x1
method..................0x1 (SubnGet)
D bit...................0x0
status..................0x0
hop_ptr.................0x0
hop_count...............0x3
trans_id................0x1c4b
attr_id.................0x11 (NodeInfo)
resv....................0x0
attr_mod................0x0
m_key...................0x0000000000000000
dr_slid.................0xFFFF
dr_dlid.................0xFFFF
Initial path: [0][1][D][C]
Return path: [0][0][0][0]
Reserved: [0][0][0][0][0][0][0]
00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
Sep 13 12:07:00 291676 [417FF970] -> osm_drop_mgr_process: ERR 0108: Unknown remote side for node 0x0002c90200402915 port 12. Adding to light sweep sampling list.
Sep 13 12:07:00 291687 [417FF970] -> Directed Path Dump of 2 hop path:
Path = [0][1][D]
Sep 13 12:07:00 292349 [417FF970] -> osm_ucast_mgr_process: Min Hop Tables configured on all switches.
Sep 13 12:07:00 298385 [40FFF970] -> __osm_trap_rcv_process_request: Received Generic Notice type:0x01 num:128 Producer:2 from LID:0x000E TID:0x000000000000001d
Sep 13 12:07:00 298519 [40FFF970] -> osm_report_notice: Reporting Generic Notice type:1 num:128 from LID:0x0000 GID:0xfe80000000000000,0x0002c90200402915
Sep 13 12:07:00 718448 [42FFF970] -> umad_receiver: ERR 5409: send completed with error (method=1 attr=11) -- dropping.
Sep 13 12:07:00 718458 [42FFF970] -> umad_receiver: ERR 5411: DR SMP hop ptr 0 hop count 3 DR SLID 0x0 DR DLID 0x0
Sep 13 12:07:00 718467 [42FFF970] -> __osm_sm_mad_ctrl_send_err_cb: ERR 3113: MAD completed in error (IB_TIMEOUT).
Sep 13 12:07:00 718502 [42FFF970] -> SMP dump:
base_ver................0x1
mgmt_class..............0x81
class_ver...............0x1
method..................0x1 (SubnGet)
D bit...................0x0
status..................0x0
hop_ptr.................0x0
hop_count...............0x3
trans_id................0x1cfc
attr_id.................0x11 (NodeInfo)
resv....................0x0
attr_mod................0x0
m_key...................0x0000000000000000
dr_slid.................0xFFFF
dr_dlid.................0xFFFF
Initial path: [0][1][D][C]
Return path: [0][0][0][0]
Reserved: [0][0][0][0][0][0][0]
00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
Sep 13 12:07:00 727677 [417FF970] -> osm_drop_mgr_process: ERR 0108: Unknown remote side for node 0x0002c90200402915 port 12. Adding to light sweep sampling list.
Sep 13 12:07:00 727688 [417FF970] -> Directed Path Dump of 2 hop path:
Path = [0][1][D]
Sep 13 12:07:00 728344 [417FF970] -> osm_ucast_mgr_process: Min Hop Tables configured on all switches.
Sep 13 12:07:00 734376 [40FFF970] -> __osm_trap_rcv_process_request: Received Generic Notice type:0x01 num:128 Producer:2 from LID:0x000E TID:0x000000000000001e
Sep 13 12:07:00 734431 [40FFF970] -> __osm_trap_rcv_process_request: ERR 3804: Received the trap 11 times continuously.
Sep 13 12:07:03 098464 [42FFF970] -> umad_receiver: ERR 5409: send completed with error (method=1 attr=15) -- dropping.
Sep 13 12:07:03 098474 [42FFF970] -> umad_receiver: ERR 5411: DR SMP hop ptr 0 hop count 3 DR SLID 0x0 DR DLID 0x0
Sep 13 12:07:03 098486 [42FFF970] -> __osm_sm_mad_ctrl_send_err_cb: ERR 3113: MAD completed in error (IB_TIMEOUT).
Sep 13 12:07:03 098520 [42FFF970] -> SMP dump:
base_ver................0x1
mgmt_class..............0x81
class_ver...............0x1
method..................0x1 (SubnGet)
D bit...................0x0
status..................0x0
hop_ptr.................0x0
hop_count...............0x3
trans_id................0x1d32
attr_id.................0x15 (PortInfo)
resv....................0x0
attr_mod................0x0
m_key...................0x0000000000000000
dr_slid.................0xFFFF
dr_dlid.................0xFFFF
Initial path: [0][1][D][C]
Return path: [0][0][0][0]
Reserved: [0][0][0][0][0][0][0]
00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
Sep 13 12:07:03 518467 [42FFF970] -> umad_receiver: ERR 5409: send completed with error (method=1 attr=11) -- dropping.
Sep 13 12:07:03 518476 [42FFF970] -> umad_receiver: ERR 5411: DR SMP hop ptr 0 hop count 3 DR SLID 0x0 DR DLID 0x0
Sep 13 12:07:03 518486 [42FFF970] -> __osm_sm_mad_ctrl_send_err_cb: ERR 3113: MAD completed in error (IB_TIMEOUT).
Sep 13 12:07:03 518520 [42FFF970] -> SMP dump:
base_ver................0x1
mgmt_class..............0x81
class_ver...............0x1
method..................0x1 (SubnGet)
D bit...................0x0
status..................0x0
hop_ptr.................0x0
hop_count...............0x3
trans_id................0x1db0
attr_id.................0x11 (NodeInfo)
resv....................0x0
attr_mod................0x0
m_key...................0x0000000000000000
dr_slid.................0xFFFF
dr_dlid.................0xFFFF
Initial path: [0][1][D][C]
Return path: [0][0][0][0]
Reserved: [0][0][0][0][0][0][0]
00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
Sep 13 12:07:03 527688 [417FF970] -> osm_drop_mgr_process: ERR 0108: Unknown remote side for node 0x0002c90200402915 port 12. Adding to light sweep sampling list.
Sep 13 12:07:03 527699 [417FF970] -> Directed Path Dump of 2 hop path:
Path = [0][1][D]
Sep 13 12:07:03 528354 [417FF970] -> osm_ucast_mgr_process: Min Hop Tables configured on all switches.
Sep 13 12:07:03 534370 [40FFF970] -> __osm_trap_rcv_process_request: Received Generic Notice type:0x01 num:128 Producer:2 from LID:0x000E TID:0x000000000000001f
Sep 13 12:07:03 534425 [40FFF970] -> __osm_trap_rcv_process_request: ERR 3804: Received the trap 12 times continuously.
Sep 13 12:07:13 102531 [42FFF970] -> umad_receiver: ERR 5409: send completed with error (method=1 attr=15) -- dropping.
Sep 13 12:07:13 102542 [42FFF970] -> umad_receiver: ERR 5411: DR SMP hop ptr 0 hop count 3 DR SLID 0x0 DR DLID 0x0
Sep 13 12:07:13 102551 [42FFF970] -> __osm_sm_mad_ctrl_send_err_cb: ERR 3113: MAD completed in error (IB_TIMEOUT).
Sep 13 12:07:13 102585 [42FFF970] -> SMP dump:
base_ver................0x1
mgmt_class..............0x81
class_ver...............0x1
method..................0x1 (SubnGet)
D bit...................0x0
status..................0x0
hop_ptr.................0x0
hop_count...............0x3
trans_id................0x1de6
attr_id.................0x15 (PortInfo)
resv....................0x0
attr_mod................0x0
m_key...................0x0000000000000000
dr_slid.................0xFFFF
dr_dlid.................0xFFFF
Initial path: [0][1][D][C]
Return path: [0][0][0][0]
Reserved: [0][0][0][0][0][0][0]
00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
Sep 13 12:07:13 522533 [42FFF970] -> umad_receiver: ERR 5409: send completed with error (method=1 attr=11) -- dropping.
Sep 13 12:07:13 522542 [42FFF970] -> umad_receiver: ERR 5411: DR SMP hop ptr 0 hop count 3 DR SLID 0x0 DR DLID 0x0
Sep 13 12:07:13 522552 [42FFF970] -> __osm_sm_mad_ctrl_send_err_cb: ERR 3113: MAD completed in error (IB_TIMEOUT).
Sep 13 12:07:13 522586 [42FFF970] -> SMP dump:
base_ver................0x1
mgmt_class..............0x81
class_ver...............0x1
method..................0x1 (SubnGet)
D bit...................0x0
status..................0x0
hop_ptr.................0x0
hop_count...............0x3
trans_id................0x1e64
attr_id.................0x11 (NodeInfo)
resv....................0x0
attr_mod................0x0
m_key...................0x0000000000000000
dr_slid.................0xFFFF
dr_dlid.................0xFFFF
Initial path: [0][1][D][C]
Return path: [0][0][0][0]
Reserved: [0][0][0][0][0][0][0]
00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
Sep 13 12:07:13 531754 [417FF970] -> osm_drop_mgr_process: ERR 0108: Unknown remote side for node 0x0002c90200402915 port 12. Adding to light sweep sampling list.
Sep 13 12:07:13 531764 [417FF970] -> Directed Path Dump of 2 hop path:
Path = [0][1][D]
Sep 13 12:07:13 532428 [417FF970] -> osm_ucast_mgr_process: Min Hop Tables configured on all switches.
Sep 13 12:07:13 538455 [40FFF970] -> __osm_trap_rcv_process_request: Received Generic Notice type:0x01 num:128 Producer:2 from LID:0x000E TID:0x0000000000000020
Sep 13 12:07:13 538583 [40FFF970] -> osm_report_notice: Reporting Generic Notice type:1 num:128 from LID:0x0000 GID:0xfe80000000000000,0x0002c90200402915
Sep 13 12:07:13 958535 [42FFF970] -> umad_receiver: ERR 5409: send completed with error (method=1 attr=11) -- dropping.
Sep 13 12:07:13 958544 [42FFF970] -> umad_receiver: ERR 5411: DR SMP hop ptr 0 hop count 3 DR SLID 0x0 DR DLID 0x0
Sep 13 12:07:13 958554 [42FFF970] -> __osm_sm_mad_ctrl_send_err_cb: ERR 3113: MAD completed in error (IB_TIMEOUT).
Sep 13 12:07:13 958588 [42FFF970] -> SMP dump:
base_ver................0x1
mgmt_class..............0x81
class_ver...............0x1
method..................0x1 (SubnGet)
D bit...................0x0
status..................0x0
hop_ptr.................0x0
hop_count...............0x3
trans_id................0x1f15
attr_id.................0x11 (NodeInfo)
resv....................0x0
attr_mod................0x0
m_key...................0x0000000000000000
dr_slid.................0xFFFF
dr_dlid.................0xFFFF
Initial path: [0][1][D][C]
Return path: [0][0][0][0]
Reserved: [0][0][0][0][0][0][0]
00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
Sep 13 12:07:13 967703 [417FF970] -> osm_drop_mgr_process: ERR 0108: Unknown remote side for node 0x0002c90200402915 port 12. Adding to light sweep sampling list.
Sep 13 12:07:13 967714 [417FF970] -> Directed Path Dump of 2 hop path:
Path = [0][1][D]
Sep 13 12:07:13 968369 [417FF970] -> osm_ucast_mgr_process: Min Hop Tables configured on all switches.
Sep 13 12:07:13 974394 [40FFF970] -> __osm_trap_rcv_process_request: Received Generic Notice type:0x01 num:128 Producer:2 from LID:0x000E TID:0x0000000000000021
Sep 13 12:07:13 974522 [40FFF970] -> osm_report_notice: Reporting Generic Notice type:1 num:128 from LID:0x0000 GID:0xfe80000000000000,0x0002c90200402915
Sep 13 12:07:14 394541 [42FFF970] -> umad_receiver: ERR 5409: send completed with error (method=1 attr=11) -- dropping.
Sep 13 12:07:14 394554 [42FFF970] -> umad_receiver: ERR 5411: DR SMP hop ptr 0 hop count 3 DR SLID 0x0 DR DLID 0x0
Sep 13 12:07:14 394563 [42FFF970] -> __osm_sm_mad_ctrl_send_err_cb: ERR 3113: MAD completed in error (IB_TIMEOUT).
Sep 13 12:07:14 394598 [42FFF970] -> SMP dump:
base_ver................0x1
mgmt_class..............0x81
class_ver...............0x1
method..................0x1 (SubnGet)
D bit...................0x0
status..................0x0
hop_ptr.................0x0
hop_count...............0x3
trans_id................0x1fc6
attr_id.................0x11 (NodeInfo)
resv....................0x0
attr_mod................0x0
m_key...................0x0000000000000000
dr_slid.................0xFFFF
dr_dlid.................0xFFFF
Initial path: [0][1][D][C]
Return path: [0][0][0][0]
Reserved: [0][0][0][0][0][0][0]
00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
Sep 13 12:07:14 403807 [40FFF970] -> osm_drop_mgr_process: ERR 0108: Unknown remote side for node 0x0002c90200402915 port 12. Adding to light sweep sampling list.
Sep 13 12:07:14 403822 [40FFF970] -> Directed Path Dump of 2 hop path:
Path = [0][1][D]
Sep 13 12:07:14 404580 [40FFF970] -> osm_ucast_mgr_process: Min Hop Tables configured on all switches.
Sep 13 12:07:14 410846 [417FF970] -> __osm_trap_rcv_process_request: Received Generic Notice type:0x01 num:128 Producer:2 from LID:0x000E TID:0x0000000000000022
Sep 13 12:07:14 410983 [417FF970] -> osm_report_notice: Reporting Generic Notice type:1 num:128 from LID:0x0000 GID:0xfe80000000000000,0x0002c90200402915
Sep 13 12:07:14 830541 [42FFF970] -> umad_receiver: ERR 5409: send completed with error (method=1 attr=11) -- dropping.
Sep 13 12:07:14 830549 [42FFF970] -> umad_receiver: ERR 5411: DR SMP hop ptr 0 hop count 3 DR SLID 0x0 DR DLID 0x0
Sep 13 12:07:14 830559 [42FFF970] -> __osm_sm_mad_ctrl_send_err_cb: ERR 3113: MAD completed in error (IB_TIMEOUT).
Sep 13 12:07:14 830592 [42FFF970] -> SMP dump:
base_ver................0x1
mgmt_class..............0x81
class_ver...............0x1
method..................0x1 (SubnGet)
D bit...................0x0
status..................0x0
hop_ptr.................0x0
hop_count...............0x3
trans_id................0x2077
attr_id.................0x11 (NodeInfo)
resv....................0x0
attr_mod................0x0
m_key...................0x0000000000000000
dr_slid.................0xFFFF
dr_dlid.................0xFFFF
Initial path: [0][1][D][C]
Return path: [0][0][0][0]
Reserved: [0][0][0][0][0][0][0]
00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
Sep 13 12:07:14 839738 [417FF970] -> osm_drop_mgr_process: ERR 0108: Unknown remote side for node 0x0002c90200402915 port 12. Adding to light sweep sampling list.
Sep 13 12:07:14 839749 [417FF970] -> Directed Path Dump of 2 hop path:
Path = [0][1][D]
Sep 13 12:07:14 840411 [417FF970] -> osm_ucast_mgr_process: Min Hop Tables configured on all switches.
Sep 13 12:07:14 846428 [40FFF970] -> __osm_trap_rcv_process_request: Received Generic Notice type:0x01 num:128 Producer:2 from LID:0x000E TID:0x0000000000000023
Sep 13 12:07:14 846560 [40FFF970] -> osm_report_notice: Reporting Generic Notice type:1 num:128 from LID:0x0000 GID:0xfe80000000000000,0x0002c90200402915
Sep 13 12:07:15 266547 [42FFF970] -> umad_receiver: ERR 5409: send completed with error (method=1 attr=11) -- dropping.
Sep 13 12:07:15 266558 [42FFF970] -> umad_receiver: ERR 5411: DR SMP hop ptr 0 hop count 3 DR SLID 0x0 DR DLID 0x0
Sep 13 12:07:15 266567 [42FFF970] -> __osm_sm_mad_ctrl_send_err_cb: ERR 3113: MAD completed in error (IB_TIMEOUT).
Sep 13 12:07:15 266601 [42FFF970] -> SMP dump:
base_ver................0x1
mgmt_class..............0x81
class_ver...............0x1
method..................0x1 (SubnGet)
D bit...................0x0
status..................0x0
hop_ptr.................0x0
hop_count...............0x3
trans_id................0x2128
attr_id.................0x11 (NodeInfo)
resv....................0x0
attr_mod................0x0
m_key...................0x0000000000000000
dr_slid.................0xFFFF
dr_dlid.................0xFFFF
Initial path: [0][1][D][C]
Return path: [0][0][0][0]
Reserved: [0][0][0][0][0][0][0]
00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
Sep 13 12:07:15 275750 [40FFF970] -> osm_drop_mgr_process: ERR 0108: Unknown remote side for node 0x0002c90200402915 port 12. Adding to light sweep sampling list.
Sep 13 12:07:15 275765 [40FFF970] -> Directed Path Dump of 2 hop path:
Path = [0][1][D]
Sep 13 12:07:15 276460 [40FFF970] -> osm_ucast_mgr_process: Min Hop Tables configured on all switches.
Sep 13 12:07:15 282727 [417FF970] -> __osm_trap_rcv_process_request: Received Generic Notice type:0x01 num:128 Producer:2 from LID:0x000E TID:0x0000000000000024
Sep 13 12:07:15 282864 [417FF970] -> osm_report_notice: Reporting Generic Notice type:1 num:128 from LID:0x0000 GID:0xfe80000000000000,0x0002c90200402915
Sep 13 12:07:15 702546 [42FFF970] -> umad_receiver: ERR 5409: send completed with error (method=1 attr=11) -- dropping.
Sep 13 12:07:15 702554 [42FFF970] -> umad_receiver: ERR 5411: DR SMP hop ptr 0 hop count 3 DR SLID 0x0 DR DLID 0x0
Sep 13 12:07:15 702564 [42FFF970] -> __osm_sm_mad_ctrl_send_err_cb: ERR 3113: MAD completed in error (IB_TIMEOUT).
Sep 13 12:07:15 702597 [42FFF970] -> SMP dump:
base_ver................0x1
mgmt_class..............0x81
class_ver...............0x1
method..................0x1 (SubnGet)
D bit...................0x0
status..................0x0
hop_ptr.................0x0
hop_count...............0x3
trans_id................0x21d9
attr_id.................0x11 (NodeInfo)
resv....................0x0
attr_mod................0x0
m_key...................0x0000000000000000
dr_slid.................0xFFFF
dr_dlid.................0xFFFF
Initial path: [0][1][D][C]
Return path: [0][0][0][0]
Reserved: [0][0][0][0][0][0][0]
00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
Sep 13 12:07:15 711774 [417FF970] -> osm_drop_mgr_process: ERR 0108: Unknown remote side for node 0x0002c90200402915 port 12. Adding to light sweep sampling list.
Sep 13 12:07:15 711785 [417FF970] -> Directed Path Dump of 2 hop path:
Path = [0][1][D]
Sep 13 12:07:15 712447 [417FF970] -> osm_ucast_mgr_process: Min Hop Tables configured on all switches.
Sep 13 12:07:15 718461 [40FFF970] -> __osm_trap_rcv_process_request: Received Generic Notice type:0x01 num:128 Producer:2 from LID:0x000E TID:0x0000000000000025
Sep 13 12:07:15 718593 [40FFF970] -> osm_report_notice: Reporting Generic Notice type:1 num:128 from LID:0x0000 GID:0xfe80000000000000,0x0002c90200402915
Sep 13 12:07:16 138551 [42FFF970] -> umad_receiver: ERR 5409: send completed with error (method=1 attr=11) -- dropping.
Sep 13 12:07:16 138561 [42FFF970] -> umad_receiver: ERR 5411: DR SMP hop ptr 0 hop count 3 DR SLID 0x0 DR DLID 0x0
Sep 13 12:07:16 138570 [42FFF970] -> __osm_sm_mad_ctrl_send_err_cb: ERR 3113: MAD completed in error (IB_TIMEOUT).
Sep 13 12:07:16 138604 [42FFF970] -> SMP dump:
base_ver................0x1
mgmt_class..............0x81
class_ver...............0x1
method..................0x1 (SubnGet)
D bit...................0x0
status..................0x0
hop_ptr.................0x0
hop_count...............0x3
trans_id................0x228a
attr_id.................0x11 (NodeInfo)
resv....................0x0
attr_mod................0x0
m_key...................0x0000000000000000
dr_slid.................0xFFFF
dr_dlid.................0xFFFF
Initial path: [0][1][D][C]
Return path: [0][0][0][0]
Reserved: [0][0][0][0][0][0][0]
00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
Sep 13 12:07:16 147713 [40FFF970] -> osm_drop_mgr_process: ERR 0108: Unknown remote side for node 0x0002c90200402915 port 12. Adding to light sweep sampling list.
Sep 13 12:07:16 147726 [40FFF970] -> Directed Path Dump of 2 hop path:
Path = [0][1][D]
Sep 13 12:07:16 148411 [40FFF970] -> osm_ucast_mgr_process: Min Hop Tables configured on all switches.
Sep 13 12:07:16 154684 [417FF970] -> __osm_trap_rcv_process_request: Received Generic Notice type:0x01 num:128 Producer:2 from LID:0x000E TID:0x0000000000000026
Sep 13 12:07:16 154813 [417FF970] -> osm_report_notice: Reporting Generic Notice type:1 num:128 from LID:0x0000 GID:0xfe80000000000000,0x0002c90200402915
Sep 13 12:07:16 574554 [42FFF970] -> umad_receiver: ERR 5409: send completed with error (method=1 attr=11) -- dropping.
Sep 13 12:07:16 574563 [42FFF970] -> umad_receiver: ERR 5411: DR SMP hop ptr 0 hop count 3 DR SLID 0x0 DR DLID 0x0
Sep 13 12:07:16 574573 [42FFF970] -> __osm_sm_mad_ctrl_send_err_cb: ERR 3113: MAD completed in error (IB_TIMEOUT).
Sep 13 12:07:16 574607 [42FFF970] -> SMP dump:
base_ver................0x1
mgmt_class..............0x81
class_ver...............0x1
method..................0x1 (SubnGet)
D bit...................0x0
status..................0x0
hop_ptr.................0x0
hop_count...............0x3
trans_id................0x233b
attr_id.................0x11 (NodeInfo)
resv....................0x0
attr_mod................0x0
m_key...................0x0000000000000000
dr_slid.................0xFFFF
dr_dlid.................0xFFFF
Initial path: [0][1][D][C]
Return path: [0][0][0][0]
Reserved: [0][0][0][0][0][0][0]
00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
Sep 13 12:07:16 583808 [417FF970] -> osm_drop_mgr_process: ERR 0108: Unknown remote side for node 0x0002c90200402915 port 12. Adding to light sweep sampling list.
Sep 13 12:07:16 583819 [417FF970] -> Directed Path Dump of 2 hop path:
Path = [0][1][D]
Sep 13 12:07:16 584481 [417FF970] -> osm_ucast_mgr_process: Min Hop Tables configured on all switches.
Sep 13 12:07:16 590518 [40FFF970] -> __osm_trap_rcv_process_request: Received Generic Notice type:0x01 num:128 Producer:2 from LID:0x000E TID:0x0000000000000027
Sep 13 12:07:16 590651 [40FFF970] -> osm_report_notice: Reporting Generic Notice type:1 num:128 from LID:0x0000 GID:0xfe80000000000000,0x0002c90200402915
Sep 13 12:07:17 010556 [42FFF970] -> umad_receiver: ERR 5409: send completed with error (method=1 attr=11) -- dropping.
Sep 13 12:07:17 010566 [42FFF970] -> umad_receiver: ERR 5411: DR SMP hop ptr 0 hop count 3 DR SLID 0x0 DR DLID 0x0
Sep 13 12:07:17 010576 [42FFF970] -> __osm_sm_mad_ctrl_send_err_cb: ERR 3113: MAD completed in error (IB_TIMEOUT).
Sep 13 12:07:17 010610 [42FFF970] -> SMP dump:
base_ver................0x1
mgmt_class..............0x81
class_ver...............0x1
method..................0x1 (SubnGet)
D bit...................0x0
status..................0x0
hop_ptr.................0x0
hop_count...............0x3
trans_id................0x23ec
attr_id.................0x11 (NodeInfo)
resv....................0x0
attr_mod................0x0
m_key...................0x0000000000000000
dr_slid.................0xFFFF
dr_dlid.................0xFFFF
Initial path: [0][1][D][C]
Return path: [0][0][0][0]
Reserved: [0][0][0][0][0][0][0]
00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
Sep 13 12:07:17 019789 [40FFF970] -> osm_drop_mgr_process: ERR 0108: Unknown remote side for node 0x0002c90200402915 port 12. Adding to light sweep sampling list.
Sep 13 12:07:17 019802 [40FFF970] -> Directed Path Dump of 2 hop path:
Path = [0][1][D]
Sep 13 12:07:17 020499 [40FFF970] -> osm_ucast_mgr_process: Min Hop Tables configured on all switches.
Sep 13 12:07:17 026766 [417FF970] -> __osm_trap_rcv_process_request: Received Generic Notice type:0x01 num:128 Producer:2 from LID:0x000E TID:0x0000000000000028
Sep 13 12:07:17 026896 [417FF970] -> osm_report_notice: Reporting Generic Notice type:1 num:128 from LID:0x0000 GID:0xfe80000000000000,0x0002c90200402915
Sep 13 12:07:17 446560 [42FFF970] -> umad_receiver: ERR 5409: send completed with error (method=1 attr=11) -- dropping.
Sep 13 12:07:17 446569 [42FFF970] -> umad_receiver: ERR 5411: DR SMP hop ptr 0 hop count 3 DR SLID 0x0 DR DLID 0x0
Sep 13 12:07:17 446579 [42FFF970] -> __osm_sm_mad_ctrl_send_err_cb: ERR 3113: MAD completed in error (IB_TIMEOUT).
Sep 13 12:07:17 446613 [42FFF970] -> SMP dump:
base_ver................0x1
mgmt_class..............0x81
class_ver...............0x1
method..................0x1 (SubnGet)
D bit...................0x0
status..................0x0
hop_ptr.................0x0
hop_count...............0x3
trans_id................0x249d
attr_id.................0x11 (NodeInfo)
resv....................0x0
attr_mod................0x0
m_key...................0x0000000000000000
dr_slid.................0xFFFF
dr_dlid.................0xFFFF
Initial path: [0][1][D][C]
Return path: [0][0][0][0]
Reserved: [0][0][0][0][0][0][0]
00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
Sep 13 12:07:17 455776 [40FFF970] -> osm_drop_mgr_process: ERR 0108: Unknown remote side for node 0x0002c90200402915 port 12. Adding to light sweep sampling list.
Sep 13 12:07:17 455790 [40FFF970] -> Directed Path Dump of 2 hop path:
Path = [0][1][D]
Sep 13 12:07:17 456467 [40FFF970] -> osm_ucast_mgr_process: Min Hop Tables configured on all switches.
Sep 13 12:07:17 462731 [417FF970] -> __osm_trap_rcv_process_request: Received Generic Notice type:0x01 num:128 Producer:2 from LID:0x000E TID:0x0000000000000029
Sep 13 12:07:17 462868 [417FF970] -> osm_report_notice: Reporting Generic Notice type:1 num:128 from LID:0x0000 GID:0xfe80000000000000,0x0002c90200402915
Sep 13 12:07:17 882562 [42FFF970] -> umad_receiver: ERR 5409: send completed with error (method=1 attr=11) -- dropping.
Sep 13 12:07:17 882570 [42FFF970] -> umad_receiver: ERR 5411: DR SMP hop ptr 0 hop count 3 DR SLID 0x0 DR DLID 0x0
Sep 13 12:07:17 882579 [42FFF970] -> __osm_sm_mad_ctrl_send_err_cb: ERR 3113: MAD completed in error (IB_TIMEOUT).
Sep 13 12:07:17 882613 [42FFF970] -> SMP dump:
base_ver................0x1
mgmt_class..............0x81
class_ver...............0x1
method..................0x1 (SubnGet)
D bit...................0x0
status..................0x0
hop_ptr.................0x0
hop_count...............0x3
trans_id................0x254e
attr_id.................0x11 (NodeInfo)
resv....................0x0
attr_mod................0x0
m_key...................0x0000000000000000
dr_slid.................0xFFFF
dr_dlid.................0xFFFF
Initial path: [0][1][D][C]
Return path: [0][0][0][0]
Reserved: [0][0][0][0][0][0][0]
00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
Sep 13 12:07:17 891770 [417FF970] -> osm_drop_mgr_process: ERR 0108: Unknown remote side for node 0x0002c90200402915 port 12. Adding to light sweep sampling list.
Sep 13 12:07:17 891781 [417FF970] -> Directed Path Dump of 2 hop path:
Path = [0][1][D]
Sep 13 12:07:17 892444 [417FF970] -> osm_ucast_mgr_process: Min Hop Tables configured on all switches.
Sep 13 12:07:17 898467 [40FFF970] -> __osm_trap_rcv_process_request: Received Generic Notice type:0x01 num:128 Producer:2 from LID:0x000E TID:0x000000000000002a
Sep 13 12:07:17 898601 [40FFF970] -> osm_report_notice: Reporting Generic Notice type:1 num:128 from LID:0x0000 GID:0xfe80000000000000,0x0002c90200402915
Sep 13 12:07:18 318566 [42FFF970] -> umad_receiver: ERR 5409: send completed with error (method=1 attr=11) -- dropping.
Sep 13 12:07:18 318577 [42FFF970] -> umad_receiver: ERR 5411: DR SMP hop ptr 0 hop count 3 DR SLID 0x0 DR DLID 0x0
Sep 13 12:07:18 318586 [42FFF970] -> __osm_sm_mad_ctrl_send_err_cb: ERR 3113: MAD completed in error (IB_TIMEOUT).
Sep 13 12:07:18 318621 [42FFF970] -> SMP dump:
base_ver................0x1
mgmt_class..............0x81
class_ver...............0x1
method..................0x1 (SubnGet)
D bit...................0x0
status..................0x0
hop_ptr.................0x0
hop_count...............0x3
trans_id................0x25ff
attr_id.................0x11 (NodeInfo)
resv....................0x0
attr_mod................0x0
m_key...................0x0000000000000000
dr_slid.................0xFFFF
dr_dlid.................0xFFFF
Initial path: [0][1][D][C]
Return path: [0][0][0][0]
Reserved: [0][0][0][0][0][0][0]
00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
Sep 13 12:07:18 327752 [40FFF970] -> osm_drop_mgr_process: ERR 0108: Unknown remote side for node 0x0002c90200402915 port 12. Adding to light sweep sampling list.
Sep 13 12:07:18 327780 [40FFF970] -> Directed Path Dump of 2 hop path:
Path = [0][1][D]
Sep 13 12:07:18 328474 [40FFF970] -> osm_ucast_mgr_process: Min Hop Tables configured on all switches.
Sep 13 12:07:18 334741 [417FF970] -> __osm_trap_rcv_process_request: Received Generic Notice type:0x01 num:128 Producer:2 from LID:0x000E TID:0x000000000000002b
Sep 13 12:07:18 334802 [417FF970] -> __osm_trap_rcv_process_request: ERR 3804: Received the trap 11 times continuously.
Sep 13 12:07:23 106596 [42FFF970] -> umad_receiver: ERR 5409: send completed with error (method=1 attr=15) -- dropping.
Sep 13 12:07:23 106605 [42FFF970] -> umad_receiver: ERR 5411: DR SMP hop ptr 0 hop count 3 DR SLID 0x0 DR DLID 0x0
Sep 13 12:07:23 106614 [42FFF970] -> __osm_sm_mad_ctrl_send_err_cb: ERR 3113: MAD completed in error (IB_TIMEOUT).
Sep 13 12:07:23 106648 [42FFF970] -> SMP dump:
base_ver................0x1
mgmt_class..............0x81
class_ver...............0x1
method..................0x1 (SubnGet)
D bit...................0x0
status..................0x0
hop_ptr.................0x0
hop_count...............0x3
trans_id................0x2635
attr_id.................0x15 (PortInfo)
resv....................0x0
attr_mod................0x0
m_key...................0x0000000000000000
dr_slid.................0xFFFF
dr_dlid.................0xFFFF
Initial path: [0][1][D][C]
Return path: [0][0][0][0]
Reserved: [0][0][0][0][0][0][0]
00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
Sep 13 12:07:23 526602 [42FFF970] -> umad_receiver: ERR 5409: send completed with error (method=1 attr=11) -- dropping.
Sep 13 12:07:23 526611 [42FFF970] -> umad_receiver: ERR 5411: DR SMP hop ptr 0 hop count 3 DR SLID 0x0 DR DLID 0x0
Sep 13 12:07:23 526623 [42FFF970] -> __osm_sm_mad_ctrl_send_err_cb: ERR 3113: MAD completed in error (IB_TIMEOUT).
Sep 13 12:07:23 526657 [42FFF970] -> SMP dump:
base_ver................0x1
mgmt_class..............0x81
class_ver...............0x1
method..................0x1 (SubnGet)
D bit...................0x0
status..................0x0
hop_ptr.................0x0
hop_count...............0x3
trans_id................0x26b3
attr_id.................0x11 (NodeInfo)
resv....................0x0
attr_mod................0x0
m_key...................0x0000000000000000
dr_slid.................0xFFFF
dr_dlid.................0xFFFF
Initial path: [0][1][D][C]
Return path: [0][0][0][0]
Reserved: [0][0][0][0][0][0][0]
00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
Sep 13 12:07:23 535916 [417FF970] -> osm_drop_mgr_process: ERR 0108: Unknown remote side for node 0x0002c90200402915 port 12. Adding to light sweep sampling list.
Sep 13 12:07:23 535926 [417FF970] -> Directed Path Dump of 2 hop path:
Path = [0][1][D]
Sep 13 12:07:23 536610 [417FF970] -> osm_ucast_mgr_process: Min Hop Tables configured on all switches.
Sep 13 12:07:23 542642 [40FFF970] -> __osm_trap_rcv_process_request: Received Generic Notice type:0x01 num:128 Producer:2 from LID:0x000E TID:0x000000000000002c
Sep 13 12:07:23 542771 [40FFF970] -> osm_report_notice: Reporting Generic Notice type:1 num:128 from LID:0x0000 GID:0xfe80000000000000,0x0002c90200402915
Sep 13 12:07:23 962601 [42FFF970] -> umad_receiver: ERR 5409: send completed with error (method=1 attr=11) -- dropping.
Sep 13 12:07:23 962611 [42FFF970] -> umad_receiver: ERR 5411: DR SMP hop ptr 0 hop count 3 DR SLID 0x0 DR DLID 0x0
Sep 13 12:07:23 962621 [42FFF970] -> __osm_sm_mad_ctrl_send_err_cb: ERR 3113: MAD completed in error (IB_TIMEOUT).
Sep 13 12:07:23 962654 [42FFF970] -> SMP dump:
base_ver................0x1
mgmt_class..............0x81
class_ver...............0x1
method..................0x1 (SubnGet)
D bit...................0x0
status..................0x0
hop_ptr.................0x0
hop_count...............0x3
trans_id................0x2764
attr_id.................0x11 (NodeInfo)
resv....................0x0
attr_mod................0x0
m_key...................0x0000000000000000
dr_slid.................0xFFFF
dr_dlid.................0xFFFF
Initial path: [0][1][D][C]
Return path: [0][0][0][0]
Reserved: [0][0][0][0][0][0][0]
00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
Sep 13 12:07:23 971816 [417FF970] -> osm_drop_mgr_process: ERR 0108: Unknown remote side for node 0x0002c90200402915 port 12. Adding to light sweep sampling list.
Sep 13 12:07:23 971826 [417FF970] -> Directed Path Dump of 2 hop path:
Path = [0][1][D]
Sep 13 12:07:23 972495 [417FF970] -> osm_ucast_mgr_process: Min Hop Tables configured on all switches.
Sep 13 12:07:23 978531 [40FFF970] -> __osm_trap_rcv_process_request: Received Generic Notice type:0x01 num:128 Producer:2 from LID:0x000E TID:0x000000000000002d
Sep 13 12:07:23 978659 [40FFF970] -> osm_report_notice: Reporting Generic Notice type:1 num:128 from LID:0x0000 GID:0xfe80000000000000,0x0002c90200402915
Sep 13 12:07:24 398606 [42FFF970] -> umad_receiver: ERR 5409: send completed with error (method=1 attr=11) -- dropping.
Sep 13 12:07:24 398616 [42FFF970] -> umad_receiver: ERR 5411: DR SMP hop ptr 0 hop count 3 DR SLID 0x0 DR DLID 0x0
Sep 13 12:07:24 398626 [42FFF970] -> __osm_sm_mad_ctrl_send_err_cb: ERR 3113: MAD completed in error (IB_TIMEOUT).
Sep 13 12:07:24 398660 [42FFF970] -> SMP dump:
base_ver................0x1
mgmt_class..............0x81
class_ver...............0x1
method..................0x1 (SubnGet)
D bit...................0x0
status..................0x0
hop_ptr.................0x0
hop_count...............0x3
trans_id................0x2815
attr_id.................0x11 (NodeInfo)
resv....................0x0
attr_mod................0x0
m_key...................0x0000000000000000
dr_slid.................0xFFFF
dr_dlid.................0xFFFF
Initial path: [0][1][D][C]
Return path: [0][0][0][0]
Reserved: [0][0][0][0][0][0][0]
00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
Sep 13 12:07:24 407894 [417FF970] -> osm_drop_mgr_process: ERR 0108: Unknown remote side for node 0x0002c90200402915 port 12. Adding to light sweep sampling list.
Sep 13 12:07:24 407904 [417FF970] -> Directed Path Dump of 2 hop path:
Path = [0][1][D]
Sep 13 12:07:24 408577 [417FF970] -> osm_ucast_mgr_process: Min Hop Tables configured on all switches.
Sep 13 12:07:24 414649 [40FFF970] -> __osm_trap_rcv_process_request: Received Generic Notice type:0x01 num:128 Producer:2 from LID:0x000E TID:0x000000000000002e
Sep 13 12:07:24 414779 [40FFF970] -> osm_report_notice: Reporting Generic Notice type:1 num:128 from LID:0x0000 GID:0xfe80000000000000,0x0002c90200402915
Sep 13 12:07:24 834608 [42FFF970] -> umad_receiver: ERR 5409: send completed with error (method=1 attr=11) -- dropping.
Sep 13 12:07:24 834617 [42FFF970] -> umad_receiver: ERR 5411: DR SMP hop ptr 0 hop count 3 DR SLID 0x0 DR DLID 0x0
Sep 13 12:07:24 834627 [42FFF970] -> __osm_sm_mad_ctrl_send_err_cb: ERR 3113: MAD completed in error (IB_TIMEOUT).
Sep 13 12:07:24 834661 [42FFF970] -> SMP dump:
base_ver................0x1
mgmt_class..............0x81
class_ver...............0x1
method..................0x1 (SubnGet)
D bit...................0x0
status..................0x0
hop_ptr.................0x0
hop_count...............0x3
trans_id................0x28c6
attr_id.................0x11 (NodeInfo)
resv....................0x0
attr_mod................0x0
m_key...................0x0000000000000000
dr_slid.................0xFFFF
dr_dlid.................0xFFFF
Initial path: [0][1][D][C]
Return path: [0][0][0][0]
Reserved: [0][0][0][0][0][0][0]
00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
Sep 13 12:07:24 843917 [40FFF970] -> osm_drop_mgr_process: ERR 0108: Unknown remote side for node 0x0002c90200402915 port 12. Adding to light sweep sampling list.
Sep 13 12:07:24 843931 [40FFF970] -> Directed Path Dump of 2 hop path:
Path = [0][1][D]
Sep 13 12:07:24 844671 [40FFF970] -> osm_ucast_mgr_process: Min Hop Tables configured on all switches.
Sep 13 12:07:24 850948 [417FF970] -> __osm_trap_rcv_process_request: Received Generic Notice type:0x01 num:128 Producer:2 from LID:0x000E TID:0x000000000000002f
Sep 13 12:07:24 851085 [417FF970] -> osm_report_notice: Reporting Generic Notice type:1 num:128 from LID:0x0000 GID:0xfe80000000000000,0x0002c90200402915
Sep 13 12:07:25 270612 [42FFF970] -> umad_receiver: ERR 5409: send completed with error (method=1 attr=11) -- dropping.
Sep 13 12:07:25 270621 [42FFF970] -> umad_receiver: ERR 5411: DR SMP hop ptr 0 hop count 3 DR SLID 0x0 DR DLID 0x0
Sep 13 12:07:25 270631 [42FFF970] -> __osm_sm_mad_ctrl_send_err_cb: ERR 3113: MAD completed in error (IB_TIMEOUT).
Sep 13 12:07:25 270665 [42FFF970] -> SMP dump:
base_ver................0x1
mgmt_class..............0x81
class_ver...............0x1
method..................0x1 (SubnGet)
D bit...................0x0
status..................0x0
hop_ptr.................0x0
hop_count...............0x3
trans_id................0x2977
attr_id.................0x11 (NodeInfo)
resv....................0x0
attr_mod................0x0
m_key...................0x0000000000000000
dr_slid.................0xFFFF
dr_dlid.................0xFFFF
Initial path: [0][1][D][C]
Return path: [0][0][0][0]
Reserved: [0][0][0][0][0][0][0]
00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
Sep 13 12:07:25 279825 [40FFF970] -> osm_drop_mgr_process: ERR 0108: Unknown remote side for node 0x0002c90200402915 port 12. Adding to light sweep sampling list.
Sep 13 12:07:25 279839 [40FFF970] -> Directed Path Dump of 2 hop path:
Path = [0][1][D]
Sep 13 12:07:25 280528 [40FFF970] -> osm_ucast_mgr_process: Min Hop Tables configured on all switches.
Sep 13 12:07:25 286786 [417FF970] -> __osm_trap_rcv_process_request: Received Generic Notice type:0x01 num:128 Producer:2 from LID:0x000E TID:0x0000000000000030
Sep 13 12:07:25 286922 [417FF970] -> osm_report_notice: Reporting Generic Notice type:1 num:128 from LID:0x0000 GID:0xfe80000000000000,0x0002c90200402915
Sep 13 12:07:25 706614 [42FFF970] -> umad_receiver: ERR 5409: send completed with error (method=1 attr=11) -- dropping.
Sep 13 12:07:25 706622 [42FFF970] -> umad_receiver: ERR 5411: DR SMP hop ptr 0 hop count 3 DR SLID 0x0 DR DLID 0x0
Sep 13 12:07:25 706631 [42FFF970] -> __osm_sm_mad_ctrl_send_err_cb: ERR 3113: MAD completed in error (IB_TIMEOUT).
Sep 13 12:07:25 706665 [42FFF970] -> SMP dump:
base_ver................0x1
mgmt_class..............0x81
class_ver...............0x1
method..................0x1 (SubnGet)
D bit...................0x0
status..................0x0
hop_ptr.................0x0
hop_count...............0x3
trans_id................0x2a28
attr_id.................0x11 (NodeInfo)
resv....................0x0
attr_mod................0x0
m_key...................0x0000000000000000
dr_slid.................0xFFFF
dr_dlid.................0xFFFF
Initial path: [0][1][D][C]
Return path: [0][0][0][0]
Reserved: [0][0][0][0][0][0][0]
00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
Sep 13 12:07:25 715859 [417FF970] -> osm_drop_mgr_process: ERR 0108: Unknown remote side for node 0x0002c90200402915 port 12. Adding to light sweep sampling list.
Sep 13 12:07:25 715870 [417FF970] -> Directed Path Dump of 2 hop path:
Path = [0][1][D]
Sep 13 12:07:25 716549 [417FF970] -> osm_ucast_mgr_process: Min Hop Tables configured on all switches.
Sep 13 12:07:25 722607 [40FFF970] -> __osm_trap_rcv_process_request: Received Generic Notice type:0x01 num:128 Producer:2 from LID:0x000E TID:0x0000000000000031
Sep 13 12:07:25 722740 [40FFF970] -> osm_report_notice: Reporting Generic Notice type:1 num:128 from LID:0x0000 GID:0xfe80000000000000,0x0002c90200402915
Sep 13 12:07:26 142616 [42FFF970] -> umad_receiver: ERR 5409: send completed with error (method=1 attr=11) -- dropping.
Sep 13 12:07:26 142626 [42FFF970] -> umad_receiver: ERR 5411: DR SMP hop ptr 0 hop count 3 DR SLID 0x0 DR DLID 0x0
Sep 13 12:07:26 142636 [42FFF970] -> __osm_sm_mad_ctrl_send_err_cb: ERR 3113: MAD completed in error (IB_TIMEOUT).
Sep 13 12:07:26 142670 [42FFF970] -> SMP dump:
base_ver................0x1
mgmt_class..............0x81
class_ver...............0x1
method..................0x1 (SubnGet)
D bit...................0x0
status..................0x0
hop_ptr.................0x0
hop_count...............0x3
trans_id................0x2ad9
attr_id.................0x11 (NodeInfo)
resv....................0x0
attr_mod................0x0
m_key...................0x0000000000000000
dr_slid.................0xFFFF
dr_dlid.................0xFFFF
Initial path: [0][1][D][C]
Return path: [0][0][0][0]
Reserved: [0][0][0][0][0][0][0]
00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
Sep 13 12:07:26 151864 [40FFF970] -> osm_drop_mgr_process: ERR 0108: Unknown remote side for node 0x0002c90200402915 port 12. Adding to light sweep sampling list.
Sep 13 12:07:26 151877 [40FFF970] -> Directed Path Dump of 2 hop path:
Path = [0][1][D]
Sep 13 12:07:26 152570 [40FFF970] -> osm_ucast_mgr_process: Min Hop Tables configured on all switches.
Sep 13 12:07:26 158845 [417FF970] -> __osm_trap_rcv_process_request: Received Generic Notice type:0x01 num:128 Producer:2 from LID:0x000E TID:0x0000000000000032
Sep 13 12:07:26 158974 [417FF970] -> osm_report_notice: Reporting Generic Notice type:1 num:128 from LID:0x0000 GID:0xfe80000000000000,0x0002c90200402915
Sep 13 12:07:26 578621 [42FFF970] -> umad_receiver: ERR 5409: send completed with error (method=1 attr=11) -- dropping.
Sep 13 12:07:26 578630 [42FFF970] -> umad_receiver: ERR 5411: DR SMP hop ptr 0 hop count 3 DR SLID 0x0 DR DLID 0x0
Sep 13 12:07:26 578640 [42FFF970] -> __osm_sm_mad_ctrl_send_err_cb: ERR 3113: MAD completed in error (IB_TIMEOUT).
Sep 13 12:07:26 578674 [42FFF970] -> SMP dump:
base_ver................0x1
mgmt_class..............0x81
class_ver...............0x1
method..................0x1 (SubnGet)
D bit...................0x0
status..................0x0
hop_ptr.................0x0
hop_count...............0x3
trans_id................0x2b8a
attr_id.................0x11 (NodeInfo)
resv....................0x0
attr_mod................0x0
m_key...................0x0000000000000000
dr_slid.................0xFFFF
dr_dlid.................0xFFFF
Initial path: [0][1][D][C]
Return path: [0][0][0][0]
Reserved: [0][0][0][0][0][0][0]
00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
Sep 13 12:07:26 587819 [40FFF970] -> osm_drop_mgr_process: ERR 0108: Unknown remote side for node 0x0002c90200402915 port 12. Adding to light sweep sampling list.
Sep 13 12:07:26 587833 [40FFF970] -> Directed Path Dump of 2 hop path:
Path = [0][1][D]
Sep 13 12:07:26 588526 [40FFF970] -> osm_ucast_mgr_process: Min Hop Tables configured on all switches.
Sep 13 12:07:26 594820 [417FF970] -> __osm_trap_rcv_process_request: Received Generic Notice type:0x01 num:128 Producer:2 from LID:0x000E TID:0x0000000000000033
Sep 13 12:07:26 594956 [417FF970] -> osm_report_notice: Reporting Generic Notice type:1 num:128 from LID:0x0000 GID:0xfe80000000000000,0x0002c90200402915
Sep 13 12:07:27 014621 [42FFF970] -> umad_receiver: ERR 5409: send completed with error (method=1 attr=11) -- dropping.
Sep 13 12:07:27 014629 [42FFF970] -> umad_receiver: ERR 5411: DR SMP hop ptr 0 hop count 3 DR SLID 0x0 DR DLID 0x0
Sep 13 12:07:27 014639 [42FFF970] -> __osm_sm_mad_ctrl_send_err_cb: ERR 3113: MAD completed in error (IB_TIMEOUT).
Sep 13 12:07:27 014672 [42FFF970] -> SMP dump:
base_ver................0x1
mgmt_class..............0x81
class_ver...............0x1
method..................0x1 (SubnGet)
D bit...................0x0
status..................0x0
hop_ptr.................0x0
hop_count...............0x3
trans_id................0x2c3b
attr_id.................0x11 (NodeInfo)
resv....................0x0
attr_mod................0x0
m_key...................0x0000000000000000
dr_slid.................0xFFFF
dr_dlid.................0xFFFF
Initial path: [0][1][D][C]
Return path: [0][0][0][0]
Reserved: [0][0][0][0][0][0][0]
00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
Sep 13 12:07:27 023896 [417FF970] -> osm_drop_mgr_process: ERR 0108: Unknown remote side for node 0x0002c90200402915 port 12. Adding to light sweep sampling list.
Sep 13 12:07:27 023906 [417FF970] -> Directed Path Dump of 2 hop path:
Path = [0][1][D]
Sep 13 12:07:27 024583 [417FF970] -> osm_ucast_mgr_process: Min Hop Tables configured on all switches.
Sep 13 12:07:27 030621 [40FFF970] -> __osm_trap_rcv_process_request: Received Generic Notice type:0x01 num:128 Producer:2 from LID:0x000E TID:0x0000000000000034
Sep 13 12:07:27 030752 [40FFF970] -> osm_report_notice: Reporting Generic Notice type:1 num:128 from LID:0x0000 GID:0xfe80000000000000,0x0002c90200402915
Sep 13 12:07:27 446626 [42FFF970] -> umad_receiver: ERR 5409: send completed with error (method=1 attr=11) -- dropping.
Sep 13 12:07:27 446636 [42FFF970] -> umad_receiver: ERR 5411: DR SMP hop ptr 0 hop count 3 DR SLID 0x0 DR DLID 0x0
Sep 13 12:07:27 446646 [42FFF970] -> __osm_sm_mad_ctrl_send_err_cb: ERR 3113: MAD completed in error (IB_TIMEOUT).
Sep 13 12:07:27 446680 [42FFF970] -> SMP dump:
base_ver................0x1
mgmt_class..............0x81
class_ver...............0x1
method..................0x1 (SubnGet)
D bit...................0x0
status..................0x0
hop_ptr.................0x0
hop_count...............0x3
trans_id................0x2cec
attr_id.................0x11 (NodeInfo)
resv....................0x0
attr_mod................0x0
m_key...................0x0000000000000000
dr_slid.................0xFFFF
dr_dlid.................0xFFFF
Initial path: [0][1][D][C]
Return path: [0][0][0][0]
Reserved: [0][0][0][0][0][0][0]
00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
Sep 13 12:07:27 455910 [40FFF970] -> osm_drop_mgr_process: ERR 0108: Unknown remote side for node 0x0002c90200402915 port 12. Adding to light sweep sampling list.
Sep 13 12:07:27 455924 [40FFF970] -> Directed Path Dump of 2 hop path:
Path = [0][1][D]
Sep 13 12:07:27 456623 [40FFF970] -> osm_ucast_mgr_process: Min Hop Tables configured on all switches.
Sep 13 12:07:27 462900 [417FF970] -> __osm_trap_rcv_process_request: Received Generic Notice type:0x01 num:128 Producer:2 from LID:0x000E TID:0x0000000000000035
Sep 13 12:07:27 463028 [417FF970] -> osm_report_notice: Reporting Generic Notice type:1 num:128 from LID:0x0000 GID:0xfe80000000000000,0x0002c90200402915
Sep 13 12:07:27 882629 [42FFF970] -> umad_receiver: ERR 5409: send completed with error (method=1 attr=11) -- dropping.
Sep 13 12:07:27 882637 [42FFF970] -> umad_receiver: ERR 5411: DR SMP hop ptr 0 hop count 3 DR SLID 0x0 DR DLID 0x0
Sep 13 12:07:27 882646 [42FFF970] -> __osm_sm_mad_ctrl_send_err_cb: ERR 3113: MAD completed in error (IB_TIMEOUT).
Sep 13 12:07:27 882680 [42FFF970] -> SMP dump:
base_ver................0x1
mgmt_class..............0x81
class_ver...............0x1
method..................0x1 (SubnGet)
D bit...................0x0
status..................0x0
hop_ptr.................0x0
hop_count...............0x3
trans_id................0x2d9d
attr_id.................0x11 (NodeInfo)
resv....................0x0
attr_mod................0x0
m_key...................0x0000000000000000
dr_slid.................0xFFFF
dr_dlid.................0xFFFF
Initial path: [0][1][D][C]
Return path: [0][0][0][0]
Reserved: [0][0][0][0][0][0][0]
00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
Sep 13 12:07:27 891749 [40FFF970] -> osm_drop_mgr_process: ERR 0108: Unknown remote side for node 0x0002c90200402915 port 12. Adding to light sweep sampling list.
Sep 13 12:07:27 891762 [40FFF970] -> Directed Path Dump of 2 hop path:
Path = [0][1][D]
Sep 13 12:07:27 892453 [40FFF970] -> osm_ucast_mgr_process: Min Hop Tables configured on all switches.
Sep 13 12:07:27 898712 [417FF970] -> __osm_trap_rcv_process_request: Received Generic Notice type:0x01 num:128 Producer:2 from LID:0x000E TID:0x0000000000000036
Sep 13 12:07:27 898849 [417FF970] -> osm_report_notice: Reporting Generic Notice type:1 num:128 from LID:0x0000 GID:0xfe80000000000000,0x0002c90200402915
Sep 13 12:07:28 318633 [42FFF970] -> umad_receiver: ERR 5409: send completed with error (method=1 attr=11) -- dropping.
Sep 13 12:07:28 318642 [42FFF970] -> umad_receiver: ERR 5411: DR SMP hop ptr 0 hop count 3 DR SLID 0x0 DR DLID 0x0
Sep 13 12:07:28 318652 [42FFF970] -> __osm_sm_mad_ctrl_send_err_cb: ERR 3113: MAD completed in error (IB_TIMEOUT).
Sep 13 12:07:28 318686 [42FFF970] -> SMP dump:
base_ver................0x1
mgmt_class..............0x81
class_ver...............0x1
method..................0x1 (SubnGet)
D bit...................0x0
status..................0x0
hop_ptr.................0x0
hop_count...............0x3
trans_id................0x2e4e
attr_id.................0x11 (NodeInfo)
resv....................0x0
attr_mod................0x0
m_key...................0x0000000000000000
dr_slid.................0xFFFF
dr_dlid.................0xFFFF
Initial path: [0][1][D][C]
Return path: [0][0][0][0]
Reserved: [0][0][0][0][0][0][0]
00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
Sep 13 12:07:28 327869 [40FFF970] -> osm_drop_mgr_process: ERR 0108: Unknown remote side for node 0x0002c90200402915 port 12. Adding to light sweep sampling list.
Sep 13 12:07:28 327883 [40FFF970] -> Directed Path Dump of 2 hop path:
Path = [0][1][D]
Sep 13 12:07:28 328570 [40FFF970] -> osm_ucast_mgr_process: Min Hop Tables configured on all switches.
Sep 13 12:07:28 334831 [417FF970] -> __osm_trap_rcv_process_request: Received Generic Notice type:0x01 num:128 Producer:2 from LID:0x000E TID:0x0000000000000037
Sep 13 12:07:28 334884 [417FF970] -> __osm_trap_rcv_process_request: ERR 3804: Received the trap 11 times continuously.
Sep 13 12:07:33 110661 [42FFF970] -> umad_receiver: ERR 5409: send completed with error (method=1 attr=15) -- dropping.
Sep 13 12:07:33 110671 [42FFF970] -> umad_receiver: ERR 5411: DR SMP hop ptr 0 hop count 3 DR SLID 0x0 DR DLID 0x0
Sep 13 12:07:33 110680 [42FFF970] -> __osm_sm_mad_ctrl_send_err_cb: ERR 3113: MAD completed in error (IB_TIMEOUT).
Sep 13 12:07:33 110714 [42FFF970] -> SMP dump:
base_ver................0x1
mgmt_class..............0x81
class_ver...............0x1
method..................0x1 (SubnGet)
D bit...................0x0
status..................0x0
hop_ptr.................0x0
hop_count...............0x3
trans_id................0x2e84
attr_id.................0x15 (PortInfo)
resv....................0x0
attr_mod................0x0
m_key...................0x0000000000000000
dr_slid.................0xFFFF
dr_dlid.................0xFFFF
Initial path: [0][1][D][C]
Return path: [0][0][0][0]
Reserved: [0][0][0][0][0][0][0]
00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
Sep 13 12:07:33 530667 [42FFF970] -> umad_receiver: ERR 5409: send completed with error (method=1 attr=11) -- dropping.
Sep 13 12:07:33 530676 [42FFF970] -> umad_receiver: ERR 5411: DR SMP hop ptr 0 hop count 3 DR SLID 0x0 DR DLID 0x0
Sep 13 12:07:33 530686 [42FFF970] -> __osm_sm_mad_ctrl_send_err_cb: ERR 3113: MAD completed in error (IB_TIMEOUT).
Sep 13 12:07:33 530720 [42FFF970] -> SMP dump:
base_ver................0x1
mgmt_class..............0x81
class_ver...............0x1
method..................0x1 (SubnGet)
D bit...................0x0
status..................0x0
hop_ptr.................0x0
hop_count...............0x3
trans_id................0x2f02
attr_id.................0x11 (NodeInfo)
resv....................0x0
attr_mod................0x0
m_key...................0x0000000000000000
dr_slid.................0xFFFF
dr_dlid.................0xFFFF
Initial path: [0][1][D][C]
Return path: [0][0][0][0]
Reserved: [0][0][0][0][0][0][0]
00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
Sep 13 12:07:33 539999 [40FFF970] -> osm_drop_mgr_process: ERR 0108: Unknown remote side for node 0x0002c90200402915 port 12. Adding to light sweep sampling list.
Sep 13 12:07:33 540014 [40FFF970] -> Directed Path Dump of 2 hop path:
Path = [0][1][D]
Sep 13 12:07:33 540697 [40FFF970] -> osm_ucast_mgr_process: Min Hop Tables configured on all switches.
Sep 13 12:07:33 546985 [417FF970] -> __osm_trap_rcv_process_request: Received Generic Notice type:0x01 num:128 Producer:2 from LID:0x000E TID:0x0000000000000038
Sep 13 12:07:33 547118 [417FF970] -> osm_report_notice: Reporting Generic Notice type:1 num:128 from LID:0x0000 GID:0xfe80000000000000,0x0002c90200402915
Sep 13 12:07:33 966667 [42FFF970] -> umad_receiver: ERR 5409: send completed with error (method=1 attr=11) -- dropping.
Sep 13 12:07:33 966675 [42FFF970] -> umad_receiver: ERR 5411: DR SMP hop ptr 0 hop count 3 DR SLID 0x0 DR DLID 0x0
Sep 13 12:07:33 966685 [42FFF970] -> __osm_sm_mad_ctrl_send_err_cb: ERR 3113: MAD completed in error (IB_TIMEOUT).
Sep 13 12:07:33 966719 [42FFF970] -> SMP dump:
base_ver................0x1
mgmt_class..............0x81
class_ver...............0x1
method..................0x1 (SubnGet)
D bit...................0x0
status..................0x0
hop_ptr.................0x0
hop_count...............0x3
trans_id................0x2fb3
attr_id.................0x11 (NodeInfo)
resv....................0x0
attr_mod................0x0
m_key...................0x0000000000000000
dr_slid.................0xFFFF
dr_dlid.................0xFFFF
Initial path: [0][1][D][C]
Return path: [0][0][0][0]
Reserved: [0][0][0][0][0][0][0]
00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
Sep 13 12:07:33 975842 [417FF970] -> osm_drop_mgr_process: ERR 0108: Unknown remote side for node 0x0002c90200402915 port 12. Adding to light sweep sampling list.
Sep 13 12:07:33 975853 [417FF970] -> Directed Path Dump of 2 hop path:
Path = [0][1][D]
Sep 13 12:07:33 976525 [417FF970] -> osm_ucast_mgr_process: Min Hop Tables configured on all switches.
Sep 13 12:07:33 982537 [40FFF970] -> __osm_trap_rcv_process_request: Received Generic Notice type:0x01 num:128 Producer:2 from LID:0x000E TID:0x0000000000000039
Sep 13 12:07:33 982673 [40FFF970] -> osm_report_notice: Reporting Generic Notice type:1 num:128 from LID:0x0000 GID:0xfe80000000000000,0x0002c90200402915
Sep 13 12:07:34 398673 [42FFF970] -> umad_receiver: ERR 5409: send completed with error (method=1 attr=11) -- dropping.
Sep 13 12:07:34 398683 [42FFF970] -> umad_receiver: ERR 5411: DR SMP hop ptr 0 hop count 3 DR SLID 0x0 DR DLID 0x0
Sep 13 12:07:34 398693 [42FFF970] -> __osm_sm_mad_ctrl_send_err_cb: ERR 3113: MAD completed in error (IB_TIMEOUT).
Sep 13 12:07:34 398728 [42FFF970] -> SMP dump:
base_ver................0x1
mgmt_class..............0x81
class_ver...............0x1
method..................0x1 (SubnGet)
D bit...................0x0
status..................0x0
hop_ptr.................0x0
hop_count...............0x3
trans_id................0x3064
attr_id.................0x11 (NodeInfo)
resv....................0x0
attr_mod................0x0
m_key...................0x0000000000000000
dr_slid.................0xFFFF
dr_dlid.................0xFFFF
Initial path: [0][1][D][C]
Return path: [0][0][0][0]
Reserved: [0][0][0][0][0][0][0]
00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
Sep 13 12:07:34 407934 [40FFF970] -> osm_drop_mgr_process: ERR 0108: Unknown remote side for node 0x0002c90200402915 port 12. Adding to light sweep sampling list.
Sep 13 12:07:34 407949 [40FFF970] -> Directed Path Dump of 2 hop path:
Path = [0][1][D]
Sep 13 12:07:34 408646 [40FFF970] -> osm_ucast_mgr_process: Min Hop Tables configured on all switches.
Sep 13 12:07:34 414922 [417FF970] -> __osm_trap_rcv_process_request: Received Generic Notice type:0x01 num:128 Producer:2 from LID:0x000E TID:0x000000000000003a
Sep 13 12:07:34 415057 [417FF970] -> osm_report_notice: Reporting Generic Notice type:1 num:128 from LID:0x0000 GID:0xfe80000000000000,0x0002c90200402915
Sep 13 12:07:34 834673 [42FFF970] -> umad_receiver: ERR 5409: send completed with error (method=1 attr=11) -- dropping.
Sep 13 12:07:34 834681 [42FFF970] -> umad_receiver: ERR 5411: DR SMP hop ptr 0 hop count 3 DR SLID 0x0 DR DLID 0x0
Sep 13 12:07:34 834691 [42FFF970] -> __osm_sm_mad_ctrl_send_err_cb: ERR 3113: MAD completed in error (IB_TIMEOUT).
Sep 13 12:07:34 834725 [42FFF970] -> SMP dump:
base_ver................0x1
mgmt_class..............0x81
class_ver...............0x1
method..................0x1 (SubnGet)
D bit...................0x0
status..................0x0
hop_ptr.................0x0
hop_count...............0x3
trans_id................0x3115
attr_id.................0x11 (NodeInfo)
resv....................0x0
attr_mod................0x0
m_key...................0x0000000000000000
dr_slid.................0xFFFF
dr_dlid.................0xFFFF
Initial path: [0][1][D][C]
Return path: [0][0][0][0]
Reserved: [0][0][0][0][0][0][0]
00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
Sep 13 12:07:34 844013 [417FF970] -> osm_drop_mgr_process: ERR 0108: Unknown remote side for node 0x0002c90200402915 port 12. Adding to light sweep sampling list.
Sep 13 12:07:34 844024 [417FF970] -> Directed Path Dump of 2 hop path:
Path = [0][1][D]
Sep 13 12:07:34 844694 [417FF970] -> osm_ucast_mgr_process: Min Hop Tables configured on all switches.
Sep 13 12:07:34 850732 [40FFF970] -> __osm_trap_rcv_process_request: Received Generic Notice type:0x01 num:128 Producer:2 from LID:0x000E TID:0x000000000000003b
Sep 13 12:07:34 850863 [40FFF970] -> osm_report_notice: Reporting Generic Notice type:1 num:128 from LID:0x0000 GID:0xfe80000000000000,0x0002c90200402915
Sep 13 12:07:35 266677 [42FFF970] -> umad_receiver: ERR 5409: send completed with error (method=1 attr=11) -- dropping.
Sep 13 12:07:35 266687 [42FFF970] -> umad_receiver: ERR 5411: DR SMP hop ptr 0 hop count 3 DR SLID 0x0 DR DLID 0x0
Sep 13 12:07:35 266696 [42FFF970] -> __osm_sm_mad_ctrl_send_err_cb: ERR 3113: MAD completed in error (IB_TIMEOUT).
Sep 13 12:07:35 266731 [42FFF970] -> SMP dump:
base_ver................0x1
mgmt_class..............0x81
class_ver...............0x1
method..................0x1 (SubnGet)
D bit...................0x0
status..................0x0
hop_ptr.................0x0
hop_count...............0x3
trans_id................0x31c6
attr_id.................0x11 (NodeInfo)
resv....................0x0
attr_mod................0x0
m_key...................0x0000000000000000
dr_slid.................0xFFFF
dr_dlid.................0xFFFF
Initial path: [0][1][D][C]
Return path: [0][0][0][0]
Reserved: [0][0][0][0][0][0][0]
00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
Sep 13 12:07:35 275935 [40FFF970] -> osm_drop_mgr_process: ERR 0108: Unknown remote side for node 0x0002c90200402915 port 12. Adding to light sweep sampling list.
Sep 13 12:07:35 275947 [40FFF970] -> Directed Path Dump of 2 hop path:
Path = [0][1][D]
Sep 13 12:07:35 276635 [40FFF970] -> osm_ucast_mgr_process: Min Hop Tables configured on all switches.
Sep 13 12:07:35 282910 [417FF970] -> __osm_trap_rcv_process_request: Received Generic Notice type:0x01 num:128 Producer:2 from LID:0x000E TID:0x000000000000003c
Sep 13 12:07:35 283039 [417FF970] -> osm_report_notice: Reporting Generic Notice type:1 num:128 from LID:0x0000 GID:0xfe80000000000000,0x0002c90200402915
Sep 13 12:07:35 702679 [42FFF970] -> umad_receiver: ERR 5409: send completed with error (method=1 attr=11) -- dropping.
Sep 13 12:07:35 702688 [42FFF970] -> umad_receiver: ERR 5411: DR SMP hop ptr 0 hop count 3 DR SLID 0x0 DR DLID 0x0
Sep 13 12:07:35 702698 [42FFF970] -> __osm_sm_mad_ctrl_send_err_cb: ERR 3113: MAD completed in error (IB_TIMEOUT).
Sep 13 12:07:35 702732 [42FFF970] -> SMP dump:
base_ver................0x1
mgmt_class..............0x81
class_ver...............0x1
method..................0x1 (SubnGet)
D bit...................0x0
status..................0x0
hop_ptr.................0x0
hop_count...............0x3
trans_id................0x3277
attr_id.................0x11 (NodeInfo)
resv....................0x0
attr_mod................0x0
m_key...................0x0000000000000000
dr_slid.................0xFFFF
dr_dlid.................0xFFFF
Initial path: [0][1][D][C]
Return path: [0][0][0][0]
Reserved: [0][0][0][0][0][0][0]
00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
Sep 13 12:07:35 711994 [40FFF970] -> osm_drop_mgr_process: ERR 0108: Unknown remote side for node 0x0002c90200402915 port 12. Adding to light sweep sampling list.
Sep 13 12:07:35 712009 [40FFF970] -> Directed Path Dump of 2 hop path:
Path = [0][1][D]
Sep 13 12:07:35 712690 [40FFF970] -> osm_ucast_mgr_process: Min Hop Tables configured on all switches.
Sep 13 12:07:35 718977 [417FF970] -> __osm_trap_rcv_process_request: Received Generic Notice type:0x01 num:128 Producer:2 from LID:0x000E TID:0x000000000000003d
Sep 13 12:07:35 719106 [417FF970] -> osm_report_notice: Reporting Generic Notice type:1 num:128 from LID:0x0000 GID:0xfe80000000000000,0x0002c90200402915
Sep 13 12:07:36 138684 [42FFF970] -> umad_receiver: ERR 5409: send completed with error (method=1 attr=11) -- dropping.
Sep 13 12:07:36 138692 [42FFF970] -> umad_receiver: ERR 5411: DR SMP hop ptr 0 hop count 3 DR SLID 0x0 DR DLID 0x0
Sep 13 12:07:36 138701 [42FFF970] -> __osm_sm_mad_ctrl_send_err_cb: ERR 3113: MAD completed in error (IB_TIMEOUT).
Sep 13 12:07:36 138735 [42FFF970] -> SMP dump:
base_ver................0x1
mgmt_class..............0x81
class_ver...............0x1
method..................0x1 (SubnGet)
D bit...................0x0
status..................0x0
hop_ptr.................0x0
hop_count...............0x3
trans_id................0x3328
attr_id.................0x11 (NodeInfo)
resv....................0x0
attr_mod................0x0
m_key...................0x0000000000000000
dr_slid.................0xFFFF
dr_dlid.................0xFFFF
Initial path: [0][1][D][C]
Return path: [0][0][0][0]
Reserved: [0][0][0][0][0][0][0]
00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
Sep 13 12:07:36 147903 [40FFF970] -> osm_drop_mgr_process: ERR 0108: Unknown remote side for node 0x0002c90200402915 port 12. Adding to light sweep sampling list.
Sep 13 12:07:36 147917 [40FFF970] -> Directed Path Dump of 2 hop path:
Path = [0][1][D]
Sep 13 12:07:36 148591 [40FFF970] -> osm_ucast_mgr_process: Min Hop Tables configured on all switches.
Sep 13 12:07:36 154866 [417FF970] -> __osm_trap_rcv_process_request: Received Generic Notice type:0x01 num:128 Producer:2 from LID:0x000E TID:0x000000000000003e
Sep 13 12:07:36 154995 [417FF970] -> osm_report_notice: Reporting Generic Notice type:1 num:128 from LID:0x0000 GID:0xfe80000000000000,0x0002c90200402915
Sep 13 12:07:36 574686 [42FFF970] -> umad_receiver: ERR 5409: send completed with error (method=1 attr=11) -- dropping.
Sep 13 12:07:36 574696 [42FFF970] -> umad_receiver: ERR 5411: DR SMP hop ptr 0 hop count 3 DR SLID 0x0 DR DLID 0x0
Sep 13 12:07:36 574705 [42FFF970] -> __osm_sm_mad_ctrl_send_err_cb: ERR 3113: MAD completed in error (IB_TIMEOUT).
Sep 13 12:07:36 574739 [42FFF970] -> SMP dump:
base_ver................0x1
mgmt_class..............0x81
class_ver...............0x1
method..................0x1 (SubnGet)
D bit...................0x0
status..................0x0
hop_ptr.................0x0
hop_count...............0x3
trans_id................0x33d9
attr_id.................0x11 (NodeInfo)
resv....................0x0
attr_mod................0x0
m_key...................0x0000000000000000
dr_slid.................0xFFFF
dr_dlid.................0xFFFF
Initial path: [0][1][D][C]
Return path: [0][0][0][0]
Reserved: [0][0][0][0][0][0][0]
00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
Sep 13 12:07:36 583962 [417FF970] -> osm_drop_mgr_process: ERR 0108: Unknown remote side for node 0x0002c90200402915 port 12. Adding to light sweep sampling list.
Sep 13 12:07:36 583973 [417FF970] -> Directed Path Dump of 2 hop path:
Path = [0][1][D]
Sep 13 12:07:36 584638 [417FF970] -> osm_ucast_mgr_process: Min Hop Tables configured on all switches.
Sep 13 12:07:36 590676 [40FFF970] -> __osm_trap_rcv_process_request: Received Generic Notice type:0x01 num:128 Producer:2 from LID:0x000E TID:0x000000000000003f
Sep 13 12:07:36 590811 [40FFF970] -> osm_report_notice: Reporting Generic Notice type:1 num:128 from LID:0x0000 GID:0xfe80000000000000,0x0002c90200402915
Sep 13 12:07:37 006688 [42FFF970] -> umad_receiver: ERR 5409: send completed with error (method=1 attr=11) -- dropping.
Sep 13 12:07:37 006698 [42FFF970] -> umad_receiver: ERR 5411: DR SMP hop ptr 0 hop count 3 DR SLID 0x0 DR DLID 0x0
Sep 13 12:07:37 006708 [42FFF970] -> __osm_sm_mad_ctrl_send_err_cb: ERR 3113: MAD completed in error (IB_TIMEOUT).
Sep 13 12:07:37 006741 [42FFF970] -> SMP dump:
base_ver................0x1
mgmt_class..............0x81
class_ver...............0x1
method..................0x1 (SubnGet)
D bit...................0x0
status..................0x0
hop_ptr.................0x0
hop_count...............0x3
trans_id................0x348a
attr_id.................0x11 (NodeInfo)
resv....................0x0
attr_mod................0x0
m_key...................0x0000000000000000
dr_slid.................0xFFFF
dr_dlid.................0xFFFF
Initial path: [0][1][D][C]
Return path: [0][0][0][0]
Reserved: [0][0][0][0][0][0][0]
00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
Sep 13 12:07:37 015886 [417FF970] -> osm_drop_mgr_process: ERR 0108: Unknown remote side for node 0x0002c90200402915 port 12. Adding to light sweep sampling list.
Sep 13 12:07:37 015896 [417FF970] -> Directed Path Dump of 2 hop path:
Path = [0][1][D]
Sep 13 12:07:37 016556 [417FF970] -> osm_ucast_mgr_process: Min Hop Tables configured on all switches.
Sep 13 12:07:37 022568 [40FFF970] -> __osm_trap_rcv_process_request: Received Generic Notice type:0x01 num:128 Producer:2 from LID:0x000E TID:0x0000000000000040
Sep 13 12:07:37 022698 [40FFF970] -> osm_report_notice: Reporting Generic Notice type:1 num:128 from LID:0x0000 GID:0xfe80000000000000,0x0002c90200402915
Sep 13 12:07:37 438692 [42FFF970] -> umad_receiver: ERR 5409: send completed with error (method=1 attr=11) -- dropping.
Sep 13 12:07:37 438702 [42FFF970] -> umad_receiver: ERR 5411: DR SMP hop ptr 0 hop count 3 DR SLID 0x0 DR DLID 0x0
Sep 13 12:07:37 438712 [42FFF970] -> __osm_sm_mad_ctrl_send_err_cb: ERR 3113: MAD completed in error (IB_TIMEOUT).
Sep 13 12:07:37 438746 [42FFF970] -> SMP dump:
base_ver................0x1
mgmt_class..............0x81
class_ver...............0x1
method..................0x1 (SubnGet)
D bit...................0x0
status..................0x0
hop_ptr.................0x0
hop_count...............0x3
trans_id................0x353b
attr_id.................0x11 (NodeInfo)
resv....................0x0
attr_mod................0x0
m_key...................0x0000000000000000
dr_slid.................0xFFFF
dr_dlid.................0xFFFF
Initial path: [0][1][D][C]
Return path: [0][0][0][0]
Reserved: [0][0][0][0][0][0][0]
00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
Sep 13 12:07:37 447934 [417FF970] -> osm_drop_mgr_process: ERR 0108: Unknown remote side for node 0x0002c90200402915 port 12. Adding to light sweep sampling list.
Sep 13 12:07:37 447944 [417FF970] -> Directed Path Dump of 2 hop path:
Path = [0][1][D]
Sep 13 12:07:37 448600 [417FF970] -> osm_ucast_mgr_process: Min Hop Tables configured on all switches.
Sep 13 12:07:37 454615 [40FFF970] -> __osm_trap_rcv_process_request: Received Generic Notice type:0x01 num:128 Producer:2 from LID:0x000E TID:0x0000000000000041
Sep 13 12:07:37 454746 [40FFF970] -> osm_report_notice: Reporting Generic Notice type:1 num:128 from LID:0x0000 GID:0xfe80000000000000,0x0002c90200402915
Sep 13 12:07:37 870693 [42FFF970] -> umad_receiver: ERR 5409: send completed with error (method=1 attr=11) -- dropping.
Sep 13 12:07:37 870703 [42FFF970] -> umad_receiver: ERR 5411: DR SMP hop ptr 0 hop count 3 DR SLID 0x0 DR DLID 0x0
Sep 13 12:07:37 870713 [42FFF970] -> __osm_sm_mad_ctrl_send_err_cb: ERR 3113: MAD completed in error (IB_TIMEOUT).
Sep 13 12:07:37 870747 [42FFF970] -> SMP dump:
base_ver................0x1
mgmt_class..............0x81
class_ver...............0x1
method..................0x1 (SubnGet)
D bit...................0x0
status..................0x0
hop_ptr.................0x0
hop_count...............0x3
trans_id................0x35ec
attr_id.................0x11 (NodeInfo)
resv....................0x0
attr_mod................0x0
m_key...................0x0000000000000000
dr_slid.................0xFFFF
dr_dlid.................0xFFFF
Initial path: [0][1][D][C]
Return path: [0][0][0][0]
Reserved: [0][0][0][0][0][0][0]
00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
Sep 13 12:07:37 879876 [417FF970] -> osm_drop_mgr_process: ERR 0108: Unknown remote side for node 0x0002c90200402915 port 12. Adding to light sweep sampling list.
Sep 13 12:07:37 879887 [417FF970] -> Directed Path Dump of 2 hop path:
Path = [0][1][D]
Sep 13 12:07:37 880542 [417FF970] -> osm_ucast_mgr_process: Min Hop Tables configured on all switches.
Sep 13 12:07:37 886564 [40FFF970] -> __osm_trap_rcv_process_request: Received Generic Notice type:0x01 num:128 Producer:2 from LID:0x000E TID:0x0000000000000042
Sep 13 12:07:37 886699 [40FFF970] -> osm_report_notice: Reporting Generic Notice type:1 num:128 from LID:0x0000 GID:0xfe80000000000000,0x0002c90200402915
Sep 13 12:07:38 302698 [42FFF970] -> umad_receiver: ERR 5409: send completed with error (method=1 attr=11) -- dropping.
Sep 13 12:07:38 302708 [42FFF970] -> umad_receiver: ERR 5411: DR SMP hop ptr 0 hop count 3 DR SLID 0x0 DR DLID 0x0
Sep 13 12:07:38 302718 [42FFF970] -> __osm_sm_mad_ctrl_send_err_cb: ERR 3113: MAD completed in error (IB_TIMEOUT).
Sep 13 12:07:38 302752 [42FFF970] -> SMP dump:
base_ver................0x1
mgmt_class..............0x81
class_ver...............0x1
method..................0x1 (SubnGet)
D bit...................0x0
status..................0x0
hop_ptr.................0x0
hop_count...............0x3
trans_id................0x369d
attr_id.................0x11 (NodeInfo)
resv....................0x0
attr_mod................0x0
m_key...................0x0000000000000000
dr_slid.................0xFFFF
dr_dlid.................0xFFFF
Initial path: [0][1][D][C]
Return path: [0][0][0][0]
Reserved: [0][0][0][0][0][0][0]
00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
Sep 13 12:07:38 313032 [417FF970] -> osm_drop_mgr_process: ERR 0108: Unknown remote side for node 0x0002c90200402915 port 12. Adding to light sweep sampling list.
Sep 13 12:07:38 313042 [417FF970] -> Directed Path Dump of 2 hop path:
Path = [0][1][D]
Sep 13 12:07:38 313696 [417FF970] -> osm_ucast_mgr_process: Min Hop Tables configured on all switches.
Sep 13 12:07:38 319714 [40FFF970] -> __osm_trap_rcv_process_request: Received Generic Notice type:0x01 num:128 Producer:2 from LID:0x000E TID:0x0000000000000043
Sep 13 12:07:38 319767 [40FFF970] -> __osm_trap_rcv_process_request: ERR 3804: Received the trap 11 times continuously.
Sep 13 12:07:43 114729 [42FFF970] -> umad_receiver: ERR 5409: send completed with error (method=1 attr=15) -- dropping.
Sep 13 12:07:43 114739 [42FFF970] -> umad_receiver: ERR 5411: DR SMP hop ptr 0 hop count 3 DR SLID 0x0 DR DLID 0x0
Sep 13 12:07:43 114749 [42FFF970] -> __osm_sm_mad_ctrl_send_err_cb: ERR 3113: MAD completed in error (IB_TIMEOUT).
Sep 13 12:07:43 114783 [42FFF970] -> SMP dump:
base_ver................0x1
mgmt_class..............0x81
class_ver...............0x1
method..................0x1 (SubnGet)
D bit...................0x0
status..................0x0
hop_ptr.................0x0
hop_count...............0x3
trans_id................0x36d3
attr_id.................0x15 (PortInfo)
resv....................0x0
attr_mod................0x0
m_key...................0x0000000000000000
dr_slid.................0xFFFF
dr_dlid.................0xFFFF
Initial path: [0][1][D][C]
Return path: [0][0][0][0]
Reserved: [0][0][0][0][0][0][0]
00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
Sep 13 12:07:43 530734 [42FFF970] -> umad_receiver: ERR 5409: send completed with error (method=1 attr=11) -- dropping.
Sep 13 12:07:43 530743 [42FFF970] -> umad_receiver: ERR 5411: DR SMP hop ptr 0 hop count 3 DR SLID 0x0 DR DLID 0x0
Sep 13 12:07:43 530752 [42FFF970] -> __osm_sm_mad_ctrl_send_err_cb: ERR 3113: MAD completed in error (IB_TIMEOUT).
Sep 13 12:07:43 530787 [42FFF970] -> SMP dump:
base_ver................0x1
mgmt_class..............0x81
class_ver...............0x1
method..................0x1 (SubnGet)
D bit...................0x0
status..................0x0
hop_ptr.................0x0
hop_count...............0x3
trans_id................0x3751
attr_id.................0x11 (NodeInfo)
resv....................0x0
attr_mod................0x0
m_key...................0x0000000000000000
dr_slid.................0xFFFF
dr_dlid.................0xFFFF
Initial path: [0][1][D][C]
Return path: [0][0][0][0]
Reserved: [0][0][0][0][0][0][0]
00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
Sep 13 12:07:43 539933 [40FFF970] -> osm_drop_mgr_process: ERR 0108: Unknown remote side for node 0x0002c90200402915 port 12. Adding to light sweep sampling list.
Sep 13 12:07:43 539950 [40FFF970] -> Directed Path Dump of 2 hop path:
Path = [0][1][D]
Sep 13 12:07:43 540688 [40FFF970] -> osm_ucast_mgr_process: Min Hop Tables configured on all switches.
Sep 13 12:07:43 546948 [417FF970] -> __osm_trap_rcv_process_request: Received Generic Notice type:0x01 num:128 Producer:2 from LID:0x000E TID:0x0000000000000044
Sep 13 12:07:43 547081 [417FF970] -> osm_report_notice: Reporting Generic Notice type:1 num:128 from LID:0x0000 GID:0xfe80000000000000,0x0002c90200402915
Sep 13 12:07:43 962734 [42FFF970] -> umad_receiver: ERR 5409: send completed with error (method=1 attr=11) -- dropping.
Sep 13 12:07:43 962743 [42FFF970] -> umad_receiver: ERR 5411: DR SMP hop ptr 0 hop count 3 DR SLID 0x0 DR DLID 0x0
Sep 13 12:07:43 962752 [42FFF970] -> __osm_sm_mad_ctrl_send_err_cb: ERR 3113: MAD completed in error (IB_TIMEOUT).
Sep 13 12:07:43 962786 [42FFF970] -> SMP dump:
base_ver................0x1
mgmt_class..............0x81
class_ver...............0x1
method..................0x1 (SubnGet)
D bit...................0x0
status..................0x0
hop_ptr.................0x0
hop_count...............0x3
trans_id................0x3802
attr_id.................0x11 (NodeInfo)
resv....................0x0
attr_mod................0x0
m_key...................0x0000000000000000
dr_slid.................0xFFFF
dr_dlid.................0xFFFF
Initial path: [0][1][D][C]
Return path: [0][0][0][0]
Reserved: [0][0][0][0][0][0][0]
00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
Sep 13 12:07:43 971861 [417FF970] -> osm_drop_mgr_process: ERR 0108: Unknown remote side for node 0x0002c90200402915 port 12. Adding to light sweep sampling list.
Sep 13 12:07:43 971871 [417FF970] -> Directed Path Dump of 2 hop path:
Path = [0][1][D]
Sep 13 12:07:43 972537 [417FF970] -> osm_ucast_mgr_process: Min Hop Tables configured on all switches.
Sep 13 12:07:43 978558 [40FFF970] -> __osm_trap_rcv_process_request: Received Generic Notice type:0x01 num:128 Producer:2 from LID:0x000E TID:0x0000000000000045
Sep 13 12:07:43 978690 [40FFF970] -> osm_report_notice: Reporting Generic Notice type:1 num:128 from LID:0x0000 GID:0xfe80000000000000,0x0002c90200402915
Sep 13 12:07:44 394739 [42FFF970] -> umad_receiver: ERR 5409: send completed with error (method=1 attr=11) -- dropping.
Sep 13 12:07:44 394750 [42FFF970] -> umad_receiver: ERR 5411: DR SMP hop ptr 0 hop count 3 DR SLID 0x0 DR DLID 0x0
Sep 13 12:07:44 394760 [42FFF970] -> __osm_sm_mad_ctrl_send_err_cb: ERR 3113: MAD completed in error (IB_TIMEOUT).
Sep 13 12:07:44 394794 [42FFF970] -> SMP dump:
base_ver................0x1
mgmt_class..............0x81
class_ver...............0x1
method..................0x1 (SubnGet)
D bit...................0x0
status..................0x0
hop_ptr.................0x0
hop_count...............0x3
trans_id................0x38b3
attr_id.................0x11 (NodeInfo)
resv....................0x0
attr_mod................0x0
m_key...................0x0000000000000000
dr_slid.................0xFFFF
dr_dlid.................0xFFFF
Initial path: [0][1][D][C]
Return path: [0][0][0][0]
Reserved: [0][0][0][0][0][0][0]
00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
Sep 13 12:07:44 403994 [40FFF970] -> osm_drop_mgr_process: ERR 0108: Unknown remote side for node 0x0002c90200402915 port 12. Adding to light sweep sampling list.
Sep 13 12:07:44 404008 [40FFF970] -> Directed Path Dump of 2 hop path:
Path = [0][1][D]
Sep 13 12:07:44 404700 [40FFF970] -> osm_ucast_mgr_process: Min Hop Tables configured on all switches.
Sep 13 12:07:44 410967 [417FF970] -> __osm_trap_rcv_process_request: Received Generic Notice type:0x01 num:128 Producer:2 from LID:0x000E TID:0x0000000000000046
Sep 13 12:07:44 411103 [417FF970] -> osm_report_notice: Reporting Generic Notice type:1 num:128 from LID:0x0000 GID:0xfe80000000000000,0x0002c90200402915
Sep 13 12:07:44 826740 [42FFF970] -> umad_receiver: ERR 5409: send completed with error (method=1 attr=11) -- dropping.
Sep 13 12:07:44 826748 [42FFF970] -> umad_receiver: ERR 5411: DR SMP hop ptr 0 hop count 3 DR SLID 0x0 DR DLID 0x0
Sep 13 12:07:44 826757 [42FFF970] -> __osm_sm_mad_ctrl_send_err_cb: ERR 3113: MAD completed in error (IB_TIMEOUT).
Sep 13 12:07:44 826791 [42FFF970] -> SMP dump:
base_ver................0x1
mgmt_class..............0x81
class_ver...............0x1
method..................0x1 (SubnGet)
D bit...................0x0
status..................0x0
hop_ptr.................0x0
hop_count...............0x3
trans_id................0x3964
attr_id.................0x11 (NodeInfo)
resv....................0x0
attr_mod................0x0
m_key...................0x0000000000000000
dr_slid.................0xFFFF
dr_dlid.................0xFFFF
Initial path: [0][1][D][C]
Return path: [0][0][0][0]
Reserved: [0][0][0][0][0][0][0]
00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
Sep 13 12:07:44 835961 [417FF970] -> osm_drop_mgr_process: ERR 0108: Unknown remote side for node 0x0002c90200402915 port 12. Adding to light sweep sampling list.
Sep 13 12:07:44 835972 [417FF970] -> Directed Path Dump of 2 hop path:
Path = [0][1][D]
Sep 13 12:07:44 836639 [417FF970] -> osm_ucast_mgr_process: Min Hop Tables configured on all switches.
Sep 13 12:07:44 842655 [40FFF970] -> __osm_trap_rcv_process_request: Received Generic Notice type:0x01 num:128 Producer:2 from LID:0x000E TID:0x0000000000000047
Sep 13 12:07:44 842787 [40FFF970] -> osm_report_notice: Reporting Generic Notice type:1 num:128 from LID:0x0000 GID:0xfe80000000000000,0x0002c90200402915
Sep 13 12:07:45 258745 [42FFF970] -> umad_receiver: ERR 5409: send completed with error (method=1 attr=11) -- dropping.
Sep 13 12:07:45 258755 [42FFF970] -> umad_receiver: ERR 5411: DR SMP hop ptr 0 hop count 3 DR SLID 0x0 DR DLID 0x0
Sep 13 12:07:45 258765 [42FFF970] -> __osm_sm_mad_ctrl_send_err_cb: ERR 3113: MAD completed in error (IB_TIMEOUT).
Sep 13 12:07:45 258799 [42FFF970] -> SMP dump:
base_ver................0x1
mgmt_class..............0x81
class_ver...............0x1
method..................0x1 (SubnGet)
D bit...................0x0
status..................0x0
hop_ptr.................0x0
hop_count...............0x3
trans_id................0x3a15
attr_id.................0x11 (NodeInfo)
resv....................0x0
attr_mod................0x0
m_key...................0x0000000000000000
dr_slid.................0xFFFF
dr_dlid.................0xFFFF
Initial path: [0][1][D][C]
Return path: [0][0][0][0]
Reserved: [0][0][0][0][0][0][0]
00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
Sep 13 12:07:45 267949 [40FFF970] -> osm_drop_mgr_process: ERR 0108: Unknown remote side for node 0x0002c90200402915 port 12. Adding to light sweep sampling list.
Sep 13 12:07:45 267965 [40FFF970] -> Directed Path Dump of 2 hop path:
Path = [0][1][D]
Sep 13 12:07:45 268652 [40FFF970] -> osm_ucast_mgr_process: Min Hop Tables configured on all switches.
Sep 13 12:07:45 274911 [417FF970] -> __osm_trap_rcv_process_request: Received Generic Notice type:0x01 num:128 Producer:2 from LID:0x000E TID:0x0000000000000048
Sep 13 12:07:45 275049 [417FF970] -> osm_report_notice: Reporting Generic Notice type:1 num:128 from LID:0x0000 GID:0xfe80000000000000,0x0002c90200402915
Sep 13 12:07:45 690746 [42FFF970] -> umad_receiver: ERR 5409: send completed with error (method=1 attr=11) -- dropping.
Sep 13 12:07:45 690756 [42FFF970] -> umad_receiver: ERR 5411: DR SMP hop ptr 0 hop count 3 DR SLID 0x0 DR DLID 0x0
Sep 13 12:07:45 690765 [42FFF970] -> __osm_sm_mad_ctrl_send_err_cb: ERR 3113: MAD completed in error (IB_TIMEOUT).
Sep 13 12:07:45 690799 [42FFF970] -> SMP dump:
base_ver................0x1
mgmt_class..............0x81
class_ver...............0x1
method..................0x1 (SubnGet)
D bit...................0x0
status..................0x0
hop_ptr.................0x0
hop_count...............0x3
trans_id................0x3ac6
attr_id.................0x11 (NodeInfo)
resv....................0x0
attr_mod................0x0
m_key...................0x0000000000000000
dr_slid.................0xFFFF
dr_dlid.................0xFFFF
Initial path: [0][1][D][C]
Return path: [0][0][0][0]
Reserved: [0][0][0][0][0][0][0]
00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
Sep 13 12:07:45 699968 [417FF970] -> osm_drop_mgr_process: ERR 0108: Unknown remote side for node 0x0002c90200402915 port 12. Adding to light sweep sampling list.
Sep 13 12:07:45 699979 [417FF970] -> Directed Path Dump of 2 hop path:
Path = [0][1][D]
Sep 13 12:07:45 700641 [417FF970] -> osm_ucast_mgr_process: Min Hop Tables configured on all switches.
Sep 13 12:07:45 706675 [40FFF970] -> __osm_trap_rcv_process_request: Received Generic Notice type:0x01 num:128 Producer:2 from LID:0x000E TID:0x0000000000000049
Sep 13 12:07:45 706808 [40FFF970] -> osm_report_notice: Reporting Generic Notice type:1 num:128 from LID:0x0000 GID:0xfe80000000000000,0x0002c90200402915
Sep 13 12:07:46 122749 [42FFF970] -> umad_receiver: ERR 5409: send completed with error (method=1 attr=11) -- dropping.
Sep 13 12:07:46 122758 [42FFF970] -> umad_receiver: ERR 5411: DR SMP hop ptr 0 hop count 3 DR SLID 0x0 DR DLID 0x0
Sep 13 12:07:46 122768 [42FFF970] -> __osm_sm_mad_ctrl_send_err_cb: ERR 3113: MAD completed in error (IB_TIMEOUT).
Sep 13 12:07:46 122802 [42FFF970] -> SMP dump:
base_ver................0x1
mgmt_class..............0x81
class_ver...............0x1
method..................0x1 (SubnGet)
D bit...................0x0
status..................0x0
hop_ptr.................0x0
hop_count...............0x3
trans_id................0x3b77
attr_id.................0x11 (NodeInfo)
resv....................0x0
attr_mod................0x0
m_key...................0x0000000000000000
dr_slid.................0xFFFF
dr_dlid.................0xFFFF
Initial path: [0][1][D][C]
Return path: [0][0][0][0]
Reserved: [0][0][0][0][0][0][0]
00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
Sep 13 12:07:46 131999 [417FF970] -> osm_drop_mgr_process: ERR 0108: Unknown remote side for node 0x0002c90200402915 port 12. Adding to light sweep sampling list.
Sep 13 12:07:46 132009 [417FF970] -> Directed Path Dump of 2 hop path:
Path = [0][1][D]
Sep 13 12:07:46 132665 [417FF970] -> osm_ucast_mgr_process: Min Hop Tables configured on all switches.
Sep 13 12:07:46 138697 [40FFF970] -> __osm_trap_rcv_process_request: Received Generic Notice type:0x01 num:128 Producer:2 from LID:0x000E TID:0x000000000000004a
Sep 13 12:07:46 138829 [40FFF970] -> osm_report_notice: Reporting Generic Notice type:1 num:128 from LID:0x0000 GID:0xfe80000000000000,0x0002c90200402915
Sep 13 12:07:46 554753 [42FFF970] -> umad_receiver: ERR 5409: send completed with error (method=1 attr=11) -- dropping.
Sep 13 12:07:46 554764 [42FFF970] -> umad_receiver: ERR 5411: DR SMP hop ptr 0 hop count 3 DR SLID 0x0 DR DLID 0x0
Sep 13 12:07:46 554774 [42FFF970] -> __osm_sm_mad_ctrl_send_err_cb: ERR 3113: MAD completed in error (IB_TIMEOUT).
Sep 13 12:07:46 554808 [42FFF970] -> SMP dump:
base_ver................0x1
mgmt_class..............0x81
class_ver...............0x1
method..................0x1 (SubnGet)
D bit...................0x0
status..................0x0
hop_ptr.................0x0
hop_count...............0x3
trans_id................0x3c28
attr_id.................0x11 (NodeInfo)
resv....................0x0
attr_mod................0x0
m_key...................0x0000000000000000
dr_slid.................0xFFFF
dr_dlid.................0xFFFF
Initial path: [0][1][D][C]
Return path: [0][0][0][0]
Reserved: [0][0][0][0][0][0][0]
00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
Sep 13 12:07:46 563953 [40FFF970] -> osm_drop_mgr_process: ERR 0108: Unknown remote side for node 0x0002c90200402915 port 12. Adding to light sweep sampling list.
Sep 13 12:07:46 563967 [40FFF970] -> Directed Path Dump of 2 hop path:
Path = [0][1][D]
Sep 13 12:07:46 564679 [40FFF970] -> osm_ucast_mgr_process: Min Hop Tables configured on all switches.
Sep 13 12:07:46 570953 [417FF970] -> __osm_trap_rcv_process_request: Received Generic Notice type:0x01 num:128 Producer:2 from LID:0x000E TID:0x000000000000004b
Sep 13 12:07:46 571090 [417FF970] -> osm_report_notice: Reporting Generic Notice type:1 num:128 from LID:0x0000 GID:0xfe80000000000000,0x0002c90200402915
Sep 13 12:07:46 986754 [42FFF970] -> umad_receiver: ERR 5409: send completed with error (method=1 attr=11) -- dropping.
Sep 13 12:07:46 986762 [42FFF970] -> umad_receiver: ERR 5411: DR SMP hop ptr 0 hop count 3 DR SLID 0x0 DR DLID 0x0
Sep 13 12:07:46 986772 [42FFF970] -> __osm_sm_mad_ctrl_send_err_cb: ERR 3113: MAD completed in error (IB_TIMEOUT).
Sep 13 12:07:46 986805 [42FFF970] -> SMP dump:
base_ver................0x1
mgmt_class..............0x81
class_ver...............0x1
method..................0x1 (SubnGet)
D bit...................0x0
status..................0x0
hop_ptr.................0x0
hop_count...............0x3
trans_id................0x3cd9
attr_id.................0x11 (NodeInfo)
resv....................0x0
attr_mod................0x0
m_key...................0x0000000000000000
dr_slid.................0xFFFF
dr_dlid.................0xFFFF
Initial path: [0][1][D][C]
Return path: [0][0][0][0]
Reserved: [0][0][0][0][0][0][0]
00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
Sep 13 12:07:46 996022 [417FF970] -> osm_drop_mgr_process: ERR 0108: Unknown remote side for node 0x0002c90200402915 port 12. Adding to light sweep sampling list.
Sep 13 12:07:46 996033 [417FF970] -> Directed Path Dump of 2 hop path:
Path = [0][1][D]
Sep 13 12:07:46 996694 [417FF970] -> osm_ucast_mgr_process: Min Hop Tables configured on all switches.
Sep 13 12:07:47 002730 [40FFF970] -> __osm_trap_rcv_process_request: Received Generic Notice type:0x01 num:128 Producer:2 from LID:0x000E TID:0x000000000000004c
Sep 13 12:07:47 002861 [40FFF970] -> osm_report_notice: Reporting Generic Notice type:1 num:128 from LID:0x0000 GID:0xfe80000000000000,0x0002c90200402915
Sep 13 12:07:47 418759 [42FFF970] -> umad_receiver: ERR 5409: send completed with error (method=1 attr=11) -- dropping.
Sep 13 12:07:47 418772 [42FFF970] -> umad_receiver: ERR 5411: DR SMP hop ptr 0 hop count 3 DR SLID 0x0 DR DLID 0x0
Sep 13 12:07:47 418781 [42FFF970] -> __osm_sm_mad_ctrl_send_err_cb: ERR 3113: MAD completed in error (IB_TIMEOUT).
Sep 13 12:07:47 418816 [42FFF970] -> SMP dump:
base_ver................0x1
mgmt_class..............0x81
class_ver...............0x1
method..................0x1 (SubnGet)
D bit...................0x0
status..................0x0
hop_ptr.................0x0
hop_count...............0x3
trans_id................0x3d8a
attr_id.................0x11 (NodeInfo)
resv....................0x0
attr_mod................0x0
m_key...................0x0000000000000000
dr_slid.................0xFFFF
dr_dlid.................0xFFFF
Initial path: [0][1][D][C]
Return path: [0][0][0][0]
Reserved: [0][0][0][0][0][0][0]
00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
Sep 13 12:07:47 428089 [40FFF970] -> osm_drop_mgr_process: ERR 0108: Unknown remote side for node 0x0002c90200402915 port 12. Adding to light sweep sampling list.
Sep 13 12:07:47 428103 [40FFF970] -> Directed Path Dump of 2 hop path:
Path = [0][1][D]
Sep 13 12:07:47 428797 [40FFF970] -> osm_ucast_mgr_process: Min Hop Tables configured on all switches.
Sep 13 12:07:47 435076 [417FF970] -> __osm_trap_rcv_process_request: Received Generic Notice type:0x01 num:128 Producer:2 from LID:0x000E TID:0x000000000000004d
Sep 13 12:07:47 435212 [417FF970] -> osm_report_notice: Reporting Generic Notice type:1 num:128 from LID:0x0000 GID:0xfe80000000000000,0x0002c90200402915
Sep 13 12:07:47 850760 [42FFF970] -> umad_receiver: ERR 5409: send completed with error (method=1 attr=11) -- dropping.
Sep 13 12:07:47 850768 [42FFF970] -> umad_receiver: ERR 5411: DR SMP hop ptr 0 hop count 3 DR SLID 0x0 DR DLID 0x0
Sep 13 12:07:47 850777 [42FFF970] -> __osm_sm_mad_ctrl_send_err_cb: ERR 3113: MAD completed in error (IB_TIMEOUT).
Sep 13 12:07:47 850811 [42FFF970] -> SMP dump:
base_ver................0x1
mgmt_class..............0x81
class_ver...............0x1
method..................0x1 (SubnGet)
D bit...................0x0
status..................0x0
hop_ptr.................0x0
hop_count...............0x3
trans_id................0x3e3b
attr_id.................0x11 (NodeInfo)
resv....................0x0
attr_mod................0x0
m_key...................0x0000000000000000
dr_slid.................0xFFFF
dr_dlid.................0xFFFF
Initial path: [0][1][D][C]
Return path: [0][0][0][0]
Reserved: [0][0][0][0][0][0][0]
00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
Sep 13 12:07:47 860066 [417FF970] -> osm_drop_mgr_process: ERR 0108: Unknown remote side for node 0x0002c90200402915 port 12. Adding to light sweep sampling list.
Sep 13 12:07:47 860077 [417FF970] -> Directed Path Dump of 2 hop path:
Path = [0][1][D]
Sep 13 12:07:47 860740 [417FF970] -> osm_ucast_mgr_process: Min Hop Tables configured on all switches.
Sep 13 12:07:47 866763 [40FFF970] -> __osm_trap_rcv_process_request: Received Generic Notice type:0x01 num:128 Producer:2 from LID:0x000E TID:0x000000000000004e
Sep 13 12:07:47 866894 [40FFF970] -> osm_report_notice: Reporting Generic Notice type:1 num:128 from LID:0x0000 GID:0xfe80000000000000,0x0002c90200402915
Sep 13 12:07:48 282763 [42FFF970] -> umad_receiver: ERR 5409: send completed with error (method=1 attr=11) -- dropping.
Sep 13 12:07:48 282772 [42FFF970] -> umad_receiver: ERR 5411: DR SMP hop ptr 0 hop count 3 DR SLID 0x0 DR DLID 0x0
Sep 13 12:07:48 282784 [42FFF970] -> __osm_sm_mad_ctrl_send_err_cb: ERR 3113: MAD completed in error (IB_TIMEOUT).
Sep 13 12:07:48 282818 [42FFF970] -> SMP dump:
base_ver................0x1
mgmt_class..............0x81
class_ver...............0x1
method..................0x1 (SubnGet)
D bit...................0x0
status..................0x0
hop_ptr.................0x0
hop_count...............0x3
trans_id................0x3eec
attr_id.................0x11 (NodeInfo)
resv....................0x0
attr_mod................0x0
m_key...................0x0000000000000000
dr_slid.................0xFFFF
dr_dlid.................0xFFFF
Initial path: [0][1][D][C]
Return path: [0][0][0][0]
Reserved: [0][0][0][0][0][0][0]
00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
Sep 13 12:07:48 291948 [417FF970] -> osm_drop_mgr_process: ERR 0108: Unknown remote side for node 0x0002c90200402915 port 12. Adding to light sweep sampling list.
Sep 13 12:07:48 291959 [417FF970] -> Directed Path Dump of 2 hop path:
Path = [0][1][D]
Sep 13 12:07:48 292615 [417FF970] -> osm_ucast_mgr_process: Min Hop Tables configured on all switches.
Sep 13 12:07:48 298632 [40FFF970] -> __osm_trap_rcv_process_request: Received Generic Notice type:0x01 num:128 Producer:2 from LID:0x000E TID:0x000000000000004f
Sep 13 12:07:48 298685 [40FFF970] -> __osm_trap_rcv_process_request: ERR 3804: Received the trap 11 times continuously.
Sep 13 12:07:53 118795 [42FFF970] -> umad_receiver: ERR 5409: send completed with error (method=1 attr=15) -- dropping.
Sep 13 12:07:53 118805 [42FFF970] -> umad_receiver: ERR 5411: DR SMP hop ptr 0 hop count 3 DR SLID 0x0 DR DLID 0x0
Sep 13 12:07:53 118815 [42FFF970] -> __osm_sm_mad_ctrl_send_err_cb: ERR 3113: MAD completed in error (IB_TIMEOUT).
Sep 13 12:07:53 118849 [42FFF970] -> SMP dump:
base_ver................0x1
mgmt_class..............0x81
class_ver...............0x1
method..................0x1 (SubnGet)
D bit...................0x0
status..................0x0
hop_ptr.................0x0
hop_count...............0x3
trans_id................0x3f22
attr_id.................0x15 (PortInfo)
resv....................0x0
attr_mod................0x0
m_key...................0x0000000000000000
dr_slid.................0xFFFF
dr_dlid.................0xFFFF
Initial path: [0][1][D][C]
Return path: [0][0][0][0]
Reserved: [0][0][0][0][0][0][0]
00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
Sep 13 12:07:53 534798 [42FFF970] -> umad_receiver: ERR 5409: send completed with error (method=1 attr=11) -- dropping.
Sep 13 12:07:53 534808 [42FFF970] -> umad_receiver: ERR 5411: DR SMP hop ptr 0 hop count 3 DR SLID 0x0 DR DLID 0x0
Sep 13 12:07:53 534817 [42FFF970] -> __osm_sm_mad_ctrl_send_err_cb: ERR 3113: MAD completed in error (IB_TIMEOUT).
Sep 13 12:07:53 534851 [42FFF970] -> SMP dump:
base_ver................0x1
mgmt_class..............0x81
class_ver...............0x1
method..................0x1 (SubnGet)
D bit...................0x0
status..................0x0
hop_ptr.................0x0
hop_count...............0x3
trans_id................0x3fa0
attr_id.................0x11 (NodeInfo)
resv....................0x0
attr_mod................0x0
m_key...................0x0000000000000000
dr_slid.................0xFFFF
dr_dlid.................0xFFFF
Initial path: [0][1][D][C]
Return path: [0][0][0][0]
Reserved: [0][0][0][0][0][0][0]
00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
Sep 13 12:07:53 544030 [417FF970] -> osm_drop_mgr_process: ERR 0108: Unknown remote side for node 0x0002c90200402915 port 12. Adding to light sweep sampling list.
Sep 13 12:07:53 544041 [417FF970] -> Directed Path Dump of 2 hop path:
Path = [0][1][D]
Sep 13 12:07:53 544705 [417FF970] -> osm_ucast_mgr_process: Min Hop Tables configured on all switches.
Sep 13 12:07:53 550724 [40FFF970] -> __osm_trap_rcv_process_request: Received Generic Notice type:0x01 num:128 Producer:2 from LID:0x000E TID:0x0000000000000050
Sep 13 12:07:53 550852 [40FFF970] -> osm_report_notice: Reporting Generic Notice type:1 num:128 from LID:0x0000 GID:0xfe80000000000000,0x0002c90200402915
Sep 13 12:07:53 966800 [42FFF970] -> umad_receiver: ERR 5409: send completed with error (method=1 attr=11) -- dropping.
Sep 13 12:07:53 966810 [42FFF970] -> umad_receiver: ERR 5411: DR SMP hop ptr 0 hop count 3 DR SLID 0x0 DR DLID 0x0
Sep 13 12:07:53 966819 [42FFF970] -> __osm_sm_mad_ctrl_send_err_cb: ERR 3113: MAD completed in error (IB_TIMEOUT).
Sep 13 12:07:53 966853 [42FFF970] -> SMP dump:
base_ver................0x1
mgmt_class..............0x81
class_ver...............0x1
method..................0x1 (SubnGet)
D bit...................0x0
status..................0x0
hop_ptr.................0x0
hop_count...............0x3
trans_id................0x4051
attr_id.................0x11 (NodeInfo)
resv....................0x0
attr_mod................0x0
m_key...................0x0000000000000000
dr_slid.................0xFFFF
dr_dlid.................0xFFFF
Initial path: [0][1][D][C]
Return path: [0][0][0][0]
Reserved: [0][0][0][0][0][0][0]
00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
Sep 13 12:07:53 975975 [417FF970] -> osm_drop_mgr_process: ERR 0108: Unknown remote side for node 0x0002c90200402915 port 12. Adding to light sweep sampling list.
Sep 13 12:07:53 975985 [417FF970] -> Directed Path Dump of 2 hop path:
Path = [0][1][D]
Sep 13 12:07:53 976641 [417FF970] -> osm_ucast_mgr_process: Min Hop Tables configured on all switches.
Sep 13 12:07:53 982670 [40FFF970] -> __osm_trap_rcv_process_request: Received Generic Notice type:0x01 num:128 Producer:2 from LID:0x000E TID:0x0000000000000051
Sep 13 12:07:53 982799 [40FFF970] -> osm_report_notice: Reporting Generic Notice type:1 num:128 from LID:0x0000 GID:0xfe80000000000000,0x0002c90200402915
Sep 13 12:07:54 398806 [42FFF970] -> umad_receiver: ERR 5409: send completed with error (method=1 attr=11) -- dropping.
Sep 13 12:07:54 398816 [42FFF970] -> umad_receiver: ERR 5411: DR SMP hop ptr 0 hop count 3 DR SLID 0x0 DR DLID 0x0
Sep 13 12:07:54 398826 [42FFF970] -> __osm_sm_mad_ctrl_send_err_cb: ERR 3113: MAD completed in error (IB_TIMEOUT).
Sep 13 12:07:54 398860 [42FFF970] -> SMP dump:
base_ver................0x1
mgmt_class..............0x81
class_ver...............0x1
method..................0x1 (SubnGet)
D bit...................0x0
status..................0x0
hop_ptr.................0x0
hop_count...............0x3
trans_id................0x4102
attr_id.................0x11 (NodeInfo)
resv....................0x0
attr_mod................0x0
m_key...................0x0000000000000000
dr_slid.................0xFFFF
dr_dlid.................0xFFFF
Initial path: [0][1][D][C]
Return path: [0][0][0][0]
Reserved: [0][0][0][0][0][0][0]
00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
Sep 13 12:07:54 408050 [40FFF970] -> osm_drop_mgr_process: ERR 0108: Unknown remote side for node 0x0002c90200402915 port 12. Adding to light sweep sampling list.
Sep 13 12:07:54 408070 [40FFF970] -> Directed Path Dump of 2 hop path:
Path = [0][1][D]
Sep 13 12:07:54 408805 [40FFF970] -> osm_ucast_mgr_process: Min Hop Tables configured on all switches.
Sep 13 12:07:54 415078 [417FF970] -> __osm_trap_rcv_process_request: Received Generic Notice type:0x01 num:128 Producer:2 from LID:0x000E TID:0x0000000000000052
Sep 13 12:07:54 415215 [417FF970] -> osm_report_notice: Reporting Generic Notice type:1 num:128 from LID:0x0000 GID:0xfe80000000000000,0x0002c90200402915
Sep 13 12:07:54 830806 [42FFF970] -> umad_receiver: ERR 5409: send completed with error (method=1 attr=11) -- dropping.
Sep 13 12:07:54 830814 [42FFF970] -> umad_receiver: ERR 5411: DR SMP hop ptr 0 hop count 3 DR SLID 0x0 DR DLID 0x0
Sep 13 12:07:54 830824 [42FFF970] -> __osm_sm_mad_ctrl_send_err_cb: ERR 3113: MAD completed in error (IB_TIMEOUT).
Sep 13 12:07:54 830857 [42FFF970] -> SMP dump:
base_ver................0x1
mgmt_class..............0x81
class_ver...............0x1
method..................0x1 (SubnGet)
D bit...................0x0
status..................0x0
hop_ptr.................0x0
hop_count...............0x3
trans_id................0x41b3
attr_id.................0x11 (NodeInfo)
resv....................0x0
attr_mod................0x0
m_key...................0x0000000000000000
dr_slid.................0xFFFF
dr_dlid.................0xFFFF
Initial path: [0][1][D][C]
Return path: [0][0][0][0]
Reserved: [0][0][0][0][0][0][0]
00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
Sep 13 12:07:54 840063 [417FF970] -> osm_drop_mgr_process: ERR 0108: Unknown remote side for node 0x0002c90200402915 port 12. Adding to light sweep sampling list.
Sep 13 12:07:54 840074 [417FF970] -> Directed Path Dump of 2 hop path:
Path = [0][1][D]
Sep 13 12:07:54 840735 [417FF970] -> osm_ucast_mgr_process: Min Hop Tables configured on all switches.
Sep 13 12:07:54 846766 [40FFF970] -> __osm_trap_rcv_process_request: Received Generic Notice type:0x01 num:128 Producer:2 from LID:0x000E TID:0x0000000000000053
Sep 13 12:07:54 846898 [40FFF970] -> osm_report_notice: Reporting Generic Notice type:1 num:128 from LID:0x0000 GID:0xfe80000000000000,0x0002c90200402915
Sep 13 12:07:55 262809 [42FFF970] -> umad_receiver: ERR 5409: send completed with error (method=1 attr=11) -- dropping.
Sep 13 12:07:55 262819 [42FFF970] -> umad_receiver: ERR 5411: DR SMP hop ptr 0 hop count 3 DR SLID 0x0 DR DLID 0x0
Sep 13 12:07:55 262829 [42FFF970] -> __osm_sm_mad_ctrl_send_err_cb: ERR 3113: MAD completed in error (IB_TIMEOUT).
Sep 13 12:07:55 262863 [42FFF970] -> SMP dump:
base_ver................0x1
mgmt_class..............0x81
class_ver...............0x1
method..................0x1 (SubnGet)
D bit...................0x0
status..................0x0
hop_ptr.................0x0
hop_count...............0x3
trans_id................0x4264
attr_id.................0x11 (NodeInfo)
resv....................0x0
attr_mod................0x0
m_key...................0x0000000000000000
dr_slid.................0xFFFF
dr_dlid.................0xFFFF
Initial path: [0][1][D][C]
Return path: [0][0][0][0]
Reserved: [0][0][0][0][0][0][0]
00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
Sep 13 12:07:55 272123 [417FF970] -> osm_drop_mgr_process: ERR 0108: Unknown remote side for node 0x0002c90200402915 port 12. Adding to light sweep sampling list.
Sep 13 12:07:55 272133 [417FF970] -> Directed Path Dump of 2 hop path:
Path = [0][1][D]
Sep 13 12:07:55 272788 [417FF970] -> osm_ucast_mgr_process: Min Hop Tables configured on all switches.
Sep 13 12:07:55 278813 [40FFF970] -> __osm_trap_rcv_process_request: Received Generic Notice type:0x01 num:128 Producer:2 from LID:0x000E TID:0x0000000000000054
Sep 13 12:07:55 278943 [40FFF970] -> osm_report_notice: Reporting Generic Notice type:1 num:128 from LID:0x0000 GID:0xfe80000000000000,0x0002c90200402915
Sep 13 12:07:55 694814 [42FFF970] -> umad_receiver: ERR 5409: send completed with error (method=1 attr=11) -- dropping.
Sep 13 12:07:55 694824 [42FFF970] -> umad_receiver: ERR 5411: DR SMP hop ptr 0 hop count 3 DR SLID 0x0 DR DLID 0x0
Sep 13 12:07:55 694834 [42FFF970] -> __osm_sm_mad_ctrl_send_err_cb: ERR 3113: MAD completed in error (IB_TIMEOUT).
Sep 13 12:07:55 694868 [42FFF970] -> SMP dump:
base_ver................0x1
mgmt_class..............0x81
class_ver...............0x1
method..................0x1 (SubnGet)
D bit...................0x0
status..................0x0
hop_ptr.................0x0
hop_count...............0x3
trans_id................0x4315
attr_id.................0x11 (NodeInfo)
resv....................0x0
attr_mod................0x0
m_key...................0x0000000000000000
dr_slid.................0xFFFF
dr_dlid.................0xFFFF
Initial path: [0][1][D][C]
Return path: [0][0][0][0]
Reserved: [0][0][0][0][0][0][0]
00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
Sep 13 12:07:55 704098 [40FFF970] -> osm_drop_mgr_process: ERR 0108: Unknown remote side for node 0x0002c90200402915 port 12. Adding to light sweep sampling list.
Sep 13 12:07:55 704112 [40FFF970] -> Directed Path Dump of 2 hop path:
Path = [0][1][D]
Sep 13 12:07:55 704813 [40FFF970] -> osm_ucast_mgr_process: Min Hop Tables configured on all switches.
Sep 13 12:07:55 711069 [417FF970] -> __osm_trap_rcv_process_request: Received Generic Notice type:0x01 num:128 Producer:2 from LID:0x000E TID:0x0000000000000055
Sep 13 12:07:55 711206 [417FF970] -> osm_report_notice: Reporting Generic Notice type:1 num:128 from LID:0x0000 GID:0xfe80000000000000,0x0002c90200402915
Sep 13 12:07:56 126814 [42FFF970] -> umad_receiver: ERR 5409: send completed with error (method=1 attr=11) -- dropping.
Sep 13 12:07:56 126822 [42FFF970] -> umad_receiver: ERR 5411: DR SMP hop ptr 0 hop count 3 DR SLID 0x0 DR DLID 0x0
Sep 13 12:07:56 126832 [42FFF970] -> __osm_sm_mad_ctrl_send_err_cb: ERR 3113: MAD completed in error (IB_TIMEOUT).
Sep 13 12:07:56 126866 [42FFF970] -> SMP dump:
base_ver................0x1
mgmt_class..............0x81
class_ver...............0x1
method..................0x1 (SubnGet)
D bit...................0x0
status..................0x0
hop_ptr.................0x0
hop_count...............0x3
trans_id................0x43c6
attr_id.................0x11 (NodeInfo)
resv....................0x0
attr_mod................0x0
m_key...................0x0000000000000000
dr_slid.................0xFFFF
dr_dlid.................0xFFFF
Initial path: [0][1][D][C]
Return path: [0][0][0][0]
Reserved: [0][0][0][0][0][0][0]
00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
Sep 13 12:07:56 136024 [417FF970] -> osm_drop_mgr_process: ERR 0108: Unknown remote side for node 0x0002c90200402915 port 12. Adding to light sweep sampling list.
Sep 13 12:07:56 136035 [417FF970] -> Directed Path Dump of 2 hop path:
Path = [0][1][D]
Sep 13 12:07:56 136696 [417FF970] -> osm_ucast_mgr_process: Min Hop Tables configured on all switches.
Sep 13 12:07:56 142734 [40FFF970] -> __osm_trap_rcv_process_request: Received Generic Notice type:0x01 num:128 Producer:2 from LID:0x000E TID:0x0000000000000056
Sep 13 12:07:56 142865 [40FFF970] -> osm_report_notice: Reporting Generic Notice type:1 num:128 from LID:0x0000 GID:0xfe80000000000000,0x0002c90200402915
Sep 13 12:07:56 558818 [42FFF970] -> umad_receiver: ERR 5409: send completed with error (method=1 attr=11) -- dropping.
Sep 13 12:07:56 558831 [42FFF970] -> umad_receiver: ERR 5411: DR SMP hop ptr 0 hop count 3 DR SLID 0x0 DR DLID 0x0
Sep 13 12:07:56 558841 [42FFF970] -> __osm_sm_mad_ctrl_send_err_cb: ERR 3113: MAD completed in error (IB_TIMEOUT).
Sep 13 12:07:56 558875 [42FFF970] -> SMP dump:
base_ver................0x1
mgmt_class..............0x81
class_ver...............0x1
method..................0x1 (SubnGet)
D bit...................0x0
status..................0x0
hop_ptr.................0x0
hop_count...............0x3
trans_id................0x4477
attr_id.................0x11 (NodeInfo)
resv....................0x0
attr_mod................0x0
m_key...................0x0000000000000000
dr_slid.................0xFFFF
dr_dlid.................0xFFFF
Initial path: [0][1][D][C]
Return path: [0][0][0][0]
Reserved: [0][0][0][0][0][0][0]
00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
Sep 13 12:07:56 568129 [417FF970] -> osm_drop_mgr_process: ERR 0108: Unknown remote side for node 0x0002c90200402915 port 12. Adding to light sweep sampling list.
Sep 13 12:07:56 568140 [417FF970] -> Directed Path Dump of 2 hop path:
Path = [0][1][D]
Sep 13 12:07:56 568797 [417FF970] -> osm_ucast_mgr_process: Min Hop Tables configured on all switches.
Sep 13 12:07:56 574831 [40FFF970] -> __osm_trap_rcv_process_request: Received Generic Notice type:0x01 num:128 Producer:2 from LID:0x000E TID:0x0000000000000057
Sep 13 12:07:56 574961 [40FFF970] -> osm_report_notice: Reporting Generic Notice type:1 num:128 from LID:0x0000 GID:0xfe80000000000000,0x0002c90200402915
Sep 13 12:07:56 719968 [417FF970] -> __osm_trap_rcv_process_request: Received Generic Notice type:0x01 num:128 Producer:2 from LID:0x000E TID:0x0000000000000058
Sep 13 12:07:56 720052 [417FF970] -> osm_report_notice: Reporting Generic Notice type:1 num:128 from LID:0x0000 GID:0xfe80000000000000,0x0002c90200402915
Sep 13 12:07:56 990820 [42FFF970] -> umad_receiver: ERR 5409: send completed with error (method=1 attr=11) -- dropping.
Sep 13 12:07:56 990828 [42FFF970] -> umad_receiver: ERR 5411: DR SMP hop ptr 0 hop count 3 DR SLID 0x0 DR DLID 0x0
Sep 13 12:07:56 990837 [42FFF970] -> __osm_sm_mad_ctrl_send_err_cb: ERR 3113: MAD completed in error (IB_TIMEOUT).
Sep 13 12:07:56 990871 [42FFF970] -> SMP dump:
base_ver................0x1
mgmt_class..............0x81
class_ver...............0x1
method..................0x1 (SubnGet)
D bit...................0x0
status..................0x0
hop_ptr.................0x0
hop_count...............0x3
trans_id................0x4528
attr_id.................0x11 (NodeInfo)
resv....................0x0
attr_mod................0x0
m_key...................0x0000000000000000
dr_slid.................0xFFFF
dr_dlid.................0xFFFF
Initial path: [0][1][D][C]
Return path: [0][0][0][0]
Reserved: [0][0][0][0][0][0][0]
00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
Sep 13 12:07:57 000088 [417FF970] -> osm_drop_mgr_process: ERR 0108: Unknown remote side for node 0x0002c90200402915 port 12. Adding to light sweep sampling list.
Sep 13 12:07:57 000098 [417FF970] -> Directed Path Dump of 2 hop path:
Path = [0][1][D]
Sep 13 12:07:57 000754 [417FF970] -> osm_ucast_mgr_process: Min Hop Tables configured on all switches.
Sep 13 12:07:57 005797 [42FFF970] -> __osm_sm_mad_ctrl_rcv_callback: ERR 3111: Error status = 0x1C00.
Sep 13 12:07:57 005832 [42FFF970] -> SMP dump:
base_ver................0x1
mgmt_class..............0x81
class_ver...............0x1
method..................0x81 (SubnGetResp)
D bit...................0x1
status..................0x1C00
hop_ptr.................0x0
hop_count...............0x2
trans_id................0x455a
attr_id.................0x15 (PortInfo)
resv....................0x0
attr_mod................0xC
m_key...................0x0000000000000000
dr_slid.................0xFFFF
dr_dlid.................0xFFFF
Initial path: [0][1][D]
Return path: [0][1][18]
Reserved: [0][0][0][0][0][0][0]
00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
00 00 00 00 00 00 00 00 00 00 00 00 18 03 03 02
31 22 00 13 40 40 00 08 08 04 F2 40 00 00 00 00
00 00 00 00 00 88 00 00 00 00 00 00 00 00 00 00
Sep 13 12:07:57 005891 [40FFF970] -> osm_pi_rcv_process_set: ERR 0F10: Received Error Status for SetResp()
Sep 13 12:07:57 005908 [40FFF970] -> PortInfo dump:
port number.............0xC
node_guid...............0x0002c90200402915
port_guid...............0x0002c90200402915
m_key...................0x0000000000000000
subnet_prefix...........0x0000000000000000
base_lid................0x0
master_sm_base_lid......0x0
capability_mask.........0x0
diag_code...............0x0
m_key_lease_period......0x0
local_port_num..........0x18
link_width_enabled......0x3
link_width_supported....0x3
link_width_active.......0x2
link_speed_supported....0x3
port_state..............DOWN
state_info2.............0x22
m_key_protect_bits......0x0
lmc.....................0x0
link_speed..............0x13
mtu_smsl................0x40
vl_cap_init_type........0x40
vl_high_limit...........0x0
vl_arb_high_cap.........0x8
vl_arb_low_cap..........0x8
init_rep_mtu_cap........0x4
vl_stall_life...........0xF2
vl_enforce..............0x40
m_key_violations........0x0
p_key_violations........0x0
q_key_violations........0x0
guid_cap................0x0
subnet_timeout..........0x0
resp_time_value.........0x0
error_threshold.........0x88
Sep 13 12:07:57 005951 [40FFF970] -> Capabilities Mask:
More information about the general
mailing list