[openib-general] [oops] recent opensm crash

Hal Rosenstock halr at voltaire.com
Fri Jul 1 10:24:37 PDT 2005


On Thu, 2005-06-30 at 18:10, Tom Duffy wrote:
> #0  stack_dump () at src/stack.c:72
> 72              if (!__builtin_frame_address(2))
> (gdb) bt
> #0  stack_dump () at src/stack.c:72
> #1  0x00002aaaaacbd1a6 in handler (x=11) at src/stack.c:151
> #2  <signal handler called>
> #3  __osm_sm_mad_ctrl_send_err_cb (bind_context=0x550dd8, p_madw=0x567820)
>     at osm_sm_mad_ctrl.c:832
> #4  0x00002aaaaaaaeeed in osm_vendor_send (h_bind=0x586920, p_madw=0x567820,
>     resp_expected=1) at osm_vendor_ibumad.c:889

I found one problem associated with this and just checked in a patch.
I'm not sure whether there is another one behind this or not. Any
reliable way to recreate this ?

-- Hal

> #5  0x000000000042ef72 in __osm_vl15_poller (p_ptr=0x552620) at osm_madw.h:933
> #6  0x00002aaaaadc911e in __cl_thread_wrapper (arg=0x0) at cl_thread.c:61
> #7  0x00000036d28060aa in start_thread () from /lib64/tls/libpthread.so.0
> #8  0x00000036d19c53d3 in clone () from /lib64/tls/libc.so.6
> #9  0x0000000000000000 in ?? ()
> 
> I was bringing up and down an node when this happened.
> 
> Attached are the last 500 lines from osm.log.
> 
> -tduffy
> 
> ______________________________________________________________________
> 
> 				base_ver................0x1
> 				mgmt_class..............0x81
> 				class_ver...............0x1
> 				method..................0x1 (SubnGet)
> 				status..................0x0
> 				hop_ptr.................0x0
> 				hop_count...............0x1
> 				trans_id................0x16c6a
> 				attr_id.................0x16 (P_KeyTable)
> 				resv....................0x0
> 				attr_mod................0x50000
> 				m_key...................0x0000000000000000
> 				dr_slid.................0xFFFF
> 				dr_dlid.................0xFFFF
> 
> 				Initial path: [0][1]
> 				Return path:  [0][0]
> 				Reserved:     [0][0][0][0][0][0][0]
> 
> 				00 00 00 00 00 00 00 00   00 00 00 00 00 00 00 00
> 
> 				00 00 00 00 00 00 00 00   00 00 00 00 00 00 00 00
> 
> 				00 00 00 00 00 00 00 00   00 00 00 00 00 00 00 00
> 
> 				00 00 00 00 00 00 00 00   00 00 00 00 00 00 00 00
> 
> Jun 30 14:59:00 [43806960] -> osm_vendor_send: [
> Jun 30 14:59:00 [43005960] -> PortInfo dump:
> 				port number.............0x8
> 				node_guid...............0x000617000000000d
> 				port_guid...............0x000617000000000d
> 				m_key...................0x0000000000000000
> 				subnet_prefix...........0x0000000000000000
> 				base_lid................0x0
> 				master_sm_base_lid......0x0
> 				capability_mask.........0x0
> 				diag_code...............0x0
> 				m_key_lease_period......0x0
> 				local_port_num..........0x7
> 				link_width_enabled......0x3
> 				link_width_supported....0x3
> 				link_width_active.......0x0
> 				link_speed_supported....0x1
> 				port_state..............DOWN
> 				state_info2.............0x22
> 				m_key_protect_bits......0x0
> 				lmc.....................0x0
> 				link_speed..............0x11
> 				mtu_smsl................0x10
> 				vl_cap..................0x40
> 				vl_high_limit...........0x0
> 				vl_arb_high_cap.........0x20
> 				vl_arb_low_cap..........0x20
> 				mtu_cap.................0x5
> 				vl_stall_life...........0x8
> 				vl_enforce..............0x10
> 				m_key_violations........0x0
> 				p_key_violations........0x0
> 				q_key_violations........0x0
> 				guid_cap................0x0
> 				subnet_timeout..........0x0
> 				resp_time_value.........0x0
> 				error_threshold.........0xFF
> Jun 30 14:59:00 [43005960] -> Capabilities Mask:
> Jun 30 14:59:00 [43005960] -> __osm_pi_rcv_process_switch_port: [
> Jun 30 14:59:00 [43005960] -> __osm_pi_rcv_process_switch_port: ]
> Jun 30 14:59:00 [43005960] -> __osm_pi_rcv_get_pkey_slvl_vla_tables: [
> Jun 30 14:59:00 [43005960] -> osm_physp_has_pkey: [
> Jun 30 14:59:00 [43005960] -> osm_req_get: [
> Jun 30 14:59:00 [43005960] -> osm_mad_pool_get: [
> Jun 30 14:59:00 [43806960] -> __osm_mtl_send_callback: Completed Sending Request MADW: 0x5b0ac0.
> Jun 30 14:59:00 [43806960] -> osm_vendor_send: ]
> Jun 30 14:59:00 [43005960] -> osm_vendor_get: [
> Jun 30 14:59:00 [44808960] -> osm_vendor_put: [
> Jun 30 14:59:00 [44808960] -> osm_vendor_put: Retiring UMAD 0x591a50.
> Jun 30 14:59:00 [44808960] -> osm_vendor_put: ]
> Jun 30 14:59:00 [44808960] -> osm_mad_pool_put: ]
> Jun 30 14:59:00 [43806960] -> __osm_vl15_poller: 1 on wire, 11 outstanding, 10 unicasts sent, 88641 sent total.
> Jun 30 14:59:00 [44808960] -> __osm_sm_mad_ctrl_process_get_resp: Posting Dispatcher message OSM_MSG_MAD_NODE_INFO.
> Jun 30 14:59:00 [44808960] -> __osm_sm_mad_ctrl_process_get_resp: ]
> Jun 30 14:59:00 [44808960] -> __osm_sm_mad_ctrl_rcv_callback: ]
> Jun 30 14:59:00 [43005960] -> osm_vendor_get: Acquiring UMAD for p_madw = 0x567428, size = 256.
> Jun 30 14:59:00 [44808960] -> osm_mad_pool_get: [
> Jun 30 14:59:00 [44808960] -> osm_vendor_get: [
> Jun 30 14:59:00 [43005960] -> osm_vendor_get: Acquired UMAD 0x592620, size = 256.
> Jun 30 14:59:00 [43005960] -> osm_vendor_get: ]
> Jun 30 14:59:00 [44808960] -> osm_vendor_get: Acquiring UMAD for p_madw = 0x560268, size = 256.
> Jun 30 14:59:00 [44808960] -> osm_vendor_get: Acquired UMAD 0x591a50, size = 256.
> Jun 30 14:59:00 [44808960] -> osm_vendor_get: ]
> Jun 30 14:59:00 [43005960] -> osm_mad_pool_get: Acquired p_madw = 0x567410, p_mad = 0x592654, size = 256.
> Jun 30 14:59:00 [43005960] -> osm_mad_pool_get: ]
> Jun 30 14:59:00 [44808960] -> osm_mad_pool_get: Acquired p_madw = 0x560250, p_mad = 0x591a84, size = 256.
> Jun 30 14:59:00 [44808960] -> osm_mad_pool_get: ]
> Jun 30 14:59:00 [44808960] -> __osm_sm_mad_ctrl_rcv_callback: [
> Jun 30 14:59:00 [44808960] -> __osm_sm_mad_ctrl_rcv_callback: 88641 QP0 MADs received.
> Jun 30 14:59:00 [43005960] -> osm_req_get: Getting P_KeyTable (0x16), modifier = 0x80000, TID = 0x16c6d.
> Jun 30 14:59:00 [43005960] -> osm_vl15_post: [
> Jun 30 14:59:00 [43005960] -> osm_vl15_post: Servicing p_madw = 0x567410 (mad 0x592654 req 1)
> Jun 30 14:59:00 [44808960] -> SMP dump:
> 				base_ver................0x1
> 				mgmt_class..............0x81
> 				class_ver...............0x1
> 				method..................0x81 (SubnGetResp)
> 				status..................0x8000
> 				hop_ptr.................0x0
> 				hop_count...............0x1
> 				trans_id................0x16c6a
> 				attr_id.................0x16 (P_KeyTable)
> 				resv....................0x0
> 				attr_mod................0x50000
> 				m_key...................0x0000000000000000
> 				dr_slid.................0xFFFF
> 				dr_dlid.................0xFFFF
> 
> 				Initial path: [0][1]
> 				Return path:  [0][7]
> 				Reserved:     [0][0][0][0][0][0][0]
> 
> 				00 00 00 00 00 00 00 00   00 00 00 00 00 00 00 00
> 
> 				00 00 00 00 00 00 00 00   00 00 00 00 00 00 00 00
> 
> 				00 00 00 00 00 00 00 00   00 00 00 00 00 00 00 00
> 
> 				00 00 00 00 00 00 00 00   00 00 00 00 00 00 00 00
> 
> Jun 30 14:59:00 [44808960] -> __osm_sm_mad_ctrl_process_get_resp: [
> Jun 30 14:59:00 [44808960] -> __osm_sm_mad_ctrl_update_wire_stats: [
> Jun 30 14:59:00 [43005960] -> osm_vl15_post: 1 MADs on wire, 12 MADs outstanding.
> Jun 30 14:59:00 [43005960] -> osm_vl15_poll: [
> Jun 30 14:59:00 [43005960] -> osm_vl15_poll: Signalling poller thread.
> Jun 30 14:59:00 [44808960] -> __osm_sm_mad_ctrl_update_wire_stats: 0 SMPs on the wire, 12 outstanding.
> Jun 30 14:59:00 [44808960] -> osm_vl15_poll: [
> Jun 30 14:59:00 [44808960] -> osm_vl15_poll: Signalling poller thread.
> Jun 30 14:59:00 [44808960] -> osm_vl15_poll: ]
> Jun 30 14:59:00 [44808960] -> __osm_sm_mad_ctrl_update_wire_stats: ]
> Jun 30 14:59:00 [44808960] -> osm_mad_pool_put: [
> Jun 30 14:59:00 [44808960] -> osm_mad_pool_put: Releasing p_madw = 0x5b0ac0, p_mad = 0x5930e4.
> Jun 30 14:59:00 [44808960] -> osm_vendor_put: [
> Jun 30 14:59:00 [44808960] -> osm_vendor_put: Retiring UMAD 0x5930b0.
> Jun 30 14:59:00 [44808960] -> osm_vendor_put: ]
> Jun 30 14:59:00 [44808960] -> osm_mad_pool_put: ]
> Jun 30 14:59:00 [43806960] -> __osm_vl15_poller: Servicing p_madw = 0x567820 (mad 0x591bc4 req 1)
> Jun 30 14:59:00 [44808960] -> __osm_sm_mad_ctrl_process_get_resp: Posting Dispatcher message OSM_MSG_MAD_PKEY.
> Jun 30 14:59:00 [44808960] -> __osm_sm_mad_ctrl_process_get_resp: ]
> Jun 30 14:59:00 [44808960] -> __osm_sm_mad_ctrl_rcv_callback: ]
> Jun 30 14:59:00 [43806960] -> SMP dump:
> 				base_ver................0x1
> 				mgmt_class..............0x81
> 				class_ver...............0x1
> 				method..................0x1 (SubnGet)
> 				status..................0x0
> 				hop_ptr.................0x0
> 				hop_count...............0x1
> 				trans_id................0x16c6b
> 				attr_id.................0x16 (P_KeyTable)
> 				resv....................0x0
> 				attr_mod................0x60000
> 				m_key...................0x0000000000000000
> 				dr_slid.................0xFFFF
> 				dr_dlid.................0xFFFF
> 
> 				Initial path: [0][1]
> 				Return path:  [0][0]
> 				Reserved:     [0][0][0][0][0][0][0]
> 
> 				00 00 00 00 00 00 00 00   00 00 00 00 00 00 00 00
> 
> 				00 00 00 00 00 00 00 00   00 00 00 00 00 00 00 00
> 
> 				00 00 00 00 00 00 00 00   00 00 00 00 00 00 00 00
> 
> 				00 00 00 00 00 00 00 00   00 00 00 00 00 00 00 00
> 
> Jun 30 14:59:00 [43005960] -> osm_vl15_poll: ]
> Jun 30 14:59:00 [43806960] -> osm_vendor_send: [
> Jun 30 14:59:00 [43005960] -> osm_vl15_post: ]
> Jun 30 14:59:00 [43005960] -> osm_req_get: ]
> Jun 30 14:59:00 [43005960] -> osm_physp_has_pkey: ]
> Jun 30 14:59:00 [43005960] -> __osm_pi_rcv_get_pkey_slvl_vla_tables: ]
> Jun 30 14:59:00 [43005960] -> osm_pi_rcv_process: ]
> Jun 30 14:59:00 [44808960] -> osm_mad_pool_get: [
> Jun 30 14:59:00 [44808960] -> osm_vendor_get: [
> Jun 30 14:59:00 [44808960] -> osm_vendor_get: Acquiring UMAD for p_madw = 0x5b0ad8, size = 256.
> Jun 30 14:59:00 [44808960] -> osm_vendor_get: Acquired UMAD 0x5930b0, size = 256.
> Jun 30 14:59:00 [44808960] -> osm_vendor_get: ]
> Jun 30 14:59:00 [44808960] -> osm_mad_pool_get: Acquired p_madw = 0x5b0ac0, p_mad = 0x5930e4, size = 256.
> Jun 30 14:59:00 [44808960] -> osm_mad_pool_get: ]
> Jun 30 14:59:00 [44808960] -> __osm_sm_mad_ctrl_rcv_callback: [
> Jun 30 14:59:00 [44808960] -> __osm_sm_mad_ctrl_rcv_callback: 88642 QP0 MADs received.
> Jun 30 14:59:00 [43806960] -> osm_vendor_send: Send failed -5 (Success).
> Jun 30 14:59:00 [43005960] -> __osm_sm_mad_ctrl_disp_done_callback: [
> Jun 30 14:59:00 [43005960] -> __osm_sm_mad_ctrl_retire_trans_mad: [
> Jun 30 14:59:00 [43806960] -> __osm_sm_mad_ctrl_send_err_cb: [
> Jun 30 14:59:00 [44808960] -> SMP dump:
> 				base_ver................0x1
> 				mgmt_class..............0x81
> 				class_ver...............0x1
> 				method..................0x81 (SubnGetResp)
> 				status..................0x8000
> 				hop_ptr.................0x0
> 				hop_count...............0x1
> 				trans_id................0x16c6b
> 				attr_id.................0x16 (P_KeyTable)
> 				resv....................0x0
> 				attr_mod................0x60000
> 				m_key...................0x0000000000000000
> 				dr_slid.................0xFFFF
> 				dr_dlid.................0xFFFF
> 
> 				Initial path: [0][1]
> 				Return path:  [0][7]
> 				Reserved:     [0][0][0][0][0][0][0]
> 
> 				00 00 00 00 00 00 00 00   00 00 00 00 00 00 00 00
> 
> 				00 00 00 00 00 00 00 00   00 00 00 00 00 00 00 00
> 
> 				00 00 00 00 00 00 00 00   00 00 00 00 00 00 00 00
> 
> 				00 00 00 00 00 00 00 00   00 00 00 00 00 00 00 00
> 
> Jun 30 14:59:00 [44808960] -> __osm_sm_mad_ctrl_process_get_resp: [
> Jun 30 14:59:00 [44808960] -> __osm_sm_mad_ctrl_update_wire_stats: [
> Jun 30 14:59:00 [44808960] -> __osm_sm_mad_ctrl_update_wire_stats: 0 SMPs on the wire, 12 outstanding.
> Jun 30 14:59:00 [44808960] -> osm_vl15_poll: [
> Jun 30 14:59:00 [44808960] -> osm_vl15_poll: Signalling poller thread.
> Jun 30 14:59:00 [44808960] -> osm_vl15_poll: ]
> Jun 30 14:59:00 [44808960] -> __osm_sm_mad_ctrl_update_wire_stats: ]
> Jun 30 14:59:00 [44808960] -> osm_mad_pool_put: [
> Jun 30 14:59:00 [44808960] -> osm_mad_pool_put: Releasing p_madw = 0x567820, p_mad = 0x591bc4.
> Jun 30 14:59:00 [44808960] -> osm_vendor_put: [
> Jun 30 14:59:00 [44808960] -> osm_vendor_put: Retiring UMAD 0x591b90.
> Jun 30 14:59:00 [44808960] -> osm_vendor_put: ]
> Jun 30 14:59:00 [44808960] -> osm_mad_pool_put: ]
> Jun 30 14:59:00 [44808960] -> __osm_sm_mad_ctrl_process_get_resp: Posting Dispatcher message OSM_MSG_MAD_PKEY.
> Jun 30 14:59:00 [44808960] -> __osm_sm_mad_ctrl_process_get_resp: ]
> Jun 30 14:59:00 [44808960] -> __osm_sm_mad_ctrl_rcv_callback: ]
> Jun 30 14:59:00 [43005960] -> __osm_sm_mad_ctrl_retire_trans_mad: Retiring MAD with TID = 0x16c62.
> Jun 30 14:59:00 [43005960] -> osm_mad_pool_put: [
> Jun 30 14:59:00 [43005960] -> osm_mad_pool_put: Releasing p_madw = 0x5b0370, p_mad = 0x5b1244.
> Jun 30 14:59:00 [43005960] -> osm_vendor_put: [
> Jun 30 14:59:00 [43005960] -> osm_vendor_put: Retiring UMAD 0x5b1210.
> Jun 30 14:59:00 [43005960] -> osm_vendor_put: ]
> Jun 30 14:59:00 [43005960] -> osm_mad_pool_put: ]
> Jun 30 14:59:00 [43005960] -> __osm_sm_mad_ctrl_retire_trans_mad: 11 QP0 MADs outstanding.
> Jun 30 14:59:00 [43005960] -> __osm_sm_mad_ctrl_retire_trans_mad: ]
> Jun 30 14:59:00 [43005960] -> __osm_sm_mad_ctrl_disp_done_callback: ]
> Jun 30 14:59:00 [43005960] -> osm_pkey_rcv_process: [
> Jun 30 14:59:00 [43005960] -> osm_pkey_rcv_process: Got GetResp(PKey) block:0 port_num 0 with GUID = 0x617000000000d for parent node GUID = 0x617000000000d, TID = 0x16c63.
> Jun 30 14:59:00 [43005960] -> P_Key table dump:
> 			port_guid...........0x000617000000000d
> 			block_num...........0x0
> 			port_num............0x0
> 	P_Key Table:  0XFFFF | 0X0 | 0X0 | 0X0 | 0X0 | 0X0 | 0X0 | 0X0 | 0X0 | 0X0 | 0X0 | 0X0 | 0X0 | 0X0 | 0X0 | 0X0 | 0X0 | 0X0 | 0X0 | 0X0 | 0X0 | 0X0 | 0X0 | 0X0 | 0X0 | 0X0 | 0X0 | 0X0 | 0X0 | 0X0 | 0X0 | 0X0 |
> Jun 30 14:59:00 [43005960] -> osm_pkey_rcv_process: ]
> Jun 30 14:59:00 [43005960] -> __osm_sm_mad_ctrl_disp_done_callback: [
> Jun 30 14:59:00 [43005960] -> __osm_sm_mad_ctrl_retire_trans_mad: [
> Jun 30 14:59:00 [43005960] -> __osm_sm_mad_ctrl_retire_trans_mad: Retiring MAD with TID = 0x16c63.
> Jun 30 14:59:00 [43005960] -> osm_mad_pool_put: [
> Jun 30 14:59:00 [43005960] -> osm_mad_pool_put: Releasing p_madw = 0x55df60, p_mad = 0x5b0c74.
> Jun 30 14:59:00 [43005960] -> osm_vendor_put: [
> Jun 30 14:59:00 [43005960] -> osm_vendor_put: Retiring UMAD 0x5b0c40.
> Jun 30 14:59:00 [43005960] -> osm_vendor_put: ]
> Jun 30 14:59:00 [43005960] -> osm_mad_pool_put: ]
> Jun 30 14:59:00 [43005960] -> __osm_sm_mad_ctrl_retire_trans_mad: 10 QP0 MADs outstanding.
> Jun 30 14:59:00 [43005960] -> __osm_sm_mad_ctrl_retire_trans_mad: ]
> Jun 30 14:59:00 [43005960] -> __osm_sm_mad_ctrl_disp_done_callback: ]
> Jun 30 14:59:00 [43005960] -> osm_pkey_rcv_process: [
> Jun 30 14:59:00 [43005960] -> osm_pkey_rcv_process: Got GetResp(PKey) block:0 port_num 1 with GUID = 0x617000000000d for parent node GUID = 0x617000000000d, TID = 0x16c64.
> Jun 30 14:59:00 [43005960] -> P_Key table dump:
> 			port_guid...........0x000617000000000d
> 			block_num...........0x0
> 			port_num............0x1
> 	P_Key Table:  0X0 | 0X0 | 0X0 | 0X0 | 0X0 | 0X0 | 0X0 | 0X0 | 0X0 | 0X0 | 0X0 | 0X0 | 0X0 | 0X0 | 0X0 | 0X0 | 0X0 | 0X0 | 0X0 | 0X0 | 0X0 | 0X0 | 0X0 | 0X0 | 0X0 | 0X0 | 0X0 | 0X0 | 0X0 | 0X0 | 0X0 | 0X0 |
> Jun 30 14:59:00 [43005960] -> osm_pkey_rcv_process: ]
> Jun 30 14:59:00 [43005960] -> __osm_sm_mad_ctrl_disp_done_callback: [
> Jun 30 14:59:00 [43005960] -> __osm_sm_mad_ctrl_retire_trans_mad: [
> Jun 30 14:59:00 [43005960] -> __osm_sm_mad_ctrl_retire_trans_mad: Retiring MAD with TID = 0x16c64.
> Jun 30 14:59:00 [43005960] -> osm_mad_pool_put: [
> Jun 30 14:59:00 [43005960] -> osm_mad_pool_put: Releasing p_madw = 0x5b0920, p_mad = 0x5916c4.
> Jun 30 14:59:00 [43005960] -> osm_vendor_put: [
> Jun 30 14:59:00 [43005960] -> osm_vendor_put: Retiring UMAD 0x591690.
> Jun 30 14:59:00 [43005960] -> osm_vendor_put: ]
> Jun 30 14:59:00 [43005960] -> osm_mad_pool_put: ]
> Jun 30 14:59:00 [43005960] -> __osm_sm_mad_ctrl_retire_trans_mad: 9 QP0 MADs outstanding.
> Jun 30 14:59:00 [43005960] -> __osm_sm_mad_ctrl_retire_trans_mad: ]
> Jun 30 14:59:00 [43005960] -> __osm_sm_mad_ctrl_disp_done_callback: ]
> Jun 30 14:59:00 [43005960] -> osm_pkey_rcv_process: [
> Jun 30 14:59:00 [43005960] -> osm_pkey_rcv_process: Got GetResp(PKey) block:0 port_num 2 with GUID = 0x617000000000d for parent node GUID = 0x617000000000d, TID = 0x16c65.
> Jun 30 14:59:00 [43005960] -> P_Key table dump:
> 			port_guid...........0x000617000000000d
> 			block_num...........0x0
> 			port_num............0x2
> 	P_Key Table:  0X0 | 0X0 | 0X0 | 0X0 | 0X0 | 0X0 | 0X0 | 0X0 | 0X0 | 0X0 | 0X0 | 0X0 | 0X0 | 0X0 | 0X0 | 0X0 | 0X0 | 0X0 | 0X0 | 0X0 | 0X0 | 0X0 | 0X0 | 0X0 | 0X0 | 0X0 | 0X0 | 0X0 | 0X0 | 0X0 | 0X0 | 0X0 |
> Jun 30 14:59:00 [43005960] -> osm_pkey_rcv_process: ]
> Jun 30 14:59:00 [43005960] -> __osm_sm_mad_ctrl_disp_done_callback: [
> Jun 30 14:59:00 [43005960] -> __osm_sm_mad_ctrl_retire_trans_mad: [
> Jun 30 14:59:00 [43005960] -> __osm_sm_mad_ctrl_retire_trans_mad: Retiring MAD with TID = 0x16c65.
> Jun 30 14:59:00 [43005960] -> osm_mad_pool_put: [
> Jun 30 14:59:00 [43005960] -> osm_mad_pool_put: Releasing p_madw = 0x55add0, p_mad = 0x591d04.
> Jun 30 14:59:00 [43005960] -> osm_vendor_put: [
> Jun 30 14:59:00 [43005960] -> osm_vendor_put: Retiring UMAD 0x591cd0.
> Jun 30 14:59:00 [43005960] -> osm_vendor_put: ]
> Jun 30 14:59:00 [43005960] -> osm_mad_pool_put: ]
> Jun 30 14:59:00 [43005960] -> __osm_sm_mad_ctrl_retire_trans_mad: 8 QP0 MADs outstanding.
> Jun 30 14:59:00 [43005960] -> __osm_sm_mad_ctrl_retire_trans_mad: ]
> Jun 30 14:59:00 [43005960] -> __osm_sm_mad_ctrl_disp_done_callback: ]
> Jun 30 14:59:00 [43005960] -> osm_ni_rcv_process: [
> Jun 30 14:59:00 [43005960] -> NodeInfo dump:
> 				base_version............0x1
> 				class_version...........0x1
> 				node_type...............Channel Adapter
> 				num_ports...............0x2
> 				sys_guid................0x0002c9000100d050
> 				node_guid...............0x0002c901097624c0
> 				port_guid...............0x0002c901097624c1
> 				partition_cap...........0x40
> 				device_id...............0x5A44
> 				revision................0xA1
> 				port_num................0x1
> 				vendor_id...............0x2C9
> Jun 30 14:59:00 [43005960] -> __osm_ni_rcv_process_existing: [
> Jun 30 14:59:00 [43005960] -> __osm_ni_rcv_process_existing: Rediscovered Channel Adapter node 0x2c901097624c0
> 				TID = 0x16c66, discovered 0 times already.
> Jun 30 14:59:00 [43005960] -> __osm_ni_rcv_process_existing_ca: [
> Jun 30 14:59:00 [43005960] -> __osm_ni_rcv_process_ca_port: [
> Jun 30 14:59:00 [43005960] -> osm_req_get: [
> Jun 30 14:59:00 [43005960] -> osm_mad_pool_get: [
> Jun 30 14:59:00 [43005960] -> osm_vendor_get: [
> Jun 30 14:59:00 [43005960] -> osm_vendor_get: Acquiring UMAD for p_madw = 0x55ade8, size = 256.
> Jun 30 14:59:00 [43005960] -> osm_vendor_get: Acquired UMAD 0x5b1210, size = 256.
> Jun 30 14:59:00 [43005960] -> osm_vendor_get: ]
> Jun 30 14:59:00 [43005960] -> osm_mad_pool_get: Acquired p_madw = 0x55add0, p_mad = 0x5b1244, size = 256.
> Jun 30 14:59:00 [43005960] -> osm_mad_pool_get: ]
> Jun 30 14:59:00 [43005960] -> osm_req_get: Getting PortInfo (0x15), modifier = 0x1, TID = 0x16c6e.
> Jun 30 14:59:00 [43005960] -> osm_vl15_post: [
> Jun 30 14:59:00 [43005960] -> osm_vl15_post: Servicing p_madw = 0x55add0 (mad 0x5b1244 req 1)
> Jun 30 14:59:00 [43005960] -> osm_vl15_post: 0 MADs on wire, 9 MADs outstanding.
> Jun 30 14:59:00 [43005960] -> osm_vl15_poll: [
> Jun 30 14:59:00 [43005960] -> osm_vl15_poll: Signalling poller thread.
> Jun 30 14:59:00 [43005960] -> osm_vl15_poll: ]
> Jun 30 14:59:00 [43005960] -> osm_vl15_post: ]
> Jun 30 14:59:00 [43005960] -> osm_req_get: ]
> Jun 30 14:59:00 [43005960] -> __osm_ni_rcv_process_ca_port: ]
> Jun 30 14:59:00 [43005960] -> __osm_ni_rcv_process_existing_ca: ]
> Jun 30 14:59:00 [43005960] -> __osm_ni_rcv_set_links: [
> Jun 30 14:59:00 [43005960] -> __osm_ni_rcv_set_links: Link already exists.
> Jun 30 14:59:00 [43005960] -> __osm_ni_rcv_set_links: ]
> Jun 30 14:59:00 [43005960] -> __osm_ni_rcv_process_existing: ]
> Jun 30 14:59:00 [43005960] -> osm_ni_rcv_process: ]
> Jun 30 14:59:00 [43005960] -> __osm_sm_mad_ctrl_disp_done_callback: [
> Jun 30 14:59:00 [43005960] -> __osm_sm_mad_ctrl_retire_trans_mad: [
> Jun 30 14:59:00 [43005960] -> __osm_sm_mad_ctrl_retire_trans_mad: Retiring MAD with TID = 0x16c66.
> Jun 30 14:59:00 [43005960] -> osm_mad_pool_put: [
> Jun 30 14:59:00 [43005960] -> osm_mad_pool_put: Releasing p_madw = 0x5b09f0, p_mad = 0x592fa4.
> Jun 30 14:59:00 [43005960] -> osm_vendor_put: [
> Jun 30 14:59:00 [43005960] -> osm_vendor_put: Retiring UMAD 0x592f70.
> Jun 30 14:59:00 [43005960] -> osm_vendor_put: ]
> Jun 30 14:59:00 [43005960] -> osm_mad_pool_put: ]
> Jun 30 14:59:00 [43005960] -> __osm_sm_mad_ctrl_retire_trans_mad: 8 QP0 MADs outstanding.
> Jun 30 14:59:00 [43005960] -> __osm_sm_mad_ctrl_retire_trans_mad: ]
> Jun 30 14:59:00 [43005960] -> __osm_sm_mad_ctrl_disp_done_callback: ]
> Jun 30 14:59:00 [43005960] -> osm_pkey_rcv_process: [
> Jun 30 14:59:00 [43005960] -> osm_pkey_rcv_process: Got GetResp(PKey) block:0 port_num 3 with GUID = 0x617000000000d for parent node GUID = 0x617000000000d, TID = 0x16c67.
> Jun 30 14:59:00 [43005960] -> P_Key table dump:
> 			port_guid...........0x000617000000000d
> 			block_num...........0x0
> 			port_num............0x3
> 	P_Key Table:  0X0 | 0X0 | 0X0 | 0X0 | 0X0 | 0X0 | 0X0 | 0X0 | 0X0 | 0X0 | 0X0 | 0X0 | 0X0 | 0X0 | 0X0 | 0X0 | 0X0 | 0X0 | 0X0 | 0X0 | 0X0 | 0X0 | 0X0 | 0X0 | 0X0 | 0X0 | 0X0 | 0X0 | 0X0 | 0X0 | 0X0 | 0X0 |
> Jun 30 14:59:00 [43005960] -> osm_pkey_rcv_process: ]
> Jun 30 14:59:00 [43005960] -> __osm_sm_mad_ctrl_disp_done_callback: [
> Jun 30 14:59:00 [43005960] -> __osm_sm_mad_ctrl_retire_trans_mad: [
> Jun 30 14:59:00 [43005960] -> __osm_sm_mad_ctrl_retire_trans_mad: Retiring MAD with TID = 0x16c67.
> Jun 30 14:59:00 [43005960] -> osm_mad_pool_put: [
> Jun 30 14:59:00 [43005960] -> osm_mad_pool_put: Releasing p_madw = 0x561f90, p_mad = 0x592e64.
> Jun 30 14:59:00 [43005960] -> osm_vendor_put: [
> Jun 30 14:59:00 [43005960] -> osm_vendor_put: Retiring UMAD 0x592e30.
> Jun 30 14:59:00 [43005960] -> osm_vendor_put: ]
> Jun 30 14:59:00 [43005960] -> osm_mad_pool_put: ]
> Jun 30 14:59:00 [43005960] -> __osm_sm_mad_ctrl_retire_trans_mad: 7 QP0 MADs outstanding.
> Jun 30 14:59:00 [43005960] -> __osm_sm_mad_ctrl_retire_trans_mad: ]
> Jun 30 14:59:00 [43005960] -> __osm_sm_mad_ctrl_disp_done_callback: ]
> Jun 30 14:59:00 [43005960] -> osm_pkey_rcv_process: [
> Jun 30 14:59:00 [43005960] -> osm_pkey_rcv_process: Got GetResp(PKey) block:0 port_num 4 with GUID = 0x617000000000d for parent node GUID = 0x617000000000d, TID = 0x16c68.
> Jun 30 14:59:00 [43005960] -> P_Key table dump:
> 			port_guid...........0x000617000000000d
> 			block_num...........0x0
> 			port_num............0x4
> 	P_Key Table:  0X0 | 0X0 | 0X0 | 0X0 | 0X0 | 0X0 | 0X0 | 0X0 | 0X0 | 0X0 | 0X0 | 0X0 | 0X0 | 0X0 | 0X0 | 0X0 | 0X0 | 0X0 | 0X0 | 0X0 | 0X0 | 0X0 | 0X0 | 0X0 | 0X0 | 0X0 | 0X0 | 0X0 | 0X0 | 0X0 | 0X0 | 0X0 |
> Jun 30 14:59:00 [43005960] -> osm_pkey_rcv_process: ]
> Jun 30 14:59:00 [43005960] -> __osm_sm_mad_ctrl_disp_done_callback: [
> Jun 30 14:59:00 [43005960] -> __osm_sm_mad_ctrl_retire_trans_mad: [
> Jun 30 14:59:00 [43005960] -> __osm_sm_mad_ctrl_retire_trans_mad: Retiring MAD with TID = 0x16c68.
> Jun 30 14:59:00 [43005960] -> osm_mad_pool_put: [
> Jun 30 14:59:00 [43005960] -> osm_mad_pool_put: Releasing p_madw = 0x5b0850, p_mad = 0x591804.
> Jun 30 14:59:00 [43005960] -> osm_vendor_put: [
> Jun 30 14:59:00 [43005960] -> osm_vendor_put: Retiring UMAD 0x5917d0.
> Jun 30 14:59:00 [43005960] -> osm_vendor_put: ]
> Jun 30 14:59:00 [43005960] -> osm_mad_pool_put: ]
> Jun 30 14:59:00 [43005960] -> __osm_sm_mad_ctrl_retire_trans_mad: 6 QP0 MADs outstanding.
> Jun 30 14:59:00 [43005960] -> __osm_sm_mad_ctrl_retire_trans_mad: ]
> Jun 30 14:59:00 [43005960] -> __osm_sm_mad_ctrl_disp_done_callback: ]
> Jun 30 14:59:00 [43005960] -> osm_ni_rcv_process: [
> Jun 30 14:59:00 [43005960] -> NodeInfo dump:
> 				base_version............0x1
> 				class_version...........0x1
> 				node_type...............Channel Adapter
> 				num_ports...............0x2
> 				sys_guid................0x0002c90109765633
> 				node_guid...............0x0002c90109765630
> 				port_guid...............0x0002c90109765631
> 				partition_cap...........0x20
> 				device_id...............0x5A44
> 				revision................0xA1
> 				port_num................0x1
> 				vendor_id...............0x2C9
> Jun 30 14:59:00 [43005960] -> __osm_ni_rcv_process_existing: [
> Jun 30 14:59:00 [43005960] -> __osm_ni_rcv_process_existing: Rediscovered Channel Adapter node 0x2c90109765630
> 				TID = 0x16c69, discovered 0 times already.
> Jun 30 14:59:00 [43005960] -> __osm_ni_rcv_process_existing_ca: [
> Jun 30 14:59:00 [43005960] -> __osm_ni_rcv_process_ca_port: [
> Jun 30 14:59:00 [43005960] -> osm_req_get: [
> Jun 30 14:59:00 [43005960] -> osm_mad_pool_get: [
> Jun 30 14:59:00 [43005960] -> osm_vendor_get: [
> Jun 30 14:59:00 [43005960] -> osm_vendor_get: Acquiring UMAD for p_madw = 0x5b0868, size = 256.
> Jun 30 14:59:00 [43005960] -> osm_vendor_get: Acquired UMAD 0x5b0c40, size = 256.
> Jun 30 14:59:00 [43005960] -> osm_vendor_get: ]
> Jun 30 14:59:00 [43005960] -> osm_mad_pool_get: Acquired p_madw = 0x5b0850, p_mad = 0x5b0c74, size = 256.
> Jun 30 14:59:00 [43005960] -> osm_mad_pool_get: ]
> Jun 30 14:59:00 [43005960] -> osm_req_get: Getting PortInfo (0x15), modifier = 0x1, TID = 0x16c6f.
> Jun 30 14:59:00 [43005960] -> osm_vl15_post: [
> Jun 30 14:59:00 [43005960] -> osm_vl15_post: Servicing p_madw = 0x5b0850 (mad 0x5b0c74 req 1)
> Jun 30 14:59:00 [43005960] -> osm_vl15_post: 0 MADs on wire, 7 MADs outstanding.
> Jun 30 14:59:00 [43005960] -> osm_vl15_poll: [
> Jun 30 14:59:00 [43005960] -> osm_vl15_poll: Signalling poller thread.
> Jun 30 14:59:00 [43005960] -> osm_vl15_poll: ]
> Jun 30 14:59:00 [43005960] -> osm_vl15_post: ]
> Jun 30 14:59:00 [43005960] -> osm_req_get: ]
> Jun 30 14:59:00 [43005960] -> __osm_ni_rcv_process_ca_port: ]
> Jun 30 14:59:00 [43005960] -> __osm_ni_rcv_process_existing_ca: ]
> Jun 30 14:59:00 [43005960] -> __osm_ni_rcv_set_links: [
> Jun 30 14:59:00 [43005960] -> __osm_ni_rcv_set_links: Link already exists.
> Jun 30 14:59:00 [43005960] -> __osm_ni_rcv_set_links: ]
> Jun 30 14:59:00 [43005960] -> __osm_ni_rcv_process_existing: ]
> Jun 30 14:59:00 [43005960] -> osm_ni_rcv_process: ]
> Jun 30 14:59:00 [43005960] -> __osm_sm_mad_ctrl_disp_done_callback: [
> Jun 30 14:59:00 [43005960] -> __osm_sm_mad_ctrl_retire_trans_mad: [
> Jun 30 14:59:00 [43005960] -> __osm_sm_mad_ctrl_retire_trans_mad: Retiring MAD with TID = 0x16c69.
> Jun 30 14:59:00 [43005960] -> osm_mad_pool_put: [
> Jun 30 14:59:00 [43005960] -> osm_mad_pool_put: Releasing p_madw = 0x5b02a0, p_mad = 0x592144.
> Jun 30 14:59:00 [43005960] -> osm_vendor_put: [
> Jun 30 14:59:00 [43005960] -> osm_vendor_put: Retiring UMAD 0x592110.
> Jun 30 14:59:00 [43005960] -> osm_vendor_put: ]
> Jun 30 14:59:00 [43005960] -> osm_mad_pool_put: ]
> Jun 30 14:59:00 [43005960] -> __osm_sm_mad_ctrl_retire_trans_mad: 6 QP0 MADs outstanding.
> Jun 30 14:59:00 [43005960] -> __osm_sm_mad_ctrl_retire_trans_mad: ]
> Jun 30 14:59:00 [43005960] -> __osm_sm_mad_ctrl_disp_done_callback: ]
> Jun 30 14:59:00 [43005960] -> osm_pkey_rcv_process: [
> Jun 30 14:59:00 [43005960] -> osm_pkey_rcv_process: Got GetResp(PKey) block:0 port_num 5 with GUID = 0x617000000000d for parent node GUID = 0x617000000000d, TID = 0x16c6a.
> Jun 30 14:59:00 [43005960] -> P_Key table dump:
> 			port_guid...........0x000617000000000d
> 			block_num...........0x0
> 			port_num............0x5
> 	P_Key Table:  0X0 | 0X0 | 0X0 | 0X0 | 0X0 | 0X0 | 0X0 | 0X0 | 0X0 | 0X0 | 0X0 | 0X0 | 0X0 | 0X0 | 0X0 | 0X0 | 0X0 | 0X0 | 0X0 | 0X0 | 0X0 | 0X0 | 0X0 | 0X0 | 0X0 | 0X0 | 0X0 | 0X0 | 0X0 | 0X0 | 0X0 | 0X0 |
> Jun 30 14:59:00 [43005960] -> osm_pkey_rcv_process: ]
> Jun 30 14:59:00 [43005960] -> __osm_sm_mad_ctrl_disp_done_callback: [
> Jun 30 14:59:00 [43005960] -> __osm_sm_mad_ctrl_retire_trans_mad: [
> Jun 30 14:59:00 [43005960] -> __osm_sm_mad_ctrl_retire_trans_mad: Retiring MAD with TID = 0x16c6a.
> Jun 30 14:59:00 [43005960] -> osm_mad_pool_put: [
> Jun 30 14:59:00 [43005960] -> osm_mad_pool_put: Releasing p_madw = 0x560250, p_mad = 0x592514.
> Jun 30 14:59:00 [43005960] -> osm_vendor_put: [
> Jun 30 14:59:00 [43005960] -> osm_vendor_put: Retiring UMAD 0x5924e0.
> Jun 30 14:59:00 [43005960] -> osm_vendor_put: ]
> Jun 30 14:59:00 [43005960] -> osm_mad_pool_put: ]
> Jun 30 14:59:00 [43005960] -> __osm_sm_mad_ctrl_retire_trans_mad: 5 QP0 MADs outstanding.
> Jun 30 14:59:00 [43005960] -> __osm_sm_mad_ctrl_retire_trans_mad: ]
> Jun 30 14:59:00 [43005960] -> __osm_sm_mad_ctrl_disp_done_callback: ]
> Jun 30 14:59:00 [43005960] -> osm_pkey_rcv_process: [
> Jun 30 14:59:00 [43005960] -> osm_pkey_rcv_process: Got GetResp(PKey) block:0 port_num 6 with GUID = 0x617000000000d for parent node GUID = 0x617000000000d, TID = 0x16c6b.
> Jun 30 14:59:00 [43005960] -> P_Key table dump:
> 			port_guid...........0x000617000000000d
> 			block_num...........0x0
> 			port_num............0x6
> 	P_Key Table:  0X0 | 0X0 | 0X0 | 0X0 | 0X0 | 0X0 | 0X0 | 0X0 | 0X0 | 0X0 | 0X0 | 0X0 | 0X0 | 0X0 | 0X0 | 0X0 | 0X0 | 0X0 | 0X0 | 0X0 | 0X0 | 0X0 | 0X0 | 0X0 | 0X0 | 0X0 | 0X0 | 0X0 | 0X0 | 0X0 | 0X0 | 0X0 |
> Jun 30 14:59:00 [43005960] -> osm_pkey_rcv_process: ]
> Jun 30 14:59:00 [43005960] -> __osm_sm_mad_ctrl_disp_done_callback: [
> Jun 30 14:59:00 [43005960] -> __osm_sm_mad_ctrl_retire_trans_mad: [
> Jun 30 14:59:00 [43005960] -> __osm_sm_mad_ctrl_retire_trans_mad: Retiring MAD with TID = 0x16c6b.
> Jun 30 14:59:00 [43005960] -> osm_mad_pool_put: [
> Jun 30 14:59:00 [43005960] -> osm_mad_pool_put: Releasing p_madw = 0x5b0ac0, p_mad = 0x591a84.
> Jun 30 14:59:00 [43005960] -> osm_vendor_put: [
> Jun 30 14:59:00 [43005960] -> osm_vendor_put: Retiring UMAD 0x591a50.
> Jun 30 14:59:00 [43005960] -> osm_vendor_put: ]
> Jun 30 14:59:00 [43005960] -> osm_mad_pool_put: ]
> Jun 30 14:59:00 [43005960] -> __osm_sm_mad_ctrl_retire_trans_mad: 4 QP0 MADs outstanding.
> Jun 30 14:59:00 [43005960] -> __osm_sm_mad_ctrl_retire_trans_mad: ]
> Jun 30 14:59:00 [43005960] -> __osm_sm_mad_ctrl_disp_done_callback: ]
> Jun 30 14:59:00 [43806960] -> __osm_sm_mad_ctrl_send_err_cb: ERR 3113: MAD completed in error (IB_SUCCESS).
> 
> ______________________________________________________________________
> 
> _______________________________________________
> openib-general mailing list
> openib-general at openib.org
> http://openib.org/mailman/listinfo/openib-general
> 
> To unsubscribe, please visit http://openib.org/mailman/listinfo/openib-general




More information about the general mailing list