[openib-general] [oops] recent opensm crash
Hal Rosenstock
halr at voltaire.com
Fri Jul 1 10:24:37 PDT 2005
On Thu, 2005-06-30 at 18:10, Tom Duffy wrote:
> #0 stack_dump () at src/stack.c:72
> 72 if (!__builtin_frame_address(2))
> (gdb) bt
> #0 stack_dump () at src/stack.c:72
> #1 0x00002aaaaacbd1a6 in handler (x=11) at src/stack.c:151
> #2 <signal handler called>
> #3 __osm_sm_mad_ctrl_send_err_cb (bind_context=0x550dd8, p_madw=0x567820)
> at osm_sm_mad_ctrl.c:832
> #4 0x00002aaaaaaaeeed in osm_vendor_send (h_bind=0x586920, p_madw=0x567820,
> resp_expected=1) at osm_vendor_ibumad.c:889
I found one problem associated with this and just checked in a patch.
I'm not sure whether there is another one behind this or not. Any
reliable way to recreate this ?
-- Hal
> #5 0x000000000042ef72 in __osm_vl15_poller (p_ptr=0x552620) at osm_madw.h:933
> #6 0x00002aaaaadc911e in __cl_thread_wrapper (arg=0x0) at cl_thread.c:61
> #7 0x00000036d28060aa in start_thread () from /lib64/tls/libpthread.so.0
> #8 0x00000036d19c53d3 in clone () from /lib64/tls/libc.so.6
> #9 0x0000000000000000 in ?? ()
>
> I was bringing up and down an node when this happened.
>
> Attached are the last 500 lines from osm.log.
>
> -tduffy
>
> ______________________________________________________________________
>
> base_ver................0x1
> mgmt_class..............0x81
> class_ver...............0x1
> method..................0x1 (SubnGet)
> status..................0x0
> hop_ptr.................0x0
> hop_count...............0x1
> trans_id................0x16c6a
> attr_id.................0x16 (P_KeyTable)
> resv....................0x0
> attr_mod................0x50000
> m_key...................0x0000000000000000
> dr_slid.................0xFFFF
> dr_dlid.................0xFFFF
>
> Initial path: [0][1]
> Return path: [0][0]
> Reserved: [0][0][0][0][0][0][0]
>
> 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
>
> 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
>
> 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
>
> 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
>
> Jun 30 14:59:00 [43806960] -> osm_vendor_send: [
> Jun 30 14:59:00 [43005960] -> PortInfo dump:
> port number.............0x8
> node_guid...............0x000617000000000d
> port_guid...............0x000617000000000d
> m_key...................0x0000000000000000
> subnet_prefix...........0x0000000000000000
> base_lid................0x0
> master_sm_base_lid......0x0
> capability_mask.........0x0
> diag_code...............0x0
> m_key_lease_period......0x0
> local_port_num..........0x7
> link_width_enabled......0x3
> link_width_supported....0x3
> link_width_active.......0x0
> link_speed_supported....0x1
> port_state..............DOWN
> state_info2.............0x22
> m_key_protect_bits......0x0
> lmc.....................0x0
> link_speed..............0x11
> mtu_smsl................0x10
> vl_cap..................0x40
> vl_high_limit...........0x0
> vl_arb_high_cap.........0x20
> vl_arb_low_cap..........0x20
> mtu_cap.................0x5
> vl_stall_life...........0x8
> vl_enforce..............0x10
> m_key_violations........0x0
> p_key_violations........0x0
> q_key_violations........0x0
> guid_cap................0x0
> subnet_timeout..........0x0
> resp_time_value.........0x0
> error_threshold.........0xFF
> Jun 30 14:59:00 [43005960] -> Capabilities Mask:
> Jun 30 14:59:00 [43005960] -> __osm_pi_rcv_process_switch_port: [
> Jun 30 14:59:00 [43005960] -> __osm_pi_rcv_process_switch_port: ]
> Jun 30 14:59:00 [43005960] -> __osm_pi_rcv_get_pkey_slvl_vla_tables: [
> Jun 30 14:59:00 [43005960] -> osm_physp_has_pkey: [
> Jun 30 14:59:00 [43005960] -> osm_req_get: [
> Jun 30 14:59:00 [43005960] -> osm_mad_pool_get: [
> Jun 30 14:59:00 [43806960] -> __osm_mtl_send_callback: Completed Sending Request MADW: 0x5b0ac0.
> Jun 30 14:59:00 [43806960] -> osm_vendor_send: ]
> Jun 30 14:59:00 [43005960] -> osm_vendor_get: [
> Jun 30 14:59:00 [44808960] -> osm_vendor_put: [
> Jun 30 14:59:00 [44808960] -> osm_vendor_put: Retiring UMAD 0x591a50.
> Jun 30 14:59:00 [44808960] -> osm_vendor_put: ]
> Jun 30 14:59:00 [44808960] -> osm_mad_pool_put: ]
> Jun 30 14:59:00 [43806960] -> __osm_vl15_poller: 1 on wire, 11 outstanding, 10 unicasts sent, 88641 sent total.
> Jun 30 14:59:00 [44808960] -> __osm_sm_mad_ctrl_process_get_resp: Posting Dispatcher message OSM_MSG_MAD_NODE_INFO.
> Jun 30 14:59:00 [44808960] -> __osm_sm_mad_ctrl_process_get_resp: ]
> Jun 30 14:59:00 [44808960] -> __osm_sm_mad_ctrl_rcv_callback: ]
> Jun 30 14:59:00 [43005960] -> osm_vendor_get: Acquiring UMAD for p_madw = 0x567428, size = 256.
> Jun 30 14:59:00 [44808960] -> osm_mad_pool_get: [
> Jun 30 14:59:00 [44808960] -> osm_vendor_get: [
> Jun 30 14:59:00 [43005960] -> osm_vendor_get: Acquired UMAD 0x592620, size = 256.
> Jun 30 14:59:00 [43005960] -> osm_vendor_get: ]
> Jun 30 14:59:00 [44808960] -> osm_vendor_get: Acquiring UMAD for p_madw = 0x560268, size = 256.
> Jun 30 14:59:00 [44808960] -> osm_vendor_get: Acquired UMAD 0x591a50, size = 256.
> Jun 30 14:59:00 [44808960] -> osm_vendor_get: ]
> Jun 30 14:59:00 [43005960] -> osm_mad_pool_get: Acquired p_madw = 0x567410, p_mad = 0x592654, size = 256.
> Jun 30 14:59:00 [43005960] -> osm_mad_pool_get: ]
> Jun 30 14:59:00 [44808960] -> osm_mad_pool_get: Acquired p_madw = 0x560250, p_mad = 0x591a84, size = 256.
> Jun 30 14:59:00 [44808960] -> osm_mad_pool_get: ]
> Jun 30 14:59:00 [44808960] -> __osm_sm_mad_ctrl_rcv_callback: [
> Jun 30 14:59:00 [44808960] -> __osm_sm_mad_ctrl_rcv_callback: 88641 QP0 MADs received.
> Jun 30 14:59:00 [43005960] -> osm_req_get: Getting P_KeyTable (0x16), modifier = 0x80000, TID = 0x16c6d.
> Jun 30 14:59:00 [43005960] -> osm_vl15_post: [
> Jun 30 14:59:00 [43005960] -> osm_vl15_post: Servicing p_madw = 0x567410 (mad 0x592654 req 1)
> Jun 30 14:59:00 [44808960] -> SMP dump:
> base_ver................0x1
> mgmt_class..............0x81
> class_ver...............0x1
> method..................0x81 (SubnGetResp)
> status..................0x8000
> hop_ptr.................0x0
> hop_count...............0x1
> trans_id................0x16c6a
> attr_id.................0x16 (P_KeyTable)
> resv....................0x0
> attr_mod................0x50000
> m_key...................0x0000000000000000
> dr_slid.................0xFFFF
> dr_dlid.................0xFFFF
>
> Initial path: [0][1]
> Return path: [0][7]
> Reserved: [0][0][0][0][0][0][0]
>
> 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
>
> 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
>
> 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
>
> 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
>
> Jun 30 14:59:00 [44808960] -> __osm_sm_mad_ctrl_process_get_resp: [
> Jun 30 14:59:00 [44808960] -> __osm_sm_mad_ctrl_update_wire_stats: [
> Jun 30 14:59:00 [43005960] -> osm_vl15_post: 1 MADs on wire, 12 MADs outstanding.
> Jun 30 14:59:00 [43005960] -> osm_vl15_poll: [
> Jun 30 14:59:00 [43005960] -> osm_vl15_poll: Signalling poller thread.
> Jun 30 14:59:00 [44808960] -> __osm_sm_mad_ctrl_update_wire_stats: 0 SMPs on the wire, 12 outstanding.
> Jun 30 14:59:00 [44808960] -> osm_vl15_poll: [
> Jun 30 14:59:00 [44808960] -> osm_vl15_poll: Signalling poller thread.
> Jun 30 14:59:00 [44808960] -> osm_vl15_poll: ]
> Jun 30 14:59:00 [44808960] -> __osm_sm_mad_ctrl_update_wire_stats: ]
> Jun 30 14:59:00 [44808960] -> osm_mad_pool_put: [
> Jun 30 14:59:00 [44808960] -> osm_mad_pool_put: Releasing p_madw = 0x5b0ac0, p_mad = 0x5930e4.
> Jun 30 14:59:00 [44808960] -> osm_vendor_put: [
> Jun 30 14:59:00 [44808960] -> osm_vendor_put: Retiring UMAD 0x5930b0.
> Jun 30 14:59:00 [44808960] -> osm_vendor_put: ]
> Jun 30 14:59:00 [44808960] -> osm_mad_pool_put: ]
> Jun 30 14:59:00 [43806960] -> __osm_vl15_poller: Servicing p_madw = 0x567820 (mad 0x591bc4 req 1)
> Jun 30 14:59:00 [44808960] -> __osm_sm_mad_ctrl_process_get_resp: Posting Dispatcher message OSM_MSG_MAD_PKEY.
> Jun 30 14:59:00 [44808960] -> __osm_sm_mad_ctrl_process_get_resp: ]
> Jun 30 14:59:00 [44808960] -> __osm_sm_mad_ctrl_rcv_callback: ]
> Jun 30 14:59:00 [43806960] -> SMP dump:
> base_ver................0x1
> mgmt_class..............0x81
> class_ver...............0x1
> method..................0x1 (SubnGet)
> status..................0x0
> hop_ptr.................0x0
> hop_count...............0x1
> trans_id................0x16c6b
> attr_id.................0x16 (P_KeyTable)
> resv....................0x0
> attr_mod................0x60000
> m_key...................0x0000000000000000
> dr_slid.................0xFFFF
> dr_dlid.................0xFFFF
>
> Initial path: [0][1]
> Return path: [0][0]
> Reserved: [0][0][0][0][0][0][0]
>
> 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
>
> 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
>
> 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
>
> 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
>
> Jun 30 14:59:00 [43005960] -> osm_vl15_poll: ]
> Jun 30 14:59:00 [43806960] -> osm_vendor_send: [
> Jun 30 14:59:00 [43005960] -> osm_vl15_post: ]
> Jun 30 14:59:00 [43005960] -> osm_req_get: ]
> Jun 30 14:59:00 [43005960] -> osm_physp_has_pkey: ]
> Jun 30 14:59:00 [43005960] -> __osm_pi_rcv_get_pkey_slvl_vla_tables: ]
> Jun 30 14:59:00 [43005960] -> osm_pi_rcv_process: ]
> Jun 30 14:59:00 [44808960] -> osm_mad_pool_get: [
> Jun 30 14:59:00 [44808960] -> osm_vendor_get: [
> Jun 30 14:59:00 [44808960] -> osm_vendor_get: Acquiring UMAD for p_madw = 0x5b0ad8, size = 256.
> Jun 30 14:59:00 [44808960] -> osm_vendor_get: Acquired UMAD 0x5930b0, size = 256.
> Jun 30 14:59:00 [44808960] -> osm_vendor_get: ]
> Jun 30 14:59:00 [44808960] -> osm_mad_pool_get: Acquired p_madw = 0x5b0ac0, p_mad = 0x5930e4, size = 256.
> Jun 30 14:59:00 [44808960] -> osm_mad_pool_get: ]
> Jun 30 14:59:00 [44808960] -> __osm_sm_mad_ctrl_rcv_callback: [
> Jun 30 14:59:00 [44808960] -> __osm_sm_mad_ctrl_rcv_callback: 88642 QP0 MADs received.
> Jun 30 14:59:00 [43806960] -> osm_vendor_send: Send failed -5 (Success).
> Jun 30 14:59:00 [43005960] -> __osm_sm_mad_ctrl_disp_done_callback: [
> Jun 30 14:59:00 [43005960] -> __osm_sm_mad_ctrl_retire_trans_mad: [
> Jun 30 14:59:00 [43806960] -> __osm_sm_mad_ctrl_send_err_cb: [
> Jun 30 14:59:00 [44808960] -> SMP dump:
> base_ver................0x1
> mgmt_class..............0x81
> class_ver...............0x1
> method..................0x81 (SubnGetResp)
> status..................0x8000
> hop_ptr.................0x0
> hop_count...............0x1
> trans_id................0x16c6b
> attr_id.................0x16 (P_KeyTable)
> resv....................0x0
> attr_mod................0x60000
> m_key...................0x0000000000000000
> dr_slid.................0xFFFF
> dr_dlid.................0xFFFF
>
> Initial path: [0][1]
> Return path: [0][7]
> Reserved: [0][0][0][0][0][0][0]
>
> 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
>
> 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
>
> 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
>
> 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
>
> Jun 30 14:59:00 [44808960] -> __osm_sm_mad_ctrl_process_get_resp: [
> Jun 30 14:59:00 [44808960] -> __osm_sm_mad_ctrl_update_wire_stats: [
> Jun 30 14:59:00 [44808960] -> __osm_sm_mad_ctrl_update_wire_stats: 0 SMPs on the wire, 12 outstanding.
> Jun 30 14:59:00 [44808960] -> osm_vl15_poll: [
> Jun 30 14:59:00 [44808960] -> osm_vl15_poll: Signalling poller thread.
> Jun 30 14:59:00 [44808960] -> osm_vl15_poll: ]
> Jun 30 14:59:00 [44808960] -> __osm_sm_mad_ctrl_update_wire_stats: ]
> Jun 30 14:59:00 [44808960] -> osm_mad_pool_put: [
> Jun 30 14:59:00 [44808960] -> osm_mad_pool_put: Releasing p_madw = 0x567820, p_mad = 0x591bc4.
> Jun 30 14:59:00 [44808960] -> osm_vendor_put: [
> Jun 30 14:59:00 [44808960] -> osm_vendor_put: Retiring UMAD 0x591b90.
> Jun 30 14:59:00 [44808960] -> osm_vendor_put: ]
> Jun 30 14:59:00 [44808960] -> osm_mad_pool_put: ]
> Jun 30 14:59:00 [44808960] -> __osm_sm_mad_ctrl_process_get_resp: Posting Dispatcher message OSM_MSG_MAD_PKEY.
> Jun 30 14:59:00 [44808960] -> __osm_sm_mad_ctrl_process_get_resp: ]
> Jun 30 14:59:00 [44808960] -> __osm_sm_mad_ctrl_rcv_callback: ]
> Jun 30 14:59:00 [43005960] -> __osm_sm_mad_ctrl_retire_trans_mad: Retiring MAD with TID = 0x16c62.
> Jun 30 14:59:00 [43005960] -> osm_mad_pool_put: [
> Jun 30 14:59:00 [43005960] -> osm_mad_pool_put: Releasing p_madw = 0x5b0370, p_mad = 0x5b1244.
> Jun 30 14:59:00 [43005960] -> osm_vendor_put: [
> Jun 30 14:59:00 [43005960] -> osm_vendor_put: Retiring UMAD 0x5b1210.
> Jun 30 14:59:00 [43005960] -> osm_vendor_put: ]
> Jun 30 14:59:00 [43005960] -> osm_mad_pool_put: ]
> Jun 30 14:59:00 [43005960] -> __osm_sm_mad_ctrl_retire_trans_mad: 11 QP0 MADs outstanding.
> Jun 30 14:59:00 [43005960] -> __osm_sm_mad_ctrl_retire_trans_mad: ]
> Jun 30 14:59:00 [43005960] -> __osm_sm_mad_ctrl_disp_done_callback: ]
> Jun 30 14:59:00 [43005960] -> osm_pkey_rcv_process: [
> Jun 30 14:59:00 [43005960] -> osm_pkey_rcv_process: Got GetResp(PKey) block:0 port_num 0 with GUID = 0x617000000000d for parent node GUID = 0x617000000000d, TID = 0x16c63.
> Jun 30 14:59:00 [43005960] -> P_Key table dump:
> port_guid...........0x000617000000000d
> block_num...........0x0
> port_num............0x0
> P_Key Table: 0XFFFF | 0X0 | 0X0 | 0X0 | 0X0 | 0X0 | 0X0 | 0X0 | 0X0 | 0X0 | 0X0 | 0X0 | 0X0 | 0X0 | 0X0 | 0X0 | 0X0 | 0X0 | 0X0 | 0X0 | 0X0 | 0X0 | 0X0 | 0X0 | 0X0 | 0X0 | 0X0 | 0X0 | 0X0 | 0X0 | 0X0 | 0X0 |
> Jun 30 14:59:00 [43005960] -> osm_pkey_rcv_process: ]
> Jun 30 14:59:00 [43005960] -> __osm_sm_mad_ctrl_disp_done_callback: [
> Jun 30 14:59:00 [43005960] -> __osm_sm_mad_ctrl_retire_trans_mad: [
> Jun 30 14:59:00 [43005960] -> __osm_sm_mad_ctrl_retire_trans_mad: Retiring MAD with TID = 0x16c63.
> Jun 30 14:59:00 [43005960] -> osm_mad_pool_put: [
> Jun 30 14:59:00 [43005960] -> osm_mad_pool_put: Releasing p_madw = 0x55df60, p_mad = 0x5b0c74.
> Jun 30 14:59:00 [43005960] -> osm_vendor_put: [
> Jun 30 14:59:00 [43005960] -> osm_vendor_put: Retiring UMAD 0x5b0c40.
> Jun 30 14:59:00 [43005960] -> osm_vendor_put: ]
> Jun 30 14:59:00 [43005960] -> osm_mad_pool_put: ]
> Jun 30 14:59:00 [43005960] -> __osm_sm_mad_ctrl_retire_trans_mad: 10 QP0 MADs outstanding.
> Jun 30 14:59:00 [43005960] -> __osm_sm_mad_ctrl_retire_trans_mad: ]
> Jun 30 14:59:00 [43005960] -> __osm_sm_mad_ctrl_disp_done_callback: ]
> Jun 30 14:59:00 [43005960] -> osm_pkey_rcv_process: [
> Jun 30 14:59:00 [43005960] -> osm_pkey_rcv_process: Got GetResp(PKey) block:0 port_num 1 with GUID = 0x617000000000d for parent node GUID = 0x617000000000d, TID = 0x16c64.
> Jun 30 14:59:00 [43005960] -> P_Key table dump:
> port_guid...........0x000617000000000d
> block_num...........0x0
> port_num............0x1
> P_Key Table: 0X0 | 0X0 | 0X0 | 0X0 | 0X0 | 0X0 | 0X0 | 0X0 | 0X0 | 0X0 | 0X0 | 0X0 | 0X0 | 0X0 | 0X0 | 0X0 | 0X0 | 0X0 | 0X0 | 0X0 | 0X0 | 0X0 | 0X0 | 0X0 | 0X0 | 0X0 | 0X0 | 0X0 | 0X0 | 0X0 | 0X0 | 0X0 |
> Jun 30 14:59:00 [43005960] -> osm_pkey_rcv_process: ]
> Jun 30 14:59:00 [43005960] -> __osm_sm_mad_ctrl_disp_done_callback: [
> Jun 30 14:59:00 [43005960] -> __osm_sm_mad_ctrl_retire_trans_mad: [
> Jun 30 14:59:00 [43005960] -> __osm_sm_mad_ctrl_retire_trans_mad: Retiring MAD with TID = 0x16c64.
> Jun 30 14:59:00 [43005960] -> osm_mad_pool_put: [
> Jun 30 14:59:00 [43005960] -> osm_mad_pool_put: Releasing p_madw = 0x5b0920, p_mad = 0x5916c4.
> Jun 30 14:59:00 [43005960] -> osm_vendor_put: [
> Jun 30 14:59:00 [43005960] -> osm_vendor_put: Retiring UMAD 0x591690.
> Jun 30 14:59:00 [43005960] -> osm_vendor_put: ]
> Jun 30 14:59:00 [43005960] -> osm_mad_pool_put: ]
> Jun 30 14:59:00 [43005960] -> __osm_sm_mad_ctrl_retire_trans_mad: 9 QP0 MADs outstanding.
> Jun 30 14:59:00 [43005960] -> __osm_sm_mad_ctrl_retire_trans_mad: ]
> Jun 30 14:59:00 [43005960] -> __osm_sm_mad_ctrl_disp_done_callback: ]
> Jun 30 14:59:00 [43005960] -> osm_pkey_rcv_process: [
> Jun 30 14:59:00 [43005960] -> osm_pkey_rcv_process: Got GetResp(PKey) block:0 port_num 2 with GUID = 0x617000000000d for parent node GUID = 0x617000000000d, TID = 0x16c65.
> Jun 30 14:59:00 [43005960] -> P_Key table dump:
> port_guid...........0x000617000000000d
> block_num...........0x0
> port_num............0x2
> P_Key Table: 0X0 | 0X0 | 0X0 | 0X0 | 0X0 | 0X0 | 0X0 | 0X0 | 0X0 | 0X0 | 0X0 | 0X0 | 0X0 | 0X0 | 0X0 | 0X0 | 0X0 | 0X0 | 0X0 | 0X0 | 0X0 | 0X0 | 0X0 | 0X0 | 0X0 | 0X0 | 0X0 | 0X0 | 0X0 | 0X0 | 0X0 | 0X0 |
> Jun 30 14:59:00 [43005960] -> osm_pkey_rcv_process: ]
> Jun 30 14:59:00 [43005960] -> __osm_sm_mad_ctrl_disp_done_callback: [
> Jun 30 14:59:00 [43005960] -> __osm_sm_mad_ctrl_retire_trans_mad: [
> Jun 30 14:59:00 [43005960] -> __osm_sm_mad_ctrl_retire_trans_mad: Retiring MAD with TID = 0x16c65.
> Jun 30 14:59:00 [43005960] -> osm_mad_pool_put: [
> Jun 30 14:59:00 [43005960] -> osm_mad_pool_put: Releasing p_madw = 0x55add0, p_mad = 0x591d04.
> Jun 30 14:59:00 [43005960] -> osm_vendor_put: [
> Jun 30 14:59:00 [43005960] -> osm_vendor_put: Retiring UMAD 0x591cd0.
> Jun 30 14:59:00 [43005960] -> osm_vendor_put: ]
> Jun 30 14:59:00 [43005960] -> osm_mad_pool_put: ]
> Jun 30 14:59:00 [43005960] -> __osm_sm_mad_ctrl_retire_trans_mad: 8 QP0 MADs outstanding.
> Jun 30 14:59:00 [43005960] -> __osm_sm_mad_ctrl_retire_trans_mad: ]
> Jun 30 14:59:00 [43005960] -> __osm_sm_mad_ctrl_disp_done_callback: ]
> Jun 30 14:59:00 [43005960] -> osm_ni_rcv_process: [
> Jun 30 14:59:00 [43005960] -> NodeInfo dump:
> base_version............0x1
> class_version...........0x1
> node_type...............Channel Adapter
> num_ports...............0x2
> sys_guid................0x0002c9000100d050
> node_guid...............0x0002c901097624c0
> port_guid...............0x0002c901097624c1
> partition_cap...........0x40
> device_id...............0x5A44
> revision................0xA1
> port_num................0x1
> vendor_id...............0x2C9
> Jun 30 14:59:00 [43005960] -> __osm_ni_rcv_process_existing: [
> Jun 30 14:59:00 [43005960] -> __osm_ni_rcv_process_existing: Rediscovered Channel Adapter node 0x2c901097624c0
> TID = 0x16c66, discovered 0 times already.
> Jun 30 14:59:00 [43005960] -> __osm_ni_rcv_process_existing_ca: [
> Jun 30 14:59:00 [43005960] -> __osm_ni_rcv_process_ca_port: [
> Jun 30 14:59:00 [43005960] -> osm_req_get: [
> Jun 30 14:59:00 [43005960] -> osm_mad_pool_get: [
> Jun 30 14:59:00 [43005960] -> osm_vendor_get: [
> Jun 30 14:59:00 [43005960] -> osm_vendor_get: Acquiring UMAD for p_madw = 0x55ade8, size = 256.
> Jun 30 14:59:00 [43005960] -> osm_vendor_get: Acquired UMAD 0x5b1210, size = 256.
> Jun 30 14:59:00 [43005960] -> osm_vendor_get: ]
> Jun 30 14:59:00 [43005960] -> osm_mad_pool_get: Acquired p_madw = 0x55add0, p_mad = 0x5b1244, size = 256.
> Jun 30 14:59:00 [43005960] -> osm_mad_pool_get: ]
> Jun 30 14:59:00 [43005960] -> osm_req_get: Getting PortInfo (0x15), modifier = 0x1, TID = 0x16c6e.
> Jun 30 14:59:00 [43005960] -> osm_vl15_post: [
> Jun 30 14:59:00 [43005960] -> osm_vl15_post: Servicing p_madw = 0x55add0 (mad 0x5b1244 req 1)
> Jun 30 14:59:00 [43005960] -> osm_vl15_post: 0 MADs on wire, 9 MADs outstanding.
> Jun 30 14:59:00 [43005960] -> osm_vl15_poll: [
> Jun 30 14:59:00 [43005960] -> osm_vl15_poll: Signalling poller thread.
> Jun 30 14:59:00 [43005960] -> osm_vl15_poll: ]
> Jun 30 14:59:00 [43005960] -> osm_vl15_post: ]
> Jun 30 14:59:00 [43005960] -> osm_req_get: ]
> Jun 30 14:59:00 [43005960] -> __osm_ni_rcv_process_ca_port: ]
> Jun 30 14:59:00 [43005960] -> __osm_ni_rcv_process_existing_ca: ]
> Jun 30 14:59:00 [43005960] -> __osm_ni_rcv_set_links: [
> Jun 30 14:59:00 [43005960] -> __osm_ni_rcv_set_links: Link already exists.
> Jun 30 14:59:00 [43005960] -> __osm_ni_rcv_set_links: ]
> Jun 30 14:59:00 [43005960] -> __osm_ni_rcv_process_existing: ]
> Jun 30 14:59:00 [43005960] -> osm_ni_rcv_process: ]
> Jun 30 14:59:00 [43005960] -> __osm_sm_mad_ctrl_disp_done_callback: [
> Jun 30 14:59:00 [43005960] -> __osm_sm_mad_ctrl_retire_trans_mad: [
> Jun 30 14:59:00 [43005960] -> __osm_sm_mad_ctrl_retire_trans_mad: Retiring MAD with TID = 0x16c66.
> Jun 30 14:59:00 [43005960] -> osm_mad_pool_put: [
> Jun 30 14:59:00 [43005960] -> osm_mad_pool_put: Releasing p_madw = 0x5b09f0, p_mad = 0x592fa4.
> Jun 30 14:59:00 [43005960] -> osm_vendor_put: [
> Jun 30 14:59:00 [43005960] -> osm_vendor_put: Retiring UMAD 0x592f70.
> Jun 30 14:59:00 [43005960] -> osm_vendor_put: ]
> Jun 30 14:59:00 [43005960] -> osm_mad_pool_put: ]
> Jun 30 14:59:00 [43005960] -> __osm_sm_mad_ctrl_retire_trans_mad: 8 QP0 MADs outstanding.
> Jun 30 14:59:00 [43005960] -> __osm_sm_mad_ctrl_retire_trans_mad: ]
> Jun 30 14:59:00 [43005960] -> __osm_sm_mad_ctrl_disp_done_callback: ]
> Jun 30 14:59:00 [43005960] -> osm_pkey_rcv_process: [
> Jun 30 14:59:00 [43005960] -> osm_pkey_rcv_process: Got GetResp(PKey) block:0 port_num 3 with GUID = 0x617000000000d for parent node GUID = 0x617000000000d, TID = 0x16c67.
> Jun 30 14:59:00 [43005960] -> P_Key table dump:
> port_guid...........0x000617000000000d
> block_num...........0x0
> port_num............0x3
> P_Key Table: 0X0 | 0X0 | 0X0 | 0X0 | 0X0 | 0X0 | 0X0 | 0X0 | 0X0 | 0X0 | 0X0 | 0X0 | 0X0 | 0X0 | 0X0 | 0X0 | 0X0 | 0X0 | 0X0 | 0X0 | 0X0 | 0X0 | 0X0 | 0X0 | 0X0 | 0X0 | 0X0 | 0X0 | 0X0 | 0X0 | 0X0 | 0X0 |
> Jun 30 14:59:00 [43005960] -> osm_pkey_rcv_process: ]
> Jun 30 14:59:00 [43005960] -> __osm_sm_mad_ctrl_disp_done_callback: [
> Jun 30 14:59:00 [43005960] -> __osm_sm_mad_ctrl_retire_trans_mad: [
> Jun 30 14:59:00 [43005960] -> __osm_sm_mad_ctrl_retire_trans_mad: Retiring MAD with TID = 0x16c67.
> Jun 30 14:59:00 [43005960] -> osm_mad_pool_put: [
> Jun 30 14:59:00 [43005960] -> osm_mad_pool_put: Releasing p_madw = 0x561f90, p_mad = 0x592e64.
> Jun 30 14:59:00 [43005960] -> osm_vendor_put: [
> Jun 30 14:59:00 [43005960] -> osm_vendor_put: Retiring UMAD 0x592e30.
> Jun 30 14:59:00 [43005960] -> osm_vendor_put: ]
> Jun 30 14:59:00 [43005960] -> osm_mad_pool_put: ]
> Jun 30 14:59:00 [43005960] -> __osm_sm_mad_ctrl_retire_trans_mad: 7 QP0 MADs outstanding.
> Jun 30 14:59:00 [43005960] -> __osm_sm_mad_ctrl_retire_trans_mad: ]
> Jun 30 14:59:00 [43005960] -> __osm_sm_mad_ctrl_disp_done_callback: ]
> Jun 30 14:59:00 [43005960] -> osm_pkey_rcv_process: [
> Jun 30 14:59:00 [43005960] -> osm_pkey_rcv_process: Got GetResp(PKey) block:0 port_num 4 with GUID = 0x617000000000d for parent node GUID = 0x617000000000d, TID = 0x16c68.
> Jun 30 14:59:00 [43005960] -> P_Key table dump:
> port_guid...........0x000617000000000d
> block_num...........0x0
> port_num............0x4
> P_Key Table: 0X0 | 0X0 | 0X0 | 0X0 | 0X0 | 0X0 | 0X0 | 0X0 | 0X0 | 0X0 | 0X0 | 0X0 | 0X0 | 0X0 | 0X0 | 0X0 | 0X0 | 0X0 | 0X0 | 0X0 | 0X0 | 0X0 | 0X0 | 0X0 | 0X0 | 0X0 | 0X0 | 0X0 | 0X0 | 0X0 | 0X0 | 0X0 |
> Jun 30 14:59:00 [43005960] -> osm_pkey_rcv_process: ]
> Jun 30 14:59:00 [43005960] -> __osm_sm_mad_ctrl_disp_done_callback: [
> Jun 30 14:59:00 [43005960] -> __osm_sm_mad_ctrl_retire_trans_mad: [
> Jun 30 14:59:00 [43005960] -> __osm_sm_mad_ctrl_retire_trans_mad: Retiring MAD with TID = 0x16c68.
> Jun 30 14:59:00 [43005960] -> osm_mad_pool_put: [
> Jun 30 14:59:00 [43005960] -> osm_mad_pool_put: Releasing p_madw = 0x5b0850, p_mad = 0x591804.
> Jun 30 14:59:00 [43005960] -> osm_vendor_put: [
> Jun 30 14:59:00 [43005960] -> osm_vendor_put: Retiring UMAD 0x5917d0.
> Jun 30 14:59:00 [43005960] -> osm_vendor_put: ]
> Jun 30 14:59:00 [43005960] -> osm_mad_pool_put: ]
> Jun 30 14:59:00 [43005960] -> __osm_sm_mad_ctrl_retire_trans_mad: 6 QP0 MADs outstanding.
> Jun 30 14:59:00 [43005960] -> __osm_sm_mad_ctrl_retire_trans_mad: ]
> Jun 30 14:59:00 [43005960] -> __osm_sm_mad_ctrl_disp_done_callback: ]
> Jun 30 14:59:00 [43005960] -> osm_ni_rcv_process: [
> Jun 30 14:59:00 [43005960] -> NodeInfo dump:
> base_version............0x1
> class_version...........0x1
> node_type...............Channel Adapter
> num_ports...............0x2
> sys_guid................0x0002c90109765633
> node_guid...............0x0002c90109765630
> port_guid...............0x0002c90109765631
> partition_cap...........0x20
> device_id...............0x5A44
> revision................0xA1
> port_num................0x1
> vendor_id...............0x2C9
> Jun 30 14:59:00 [43005960] -> __osm_ni_rcv_process_existing: [
> Jun 30 14:59:00 [43005960] -> __osm_ni_rcv_process_existing: Rediscovered Channel Adapter node 0x2c90109765630
> TID = 0x16c69, discovered 0 times already.
> Jun 30 14:59:00 [43005960] -> __osm_ni_rcv_process_existing_ca: [
> Jun 30 14:59:00 [43005960] -> __osm_ni_rcv_process_ca_port: [
> Jun 30 14:59:00 [43005960] -> osm_req_get: [
> Jun 30 14:59:00 [43005960] -> osm_mad_pool_get: [
> Jun 30 14:59:00 [43005960] -> osm_vendor_get: [
> Jun 30 14:59:00 [43005960] -> osm_vendor_get: Acquiring UMAD for p_madw = 0x5b0868, size = 256.
> Jun 30 14:59:00 [43005960] -> osm_vendor_get: Acquired UMAD 0x5b0c40, size = 256.
> Jun 30 14:59:00 [43005960] -> osm_vendor_get: ]
> Jun 30 14:59:00 [43005960] -> osm_mad_pool_get: Acquired p_madw = 0x5b0850, p_mad = 0x5b0c74, size = 256.
> Jun 30 14:59:00 [43005960] -> osm_mad_pool_get: ]
> Jun 30 14:59:00 [43005960] -> osm_req_get: Getting PortInfo (0x15), modifier = 0x1, TID = 0x16c6f.
> Jun 30 14:59:00 [43005960] -> osm_vl15_post: [
> Jun 30 14:59:00 [43005960] -> osm_vl15_post: Servicing p_madw = 0x5b0850 (mad 0x5b0c74 req 1)
> Jun 30 14:59:00 [43005960] -> osm_vl15_post: 0 MADs on wire, 7 MADs outstanding.
> Jun 30 14:59:00 [43005960] -> osm_vl15_poll: [
> Jun 30 14:59:00 [43005960] -> osm_vl15_poll: Signalling poller thread.
> Jun 30 14:59:00 [43005960] -> osm_vl15_poll: ]
> Jun 30 14:59:00 [43005960] -> osm_vl15_post: ]
> Jun 30 14:59:00 [43005960] -> osm_req_get: ]
> Jun 30 14:59:00 [43005960] -> __osm_ni_rcv_process_ca_port: ]
> Jun 30 14:59:00 [43005960] -> __osm_ni_rcv_process_existing_ca: ]
> Jun 30 14:59:00 [43005960] -> __osm_ni_rcv_set_links: [
> Jun 30 14:59:00 [43005960] -> __osm_ni_rcv_set_links: Link already exists.
> Jun 30 14:59:00 [43005960] -> __osm_ni_rcv_set_links: ]
> Jun 30 14:59:00 [43005960] -> __osm_ni_rcv_process_existing: ]
> Jun 30 14:59:00 [43005960] -> osm_ni_rcv_process: ]
> Jun 30 14:59:00 [43005960] -> __osm_sm_mad_ctrl_disp_done_callback: [
> Jun 30 14:59:00 [43005960] -> __osm_sm_mad_ctrl_retire_trans_mad: [
> Jun 30 14:59:00 [43005960] -> __osm_sm_mad_ctrl_retire_trans_mad: Retiring MAD with TID = 0x16c69.
> Jun 30 14:59:00 [43005960] -> osm_mad_pool_put: [
> Jun 30 14:59:00 [43005960] -> osm_mad_pool_put: Releasing p_madw = 0x5b02a0, p_mad = 0x592144.
> Jun 30 14:59:00 [43005960] -> osm_vendor_put: [
> Jun 30 14:59:00 [43005960] -> osm_vendor_put: Retiring UMAD 0x592110.
> Jun 30 14:59:00 [43005960] -> osm_vendor_put: ]
> Jun 30 14:59:00 [43005960] -> osm_mad_pool_put: ]
> Jun 30 14:59:00 [43005960] -> __osm_sm_mad_ctrl_retire_trans_mad: 6 QP0 MADs outstanding.
> Jun 30 14:59:00 [43005960] -> __osm_sm_mad_ctrl_retire_trans_mad: ]
> Jun 30 14:59:00 [43005960] -> __osm_sm_mad_ctrl_disp_done_callback: ]
> Jun 30 14:59:00 [43005960] -> osm_pkey_rcv_process: [
> Jun 30 14:59:00 [43005960] -> osm_pkey_rcv_process: Got GetResp(PKey) block:0 port_num 5 with GUID = 0x617000000000d for parent node GUID = 0x617000000000d, TID = 0x16c6a.
> Jun 30 14:59:00 [43005960] -> P_Key table dump:
> port_guid...........0x000617000000000d
> block_num...........0x0
> port_num............0x5
> P_Key Table: 0X0 | 0X0 | 0X0 | 0X0 | 0X0 | 0X0 | 0X0 | 0X0 | 0X0 | 0X0 | 0X0 | 0X0 | 0X0 | 0X0 | 0X0 | 0X0 | 0X0 | 0X0 | 0X0 | 0X0 | 0X0 | 0X0 | 0X0 | 0X0 | 0X0 | 0X0 | 0X0 | 0X0 | 0X0 | 0X0 | 0X0 | 0X0 |
> Jun 30 14:59:00 [43005960] -> osm_pkey_rcv_process: ]
> Jun 30 14:59:00 [43005960] -> __osm_sm_mad_ctrl_disp_done_callback: [
> Jun 30 14:59:00 [43005960] -> __osm_sm_mad_ctrl_retire_trans_mad: [
> Jun 30 14:59:00 [43005960] -> __osm_sm_mad_ctrl_retire_trans_mad: Retiring MAD with TID = 0x16c6a.
> Jun 30 14:59:00 [43005960] -> osm_mad_pool_put: [
> Jun 30 14:59:00 [43005960] -> osm_mad_pool_put: Releasing p_madw = 0x560250, p_mad = 0x592514.
> Jun 30 14:59:00 [43005960] -> osm_vendor_put: [
> Jun 30 14:59:00 [43005960] -> osm_vendor_put: Retiring UMAD 0x5924e0.
> Jun 30 14:59:00 [43005960] -> osm_vendor_put: ]
> Jun 30 14:59:00 [43005960] -> osm_mad_pool_put: ]
> Jun 30 14:59:00 [43005960] -> __osm_sm_mad_ctrl_retire_trans_mad: 5 QP0 MADs outstanding.
> Jun 30 14:59:00 [43005960] -> __osm_sm_mad_ctrl_retire_trans_mad: ]
> Jun 30 14:59:00 [43005960] -> __osm_sm_mad_ctrl_disp_done_callback: ]
> Jun 30 14:59:00 [43005960] -> osm_pkey_rcv_process: [
> Jun 30 14:59:00 [43005960] -> osm_pkey_rcv_process: Got GetResp(PKey) block:0 port_num 6 with GUID = 0x617000000000d for parent node GUID = 0x617000000000d, TID = 0x16c6b.
> Jun 30 14:59:00 [43005960] -> P_Key table dump:
> port_guid...........0x000617000000000d
> block_num...........0x0
> port_num............0x6
> P_Key Table: 0X0 | 0X0 | 0X0 | 0X0 | 0X0 | 0X0 | 0X0 | 0X0 | 0X0 | 0X0 | 0X0 | 0X0 | 0X0 | 0X0 | 0X0 | 0X0 | 0X0 | 0X0 | 0X0 | 0X0 | 0X0 | 0X0 | 0X0 | 0X0 | 0X0 | 0X0 | 0X0 | 0X0 | 0X0 | 0X0 | 0X0 | 0X0 |
> Jun 30 14:59:00 [43005960] -> osm_pkey_rcv_process: ]
> Jun 30 14:59:00 [43005960] -> __osm_sm_mad_ctrl_disp_done_callback: [
> Jun 30 14:59:00 [43005960] -> __osm_sm_mad_ctrl_retire_trans_mad: [
> Jun 30 14:59:00 [43005960] -> __osm_sm_mad_ctrl_retire_trans_mad: Retiring MAD with TID = 0x16c6b.
> Jun 30 14:59:00 [43005960] -> osm_mad_pool_put: [
> Jun 30 14:59:00 [43005960] -> osm_mad_pool_put: Releasing p_madw = 0x5b0ac0, p_mad = 0x591a84.
> Jun 30 14:59:00 [43005960] -> osm_vendor_put: [
> Jun 30 14:59:00 [43005960] -> osm_vendor_put: Retiring UMAD 0x591a50.
> Jun 30 14:59:00 [43005960] -> osm_vendor_put: ]
> Jun 30 14:59:00 [43005960] -> osm_mad_pool_put: ]
> Jun 30 14:59:00 [43005960] -> __osm_sm_mad_ctrl_retire_trans_mad: 4 QP0 MADs outstanding.
> Jun 30 14:59:00 [43005960] -> __osm_sm_mad_ctrl_retire_trans_mad: ]
> Jun 30 14:59:00 [43005960] -> __osm_sm_mad_ctrl_disp_done_callback: ]
> Jun 30 14:59:00 [43806960] -> __osm_sm_mad_ctrl_send_err_cb: ERR 3113: MAD completed in error (IB_SUCCESS).
>
> ______________________________________________________________________
>
> _______________________________________________
> openib-general mailing list
> openib-general at openib.org
> http://openib.org/mailman/listinfo/openib-general
>
> To unsubscribe, please visit http://openib.org/mailman/listinfo/openib-general
More information about the general
mailing list