[openib-general] Re: opensm failure

Tue Dec 28 09:09:55 PST 2004

Quoting r. shaharf (shaharf at voltaire.com) "RE: opensm failure":
> > Hi!
> Please update your SM, and run it again with "-V -d5" and send me also
> the stdout/stderr of the SM.
> 
> Shahar

Here.

-------------------------------------------------
OpenSM Rev:1.6.0-rc2
Command Line Arguments:
 Big V selected
 d level = 0x5
 Log File: /tmp/osm.log
-------------------------------------------------
warn: [32666] umad_init:
warn: [32666] umad_get_cas_names: max 10
warn: [32666] umad_get_cas_names: return 1 ca
warn: [32666] umad_get_ca_portguids: ca name mthca0 max port guids
warn: [32666] umad_get_ca: ca_name mthca0
warn: [32666] umad_get_ca: opened mthca0
warn: [32666] umad_get_ca_portguids: mthca0: 3 ports
using default guid 0xc900012a3981
warn: [32666] umad_get_ca_portguids: ca name mthca0 max port guids
warn: [32666] umad_get_ca: ca_name mthca0
warn: [32666] umad_get_ca: opened mthca0
warn: [32666] umad_get_ca_portguids: mthca0: 3 ports
warn: [32666] umad_get_ca: ca_name mthca0
warn: [32666] umad_get_ca: opened mthca0
warn: [32666] umad_get_port: ca_name mthca0 portnum 1
warn: [32666] umad_open_port: ca mthca0 port 1
warn: [32666] umad_open_port: open /dev/infiniband/mthca0/ports/1/mad failed

Error from osm_opensm_bind (0x2A)
warn: [32666] umad_done:
warn: [32666] umad_recv: portid -5 umad 0x80970e8 timeout 0

-------------- next part --------------
[1104201592:000212673][40188080] -> OpenSM Rev:1.6.0-rc2
[1104201592:000212887][40188080] -> osm_opensm_init: [
[1104201592:000212905][40188080] -> osm_opensm_init: Forcing single threaded dispatcher.
[1104201592:000213005][40188080] -> osm_vendor_new: [
[1104201592:000213016][40188080] -> osm_vendor_init: [
[1104201592:000213309][40188080] -> osm_vendor_init: ]
[1104201592:000213319][40188080] -> osm_vendor_new: ]
[1104201592:000213340][40188080] -> osm_mad_pool_init: [
[1104201592:000213455][40188080] -> osm_mad_pool_init: ]
[1104201592:000213464][40188080] -> osm_vl15_init: [
[1104201592:000213499][40188080] -> osm_vl15_init: ]
[1104201592:000213511][40188080] -> osm_sm_init: [
[1104201592:000213530][40188080] -> osm_sm_mad_ctrl_init: [
[1104201592:000213540][40188080] -> osm_sm_mad_ctrl_init: ]
[1104201592:000213550][40188080] -> osm_req_init: [
[1104201592:000213557][40188080] -> osm_req_init: ]
[1104201592:000213564][40188080] -> osm_req_ctrl_init: [
[1104201592:000213571][40188080] -> osm_req_ctrl_init: ]
[1104201592:000213579][40188080] -> osm_resp_init: [
[1104201592:000213586][40188080] -> osm_resp_init: ]
[1104201592:000213595][40188080] -> osm_ni_rcv_init: [
[1104201592:000213602][40188080] -> osm_ni_rcv_init: ]
[1104201592:000213609][40188080] -> osm_ni_rcv_ctrl_init: [
[1104201592:000213617][40188080] -> osm_ni_rcv_ctrl_init: ]
[1104201592:000213624][40188080] -> osm_pi_rcv_init: [
[1104201592:000213631][40188080] -> osm_pi_rcv_init: ]
[1104201592:000213638][40188080] -> osm_pi_rcv_ctrl_init: [
[1104201592:000213645][40188080] -> osm_pi_rcv_ctrl_init: ]
[1104201592:000213652][40188080] -> osm_si_rcv_init: [
[1104201592:000213659][40188080] -> osm_si_rcv_init: ]
[1104201592:000213666][40188080] -> osm_si_rcv_ctrl_init: [
[1104201592:000213673][40188080] -> osm_si_rcv_ctrl_init: ]
[1104201592:000213680][40188080] -> osm_nd_rcv_init: [
[1104201592:000213687][40188080] -> osm_nd_rcv_init: ]
[1104201592:000213694][40188080] -> osm_nd_rcv_ctrl_init: [
[1104201592:000213701][40188080] -> osm_nd_rcv_ctrl_init: ]
[1104201592:000213708][40188080] -> osm_lid_mgr_init: [
[1104201592:000213742][40188080] -> osm_lid_mgr_init: ]
[1104201592:000213753][40188080] -> osm_ucast_mgr_init: [
[1104201592:000213760][40188080] -> osm_ucast_mgr_init: ]
[1104201592:000213767][40188080] -> osm_link_mgr_init: [
[1104201592:000213774][40188080] -> osm_link_mgr_init: ]
[1104201592:000213783][40188080] -> osm_state_mgr_init: [
[1104201592:000213791][40188080] -> osm_state_mgr_init: ]
[1104201592:000213798][40188080] -> osm_state_mgr_ctrl_init: [
[1104201592:000213805][40188080] -> osm_state_mgr_ctrl_init: ]
[1104201592:000213812][40188080] -> osm_drop_mgr_init: [
[1104201592:000213819][40188080] -> osm_drop_mgr_init: ]
[1104201592:000213826][40188080] -> osm_lft_rcv_init: [
[1104201592:000213833][40188080] -> osm_lft_rcv_init: ]
[1104201592:000213840][40188080] -> osm_lft_rcv_ctrl_init: [
[1104201592:000213853][40188080] -> osm_lft_rcv_ctrl_init: ]
[1104201592:000213860][40188080] -> osm_mft_rcv_init: [
[1104201592:000213867][40188080] -> osm_mft_rcv_init: ]
[1104201592:000213874][40188080] -> osm_mft_rcv_ctrl_init: [
[1104201592:000213885][40188080] -> osm_mft_rcv_ctrl_init: ]
[1104201592:000213893][40188080] -> osm_sweep_fail_ctrl_init: [
[1104201592:000213900][40188080] -> osm_sweep_fail_ctrl_init: ]
[1104201592:000213911][40188080] -> osm_sminfo_rcv_init: [
[1104201592:000213918][40188080] -> osm_sminfo_rcv_init: ]
[1104201592:000213925][40188080] -> osm_sminfo_rcv_ctrl_init: [
[1104201592:000213933][40188080] -> osm_sminfo_rcv_ctrl_init: ]
[1104201592:000213940][40188080] -> osm_trap_rcv_init: [
[1104201592:000213948][40188080] -> cl_event_wheel_init: [
[1104201592:000213958][40188080] -> cl_event_wheel_init: ]
[1104201592:000213966][40188080] -> osm_trap_rcv_init: ]
[1104201592:000213973][40188080] -> osm_trap_rcv_ctrl_init: [
[1104201592:000213980][40188080] -> osm_trap_rcv_ctrl_init: ]
[1104201592:000213988][40188080] -> osm_sm_state_mgr_init: [
[1104201592:000213995][40188080] -> osm_sm_state_mgr_init: ]
[1104201592:000214003][40188080] -> osm_mcast_mgr_init: [
[1104201592:000214022][40188080] -> osm_mcast_mgr_init: ]
[1104201592:000214031][40188080] -> osm_slvl_rcv_init: [
[1104201592:000214038][40188080] -> osm_slvl_rcv_init: ]
[1104201592:000214045][40188080] -> osm_slvl_rcv_ctrl_init: [
[1104201592:000214053][40188080] -> osm_slvl_rcv_ctrl_init: ]
[1104201592:000214060][40188080] -> osm_vla_rcv_init: [
[1104201592:000214067][40188080] -> osm_vla_rcv_init: ]
[1104201592:000214074][40188080] -> osm_vla_rcv_ctrl_init: [
[1104201592:000214082][40188080] -> osm_vla_rcv_ctrl_init: ]
[1104201592:000214089][40188080] -> osm_pkey_rcv_init: [
[1104201592:000214096][40188080] -> osm_pkey_rcv_init: ]
[1104201592:000214103][40188080] -> osm_pkey_rcv_ctrl_init: [
[1104201592:000214110][40188080] -> osm_pkey_rcv_ctrl_init: ]
[1104201592:000214156][40F91BB0] -> __osm_vl15_poller: [
[1104201592:000214186][41192BB0] -> __osm_sm_sweeper: [
[1104201592:000214195][41192BB0] -> __osm_sm_sweeper: Masking ^C Signals.
[1104201592:000214217][40188080] -> osm_sm_init: ]
[1104201592:000214225][40188080] -> osm_sa_init: [
[1104201592:000214234][40188080] -> osm_sa_resp_init: [
[1104201592:000214242][40188080] -> osm_sa_resp_init: ]
[1104201592:000214252][40188080] -> osm_sa_mad_ctrl_init: [
[1104201592:000214260][40188080] -> osm_sa_mad_ctrl_init: ]
[1104201592:000214267][40188080] -> osm_cpi_rcv_init: [
[1104201592:000214273][40188080] -> osm_cpi_rcv_init: ]
[1104201592:000214281][40188080] -> osm_cpi_rcv_ctrl_init: [
[1104201592:000214288][40188080] -> osm_cpi_rcv_ctrl_init: ]
[1104201592:000214298][40188080] -> osm_nr_rcv_init: [
[1104201592:000214319][40188080] -> osm_nr_rcv_init: ]
[1104201592:000214339][40188080] -> osm_nr_rcv_ctrl_init: [
[1104201592:000214347][40188080] -> osm_nr_rcv_ctrl_init: ]
[1104201592:000214354][40188080] -> osm_pir_rcv_init: [
[1104201592:000214370][40188080] -> osm_pir_rcv_init: ]
[1104201592:000214378][40188080] -> osm_pir_rcv_ctrl_init: [
[1104201592:000214385][40188080] -> osm_pir_rcv_ctrl_init: ]
[1104201592:000214392][40188080] -> osm_lr_rcv_init: [
[1104201592:000214402][40188080] -> osm_lr_rcv_init: ]
[1104201592:000214409][40188080] -> osm_lr_rcv_ctrl_init: [
[1104201592:000214416][40188080] -> osm_lr_rcv_ctrl_init: ]
[1104201592:000214423][40188080] -> osm_pr_rcv_init: [
[1104201592:000214438][40188080] -> osm_pr_rcv_init: ]
[1104201592:000214447][40188080] -> osm_pr_rcv_ctrl_init: [
[1104201592:000214454][40188080] -> osm_pr_rcv_ctrl_init: ]
[1104201592:000214461][40188080] -> osm_smir_rcv_init: [
[1104201592:000214468][40188080] -> osm_smir_rcv_init: ]
[1104201592:000214475][40188080] -> osm_smir_ctrl_init: [
[1104201592:000214482][40188080] -> osm_smir_ctrl_init: ]
[1104201592:000214490][40188080] -> osm_mcmr_rcv_init: [
[1104201592:000214506][40188080] -> osm_mcmr_rcv_init: ]
[1104201592:000214514][40188080] -> osm_mcmr_rcv_ctrl_init: [
[1104201592:000214521][40188080] -> osm_mcmr_rcv_ctrl_init: ]
[1104201592:000214528][40188080] -> osm_sr_rcv_init: [
[1104201592:000214560][40188080] -> osm_sr_rcv_init: ]
[1104201592:000214568][40188080] -> osm_sr_rcv_ctrl_init: [
[1104201592:000214575][40188080] -> osm_sr_rcv_ctrl_init: ]
[1104201592:000214582][40188080] -> osm_infr_rcv_init: [
[1104201592:000214589][40188080] -> osm_infr_rcv_init: ]
[1104201592:000214596][40188080] -> osm_infr_rcv_ctrl_init: [
[1104201592:000214604][40188080] -> osm_infr_rcv_ctrl_init: ]
[1104201592:000214611][40188080] -> osm_vlarb_rec_rcv_init: [
[1104201592:000214620][40188080] -> osm_vlarb_rec_rcv_init: ]
[1104201592:000214628][40188080] -> osm_vlarb_rec_rcv_ctrl_init: [
[1104201592:000214636][40188080] -> osm_vlarb_rec_rcv_ctrl_init: ]
[1104201592:000214643][40188080] -> osm_slvl_rec_rcv_init: [
[1104201592:000214652][40188080] -> osm_slvl_rec_rcv_init: ]
[1104201592:000214660][40188080] -> osm_slvl_rec_rcv_ctrl_init: [
[1104201592:000214667][40188080] -> osm_slvl_rec_rcv_ctrl_init: ]
[1104201592:000214675][40188080] -> osm_pkey_rec_rcv_init: [
[1104201592:000214690][40188080] -> osm_pkey_rec_rcv_init: ]
[1104201592:000214698][40188080] -> osm_pkey_rec_rcv_ctrl_init: [
[1104201592:000214713][40188080] -> osm_pkey_rec_rcv_ctrl_init: ]
[1104201592:000214721][40188080] -> osm_lftr_rcv_init: [
[1104201592:000214736][40188080] -> osm_lftr_rcv_init: ]
[1104201592:000214744][40188080] -> osm_lftr_rcv_ctrl_init: [
[1104201592:000214752][40188080] -> osm_lftr_rcv_ctrl_init: ]
[1104201592:000214758][40188080] -> osm_sa_init: ]
[1104201592:000214766][40188080] -> osm_opensm_create_mcgroups: [
[1104201592:000214774][40188080] -> osm_sa_create_template_record_ipoib: [
[1104201592:000214786][40188080] -> osm_mcmr_rcv_create_new_mgrp: [
[1104201592:000214794][40188080] -> osm_mcmr_rcv_create_new_mgrp: Getting new mlid 0xc000.
[1104201592:000214802][40188080] -> __validate_requested_mgid: [
[1104201592:000214811][40188080] -> __validate_requested_mgid:  MGID Signed as 0x401B.
[1104201592:000214820][40188080] -> __validate_requested_mgid:  Skeeping MGID Validation for IPoIB Signed (0x401B) MGIDs.
[1104201592:000214828][40188080] -> __validate_requested_mgid: ]
[1104201592:000214838][40188080] -> osm_mgrp_send_create_notice: [
[1104201592:000214849][40188080] -> osm_report_notice: [
[1104201592:000214863][40188080] -> osm_report_notice: Reporting Generic Notice type:3 num:66 from LID:0x0000 GUID:0xfe80000000000000,0x0000000000000000
[1104201592:000214877][40188080] -> osm_report_notice: ]
[1104201592:000214887][40188080] -> osm_mgrp_send_create_notice: ]
[1104201592:000214894][40188080] -> osm_mcmr_rcv_create_new_mgrp: ]
[1104201592:000214901][40188080] -> osm_mcmr_rcv_create_new_mgrp: [
[1104201592:000214908][40188080] -> osm_mcmr_rcv_create_new_mgrp: Getting new mlid 0xc001.
[1104201592:000214915][40188080] -> __validate_requested_mgid: [
[1104201592:000214923][40188080] -> __validate_requested_mgid:  MGID Signed as 0x401B.
[1104201592:000214930][40188080] -> __validate_requested_mgid:  Skeeping MGID Validation for IPoIB Signed (0x401B) MGIDs.
[1104201592:000214937][40188080] -> __validate_requested_mgid: ]
[1104201592:000214945][40188080] -> osm_mgrp_send_create_notice: [
[1104201592:000214953][40188080] -> osm_report_notice: [
[1104201592:000214962][40188080] -> osm_report_notice: Reporting Generic Notice type:3 num:66 from LID:0x0000 GUID:0xfe80000000000000,0x0000000000000000
[1104201592:000214971][40188080] -> osm_report_notice: ]
[1104201592:000214978][40188080] -> osm_mgrp_send_create_notice: ]
[1104201592:000214985][40188080] -> osm_mcmr_rcv_create_new_mgrp: ]
[1104201592:000214992][40188080] -> osm_sa_create_template_record_ipoib: ]
[1104201592:000215000][40188080] -> osm_opensm_create_mcgroups: ]
[1104201592:000215011][40188080] -> updn_construct: [
[1104201592:000215019][40188080] -> updn_construct: ]
[1104201592:000215026][40188080] -> updn_init: [
[1104201592:000215038][40188080] -> updn_init: ]
[1104201592:000215046][40188080] -> osm_opensm_init: ]
[1104201592:000215070][40188080] -> osm_vendor_get_all_port_attr: [
[1104201592:000216450][40188080] -> osm_vendor_get_all_port_attr: assign CA  0x8098100ort 1 guid (0xc900012a3981) as the default port.
[1104201592:000216461][40188080] -> osm_vendor_get_all_port_attr: ]
[1104201592:000216544][40188080] -> osm_opensm_bind: [
[1104201592:000216553][40188080] -> osm_sm_bind: [
[1104201592:000216560][40188080] -> osm_sm_mad_ctrl_bind: [
[1104201592:000216568][40188080] -> osm_sm_mad_ctrl_bind: Binding to port 0xc900012a3981.
[1104201592:000216576][40188080] -> osm_vendor_bind: [
[1104201592:000216583][40188080] -> osm_vendor_bind: Binding to port 0xc900012a3981.
[1104201592:000216591][40188080] -> osm_vendor_open_port: [
[1104201592:000219904][40188080] -> osm_vl15_init: [
[1104201592:000219967][40188080] -> osm_vl15_init: ]
[1104201592:000219976][40188080] -> osm_vendor_open_port: ]
[1104201592:000219985][40188080] -> osm_vendor_bind: Unable to Open Port 0xc900012a3981.
[1104201592:000219992][40188080] -> osm_vendor_bind: ]
[1104201592:000220000][40188080] -> osm_sm_mad_ctrl_bind: ERR 3118: Vendor specific bind() failed.
[1104201592:000220009][40188080] -> osm_sm_mad_ctrl_bind: ]
[1104201592:000220017][40188080] -> osm_sm_bind: ERR 2E10: SM MAD Controller bind() failed (IB_ERROR).
[1104201592:000220037][40188080] -> osm_sm_bind: ]
[1104201592:000220045][40188080] -> osm_opensm_bind: ]
[1104201592:000220170][40188080] -> osm_vl15_shutdown: [
[1104201592:000220179][40188080] -> osm_vl15_shutdown: ]
[1104201592:000220259][40188080] -> osm_sm_destroy: [
[1104201592:000220287][41393BB0] -> umad_receiver: [
[1104201592:000220414][41393BB0] -> umad_receiver: recv error Success
[1104201592:000220450][41192BB0] -> __osm_sm_sweeper: Off schedule sweep signalled.
[1104201592:000220460][41192BB0] -> __osm_sm_sweeper: ]
[1104201592:000220477][40188080] -> osm_trap_rcv_destroy: [
[1104201592:000220486][40188080] -> cl_event_wheel_destroy: [
[1104201592:000220493][40188080] -> cl_event_wheel_dump: [
[1104201592:000220501][40188080] -> cl_event_wheel_dump: event_wheel ptr:0x8093f18 
[1104201592:000220508][40188080] -> cl_event_wheel_dump: ]
[1104201592:000220523][40188080] -> cl_event_wheel_destroy: ]
[1104201592:000220530][40188080] -> osm_trap_rcv_destroy: ]
[1104201592:000220539][40188080] -> osm_sminfo_rcv_destroy: [
[1104201592:000220547][40188080] -> osm_sminfo_rcv_destroy: ]
[1104201592:000220555][40188080] -> osm_ni_rcv_destroy: [
[1104201592:000220562][40188080] -> osm_ni_rcv_destroy: ]
[1104201592:000220569][40188080] -> osm_pi_rcv_destroy: [
[1104201592:000220576][40188080] -> osm_pi_rcv_destroy: ]
[1104201592:000220583][40188080] -> osm_si_rcv_destroy: [
[1104201592:000220590][40188080] -> osm_si_rcv_destroy: ]
[1104201592:000220598][40188080] -> osm_nd_rcv_destroy: [
[1104201592:000220604][40188080] -> osm_nd_rcv_destroy: ]
[1104201592:000220611][40188080] -> osm_lid_mgr_destroy: [
[1104201592:000220620][40188080] -> osm_lid_mgr_destroy: ]
[1104201592:000220627][40188080] -> osm_ucast_mgr_destroy: [
[1104201592:000220634][40188080] -> osm_ucast_mgr_destroy: ]
[1104201592:000220641][40188080] -> osm_link_mgr_destroy: [
[1104201592:000220648][40188080] -> osm_link_mgr_destroy: ]
[1104201592:000220655][40188080] -> osm_drop_mgr_destroy: [
[1104201592:000220661][40188080] -> osm_drop_mgr_destroy: ]
[1104201592:000220669][40188080] -> osm_lft_rcv_destroy: [
[1104201592:000220676][40188080] -> osm_lft_rcv_destroy: ]
[1104201592:000220683][40188080] -> osm_mft_rcv_destroy: [
[1104201592:000220690][40188080] -> osm_mft_rcv_destroy: ]
[1104201592:000220698][40188080] -> osm_slvl_rcv_destroy: [
[1104201592:000220704][40188080] -> osm_slvl_rcv_destroy: ]
[1104201592:000220712][40188080] -> osm_vla_rcv_destroy: [
[1104201592:000220719][40188080] -> osm_vla_rcv_destroy: ]
[1104201592:000220727][40188080] -> osm_pkey_rcv_destroy: [
[1104201592:000220733][40188080] -> osm_pkey_rcv_destroy: ]
[1104201592:000220741][40188080] -> osm_state_mgr_destroy: [
[1104201592:000220750][40188080] -> osm_state_mgr_destroy: ]
[1104201592:000220758][40188080] -> osm_sm_state_mgr_destroy: [
[1104201592:000220765][40188080] -> osm_sm_state_mgr_destroy: ]
[1104201592:000220773][40188080] -> osm_mcast_mgr_destroy: [
[1104201592:000220780][40188080] -> osm_mcast_mgr_destroy: ]
[1104201592:000220834][40188080] -> osm_sm_destroy: ]
[1104201592:000220843][40188080] -> osm_sa_destroy: [
[1104201592:000220852][40188080] -> osm_nr_rcv_destroy: [
[1104201592:000220861][40188080] -> osm_nr_rcv_destroy: ]
[1104201592:000220870][40188080] -> osm_pir_rcv_destroy: [
[1104201592:000220880][40188080] -> osm_pir_rcv_destroy: ]
[1104201592:000220888][40188080] -> osm_lr_rcv_destroy: [
[1104201592:000220897][40188080] -> osm_lr_rcv_destroy: ]
[1104201592:000220904][40188080] -> osm_pr_rcv_destroy: [
[1104201592:000220912][40188080] -> osm_pr_rcv_destroy: ]
[1104201592:000220919][40188080] -> osm_smir_rcv_destroy: [
[1104201592:000220926][40188080] -> osm_smir_rcv_destroy: ]
[1104201592:000220934][40188080] -> osm_mcmr_rcv_destroy: [
[1104201592:000220941][40188080] -> osm_mcmr_rcv_destroy: ]
[1104201592:000220949][40188080] -> osm_sr_rcv_destroy: [
[1104201592:000220969][40188080] -> osm_sr_rcv_destroy: ]
[1104201592:000220978][40188080] -> osm_infr_rcv_destroy: [
[1104201592:000220985][40188080] -> osm_infr_rcv_destroy: ]
[1104201592:000221001][40188080] -> osm_vlarb_rec_rcv_destroy: [
[1104201592:000221010][40188080] -> osm_vlarb_rec_rcv_destroy: ]
[1104201592:000221018][40188080] -> osm_slvl_rec_rcv_destroy: [
[1104201592:000221026][40188080] -> osm_slvl_rec_rcv_destroy: ]
[1104201592:000221034][40188080] -> osm_pkey_rec_rcv_destroy: [
[1104201592:000221042][40188080] -> osm_pkey_rec_rcv_destroy: ]
[1104201592:000221050][40188080] -> osm_lftr_rcv_destroy: [
[1104201592:000221057][40188080] -> osm_lftr_rcv_destroy: ]
[1104201592:000221064][40188080] -> osm_sa_destroy: ]