[ofa-general] possible irq lock inversion dependency detected
Bernd Schubert
bs at q-leap.de
Tue May 15 02:48:50 PDT 2007
Hi,
with 2.6.20 I get this message:
[263206.999448] =========================================================
[263207.007607] [ INFO: possible irq lock inversion dependency detected ]
[263207.014153] 2.6.20.3-debug #9
[263207.017230] ---------------------------------------------------------
[263207.023775] ipoib/6662 just changed the state of lock:
[263207.029020] (&idev->n_mcast_grps_lock){-...}, at: [_end+122750808/2124917936] ipath_multicast_detach+0x211/0x231 [ib_ipath]
[263207.039866] but this lock was taken by another, hard-irq-safe lock in the past:
[263207.047294] (mcast_lock){++..}
[263207.050380]
[263207.050381] and interrupts could create inverse lock ordering between them.
[263207.050382]
[263207.060846]
[263207.060847] other info that might help us debug this:
[263207.067609] 1 lock held by ipoib/6662:
[263207.071468] #0: (&priv->mcast_mutex){--..}, at: [mutex_lock+35/39] mutex_lock+0x23/0x27
[263207.080061]
[263207.080062] the first lock's dependencies:
[263207.085862] -> (&idev->n_mcast_grps_lock){-...} ops: 3 {
[263207.091371] initial-use at:
[263207.094647] [mark_lock+135/1127] mark_lock+0x87/0x467
[263207.104352] [__lock_acquire+1476/3168] __lock_acquire+0x5c4/0xc60
[263207.114571] [_end+122750035/2124917936] ipath_multicast_attach+0x178/0x26c [ib_ipath]
[263207.126446] [lock_acquire+124/160] lock_acquire+0x7c/0xa0
[263207.136317] [_end+122750035/2124917936] ipath_multicast_attach+0x178/0x26c [ib_ipath]
[263207.148192] [_spin_lock+34/46] _spin_lock+0x22/0x2e
[263207.157891] [_end+122750035/2124917936] ipath_multicast_attach+0x178/0x26c [ib_ipath]
[263207.169766] [_end+122587907/2124917936] ib_attach_mcast+0x2f/0x31 [ib_core]
[263207.187046] [_end+123887919/2124917936] ipoib_mcast_attach+0xe3/0x123 [ib_ipoib]
[263207.198481] [_end+123880632/2124917936] ipoib_mcast_join_finish+0x125/0x3fa [ib_ipoib]
[263207.210441] [check_usage+53/661] check_usage+0x35/0x295
[263207.220328] [lock_timer_base+35/72] lock_timer_base+0x23/0x48
[263207.230454] [__lock_acquire+3080/3168] __lock_acquire+0xc08/0xc60
[263207.240674] [_end+123881884/2124917936] ipoib_mcast_join_complete+0xcb/0x31a [ib_ipoib]
[263207.252720] [_read_unlock_irqrestore+56/71] _read_unlock_irqrestore+0x38/0x47
[263207.263544] [_end+122585085/2124917936] ib_unpack+0xad/0x11c [ib_core]
[263207.274117] [_end+123844451/2124917936] ib_sa_mcmember_rec_callback+0x4c/0x57 [ib_sa]
[263207.285983] [_end+123844738/2124917936] recv_handler+0x3f/0x4b [ib_sa]
[263207.296547] [_end+123718406/2124917936] ib_mad_completion_handler+0x3c7/0x59b [ib_mad]
[263207.308511] [run_workqueue+134/380] run_workqueue+0x86/0x17c
[263207.318552] [_end+123717439/2124917936] ib_mad_completion_handler+0x0/0x59b [ib_mad]
[263207.330333] [run_workqueue+177/380] run_workqueue+0xb1/0x17c
[263207.340377] [worker_thread+0/349] worker_thread+0x0/0x15d
[263207.350336] [keventd_create_kthread+0/122] keventd_create_kthread+0x0/0x7a
[263207.360983] [worker_thread+294/349] worker_thread+0x126/0x15d
[263207.371113] [default_wake_function+0/15] default_wake_function+0x0/0xf
[263207.381584] [default_wake_function+0/15] default_wake_function+0x0/0xf
[263207.392063] [worker_thread+0/349] worker_thread+0x0/0x15d
[263207.402020] [kthread+208/252] kthread+0xd0/0xfc
[263207.411459] [child_rip+10/18] child_rip+0xa/0x12
[263207.420985] [_spin_unlock_irq+40/45] _spin_unlock_irq+0x28/0x2d
[263207.431204] [restore_args+0/48] restore_args+0x0/0x30
[263207.440998] [kthread+0/252] kthread+0x0/0xfc
[263207.450344] [child_rip+0/18] child_rip+0x0/0x12
[263207.459862] [<ffffffffffffffff>] 0xffffffffffffffff
[263207.469386] hardirq-on-W at:
[263207.472661] [mark_lock+135/1127] mark_lock+0x87/0x467
[263207.482357] [__lock_acquire+1412/3168] __lock_acquire+0x584/0xc60
[263207.492567] [kfree+525/541] kfree+0x20d/0x21d
[263207.501996] [_end+122750808/2124917936] ipath_multicast_detach+0x211/0x231 [ib_ipath]
[263207.513865] [lock_acquire+124/160] lock_acquire+0x7c/0xa0
[263207.523735] [_end+122750808/2124917936] ipath_multicast_detach+0x211/0x231 [ib_ipath]
[263207.535601] [_spin_lock+34/46] _spin_lock+0x22/0x2e
[263207.545298] [_end+122750808/2124917936] ipath_multicast_detach+0x211/0x231 [ib_ipath]
[263207.557167] [debug_mutex_free_waiter+88/92] debug_mutex_free_waiter+0x58/0x5c
[263207.567990] [__mutex_lock_slowpath+624/637] __mutex_lock_slowpath+0x270/0x27d
[263207.578814] [wait_for_completion+189/198] wait_for_completion+0xbd/0xc6
[263207.589292] [_end+122587956/2124917936] ib_detach_mcast+0x2f/0x33 [ib_core]
[263207.600298] [_end+123888044/2124917936] ipoib_mcast_detach+0x3d/0x6e [ib_ipoib]
[263207.611650] [_end+123884728/2124917936] ipoib_mcast_leave+0x12d/0x1c8 [ib_ipoib]
[263207.623091] [_end+123886246/2124917936] ipoib_mcast_dev_flush+0x100/0x14e [ib_ipoib]
[263207.634877] [_end+123886283/2124917936] ipoib_mcast_dev_flush+0x125/0x14e [ib_ipoib]
[263207.646661] [_end+123878948/2124917936] ipoib_ib_dev_flush+0x0/0x11f [ib_ipoib]
[263207.658013] [_end+123877784/2124917936] ipoib_ib_dev_down+0xa8/0xb7 [ib_ipoib]
[263207.669300] [_end+123879087/2124917936] ipoib_ib_dev_flush+0x8b/0x11f [ib_ipoib]
[263207.680739] [run_workqueue+177/380] run_workqueue+0xb1/0x17c
[263207.690781] [worker_thread+0/349] worker_thread+0x0/0x15d
[263207.700732] [keventd_create_kthread+0/122] keventd_create_kthread+0x0/0x7a
[263207.711393] [worker_thread+294/349] worker_thread+0x126/0x15d
[263207.721523] [default_wake_function+0/15] default_wake_function+0x0/0xf
[263207.732000] [default_wake_function+0/15] default_wake_function+0x0/0xf
[263207.742478] [worker_thread+0/349] worker_thread+0x0/0x15d
[263207.752436] [kthread+208/252] kthread+0xd0/0xfc
[263207.761874] [child_rip+10/18] child_rip+0xa/0x12
[263207.771393] [_spin_unlock_irq+40/45] _spin_unlock_irq+0x28/0x2d
[263207.781603] [restore_args+0/48] restore_args+0x0/0x30
[263207.791388] [kthread+0/252] kthread+0x0/0xfc
[263207.800730] [child_rip+0/18] child_rip+0x0/0x12
[263207.810251] [<ffffffffffffffff>] 0xffffffffffffffff
[263207.819776] }
[263207.821548] ... key at: [_end+122893256/2124917936] __key.5+0x0/0xfffffffffffe369f [ib_ipath]
[263207.830154]
[263207.830155] the second lock's dependencies:
[263207.836049] -> (mcast_lock){++..} ops: 15329 {
[263207.840692] initial-use at:
[263207.843966] [<ffffffffffffffff>] 0xffffffffffffffff
[263207.853492] in-hardirq-W at:
[263207.856757] [<ffffffffffffffff>] 0xffffffffffffffff
[263207.866283] in-softirq-W at:
[263207.869550] [<ffffffffffffffff>] 0xffffffffffffffff
[263207.879071] }
[263207.880843] ... key at: [_end+122888936/2124917936] mcast_lock+0x18/0xfffffffffffe4797 [ib_ipath]
[263207.889798] -> (&idev->n_mcast_grps_lock){-...} ops: 3 {
[263207.895398] initial-use at:
[263207.898758] [mark_lock+135/1127] mark_lock+0x87/0x467
[263207.908679] [__lock_acquire+1476/3168] __lock_acquire+0x5c4/0xc60
[263207.919118] [_end+122750035/2124917936] ipath_multicast_attach+0x178/0x26c [ib_ipath]
[263207.931220] [lock_acquire+124/160] lock_acquire+0x7c/0xa0
[263207.941313] [_end+122750035/2124917936] ipath_multicast_attach+0x178/0x26c [ib_ipath]
[263207.953414] [_spin_lock+34/46] _spin_lock+0x22/0x2e
[263207.963335] [_end+122750035/2124917936] ipath_multicast_attach+0x178/0x26c [ib_ipath]
[263207.975429] [_end+122587907/2124917936] ib_attach_mcast+0x2f/0x31 [ib_core]
[263207.986661] [_end+123887919/2124917936] ipoib_mcast_attach+0xe3/0x123 [ib_ipoib]
[263207.998334] [_end+123880632/2124917936] ipoib_mcast_join_finish+0x125/0x3fa [ib_ipoib]
[263208.010527] [check_usage+53/661] check_usage+0x35/0x295
[263208.020622] [lock_timer_base+35/72] lock_timer_base+0x23/0x48
[263208.030971] [__lock_acquire+3080/3168] __lock_acquire+0xc08/0xc60
[263208.041409] [_end+123881884/2124917936] ipoib_mcast_join_complete+0xcb/0x31a [ib_ipoib]
[263208.053680] [_read_unlock_irqrestore+56/71] _read_unlock_irqrestore+0x38/0x47
[263208.064729] [_end+122585085/2124917936] ib_unpack+0xad/0x11c [ib_core]
[263208.075526] [_end+123844451/2124917936] ib_sa_mcmember_rec_callback+0x4c/0x57 [ib_sa]
[263208.087611] [_end+123844738/2124917936] recv_handler+0x3f/0x4b [ib_sa]
[263208.098399] [_end+123718406/2124917936] ib_mad_completion_handler+0x3c7/0x59b [ib_mad]
[263208.110585] [run_workqueue+134/380] run_workqueue+0x86/0x17c
[263208.127122] [_end+123717439/2124917936] ib_mad_completion_handler+0x0/0x59b [ib_mad]
[263208.139119] [run_workqueue+177/380] run_workqueue+0xb1/0x17c
[263208.149389] [worker_thread+0/349] worker_thread+0x0/0x15d
[263208.159571] [keventd_create_kthread+0/122] keventd_create_kthread+0x0/0x7a
[263208.170456] [worker_thread+294/349] worker_thread+0x126/0x15d
[263208.180814] [default_wake_function+0/15] default_wake_function+0x0/0xf
[263208.191516] [default_wake_function+0/15] default_wake_function+0x0/0xf
[263208.202231] [worker_thread+0/349] worker_thread+0x0/0x15d
[263208.212414] [kthread+208/252] kthread+0xd0/0xfc
[263208.222076] [child_rip+10/18] child_rip+0xa/0x12
[263208.231818] [_spin_unlock_irq+40/45] _spin_unlock_irq+0x28/0x2d
[263208.242254] [restore_args+0/48] restore_args+0x0/0x30
[263208.252263] [kthread+0/252] kthread+0x0/0xfc
[263208.261833] [child_rip+0/18] child_rip+0x0/0x12
[263208.271575] [<ffffffffffffffff>] 0xffffffffffffffff
[263208.281317] hardirq-on-W at:
[263208.284671] [mark_lock+135/1127] mark_lock+0x87/0x467
[263208.294593] [__lock_acquire+1412/3168] __lock_acquire+0x584/0xc60
[263208.305037] [kfree+525/541] kfree+0x20d/0x21d
[263208.314692] [_end+122750808/2124917936] ipath_multicast_detach+0x211/0x231 [ib_ipath]
[263208.326786] [lock_acquire+124/160] lock_acquire+0x7c/0xa0
[263208.336880] [_end+122750808/2124917936] ipath_multicast_detach+0x211/0x231 [ib_ipath]
[263208.348974] [_spin_lock+34/46] _spin_lock+0x22/0x2e
[263208.358896] [_end+122750808/2124917936] ipath_multicast_detach+0x211/0x231 [ib_ipath]
[263208.370987] [debug_mutex_free_waiter+88/92] debug_mutex_free_waiter+0x58/0x5c
[263208.382037] [__mutex_lock_slowpath+624/637] __mutex_lock_slowpath+0x270/0x27d
[263208.393086] [wait_for_completion+189/198] wait_for_completion+0xbd/0xc6
[263208.403789] [_end+122587956/2124917936] ib_detach_mcast+0x2f/0x33 [ib_core]
[263208.415020] [_end+123888044/2124917936] ipoib_mcast_detach+0x3d/0x6e [ib_ipoib]
[263208.426608] [_end+123884728/2124917936] ipoib_mcast_leave+0x12d/0x1c8 [ib_ipoib]
[263208.438274] [_end+123886246/2124917936] ipoib_mcast_dev_flush+0x100/0x14e [ib_ipoib]
[263208.450286] [_end+123886283/2124917936] ipoib_mcast_dev_flush+0x125/0x14e [ib_ipoib]
[263208.462296] [_end+123878948/2124917936] ipoib_ib_dev_flush+0x0/0x11f [ib_ipoib]
[263208.473872] [_end+123877784/2124917936] ipoib_ib_dev_down+0xa8/0xb7 [ib_ipoib]
[263208.485366] [_end+123879087/2124917936] ipoib_ib_dev_flush+0x8b/0x11f [ib_ipoib]
[263208.497030] [run_workqueue+177/380] run_workqueue+0xb1/0x17c
[263208.507298] [worker_thread+0/349] worker_thread+0x0/0x15d
[263208.517475] [keventd_create_kthread+0/122] keventd_create_kthread+0x0/0x7a
[263208.528355] [worker_thread+294/349] worker_thread+0x126/0x15d
[263208.538708] [default_wake_function+0/15] default_wake_function+0x0/0xf
[263208.549412] [default_wake_function+0/15] default_wake_function+0x0/0xf
[263208.560116] [worker_thread+0/349] worker_thread+0x0/0x15d
[263208.570299] [kthread+208/252] kthread+0xd0/0xfc
[263208.579968] [child_rip+10/18] child_rip+0xa/0x12
[263208.589713] [_spin_unlock_irq+40/45] _spin_unlock_irq+0x28/0x2d
[263208.600150] [restore_args+0/48] restore_args+0x0/0x30
[263208.610157] [kthread+0/252] kthread+0x0/0xfc
[263208.619724] [child_rip+0/18] child_rip+0x0/0x12
[263208.629469] [<ffffffffffffffff>] 0xffffffffffffffff
[263208.639220] }
[263208.641084] ... key at: [_end+122893256/2124917936] __key.5+0x0/0xfffffffffffe369f [ib_ipath]
[263208.649786] ... acquired at:
[263208.652862] [add_lock_to_list+125/169] add_lock_to_list+0x7d/0xa9
[263208.658899] [__lock_acquire+2822/3168] __lock_acquire+0xb06/0xc60
[263208.664936] [_end+122750035/2124917936] ipath_multicast_attach+0x178/0x26c [ib_ipath]
[263208.672640] [lock_acquire+124/160] lock_acquire+0x7c/0xa0
[263208.678330] [_end+122750035/2124917936] ipath_multicast_attach+0x178/0x26c [ib_ipath]
[263208.686036] [_spin_lock+34/46] _spin_lock+0x22/0x2e
[263208.691553] [_end+122750035/2124917936] ipath_multicast_attach+0x178/0x26c [ib_ipath]
[263208.699261] [_end+122587907/2124917936] ib_attach_mcast+0x2f/0x31 [ib_core]
[263208.706086] [_end+123887919/2124917936] ipoib_mcast_attach+0xe3/0x123 [ib_ipoib]
[263208.713342] [_end+123880632/2124917936] ipoib_mcast_join_finish+0x125/0x3fa [ib_ipoib]
[263208.721134] [check_usage+53/661] check_usage+0x35/0x295
[263208.726832] [lock_timer_base+35/72] lock_timer_base+0x23/0x48
[263208.732782] [__lock_acquire+3080/3168] __lock_acquire+0xc08/0xc60
[263208.738818] [_end+123881884/2124917936] ipoib_mcast_join_complete+0xcb/0x31a [ib_ipoib]
[263208.746688] [_read_unlock_irqrestore+56/71] _read_unlock_irqrestore+0x38/0x47
[263208.753333] [_end+122585085/2124917936] ib_unpack+0xad/0x11c [ib_core]
[263208.759722] [_end+123844451/2124917936] ib_sa_mcmember_rec_callback+0x4c/0x57 [ib_sa]
[263208.767435] [_end+123844738/2124917936] recv_handler+0x3f/0x4b [ib_sa]
[263208.773822] [_end+123718406/2124917936] ib_mad_completion_handler+0x3c7/0x59b [ib_mad]
[263208.781622] [run_workqueue+134/380] run_workqueue+0x86/0x17c
[263208.787486] [_end+123717439/2124917936] ib_mad_completion_handler+0x0/0x59b [ib_mad]
[263208.795106] [run_workqueue+177/380] run_workqueue+0xb1/0x17c
[263208.800969] [worker_thread+0/349] worker_thread+0x0/0x15d
[263208.806744] [keventd_create_kthread+0/122] keventd_create_kthread+0x0/0x7a
[263208.813215] [worker_thread+294/349] worker_thread+0x126/0x15d
[263208.819164] [default_wake_function+0/15] default_wake_function+0x0/0xf
[263208.825469] [default_wake_function+0/15] default_wake_function+0x0/0xf
[263208.831764] [worker_thread+0/349] worker_thread+0x0/0x15d
[263208.837541] [kthread+208/252] kthread+0xd0/0xfc
[263208.842796] [child_rip+10/18] child_rip+0xa/0x12
[263208.848137] [_spin_unlock_irq+40/45] _spin_unlock_irq+0x28/0x2d
[263208.854174] [restore_args+0/48] restore_args+0x0/0x30
[263208.859778] [kthread+0/252] kthread+0x0/0xfc
[263208.864946] [child_rip+0/18] child_rip+0x0/0x12
[263208.870289] [<ffffffffffffffff>] 0xffffffffffffffff
[263208.875632]
[263208.877236]
[263208.877236] stack backtrace:
[263208.881817]
[263208.881818] Call Trace:
[263208.885968] [print_irq_inversion_bug+292/307] print_irq_inversion_bug+0x124/0x133
[263208.892513] [check_usage_backwards+65/74] check_usage_backwards+0x41/0x4a
[263208.898711] [mark_lock+630/1127] mark_lock+0x276/0x467
[263208.904045] [__lock_acquire+1412/3168] __lock_acquire+0x584/0xc60
[263208.909813] [kfree+525/541] kfree+0x20d/0x21d
[263208.914808] [_end+122750808/2124917936] :ib_ipath:ipath_multicast_detach+0x211/0x231
[263208.922153] [lock_acquire+124/160] lock_acquire+0x7c/0xa0
[263208.927582] [_end+122750808/2124917936] :ib_ipath:ipath_multicast_detach+0x211/0x231
[263208.934927] [_spin_lock+34/46] _spin_lock+0x22/0x2e
[263208.940182] [_end+122750808/2124917936] :ib_ipath:ipath_multicast_detach+0x211/0x231
[263208.947526] [debug_mutex_free_waiter+88/92] debug_mutex_free_waiter+0x58/0x5c
[263208.953904] [__mutex_lock_slowpath+624/637] __mutex_lock_slowpath+0x270/0x27d
[263208.960277] [wait_for_completion+189/198] wait_for_completion+0xbd/0xc6
[263208.966307] [_end+122587956/2124917936] :ib_core:ib_detach_mcast+0x2f/0x33
[263208.972766] [_end+123888044/2124917936] :ib_ipoib:ipoib_mcast_detach+0x3d/0x6e
[263208.979575] [_end+123884728/2124917936] :ib_ipoib:ipoib_mcast_leave+0x12d/0x1c8
[263208.986469] [_end+123886246/2124917936] :ib_ipoib:ipoib_mcast_dev_flush+0x100/0x14e
[263208.993724] [_end+123886283/2124917936] :ib_ipoib:ipoib_mcast_dev_flush+0x125/0x14e
[263209.000981] [_end+123878948/2124917936] :ib_ipoib:ipoib_ib_dev_flush+0x0/0x11f
[263209.007790] [_end+123877784/2124917936] :ib_ipoib:ipoib_ib_dev_down+0xa8/0xb7
[263209.014510] [_end+123879087/2124917936] :ib_ipoib:ipoib_ib_dev_flush+0x8b/0x11f
[263209.021409] [run_workqueue+177/380] run_workqueue+0xb1/0x17c
[263209.027001] [worker_thread+0/349] worker_thread+0x0/0x15d
[263209.032508] [keventd_create_kthread+0/122] keventd_create_kthread+0x0/0x7a
[263209.038711] [worker_thread+294/349] worker_thread+0x126/0x15d
[263209.050667] [default_wake_function+0/15] default_wake_function+0x0/0xf
[263209.056697] [default_wake_function+0/15] default_wake_function+0x0/0xf
[263209.062726] [worker_thread+0/349] worker_thread+0x0/0x15d
[263209.068233] [kthread+208/252] kthread+0xd0/0xfc
[263209.073219] [child_rip+10/18] child_rip+0xa/0x12
[263209.078294] [_spin_unlock_irq+40/45] _spin_unlock_irq+0x28/0x2d
[263209.084062] [restore_args+0/48] restore_args+0x0/0x30
[263209.089396] [kthread+0/252] kthread+0x0/0xfc
[263209.094294] [child_rip+0/18] child_rip+0x0/0x12
[263209.099362]
[263209.101175] BUG: workqueue leaked lock or atomic: ipoib/0x00000000/6662
[263209.107904] last function: ipoib_ib_dev_flush+0x0/0x11f [ib_ipoib]
[263209.114575] 1 lock held by ipoib/6662:
[263209.118430] #0: (&priv->mcast_mutex){--..}, at: [mutex_lock+35/39] mutex_lock+0x23/0x27
[263209.127012]
[263209.127013] Call Trace:
[263209.131160] [run_workqueue+302/380] run_workqueue+0x12e/0x17c
[263209.136841] [worker_thread+0/349] worker_thread+0x0/0x15d
[263209.142354] [keventd_create_kthread+0/122] keventd_create_kthread+0x0/0x7a
[263209.148554] [worker_thread+294/349] worker_thread+0x126/0x15d
[263209.154237] [default_wake_function+0/15] default_wake_function+0x0/0xf
[263209.160264] [default_wake_function+0/15] default_wake_function+0x0/0xf
[263209.166297] [worker_thread+0/349] worker_thread+0x0/0x15d
[263209.171807] [kthread+208/252] kthread+0xd0/0xfc
[263209.176801] [child_rip+10/18] child_rip+0xa/0x12
[263209.181875] [_spin_unlock_irq+40/45] _spin_unlock_irq+0x28/0x2d
[263209.187646] [restore_args+0/48] restore_args+0x0/0x30
[263209.192977] [kthread+0/252] kthread+0x0/0xfc
[263209.197875] [child_rip+0/18] child_rip+0x0/0x12
[263209.202951]
[263209.204559] BUG: workqueue leaked lock or atomic: ipoib/0x00000000/6662
[263209.211282] last function: ipoib_ib_dev_flush+0x0/0x11f [ib_ipoib]
[263209.217946] 1 lock held by ipoib/6662:
[263209.221807] #0: (&priv->mcast_mutex){--..}, at: [mutex_lock+35/39] mutex_lock+0x23/0x27
...
[many more repeating traces]
--
Bernd Schubert
Q-Leap Networks GmbH
More information about the general
mailing list