[ofa-general] lock dependency in ib_user_mad

[Sean Hefty]

I turned on lock checking and got the following possible locking dependency.
(Running on 2.6.24-rc3.)

I have two connectX cards connected back to back and was trying to run opensm in
QoS mode.  Opensm failed to bring the subnet up, and hitting ctrl-C to kill
opensm resulted in it hanging.  Using kill -9 resulted in this being written to
/var/log/messages.

Jan  7 12:23:35 mshefty-linux3 kernel: 
Jan  7 12:23:35 mshefty-linux3 kernel:
=======================================================
Jan  7 12:23:35 mshefty-linux3 kernel: [ INFO: possible circular locking
dependency detected ]
Jan  7 12:23:35 mshefty-linux3 kernel: 2.6.24-rc3 #3
Jan  7 12:23:35 mshefty-linux3 kernel:
-------------------------------------------------------
Jan  7 12:23:35 mshefty-linux3 kernel: opensm/7164 is trying to acquire lock:
Jan  7 12:23:35 mshefty-linux3 kernel:  (¥	
){--..}, at:
[<ffffffff8024d7b0>] flush_workqueue+0x0/0xa0
Jan  7 12:23:35 mshefty-linux3 kernel: 
Jan  7 12:23:35 mshefty-linux3 kernel: but task is already holding lock:
Jan  7 12:23:35 mshefty-linux3 kernel:  (&port->mutex){----}, at:
[<ffffffff8827840d>] ib_umad_close+0x2d/0x100 [ib_umad]
Jan  7 12:23:35 mshefty-linux3 kernel: 
Jan  7 12:23:35 mshefty-linux3 kernel: which lock already depends on the new
lock.
Jan  7 12:23:35 mshefty-linux3 kernel: 
Jan  7 12:23:35 mshefty-linux3 kernel: 
Jan  7 12:23:35 mshefty-linux3 kernel: the existing dependency chain (in reverse
order) is:
Jan  7 12:23:35 mshefty-linux3 kernel: 
Jan  7 12:23:35 mshefty-linux3 kernel: -> #2 (&port->mutex){----}:
Jan  7 12:23:35 mshefty-linux3 kernel:        [<ffffffff8025d237>]
__lock_acquire+0x7a7/0x10c0
Jan  7 12:23:35 mshefty-linux3 kernel:        [<ffffffff8025de63>]
lock_acquire+0x53/0x70
Jan  7 12:23:35 mshefty-linux3 kernel:        [<ffffffff88277099>]
queue_packet+0x49/0xf0 [ib_umad]
Jan  7 12:23:35 mshefty-linux3 kernel:        [<ffffffff8029d655>]
kmem_cache_alloc+0xa5/0xe0
Jan  7 12:23:35 mshefty-linux3 kernel:        [<ffffffff80255222>]
down_read+0x32/0x40
Jan  7 12:23:35 mshefty-linux3 kernel:        [<ffffffff88277099>]
queue_packet+0x49/0xf0 [ib_umad]
Jan  7 12:23:35 mshefty-linux3 kernel:        [<ffffffff8827730b>]
recv_handler+0xdb/0x170 [ib_umad]
Jan  7 12:23:35 mshefty-linux3 kernel:        [<ffffffff8025c2df>]
trace_hardirqs_on+0xbf/0x160
Jan  7 12:23:35 mshefty-linux3 kernel:        [<ffffffff881e5225>]
ib_mad_completion_handler+0x2d5/0x6f0 [ib_mad]
Jan  7 12:23:35 mshefty-linux3 kernel:        [<ffffffff8025d377>]
__lock_acquire+0x8e7/0x10c0
Jan  7 12:23:35 mshefty-linux3 kernel:        [<ffffffff881e4f50>]
ib_mad_completion_handler+0x0/0x6f0 [ib_mad]
Jan  7 12:23:35 mshefty-linux3 kernel:        [<ffffffff881e4f50>]
ib_mad_completion_handler+0x0/0x6f0 [ib_mad]
Jan  7 12:23:35 mshefty-linux3 kernel:        [<ffffffff8024d3bd>]
run_workqueue+0xed/0x220
Jan  7 12:23:35 mshefty-linux3 kernel:        [<ffffffff8025c2df>]
trace_hardirqs_on+0xbf/0x160
Jan  7 12:23:35 mshefty-linux3 kernel:        [<ffffffff8024d5c3>]
worker_thread+0xd3/0x140
Jan  7 12:23:35 mshefty-linux3 kernel:        [<ffffffff80252050>]
autoremove_wake_function+0x0/0x30
Jan  7 12:23:35 mshefty-linux3 kernel:        [<ffffffff80252050>]
autoremove_wake_function+0x0/0x30
Jan  7 12:23:35 mshefty-linux3 kernel:        [<ffffffff8024d4f0>]
worker_thread+0x0/0x140
Jan  7 12:23:35 mshefty-linux3 kernel:        [<ffffffff8025199c>]
kthread+0x6c/0xa0
Jan  7 12:23:35 mshefty-linux3 kernel:        [<ffffffff8020cad8>]
child_rip+0xa/0x12
Jan  7 12:23:35 mshefty-linux3 kernel:        [<ffffffff8020c1ef>]
restore_args+0x0/0x30
Jan  7 12:23:35 mshefty-linux3 kernel:        [<ffffffff80251cae>]
kthreadd+0x8e/0x170
Jan  7 12:23:35 mshefty-linux3 kernel:        [<ffffffff80251930>]
kthread+0x0/0xa0
Jan  7 12:23:35 mshefty-linux3 kernel:        [<ffffffff8020cace>]
child_rip+0x0/0x12
Jan  7 12:23:35 mshefty-linux3 kernel:        [<ffffffffffffffff>]
0xffffffffffffffff
Jan  7 12:23:35 mshefty-linux3 kernel: 
Jan  7 12:23:35 mshefty-linux3 kernel: -> #1 (&port_priv->work){--..}:
Jan  7 12:23:35 mshefty-linux3 kernel:        [<ffffffff8025d237>]
__lock_acquire+0x7a7/0x10c0
Jan  7 12:23:35 mshefty-linux3 kernel:        [<ffffffff881e4f50>]
ib_mad_completion_handler+0x0/0x6f0 [ib_mad]
Jan  7 12:23:35 mshefty-linux3 kernel:        [<ffffffff8025de63>]
lock_acquire+0x53/0x70
Jan  7 12:23:35 mshefty-linux3 kernel:        [<ffffffff8024d372>]
run_workqueue+0xa2/0x220
Jan  7 12:23:35 mshefty-linux3 kernel:        [<ffffffff804d04e4>]
_spin_unlock_irq+0x24/0x30
Jan  7 12:23:35 mshefty-linux3 kernel:        [<ffffffff8024d3b7>]
run_workqueue+0xe7/0x220
Jan  7 12:23:35 mshefty-linux3 kernel:        [<ffffffff8025c2df>]
trace_hardirqs_on+0xbf/0x160
Jan  7 12:23:35 mshefty-linux3 kernel:        [<ffffffff8024d5c3>]
worker_thread+0xd3/0x140
Jan  7 12:23:35 mshefty-linux3 kernel:        [<ffffffff80252050>]
autoremove_wake_function+0x0/0x30
Jan  7 12:23:35 mshefty-linux3 kernel:        [<ffffffff80252050>]
autoremove_wake_function+0x0/0x30
Jan  7 12:23:35 mshefty-linux3 kernel:        [<ffffffff8024d4f0>]
worker_thread+0x0/0x140
Jan  7 12:23:35 mshefty-linux3 kernel:        [<ffffffff8025199c>]
kthread+0x6c/0xa0
Jan  7 12:23:35 mshefty-linux3 kernel:        [<ffffffff8020cad8>]
child_rip+0xa/0x12
Jan  7 12:23:35 mshefty-linux3 kernel:        [<ffffffff8020c1ef>]
restore_args+0x0/0x30
Jan  7 12:23:35 mshefty-linux3 kernel:        [<ffffffff80251cae>]
kthreadd+0x8e/0x170
Jan  7 12:23:35 mshefty-linux3 kernel:        [<ffffffff80251930>]
kthread+0x0/0xa0
Jan  7 12:23:35 mshefty-linux3 kernel:        [<ffffffff8020cace>]
child_rip+0x0/0x12
Jan  7 12:23:35 mshefty-linux3 kernel:        [<ffffffffffffffff>]
0xffffffffffffffff
Jan  7 12:23:35 mshefty-linux3 kernel: 
Jan  7 12:23:35 mshefty-linux3 kernel: -> #0 (¥	
){--..}:
Jan  7 12:23:35 mshefty-linux3 kernel:        [<ffffffff80256d9a>]
print_stack_trace+0x6a/0x80
Jan  7 12:23:35 mshefty-linux3 kernel:        [<ffffffff8025d0a2>]
__lock_acquire+0x612/0x10c0
Jan  7 12:23:35 mshefty-linux3 kernel:        [<ffffffff8025d377>]
__lock_acquire+0x8e7/0x10c0
Jan  7 12:23:35 mshefty-linux3 kernel:        [<ffffffff8025de63>]
lock_acquire+0x53/0x70
Jan  7 12:23:35 mshefty-linux3 kernel:        [<ffffffff8024d7b0>]
flush_workqueue+0x0/0xa0
Jan  7 12:23:35 mshefty-linux3 kernel:        [<ffffffff804d0445>]
_spin_unlock_irqrestore+0x55/0x70
Jan  7 12:23:35 mshefty-linux3 kernel:        [<ffffffff8024d7f3>]
flush_workqueue+0x43/0xa0
Jan  7 12:23:35 mshefty-linux3 kernel:        [<ffffffff881e37b7>]
ib_unregister_mad_agent+0x297/0x460 [ib_mad]
Jan  7 12:23:35 mshefty-linux3 kernel:        [<ffffffff8827849e>]
ib_umad_close+0xbe/0x100 [ib_umad]
Jan  7 12:23:35 mshefty-linux3 kernel:        [<ffffffff802a41ab>]
__fput+0x1cb/0x200
Jan  7 12:23:35 mshefty-linux3 kernel:        [<ffffffff802a243b>]
filp_close+0x4b/0xa0
Jan  7 12:23:35 mshefty-linux3 kernel:        [<ffffffff8023d0e0>]
put_files_struct+0x70/0xc0
Jan  7 12:23:35 mshefty-linux3 kernel:        [<ffffffff8023db08>]
do_exit+0x1d8/0x8d0
Jan  7 12:23:35 mshefty-linux3 kernel:        [<ffffffff802463f7>]
__dequeue_signal+0x27/0x1e0
Jan  7 12:23:35 mshefty-linux3 kernel:        [<ffffffff8023e270>]
do_group_exit+0x30/0x90
Jan  7 12:23:35 mshefty-linux3 kernel:        [<ffffffff8024868e>]
get_signal_to_deliver+0x2fe/0x4f0
Jan  7 12:23:35 mshefty-linux3 kernel:        [<ffffffff8020afe5>]
do_notify_resume+0xc5/0x750
Jan  7 12:23:35 mshefty-linux3 kernel:        [<ffffffff804cfca4>]
trace_hardirqs_on_thunk+0x35/0x3a
Jan  7 12:23:35 mshefty-linux3 kernel:        [<ffffffff8025c2df>]
trace_hardirqs_on+0xbf/0x160
Jan  7 12:23:35 mshefty-linux3 kernel:        [<ffffffff8020bc40>]
sysret_signal+0x21/0x31
Jan  7 12:23:35 mshefty-linux3 kernel:        [<ffffffff8020bf47>]
ptregscall_common+0x67/0xb0
Jan  7 12:23:35 mshefty-linux3 kernel:        [<ffffffffffffffff>]
0xffffffffffffffff
Jan  7 12:23:35 mshefty-linux3 kernel: 
Jan  7 12:23:35 mshefty-linux3 kernel: other info that might help us debug this:
Jan  7 12:23:35 mshefty-linux3 kernel: 
Jan  7 12:23:35 mshefty-linux3 kernel: 1 lock held by opensm/7164:
Jan  7 12:23:35 mshefty-linux3 kernel:  #0:  (&port->mutex){----}, at:
[<ffffffff8827840d>] ib_umad_close+0x2d/0x100 [ib_umad]
Jan  7 12:23:35 mshefty-linux3 kernel: 
Jan  7 12:23:35 mshefty-linux3 kernel: stack backtrace:
Jan  7 12:23:35 mshefty-linux3 kernel: 
Jan  7 12:23:35 mshefty-linux3 kernel: Call Trace:
Jan  7 12:23:35 mshefty-linux3 kernel:  [<ffffffff8025afc5>]
print_circular_bug_tail+0x85/0x90
Jan  7 12:23:35 mshefty-linux3 kernel:  [<ffffffff80256d9a>]
print_stack_trace+0x6a/0x80
Jan  7 12:23:35 mshefty-linux3 kernel:  [<ffffffff8025d0a2>]
__lock_acquire+0x612/0x10c0
Jan  7 12:23:35 mshefty-linux3 kernel:  [<ffffffff8025d377>]
__lock_acquire+0x8e7/0x10c0
Jan  7 12:23:35 mshefty-linux3 kernel:  [<ffffffff8025de63>]
lock_acquire+0x53/0x70
Jan  7 12:23:35 mshefty-linux3 kernel:  [<ffffffff8024d7b0>]
flush_workqueue+0x0/0xa0
Jan  7 12:23:35 mshefty-linux3 kernel:  [<ffffffff804d0445>]
_spin_unlock_irqrestore+0x55/0x70
Jan  7 12:23:35 mshefty-linux3 kernel:  [<ffffffff8024d7f3>]
flush_workqueue+0x43/0xa0
Jan  7 12:23:35 mshefty-linux3 kernel:  [<ffffffff881e37b7>]
:ib_mad:ib_unregister_mad_agent+0x297/0x460
Jan  7 12:23:35 mshefty-linux3 kernel:  [<ffffffff8827849e>]
:ib_umad:ib_umad_close+0xbe/0x100
Jan  7 12:23:35 mshefty-linux3 kernel:  [<ffffffff802a41ab>] __fput+0x1cb/0x200
Jan  7 12:23:35 mshefty-linux3 kernel:  [<ffffffff802a243b>]
filp_close+0x4b/0xa0
Jan  7 12:23:35 mshefty-linux3 kernel:  [<ffffffff8023d0e0>]
put_files_struct+0x70/0xc0
Jan  7 12:23:35 mshefty-linux3 kernel:  [<ffffffff8023db08>] do_exit+0x1d8/0x8d0
Jan  7 12:23:35 mshefty-linux3 kernel:  [<ffffffff802463f7>]
__dequeue_signal+0x27/0x1e0
Jan  7 12:23:35 mshefty-linux3 kernel:  [<ffffffff8023e270>]
do_group_exit+0x30/0x90
Jan  7 12:23:35 mshefty-linux3 kernel:  [<ffffffff8024868e>]
get_signal_to_deliver+0x2fe/0x4f0
Jan  7 12:23:35 mshefty-linux3 kernel:  [<ffffffff8020afe5>]
do_notify_resume+0xc5/0x750
Jan  7 12:23:35 mshefty-linux3 kernel:  [<ffffffff804cfca4>]
trace_hardirqs_on_thunk+0x35/0x3a
Jan  7 12:23:35 mshefty-linux3 kernel:  [<ffffffff8025c2df>]
trace_hardirqs_on+0xbf/0x160
Jan  7 12:23:35 mshefty-linux3 kernel:  [<ffffffff8020bc40>]
sysret_signal+0x21/0x31
Jan  7 12:23:35 mshefty-linux3 kernel:  [<ffffffff8020bf47>]
ptregscall_common+0x67/0xb0
Jan  7 12:23:35 mshefty-linux3 kernel: