[ofa-general] Re: 2.6.30.1: possible irq lock inversion dependency detected

[Bart Van Assche]

On Wed, Aug 5, 2009 at 8:31 PM, Roland Dreier<rdreier at cisco.com> wrote:
> So I queued up the patch below for 2.6.32... this is almost the same as
> the patch I proposed before except that I fixed two places where I
> dropped the lock *after* calling ipoib_send() -- which missed the whole
> point of what I was trying to do.  So this patch has a much better
> chance of actually working!

After having applied this patch it took somewhat longer before a
locking inversion report was generated, but unfortunately there still
was a locking inversion report generated (see also
http://bugzilla.kernel.org/show_bug.cgi?id=13757 for the details):

=========================================================
[ INFO: possible irq lock inversion dependency detected ]
2.6.30.4-scst-debug #1
---------------------------------------------------------
swapper/0 just changed the state of lock:
 (&priv->lock){-.-...}, at: [<ffffffffa050cc8f>]
ipoib_cm_rx_event_handler+0x4f/0xa0 [ib_ipoib]
but this lock took another, HARDIRQ-unsafe lock in the past:
 (&(&mad_agent_priv->timed_work)->timer){+.-...}

and interrupts could create inverse lock ordering between them.

[ ... ]

stack backtrace:
Pid: 0, comm: swapper Not tainted 2.6.30.4-scst-debug #1
Call Trace:
 <IRQ>  [<ffffffff80272bec>] print_irq_inversion_bug+0x14c/0x1c0
 [<ffffffff80272cdd>] check_usage_forwards+0x7d/0xc0
 [<ffffffff80271faf>] mark_lock+0x20f/0x6a0
 [<ffffffff80272c60>] ? check_usage_forwards+0x0/0xc0
 [<ffffffff802743e4>] __lock_acquire+0xce4/0x1c80
 [<ffffffff80272c60>] ? check_usage_forwards+0x0/0xc0
 [<ffffffff80275488>] lock_acquire+0x108/0x150
 [<ffffffffa050cc8f>] ? ipoib_cm_rx_event_handler+0x4f/0xa0 [ib_ipoib]
 [<ffffffff80515101>] _spin_lock_irqsave+0x41/0x60
 [<ffffffffa050cc8f>] ? ipoib_cm_rx_event_handler+0x4f/0xa0 [ib_ipoib]
 [<ffffffffa050cc8f>] ipoib_cm_rx_event_handler+0x4f/0xa0 [ib_ipoib]
 [<ffffffffa04e56aa>] mlx4_ib_qp_event+0x7a/0xf0 [mlx4_ib]
 [<ffffffffa0252d4f>] mlx4_qp_event+0x6f/0xe0 [mlx4_core]
 [<ffffffffa024a659>] mlx4_eq_int+0x289/0x2e0 [mlx4_core]
 [<ffffffffa024a73f>] mlx4_msi_x_interrupt+0xf/0x20 [mlx4_core]
 [<ffffffff8028bf35>] handle_IRQ_event+0x95/0x200
 [<ffffffff8028e3d8>] handle_edge_irq+0xc8/0x170
 [<ffffffff8020eeef>] handle_irq+0x1f/0x30
 [<ffffffff8020e5fe>] do_IRQ+0x6e/0xf0
 [<ffffffff8020c913>] ret_from_intr+0x0/0xf
 <EOI>  [<ffffffffa0012d9e>] ? acpi_idle_enter_bm+0x27d/0x2ad [processor]
 [<ffffffffa0012d94>] ? acpi_idle_enter_bm+0x273/0x2ad [processor]
 [<ffffffff8046eae5>] ? cpuidle_idle_call+0xa5/0x100
 [<ffffffff8020b144>] ? cpu_idle+0x64/0xd0
 [<ffffffff8050de61>] ? start_secondary+0x188/0x1e7

Bart.