[Users] ibacm preventing sysrq poweroff in OFED 3.18

[Marcus Epperson]

It looks like a change to the mlx4 driver between OFED 3.12 and 3.18 broke
sysrq poweroff for us, but only when ibacm is running. Here's what I see
when I issue sysrq 'o':

SysRq : Power Off
igb 0000:02:00.1: PCI INT B disabled
igb 0000:02:00.0: PCI INT A disabled
INFO: task events/0:67 blocked for more than 120 seconds.
      Tainted: P           ---------------    2.6.32-504.16.2.el6.x86_64 #1
"echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
events/0      D 0000000000000000     0    67      2 0x00000000
 ffff881029697b40 0000000000000046 0000000000000000 ffff881000000000
 ffff881029697a90 ffff881029697a90 0000f1546265231e ffff881029697ae0
 ffff88100af7e520 000000010fcaaf92 ffff881029695ad8 ffff881029697fd8
Call Trace:
 [<ffffffff8152ad25>] schedule_timeout+0x215/0x2e0
 [<ffffffff8152a9a3>] wait_for_common+0x123/0x180
 [<ffffffff81064bc0>] ? default_wake_function+0x0/0x20
 [<ffffffff8152aabd>] wait_for_completion+0x1d/0x20
 [<ffffffffa037b0b3>] ib_uverbs_remove_one+0x73/0xa0 [ib_uverbs]
 [<ffffffffa0080e5f>] ib_unregister_device+0x4f/0x100 [ib_core]
 [<ffffffffa01354e1>] mlx4_ib_remove+0x41/0x270 [mlx4_ib]
 [<ffffffffa00ef5d1>] mlx4_remove_device+0x71/0x90 [mlx4_core]
 [<ffffffffa00ef633>] mlx4_unregister_device+0x43/0x90 [mlx4_core]
 [<ffffffffa00f17ff>] mlx4_unload_one+0xdf/0x390 [mlx4_core]
 [<ffffffff810c7e50>] ? do_poweroff+0x0/0x10
 [<ffffffff812af4bc>] pci_device_shutdown+0x2c/0x50
 [<ffffffff81367dfb>] device_shutdown+0x4b/0x120
 [<ffffffff81095eb6>] kernel_shutdown_prepare+0x36/0x40
 [<ffffffff81095ed3>] kernel_power_off+0x13/0x50
 [<ffffffff810c7e5e>] do_poweroff+0xe/0x10
 [<ffffffff81098090>] worker_thread+0x170/0x2a0
 [<ffffffff8109ebb0>] ? autoremove_wake_function+0x0/0x40
 [<ffffffff81097f20>] ? worker_thread+0x0/0x2a0
 [<ffffffff8109e71e>] kthread+0x9e/0xc0
 [<ffffffff8100c20a>] child_rip+0xa/0x20
 [<ffffffff8109e680>] ? kthread+0x0/0xc0
 [<ffffffff8100c200>] ? child_rip+0x0/0x20

The node will stay powered on indefinitely at this point, unless I kill
ibacm. Once ibacm is gone, I see the expected "Power down." message and the
node powers off right away.

Some of our own processes were also preventing the poweroff from happening.
We had to add handing for RDMA_CM_EVENT_DEVICE_REMOVAL to get around it --
we just exit immediately when we see that. I don't know how to do the
equivalent for ibacm though.

In general it seems wrong that a "power off right now" mechanism should be
defeated by userspace processes like this. I thought about removing the
'shutdown' hook in our mlx4 driver, as show below, going back to how it was
in OFED 3.12. Is there any downside to that, other than having to maintain
my own patch forever?

Thanks for any input,
-Marcus


--- compat-rdma-3.18-1/drivers/net/ethernet/mellanox/mlx4/main.c.orig
2015-11-24 01:15:01.000000000 -0800
+++ compat-rdma-3.18-1/drivers/net/ethernet/mellanox/mlx4/main.c
2015-11-24 01:15:01.000000000 -0800

@@ -2951,7 +2539,6 @@ static struct pci_driver mlx4_driver = {
    .name       = DRV_NAME,
    .id_table   = mlx4_pci_table,
    .probe      = mlx4_init_one,
-   .shutdown   = mlx4_unload_one,
    .remove     = mlx4_remove_one,
    .err_handler    = &mlx4_err_handler,
 };
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.openfabrics.org/pipermail/users/attachments/20160222/ee1ddd85/attachment.html>