....

Feb 12 22:07:56 mellanox-2 kernel: INFO: task bigfile:28246 blocked for more than 480 seconds.
Feb 12 22:07:56 mellanox-2 kernel: "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
Feb 12 22:07:56 mellanox-2 kernel: bigfile       D ffff880028057780     0 28246  27786
Feb 12 22:07:56 mellanox-2 kernel:  ffffffff805b5360 0000000000000082 0000000000000001 ffffffffa03184b1
Feb 12 22:07:56 mellanox-2 kernel:  ffffffffa0344f70 ffff88013a92f120 ffff88013a92f480 0000000100638e67
Feb 12 22:07:56 mellanox-2 kernel:  ffff880119557c98 ffff8801024381e8 ffff880139999740 ffffffffa03102cf
Feb 12 22:07:56 mellanox-2 kernel: Call Trace:
Feb 12 22:07:56 mellanox-2 kernel:  [<ffffffffa03184b1>] nfs_readdata_alloc+0x30/0x92 [nfs]
Feb 12 22:07:56 mellanox-2 kernel:  [<ffffffffa03102cf>] __put_nfs_open_context+0x23/0x7f [nfs]
Feb 12 22:07:56 mellanox-2 kernel:  [<ffffffff8024fc3e>] getnstimeofday+0x57/0xb6
Feb 12 22:07:56 mellanox-2 kernel:  [<ffffffff80273472>] sync_page+0x0/0x40
Feb 12 22:07:56 mellanox-2 kernel:  [<ffffffff80498e4d>] io_schedule+0x5d/0x9e
Feb 12 22:07:56 mellanox-2 kernel:  [<ffffffff802734ae>] sync_page+0x3c/0x40
Feb 12 22:07:56 mellanox-2 kernel:  [<ffffffff80498fc7>] __wait_on_bit_lock+0x3e/0x84
Feb 12 22:07:56 mellanox-2 kernel:  [<ffffffff8027345e>] __lock_page+0x5e/0x64
Feb 12 22:07:56 mellanox-2 kernel:  [<ffffffff8024a4a8>] wake_bit_function+0x0/0x23
Feb 12 22:07:56 mellanox-2 kernel:  [<ffffffff80273502>] find_lock_page+0x35/0x55
Feb 12 22:07:56 mellanox-2 kernel:  [<ffffffff8027505c>] filemap_fault+0x169/0x335
Feb 12 22:07:56 mellanox-2 kernel:  [<ffffffff803360d4>] prio_tree_insert+0x23/0x22a
Feb 12 22:07:56 mellanox-2 kernel:  [<ffffffff80283ed6>] __do_fault+0x51/0x3bd
Feb 12 22:07:56 mellanox-2 kernel:  [<ffffffff80281a68>] vma_prio_tree_insert+0x20/0x39
Feb 12 22:07:56 mellanox-2 kernel:  [<ffffffff80285e40>] handle_mm_fault+0x328/0x6f2
Feb 12 22:07:56 mellanox-2 kernel:  [<ffffffff8049c297>] do_page_fault+0x3c7/0x74e
Feb 12 22:07:56 mellanox-2 kernel:  [<ffffffff8049a41f>] page_fault+0x1f/0x30
Feb 12 22:09:02 mellanox-2 kernel: INFO: task tlock64:29564 blocked for more than 480 seconds.
Feb 12 22:09:02 mellanox-2 kernel: "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
Feb 12 22:09:02 mellanox-2 kernel: tlock64       D ffff880028097780     0 29564  29551
Feb 12 22:09:02 mellanox-2 kernel:  ffff880128488de0 0000000000000082 ffff88013d5a8b80 ffff8801111cbd18
Feb 12 22:09:02 mellanox-2 kernel:  0000000000000000 ffff88013a985220 ffff88013a985580 ffff8801111cbd90
Feb 12 22:09:02 mellanox-2 kernel:  ffff8801024a4840 ffffffff80273220 ffff88010fc18160 000000000000000e
Feb 12 22:09:02 mellanox-2 kernel: Call Trace:
Feb 12 22:09:02 mellanox-2 kernel:  [<ffffffff80273220>] find_get_pages_tag+0x43/0xdd
Feb 12 22:09:02 mellanox-2 kernel:  [<ffffffffa0319049>] nfs_block_sillyrename+0x82/0x99 [nfs]
Feb 12 22:09:02 mellanox-2 kernel:  [<ffffffff8024a47a>] autoremove_wake_function+0x0/0x2e
Feb 12 22:09:02 mellanox-2 kernel:  [<ffffffffa030b9c5>] nfs_lookup+0xd2/0x170 [nfs]
Feb 12 22:09:02 mellanox-2 kernel:  [<ffffffffa028c1e6>] put_rpccred+0x3b/0xf3 [sunrpc]
Feb 12 22:09:02 mellanox-2 kernel:  [<ffffffffa030c399>] nfs_permission+0xe6/0x16c [nfs]
Feb 12 22:09:02 mellanox-2 kernel:  [<ffffffff8022d9a4>] update_curr+0x6f/0xaa
Feb 12 22:09:02 mellanox-2 kernel:  [<ffffffff80235a99>] dequeue_task_fair+0x93/0x105
Feb 12 22:09:02 mellanox-2 kernel:  [<ffffffffa030c0c8>] nfs_do_access+0xd0/0x2bb [nfs]
Feb 12 22:09:02 mellanox-2 kernel:  [<ffffffff802ada87>] d_alloc+0x15f/0x190
Feb 12 22:09:02 mellanox-2 kernel:  [<ffffffff802a6114>] __lookup_hash+0xf9/0x11d
Feb 12 22:09:02 mellanox-2 kernel:  [<ffffffff802a8aff>] do_unlinkat+0x59/0x14e
Feb 12 22:09:02 mellanox-2 kernel:  [<ffffffff80263e08>] audit_syscall_entry+0x151/0x17c
Feb 12 22:09:02 mellanox-2 kernel:  [<ffffffff8020bedb>] system_call_fastpath+0x16/0x1b
Feb 12 22:16:03 mellanox-2 hald: mounted /dev/scd0 on behalf of uid 0
Feb 12 22:16:48 mellanox-2 kernel: INFO: task bigfile:28246 blocked for more than 480 seconds.
Feb 12 22:16:48 mellanox-2 kernel: "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
Feb 12 22:16:48 mellanox-2 kernel: bigfile       D ffff880028057780     0 28246  27786
Feb 12 22:16:48 mellanox-2 kernel:  ffffffff805b5360 0000000000000082 0000000000000001 ffffffffa03184b1
Feb 12 22:16:48 mellanox-2 kernel:  ffffffffa0344f70 ffff88013a92f120 ffff88013a92f480 0000000100638e67
Feb 12 22:16:48 mellanox-2 kernel:  ffff880119557c98 ffff8801024381e8 ffff880139999740 ffffffffa03102cf
Feb 12 22:16:48 mellanox-2 kernel: Call Trace:
Feb 12 22:16:48 mellanox-2 kernel:  [<ffffffffa03184b1>] nfs_readdata_alloc+0x30/0x92 [nfs]
Feb 12 22:16:48 mellanox-2 kernel:  [<ffffffffa03102cf>] __put_nfs_open_context+0x23/0x7f [nfs]
Feb 12 22:16:48 mellanox-2 kernel:  [<ffffffff8024fc3e>] getnstimeofday+0x57/0xb6
Feb 12 22:16:48 mellanox-2 kernel:  [<ffffffff80273472>] sync_page+0x0/0x40
Feb 12 22:16:48 mellanox-2 kernel:  [<ffffffff80498e4d>] io_schedule+0x5d/0x9e
Feb 12 22:16:48 mellanox-2 kernel:  [<ffffffff802734ae>] sync_page+0x3c/0x40
Feb 12 22:16:48 mellanox-2 kernel:  [<ffffffff80498fc7>] __wait_on_bit_lock+0x3e/0x84
Feb 12 22:16:48 mellanox-2 kernel:  [<ffffffff8027345e>] __lock_page+0x5e/0x64
Feb 12 22:16:48 mellanox-2 kernel:  [<ffffffff8024a4a8>] wake_bit_function+0x0/0x23
Feb 12 22:16:48 mellanox-2 kernel:  [<ffffffff80273502>] find_lock_page+0x35/0x55
Feb 12 22:16:48 mellanox-2 kernel:  [<ffffffff8027505c>] filemap_fault+0x169/0x335
Feb 12 22:16:48 mellanox-2 kernel:  [<ffffffff803360d4>] prio_tree_insert+0x23/0x22a
Feb 12 22:16:48 mellanox-2 kernel:  [<ffffffff80283ed6>] __do_fault+0x51/0x3bd
Feb 12 22:16:49 mellanox-2 kernel:  [<ffffffff80281a68>] vma_prio_tree_insert+0x20/0x39
Feb 12 22:16:49 mellanox-2 kernel:  [<ffffffff80285e40>] handle_mm_fault+0x328/0x6f2
Feb 12 22:16:49 mellanox-2 kernel:  [<ffffffff8049c297>] do_page_fault+0x3c7/0x74e
Feb 12 22:16:49 mellanox-2 kernel:  [<ffffffff8049a41f>] page_fault+0x1f/0x30
Feb 12 22:17:55 mellanox-2 kernel: INFO: task tlock64:29564 blocked for more than 480 seconds.
Feb 12 22:17:55 mellanox-2 kernel: "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
Feb 12 22:17:55 mellanox-2 kernel: tlock64       D ffff880028097780     0 29564  29551
Feb 12 22:17:55 mellanox-2 kernel:  ffff880128488de0 0000000000000082 ffff88013d5a8b80 ffff8801111cbd18
Feb 12 22:17:55 mellanox-2 kernel:  0000000000000000 ffff88013a985220 ffff88013a985580 ffff8801111cbd90
Feb 12 22:17:55 mellanox-2 kernel:  ffff8801024a4840 ffffffff80273220 ffff88010fc18160 000000000000000e
Feb 12 22:17:55 mellanox-2 kernel: Call Trace:
Feb 12 22:17:55 mellanox-2 kernel:  [<ffffffff80273220>] find_get_pages_tag+0x43/0xdd
Feb 12 22:17:55 mellanox-2 kernel:  [<ffffffffa0319049>] nfs_block_sillyrename+0x82/0x99 [nfs]
Feb 12 22:17:55 mellanox-2 kernel:  [<ffffffff8024a47a>] autoremove_wake_function+0x0/0x2e
Feb 12 22:17:55 mellanox-2 kernel:  [<ffffffffa030b9c5>] nfs_lookup+0xd2/0x170 [nfs]
Feb 12 22:17:55 mellanox-2 kernel:  [<ffffffffa028c1e6>] put_rpccred+0x3b/0xf3 [sunrpc]
Feb 12 22:17:55 mellanox-2 kernel:  [<ffffffffa030c399>] nfs_permission+0xe6/0x16c [nfs]
Feb 12 22:17:55 mellanox-2 kernel:  [<ffffffff8022d9a4>] update_curr+0x6f/0xaa
Feb 12 22:17:55 mellanox-2 kernel:  [<ffffffff80235a99>] dequeue_task_fair+0x93/0x105
Feb 12 22:17:55 mellanox-2 kernel:  [<ffffffffa030c0c8>] nfs_do_access+0xd0/0x2bb [nfs]
Feb 12 22:17:55 mellanox-2 kernel:  [<ffffffff802ada87>] d_alloc+0x15f/0x190
Feb 12 22:17:55 mellanox-2 kernel:  [<ffffffff802a6114>] __lookup_hash+0xf9/0x11d
Feb 12 22:17:55 mellanox-2 kernel:  [<ffffffff802a8aff>] do_unlinkat+0x59/0x14e
Feb 12 22:17:55 mellanox-2 kernel:  [<ffffffff80263e08>] audit_syscall_entry+0x151/0x17c
Feb 12 22:17:55 mellanox-2 kernel:  [<ffffffff8020bedb>] system_call_fastpath+0x16/0x1b
Feb 12 22:21:03 mellanox-2 kernel: rpcrdma: connection to 11.20.1.7:20049 closed (-103)
Feb 12 22:25:41 mellanox-2 kernel: INFO: task bigfile:28246 blocked for more than 480 seconds.
Feb 12 22:25:41 mellanox-2 kernel: "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
Feb 12 22:25:41 mellanox-2 kernel: bigfile       D ffff880028057780     0 28246  27786
Feb 12 22:25:41 mellanox-2 kernel:  ffffffff805b5360 0000000000000082 0000000000000001 ffffffffa03184b1
Feb 12 22:25:41 mellanox-2 kernel:  ffffffffa0344f70 ffff88013a92f120 ffff88013a92f480 0000000100638e67
Feb 12 22:25:41 mellanox-2 kernel:  ffff880119557c98 ffff8801024381e8 ffff880139999740 ffffffffa03102cf
Feb 12 22:25:41 mellanox-2 kernel: Call Trace:
Feb 12 22:25:41 mellanox-2 kernel:  [<ffffffffa03184b1>] nfs_readdata_alloc+0x30/0x92 [nfs]
Feb 12 22:25:41 mellanox-2 kernel:  [<ffffffffa03102cf>] __put_nfs_open_context+0x23/0x7f [nfs]
Feb 12 22:25:41 mellanox-2 kernel:  [<ffffffff8024fc3e>] getnstimeofday+0x57/0xb6
Feb 12 22:25:41 mellanox-2 kernel:  [<ffffffff80273472>] sync_page+0x0/0x40
Feb 12 22:25:41 mellanox-2 kernel:  [<ffffffff80498e4d>] io_schedule+0x5d/0x9e
Feb 12 22:25:41 mellanox-2 kernel:  [<ffffffff802734ae>] sync_page+0x3c/0x40
Feb 12 22:25:41 mellanox-2 kernel:  [<ffffffff80498fc7>] __wait_on_bit_lock+0x3e/0x84
Feb 12 22:25:41 mellanox-2 kernel:  [<ffffffff8027345e>] __lock_page+0x5e/0x64
Feb 12 22:25:41 mellanox-2 kernel:  [<ffffffff8024a4a8>] wake_bit_function+0x0/0x23
Feb 12 22:25:41 mellanox-2 kernel:  [<ffffffff80273502>] find_lock_page+0x35/0x55
Feb 12 22:25:41 mellanox-2 kernel:  [<ffffffff8027505c>] filemap_fault+0x169/0x335
Feb 12 22:25:41 mellanox-2 kernel:  [<ffffffff803360d4>] prio_tree_insert+0x23/0x22a
Feb 12 22:25:41 mellanox-2 kernel:  [<ffffffff80283ed6>] __do_fault+0x51/0x3bd
Feb 12 22:25:41 mellanox-2 kernel:  [<ffffffff80281a68>] vma_prio_tree_insert+0x20/0x39
Feb 12 22:25:41 mellanox-2 kernel:  [<ffffffff80285e40>] handle_mm_fault+0x328/0x6f2
Feb 12 22:25:41 mellanox-2 kernel:  [<ffffffff8049c297>] do_page_fault+0x3c7/0x74e
Feb 12 22:25:41 mellanox-2 kernel:  [<ffffffff8049a41f>] page_fault+0x1f/0x30
Feb 12 22:26:48 mellanox-2 kernel: INFO: task tlock64:29564 blocked for more than 480 seconds.
Feb 12 22:26:48 mellanox-2 kernel: "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
Feb 12 22:26:48 mellanox-2 kernel: tlock64       D ffff880028097780     0 29564  29551
Feb 12 22:26:48 mellanox-2 kernel:  ffff880128488de0 0000000000000082 ffff88013d5a8b80 ffff8801111cbd18
Feb 12 22:26:48 mellanox-2 kernel:  0000000000000000 ffff88013a985220 ffff88013a985580 ffff8801111cbd90
Feb 12 22:26:48 mellanox-2 kernel:  ffff8801024a4840 ffffffff80273220 ffff88010fc18160 000000000000000e
Feb 12 22:26:48 mellanox-2 kernel: Call Trace:
Feb 12 22:26:48 mellanox-2 kernel:  [<ffffffff80273220>] find_get_pages_tag+0x43/0xdd
Feb 12 22:26:48 mellanox-2 kernel:  [<ffffffffa0319049>] nfs_block_sillyrename+0x82/0x99 [nfs]
Feb 12 22:26:48 mellanox-2 kernel:  [<ffffffff8024a47a>] autoremove_wake_function+0x0/0x2e
Feb 12 22:26:48 mellanox-2 kernel:  [<ffffffffa030b9c5>] nfs_lookup+0xd2/0x170 [nfs]
Feb 12 22:26:48 mellanox-2 kernel:  [<ffffffffa028c1e6>] put_rpccred+0x3b/0xf3 [sunrpc]
Feb 12 22:26:48 mellanox-2 kernel:  [<ffffffffa030c399>] nfs_permission+0xe6/0x16c [nfs]
Feb 12 22:26:48 mellanox-2 kernel:  [<ffffffff8022d9a4>] update_curr+0x6f/0xaa
Feb 12 22:26:48 mellanox-2 kernel:  [<ffffffff80235a99>] dequeue_task_fair+0x93/0x105
Feb 12 22:26:48 mellanox-2 kernel:  [<ffffffffa030c0c8>] nfs_do_access+0xd0/0x2bb [nfs]
Feb 12 22:26:48 mellanox-2 kernel:  [<ffffffff802ada87>] d_alloc+0x15f/0x190
Feb 12 22:26:48 mellanox-2 kernel:  [<ffffffff802a6114>] __lookup_hash+0xf9/0x11d
Feb 12 22:26:48 mellanox-2 kernel:  [<ffffffff802a8aff>] do_unlinkat+0x59/0x14e
Feb 12 22:26:48 mellanox-2 kernel:  [<ffffffff80263e08>] audit_syscall_entry+0x151/0x17c
Feb 12 22:26:48 mellanox-2 kernel:  [<ffffffff8020bedb>] system_call_fastpath+0x16/0x1b
Feb 12 22:28:16 mellanox-2 kernel: RPC: failed to contact local rpcbind server (errno 5). 
