[openib-general] uAT issues after SM node bounced
Sean Hefty
sean.hefty at intel.com
Fri Aug 19 14:36:44 PDT 2005
>Sean and I are seeing some issues with uAT when our dedicated SM node
>bounces. Sean saw a kernel oops (I will let him send output) and I see
>the following console message with my failing ib_at_ips_by_gid requests :
Here's the bug check that I saw. I haven't spent anytime debugging this yet.
- Sean
Aug 19 11:49:49 mshefty-linux1 kernel: ib_at: ib_dev_ats_op: dev (f8e68f40) ib0
already has pending op 2
Aug 19 11:50:00 mshefty-linux1 kernel: ib0: no IPv6 routers present
Aug 19 11:53:49 mshefty-linux1 kernel: Debug: sleeping function called from
invalid context at mm/rmap.c:86
Aug 19 11:53:49 mshefty-linux1 kernel: in_atomic():0, irqs_disabled():1
Aug 19 11:53:49 mshefty-linux1 kernel: [dump_stack+21/32] dump_stack+0x15/0x20
Aug 19 11:53:49 mshefty-linux1 kernel: [<c0105595>] dump_stack+0x15/0x20
Aug 19 11:53:49 mshefty-linux1 kernel: [__might_sleep+150/176]
__might_sleep+0x96/0xb0
Aug 19 11:53:49 mshefty-linux1 kernel: [<c011d246>] __might_sleep+0x96/0xb0
Aug 19 11:53:49 mshefty-linux1 kernel: [anon_vma_prepare+29/224]
anon_vma_prepare+0x1d/0xe0
Aug 19 11:53:49 mshefty-linux1 kernel: [<c0154b7d>] anon_vma_prepare+0x1d/0xe0
Aug 19 11:53:49 mshefty-linux1 kernel: [expand_stack+16/128]
expand_stack+0x10/0x80
Aug 19 11:53:49 mshefty-linux1 kernel: [<c0152ea0>] expand_stack+0x10/0x80
Aug 19 11:53:49 mshefty-linux1 kernel: [do_page_fault+373/1648]
do_page_fault+0x175/0x670
Aug 19 11:53:49 mshefty-linux1 kernel: [<c0117cd5>] do_page_fault+0x175/0x670
Aug 19 11:53:49 mshefty-linux1 kernel: [error_code+79/96] error_code+0x4f/0x60
Aug 19 11:53:49 mshefty-linux1 kernel: [<c010514f>] error_code+0x4f/0x60
Aug 19 11:53:49 mshefty-linux1 kernel: [pg0+949458380/1069249536]
ib_get_client_data+0x1c/0x60 [ib_core]
Aug 19 11:53:49 mshefty-linux1 kernel: [<f8dc05cc>]
ib_get_client_data+0x1c/0x60 [ib_core]
Aug 19 11:53:49 mshefty-linux1 kernel: [pg0+949630151/1069249536]
ib_sa_path_rec_get+0x27/0x170 [ib_sa]
Aug 19 11:53:49 mshefty-linux1 kernel: [<f8dea4c7>]
ib_sa_path_rec_get+0x27/0x170 [ib_sa]
Aug 19 11:53:49 mshefty-linux1 kernel: [pg0+950134476/1069249536]
resolve_path+0x4c/0x100 [ib_at]
Aug 19 11:53:49 mshefty-linux1 kernel: [<f8e656cc>] resolve_path+0x4c/0x100
[ib_at]
Aug 19 11:53:49 mshefty-linux1 kernel: [pg0+950135658/1069249536]
ib_at_paths_by_route+0xba/0xf0 [ib_at]
Aug 19 11:53:49 mshefty-linux1 kernel: [<f8e65b6a>]
ib_at_paths_by_route+0xba/0xf0 [ib_at]
Aug 19 11:53:49 mshefty-linux1 kernel: [pg0+949733048/1069249536]
ib_uat_paths_by_route+0xf8/0x1e0 [ib_uat]
Aug 19 11:53:49 mshefty-linux1 kernel: [<f8e036b8>]
ib_uat_paths_by_route+0xf8/0x1e0 [ib_uat]
Aug 19 11:53:49 mshefty-linux1 kernel: [pg0+949734940/1069249536]
ib_uat_write+0x9c/0xb0 [ib_uat]
Aug 19 11:53:49 mshefty-linux1 kernel: [<f8e03e1c>] ib_uat_write+0x9c/0xb0
[ib_uat]
Aug 19 11:53:49 mshefty-linux1 kernel: [vfs_write+176/272] vfs_write+0xb0/0x110
Aug 19 11:53:49 mshefty-linux1 kernel: [<c015e640>] vfs_write+0xb0/0x110
Aug 19 11:53:49 mshefty-linux1 kernel: [sys_write+59/112] sys_write+0x3b/0x70
Aug 19 11:53:49 mshefty-linux1 kernel: [<c015e74b>] sys_write+0x3b/0x70
Aug 19 11:53:49 mshefty-linux1 kernel: [sysenter_past_esp+84/121]
sysenter_past_esp+0x54/0x79
Aug 19 11:53:49 mshefty-linux1 kernel: [<c0103fcb>] sysenter_past_esp+0x54/0x79
Aug 19 11:53:49 mshefty-linux1 kernel: eip: f8dc05cc
Aug 19 11:53:49 mshefty-linux1 kernel: ------------[ cut here ]------------
Aug 19 11:53:49 mshefty-linux1 kernel: kernel BUG at include/asm/spinlock.h:149!
Aug 19 11:53:49 mshefty-linux1 kernel: invalid operand: 0000 [#1]
Aug 19 11:53:49 mshefty-linux1 kernel: SMP
Aug 19 11:53:49 mshefty-linux1 kernel: Modules linked in: ib_madeye ib_ipoib
ib_uat ib_at ib_ucm ib_cm ib_sa ib_uverbs ib_mthca ib_mad ib_core edd joydev st
sr_mod ide_cd cdrom nvram usbserial parport_pc lp parport thermal processor fan
button battery ac ipv6 af_packet e1000 i2c_i801 i2c_core hw_random uhci_hcd
usbcore evdev reiserfs aic7xxx scsi_transport_spi sd_mod scsi_mod
Aug 19 11:53:49 mshefty-linux1 kernel: CPU: 0
Aug 19 11:53:49 mshefty-linux1 kernel: EIP: 0060:[_spin_lock_irqsave+68/80]
Not tainted VLI
Aug 19 11:53:49 mshefty-linux1 kernel: EIP: 0060:[<c0307584>] Not tainted
VLI
Aug 19 11:53:49 mshefty-linux1 kernel: EFLAGS: 00010046 (2.6.12.1)
Aug 19 11:53:49 mshefty-linux1 kernel: EIP is at _spin_lock_irqsave+0x44/0x50
Aug 19 11:53:49 mshefty-linux1 kernel: eax: c031f296 ebx: 00000286 ecx:
c035a344 edx: f8dc05cc
Aug 19 11:53:49 mshefty-linux1 kernel: esi: 5a5a5abe edi: 5a5a5abe ebp:
f2f55e10 esp: f2f55e08
Aug 19 11:53:49 mshefty-linux1 kernel: ds: 007b es: 007b ss: 0068
Aug 19 11:53:49 mshefty-linux1 kernel: Process lt-ucmpost (pid: 8350,
threadinfo=f2f54000 task=f7762a60)
Aug 19 11:53:49 mshefty-linux1 kernel: Stack: 5a5a5a5a f8decf6c f2f55e28
f8dc05cc 00000000 0c30005a 000000d0 f2f55eb0
Aug 19 11:53:49 mshefty-linux1 kernel: f2f55e4c f8dea4c7 c0148b61
00000000 0c300000 f2f55e70 f20cdbfc f2f55e70
Aug 19 11:53:49 mshefty-linux1 kernel: f2f55eb0 f2f55ebc f8e656cc
00000000 0c300000 00000064 000000d0 f8e652c0
Aug 19 11:53:49 mshefty-linux1 kernel: Call Trace:
Aug 19 11:53:49 mshefty-linux1 kernel: [show_stack+155/176]
show_stack+0x9b/0xb0
Aug 19 11:53:49 mshefty-linux1 kernel: [<c010556b>] show_stack+0x9b/0xb0
Aug 19 11:53:49 mshefty-linux1 kernel: [show_registers+287/400]
show_registers+0x11f/0x190
Aug 19 11:53:49 mshefty-linux1 kernel: [<c01056bf>] show_registers+0x11f/0x190
Aug 19 11:53:49 mshefty-linux1 kernel: [die+227/352] die+0xe3/0x160
Aug 19 11:53:49 mshefty-linux1 kernel: [<c01058a3>] die+0xe3/0x160
Aug 19 11:53:49 mshefty-linux1 kernel: [do_invalid_op+149/160]
do_invalid_op+0x95/0xa0
Aug 19 11:53:49 mshefty-linux1 kernel: [<c0105c55>] do_invalid_op+0x95/0xa0
Aug 19 11:53:49 mshefty-linux1 kernel: [error_code+79/96] error_code+0x4f/0x60
Aug 19 11:53:49 mshefty-linux1 kernel: [<c010514f>] error_code+0x4f/0x60
Aug 19 11:53:49 mshefty-linux1 kernel: [pg0+949458380/1069249536]
ib_get_client_data+0x1c/0x60 [ib_core]
Aug 19 11:53:49 mshefty-linux1 kernel: [<f8dc05cc>]
ib_get_client_data+0x1c/0x60 [ib_core]
Aug 19 11:53:49 mshefty-linux1 kernel: [pg0+949630151/1069249536]
ib_sa_path_rec_get+0x27/0x170 [ib_sa]
Aug 19 11:53:49 mshefty-linux1 kernel: [<f8dea4c7>]
ib_sa_path_rec_get+0x27/0x170 [ib_sa]
Aug 19 11:53:49 mshefty-linux1 kernel: [pg0+950134476/1069249536]
resolve_path+0x4c/0x100 [ib_at]
Aug 19 11:53:49 mshefty-linux1 kernel: [<f8e656cc>] resolve_path+0x4c/0x100
[ib_at]
Aug 19 11:53:49 mshefty-linux1 kernel: [pg0+950135658/1069249536]
ib_at_paths_by_route+0xba/0xf0 [ib_at]
Aug 19 11:53:49 mshefty-linux1 kernel: [<f8e65b6a>]
ib_at_paths_by_route+0xba/0xf0 [ib_at]
Aug 19 11:53:49 mshefty-linux1 kernel: [pg0+949733048/1069249536]
ib_uat_paths_by_route+0xf8/0x1e0 [ib_uat]
Aug 19 11:53:49 mshefty-linux1 kernel: [<f8e036b8>]
ib_uat_paths_by_route+0xf8/0x1e0 [ib_uat]
Aug 19 11:53:49 mshefty-linux1 kernel: [pg0+949734940/1069249536]
ib_uat_write+0x9c/0xb0 [ib_uat]
Aug 19 11:53:49 mshefty-linux1 kernel: [<f8e03e1c>] ib_uat_write+0x9c/0xb0
[ib_uat]
Aug 19 11:53:49 mshefty-linux1 kernel: [vfs_write+176/272] vfs_write+0xb0/0x110
Aug 19 11:53:49 mshefty-linux1 kernel: [<c015e640>] vfs_write+0xb0/0x110
Aug 19 11:53:49 mshefty-linux1 kernel: [sys_write+59/112] sys_write+0x3b/0x70
Aug 19 11:53:49 mshefty-linux1 kernel: [<c015e74b>] sys_write+0x3b/0x70
Aug 19 11:53:49 mshefty-linux1 kernel: [sysenter_past_esp+84/121]
sysenter_past_esp+0x54/0x79
Aug 19 11:53:49 mshefty-linux1 kernel: [<c0103fcb>] sysenter_past_esp+0x54/0x79
Aug 19 11:53:49 mshefty-linux1 kernel: Code: c3 00 02 00 00 74 01 fb f3 90 80 3e
00 7e f9 fa eb e8 8d 65 f8 89 d8 5b 5e 5d c3 8b 4d 04 51 68 96 f2 31 c0 e8 3e 87
e1 ff 58 5a <0f> 0b 95 00 8d ea 31 c0 eb c5 89 f6 55 89 e5 53 89 c3 fa 81 78
Aug 19 11:54:04 mshefty-linux1 kernel: <3>kfree_debugcheck: bad ptr f8a560f4h.
Aug 19 11:54:04 mshefty-linux1 kernel: ------------[ cut here ]------------
Aug 19 11:54:04 mshefty-linux1 kernel: kernel BUG at mm/slab.c:1892!
Aug 19 11:54:04 mshefty-linux1 kernel: invalid operand: 0000 [#2]
Aug 19 11:54:04 mshefty-linux1 kernel: SMP
Aug 19 11:54:04 mshefty-linux1 kernel: Modules linked in: ib_madeye ib_ipoib
ib_uat ib_at ib_ucm ib_cm ib_sa ib_uverbs ib_mthca ib_mad ib_core edd joydev st
sr_mod ide_cd cdrom nvram usbserial parport_pc lp parport thermal processor fan
button battery ac ipv6 af_packet e1000 i2c_i801 i2c_core hw_random uhci_hcd
usbcore evdev reiserfs aic7xxx scsi_transport_spi sd_mod scsi_mod
Aug 19 11:54:04 mshefty-linux1 kernel: CPU: 1
Aug 19 11:54:04 mshefty-linux1 kernel: EIP: 0060:[kfree_debugcheck+107/128]
Not tainted VLI
Aug 19 11:54:04 mshefty-linux1 kernel: EIP: 0060:[<c0148e2b>] Not tainted
VLI
Aug 19 11:54:04 mshefty-linux1 kernel: EFLAGS: 00010006 (2.6.12.1)
Aug 19 11:54:04 mshefty-linux1 kernel: EIP is at kfree_debugcheck+0x6b/0x80
Aug 19 11:54:04 mshefty-linux1 kernel: eax: 00000028 ebx: c1714ac0 ecx:
c035a30c edx: 00000000
Aug 19 11:54:04 mshefty-linux1 kernel: esi: f8a560f4 edi: f20cdc50 ebp:
f2557ef4 esp: f2557ee4
Aug 19 11:54:04 mshefty-linux1 kernel: ds: 007b es: 007b ss: 0068
Aug 19 11:54:04 mshefty-linux1 kernel: Process ib_at_wq/1 (pid: 8195,
threadinfo=f2556000 task=f6a22a60)
Aug 19 11:54:04 mshefty-linux1 kernel: Stack: c032162c f8a560f4 f8a560f4
43062abd f2557f0c c0149af7 00000282 f20c9f18
Aug 19 11:54:04 mshefty-linux1 kernel: 43062abd f20cdc50 f2557f2c
f8e03372 00000002 00000000 00000000 f20cdc2c
Aug 19 11:54:04 mshefty-linux1 kernel: f253f23c f20cdc50 f2557f3c
f8e033f2 f20c9f18 ffffff92 f2557f4c f8e64af5
Aug 19 11:54:04 mshefty-linux1 kernel: Call Trace:
Aug 19 11:54:04 mshefty-linux1 kernel: [show_stack+155/176]
show_stack+0x9b/0xb0
Aug 19 11:54:04 mshefty-linux1 kernel: [<c010556b>] show_stack+0x9b/0xb0
Aug 19 11:54:04 mshefty-linux1 kernel: [show_registers+287/400]
show_registers+0x11f/0x190
Aug 19 11:54:04 mshefty-linux1 kernel: [<c01056bf>] show_registers+0x11f/0x190
Aug 19 11:54:04 mshefty-linux1 kernel: [die+227/352] die+0xe3/0x160
Aug 19 11:54:04 mshefty-linux1 kernel: [<c01058a3>] die+0xe3/0x160
Aug 19 11:54:04 mshefty-linux1 kernel: [do_invalid_op+149/160]
do_invalid_op+0x95/0xa0
Aug 19 11:54:04 mshefty-linux1 kernel: [<c0105c55>] do_invalid_op+0x95/0xa0
Aug 19 11:54:04 mshefty-linux1 kernel: [error_code+79/96] error_code+0x4f/0x60
Aug 19 11:54:04 mshefty-linux1 kernel: [<c010514f>] error_code+0x4f/0x60
Aug 19 11:54:04 mshefty-linux1 kernel: [kfree+23/144] kfree+0x17/0x90
Aug 19 11:54:04 mshefty-linux1 kernel: [<c0149af7>] kfree+0x17/0x90
Aug 19 11:54:04 mshefty-linux1 kernel: [pg0+949732210/1069249536]
ib_uat_callback+0x42/0x70 [ib_uat]
Aug 19 11:54:04 mshefty-linux1 kernel: [<f8e03372>] ib_uat_callback+0x42/0x70
[ib_uat]
Aug 19 11:54:04 mshefty-linux1 kernel: [pg0+949732338/1069249536]
ib_uat_path_callback+0x12/0x20 [ib_uat]
Aug 19 11:54:04 mshefty-linux1 kernel: [<f8e033f2>]
ib_uat_path_callback+0x12/0x20 [ib_uat]
Aug 19 11:54:04 mshefty-linux1 kernel: [pg0+950131445/1069249536]
req_comp_work+0x15/0x30 [ib_at]
Aug 19 11:54:04 mshefty-linux1 kernel: [<f8e64af5>] req_comp_work+0x15/0x30
[ib_at]
Aug 19 11:54:04 mshefty-linux1 kernel: [worker_thread+387/528]
worker_thread+0x183/0x210
Aug 19 11:54:04 mshefty-linux1 kernel: [<c012f173>] worker_thread+0x183/0x210
Aug 19 11:54:04 mshefty-linux1 kernel: [kthread+139/192] kthread+0x8b/0xc0
Aug 19 11:54:04 mshefty-linux1 kernel: [<c013315b>] kthread+0x8b/0xc0
Aug 19 11:54:04 mshefty-linux1 kernel: [kernel_thread_helper+5/12]
kernel_thread_helper+0x5/0xc
Aug 19 11:54:04 mshefty-linux1 kernel: [<c0102419>]
kernel_thread_helper+0x5/0xc
Aug 19 11:54:04 mshefty-linux1 kernel: Code: 07 7d 0b 32 c0 58 5a c1 e3 05 a1 10
67 42 c0 01 c3 8b 03 a9 80 00 00 00 75 d1 8d b6 00 00 00 00 56 68 2c 16 32 c0 e8
95 6e fd ff <0f> 0b 64 07 7d 0b 32 c0 59 5b 8d 65 f8 5b 5e 5d c3 8d 74 26 00
More information about the general
mailing list