[ofa-general] SRP/mlx4 interrupts throttling performance

[Vladislav Bolkhovitin]

Cameron Harr wrote:
> Vladislav Bolkhovitin wrote:
>>> ** Sometimes the benchmark "zombied" (process doing no work, but 
>>> process can't be killed) after running a certain amount of time. 
>>> However, it wasn't repeatable in a reliable way, so I mark that this 
>>> particular run has zombied before.
>> That means that there is a bug somewhere. Usually such bugs are found 
>> in few hours of code auditing (srpt driver is pretty simple) or by 
>> using kernel debug facilities (example diff to .config attached). I 
>> personally always prefer put my effort on fixing real things, not 
>> inventing various workarounds, like srpt_thread in this case.
>>
>> So I would:
>>
>>   1. Completely remove srpt thread and all related code. It doesn't do
>> anything, which can't be done in SIRQ context (tasklet)
>>
>>   2. Audit the code to check if it does any action, which it shouldn't 
>> do on SIRQ and fix it. This step isn't required, but usually it saves 
>> a lot of time of puzzled debugging in the future.
>>
>>   3. Change in srpt_handle_rdma_comp() and  srpt_handle_new_iu()
>> SCST_CONTEXT_THREAD to SCST_CONTEXT_DIRECT_ATOMIC.
> 
> I also changed it in srpt_handle_err_comp()
>> Then I would run the problematic tests (heavy tpc-h workload, e.g.) on 
>> debug kernel and fix found problems.
>>
>> Anyway, Cameron, can you get the latest code from SCST trunk and try 
>> with it? It was recently updated. Also please add the case with 
>> changes from (3) above.
> This is all with version 1.0.1 of SCST (v532).
> In my fio test, I do runs with srpt thread=1 and then =0. When it was 
> set to zero during the test, I got many errors printed out by FIO, and 
> the target eventually crashed. This is the first part of a long call trace.
> 
> NMI Watchdog detected LOCKUP on CPU 0
> CPU 0
> Modules linked in: ib_srpt(U) scst_vdisk(U) scst(U) fio_driver(PU) 
> fio_port(PU) autofs4 hidp rfcomm l2cap bluetooth sunrpc ib_ipoib mlx4_ib 
> ib_cm ib_sa ib_mad ib_core ipv6 xfrm_nalgo crypto_api nls_utf8 hfsplus 
> dm_mirror dm_multipath dm_mod video sbs backlight i2c_ec button battery 
> asus_acpi acpi_memhotplug ac parport_pc lp parport i2c_i801 shpchp 
> i2c_core e1000e mlx4_core i5000_edac edac_mc pcspkr ata_piix libata 
> sd_mod scsi_mod ext3 jbd uhci_hcd ohci_hcd ehci_hcd
> Pid: 25732, comm: scsi_tgt0 Tainted: P      2.6.18-92.1.13.el5 #1
> RIP: 0010:[<ffffffff80064bcb>]  [<ffffffff80064bcb>] 
> .text.lock.spinlock+0x29/0x30
> RSP: 0018:ffffffff80418a88  EFLAGS: 00000086
> RAX: ffff810785307fd8 RBX: ffffffff884e68a0 RCX: 0000000000000000
> RDX: 0000000000000001 RSI: 0000000000000001 RDI: ffffffff884e68a0
> RBP: ffffffff884e62a0 R08: ffff810790926900 R09: ffff8107909268e8
> R10: 0000000000000018 R11: ffffffff884fcab3 R12: 0000000000000001
> R13: 0000000000000001 R14: 0000000000000000 R15: ffff8107f0f374c0
> FS:  0000000000000000(0000) GS:ffffffff803a0000(0000) knlGS:0000000000000000
> CS:  0010 DS: 0018 ES: 0018 CR0: 000000008005003b
> CR2: 00000037bc0986d0 CR3: 0000000000201000 CR4: 00000000000006e0
> Process scsi_tgt0 (pid: 25732, threadinfo ffff810785306000, task 
> ffff810810852100)
> Stack:  0000000000000000 ffffffff884c509d ffff8107909268e8 ffff810790926900
>  00000002071dd688 0000020000000220 0000000000000200 00000000da984c08
>  0000000000000000 ffff8107909267f0 ffff810806ceee20 0000000000000001
> Call Trace:
>  <IRQ>  [<ffffffff884c509d>] :scst:sgv_pool_alloc+0x10c/0x5d3
>  [<ffffffff884c1f85>] :scst:scst_alloc_space+0x5b/0x106
>  [<ffffffff884bdc90>] :scst:scst_process_active_cmd+0x4fc/0x131c
>  [<ffffffff884bee46>] :scst:scst_cmd_init_done+0x17f/0x3ef
>  [<ffffffff884fb1ff>] :ib_srpt:srpt_handle_new_iu+0x281/0x4e7
>  [<ffffffff8835ec3d>] :mlx4_ib:mlx4_ib_free_srq_wqe+0x27/0x4f
>  [<ffffffff883591da>] :mlx4_ib:get_sw_cqe+0x12/0x30
>  [<ffffffff88359c97>] :mlx4_ib:mlx4_ib_poll_cq+0x432/0x48f
>  [<ffffffff884fcc43>] :ib_srpt:srpt_completion+0x190/0x250
>  [<ffffffff8811aa5b>] :mlx4_core:mlx4_eq_int+0x3b/0x26f
>  [<ffffffff8811ac9e>] :mlx4_core:mlx4_msi_x_interrupt+0xf/0x17

According to this trace, Vu was incorrect when he wrote that 
srpt_handle_new_iu called on tasklet context. It at least sometimes 
called from IRQ context. Try with the attached patch. It's against the 
latest trunk.
-------------- next part --------------
A non-text attachment was scrubbed...
Name: srpt_context.diff
Type: text/x-patch
Size: 837 bytes
Desc: not available
URL: <http://lists.openfabrics.org/pipermail/general/attachments/20081031/9e0a3f49/attachment.bin>