[ofa-general] SRP/mlx4 interrupts throttling performance

[Cameron Harr]

Vlad,
Thanks for the suggestion. As I look via vmstat, my CSw/s rate is fairly 
constant around 280K when scst_threads=1 (per Vu's suggestion) and pops 
up to ~330-340K CSw/s when scst_threads is set to 8. I'm currently doing 
512B writes, and this gives me about a 4:1 ratio of context switches to 
IOPs with 1 SCST thread (70K IOPs) and around 4.5:1 when there are 8 
SCST threads (75K IOPs). You say those numbers could be overkill - do 
you know of a way to drop the number? I'm very interested in trying Vu's 
other suggestions (multiple initiators and multiple QPs, but my other 
initiator has been too busy all weekend to run on.

Debug, tracing, and all that was turned off in the SCST Makefiles.

-Cameron

Vladislav Bolkhovitin wrote:
> Cameron Harr wrote:
>> I was able to get the latest scst code working with Vu's standalone 
>> ib_srpt and the kernel IB modules, and dropped my ib_srpt thread 
>> count to 2. However, I still get about the same IOP performance on 
>> the target although interrupts on the "busy" cpu have gone up to 
>> around 140K. Interesting, but now I'm at a bit of a loss as to where 
>> the bottleneck could be. I figured it was Interrupts, but if the CPU 
>> is handling more right now, perhaps the problem is elsewhere?
>
> How many context switches per second do you have during your test on 
> the target?
>
> Once in scst-devel mailing list there was a thread about observation 
> that SRP target driver produces 10 context switches per command. See 
> http://sourceforge.net/mailarchive/message.php?msg_id=e2e108260802070110q1fa084a1j54945d06c16c94f2%40mail.gmail.com 
>
>
> If it is so in your case as well, it would very well explain your 
> issue. 10 CS/cmd is a definite overkill, it should be 1 or, at max, 2 
> CS/cmd.
>
> BTW, I suppose you don't use the debug SCST build, do you?
>
> Vlad
>
>> Cameron
>>
>> Cameron Harr wrote:
>>> Cameron Harr wrote:
>>>> Additionally, I found that I can load the newer scst code if I use 
>>>> the kernel-supplied modules and the standalone srpt-1.0.0 package 
>>>> that I think you provide Vu. I was about to try it along with 
>>>> dropping a module param for ib_srpt (I was using a thread count of 
>>>> 32 that had given me better performance on an earlier test). I'll 
>>>> report back on this.
>>> Not much luck using the newer scst code and default kernel modules 
>>> (Running CentOS 5.2). If I try using the default kernel modules on 
>>> the initiator, I can't get them to see anything (the ofed SM pkg 
>>> doesn't see any devices to run on). When using the regular OFED on 
>>> the initiator, my target dies when I try to attach to the target on 
>>> the initiator:
>>> ---------------------------------
>>>  ib_srpt: Host login i_port_id=0x0:0x2c90300026053 
>>> t_port_id=0x2c90300026046:0x2c90300026046 it_iu_len=996
>>> Oct  3 13:44:23 test05 kernel: i[4127]: scst: 
>>> scst_mgmt_thread:5187:***CRITICAL ERROR*** session ffff8107f3222b88 
>>> is in scst_sess_shut_list, but in unknown shut phase 0
>>> BUG at /usr/src/scst.tot/src/scst_targ.c:5188
>>> ----------- [cut here ] --------- [please bite here ] ---------
>>> Kernel BUG at /usr/src/scst.tot/src/scst_targ.c:5188
>>> invalid opcode: 0000 [1] SMP
>>> last sysfs file: /devices/pci0000:00/0000:00:00.0/class
>>> CPU 2
>>> Modules linked in: ib_srpt(U) ib_cm ib_sa scst_vdisk(U) scst(U) 
>>> fio_driver(PU) fio_port(PU) mlx4_ib ib_mad ib_core ipv6 xfrm_nalgo 
>>> crypto_api autofs4 hidp rfcomm l2cap bluetooth sunrpc nls_utf8 
>>> hfsplus dm_mirror dm_multipath dm_mod video sbs backlight i2c_ec 
>>> button battery asus_acpi acpi_memhotplug ac parport_pc lp parport 
>>> i2c_i801 i5000_edac i2c_core edac_mc pcspkr shpchp mlx4_core e1000e 
>>> ata_piix libata sd_mod scsi_mod ext3 jbd uhci_hcd ohci_hcd ehci_hcd
>>> Pid: 4127, comm: scsi_tgt_mgmt Tainted: P      2.6.18-92.1.13.el5 #1
>>> RIP: 0010:[<ffffffff88488a56>]  [<ffffffff88488a56>] 
>>> :scst:scst_mgmt_thread+0x3ff/0x577
>>> ---------------------------------
>>>
>> _______________________________________________
>> general mailing list
>> general at lists.openfabrics.org
>> http://lists.openfabrics.org/cgi-bin/mailman/listinfo/general
>>
>> To unsubscribe, please visit 
>> http://openib.org/mailman/listinfo/openib-general
>>
>