[ofa-general] SRP/mlx4 interrupts throttling performance

[Vladislav Bolkhovitin]

Cameron Harr wrote:
> Vlad,
> Thanks for the suggestion. As I look via vmstat, my CSw/s rate is fairly 
> constant around 280K when scst_threads=1 (per Vu's suggestion) and pops 
> up to ~330-340K CSw/s when scst_threads is set to 8. I'm currently doing 
> 512B writes, and this gives me about a 4:1 ratio of context switches to 
> IOPs with 1 SCST thread (70K IOPs) and around 4.5:1 when there are 8 
> SCST threads (75K IOPs).

This is still too high. Considering that each CS is about 1 microsecond 
you can estimate how many IOPS's it costs you.

> You say those numbers could be overkill - do 
> you know of a way to drop the number?

Sorry, I don't. I can only say that too many CSs problem is in SRPT 
driver. With qla2x00t driver and BLOCKIO backstorage you will have 1 
CS/sec or less in average.

> I'm very interested in trying Vu's 
> other suggestions (multiple initiators and multiple QPs, but my other 
> initiator has been too busy all weekend to run on.
> 
> Debug, tracing, and all that was turned off in the SCST Makefiles.
> 
> -Cameron
> 
> Vladislav Bolkhovitin wrote:
>> Cameron Harr wrote:
>>> I was able to get the latest scst code working with Vu's standalone 
>>> ib_srpt and the kernel IB modules, and dropped my ib_srpt thread 
>>> count to 2. However, I still get about the same IOP performance on 
>>> the target although interrupts on the "busy" cpu have gone up to 
>>> around 140K. Interesting, but now I'm at a bit of a loss as to where 
>>> the bottleneck could be. I figured it was Interrupts, but if the CPU 
>>> is handling more right now, perhaps the problem is elsewhere?
>> How many context switches per second do you have during your test on 
>> the target?
>>
>> Once in scst-devel mailing list there was a thread about observation 
>> that SRP target driver produces 10 context switches per command. See 
>> http://sourceforge.net/mailarchive/message.php?msg_id=e2e108260802070110q1fa084a1j54945d06c16c94f2%40mail.gmail.com 
>>
>>
>> If it is so in your case as well, it would very well explain your 
>> issue. 10 CS/cmd is a definite overkill, it should be 1 or, at max, 2 
>> CS/cmd.
>>
>> BTW, I suppose you don't use the debug SCST build, do you?
>>
>> Vlad