[ofa-general] SRP/mlx4 interrupts throttling performance

[Vladislav Bolkhovitin]

Cameron Harr wrote:
> Bart Van Assche wrote:
>> On Mon, Oct 6, 2008 at 5:31 PM, Cameron Harr <cameron at harr.org> wrote:
>>   
>>> Thanks for the suggestion. As I look via vmstat, my CSw/s rate is fairly
>>> constant around 280K when scst_threads=1 (per Vu's suggestion) and pops up
>>> to ~330-340K CSw/s when scst_threads is set to 8.
>>>     
>>
>> Which threads are causing all those context switches ? You can find
>> this out by making sure that CONFIG_SCHEDSTATS=y is enabled in the
>> kernel .config and by running the following bash command:
>>
>> ( cd /proc && for p in [1-9]* ; do echo "$(<${p}/cmdline)
>> $(<${p}/schedstat)" ; done ) | sort -rn -k 3 | head
>>
>>
>>   
> Thanks for the bash lesson :). It wasn't working how I think you had 
> planned because many processes have nothing in the cmdline file. So, I 
> touched up the command a bit, putting in the pid and displaying the 
> cmdline at the end so as not to mess up the sort:
> ( cd /proc && for p in [1-9]* ; do echo -e "$p:\t $(<${p}/schedstat) 
> \t\t$(<${p}/cmdline)" ; done ) | sort -rn -k 3 | head
> 
> Using that, and watching who's moving up in amount of time waiting, the 
> main culprits are all of the scst_threads when scst_threads=8, and when 
> threads=2, the culprit is srpt_thread.

After some code examination, I figured out that Vu has chosen a 
"defensive programming" way ;): always switch to another thread.

I personally don't see why srpt_thread is needed at all. Vu, if you 
think that the processing is too heavy weighted, you should rather use 
tasklets instead.

SCST functions scst_cmd_init_done() and scst_rx_data() should be called 
with context SCST_CONTEXT_DIRECT_ATOMIC from interrupt context or 
SCST_CONTEXT_DIRECT from thread context. Then amount of context switches 
per cmd will go to the same reasonable level <=1 as for qla2x00t.

> -Cameron
> 
> 
> ------------------------------------------------------------------------
> 
> _______________________________________________
> general mailing list
> general at lists.openfabrics.org
> http://lists.openfabrics.org/cgi-bin/mailman/listinfo/general
> 
> To unsubscribe, please visit http://openib.org/mailman/listinfo/openib-general