[ofa-general] SRP/mlx4 interrupts throttling performance

[Vu Pham]

Vladislav Bolkhovitin wrote:
> Cameron Harr wrote:
>> Bart Van Assche wrote:
>>> On Mon, Oct 6, 2008 at 5:31 PM, Cameron Harr <cameron at harr.org> wrote:
>>>  
>>>> Thanks for the suggestion. As I look via vmstat, my CSw/s rate is 
>>>> fairly
>>>> constant around 280K when scst_threads=1 (per Vu's suggestion) and 
>>>> pops up
>>>> to ~330-340K CSw/s when scst_threads is set to 8.
>>>>     
>>>
>>> Which threads are causing all those context switches ? You can find
>>> this out by making sure that CONFIG_SCHEDSTATS=y is enabled in the
>>> kernel .config and by running the following bash command:
>>>
>>> ( cd /proc && for p in [1-9]* ; do echo "$(<${p}/cmdline)
>>> $(<${p}/schedstat)" ; done ) | sort -rn -k 3 | head
>>>
>>>
>>>   
>> Thanks for the bash lesson :). It wasn't working how I think you had 
>> planned because many processes have nothing in the cmdline file. So, 
>> I touched up the command a bit, putting in the pid and displaying the 
>> cmdline at the end so as not to mess up the sort:
>> ( cd /proc && for p in [1-9]* ; do echo -e "$p:\t $(<${p}/schedstat) 
>> \t\t$(<${p}/cmdline)" ; done ) | sort -rn -k 3 | head
>>
>> Using that, and watching who's moving up in amount of time waiting, 
>> the main culprits are all of the scst_threads when scst_threads=8, 
>> and when threads=2, the culprit is srpt_thread.
>
> After some code examination, I figured out that Vu has chosen a 
> "defensive programming" way ;): always switch to another thread.
>
> I personally don't see why srpt_thread is needed at all. Vu, if you 
> think that the processing is too heavy weighted, you should rather use 
> tasklets instead.
>
> SCST functions scst_cmd_init_done() and scst_rx_data() should be 
> called with context SCST_CONTEXT_DIRECT_ATOMIC from interrupt context 
> or SCST_CONTEXT_DIRECT from thread context. Then amount of context 
> switches per cmd will go to the same reasonable level <=1 as for 
> qla2x00t. \

You are correct - by default srp run in thread mode - srp can also run 
in tasklet mode (parameter thread=0); however, the main trade of is 
instability (in heavy tpc-h workload)

I already let Cameron know about this. We should have some prelim. 
number from him soon (running with thread=0) and we need some quality 
time to debug/fix the instability of some special workload


>
>> -Cameron
>>
>>
>> ------------------------------------------------------------------------
>>
>> _______________________________________________
>> general mailing list
>> general at lists.openfabrics.org
>> http://lists.openfabrics.org/cgi-bin/mailman/listinfo/general
>>
>> To unsubscribe, please visit 
>> http://openib.org/mailman/listinfo/openib-general
>