[ofa-general] SRP/mlx4 interrupts throttling performance

[Vladislav Bolkhovitin]

Cameron Harr wrote:
> Cameron Harr wrote:
>>> This is still too high. Considering that each CS is about 1 
>>> microsecond you can estimate how many IOPS's it costs you.
>> Dropping scst_threads down to 2, from 8, with 2 initiators, seems to 
>> make a fairly significant difference, propelling me to a little over 
>> 100K IOPs and putting the CS rate around 2:1, sometimes lower. 2 
>> threads gave the best performance compared to 1, 4 and 8.
> 
> Just as a status update, I've gotten my best performance with 
> scst_threads=3 on 2 initiators, and using a separate QP for each drive 
> an initiator is writing to. I'm getting pretty consistent 112-115K IOPs 
> using two initiators, each writing with 2 processes to the same 2 
> physical targets, using 512B blocks. Adding the second initiator only 
> bumps me up by about 20K IOPs, but as all the CPUs are pegged around 
> 99%, I'll take that as a bottleneck. Also, as a note from Vlad's advice, 
> the CS rate is now around 70K/s on 115K IOPs, so it's not too bad. 
> Interrupts (where this thread started), are around 200K/s - a lot higher 
> than I thought they'd go, but I'm not complaining. :)

Actually, what you did is tune your workload so it put nicely on all the 
participating threads and CPU cores, so all the threads stay each on its 
own CPU core and gracefully pass commands during processing to each 
other being busy almost all the time. I.e. you put your system in some 
kind of resonance. If you change your workload just a bit or Linux 
scheduler changed in the next kernel version, your tuning would be 
destroyed.

So, I wouldn't overestimate your results. As I already wrote, the only 
real fix is to remove all the unneeded context switches between threads 
during commands processing. This fix would work not only on carefully 
tuned artificial workloads, but on real life ones too. 5-10 threads 
participating in a single command processing reminds me the famous set 
of histories about how many people of some kind is necessary to change a 
burnt out lamp ;)

> Thanks for the help.
> Cameron
> _______________________________________________
> general mailing list
> general at lists.openfabrics.org
> http://lists.openfabrics.org/cgi-bin/mailman/listinfo/general
> 
> To unsubscribe, please visit http://openib.org/mailman/listinfo/openib-general
>