[ofa-general] SRP/mlx4 interrupts throttling performance

[Cameron Harr]

Cameron Harr wrote:
>> This is still too high. Considering that each CS is about 1 
>> microsecond you can estimate how many IOPS's it costs you.
>
> Dropping scst_threads down to 2, from 8, with 2 initiators, seems to 
> make a fairly significant difference, propelling me to a little over 
> 100K IOPs and putting the CS rate around 2:1, sometimes lower. 2 
> threads gave the best performance compared to 1, 4 and 8.

Just as a status update, I've gotten my best performance with 
scst_threads=3 on 2 initiators, and using a separate QP for each drive 
an initiator is writing to. I'm getting pretty consistent 112-115K IOPs 
using two initiators, each writing with 2 processes to the same 2 
physical targets, using 512B blocks. Adding the second initiator only 
bumps me up by about 20K IOPs, but as all the CPUs are pegged around 
99%, I'll take that as a bottleneck. Also, as a note from Vlad's advice, 
the CS rate is now around 70K/s on 115K IOPs, so it's not too bad. 
Interrupts (where this thread started), are around 200K/s - a lot higher 
than I thought they'd go, but I'm not complaining. :)

Thanks for the help.
Cameron