[ofa-general] SRP/mlx4 interrupts throttling performance

[Cameron Harr]

Cameron Harr wrote:
> I tried with the latest scst and srpt as of this morning. Previously, 
> I had used srpt-1.0.0. The following results are with BLOCKIO, and 
> I'll have a NULLIO in a bit. You can see from here that I don't hang 
> any more, but the srpt thread=0 are a little lower.
>
> As before this run was done with ioengine=libaio and iodepth=16. I 
> pretty much always get significantly better performance with libaio 
> than with sync or other engines. Also, the iodepth setting tended to 
> give me better results.
> ----------------------------------------------
> type=randwrite  bs=512  drives=1 scst_threads=1 srptthread=1 
> iops=67073.48
> type=randwrite  bs=4k   drives=1 scst_threads=1 srptthread=1 
> iops=54876.82
> type=randwrite  bs=512  drives=2 scst_threads=1 srptthread=1 
> iops=74858.00
> type=randwrite  bs=4k   drives=2 scst_threads=1 srptthread=1 
> iops=75357.15
> type=randwrite  bs=512  drives=3 scst_threads=1 srptthread=1 
> iops=83257.72
> type=randwrite  bs=4k   drives=3 scst_threads=1 srptthread=1 
> iops=82186.79
> type=randwrite  bs=512  drives=1 scst_threads=2 srptthread=1 
> iops=59908.06
> type=randwrite  bs=4k   drives=1 scst_threads=2 srptthread=1 
> iops=50982.91
> type=randwrite  bs=512  drives=2 scst_threads=2 srptthread=1 
> iops=99243.07
> type=randwrite  bs=4k   drives=2 scst_threads=2 srptthread=1 
> iops=79670.62
> type=randwrite  bs=512  drives=3 scst_threads=2 srptthread=1 
> iops=102898.37
> type=randwrite  bs=4k   drives=3 scst_threads=2 srptthread=1 
> iops=92248.25
> type=randwrite  bs=512  drives=1 scst_threads=3 srptthread=1 
> iops=63086.77
> type=randwrite  bs=4k   drives=1 scst_threads=3 srptthread=1 
> iops=53020.41
> type=randwrite  bs=512  drives=2 scst_threads=3 srptthread=1 
> iops=95990.06
> type=randwrite  bs=4k   drives=2 scst_threads=3 srptthread=1 
> iops=77487.26
> type=randwrite  bs=512  drives=3 scst_threads=3 srptthread=1 
> iops=105945.85
> type=randwrite  bs=4k   drives=3 scst_threads=3 srptthread=1 
> iops=95389.01
> type=randwrite  bs=512  drives=1 scst_threads=1 srptthread=0 
> iops=50299.36
> type=randwrite  bs=4k   drives=1 scst_threads=1 srptthread=0 
> iops=48070.11
> type=randwrite  bs=512  drives=2 scst_threads=1 srptthread=0 
> iops=54017.21
> type=randwrite  bs=4k   drives=2 scst_threads=1 srptthread=0 
> iops=50407.20
> type=randwrite  bs=512  drives=3 scst_threads=1 srptthread=0 
> iops=55822.11
> type=randwrite  bs=4k   drives=3 scst_threads=1 srptthread=0 
> iops=50447.82
> type=randwrite  bs=512  drives=1 scst_threads=2 srptthread=0 
> iops=60672.48
> type=randwrite  bs=4k   drives=1 scst_threads=2 srptthread=0 
> iops=48811.93
> type=randwrite  bs=512  drives=2 scst_threads=2 srptthread=0 
> iops=81919.87
> type=randwrite  bs=4k   drives=2 scst_threads=2 srptthread=0 
> iops=72912.99
> type=randwrite  bs=512  drives=3 scst_threads=2 srptthread=0 
> iops=91036.45
> type=randwrite  bs=4k   drives=3 scst_threads=2 srptthread=0 
> iops=88994.63
> type=randwrite  bs=512  drives=1 scst_threads=3 srptthread=0 
> iops=58929.21
> type=randwrite  bs=4k   drives=1 scst_threads=3 srptthread=0 
> iops=48698.90
> type=randwrite  bs=512  drives=2 scst_threads=3 srptthread=0 
> iops=83967.58
> type=randwrite  bs=4k   drives=2 scst_threads=3 srptthread=0 
> iops=73932.36
> type=randwrite  bs=512  drives=3 scst_threads=3 srptthread=0 
> iops=96686.46
> type=randwrite  bs=4k   drives=3 scst_threads=3 srptthread=0 
> iops=88689.27
>

And here are the results with NULLIO, sorted by block size. Having the 
SRPT thread=0 actually shows some benefit here:
-------------------------------------------
type=randwrite  bs=4k   drives=1 scst_threads=1 srptthread=0 iops=140700.40
type=randwrite  bs=4k   drives=1 scst_threads=1 srptthread=1 iops=89167.67
type=randwrite  bs=4k   drives=1 scst_threads=2 srptthread=0 iops=125166.68
type=randwrite  bs=4k   drives=1 scst_threads=2 srptthread=1 iops=136699.05
type=randwrite  bs=4k   drives=1 scst_threads=3 srptthread=0 iops=127363.18
type=randwrite  bs=4k   drives=1 scst_threads=3 srptthread=1 iops=91205.03
type=randwrite  bs=4k   drives=2 scst_threads=1 srptthread=0 iops=94412.46
type=randwrite  bs=4k   drives=2 scst_threads=1 srptthread=1 iops=84354.34
type=randwrite  bs=4k   drives=2 scst_threads=2 srptthread=0 iops=155053.30
type=randwrite  bs=4k   drives=2 scst_threads=2 srptthread=1 iops=102480.27
type=randwrite  bs=4k   drives=2 scst_threads=3 srptthread=0 iops=141045.50
type=randwrite  bs=4k   drives=2 scst_threads=3 srptthread=1 iops=99681.15
type=randwrite  bs=4k   drives=3 scst_threads=1 srptthread=0 iops=173182.91
type=randwrite  bs=4k   drives=3 scst_threads=1 srptthread=1 iops=117629.27
type=randwrite  bs=4k   drives=3 scst_threads=2 srptthread=0 iops=99960.51
type=randwrite  bs=4k   drives=3 scst_threads=2 srptthread=1 iops=103412.00
type=randwrite  bs=4k   drives=3 scst_threads=3 srptthread=0 iops=120926.77
type=randwrite  bs=4k   drives=3 scst_threads=3 srptthread=1 iops=100368.32
type=randwrite  bs=512  drives=1 scst_threads=1 srptthread=0 iops=102232.77
type=randwrite  bs=512  drives=1 scst_threads=1 srptthread=1 iops=139095.94
type=randwrite  bs=512  drives=1 scst_threads=2 srptthread=0 iops=130327.29
type=randwrite  bs=512  drives=1 scst_threads=2 srptthread=1 iops=159158.20
type=randwrite  bs=512  drives=1 scst_threads=3 srptthread=0 iops=136153.84
type=randwrite  bs=512  drives=1 scst_threads=3 srptthread=1 iops=92417.19
type=randwrite  bs=512  drives=2 scst_threads=1 srptthread=0 iops=126892.60
type=randwrite  bs=512  drives=2 scst_threads=1 srptthread=1 iops=99436.74
type=randwrite  bs=512  drives=2 scst_threads=2 srptthread=0 iops=101566.13
type=randwrite  bs=512  drives=2 scst_threads=2 srptthread=1 iops=142292.97
type=randwrite  bs=512  drives=2 scst_threads=3 srptthread=0 iops=166114.78
type=randwrite  bs=512  drives=2 scst_threads=3 srptthread=1 iops=155634.89
type=randwrite  bs=512  drives=3 scst_threads=1 srptthread=0 iops=131368.01
type=randwrite  bs=512  drives=3 scst_threads=1 srptthread=1 iops=186550.24
type=randwrite  bs=512  drives=3 scst_threads=2 srptthread=0 iops=139813.79
type=randwrite  bs=512  drives=3 scst_threads=2 srptthread=1 iops=162499.08
type=randwrite  bs=512  drives=3 scst_threads=3 srptthread=0 iops=154777.28
type=randwrite  bs=512  drives=3 scst_threads=3 srptthread=1 iops=187425.87