[Scst-devel] [ofa-general] SRP/mlx4 interrupts throttling performance

[Vladislav Bolkhovitin]

Cameron Harr, on 01/13/2009 09:39 PM wrote:
> Vladislav Bolkhovitin wrote:
>> Cameron Harr, on 01/13/2009 07:42 PM wrote:
>>> Vlad, you've got a good eye. Unfortunately, those results can't 
>>> really be compared because I believe the previous results were 
>>> intentionally run in a worse-case performance scenario. However I did 
>>> run no-affinity runs before the affinity runs and would say 
>>> performance increase is variable and somewhat inconclusive:
>>>
>>> type=randwrite  bs=4k   drives=1 scst_threads=1 srptthread=1 
>>> iops=76724.08
>>> type=randwrite  bs=4k   drives=2 scst_threads=1 srptthread=1 
>>> iops=91318.28
>>> type=randwrite  bs=4k   drives=1 scst_threads=2 srptthread=1 
>>> iops=60374.94
>>> type=randwrite  bs=4k   drives=2 scst_threads=2 srptthread=1 
>>> iops=91618.18
>>> type=randwrite  bs=4k   drives=1 scst_threads=3 srptthread=1 
>>> iops=63076.21
>>> type=randwrite  bs=4k   drives=2 scst_threads=3 srptthread=1 
>>> iops=92251.24
>>> type=randwrite  bs=4k   drives=1 scst_threads=1 srptthread=0 
>>> iops=50539.96
>>> type=randwrite  bs=4k   drives=2 scst_threads=1 srptthread=0 
>>> iops=57884.80
>>> type=randwrite  bs=4k   drives=1 scst_threads=2 srptthread=0 
>>> iops=54502.85
>>> type=randwrite  bs=4k   drives=2 scst_threads=2 srptthread=0 
>>> iops=93230.44
>>> type=randwrite  bs=4k   drives=1 scst_threads=3 srptthread=0 
>>> iops=55941.89
>>> type=randwrite  bs=4k   drives=2 scst_threads=3 srptthread=0 
>>> iops=94480.92
>> For srptthread=0 case there is a consistent quite big increase.
> For srptthread=0, there is indeed a difference between no-affinity and 
> affinity. However, here I meant there's not much of a difference between 
> srptthread=[01] on a number of the points - and overall on this 
> particular run it still seems like having the srp thread enabled still 
> gives better performance.

We made a progress, but the affinity you set wasn't optimal, hence the 
results. We need to find out the optimal affinity layout. With it 
results for both srptthread=[01] should be at least the same.

>>> My CPU config on the target (where I did the affinity) is 2 quad-core 
>>> Xeon E5440 @ 2.83GHz. I didn't have my script configured to dump top 
>>> and vmstat, so here's data from a rerun (and I have attached 
>>> requested info). I'm not sure what is accounting for the spike at the 
>>> beginning, but it seems consistent.
>>>
>>> type=randwrite  bs=4k   drives=1 scst_threads=1 srptthread=1 
>>> iops=104699.43
>>> type=randwrite  bs=4k   drives=2 scst_threads=1 srptthread=1 
>>> iops=133928.98
>>> type=randwrite  bs=4k   drives=1 scst_threads=2 srptthread=1 
>>> iops=82736.73
>>> type=randwrite  bs=4k   drives=2 scst_threads=2 srptthread=1 
>>> iops=82221.42
>>> type=randwrite  bs=4k   drives=1 scst_threads=3 srptthread=1 
>>> iops=70203.53
>>> type=randwrite  bs=4k   drives=2 scst_threads=3 srptthread=1 
>>> iops=85628.45
>>> type=randwrite  bs=4k   drives=1 scst_threads=1 srptthread=0 
>>> iops=75646.90
>>> type=randwrite  bs=4k   drives=2 scst_threads=1 srptthread=0 
>>> iops=87124.32
>>> type=randwrite  bs=4k   drives=1 scst_threads=2 srptthread=0 
>>> iops=74545.84
>>> type=randwrite  bs=4k   drives=2 scst_threads=2 srptthread=0 
>>> iops=88348.71
>>> type=randwrite  bs=4k   drives=1 scst_threads=3 srptthread=0 
>>> iops=71837.15
>>> type=randwrite  bs=4k   drives=2 scst_threads=3 srptthread=0 
>>> iops=84387.22
>> Why there is such a huge difference with the results you sent in the 
>> previous e-mail? For instance, for case drives=1 scst_threads=1 
>> srptthread=1 104K vs 74K. What did you changed?
> That's what I meant when I wasn't sure what was causing the spike at the 
> beginning. I didn't really do anything different other than rebooting. 
> One factor may be due to the supply of free blocks in the flash media. 
> As the number of free blocks decreases, garbage collection can increase 
> to free up previously-used blocks. However, there appears to be some 
> variance in the numbers that I can't account for. I just did another run 
> after forcing free blocks to a critical level and got 64K IOPs.
>> What is content of /proc/interrupts after the tests?
> You can see the HCA being interrupt-intensive, as well as iodrive, which 
> I'm surprised to see because I locked its worker threads to cpus 6 and 7.

Try the following variants:

1. Affine IRQ 82, scsi_tgt0 to CPU0, fct0-worker to CPU2, IRQs 169 and 
177 to CPU4, scsi_tgt1 to CPU1, fct1-worker to CPU3, scsi_tgt2 to CPU5, 
fct2-worker to CPU7

2. Affine IRQ 82 to CPU0, fct0-worker to CPU2, IRQs 169 and 177 to CPU4, 
fct1-worker to CPU3, fct2-worker to CPU7, no affinity for other processes.

3. Affine IRQ 82 to CPU0, IRQs 169 and 177 to CPU4, fct1-worker's to all 
CPUs, except CPU0 and CPU4, no affinity for other processes.

Or other similar variants you'd like (even CPUs relate to physical CPU0, 
odd CPUs relate to physical CPU1). For instance, you can try to affine 
IRQs 169 and 177 to CPU1.

No points to run for srptthread=1, for it just produce a baseline with 
no affinity at all.

Please do each run several times and write down an average result 
between runs and approximate variation between them in %%. Otherwise we 
can't make any reliable conclusions.

Thanks,
Vlad