[Scst-devel] [ofa-general] SRP/mlx4 interrupts throttling performance
Cameron Harr
cameron at harr.org
Tue Jan 13 10:39:28 PST 2009
Vladislav Bolkhovitin wrote:
> Cameron Harr, on 01/13/2009 07:42 PM wrote:
>> Vlad, you've got a good eye. Unfortunately, those results can't
>> really be compared because I believe the previous results were
>> intentionally run in a worse-case performance scenario. However I did
>> run no-affinity runs before the affinity runs and would say
>> performance increase is variable and somewhat inconclusive:
>>
>> type=randwrite bs=4k drives=1 scst_threads=1 srptthread=1
>> iops=76724.08
>> type=randwrite bs=4k drives=2 scst_threads=1 srptthread=1
>> iops=91318.28
>> type=randwrite bs=4k drives=1 scst_threads=2 srptthread=1
>> iops=60374.94
>> type=randwrite bs=4k drives=2 scst_threads=2 srptthread=1
>> iops=91618.18
>> type=randwrite bs=4k drives=1 scst_threads=3 srptthread=1
>> iops=63076.21
>> type=randwrite bs=4k drives=2 scst_threads=3 srptthread=1
>> iops=92251.24
>> type=randwrite bs=4k drives=1 scst_threads=1 srptthread=0
>> iops=50539.96
>> type=randwrite bs=4k drives=2 scst_threads=1 srptthread=0
>> iops=57884.80
>> type=randwrite bs=4k drives=1 scst_threads=2 srptthread=0
>> iops=54502.85
>> type=randwrite bs=4k drives=2 scst_threads=2 srptthread=0
>> iops=93230.44
>> type=randwrite bs=4k drives=1 scst_threads=3 srptthread=0
>> iops=55941.89
>> type=randwrite bs=4k drives=2 scst_threads=3 srptthread=0
>> iops=94480.92
>
> For srptthread=0 case there is a consistent quite big increase.
For srptthread=0, there is indeed a difference between no-affinity and
affinity. However, here I meant there's not much of a difference between
srptthread=[01] on a number of the points - and overall on this
particular run it still seems like having the srp thread enabled still
gives better performance.
>> My CPU config on the target (where I did the affinity) is 2 quad-core
>> Xeon E5440 @ 2.83GHz. I didn't have my script configured to dump top
>> and vmstat, so here's data from a rerun (and I have attached
>> requested info). I'm not sure what is accounting for the spike at the
>> beginning, but it seems consistent.
>>
>> type=randwrite bs=4k drives=1 scst_threads=1 srptthread=1
>> iops=104699.43
>> type=randwrite bs=4k drives=2 scst_threads=1 srptthread=1
>> iops=133928.98
>> type=randwrite bs=4k drives=1 scst_threads=2 srptthread=1
>> iops=82736.73
>> type=randwrite bs=4k drives=2 scst_threads=2 srptthread=1
>> iops=82221.42
>> type=randwrite bs=4k drives=1 scst_threads=3 srptthread=1
>> iops=70203.53
>> type=randwrite bs=4k drives=2 scst_threads=3 srptthread=1
>> iops=85628.45
>> type=randwrite bs=4k drives=1 scst_threads=1 srptthread=0
>> iops=75646.90
>> type=randwrite bs=4k drives=2 scst_threads=1 srptthread=0
>> iops=87124.32
>> type=randwrite bs=4k drives=1 scst_threads=2 srptthread=0
>> iops=74545.84
>> type=randwrite bs=4k drives=2 scst_threads=2 srptthread=0
>> iops=88348.71
>> type=randwrite bs=4k drives=1 scst_threads=3 srptthread=0
>> iops=71837.15
>> type=randwrite bs=4k drives=2 scst_threads=3 srptthread=0
>> iops=84387.22
>
> Why there is such a huge difference with the results you sent in the
> previous e-mail? For instance, for case drives=1 scst_threads=1
> srptthread=1 104K vs 74K. What did you changed?
That's what I meant when I wasn't sure what was causing the spike at the
beginning. I didn't really do anything different other than rebooting.
One factor may be due to the supply of free blocks in the flash media.
As the number of free blocks decreases, garbage collection can increase
to free up previously-used blocks. However, there appears to be some
variance in the numbers that I can't account for. I just did another run
after forcing free blocks to a critical level and got 64K IOPs.
>
> What is content of /proc/interrupts after the tests?
You can see the HCA being interrupt-intensive, as well as iodrive, which
I'm surprised to see because I locked its worker threads to cpus 6 and 7.
CPU0 CPU1 CPU2 CPU3 CPU4
CPU5 CPU6 CPU7
0: 11173938 0 0 0 0
0 0 0 IO-APIC-edge timer
1: 9 0 0 0 0
0 0 0 IO-APIC-edge i8042
4: 14 0 0 0 0
0 0 0 IO-APIC-edge serial
8: 0 0 0 0 0
0 0 0 IO-APIC-edge rtc
9: 0 0 0 0 0
0 0 0 IO-APIC-level acpi
12: 106 0 0 0 0
0 0 0 IO-APIC-edge i8042
14: 35021 3114 0 0 47867
13692 0 0 IO-APIC-edge ide0
66: 19587 818 0 0 3743
3171 0 0 IO-APIC-level uhci_hcd:usb3, libata
74: 4209 159 0 0 0
0 0 0 PCI-MSI-X mlx4_core (async)
82: 273658543 63380024 0 0 0
0 0 0 PCI-MSI-X mlx4_core (comp)
90: 1 0 0 0 0
0 0 0 PCI-MSI-X dev23945
98: 2 0 0 0 0
0 0 0 PCI-MSI-X dev23945 (queue 0)
106: 0 0 0 0 0
0 0 0 PCI-MSI-X dev23945 (queue 1)
114: 0 0 0 0 0
0 0 0 PCI-MSI-X dev23945 (queue 2)
122: 0 0 0 0 0
0 0 0 PCI-MSI-X dev23945 (queue 3)
130: 0 0 0 0 0
0 0 0 PCI-MSI-X eth4 (queue 4)
138: 0 0 0 0 0
0 0 0 PCI-MSI-X eth4 (queue 5)
146: 0 0 0 0 0
0 0 0 PCI-MSI-X eth4 (queue 6)
154: 0 0 0 0 0
0 0 0 PCI-MSI-X eth4 (queue 7)
162: 58 0 0 0 0
0 48424 32510 PCI-MSI eth2
169: 149935 418 0 179925438 5982077 2103882
27511574 0 IO-APIC-level iodrive
177: 84565 10300 0 24953166 6645416 269061
38519676 28147 IO-APIC-level ehci_hcd:usb1, uhci_hcd:usb2, iodrive
185: 0 0 0 0 0
0 0 0 IO-APIC-level uhci_hcd:usb4
NMI: 4740 1805 1330 3495 2607
3838 3233 5481
LOC: 11171500 11171423 11171374 11171279 11171219 11171128
11171069 11170985
ERR: 0
MIS: 0
More information about the general
mailing list