[ofa-general] SRP/mlx4 interrupts throttling performance

[Vu Pham]

Cameron Harr wrote:
>
>
> Vu Pham wrote:
>> Cameron Harr wrote:
>>> Vu Pham wrote:
>>>>
>>>>> Alternatively, is there anything in the SCST layer I should tweak. 
>>>>> I'm
>>>>> still running rev 245 of that code (kinda old, but works with OFED 
>>>>> 1.3.1
>>>>> w/o hacks).
>>
>> With blockio I get the best performance + stability with scst_threads=1
>
> I got best performance with threads=2 or 3, and I've noticed that the 
> srpt_thread is often at 99%, though if I increase/decrease the 
> "thread=?" parameter for ib_srpt, it doesn't seem to make a 
> difference. A second initiator doesn't seem to help much either, with 
> a single initiator writing to two targets, can now usually get between 
> 95K and 105K IOPs.

ib_srpt's "thread=?" parameter does not mean number of thread but 
indicating you are in thread context or not
thread=0 --> you will avoid one context switch between ib_srpt's thread 
and scst's threads. You may get better result with thread=0; however, 
there is stability risk

>>>>>>
>>>>>> My target server (with DAS) contains 8 2.8 GHz CPU cores and can 
>>>>>> sustain over 200K IOPs locally, but only around 73K IOPs over SRP.
>>>>
>>>> Is this number from one initiator or multiple?
>>> One initiator. At first I thought it might be a limitation of the 
>>> SRP, and added a second initiator, but the aggregate performance of 
>>> the two was about equal to that of a single initiator.
>>
>> Try again with scst_threads=1. I expect that you can get ~140K with 
>> two initiators
>>
> Unfortunately, I'm nowhere close that high, though I am significantly 
> higher than before. 2 initiators does seem to reduce the context 
> switching rate however, which is good.

Could you try again with ib_srpt' thread=0?

>>>>>> Looking at /proc/interrupts, I see that the mlx_core (comp) 
>>>>>> device is pushing about 135K Int/s on 1 of 2 CPUs. All CPUs are 
>>>>>> enabled for that PCI-E slot, but it only ever uses 2 of the CPUs, 
>>>>>> and only 1 at a time. None of the other CPUs has an interrupt 
>>>>>> rate more than about 40-50K/s.
>>>> The number of interrupt can be cut down if there are more 
>>>> completions to be processed by sw. ie. please test with multiple 
>>>> QPs between one initiator vs. your target and multiple initiators 
>>>> vs. your target
> Interrupts are still pretty high (around 160K/s now), but that seems 
> to not be my bottleneck. Context switching seems to be about 2-2.5 for 
> every IOP and sometimes less - not perfect, but not horrible either.
>>
>> ib_srpt process completions in event callback handler. With more QPs 
>> there are more completions pending per interrupt instead of one 
>> completion event per interrupt.
>> You can have multiple QPs between initiator vs. target by using 
>> different initiator_id_ext ie.
>> echo id_ext=xxx,ioc_guid=yyy,....initiator_ext=1 > 
>> /sys/class/infiniband_srp/.../add_target
>> echo id_ext=xxx,ioc_guid=yyy,....initiator_ext=2 > 
>> /sys/class/infiniband_srp/.../add_target
>> echo id_ext=xxx,ioc_guid=yyy,....initiator_ext=3 > 
>> /sys/class/infiniband_srp/.../add_target
> This doesn't seem to net much of an improvement, though I understand 
> the reasoning behind it. My hunch is there's another bottleneck now to 
> look for.
>
> Cameron
> _______________________________________________
> general mailing list
> general at lists.openfabrics.org
> http://lists.openfabrics.org/cgi-bin/mailman/listinfo/general
>
> To unsubscribe, please visit 
> http://openib.org/mailman/listinfo/openib-general