[ofa-general] SRP/mlx4 interrupts throttling performance

[Cameron Harr]

Vu Pham wrote:
> Vladislav Bolkhovitin wrote:
>> Cameron Harr wrote:
>>>>>>> Vladislav Bolkhovitin wrote:
>>>>>>>> I guess, you use a regular caching IO? The lowest packet size 
>>>>>>>> it can produce is a PAGE_SIZE (4K). Target can't change it. You 
>>>>>>>> can have lower packets only with O_DIRECT or sg interface. But 
>>>>>>>> I'm not sure it will be performance effective.
>>>>>>> I do everything with Direct IO, which is automatic when using 
>>>>>>> the BLOCKIO method in SCST.
>>>>>> I meant on initiator(s), not on the target.
>>>>>>
>>>>> Sorry - but yes, I always run the benchmark apps with direct IO
>>>> Then, there's one more reason why we should find out the cause of 
>>>> such a big variation between runs. Can you repeat all the tests 
>>>> with the latest SCST SVN trunk/ including SRPT driver with each run 
>>>> for at least few minutes?
>>>
>>>  From a little testing, the updated SCST tree doesn't work with the 
>>> OFED-1.3.1 SRP stack, though I have gotten it working with the 
>>> infiniband drivers in the normal distribution kernel. Shall I use 
>>> those modules?
>>
>> I think, only Vu can answer it.
>
> ofed-1.3.1 srpt only works with scst-1.0.0
> You can run 2.6.26 + its IB stack + latest SVN scst and srpt
>
> -vu

I think I'm stuck with the default CentOS kernel: 2.6.18-92.1.13.el5, 
but I can use that.
>
>>
>>> Also, as I mentioned before, my time is going to be fairly limited 
>>> for a while, but I'll try to squeeze this in and will make sure I 
>>> run for longer periods of time. I'll also try to calculate an exact 
>>> end number based on iop/runtime.
>>>> Which backstorage do you use for BLOCKIO?
>>> Fusion IO ioDrive. I generally use 1 or 2, but somtimes up to 4 at a 
>>> time.
>>
>> It might be a reason of not stable results. Can you try with NULLIO 
>> to narrow things down a bit?

I actually tested NULLIO a while back at Vu's suggestion, but I believe 
I forgot to report on it. From my recollection, NULLIO gave me pretty 
consistent block sizes of 512B, which is the BS I used in the benchmark.