[ofa-general] SRP/mlx4 interrupts throttling performance

[Cameron Harr]

Alternatively, is there anything in the SCST layer I should tweak. I'm
still running rev 245 of that code (kinda old, but works with OFED 1.3.1
w/o hacks).

And sorry about the yucky signature on my past email - I used to be able
to send w/o that.

Cameron Harr wrote:
> Greetings,
> While trying to maximize IOPs over SRP, I ran into a pretty 
> significant bottleneck. After digging into the matter, I believe the 
> problem is coming from the mlx4 drivers overwhelming one of the cpus, 
> and thus cutting performance to that level (since no more IB traffic 
> means no more data for the disks).
>
> My target server (with DAS) contains 8 2.8 GHz CPU cores and can 
> sustain over 200K IOPs locally, but only around 73K IOPs over SRP. 
> Looking at /proc/interrupts, I see that the mlx_core (comp) device is 
> pushing about 135K Int/s on 1 of 2 CPUs. All CPUs are enabled for that 
> PCI-E slot, but it only ever uses 2 of the CPUs, and only 1 at a time. 
> None of the other CPUs has an interrupt rate more than about 40-50K/s.
>
> Does anyone know of a trick to spread those interrupts out more (which 
> I realize might be bad due to context switching), or something else 
> that will reduce my interrupts on that cpu? The mlx4 is a MSI-X 
> interrupt. I've changed it to an APIC int, but it seems to give 
> slightly lower performance.
>
> Thanks,
> Cameron
>