[ofa-general] SRP/mlx4 interrupts throttling performance

[Cameron Harr]

Greetings,
While trying to maximize IOPs over SRP, I ran into a pretty significant 
bottleneck. After digging into the matter, I believe the problem is 
coming from the mlx4 drivers overwhelming one of the cpus, and thus 
cutting performance to that level (since no more IB traffic means no 
more data for the disks).

My target server (with DAS) contains 8 2.8 GHz CPU cores and can sustain 
over 200K IOPs locally, but only around 73K IOPs over SRP. Looking at 
/proc/interrupts, I see that the mlx_core (comp) device is pushing about 
135K Int/s on 1 of 2 CPUs. All CPUs are enabled for that PCI-E slot, but 
it only ever uses 2 of the CPUs, and only 1 at a time. None of the other 
CPUs has an interrupt rate more than about 40-50K/s.

Does anyone know of a trick to spread those interrupts out more (which I 
realize might be bad due to context switching), or something else that 
will reduce my interrupts on that cpu? The mlx4 is a MSI-X interrupt. 
I've changed it to an APIC int, but it seems to give slightly lower 
performance.

Thanks,
Cameron



CONFIDENTIAL

This document and attachments contain information from Fusion-io, Inc. which is confidential and/or legally privileged. 
The information is intended only for the use of the individual or entity named on this transmission. 
If you are not the intended recipient, you are hereby notified that any disclosure, copying, distribution or taking of any 
action in reliance on the contents of this emailed information is strictly prohibited, and that the documents should be returned to Fusion-io, Inc. immediately. 
In this regard, if you have received this email in error, please notify us by return email immediately.