[ofa-general] Distributing ib_mthca interrupts

[Charles Taylor]

We have servers that have both IB and ethernet-only clients.    The  
ethernet-only clients access the servers via IPoIB through a  
gateway.    For small MTUs, the IpoIB software interrupt load can get  
pretty unmanageable so we would like to be able to spread the load  
over multiple cores.    Our current /proc/interrupts looks like...

cat /proc/interrupts
            CPU0       CPU1       CPU2       CPU3       CPU4        
CPU5       CPU6       CPU7
snip...
  67:        946          0          0          0 1364412993           
0          0          0       PCI-MSI-X  ib_mthca (comp)
  75:       1785          0          0          0    6286882           
0          0          0       PCI-MSI-X  ib_mthca (async)
snip...

We'd like to be able to spread the irq 67 interrupts across several or  
all CPUs but

echo f0 > /proc/irq/67/smp_affinity

seems to have no effect.    We  can move the interrupts to any  
*single* cpu we like but we cannot distribute them among multiple  
cpus.   The results is that under heavy load from the ethernet-only  
clients, the IPoIB traffic becomes interrupt bound.

We found a site (http://www.alexonlinux.com/smp-affinity-and-proper-interrupt-handling-in-linux 
)

that suggests that disabling CONFIG_HOTPLUG_CPU will allow /proc/irq/ 
nn/smp_affinity to work as expected.     My efforts so far with the  
2.6.18-92 (RH 5.2) and the 2.6.18-128 kernels have resulted in kernels  
that don't boot (after simply configuring CONFIG_HOTPLUG_CPU=n).

I've googled around a bit but have not  come up with much.    I  
thought I'd ask here since we are specifically interested in being  
able to do this for our IB HCAs.

Is this a known issue and is there a way to make it work?

Thanks,

Charlie Taylor
UF HPC Center

-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.openfabrics.org/pipermail/general/attachments/20090601/35dc605a/attachment.html>