<html><body style="word-wrap: break-word; -webkit-nbsp-mode: space; -webkit-line-break: after-white-space; "><div><br></div><div>We have servers that have both IB and ethernet-only clients. The ethernet-only clients access the servers via IPoIB through a gateway. For small MTUs, the IpoIB software interrupt load can get pretty unmanageable so we would like to be able to spread the load over multiple cores. Our current /proc/interrupts looks like...</div><div><br></div><div><div><font class="Apple-style-span" face="Courier">cat /proc/interrupts </font></div><div><font class="Apple-style-span" face="Courier"> CPU0 CPU1 CPU2 CPU3 CPU4 CPU5 CPU6 CPU7 </font></div><div><font class="Apple-style-span" face="Courier">snip...</font></div><div><font class="Apple-style-span" face="Courier"> 67: 946 0 0 0 1364412993 0 0 0 PCI-MSI-X ib_mthca (comp)</font></div><div><font class="Apple-style-span" face="Courier"> 75: 1785 0 0 0 6286882 0 0 0 PCI-MSI-X ib_mthca (async)</font></div><div><font class="Apple-style-span" face="Courier">snip...</font></div><div><font class="Apple-style-span" face="Courier"><br></font></div><div><font class="Apple-style-span" face="Courier">We'd like to be able to spread the irq 67 interrupts across several or all CPUs but </font></div><div><font class="Apple-style-span" face="Courier"><br></font></div><div><font class="Apple-style-span" face="Courier">echo f0 > <span class="Apple-style-span" style="font-family: Helvetica; ">/proc/irq/67/smp_affinity</span></font></div><div><br></div><div>seems to have no effect. We can move the interrupts to any *single* cpu we like but we cannot distribute them among multiple cpus. The results is that under heavy load from the ethernet-only clients, the IPoIB traffic becomes interrupt bound. </div><div><br></div><div>We found a site (<a href="http://www.alexonlinux.com/smp-affinity-and-proper-interrupt-handling-in-linux">http://www.alexonlinux.com/smp-affinity-and-proper-interrupt-handling-in-linux)</a></div><div><br></div><div>that suggests that disabling CONFIG_HOTPLUG_CPU will allow /proc/irq/nn/smp_affinity to work as expected. My efforts so far with the 2.6.18-92 (RH 5.2) and the 2.6.18-128 kernels have resulted in kernels that don't boot (after simply configuring CONFIG_HOTPLUG_CPU=n).</div><div><br></div><div>I've googled around a bit but have not come up with much. I thought I'd ask here since we are specifically interested in being able to do this for our IB HCAs. </div><div><br></div><div>Is this a known issue and is there a way to make it work?</div><div><br></div><div>Thanks,</div><div><br></div><div>Charlie Taylor</div><div>UF HPC Center</div><div><br></div></div></body></html>