[openib-general] ip over ib throughtput

[Grant Grundler]

On Tue, May 09, 2006 at 12:41:30PM -0700, Shirley Ma wrote:
> Grant Grundler <iod00d at hp.com> wrote on 05/09/2006 12:05:38 PM:
> 
> > On Tue, May 09, 2006 at 11:36:45AM -0700, Shirley Ma wrote:
> > > What throughput did you get on two CPUs?
> > 
> > With one CPU, I get ~2.5-2.8 Gb/s. With two CPUs: 3.5-3.6 Gb/s.
> > The last SVN version I tested was 2.6.15 + r4929 (several monthes ago).
> 
> I got 3.5-3.66Gb/s on UP with splitting CQ + threads supports on send/recv 
> CQ polling, and didn't get any better on SMP.

I was testing with a 3 year old 1.5 Ghz IA64 box.
That might explain the difference between our results.

> > To be clear, "one CPU" means the netperf process is bound to the same
> > CPU as the one handling mthca interrupts with taskset. Ditto for 
> netserver
> > on the other system. "Two CPU" means bind netperf/netserver processes
> > to a different CPU that is NOT handling the mthca interrupt.
> 
> I tried interrupts affinity on mthca, it didn't work by changing 
> /proc/irq/XXX/smp_affinity. The interrupts distruction kept bouncing 
> between there two CPUs. 

That sounds like a bug in the Local xAPIC and/or arch specific code.
Setting the smp_affinity mask should bind the IRQ to a specific CPU.

> What did you do to handle mthca interrupt on a fixed cpu?

Used an HP IA64 ZX1 machine. HP IA64 chipsets do not implement
"automatic" (read HW level) interrupt redirection. IIRC, it
was the "XPR" functionality in the Local xAPIC that HP didn't
implement (by design). Should be some way in SW to avoid that
too. I don't know offhand what it is though.

grant