[openib-general] Performance optimization

[Grant Grundler]

On Mon, Feb 27, 2006 at 06:28:18PM -0800, Frithjof Kruggel wrote:
> 
> Hi,
> 
> thank you for your advice. However, using
> 
> taskset 0x1 ibv_uc_pingpong xxxxx

Is this test a bandwidth test or a latency test?
Anything that says "pingpong" suggests latency to me.

> did not report rates beyond 1850 MBit/s.
> If I raise the size (--size 65536), rates are about
> 3500 MBit/s. There is no significant difference if
> I try this within the same node or across node,
> with or without a switch.

Well, that suggests the system is CPU bound or
bottlenecking on control traffic somehow.
Fairly normal for a latency test.

What is the PCI-X bus frequency?
(Check HW Owners Guide or system data sheet)
3500 Mbit/s would be about right for a 66Mhz/64-bit bus.

Is MSI or MSI-X enabled?
Can you remind me which model of box/motherboard is being used?
(ISTR Opteron and the issues older Operton can't support MSI/MSI-X.
That will cause substantial performance penalty on smaller transactions
that involve lots of interrupts).

> I would like to know whether there is any step-by-step
> list (even checking BIOS parameters?) in order to
> understand where something might go wrong.

I'm not aware of one since so many different things impact
performance. Everything from DIMM types, PCI bus frequencies,
CPU cache size, chipset features (e.g. write coalescing or MSI
support), to how workloads have different cache "footprint".

sorry that it's not so simple.

grant