[openib-general] ip over ib throughtput

[Grant Grundler]

On Thu, Jan 13, 2005 at 08:43:40AM -0800, Woodruff, Robert J wrote:
> I found this paper about 10gig ethernet on Itanium where they were
> not seeing that good of performance and were able to increase it by
> changing some BIOS setting that controlled PCI-X burst size.

Yeah, I have mail from April 2003 discussing MMRBC issues with CERN.
Unfortunately, the paper doesn't describe how to change it and
I had to look that up again:
	setpci -d 15b3:5a44 0x72	# read low byte of PCI-X command

lspci tells me the PCI-X capability starts at offset 0x70 for this device.
0x72 is the low byte of the PCI-X command field.
"lspci -vv" as root decodes it for me:
	        Capabilities: [70] PCI-X non-bridge device.
			Command: DPERE+ ERO+ RBC=3 OST=4

RBC=3 means it's already set to 4k block. :^(
But if it weren't, I could modify it with:
	setpci -d 15b3:5a44 0x72.W=0x1c


Firmware (by default) sets "Command" field to 0x4e.

But when I "modprobe ibmthca" and run "lspci -vv":
                Command: DPERE- ERO- RBC=3 OST=1

Hrm..setting 0x72 to 0x1f (was 0x1c) gives me:
                Command: DPERE+ ERO+ RBC=3 OST=1

ISTR the ZX1 chipset ignores ERO settings. And the driver isn't going
to handle Data parity errors...so 0x72 can stay 0x1c.

Rerun with mthca using IRQ lines and this setting: 1357 Mbps
Reload mthca with msi_x=1 and same settings:       1575 Mbps

I wasn't using taskset and expect thats' why I'm not getting 1590 like before.
Anyway, I expect core problem is something to do with CPU behavior.
I'll be trying to chase this more with pfmon.

I also wondered why the OST (Outstanding Split Transactions)
was reduced to 1 (firmware sets this to 4). Bumping it back to 4
didn't help either.

> Perhaps this might also help IB, although they state in the paper
> that this seemed to help more with larger MTUs, so with our 2K MTU,
> not sure if it will help or not.

MTU size helps a bit.
Using 4k or larger gives me ~1590 Mbps vs ~1560 for 2k MTU.
But I don't see improvement for larger MTUs.

hth,
grant