[ofa-general] performance drop for datagram mode with the new connectx FW

[Or Gerlitz]

Eli,

Using the new connectx FW (2.5), I see performance drop to almost
zero with ipoib datagram mode. The code that runs on these systems
is ofed 1.3 and not mainline kernel, details below.

Running netperf With connected mode (64k MTU) I get about 950MB/s
where with datagram mode (2k MTU) I get only 20-40MB/s. I used to
see about 650MB/s and above with FW 2.3 and datagram mode. Not that
it could explain the drop, but the NIC reports to the OS stateless
offload support - /sys/class/net/ib1/features is 0x11423

I have opened the ipoib and mlx4 debug prints, and I don't see anything
special other then the dmesg get quite filled with

	ib1: TX ring full, stopping kernel net queue

any idea what can explain this? ibv_ud_pingpong gives about 2Gb/s which
is about five times what I see with ipoib.


Or.

git://git.openfabrics.org/ofed_1_3/linux-2.6.git ofed_kernel
commit 564e9e9383272f4311fd87ff4e5447cfcebad73a

# uname -a
Linux gen2-1 2.6.18-53.el5 #1 SMP Wed Oct 10 16:34:19 EDT 2007 x86_64 x86_64 x86_64 GNU/Linux

# cat /etc/redhat-release
Red Hat Enterprise Linux Server release 5.1 (Tikanga)