[ofa-general] performance drop for datagram mode with the new connectx FW

[Eli Cohen]

On Mon, Jun 23, 2008 at 02:36:39PM +0300, Or Gerlitz wrote:
> Eli,
> 
> Using the new connectx FW (2.5), I see performance drop to almost
> zero with ipoib datagram mode. The code that runs on these systems
> is ofed 1.3 and not mainline kernel, details below.
> 
> Running netperf With connected mode (64k MTU) I get about 950MB/s
> where with datagram mode (2k MTU) I get only 20-40MB/s. I used to
> see about 650MB/s and above with FW 2.3 and datagram mode. Not that
> it could explain the drop, but the NIC reports to the OS stateless
> offload support - /sys/class/net/ib1/features is 0x11423
> 
> I have opened the ipoib and mlx4 debug prints, and I don't see anything
> special other then the dmesg get quite filled with
> 
> 	ib1: TX ring full, stopping kernel net queue
> 
> any idea what can explain this? ibv_ud_pingpong gives about 2Gb/s which
> is about five times what I see with ipoib.
> 
> 
> Or.
> 
> git://git.openfabrics.org/ofed_1_3/linux-2.6.git ofed_kernel
> commit 564e9e9383272f4311fd87ff4e5447cfcebad73a
> 
> # uname -a
> Linux gen2-1 2.6.18-53.el5 #1 SMP Wed Oct 10 16:34:19 EDT 2007 x86_64 x86_64 x86_64 GNU/Linux
> 
> # cat /etc/redhat-release
> Red Hat Enterprise Linux Server release 5.1 (Tikanga)

Can you tell if changing the FW to 2.3 gives more reasonable results?
I don't believe such a drop in performance would have passed the QA
tests but I'll check that.