[ofa-general] performance drop for datagram mode with the new connectx FW

[Oren Meron]

Or Gerlitz wrote:
> Eli,
> 
> Using the new connectx FW (2.5), I see performance drop to almost
> zero with ipoib datagram mode. The code that runs on these systems
> is ofed 1.3 and not mainline kernel, details below.
> 
> Running netperf With connected mode (64k MTU) I get about 950MB/s
> where with datagram mode (2k MTU) I get only 20-40MB/s. I used to
> see about 650MB/s and above with FW 2.3 and datagram mode. Not that
> it could explain the drop, but the NIC reports to the OS stateless
> offload support - /sys/class/net/ib1/features is 0x11423
> 
> I have opened the ipoib and mlx4 debug prints, and I don't see anything
> special other then the dmesg get quite filled with
> 
> 	ib1: TX ring full, stopping kernel net queue
> 
> any idea what can explain this? ibv_ud_pingpong gives about 2Gb/s which
> is about five times what I see with ipoib.
> 
> 
> Or.
> 
> git://git.openfabrics.org/ofed_1_3/linux-2.6.git ofed_kernel
> commit 564e9e9383272f4311fd87ff4e5447cfcebad73a
> 
> # uname -a
> Linux gen2-1 2.6.18-53.el5 #1 SMP Wed Oct 10 16:34:19 EDT 2007 x86_64 x86_64 x86_64 GNU/Linux
> 
> # cat /etc/redhat-release
> Red Hat Enterprise Linux Server release 5.1 (Tikanga)
> _______________________________________________
> general mailing list
> general at lists.openfabrics.org
> http://lists.openfabrics.org/cgi-bin/mailman/listinfo/general
> 
> To unsubscribe, please visit http://openib.org/mailman/listinfo/openib-general
> 
Hi Or,
Trying to reproduce it i encountered the opposite result:
datagram mode produced perfectly normal results (450-600 MBytes/s),
connected mode produced extremely low results (20-30 MBytes/s).
run parameters:
dell intel servers, sles10, fw 2.5.0, MLNX_OFED_LINUX-1.3.1 

will look into it with eli.

oren