[ofa-general] performance drop for datagram mode with the newconnectx FW

[Suresh Shelvapille]

I get about 520MB with the latest FW 2.5.0, it is definitely worse than 2.3.0


> -----Original Message-----
> From: Suresh Shelvapille [mailto:suri at baymicrosystems.com]
> Sent: Tuesday, June 24, 2008 12:01 PM
> To: 'Eli Cohen'; 'Or Gerlitz'
> Cc: 'general at lists.openfabrics.org'
> Subject: RE: [ofa-general] performance drop for datagram mode with the newconnectx FW
> 
> 
> I also get about 650MB for datagram mode with 2.3 FW.
> 
> > -----Original Message-----
> > From: general-bounces at lists.openfabrics.org [mailto:general-bounces at lists.openfabrics.org] On
> Behalf
> > Of Eli Cohen
> > Sent: Monday, June 23, 2008 7:43 AM
> > To: Or Gerlitz
> > Cc: Eli Cohen; Roland Dreier; general at lists.openfabrics.org
> > Subject: Re: [ofa-general] performance drop for datagram mode with the newconnectx FW
> >
> > On Mon, Jun 23, 2008 at 02:36:39PM +0300, Or Gerlitz wrote:
> > > Eli,
> > >
> > > Using the new connectx FW (2.5), I see performance drop to almost
> > > zero with ipoib datagram mode. The code that runs on these systems
> > > is ofed 1.3 and not mainline kernel, details below.
> > >
> > > Running netperf With connected mode (64k MTU) I get about 950MB/s
> > > where with datagram mode (2k MTU) I get only 20-40MB/s. I used to
> > > see about 650MB/s and above with FW 2.3 and datagram mode. Not that
> > > it could explain the drop, but the NIC reports to the OS stateless
> > > offload support - /sys/class/net/ib1/features is 0x11423
> > >
> > > I have opened the ipoib and mlx4 debug prints, and I don't see anything
> > > special other then the dmesg get quite filled with
> > >
> > > 	ib1: TX ring full, stopping kernel net queue
> > >
> > > any idea what can explain this? ibv_ud_pingpong gives about 2Gb/s which
> > > is about five times what I see with ipoib.
> > >
> > >
> > > Or.
> > >
> > > git://git.openfabrics.org/ofed_1_3/linux-2.6.git ofed_kernel
> > > commit 564e9e9383272f4311fd87ff4e5447cfcebad73a
> > >
> > > # uname -a
> > > Linux gen2-1 2.6.18-53.el5 #1 SMP Wed Oct 10 16:34:19 EDT 2007 x86_64 x86_64 x86_64 GNU/Linux
> > >
> > > # cat /etc/redhat-release
> > > Red Hat Enterprise Linux Server release 5.1 (Tikanga)
> >
> > Can you tell if changing the FW to 2.3 gives more reasonable results?
> > I don't believe such a drop in performance would have passed the QA
> > tests but I'll check that.
> > _______________________________________________
> > general mailing list
> > general at lists.openfabrics.org
> > http://lists.openfabrics.org/cgi-bin/mailman/listinfo/general
> >
> > To unsubscribe, please visit http://openib.org/mailman/listinfo/openib-general