[ofa-general] Re: [PATCH 0/5]: Improve small UDP messages

[Eli Cohen]

On Thu, 2008-01-31 at 21:57 -0800, Shirley Ma wrote:
> Hello Eli,
> 
> 	I am going to verify this in our lab. I assume this patchset is built
> against 2.6.25?

Well, this patch is against the ofed 1.3 tree.
> 
> On Fri, 2008-02-01 at 15:40 +0200, Eli Cohen wrote:
> > The following patches, based on ofed 1.3, are intended to address bugs
> > https://bugs.openfabrics.org/show_bug.cgi?id=760 and
> > https://bugs.openfabrics.org/show_bug.cgi?id=761. They address UD mode
> > both send and receive and improve performance when using small
> > messages
> > UDP traffic. The observation we had is that at small UDP messages, the
> > message rate is high and so what limits throughput is CPU, e.g. CPU is
> > 100% busy.
> What's the configuration for this test? How many CPUs?

The test setup is two machines back to back. One is running netserver
and the other runs:
netperf -H <remote ip address> -t UDP_STREAM -- -m 128

> 
> > In the send flow I use a dedicated CQ for the send flow which in turn
> > is
> > never armed. CQEs consumption is done by polling after posting a send
> > message. Also, the QP is configured for selective signaling and
> > polling
> > the CQ is done once in 16 messages.
> 
> I did see selective signaling impact the performance. Depends on how
> many packets you want to pull once, the performance could be good, could
> be worse from my experience based on how many you want to pull once. I
> haven't fully understood this yet. I have a similar patch. But why you
> pick up 16 messages?

I chose this number since it showed good results. It is a macro defined
in ipoib.h and can be changed to other values.

> 
> > On the receive side the code is changed to post to receive queue once
> > in
> > 16 completions. This is done in for both UD and and CM.
> 
> Ohmm, have you tested latency? I think it will increase latency for
> small messages.
> 

I see also improvement in latency. I used netperf -t UDP_RR and -t
TCP_RR and in both cases I improvement.