[openib-general] ip over ib throughtput

[Grant Grundler]

On Mon, May 08, 2006 at 05:52:17PM -0500, Hassan M. Jafri wrote:
> I cant crank out more than 150 MB/sec with my 2.0 GHz xeons. verbs level 
> benchmarks, however give decent numbers for bandwidth. With netperf, the 
> server side CPU usage is 99% which is much higher than other posted 
> bandwidth results on this thread. Any suggestions?
> 
> Here is the complete configuration for my bandwidth tests
> 
> Kernel-2.6.15.4
> netperf-2.3-3

FWIW, 2.4.1 is the latest netperf version.
See http://www.netperf.org/svn/netperf2

> OpenIB rev 6552
> MTLP23108-CF128
> Firmware 3.4.0

I was using 3.3.3...looks like I should update.

> MSI-X is enabled for the HCA
>
> ------------------------------
> Here is the netperf output
> 
> 
> 
> TCP STREAM TEST to 192.168.2.2
> Recv   Send    Send                          Utilization       Service 
> Demand
> Socket Socket  Message  Elapsed              Send     Recv     Send    Recv
> Size   Size    Size     Time     Throughput  local    remote   local 
> remote
> bytes  bytes   bytes    secs.    MBytes  /s  % T      % T      us/KB   us/KB
> 
> 262142 262142  32768    10.01       151.32   59.66    99.84    7.700 
> 12.886

I think Michael is probably right - NETFILTER or something else
is burning additional CPU cycles.  The "Send local" service demand
is nearly 2X of what I was seeing a few monthes back. You need to
use perfmon or oprofile to figure out where the time is being spent.

You could also test SDP (see openib.org for notes on LD_PRELOAD=libsdp.so)
to see how that performs - doesn't use NETFILTER.

You might also try the netperf "-T" option to bind the netserver
process to a different CPU.  If two CPUs are available, IPoIB throughput
is better with the interrupts handled on a different CPU than the one
handling the data.

grant