[ewg] Re: [ofa-general] Running netperf and iperf over SDP

[Rick Jones]

Moshe Kazir wrote:
> While running netperf and iperf over SDP I get unstable Performance
> results.
> 
> On iperf I get more then 25 % difference between minimum and maximum.
> 
> On netperf I get the following amazing result. ->
> 
> ++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
>  # while   LD_PRELOAD=/usr/lib64/libsdp.so ./netperf  -H 192.168.7.172
> -- -m 512 -M 1047 ; do echo . ; done
> TCP STREAM TEST to 192.168.7.172
> Recv   Send    Send
> Socket Socket  Message  Elapsed
> Size   Size    Size     Time     Throughput
> bytes  bytes   bytes    secs.    10^6bits/sec
> 
> 126976 126976    512    10.00    3247.39
> .
> TCP STREAM TEST to 192.168.7.172
> Recv   Send    Send
> Socket Socket  Message  Elapsed
> Size   Size    Size     Time     Throughput
> bytes  bytes   bytes    secs.    10^6bits/sec
> 
> 126976 126976    512    10.00    1222.48
> .
> ++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
> 
> Can some one spar me a hint ?
> What I'm doing wrong ?

I would start by telling netperf you want CPU utilization measured.  That way we 
can see if there is a correlation between the CPU utilization and the throughput.

Also, if this were a "pure" TCP test, you would have a race between the nagle 
algorithm and the speed at which ACK's come-back from the receiver affecting the 
distribution of TCP segment sizes being transmitted since your send size is so 
much smaller than the MTU of the link.  IIRC for IPoIB in 1.2.mumble the MTU is 
65520 or something like that.  You might consider taking snapshots of the 
link-level statistics (does ethtool -S work for an IB interface?) from before 
and after each netperf test and run them through beforeafter:

ftp://ftp.cup.hp.com/dist/networking/tools/

You might also experiment with setting TCP_NODELAY - although since this is 
LD_PRELOADED SDP I'm not sure what that really means/does.

Any particular reason you are telling the netperf side to post 1047 byte 
receives when you are making 512 byte calls to send()?

> 
> The test run on x86_64 , sles 10 sp1 , OFED-1.2.5  

I'm guessing you have multiple cores - how do interrutps from the HCA (?) get 
distributed?  What happens when you use the -T option of netperf to vary the CPU 
binding of either netperf or netserver:

netperf -T N,M    #bind netperf to CPU N, netserver to CPU M
netperf -T N,     #just bind netperf to CPU N, netserver unbound
netperf -T  ,M    #netperf unbound, netserver bound to CPU M

relative to where the interrupts from the HCA go?

Finally, well for now :), there are "direct" SDP tests in netperf.  Make sure 
you are on say 2.4.4:

http://www.netperf.org/svn/netperf2/tags/netperf-2.4.4/   or
ftp://ftp.netperf.org/netperf

and add --enable-sdb to the ./configure command.

happy benchmarking,

rick jones