[ofa-general] RE: [ewg] Not seeing any SDP performance changes
Jim Mott
jimmott at austin.rr.com
Thu Jan 24 10:29:27 PST 2008
Attached is the output of a run of the shell script I described. The source machine was the big 2-socket dual core, what the target
(running netserver) was the AMD 6000+ system.
I know it is a little dense, but the format of the data is pretty self explanatory.
In this run you see that for the 64 byte latency test:
SDP 59628.17 TPS with 41.4 uS/transaction (Local 22.604 + Remote 18.796)
IP 17703.55 TPS with 50.6 uS/transaction (Local 15.680 + Remote 34.380)
SDP provided about 3.37x total transactions/second with 18% less CPU/transaction.
On the Bandwidth side for 1,000,000 byte messages you get:
IP-CM 3802.80 Mb/sec with 2.4 uS/KB (local 1.297 + remote 1.103)
SDP 6841.07 Mb/sec with 2.6 uS/KB (local 1.346 + remote 1.217)
Bzcopy 10573.41 Mb/sec with 1.5 uS/KB (local 0.733 + remote 0.796)
BZcopy provided about 2.8x the bandwidth of IPoIB-CM with 38% less CPU/Kb.
Bzcopy provided about 1.5x the bandwidth of SDP with 42% less CPU/Kb.
These are the sorts of numbers I keep getting.
# date ==> Thu Jan 24 11:38:37 CST 2008
# Tests run for 60 seconds
# Source IP is 193.168.10.198 remote IP is 193.168.10.125
# Local OS is 2.6.9-42.ELsmp x86_64
# sdp_zcopy_thresh is 0
ofa_1_3_dev-20080121-0855
User:
libibverbs:
git://git.openfabrics.org/ofed_1_3/libibverbs.git ofed_1_3
commit 018c44a44ff0344dfe7cf5f6598f81d81769164e
libmthca:
git://git.openfabrics.org/ofed_1_3/libmthca.git ofed_1_3
commit ec00c5b0887888bb62515961205a1e6e61bfea5f
libmlx4:
git://git.openfabrics.org/ofed_1_3/libmlx4.git ofed_1_3
commit e3b9b75bdb024cf4af9ecac96b5aa14488ea5c72
libehca:
git://git.openfabrics.org/ofed_1_3/libehca.git ofed_1_3
commit f159085910b42d7118e536b0ad40b8fc2b8e5c27
libipathverbs:
git://git.openfabrics.org/~ralphc/libipathverbs/.git master
commit d47f13b02acab6129e719155f4f90d743229685a
libcxgb3:
git://git.openfabrics.org/ofed_1_3/libcxgb3.git ofed_1_3
commit 10893a7a45e6913483d90023bb90bf9cb2420384
libnes:
git://git.openfabrics.org/ofed_1_3/libnes.git ofed_1_3
commit 27ebf415cb65918237c7c21fd8b28bc1dbf4fca6
libibcm:
git://git.openfabrics.org/~shefty/libibcm.git master
commit a45e43483ac29a26c1803f217ca21a07534494c4
librdmacm:
git://git.openfabrics.org/ofed_1_3/librdmacm.git ofed_1_3
commit afe87c16f40fe4a3622f231672737950c0ebf9fa
dapl:
git://git.openfabrics.org/~ardavis/dapl.git master
commit 6dcf1763c153c27c29ba76bac35be4f6935ddd96
libsdp:
git://git.openfabrics.org/ofed_1_3/libsdp.git ofed_1_3
commit 47801f8f1e2168c34690b93edaccadc2ece936ef
sdpnetstat:
git://git.openfabrics.org/ofed_1_3/sdpnetstat.git ofed_1_3
commit 3341620a7259c4f7bdd4180864b98e260c3dc223
srptools:
git://git.openfabrics.org/~ishai/srptools.git master
commit 79ce808b9e181559c08495e1698c58bd49155ae4
perftest:
git://git.openfabrics.org/~tziporet/perftest.git master
commit 07343734cb4ae15ea5b6aaabcd3ad57e2a36806b
qlvnictools:
git://git.openfabrics.org/ofed_1_3/qlvnictools.git ofed_1_3
commit 41a148393a602810df80109e71086970a91c1d8d
tvflash:
git://git.openfabrics.org/~rdreier/tvflash.git master
commit 39a63301f0344b6b3d45bc4b16d76be81f4377c0
mstflint:
git://git.openfabrics.org/~orenk/mstflint.git master
commit 3c711303e6474186920a24aadcc262f6fa6c9177
qperf:
git://git.openfabrics.org/ofed_1_3/qperf.git ofed_1_3
commit 317ca959ec2bd978e6c51fd304ac546fdf5397d8
management:
git://git.openfabrics.org/ofed_1_3/management.git ofed_1_3
commit 88b853b0f2a463a8335f7451c809d5ae5d1e14ee
ibutils:
git://git.openfabrics.org/~orenk/ibutils.git master
commit 0225143c82416d02d6f00cf93bb0f38915557a12
imgen:
git://git.openfabrics.org/~mst/imgen.git master
commit a309109bebcc1ae94720c6bb8be5b0b974b93324
ofed_scripts:
git://git.openfabrics.org/ofed_1_3/ofascripts.git ofed_1_3
commit efcd4c7ab3d3a4b0b8d75165d101825cb73093ff
Kernel:
Git:
git://git.openfabrics.org/ofed_1_3/linux-2.6.git ofed_kernel
commit dbdcdc7b6c699f8634c58f18e83c1d2caa75f4c2
CA 'mlx4_0'
CA type: MT25418
Number of ports: 2
Firmware version: 2.3.914
Hardware version: 0
Node GUID: 0x0002c90300002078
System image GUID: 0x0002c9030000207b
Port 1:
State: Active
Physical state: LinkUp
Rate: 20
Base lid: 1
LMC: 0
SM lid: 8
Capability mask: 0x02510868
Port GUID: 0x0002c90300002079
Port 2:
State: Down
Physical state: Polling
Rate: 10
Base lid: 0
LMC: 0
SM lid: 0
Capability mask: 0x02510868
Port GUID: 0x0002c9030000207a
# Format of latency record
# LAT, mode, sdp, size, transaction/sec, local CPU, rmt CPU, local uS/tr, rmt uS/tr
# Format of bandwith record
# BW, mode, sdp, size, bandwith (Mb/sec), local CPU, rmt CPU, local uS/KB, rmt uS/KB
----- Start -----
LAT, IPoIB, No SDP, 64, 17703.55, 3.47, 30.43, 15.680, 34.380
LAT, IPoIB-CM, No SDP, 64, 16147.11, 3.48, 29.73, 17.220, 36.826
LAT, IPoIB-CM, SDP, 64, 59628.17, 16.85, 56.04, 22.604, 18.796
BW, IPoIB, No SDP, 64, 731.88, 15.49, 29.44, 13.873, 6.592
BW, IPoIB-CM, No SDP, 64, 832.68, 13.91, 21.11, 10.950, 4.155
BW, IPoIB-CM, SDP, 64, 149.59, 17.22, 54.04, 75.452, 59.186
BW, IPoIB-CM, SDP-Bzcopy(64), 64, 75.73, 16.62, 56.30, 143.795, 121.818
LAT, IPoIB, No SDP, 128, 17531.94, 2.60, 28.50, 11.855, 32.514
LAT, IPoIB-CM, No SDP, 128, 16148.53, 2.42, 29.23, 11.994, 36.205
LAT, IPoIB-CM, SDP, 128, 59596.38, 15.87, 55.70, 21.305, 18.694
BW, IPoIB, No SDP, 128, 1500.18, 13.61, 35.66, 5.944, 3.895
BW, IPoIB-CM, No SDP, 128, 1843.86, 13.44, 28.70, 4.776, 2.551
BW, IPoIB-CM, SDP, 128, 312.79, 17.96, 52.81, 37.625, 27.664
BW, IPoIB-CM, SDP-Bzcopy(128), 128, 137.33, 17.18, 55.45, 82.003, 66.151
LAT, IPoIB, No SDP, 512, 15950.67, 3.56, 28.34, 17.840, 35.533
LAT, IPoIB-CM, No SDP, 512, 15248.76, 3.53, 30.09, 18.534, 39.466
LAT, IPoIB-CM, SDP, 512, 58716.25, 15.46, 55.70, 21.061, 18.974
BW, IPoIB, No SDP, 512, 2649.21, 13.25, 40.64, 3.277, 2.513
BW, IPoIB-CM, No SDP, 512, 3672.23, 12.55, 28.03, 2.239, 1.251
BW, IPoIB-CM, SDP, 512, 1606.05, 17.83, 51.98, 7.277, 5.303
BW, IPoIB-CM, SDP-Bzcopy(512), 512, 531.12, 17.24, 55.45, 21.276, 17.104
LAT, IPoIB, No SDP, 1024, 15460.70, 3.41, 27.34, 17.668, 35.366
LAT, IPoIB-CM, No SDP, 1024, 14610.25, 3.52, 29.12, 19.247, 39.856
LAT, IPoIB-CM, SDP, 1024, 52837.54, 14.88, 55.30, 22.526, 20.933
BW, IPoIB, No SDP, 1024, 3202.60, 12.85, 44.53, 2.630, 2.278
BW, IPoIB-CM, No SDP, 1024, 4323.63, 12.02, 30.21, 1.822, 1.145
BW, IPoIB-CM, SDP, 1024, 2670.95, 17.53, 47.30, 4.302, 2.901
BW, IPoIB-CM, SDP-Bzcopy(1024), 1024, 957.09, 16.81, 54.80, 11.511, 9.380
LAT, IPoIB, No SDP, 2048, 13176.44, 4.21, 31.66, 25.547, 48.053
LAT, IPoIB-CM, No SDP, 2048, 13654.42, 3.10, 27.27, 18.165, 39.936
LAT, IPoIB-CM, SDP, 2048, 41861.31, 15.71, 54.07, 30.031, 25.831
BW, IPoIB, No SDP, 2048, 3375.68, 13.23, 44.40, 2.568, 2.155
BW, IPoIB-CM, No SDP, 2048, 4790.28, 10.69, 32.74, 1.463, 1.120
BW, IPoIB-CM, SDP, 2048, 3066.77, 18.10, 44.72, 3.868, 2.389
BW, IPoIB-CM, SDP-Bzcopy(2048), 2048, 1456.18, 14.98, 54.94, 6.744, 6.181
LAT, IPoIB, No SDP, 4096, 11710.43, 4.33, 29.50, 29.592, 50.377
LAT, IPoIB-CM, No SDP, 4096, 11723.73, 3.79, 28.05, 25.878, 47.855
LAT, IPoIB-CM, SDP, 4096, 31206.42, 14.57, 54.01, 37.355, 34.616
BW, IPoIB, No SDP, 4096, 3564.14, 12.81, 45.62, 2.356, 2.097
BW, IPoIB-CM, No SDP, 4096, 4711.42, 9.81, 33.34, 1.365, 1.159
BW, IPoIB-CM, SDP, 4096, 4036.12, 17.60, 48.64, 2.857, 1.974
BW, IPoIB-CM, SDP-Bzcopy(4096), 4096, 2716.29, 14.92, 54.36, 3.600, 3.279
LAT, IPoIB, No SDP, 8192, 9327.63, 5.38, 29.40, 46.101, 63.036
LAT, IPoIB-CM, No SDP, 8192, 9585.17, 3.50, 26.79, 29.188, 55.891
LAT, IPoIB-CM, SDP, 8192, 21154.69, 14.24, 54.07, 53.849, 51.120
BW, IPoIB, No SDP, 8192, 3936.99, 12.67, 47.47, 2.109, 1.975
BW, IPoIB-CM, No SDP, 8192, 4716.53, 9.58, 33.50, 1.330, 1.164
BW, IPoIB-CM, SDP, 8192, 5688.79, 15.64, 51.24, 1.802, 1.476
BW, IPoIB-CM, SDP-Bzcopy(8192), 8192, 4582.61, 14.39, 53.10, 2.058, 1.898
LAT, IPoIB, No SDP, 16000, 7486.21, 7.13, 37.44, 76.157, 100.034
LAT, IPoIB-CM, No SDP, 16000, 7328.59, 2.69, 26.01, 29.416, 70.975
LAT, IPoIB-CM, SDP, 16000, 13124.20, 14.93, 53.32, 91.027, 81.259
BW, IPoIB, No SDP, 16000, 3621.30, 13.00, 46.12, 2.353, 2.087
BW, IPoIB-CM, No SDP, 16000, 5281.13, 9.55, 35.28, 1.186, 1.095
BW, IPoIB-CM, SDP, 16000, 6851.09, 15.01, 52.01, 1.436, 1.244
BW, IPoIB-CM, SDP-Bzcopy(16000), 16000, 6633.81, 14.23, 52.49, 1.406, 1.296
LAT, IPoIB, No SDP, 32768, 4328.65, 7.04, 33.37, 130.028, 154.172
LAT, IPoIB-CM, No SDP, 32768, 4540.68, 3.63, 25.96, 63.984, 114.345
LAT, IPoIB-CM, SDP, 32768, 8449.69, 13.97, 53.27, 132.230, 126.086
BW, IPoIB, No SDP, 32768, 3932.05, 12.57, 47.40, 2.096, 1.975
BW, IPoIB-CM, No SDP, 32768, 4790.61, 9.34, 33.79, 1.278, 1.156
BW, IPoIB-CM, SDP, 32768, 7700.25, 15.04, 51.26, 1.280, 1.091
BW, IPoIB-CM, SDP-Bzcopy(32768), 32768, 7552.72, 13.40, 52.51, 1.163, 1.139
LAT, IPoIB, No SDP, 65536, 2821.14, 8.01, 34.68, 227.079, 245.879
LAT, IPoIB-CM, No SDP, 65536, 2494.38, 3.52, 30.69, 112.742, 246.042
LAT, IPoIB-CM, SDP, 65536, 4972.83, 15.00, 52.92, 241.327, 212.849
BW, IPoIB, No SDP, 65536, 3759.02, 12.83, 46.25, 2.236, 2.016
BW, IPoIB-CM, No SDP, 65536, 4824.50, 8.37, 34.40, 1.138, 1.168
BW, IPoIB-CM, SDP, 65536, 8388.52, 13.00, 51.51, 1.015, 1.006
BW, IPoIB-CM, SDP-Bzcopy(65536), 65536, 9070.82, 13.10, 52.44, 0.946, 0.947
LAT, IPoIB, No SDP, 131072, 1941.87, 10.31, 38.14, 424.560, 392.832
LAT, IPoIB-CM, No SDP, 131072, 1559.19, 5.05, 38.05, 259.204, 488.052
LAT, IPoIB-CM, SDP, 131072, 2897.77, 12.64, 49.54, 349.040, 341.949
BW, IPoIB, No SDP, 131072, 3761.74, 12.54, 46.65, 2.185, 2.032
BW, IPoIB-CM, No SDP, 131072, 3749.70, 7.69, 27.40, 1.345, 1.197
BW, IPoIB-CM, SDP, 131072, 6462.24, 14.25, 51.22, 1.445, 1.299
BW, IPoIB-CM, SDP-Bzcopy(131072), 131072, 9807.54, 13.94, 52.07, 0.931, 0.870
LAT, IPoIB, No SDP, 1000000, 203.55, 11.29, 42.55, 4437.527, 4180.463
LAT, IPoIB-CM, No SDP, 1000000, 164.28, 4.53, 35.14, 2205.818, 4278.296
LAT, IPoIB-CM, SDP, 1000000, 470.33, 13.48, 49.18, 2292.254, 2091.181
BW, IPoIB, No SDP, 1000000, 3795.27, 12.94, 46.85, 2.234, 2.022
BW, IPoIB-CM, No SDP, 1000000, 3802.80, 7.52, 25.61, 1.297, 1.103
BW, IPoIB-CM, SDP, 1000000, 6841.07, 14.05, 50.82, 1.346, 1.217
BW, IPoIB-CM, SDP-Bzcopy(1000000), 1000000, 10573.41, 11.82, 51.34, 0.733, 0.796
-----Original Message-----
From: general-bounces at lists.openfabrics.org [mailto:general-bounces at lists.openfabrics.org] On Behalf Of Jim Mott
Sent: Thursday, January 24, 2008 11:47 AM
To: Scott Weitzenkamp (sweitzen); Weikuan Yu
Cc: general at lists.openfabrics.org
Subject: RE: [ofa-general] RE: [ewg] Not seeing any SDP performance changes inOFED 1.3 beta, and I get Oops when enabling
sdp_zcopy_thresh
I am really puzzled. The majority of my testing has been between
Rhat4U4 and Rhat5. Using netperf command lines of the form:
netperf -C -c -P 0 -t TCP_RR -H 193.168.10.143 -l 60 ---r 64
netperf -C -c -P 0 -t TCP_STREAM -H 193.168.10.143 -l 60 ---r 1000000
and a process of:
- set sdp_zcopy_thresh=0, run bandwidth test
- set sdp_zcopy_thresh=size, run bandwidth test
I repeatedly get results that look like this:
size SDP Bzcopy
65536 7375.00 7515.98
131072 7465.70 8105.58
1000000 6541.87 9948.76
These numbers are from high end (2-socket, quad-core) machines. When
you
use smaller machines, like the AMD dual-core shown below, the
differences
between SDP with and without bzcopy are more striking.
The process to start the netserver is:
export LD_LIBRARY_PATH=/usr/local/ofed/lib64:/usr/local/ofed/lib
export LD_PRELOAD=libsdp.so
export LIBSDP_CONFIG_FILE=/etc/infiniband/libsdp.conf
netserver
The process to start the netperf is similar:
export LD_LIBRARY_PATH=/usr/local/ofed/lib64:/usr/local/ofed/lib
export LD_PRELOAD=libsdp.so
export LIBSDP_CONFIG_FILE=/etc/infiniband/libsdp.conf
netperf -C -c -P 0 -t TCP_STREAM -H 193.168.10.143 -l 60 ---r 1000000
You and unload and reload ib_sdp between tests, but I just echo 0 and
echo size into sdp_zcopy_thresh on the sending side. Note that it is
in a different place on Rhat4u4 and Rhat5.
My libsdp.conf is the default that ships with OFED. Stripping the
comments (grep -v), it is just:
log min-level 9 destination file libsdp.log
use both server * *:*
use both client * *:*
Note that if you build locally:
cd /tmp/openib_gen2/xxxx/ofa_1_3_dev_kernel
make install
the libsdp.conf file seems to get lost. You must restore it by
hand.
I have a shell script that automates this testing for a
wide range of message sizes:
64 128 512 1024 2048 4096 8192 16000 32768 65536 131072 1000000
on multiple transports:
IP both "echo datagram > /sys/class/net/ib0/mode"
IP-CM both "echo connected > /sys/class/net/ib0/mode"
SDP both
Bzcopy TCP_STREAM
Where both is TCP_RR and TCP_STREAM testing.
The variance in SDP bandwidth results can be 10%-15% between runs. The
difference between Bzcopy and non-Bzcopy is always very visible for 128K
and up tests though.
Could some other people please try to run some of these tests? If only
help me know if I am crazy?
Thanks,
JIm
Jim Mott
Mellanox Technologies Ltd.
mail: jim at mellanox.com
Phone: 512-294-5481
-----Original Message-----
From: Scott Weitzenkamp (sweitzen) [mailto:sweitzen at cisco.com]
Sent: Thursday, January 24, 2008 11:17 AM
To: Jim Mott; Weikuan Yu
Cc: ewg at lists.openfabrics.org; general at lists.openfabrics.org
Subject: RE: [ofa-general] RE: [ewg] Not seeing any SDP performance
changes inOFED 1.3 beta, and I get Oops when enabling sdp_zcopy_thresh
I've tested on RHEL4 and RHEL5, and see no sdp_zcopy_thresh improvement
for any message size, as measured with netperf, for any Arbel or
ConnectX HCA.
Scott
> -----Original Message-----
> From: Jim Mott [mailto:jim at mellanox.com]
> Sent: Thursday, January 24, 2008 7:57 AM
> To: Weikuan Yu; Scott Weitzenkamp (sweitzen)
> Cc: ewg at lists.openfabrics.org; general at lists.openfabrics.org
> Subject: RE: [ofa-general] RE: [ewg] Not seeing any SDP
> performance changes inOFED 1.3 beta, and I get Oops when
> enabling sdp_zcopy_thresh
>
> Hi,
> 64K is borderline for seeing bzcopy effect. Using an AMD
> 6000+ (3 Ghz
> dual core) in Asus M2A-VM motherboard with ConnectX running
> 2.3 firmware
> and OFED 1.3-rc3 stack running on 2.6.23.8 kernel.org kernel,
> I ran the
> test for 128K:
> 5546 sdp_zcopy_thresh=0 (off)
> 8709 sdp_zcopy_thresh=65536
>
> For these tests, I just have LD_PRELOAD set in my environment.
>
> =======================
>
> I see that TCP_MAXSEG is not being handled by libsdp and will
> look into
> it.
>
>
> [root at dirk ~]# modprobe ib_sdp
> [root at dirk ~]# netperf -v2 -4 -H 193.168.10.198 -l 30 -t TCP_STREAM -c
> -C -- -m 128K
> TCP STREAM TEST from 0.0.0.0 (0.0.0.0) port 0 AF_INET to
> 193.168.10.198
> (193.168.10.198) port 0 AF_INET
> netperf: get_tcp_info: getsockopt TCP_MAXSEG: errno 92
> Recv Send Send Utilization Service
> Demand
> Socket Socket Message Elapsed Send Recv Send
> Recv
> Size Size Size Time Throughput local remote local
> remote
> bytes bytes bytes secs. 10^6bits/s % S % S us/KB
> us/KB
>
> 87380 16384 131072 30.01 5545.69 51.47 14.43 1.521
> 1.706
>
> Alignment Offset Bytes Bytes Sends Bytes
> Recvs
> Local Remote Local Remote Xfered Per Per
> Send Recv Send Recv Send (avg) Recv (avg)
> 8 8 0 0 2.08e+10 131072.00 158690 33135.60
> 627718
>
> Maximum
> Segment
> Size (bytes)
> -1
> [root at dirk ~]# echo 65536
> >/sys/module/ib_sdp/parameters/sdp_zcopy_thresh
> [root at dirk ~]# netperf -v2 -4 -H 193.168.10.198 -l 30 -t TCP_STREAM -c
> -C -- -m 128K
> TCP STREAM TEST from 0.0.0.0 (0.0.0.0) port 0 AF_INET to
> 193.168.10.198
> (193.168.10.198) port 0 AF_INET
> netperf: get_tcp_info: getsockopt TCP_MAXSEG: errno 92
> Recv Send Send Utilization Service
> Demand
> Socket Socket Message Elapsed Send Recv Send
> Recv
> Size Size Size Time Throughput local remote local
> remote
> bytes bytes bytes secs. 10^6bits/s % S % S us/KB
> us/KB
>
> 87380 16384 131072 30.01 8708.58 50.63 14.55 0.953
> 1.095
>
> Alignment Offset Bytes Bytes Sends Bytes
> Recvs
> Local Remote Local Remote Xfered Per Per
> Send Recv Send Recv Send (avg) Recv (avg)
> 8 8 0 0 3.267e+10 131072.00 249228 26348.30
> 1239807
>
> Maximum
> Segment
> Size (bytes)
> -1
>
> Thanks,
> JIm
>
> Jim Mott
> Mellanox Technologies Ltd.
> mail: jim at mellanox.com
> Phone: 512-294-5481
>
>
> -----Original Message-----
> From: Weikuan Yu [mailto:weikuan.yu at gmail.com]
> Sent: Thursday, January 24, 2008 9:09 AM
> To: Scott Weitzenkamp (sweitzen)
> Cc: Jim Mott; ewg at lists.openfabrics.org; general at lists.openfabrics.org
> Subject: Re: [ofa-general] RE: [ewg] Not seeing any SDP performance
> changes inOFED 1.3 beta, and I get Oops when enabling sdp_zcopy_thresh
>
> Hi, Scott,
>
> I have been running SDP tests across two woodcrest nodes with 4x DDR
> cards using OFED-1.2.5.4. The card/firmware info is below.
>
> CA 'mthca0'
> CA type: MT25208
> Number of ports: 2
> Firmware version: 5.1.400
> Hardware version: a0
> Node GUID: 0x0002c90200228e0c
> System image GUID: 0x0002c90200228e0f
>
> I could not get a bandwidth more than 5Gbps like you have shown here.
> Wonder if I need to upgrade to the latest software or firmware? Any
> suggestions?
>
> Thanks,
> --Weikuan
>
>
> TCP STREAM TEST from 0.0.0.0 (0.0.0.0) port 0 AF_INET to
> 192.168.225.77
> (192.168
> .225.77) port 0 AF_INET
> Recv Send Send Utilization
> Service
> Demand
> Socket Socket Message Elapsed Send Recv Send
> Recv
> Size Size Size Time Throughput local remote local
> remote
> bytes bytes bytes secs. 10^6bits/s % S % S us/KB
> us/KB
>
> 131072 131072 131072 10.00 4918.95 21.29 24.99 1.418
> 1.665
>
>
> Scott Weitzenkamp (sweitzen) wrote:
> > Jim,
> >
> > I am trying OFED-1.3-20071231-0600 and RHEL4 x86_64 on a dual CPU
> > (single core each CPU) Xeon system. I do not see any performance
> > improvement (either throughput or CPU utilization) using
> netperf when
> I
> > set /sys/module/ib_sdp/sdp_zcopy_thresh to 16384. Can you elaborate
> on
> > your HCA type, and performance improvement you see?
> >
> > Here's an example netperf command line when using a Cheetah DDR HCA
> and
> > 1.2.917 firmware (I have also tried ConnectX and 2.3.000 firmware
> too):
> >
> > [releng at svbu-qa1850-2 ~]$ LD_PRELOAD=libsdp.so netperf241 -v2 -4 -H
> > 192.168.1.201 -l 30 -t TCP_STREAM -c -C -- -m 65536
> > TCP STREAM TEST from 0.0.0.0 (0.0.0.0) port 0 AF_INET to
> 192.168.1.201
> > (192.168.1.201) port 0 AF_INET : histogram : demo
> >
> > Recv Send Send Utilization
> Service
> > Demand
> > Socket Socket Message Elapsed Send Recv Send
> > Recv
> > Size Size Size Time Throughput local remote local
> > remote
> > bytes bytes bytes secs. 10^6bits/s % S % S us/KB
> > us/KB
> >
> > 87380 16384 65536 30.01 7267.70 55.06 61.27 1.241
> > 1.381
> >
> > Alignment Offset Bytes Bytes Sends Bytes
> > Recvs
> > Local Remote Local Remote Xfered Per Per
> > Send Recv Send Recv Send (avg)
> Recv (avg)
> > 8 8 0 0 2.726e+10 65536.00 415942
> 48106.01
> > 566648
> >
>
_______________________________________________
general mailing list
general at lists.openfabrics.org
http://lists.openfabrics.org/cgi-bin/mailman/listinfo/general
To unsubscribe, please visit http://openib.org/mailman/listinfo/openib-general
More information about the general
mailing list