[ofa-general] RE: [ewg] Not seeing any SDP performance changes inOFED 1.3 beta, and I get Oops when enabling sdp_zcopy_thresh
Jim Mott
jim at mellanox.com
Thu Jan 24 07:56:39 PST 2008
Hi,
64K is borderline for seeing bzcopy effect. Using an AMD 6000+ (3 Ghz
dual core) in Asus M2A-VM motherboard with ConnectX running 2.3 firmware
and OFED 1.3-rc3 stack running on 2.6.23.8 kernel.org kernel, I ran the
test for 128K:
5546 sdp_zcopy_thresh=0 (off)
8709 sdp_zcopy_thresh=65536
For these tests, I just have LD_PRELOAD set in my environment.
=======================
I see that TCP_MAXSEG is not being handled by libsdp and will look into
it.
[root at dirk ~]# modprobe ib_sdp
[root at dirk ~]# netperf -v2 -4 -H 193.168.10.198 -l 30 -t TCP_STREAM -c
-C -- -m 128K
TCP STREAM TEST from 0.0.0.0 (0.0.0.0) port 0 AF_INET to 193.168.10.198
(193.168.10.198) port 0 AF_INET
netperf: get_tcp_info: getsockopt TCP_MAXSEG: errno 92
Recv Send Send Utilization Service
Demand
Socket Socket Message Elapsed Send Recv Send
Recv
Size Size Size Time Throughput local remote local
remote
bytes bytes bytes secs. 10^6bits/s % S % S us/KB
us/KB
87380 16384 131072 30.01 5545.69 51.47 14.43 1.521
1.706
Alignment Offset Bytes Bytes Sends Bytes
Recvs
Local Remote Local Remote Xfered Per Per
Send Recv Send Recv Send (avg) Recv (avg)
8 8 0 0 2.08e+10 131072.00 158690 33135.60
627718
Maximum
Segment
Size (bytes)
-1
[root at dirk ~]# echo 65536
>/sys/module/ib_sdp/parameters/sdp_zcopy_thresh
[root at dirk ~]# netperf -v2 -4 -H 193.168.10.198 -l 30 -t TCP_STREAM -c
-C -- -m 128K
TCP STREAM TEST from 0.0.0.0 (0.0.0.0) port 0 AF_INET to 193.168.10.198
(193.168.10.198) port 0 AF_INET
netperf: get_tcp_info: getsockopt TCP_MAXSEG: errno 92
Recv Send Send Utilization Service
Demand
Socket Socket Message Elapsed Send Recv Send
Recv
Size Size Size Time Throughput local remote local
remote
bytes bytes bytes secs. 10^6bits/s % S % S us/KB
us/KB
87380 16384 131072 30.01 8708.58 50.63 14.55 0.953
1.095
Alignment Offset Bytes Bytes Sends Bytes
Recvs
Local Remote Local Remote Xfered Per Per
Send Recv Send Recv Send (avg) Recv (avg)
8 8 0 0 3.267e+10 131072.00 249228 26348.30
1239807
Maximum
Segment
Size (bytes)
-1
Thanks,
JIm
Jim Mott
Mellanox Technologies Ltd.
mail: jim at mellanox.com
Phone: 512-294-5481
-----Original Message-----
From: Weikuan Yu [mailto:weikuan.yu at gmail.com]
Sent: Thursday, January 24, 2008 9:09 AM
To: Scott Weitzenkamp (sweitzen)
Cc: Jim Mott; ewg at lists.openfabrics.org; general at lists.openfabrics.org
Subject: Re: [ofa-general] RE: [ewg] Not seeing any SDP performance
changes inOFED 1.3 beta, and I get Oops when enabling sdp_zcopy_thresh
Hi, Scott,
I have been running SDP tests across two woodcrest nodes with 4x DDR
cards using OFED-1.2.5.4. The card/firmware info is below.
CA 'mthca0'
CA type: MT25208
Number of ports: 2
Firmware version: 5.1.400
Hardware version: a0
Node GUID: 0x0002c90200228e0c
System image GUID: 0x0002c90200228e0f
I could not get a bandwidth more than 5Gbps like you have shown here.
Wonder if I need to upgrade to the latest software or firmware? Any
suggestions?
Thanks,
--Weikuan
TCP STREAM TEST from 0.0.0.0 (0.0.0.0) port 0 AF_INET to 192.168.225.77
(192.168
.225.77) port 0 AF_INET
Recv Send Send Utilization Service
Demand
Socket Socket Message Elapsed Send Recv Send
Recv
Size Size Size Time Throughput local remote local
remote
bytes bytes bytes secs. 10^6bits/s % S % S us/KB
us/KB
131072 131072 131072 10.00 4918.95 21.29 24.99 1.418
1.665
Scott Weitzenkamp (sweitzen) wrote:
> Jim,
>
> I am trying OFED-1.3-20071231-0600 and RHEL4 x86_64 on a dual CPU
> (single core each CPU) Xeon system. I do not see any performance
> improvement (either throughput or CPU utilization) using netperf when
I
> set /sys/module/ib_sdp/sdp_zcopy_thresh to 16384. Can you elaborate
on
> your HCA type, and performance improvement you see?
>
> Here's an example netperf command line when using a Cheetah DDR HCA
and
> 1.2.917 firmware (I have also tried ConnectX and 2.3.000 firmware
too):
>
> [releng at svbu-qa1850-2 ~]$ LD_PRELOAD=libsdp.so netperf241 -v2 -4 -H
> 192.168.1.201 -l 30 -t TCP_STREAM -c -C -- -m 65536
> TCP STREAM TEST from 0.0.0.0 (0.0.0.0) port 0 AF_INET to 192.168.1.201
> (192.168.1.201) port 0 AF_INET : histogram : demo
>
> Recv Send Send Utilization Service
> Demand
> Socket Socket Message Elapsed Send Recv Send
> Recv
> Size Size Size Time Throughput local remote local
> remote
> bytes bytes bytes secs. 10^6bits/s % S % S us/KB
> us/KB
>
> 87380 16384 65536 30.01 7267.70 55.06 61.27 1.241
> 1.381
>
> Alignment Offset Bytes Bytes Sends Bytes
> Recvs
> Local Remote Local Remote Xfered Per Per
> Send Recv Send Recv Send (avg) Recv (avg)
> 8 8 0 0 2.726e+10 65536.00 415942 48106.01
> 566648
>
More information about the ewg
mailing list