[ofa-general] RE: [ewg] Not seeing any SDP performance changes inOFED 1.3 beta, and I get Oops when enabling sdp_zcopy_thresh
Scott Weitzenkamp (sweitzen)
sweitzen at cisco.com
Thu Jan 24 09:16:45 PST 2008
I've tested on RHEL4 and RHEL5, and see no sdp_zcopy_thresh improvement
for any message size, as measured with netperf, for any Arbel or
ConnectX HCA.
Scott
> -----Original Message-----
> From: Jim Mott [mailto:jim at mellanox.com]
> Sent: Thursday, January 24, 2008 7:57 AM
> To: Weikuan Yu; Scott Weitzenkamp (sweitzen)
> Cc: ewg at lists.openfabrics.org; general at lists.openfabrics.org
> Subject: RE: [ofa-general] RE: [ewg] Not seeing any SDP
> performance changes inOFED 1.3 beta, and I get Oops when
> enabling sdp_zcopy_thresh
>
> Hi,
> 64K is borderline for seeing bzcopy effect. Using an AMD
> 6000+ (3 Ghz
> dual core) in Asus M2A-VM motherboard with ConnectX running
> 2.3 firmware
> and OFED 1.3-rc3 stack running on 2.6.23.8 kernel.org kernel,
> I ran the
> test for 128K:
> 5546 sdp_zcopy_thresh=0 (off)
> 8709 sdp_zcopy_thresh=65536
>
> For these tests, I just have LD_PRELOAD set in my environment.
>
> =======================
>
> I see that TCP_MAXSEG is not being handled by libsdp and will
> look into
> it.
>
>
> [root at dirk ~]# modprobe ib_sdp
> [root at dirk ~]# netperf -v2 -4 -H 193.168.10.198 -l 30 -t TCP_STREAM -c
> -C -- -m 128K
> TCP STREAM TEST from 0.0.0.0 (0.0.0.0) port 0 AF_INET to
> 193.168.10.198
> (193.168.10.198) port 0 AF_INET
> netperf: get_tcp_info: getsockopt TCP_MAXSEG: errno 92
> Recv Send Send Utilization Service
> Demand
> Socket Socket Message Elapsed Send Recv Send
> Recv
> Size Size Size Time Throughput local remote local
> remote
> bytes bytes bytes secs. 10^6bits/s % S % S us/KB
> us/KB
>
> 87380 16384 131072 30.01 5545.69 51.47 14.43 1.521
> 1.706
>
> Alignment Offset Bytes Bytes Sends Bytes
> Recvs
> Local Remote Local Remote Xfered Per Per
> Send Recv Send Recv Send (avg) Recv (avg)
> 8 8 0 0 2.08e+10 131072.00 158690 33135.60
> 627718
>
> Maximum
> Segment
> Size (bytes)
> -1
> [root at dirk ~]# echo 65536
> >/sys/module/ib_sdp/parameters/sdp_zcopy_thresh
> [root at dirk ~]# netperf -v2 -4 -H 193.168.10.198 -l 30 -t TCP_STREAM -c
> -C -- -m 128K
> TCP STREAM TEST from 0.0.0.0 (0.0.0.0) port 0 AF_INET to
> 193.168.10.198
> (193.168.10.198) port 0 AF_INET
> netperf: get_tcp_info: getsockopt TCP_MAXSEG: errno 92
> Recv Send Send Utilization Service
> Demand
> Socket Socket Message Elapsed Send Recv Send
> Recv
> Size Size Size Time Throughput local remote local
> remote
> bytes bytes bytes secs. 10^6bits/s % S % S us/KB
> us/KB
>
> 87380 16384 131072 30.01 8708.58 50.63 14.55 0.953
> 1.095
>
> Alignment Offset Bytes Bytes Sends Bytes
> Recvs
> Local Remote Local Remote Xfered Per Per
> Send Recv Send Recv Send (avg) Recv (avg)
> 8 8 0 0 3.267e+10 131072.00 249228 26348.30
> 1239807
>
> Maximum
> Segment
> Size (bytes)
> -1
>
> Thanks,
> JIm
>
> Jim Mott
> Mellanox Technologies Ltd.
> mail: jim at mellanox.com
> Phone: 512-294-5481
>
>
> -----Original Message-----
> From: Weikuan Yu [mailto:weikuan.yu at gmail.com]
> Sent: Thursday, January 24, 2008 9:09 AM
> To: Scott Weitzenkamp (sweitzen)
> Cc: Jim Mott; ewg at lists.openfabrics.org; general at lists.openfabrics.org
> Subject: Re: [ofa-general] RE: [ewg] Not seeing any SDP performance
> changes inOFED 1.3 beta, and I get Oops when enabling sdp_zcopy_thresh
>
> Hi, Scott,
>
> I have been running SDP tests across two woodcrest nodes with 4x DDR
> cards using OFED-1.2.5.4. The card/firmware info is below.
>
> CA 'mthca0'
> CA type: MT25208
> Number of ports: 2
> Firmware version: 5.1.400
> Hardware version: a0
> Node GUID: 0x0002c90200228e0c
> System image GUID: 0x0002c90200228e0f
>
> I could not get a bandwidth more than 5Gbps like you have shown here.
> Wonder if I need to upgrade to the latest software or firmware? Any
> suggestions?
>
> Thanks,
> --Weikuan
>
>
> TCP STREAM TEST from 0.0.0.0 (0.0.0.0) port 0 AF_INET to
> 192.168.225.77
> (192.168
> .225.77) port 0 AF_INET
> Recv Send Send Utilization
> Service
> Demand
> Socket Socket Message Elapsed Send Recv Send
> Recv
> Size Size Size Time Throughput local remote local
> remote
> bytes bytes bytes secs. 10^6bits/s % S % S us/KB
> us/KB
>
> 131072 131072 131072 10.00 4918.95 21.29 24.99 1.418
> 1.665
>
>
> Scott Weitzenkamp (sweitzen) wrote:
> > Jim,
> >
> > I am trying OFED-1.3-20071231-0600 and RHEL4 x86_64 on a dual CPU
> > (single core each CPU) Xeon system. I do not see any performance
> > improvement (either throughput or CPU utilization) using
> netperf when
> I
> > set /sys/module/ib_sdp/sdp_zcopy_thresh to 16384. Can you elaborate
> on
> > your HCA type, and performance improvement you see?
> >
> > Here's an example netperf command line when using a Cheetah DDR HCA
> and
> > 1.2.917 firmware (I have also tried ConnectX and 2.3.000 firmware
> too):
> >
> > [releng at svbu-qa1850-2 ~]$ LD_PRELOAD=libsdp.so netperf241 -v2 -4 -H
> > 192.168.1.201 -l 30 -t TCP_STREAM -c -C -- -m 65536
> > TCP STREAM TEST from 0.0.0.0 (0.0.0.0) port 0 AF_INET to
> 192.168.1.201
> > (192.168.1.201) port 0 AF_INET : histogram : demo
> >
> > Recv Send Send Utilization
> Service
> > Demand
> > Socket Socket Message Elapsed Send Recv Send
> > Recv
> > Size Size Size Time Throughput local remote local
> > remote
> > bytes bytes bytes secs. 10^6bits/s % S % S us/KB
> > us/KB
> >
> > 87380 16384 65536 30.01 7267.70 55.06 61.27 1.241
> > 1.381
> >
> > Alignment Offset Bytes Bytes Sends Bytes
> > Recvs
> > Local Remote Local Remote Xfered Per Per
> > Send Recv Send Recv Send (avg)
> Recv (avg)
> > 8 8 0 0 2.726e+10 65536.00 415942
> 48106.01
> > 566648
> >
>
More information about the ewg
mailing list