[ofa-general] RE: [ewg] Not seeing any SDP performance changes inOFED 1.3 beta, and I get Oops when enabling sdp_zcopy_thresh

Jim Mott jim at mellanox.com
Thu Jan 24 07:56:39 PST 2008


Hi,
  64K is borderline for seeing bzcopy effect.  Using an AMD 6000+ (3 Ghz
dual core) in Asus M2A-VM motherboard with ConnectX running 2.3 firmware
and OFED 1.3-rc3 stack running on 2.6.23.8 kernel.org kernel, I ran the
test for 128K:
  5546  sdp_zcopy_thresh=0 (off)
  8709  sdp_zcopy_thresh=65536

For these tests, I just have LD_PRELOAD set in my environment.

=======================

I see that TCP_MAXSEG is not being handled by libsdp and will look into
it.


[root at dirk ~]# modprobe ib_sdp
[root at dirk ~]# netperf -v2 -4 -H 193.168.10.198 -l 30 -t TCP_STREAM -c
-C -- -m 128K
TCP STREAM TEST from 0.0.0.0 (0.0.0.0) port 0 AF_INET to 193.168.10.198
(193.168.10.198) port 0 AF_INET
netperf: get_tcp_info: getsockopt TCP_MAXSEG: errno 92
Recv   Send    Send                          Utilization       Service
Demand
Socket Socket  Message  Elapsed              Send     Recv     Send
Recv
Size   Size    Size     Time     Throughput  local    remote   local
remote
bytes  bytes   bytes    secs.    10^6bits/s  % S      % S      us/KB
us/KB

 87380  16384 131072    30.01      5545.69   51.47    14.43    1.521
1.706  

Alignment      Offset         Bytes    Bytes       Sends   Bytes
Recvs
Local  Remote  Local  Remote  Xfered   Per                 Per
Send   Recv    Send   Recv             Send (avg)          Recv (avg)
    8       8      0       0 2.08e+10  131072.00    158690   33135.60
627718

Maximum
Segment
Size (bytes)
    -1
[root at dirk ~]# echo 65536
>/sys/module/ib_sdp/parameters/sdp_zcopy_thresh 
[root at dirk ~]# netperf -v2 -4 -H 193.168.10.198 -l 30 -t TCP_STREAM -c
-C -- -m 128K
TCP STREAM TEST from 0.0.0.0 (0.0.0.0) port 0 AF_INET to 193.168.10.198
(193.168.10.198) port 0 AF_INET
netperf: get_tcp_info: getsockopt TCP_MAXSEG: errno 92
Recv   Send    Send                          Utilization       Service
Demand
Socket Socket  Message  Elapsed              Send     Recv     Send
Recv
Size   Size    Size     Time     Throughput  local    remote   local
remote
bytes  bytes   bytes    secs.    10^6bits/s  % S      % S      us/KB
us/KB

 87380  16384 131072    30.01      8708.58   50.63    14.55    0.953
1.095  

Alignment      Offset         Bytes    Bytes       Sends   Bytes
Recvs
Local  Remote  Local  Remote  Xfered   Per                 Per
Send   Recv    Send   Recv             Send (avg)          Recv (avg)
    8       8      0       0 3.267e+10  131072.00    249228   26348.30
1239807

Maximum
Segment
Size (bytes)
    -1

Thanks,
JIm

Jim Mott
Mellanox Technologies Ltd.
mail: jim at mellanox.com
Phone: 512-294-5481


-----Original Message-----
From: Weikuan Yu [mailto:weikuan.yu at gmail.com] 
Sent: Thursday, January 24, 2008 9:09 AM
To: Scott Weitzenkamp (sweitzen)
Cc: Jim Mott; ewg at lists.openfabrics.org; general at lists.openfabrics.org
Subject: Re: [ofa-general] RE: [ewg] Not seeing any SDP performance
changes inOFED 1.3 beta, and I get Oops when enabling sdp_zcopy_thresh

Hi, Scott,

I have been running SDP tests across two woodcrest nodes with 4x DDR 
cards using OFED-1.2.5.4. The card/firmware info is below.

CA 'mthca0'
         CA type: MT25208
         Number of ports: 2
         Firmware version: 5.1.400
         Hardware version: a0
         Node GUID: 0x0002c90200228e0c
         System image GUID: 0x0002c90200228e0f

I could not get a bandwidth more than 5Gbps like you have shown here. 
Wonder if I need to upgrade to the latest software or firmware? Any 
suggestions?

Thanks,
--Weikuan


TCP STREAM TEST from 0.0.0.0 (0.0.0.0) port 0 AF_INET to 192.168.225.77 
(192.168
.225.77) port 0 AF_INET
Recv   Send    Send                          Utilization       Service 
Demand
Socket Socket  Message  Elapsed              Send     Recv     Send
Recv
Size   Size    Size     Time     Throughput  local    remote   local 
remote
bytes  bytes   bytes    secs.    10^6bits/s  % S      % S      us/KB
us/KB

131072 131072 131072    10.00      4918.95   21.29    24.99    1.418 
1.665


Scott Weitzenkamp (sweitzen) wrote:
> Jim,
> 
> I am trying OFED-1.3-20071231-0600 and RHEL4 x86_64 on a dual CPU
> (single core each CPU) Xeon system.  I do not see any performance
> improvement (either throughput or CPU utilization) using netperf when
I
> set /sys/module/ib_sdp/sdp_zcopy_thresh to 16384.  Can you elaborate
on
> your HCA type, and performance improvement you see?
> 
> Here's an example netperf command line when using a Cheetah DDR HCA
and
> 1.2.917 firmware (I have also tried ConnectX and 2.3.000 firmware
too):
> 
> [releng at svbu-qa1850-2 ~]$ LD_PRELOAD=libsdp.so netperf241 -v2 -4 -H
> 192.168.1.201 -l 30 -t TCP_STREAM -c -C --   -m 65536
> TCP STREAM TEST from 0.0.0.0 (0.0.0.0) port 0 AF_INET to 192.168.1.201
> (192.168.1.201) port 0 AF_INET : histogram : demo
> 
> Recv   Send    Send                          Utilization       Service
> Demand
> Socket Socket  Message  Elapsed              Send     Recv     Send
> Recv
> Size   Size    Size     Time     Throughput  local    remote   local
> remote
> bytes  bytes   bytes    secs.    10^6bits/s  % S      % S      us/KB
> us/KB
> 
>  87380  16384  65536    30.01      7267.70   55.06    61.27    1.241
> 1.381 
> 
> Alignment      Offset         Bytes    Bytes       Sends   Bytes
> Recvs
> Local  Remote  Local  Remote  Xfered   Per                 Per
> Send   Recv    Send   Recv             Send (avg)          Recv (avg)
>     8       8      0       0 2.726e+10  65536.00    415942   48106.01
> 566648
> 



More information about the ewg mailing list