[ofa-general] Re: performance drop for datagram mode with the new connectx FW
Or Gerlitz
ogerlitz at voltaire.com
Mon Jul 7 01:20:17 PDT 2008
> manage narrowing this a little further, the issue seems most
> notable under long messages sent over datagram mode, could be
> an issue in the LSO engine ?! see in this table,
>
> ====================================================
> mode mtu size bw note
> ====================================================
> datagram 2044 64000 30 <---------- problem
> # netperf -H 10.10.0.90 -fM -l 20 -D 1, -- -m 64000
OK, it turns out that this specific parameters set to netperf / TCP_STREAM was
the source for the performance drop I saw, setting also the socket send buffer
changes everything (all tests use datagram mode, mtu 2044, msg size 64000)
====================================================
send_buf recv_buf BW (MB/s)
====================================================
NA NA 30
32000 NA 640
64000 NA 750
128000 NA 790
128000 128000 840
On this system / current settings, I get quite the same numbers
for both firmware version (2.3 and 2.5)
Or.
# netperf -H 10.10.0.90 -fM -l 20 -D 1, -- -m 64000
TCP STREAM TEST from 0.0.0.0 (0.0.0.0) port 0 AF_INET to 10.10.0.90 (10.10.0.90) port 0 AF_INET : demo
Interim result: 44.11 MBytes/s over 10.68 seconds
Interim result: 34.25 MBytes/s over 2.49 seconds
Interim result: 67.30 MBytes/s over 1.01 seconds
Interim result: 67.28 MBytes/s over 1.00 seconds
Interim result: 66.72 MBytes/s over 1.01 seconds
Interim result: 33.31 MBytes/s over 2.00 seconds
Interim result: 22.63 MBytes/s over 1.47 seconds
Recv Send Send
Socket Socket Message Elapsed
Size Size Size Time Throughput
bytes bytes bytes secs. MBytes/sec
87380 16384 64000 20.66 42.95
# netperf -H 10.10.0.90 -fM -l 20 -D 1, -- -m 64000 -s 32000
TCP STREAM TEST from 0.0.0.0 (0.0.0.0) port 0 AF_INET to 10.10.0.90 (10.10.0.90) port 0 AF_INET : demo
Interim result: 643.92 MBytes/s over 1.00 seconds
Interim result: 646.37 MBytes/s over 1.00 seconds
Interim result: 644.34 MBytes/s over 1.00 seconds
Interim result: 644.39 MBytes/s over 1.00 seconds
Interim result: 643.47 MBytes/s over 1.00 seconds
Interim result: 643.22 MBytes/s over 1.00 seconds
Interim result: 643.55 MBytes/s over 1.00 seconds
Interim result: 642.41 MBytes/s over 1.00 seconds
Interim result: 640.98 MBytes/s over 1.00 seconds
Interim result: 641.88 MBytes/s over 1.00 seconds
Interim result: 642.67 MBytes/s over 1.00 seconds
Interim result: 642.45 MBytes/s over 1.00 seconds
Interim result: 642.25 MBytes/s over 1.00 seconds
Interim result: 642.23 MBytes/s over 1.00 seconds
Interim result: 640.45 MBytes/s over 1.00 seconds
Interim result: 641.84 MBytes/s over 1.00 seconds
Interim result: 640.92 MBytes/s over 1.00 seconds
Interim result: 639.94 MBytes/s over 1.00 seconds
Interim result: 639.54 MBytes/s over 1.00 seconds
Recv Send Send
Socket Socket Message Elapsed
Size Size Size Time Throughput
bytes bytes bytes secs. MBytes/sec
87380 64000 64000 20.00 642.22
[root at cto-gen2-1 ~]# netperf -H 10.10.0.90 -fM -l 20 -D 1, -- -m 64000 -s 64000
TCP STREAM TEST from 0.0.0.0 (0.0.0.0) port 0 AF_INET to 10.10.0.90 (10.10.0.90) port 0 AF_INET : demo
Interim result: 753.91 MBytes/s over 1.00 seconds
Interim result: 754.28 MBytes/s over 1.00 seconds
Interim result: 752.74 MBytes/s over 1.00 seconds
Interim result: 747.04 MBytes/s over 1.01 seconds
Interim result: 747.66 MBytes/s over 1.00 seconds
Interim result: 746.79 MBytes/s over 1.00 seconds
Interim result: 749.77 MBytes/s over 1.00 seconds
Interim result: 751.40 MBytes/s over 1.00 seconds
Interim result: 748.91 MBytes/s over 1.00 seconds
Interim result: 750.36 MBytes/s over 1.00 seconds
Interim result: 747.39 MBytes/s over 1.00 seconds
Interim result: 749.42 MBytes/s over 1.00 seconds
Interim result: 750.04 MBytes/s over 1.00 seconds
Interim result: 751.30 MBytes/s over 1.00 seconds
Interim result: 750.37 MBytes/s over 1.00 seconds
Interim result: 749.90 MBytes/s over 1.00 seconds
Interim result: 747.94 MBytes/s over 1.00 seconds
Interim result: 747.81 MBytes/s over 1.00 seconds
Interim result: 744.92 MBytes/s over 1.00 seconds
Recv Send Send
Socket Socket Message Elapsed
Size Size Size Time Throughput
bytes bytes bytes secs. MBytes/sec
87380 128000 64000 20.00 749.34
[root at cto-gen2-1 ~]# netperf -H 10.10.0.90 -fM -l 20 -D 1, -- -m 64000 -s 128000
TCP STREAM TEST from 0.0.0.0 (0.0.0.0) port 0 AF_INET to 10.10.0.90 (10.10.0.90) port 0 AF_INET : demo
Interim result: 774.11 MBytes/s over 1.00 seconds
Interim result: 786.26 MBytes/s over 1.00 seconds
Interim result: 795.26 MBytes/s over 1.00 seconds
Interim result: 795.31 MBytes/s over 1.00 seconds
Interim result: 791.34 MBytes/s over 1.00 seconds
Interim result: 792.93 MBytes/s over 1.00 seconds
Interim result: 793.18 MBytes/s over 1.00 seconds
Interim result: 792.43 MBytes/s over 1.00 seconds
Interim result: 794.03 MBytes/s over 1.00 seconds
Interim result: 791.21 MBytes/s over 1.00 seconds
Interim result: 791.28 MBytes/s over 1.00 seconds
Interim result: 790.57 MBytes/s over 1.00 seconds
Interim result: 791.86 MBytes/s over 1.00 seconds
Interim result: 792.43 MBytes/s over 1.00 seconds
Interim result: 792.40 MBytes/s over 1.00 seconds
Interim result: 793.32 MBytes/s over 1.00 seconds
Interim result: 791.92 MBytes/s over 1.00 seconds
Interim result: 793.45 MBytes/s over 1.00 seconds
Interim result: 792.25 MBytes/s over 1.00 seconds
Recv Send Send
Socket Socket Message Elapsed
Size Size Size Time Throughput
bytes bytes bytes secs. MBytes/sec
87380 256000 64000 20.00 791.32
# netperf -H 10.10.0.90 -fM -l 20 -D 1, -- -m 64000 -s 128000 -S 128000
TCP STREAM TEST from 0.0.0.0 (0.0.0.0) port 0 AF_INET to 10.10.0.90 (10.10.0.90) port 0 AF_INET : demo
Interim result: 842.25 MBytes/s over 1.00 seconds
Interim result: 845.21 MBytes/s over 1.00 seconds
Interim result: 845.46 MBytes/s over 1.00 seconds
Interim result: 844.63 MBytes/s over 1.00 seconds
Interim result: 844.32 MBytes/s over 1.00 seconds
Interim result: 842.70 MBytes/s over 1.00 seconds
Interim result: 845.30 MBytes/s over 1.00 seconds
Interim result: 843.44 MBytes/s over 1.00 seconds
Interim result: 844.67 MBytes/s over 1.00 seconds
Interim result: 843.80 MBytes/s over 1.00 seconds
Interim result: 842.33 MBytes/s over 1.00 seconds
Interim result: 842.04 MBytes/s over 1.00 seconds
Interim result: 843.81 MBytes/s over 1.00 seconds
Interim result: 841.76 MBytes/s over 1.00 seconds
Interim result: 845.92 MBytes/s over 1.00 seconds
Interim result: 843.63 MBytes/s over 1.00 seconds
Interim result: 840.97 MBytes/s over 1.00 seconds
Interim result: 841.30 MBytes/s over 1.00 seconds
Interim result: 842.38 MBytes/s over 1.00 seconds
Recv Send Send
Socket Socket Message Elapsed
Size Size Size Time Throughput
bytes bytes bytes secs. MBytes/sec
256000 256000 64000 20.00 843.31
More information about the general
mailing list