[ofa-general] Re: performance drop for datagram mode with the newconnectx FW
Sagi Rotem
Sagir at mellanox.co.il
Tue Jul 1 00:06:26 PDT 2008
Or,
Are these results after you have applied Eli's fix ?
Sagi
-----Original Message-----
From: general-bounces at lists.openfabrics.org
[mailto:general-bounces at lists.openfabrics.org] On Behalf Of Or Gerlitz
Sent: Tuesday, July 01, 2008 9:42 AM
To: Oren Meron
Cc: Eli Cohen; general at lists.openfabrics.org
Subject: [ofa-general] Re: performance drop for datagram mode with the
newconnectx FW
> Using the new connectx FW (2.5), I see performance drop to almost zero
> with ipoib datagram mode. The code that runs on these systems is ofed
> 1.3 and not mainline kernel, details below.
>
> Running netperf With connected mode (64k MTU) I get about 950MB/s
> where with datagram mode (2k MTU) I get only 20-40MB/s. I used to see
> about 650MB/s and above with FW 2.3 and datagram mode. Not that it
> could explain the drop, but the NIC reports to the OS stateless
> offload support - /sys/class/net/ib1/features is 0x11423
I think to manage narrowing this a little further, the issue seems most
notable under long messages sent over datagram mode, could be an issue
in the LSO engine ?! see in this table,
====================================================
mode mtu size bw note
====================================================
datagram 2044 64000 30 <---------- problem
datagram 2044 2000 430
datagram 2044 2000 300 TCP_NODELAY
-----------------------------------------------------
connected 2044 64000 450
connected 2044 2000 450
connected 2044 2000 300 TCP_NODELAY
---------------------------------------------------
connected 64000 64000 930
connected 64000 2000 930
connected 64000 2000 470 TCP_NODELAY
====================================================
notes:
- in all cases, I have set the mtu manually
- verbs tests of bidirectional bandwidth shows that the HCA UD and RC
engines work very well, and
that the cables/switch etc are operating fine as the SDR limit is
easily reached.
# qperf -li mlx4_0:2 -ri mlx4_0:1 172.25.5.77 -m 2000 -t 10 rc_bi_bw
rc_bi_bw:
bw = 1.91 GB/sec
# qperf -li mlx4_0:2 -ri mlx4_0:1 172.25.5.77 -m 2000 -t 10 ud_bi_bw
ud_bi_bw:
send_bw = 1.95 GB/sec
recv_bw = 1.95 GB/sec
datagram mode mtu 2044
# netperf -H 10.10.0.90 -fM -l 600 -D 1, -- -m 64000 TCP STREAM TEST
from 0.0.0.0 (0.0.0.0) port 0 AF_INET to 10.10.0.90 (10.10.0.90) port 0
AF_INET : demo
Interim result: 33.00 MBytes/s over 10.31 seconds
Interim result: 22.27 MBytes/s over 1.48 seconds
Interim result: 34.22 MBytes/s over 2.51 seconds
Interim result: 34.05 MBytes/s over 1.01 seconds
Interim result: 22.88 MBytes/s over 1.49 seconds
Interim result: 30.03 MBytes/s over 1.00 seconds
Interim result: 28.26 MBytes/s over 1.01 seconds
Interim result: 28.21 MBytes/s over 1.00 seconds
Interim result: 14.26 MBytes/s over 1.98 seconds
# netperf -H 10.10.0.90 -fM -l 600 -D 1, -- -m 2000 TCP STREAM TEST
from 0.0.0.0 (0.0.0.0) port 0 AF_INET to 10.10.0.90 (10.10.0.90) port 0
AF_INET : demo Interim result: 429.54 MBytes/s over 1.00 seconds
Interim result: 429.22 MBytes/s over 1.00 seconds Interim result:
429.50 MBytes/s over 1.00 seconds Interim result: 429.47 MBytes/s over
1.00 seconds Interim result: 429.35 MBytes/s over 1.00 seconds Interim
result: 429.10 MBytes/s over 1.00 seconds Interim result: 429.16
MBytes/s over 1.00 seconds
# netperf -H 10.10.0.90 -fM -l 600 -D 1, -- -m 2000 -D L,R TCP STREAM
TEST from 0.0.0.0 (0.0.0.0) port 0 AF_INET to 10.10.0.90 (10.10.0.90)
port 0 AF_INET : nodelay : demo Interim result: 294.15 MBytes/s over
1.32 seconds Interim result: 294.27 MBytes/s over 1.00 seconds Interim
result: 294.98 MBytes/s over 1.00 seconds Interim result: 294.74
MBytes/s over 1.00 seconds Interim result: 294.45 MBytes/s over 1.00
seconds Interim result: 294.53 MBytes/s over 1.00 seconds Interim
result: 294.35 MBytes/s over 1.00 seconds Interim result: 294.43
MBytes/s over 1.00 seconds
connected mode mtu 2044
# netperf -H 10.10.0.90 -fM -l 600 -D 1, -- -m 64000
TCP STREAM TEST from 0.0.0.0 (0.0.0.0) port 0 AF_INET to 10.10.0.90
(10.10.0.90) port 0 AF_INET : demo Interim result: 161.05 MBytes/s over
1.71 seconds Interim result: 231.38 MBytes/s over 1.00 seconds Interim
result: 231.16 MBytes/s over 1.00 seconds Interim result: 232.40
MBytes/s over 1.00 seconds Interim result: 322.28 MBytes/s over 1.00
seconds Interim result: 448.25 MBytes/s over 1.00 seconds Interim
result: 448.54 MBytes/s over 1.00 seconds Interim result: 448.36
MBytes/s over 1.00 seconds Interim result: 448.52 MBytes/s over 1.00
seconds Interim result: 448.18 MBytes/s over 1.00 seconds Interim
result: 448.06 MBytes/s over 1.00 seconds Interim result: 448.26
MBytes/s over 1.00 seconds Interim result: 448.41 MBytes/s over 1.00
seconds
# netperf -H 10.10.0.90 -fM -l 600 -D 1, -- -m 2000
TCP STREAM TEST from 0.0.0.0 (0.0.0.0) port 0 AF_INET to 10.10.0.90
(10.10.0.90) port 0 AF_INET : demo Interim result: 454.45 MBytes/s over
1.00 seconds Interim result: 455.10 MBytes/s over 1.00 seconds Interim
result: 454.87 MBytes/s over 1.00 seconds Interim result: 454.64
MBytes/s over 1.00 seconds Interim result: 454.30 MBytes/s over 1.00
seconds Interim result: 454.11 MBytes/s over 1.00 seconds Interim
result: 454.08 MBytes/s over 1.00 seconds Interim result: 454.07
MBytes/s over 1.00 seconds Interim result: 454.06 MBytes/s over 1.00
seconds Interim result: 454.19 MBytes/s over 1.00 seconds Interim
result: 454.42 MBytes/s over 1.00 seconds Interim result: 454.31
MBytes/s over 1.00 seconds
# netperf -H 10.10.0.90 -fM -l 600 -D 1, -- -m 2000 -D L,R
TCP STREAM TEST from 0.0.0.0 (0.0.0.0) port 0 AF_INET to 10.10.0.90
(10.10.0.90) port 0 AF_INET : nodelay : demo Interim result: 305.93
MBytes/s over 1.00 seconds Interim result: 306.84 MBytes/s over 1.00
seconds Interim result: 306.88 MBytes/s over 1.00 seconds Interim
result: 306.80 MBytes/s over 1.00 seconds Interim result: 306.69
MBytes/s over 1.00 seconds Interim result: 306.41 MBytes/s over 1.00
seconds Interim result: 307.25 MBytes/s over 1.00 seconds Interim
result: 307.56 MBytes/s over 1.00 seconds Interim result: 307.51
MBytes/s over 1.00 seconds Interim result: 307.58 MBytes/s over 1.00
seconds
connected mode mtu 64000
# netperf -H 10.10.0.90 -fM -l 600 -D 1, -- -m 64000
TCP STREAM TEST from 0.0.0.0 (0.0.0.0) port 0 AF_INET to 10.10.0.90
(10.10.0.90) port 0 AF_INET : demo Interim result: 932.87 MBytes/s over
1.21 seconds Interim result: 931.33 MBytes/s over 1.00 seconds Interim
result: 930.20 MBytes/s over 1.00 seconds Interim result: 929.64
MBytes/s over 1.00 seconds Interim result: 931.16 MBytes/s over 1.00
seconds Interim result: 929.96 MBytes/s over 1.00 seconds
# netperf -H 10.10.0.90 -fM -l 600 -D 1, -- -m 2000
TCP STREAM TEST from 0.0.0.0 (0.0.0.0) port 0 AF_INET to 10.10.0.90
(10.10.0.90) port 0 AF_INET : demo Interim result: 932.82 MBytes/s over
1.46 seconds Interim result: 930.69 MBytes/s over 1.00 seconds Interim
result: 931.36 MBytes/s over 1.00 seconds Interim result: 930.05
MBytes/s over 1.00 seconds Interim result: 931.40 MBytes/s over 1.00
seconds Interim result: 930.07 MBytes/s over 1.00 seconds
# netperf -H 10.10.0.90 -fM -l 600 -D 1, -- -m 2000 -D L,R
TCP STREAM TEST from 0.0.0.0 (0.0.0.0) port 0 AF_INET to 10.10.0.90
(10.10.0.90) port 0 AF_INET : nodelay : demo Interim result: 475.01
MBytes/s over 1.58 seconds Interim result: 474.22 MBytes/s over 1.00
seconds Interim result: 474.25 MBytes/s over 1.00 seconds Interim
result: 473.14 MBytes/s over 1.00 seconds Interim result: 473.62
MBytes/s over 1.00 seconds Interim result: 474.37 MBytes/s over 1.00
seconds Interim result: 473.81 MBytes/s over 1.00 seconds
_______________________________________________
general mailing list
general at lists.openfabrics.org
http://lists.openfabrics.org/cgi-bin/mailman/listinfo/general
To unsubscribe, please visit
http://openib.org/mailman/listinfo/openib-general
More information about the general
mailing list