[ofa-general] Re: performance drop for datagram mode with the new connectx FW
Or Gerlitz
ogerlitz at voltaire.com
Mon Jun 30 23:41:30 PDT 2008
> Using the new connectx FW (2.5), I see performance drop to almost
> zero with ipoib datagram mode. The code that runs on these systems
> is ofed 1.3 and not mainline kernel, details below.
>
> Running netperf With connected mode (64k MTU) I get about 950MB/s
> where with datagram mode (2k MTU) I get only 20-40MB/s. I used to
> see about 650MB/s and above with FW 2.3 and datagram mode. Not that
> it could explain the drop, but the NIC reports to the OS stateless
> offload support - /sys/class/net/ib1/features is 0x11423
I think to manage narrowing this a little further, the issue seems
most notable under long messages sent over datagram mode, could be
an issue in the LSO engine ?! see in this table,
====================================================
mode mtu size bw note
====================================================
datagram 2044 64000 30 <---------- problem
datagram 2044 2000 430
datagram 2044 2000 300 TCP_NODELAY
-----------------------------------------------------
connected 2044 64000 450
connected 2044 2000 450
connected 2044 2000 300 TCP_NODELAY
---------------------------------------------------
connected 64000 64000 930
connected 64000 2000 930
connected 64000 2000 470 TCP_NODELAY
====================================================
notes:
- in all cases, I have set the mtu manually
- verbs tests of bidirectional bandwidth shows that the HCA UD and RC engines work very well, and
that the cables/switch etc are operating fine as the SDR limit is easily reached.
# qperf -li mlx4_0:2 -ri mlx4_0:1 172.25.5.77 -m 2000 -t 10 rc_bi_bw
rc_bi_bw:
bw = 1.91 GB/sec
# qperf -li mlx4_0:2 -ri mlx4_0:1 172.25.5.77 -m 2000 -t 10 ud_bi_bw
ud_bi_bw:
send_bw = 1.95 GB/sec
recv_bw = 1.95 GB/sec
datagram mode mtu 2044
# netperf -H 10.10.0.90 -fM -l 600 -D 1, -- -m 64000
TCP STREAM TEST from 0.0.0.0 (0.0.0.0) port 0 AF_INET to 10.10.0.90 (10.10.0.90) port 0 AF_INET : demo
Interim result: 33.00 MBytes/s over 10.31 seconds
Interim result: 22.27 MBytes/s over 1.48 seconds
Interim result: 34.22 MBytes/s over 2.51 seconds
Interim result: 34.05 MBytes/s over 1.01 seconds
Interim result: 22.88 MBytes/s over 1.49 seconds
Interim result: 30.03 MBytes/s over 1.00 seconds
Interim result: 28.26 MBytes/s over 1.01 seconds
Interim result: 28.21 MBytes/s over 1.00 seconds
Interim result: 14.26 MBytes/s over 1.98 seconds
# netperf -H 10.10.0.90 -fM -l 600 -D 1, -- -m 2000
TCP STREAM TEST from 0.0.0.0 (0.0.0.0) port 0 AF_INET to 10.10.0.90 (10.10.0.90) port 0 AF_INET : demo
Interim result: 429.54 MBytes/s over 1.00 seconds
Interim result: 429.22 MBytes/s over 1.00 seconds
Interim result: 429.50 MBytes/s over 1.00 seconds
Interim result: 429.47 MBytes/s over 1.00 seconds
Interim result: 429.35 MBytes/s over 1.00 seconds
Interim result: 429.10 MBytes/s over 1.00 seconds
Interim result: 429.16 MBytes/s over 1.00 seconds
# netperf -H 10.10.0.90 -fM -l 600 -D 1, -- -m 2000 -D L,R
TCP STREAM TEST from 0.0.0.0 (0.0.0.0) port 0 AF_INET to 10.10.0.90 (10.10.0.90) port 0 AF_INET : nodelay : demo
Interim result: 294.15 MBytes/s over 1.32 seconds
Interim result: 294.27 MBytes/s over 1.00 seconds
Interim result: 294.98 MBytes/s over 1.00 seconds
Interim result: 294.74 MBytes/s over 1.00 seconds
Interim result: 294.45 MBytes/s over 1.00 seconds
Interim result: 294.53 MBytes/s over 1.00 seconds
Interim result: 294.35 MBytes/s over 1.00 seconds
Interim result: 294.43 MBytes/s over 1.00 seconds
connected mode mtu 2044
# netperf -H 10.10.0.90 -fM -l 600 -D 1, -- -m 64000
TCP STREAM TEST from 0.0.0.0 (0.0.0.0) port 0 AF_INET to 10.10.0.90 (10.10.0.90) port 0 AF_INET : demo
Interim result: 161.05 MBytes/s over 1.71 seconds
Interim result: 231.38 MBytes/s over 1.00 seconds
Interim result: 231.16 MBytes/s over 1.00 seconds
Interim result: 232.40 MBytes/s over 1.00 seconds
Interim result: 322.28 MBytes/s over 1.00 seconds
Interim result: 448.25 MBytes/s over 1.00 seconds
Interim result: 448.54 MBytes/s over 1.00 seconds
Interim result: 448.36 MBytes/s over 1.00 seconds
Interim result: 448.52 MBytes/s over 1.00 seconds
Interim result: 448.18 MBytes/s over 1.00 seconds
Interim result: 448.06 MBytes/s over 1.00 seconds
Interim result: 448.26 MBytes/s over 1.00 seconds
Interim result: 448.41 MBytes/s over 1.00 seconds
# netperf -H 10.10.0.90 -fM -l 600 -D 1, -- -m 2000
TCP STREAM TEST from 0.0.0.0 (0.0.0.0) port 0 AF_INET to 10.10.0.90 (10.10.0.90) port 0 AF_INET : demo
Interim result: 454.45 MBytes/s over 1.00 seconds
Interim result: 455.10 MBytes/s over 1.00 seconds
Interim result: 454.87 MBytes/s over 1.00 seconds
Interim result: 454.64 MBytes/s over 1.00 seconds
Interim result: 454.30 MBytes/s over 1.00 seconds
Interim result: 454.11 MBytes/s over 1.00 seconds
Interim result: 454.08 MBytes/s over 1.00 seconds
Interim result: 454.07 MBytes/s over 1.00 seconds
Interim result: 454.06 MBytes/s over 1.00 seconds
Interim result: 454.19 MBytes/s over 1.00 seconds
Interim result: 454.42 MBytes/s over 1.00 seconds
Interim result: 454.31 MBytes/s over 1.00 seconds
# netperf -H 10.10.0.90 -fM -l 600 -D 1, -- -m 2000 -D L,R
TCP STREAM TEST from 0.0.0.0 (0.0.0.0) port 0 AF_INET to 10.10.0.90 (10.10.0.90) port 0 AF_INET : nodelay : demo
Interim result: 305.93 MBytes/s over 1.00 seconds
Interim result: 306.84 MBytes/s over 1.00 seconds
Interim result: 306.88 MBytes/s over 1.00 seconds
Interim result: 306.80 MBytes/s over 1.00 seconds
Interim result: 306.69 MBytes/s over 1.00 seconds
Interim result: 306.41 MBytes/s over 1.00 seconds
Interim result: 307.25 MBytes/s over 1.00 seconds
Interim result: 307.56 MBytes/s over 1.00 seconds
Interim result: 307.51 MBytes/s over 1.00 seconds
Interim result: 307.58 MBytes/s over 1.00 seconds
connected mode mtu 64000
# netperf -H 10.10.0.90 -fM -l 600 -D 1, -- -m 64000
TCP STREAM TEST from 0.0.0.0 (0.0.0.0) port 0 AF_INET to 10.10.0.90 (10.10.0.90) port 0 AF_INET : demo
Interim result: 932.87 MBytes/s over 1.21 seconds
Interim result: 931.33 MBytes/s over 1.00 seconds
Interim result: 930.20 MBytes/s over 1.00 seconds
Interim result: 929.64 MBytes/s over 1.00 seconds
Interim result: 931.16 MBytes/s over 1.00 seconds
Interim result: 929.96 MBytes/s over 1.00 seconds
# netperf -H 10.10.0.90 -fM -l 600 -D 1, -- -m 2000
TCP STREAM TEST from 0.0.0.0 (0.0.0.0) port 0 AF_INET to 10.10.0.90 (10.10.0.90) port 0 AF_INET : demo
Interim result: 932.82 MBytes/s over 1.46 seconds
Interim result: 930.69 MBytes/s over 1.00 seconds
Interim result: 931.36 MBytes/s over 1.00 seconds
Interim result: 930.05 MBytes/s over 1.00 seconds
Interim result: 931.40 MBytes/s over 1.00 seconds
Interim result: 930.07 MBytes/s over 1.00 seconds
# netperf -H 10.10.0.90 -fM -l 600 -D 1, -- -m 2000 -D L,R
TCP STREAM TEST from 0.0.0.0 (0.0.0.0) port 0 AF_INET to 10.10.0.90 (10.10.0.90) port 0 AF_INET : nodelay : demo
Interim result: 475.01 MBytes/s over 1.58 seconds
Interim result: 474.22 MBytes/s over 1.00 seconds
Interim result: 474.25 MBytes/s over 1.00 seconds
Interim result: 473.14 MBytes/s over 1.00 seconds
Interim result: 473.62 MBytes/s over 1.00 seconds
Interim result: 474.37 MBytes/s over 1.00 seconds
Interim result: 473.81 MBytes/s over 1.00 seconds
More information about the general
mailing list