[ofa-general] Re: performance drop for datagram mode with the new connectx FW

Or Gerlitz ogerlitz at voltaire.com
Mon Jun 30 23:41:30 PDT 2008


> Using the new connectx FW (2.5), I see performance drop to almost
> zero with ipoib datagram mode. The code that runs on these systems
> is ofed 1.3 and not mainline kernel, details below.
>
> Running netperf With connected mode (64k MTU) I get about 950MB/s
> where with datagram mode (2k MTU) I get only 20-40MB/s. I used to
> see about 650MB/s and above with FW 2.3 and datagram mode. Not that
> it could explain the drop, but the NIC reports to the OS stateless
> offload support - /sys/class/net/ib1/features is 0x11423

I think to manage narrowing this a little further, the issue seems
most notable under long messages sent over datagram mode, could be
an issue in the LSO engine ?! see in this table,

====================================================
mode		mtu	size	bw	note
====================================================
datagram	2044	64000	30 <---------- problem
datagram	2044	2000	430
datagram	2044	2000	300	TCP_NODELAY
-----------------------------------------------------
connected	2044	64000	450
connected	2044	2000	450
connected	2044	2000	300	TCP_NODELAY
---------------------------------------------------
connected	64000	64000	930
connected	64000	2000	930
connected	64000	2000	470	TCP_NODELAY
====================================================

notes:

- in all cases, I have set the mtu manually
- verbs tests of bidirectional bandwidth shows that the HCA UD and RC engines work very well, and
  that the cables/switch etc are operating fine as the SDR limit is easily reached.


# qperf -li mlx4_0:2 -ri mlx4_0:1 172.25.5.77 -m 2000 -t 10 rc_bi_bw

rc_bi_bw:
    bw  =  1.91 GB/sec

# qperf -li mlx4_0:2 -ri mlx4_0:1 172.25.5.77 -m 2000 -t 10 ud_bi_bw
ud_bi_bw:
    send_bw  =  1.95 GB/sec
    recv_bw  =  1.95 GB/sec



datagram mode mtu 2044

# netperf -H 10.10.0.90 -fM -l 600 -D 1,  -- -m 64000
TCP STREAM TEST from 0.0.0.0 (0.0.0.0) port 0 AF_INET to 10.10.0.90 (10.10.0.90) port 0 AF_INET : demo
Interim result:   33.00 MBytes/s over 10.31 seconds
Interim result:   22.27 MBytes/s over 1.48 seconds
Interim result:   34.22 MBytes/s over 2.51 seconds
Interim result:   34.05 MBytes/s over 1.01 seconds
Interim result:   22.88 MBytes/s over 1.49 seconds
Interim result:   30.03 MBytes/s over 1.00 seconds
Interim result:   28.26 MBytes/s over 1.01 seconds
Interim result:   28.21 MBytes/s over 1.00 seconds
Interim result:   14.26 MBytes/s over 1.98 seconds

# netperf -H 10.10.0.90 -fM -l 600 -D 1,  -- -m 2000
TCP STREAM TEST from 0.0.0.0 (0.0.0.0) port 0 AF_INET to 10.10.0.90 (10.10.0.90) port 0 AF_INET : demo
Interim result:  429.54 MBytes/s over 1.00 seconds
Interim result:  429.22 MBytes/s over 1.00 seconds
Interim result:  429.50 MBytes/s over 1.00 seconds
Interim result:  429.47 MBytes/s over 1.00 seconds
Interim result:  429.35 MBytes/s over 1.00 seconds
Interim result:  429.10 MBytes/s over 1.00 seconds
Interim result:  429.16 MBytes/s over 1.00 seconds

# netperf -H 10.10.0.90 -fM -l 600 -D 1,  -- -m 2000 -D L,R
TCP STREAM TEST from 0.0.0.0 (0.0.0.0) port 0 AF_INET to 10.10.0.90 (10.10.0.90) port 0 AF_INET : nodelay : demo
Interim result:  294.15 MBytes/s over 1.32 seconds
Interim result:  294.27 MBytes/s over 1.00 seconds
Interim result:  294.98 MBytes/s over 1.00 seconds
Interim result:  294.74 MBytes/s over 1.00 seconds
Interim result:  294.45 MBytes/s over 1.00 seconds
Interim result:  294.53 MBytes/s over 1.00 seconds
Interim result:  294.35 MBytes/s over 1.00 seconds
Interim result:  294.43 MBytes/s over 1.00 seconds


connected mode mtu 2044

# netperf -H 10.10.0.90 -fM -l 600 -D 1,   -- -m 64000
TCP STREAM TEST from 0.0.0.0 (0.0.0.0) port 0 AF_INET to 10.10.0.90 (10.10.0.90) port 0 AF_INET : demo
Interim result:  161.05 MBytes/s over 1.71 seconds
Interim result:  231.38 MBytes/s over 1.00 seconds
Interim result:  231.16 MBytes/s over 1.00 seconds
Interim result:  232.40 MBytes/s over 1.00 seconds
Interim result:  322.28 MBytes/s over 1.00 seconds
Interim result:  448.25 MBytes/s over 1.00 seconds
Interim result:  448.54 MBytes/s over 1.00 seconds
Interim result:  448.36 MBytes/s over 1.00 seconds
Interim result:  448.52 MBytes/s over 1.00 seconds
Interim result:  448.18 MBytes/s over 1.00 seconds
Interim result:  448.06 MBytes/s over 1.00 seconds
Interim result:  448.26 MBytes/s over 1.00 seconds
Interim result:  448.41 MBytes/s over 1.00 seconds


# netperf -H 10.10.0.90 -fM -l 600 -D 1,   -- -m 2000
TCP STREAM TEST from 0.0.0.0 (0.0.0.0) port 0 AF_INET to 10.10.0.90 (10.10.0.90) port 0 AF_INET : demo
Interim result:  454.45 MBytes/s over 1.00 seconds
Interim result:  455.10 MBytes/s over 1.00 seconds
Interim result:  454.87 MBytes/s over 1.00 seconds
Interim result:  454.64 MBytes/s over 1.00 seconds
Interim result:  454.30 MBytes/s over 1.00 seconds
Interim result:  454.11 MBytes/s over 1.00 seconds
Interim result:  454.08 MBytes/s over 1.00 seconds
Interim result:  454.07 MBytes/s over 1.00 seconds
Interim result:  454.06 MBytes/s over 1.00 seconds
Interim result:  454.19 MBytes/s over 1.00 seconds
Interim result:  454.42 MBytes/s over 1.00 seconds
Interim result:  454.31 MBytes/s over 1.00 seconds

# netperf -H 10.10.0.90 -fM -l 600 -D 1,   -- -m 2000 -D L,R
TCP STREAM TEST from 0.0.0.0 (0.0.0.0) port 0 AF_INET to 10.10.0.90 (10.10.0.90) port 0 AF_INET : nodelay : demo
Interim result:  305.93 MBytes/s over 1.00 seconds
Interim result:  306.84 MBytes/s over 1.00 seconds
Interim result:  306.88 MBytes/s over 1.00 seconds
Interim result:  306.80 MBytes/s over 1.00 seconds
Interim result:  306.69 MBytes/s over 1.00 seconds
Interim result:  306.41 MBytes/s over 1.00 seconds
Interim result:  307.25 MBytes/s over 1.00 seconds
Interim result:  307.56 MBytes/s over 1.00 seconds
Interim result:  307.51 MBytes/s over 1.00 seconds
Interim result:  307.58 MBytes/s over 1.00 seconds

connected mode mtu 64000

# netperf -H 10.10.0.90 -fM -l 600 -D 1,   -- -m 64000
TCP STREAM TEST from 0.0.0.0 (0.0.0.0) port 0 AF_INET to 10.10.0.90 (10.10.0.90) port 0 AF_INET : demo
Interim result:  932.87 MBytes/s over 1.21 seconds
Interim result:  931.33 MBytes/s over 1.00 seconds
Interim result:  930.20 MBytes/s over 1.00 seconds
Interim result:  929.64 MBytes/s over 1.00 seconds
Interim result:  931.16 MBytes/s over 1.00 seconds
Interim result:  929.96 MBytes/s over 1.00 seconds

# netperf -H 10.10.0.90 -fM -l 600 -D 1,   -- -m 2000
TCP STREAM TEST from 0.0.0.0 (0.0.0.0) port 0 AF_INET to 10.10.0.90 (10.10.0.90) port 0 AF_INET : demo
Interim result:  932.82 MBytes/s over 1.46 seconds
Interim result:  930.69 MBytes/s over 1.00 seconds
Interim result:  931.36 MBytes/s over 1.00 seconds
Interim result:  930.05 MBytes/s over 1.00 seconds
Interim result:  931.40 MBytes/s over 1.00 seconds
Interim result:  930.07 MBytes/s over 1.00 seconds

# netperf -H 10.10.0.90 -fM -l 600 -D 1,   -- -m 2000 -D L,R
TCP STREAM TEST from 0.0.0.0 (0.0.0.0) port 0 AF_INET to 10.10.0.90 (10.10.0.90) port 0 AF_INET : nodelay : demo
Interim result:  475.01 MBytes/s over 1.58 seconds
Interim result:  474.22 MBytes/s over 1.00 seconds
Interim result:  474.25 MBytes/s over 1.00 seconds
Interim result:  473.14 MBytes/s over 1.00 seconds
Interim result:  473.62 MBytes/s over 1.00 seconds
Interim result:  474.37 MBytes/s over 1.00 seconds
Interim result:  473.81 MBytes/s over 1.00 seconds





More information about the general mailing list