[ofa-general] SDP and stock kernel gets BUG?
Maarten van Malland
m.van.malland at linvision.com
Tue Jul 14 08:08:31 PDT 2009
Hi Amir,
I'm testing with Jason to get this SDP working properly on a 2.6.30.1 stock
kernel.
With the SDP from OFED 1.5 this is the performance we're getting:
inftsttwin03 ~ # LD_PRELOAD=/opt/ofa-1.5/lib/libsdp.so nttcp -r -T -l 128000
twin1-ibl
Bytes Real s CPU s Real-MBit/s CPU-MBit/s Calls Real-C/s
CPU-C/s
l262144000 0.98 0.28 2142.7300 7599.5333 3167 3235.83
11476.4
1262144000 0.98 0.02 2143.1132 131096.5806 2048 2092.88
128024.0
inftsttwin03 ~ # LD_PRELOAD=/opt/ofa-1.5/lib/libsdp.so nttcp -r -T -l 128000
twin1-ibl
Bytes Real s CPU s Real-MBit/s CPU-MBit/s Calls Real-C/s
CPU-C/s
l262144000 1.20 0.05 1748.5024 41948.9128 3345 2788.90
66909.4
1262144000 1.14 0.01 1847.3367 299678.7654 2048 1804.04
292655.0
However, with the SDP from OFED 1.4 we're getting this performance:
inftsttwin03 ~ # LD_PRELOAD=/opt/ofa-1.5/lib/libsdp.so nttcp -r -T -l 128000
twin1-ibl
Bytes Real s CPU s Real-MBit/s CPU-MBit/s Calls Real-C/s
CPU-C/s
l262144000 0.15 0.14 14411.9300 14565.7809 7007 48153.11
48667.2
1262144000 0.15 0.10 14410.4446 21848.5196 2048 14072.70
21336.4
inftsttwin03 ~ # LD_PRELOAD=/opt/ofa-1.5/lib/libsdp.so nttcp -r -T -l 128000
twin1-ibl
Bytes Real s CPU s Real-MBit/s CPU-MBit/s Calls Real-C/s
CPU-C/s
l262144000 0.15 0.14 14286.1658 14465.3120 6614 45055.72
45620.7
1262144000 0.15 0.09 14312.5883 22798.5998 2048 13977.14
22264.3
The output from /proc/net/sdpstats is:
inftsttwin03 ~ # cat /proc/net/sdpstats
SDP statistics:
sendmsg_seglen:
1 | - 0
2 | - 0
4 | - 0
8 | ************************************************** - 1
16 | - 0
32 | - 0
64 | ************************************************** - 1
128 | - 0
256 | - 0
512 | - 0
1024 | - 0
2048 | - 0
4096 | - 0
8192 | - 0
16384 | - 0
32768 | - 0
65536 | - 0
131072 | - 0
262144 | - 0
524288 | - 0
1048576 | - 0
2097152 | - 0
4194304 | - 0
8388608 | - 0
0 | - 0
send_size:
1 | - 0
2 | - 0
4 | - 0
8 | - 0
16 | ************************************************** - 325
32 | - 0
64 | - 1
128 | - 0
256 | - 0
512 | - 0
1024 | - 0
2048 | - 0
4096 | - 0
8192 | - 0
16384 | - 0
32768 | - 0
65536 | - 0
131072 | - 0
262144 | - 0
524288 | - 0
1048576 | - 0
2097152 | - 0
4194304 | - 0
8388608 | - 0
0 | - 0
credits_before_update:
0 | - 0
1 | - 0
2 | - 0
3 | - 0
4 | - 0
5 | - 0
6 | - 0
7 | - 0
8 | - 0
9 | - 0
10 | - 0
11 | - 0
12 | - 0
13 | - 0
14 | - 0
15 | - 0
16 | - 0
17 | - 0
18 | - 0
19 | - 0
20 | - 0
21 | - 0
22 | - 0
23 | - 0
24 | - 0
25 | - 0
26 | - 0
27 | - 0
28 | - 0
29 | - 0
30 | - 0
31 | - 0
32 | - 0
33 | - 0
34 | - 0
35 | - 0
36 | - 0
37 | - 0
38 | - 0
39 | - 0
40 | - 0
41 | - 0
42 | - 0
43 | - 0
44 | - 0
45 | - 0
46 | - 0
47 | - 0
48 | - 0
49 | - 0
50 | - 0
51 | - 0
52 | - 0
53 | - 1
54 | ************ - 1619
55 | ************************************************** - 6404
56 | - 0
57 | - 0
58 | - 0
59 | - 0
60 | - 0
61 | - 0
62 | - 0
63 | - 0
sdp_sendmsg() calls : 2
bcopy segments : 2
bzcopy segments : 0
post_send_credits : 322
memcpy_count : 98
post_send SDP_MID_HELLO : 0
post_send SDP_MID_HELLO_ACK : 0
post_send SDP_MID_DISCONN : 2
post_send SDP_MID_CHRCVBUF : 0
post_send SDP_MID_CHRCVBUF_ACK : 0
post_send SDP_MID_DATA : 324
post_recv : 8134
BZCopy poll miss : 0
send_wait_for_mem : 0
send_miss_no_credits : 0
rx_poll_miss : 0
tx_poll_miss : 3
tx_poll_busy : 0
tx_poll_hit : 3
CQ stats:
- RX interrupts : 3151
- TX interrupts : 0
bz_clean : 180
bz_setup : 345
tx_copy : 2756
sendmsg : 535432
The first 100 lines from /proc/net/sdpprf:
inftsttwin03 ~ # head -100 /proc/net/sdpprf
0 : [ 0.000000] TX: SDP_MID_DATA bufs: 55 mseq:1 ack:0 -
[7916{3} 48477:5037] skb: ffff880123186d80 sdp_post_send:99
1 : [ 0.000248] tx completion. mseq:1 -
[7916{3} 48477:5037] skb: ffff880123186d80 sdp_handle_send_comp:220
2 : [ 0.099838] TX: SDP_MID_DATA bufs: 55 mseq:2 ack:0 -
[0{3} 48477:5037] skb: ffff880123186bc0 sdp_post_send:99
3 : [ 0.102805] RX SDP_MID_DATA +55 c:55->55 mseq:1 ack:0 -
[0{0} 5038:51962] skb: ffff88011e055100 sdp_process_rx_skb:489
4 : [ 0.102807] Waking up sleepers -
[0{0} 5038:51962] skb: (null) sdp_bzcopy_write_space:602
5 : [ 0.102886] READ finished. mseq: 1 mseq_ack:0 -
[7916{0} 5038:51962] skb: ffff88011e055100 sdp_recvmsg:2146
6 : [ 0.102862] RX SDP_MID_DATA +55 c:55->55 mseq:2 ack:0 -
[7916{0} 5038:51962] skb: ffff88011dc78dc0 sdp_process_rx_skb:489
7 : [ 0.102862] Waking up sleepers -
[7916{0} 5038:51962] skb: (null) sdp_bzcopy_write_space:602
8 : [ 0.102960] READ finished. mseq: 2 mseq_ack:0 -
[7916{0} 5038:51962] skb: ffff88011dc78dc0 sdp_recvmsg:2146
9 : [ 0.102927] RX SDP_MID_DATA +55 c:55->55 mseq:3 ack:0 -
[7916{0} 5038:51962] skb: ffff88011dc78c00 sdp_process_rx_skb:489
10 : [ 0.102927] Waking up sleepers -
[7916{0} 5038:51962] skb: (null) sdp_bzcopy_write_space:602
11 : [ 0.103024] READ finished. mseq: 3 mseq_ack:0 -
[7916{0} 5038:51962] skb: ffff88011dc78c00 sdp_recvmsg:2146
12 : [ 0.103264] RX SDP_MID_DATA +55 c:55->55 mseq:4 ack:0 -
[0{0} 5038:51962] skb: ffff88011dc78a40 sdp_process_rx_skb:489
13 : [ 0.103265] Waking up sleepers -
[0{0} 5038:51962] skb: (null) sdp_bzcopy_write_space:602
14 : [ 0.103320] READ finished. mseq: 4 mseq_ack:0 -
[7916{0} 5038:51962] skb: ffff88011dc78a40 sdp_recvmsg:2146
15 : [ 0.103352] RX SDP_MID_DATA +55 c:55->55 mseq:5 ack:0 -
[0{0} 5038:51962] skb: ffff88011dc78880 sdp_process_rx_skb:489
16 : [ 0.103353] Waking up sleepers -
[0{0} 5038:51962] skb: (null) sdp_bzcopy_write_space:602
17 : [ 0.103362] RX SDP_MID_DATA +55 c:55->55 mseq:6 ack:0 -
[7916{0} 5038:51962] skb: ffff88011dc786c0 sdp_process_rx_skb:489
18 : [ 0.103363] Waking up sleepers -
[7916{0} 5038:51962] skb: (null) sdp_bzcopy_write_space:602
19 : [ 0.103422] READ finished. mseq: 5 mseq_ack:0 -
[7916{0} 5038:51962] skb: ffff88011dc78880 sdp_recvmsg:2146
20 : [ 0.103376] RX SDP_MID_DATA +55 c:55->55 mseq:7 ack:0 -
[7916{0} 5038:51962] skb: ffff88011dc78500 sdp_process_rx_skb:489
21 : [ 0.103377] Waking up sleepers -
[7916{0} 5038:51962] skb: (null) sdp_bzcopy_write_space:602
22 : [ 0.103398] RX SDP_MID_DATA +55 c:55->55 mseq:8 ack:0 -
[7916{0} 5038:51962] skb: ffff88011dc78340 sdp_process_rx_skb:489
23 : [ 0.103398] Waking up sleepers -
[7916{0} 5038:51962] skb: (null) sdp_bzcopy_write_space:602
24 : [ 0.103408] RX SDP_MID_DATA +55 c:55->55 mseq:9 ack:0 -
[7916{0} 5038:51962] skb: ffff88011dc78180 sdp_process_rx_skb:489
25 : [ 0.103408] Waking up sleepers -
[7916{0} 5038:51962] skb: (null) sdp_bzcopy_write_space:602
26 : [ 0.103420] RX SDP_MID_DATA +55 c:55->55 mseq:10 ack:0 -
[7916{0} 5038:51962] skb: ffff8801231b3e00 sdp_process_rx_skb:489
27 : [ 0.103421] Waking up sleepers -
[7916{0} 5038:51962] skb: (null) sdp_bzcopy_write_space:602
28 : [ 0.103429] RX SDP_MID_DATA +55 c:55->55 mseq:11 ack:0 -
[7916{0} 5038:51962] skb: ffff8801231b3c40 sdp_process_rx_skb:489
29 : [ 0.103429] Waking up sleepers -
[7916{0} 5038:51962] skb: (null) sdp_bzcopy_write_space:602
30 : [ 0.103479] READ finished. mseq: 6 mseq_ack:0 -
[7916{0} 5038:51962] skb: ffff88011dc786c0 sdp_recvmsg:2146
31 : [ 0.103532] READ finished. mseq: 7 mseq_ack:0 -
[7916{0} 5038:51962] skb: ffff88011dc78500 sdp_recvmsg:2146
32 : [ 0.103598] READ finished. mseq: 8 mseq_ack:0 -
[7916{0} 5038:51962] skb: ffff88011dc78340 sdp_recvmsg:2146
33 : [ 0.103612] READ finished. mseq: 9 mseq_ack:0 -
[7916{0} 5038:51962] skb: ffff88011dc78180 sdp_recvmsg:2146
34 : [ 0.103626] READ finished. mseq: 10 mseq_ack:0 -
[7916{0} 5038:51962] skb: ffff8801231b3e00 sdp_recvmsg:2146
35 : [ 0.103640] RX SDP_MID_DATA +55 c:55->55 mseq:12 ack:0 -
[7916{0} 5038:51962] skb: ffff8801231b3a80 sdp_process_rx_skb:489
36 : [ 0.103640] Waking up sleepers -
[7916{0} 5038:51962] skb: (null) sdp_bzcopy_write_space:602
37 : [ 0.103644] READ finished. mseq: 11 mseq_ack:0 -
[7916{0} 5038:51962] skb: ffff8801231b3c40 sdp_recvmsg:2146
38 : [ 0.103655] RX SDP_MID_DATA +55 c:55->55 mseq:13 ack:0 -
[7916{0} 5038:51962] skb: ffff8801231b38c0 sdp_process_rx_skb:489
39 : [ 0.103656] Waking up sleepers -
[7916{0} 5038:51962] skb: (null) sdp_bzcopy_write_space:602
40 : [ 0.103661] RX SDP_MID_DATA +55 c:55->55 mseq:14 ack:0 -
[7916{0} 5038:51962] skb: ffff8801231b3700 sdp_process_rx_skb:489
41 : [ 0.103661] Waking up sleepers -
[7916{0} 5038:51962] skb: (null) sdp_bzcopy_write_space:602
42 : [ 0.103671] RX SDP_MID_DATA +55 c:55->55 mseq:15 ack:0 -
[7916{0} 5038:51962] skb: ffff8801231b3540 sdp_process_rx_skb:489
43 : [ 0.103671] Waking up sleepers -
[7916{0} 5038:51962] skb: (null) sdp_bzcopy_write_space:602
44 : [ 0.103679] READ finished. mseq: 12 mseq_ack:0 -
[7916{0} 5038:51962] skb: ffff8801231b3a80 sdp_recvmsg:2146
45 : [ 0.103695] READ finished. mseq: 13 mseq_ack:0 -
[7916{0} 5038:51962] skb: ffff8801231b38c0 sdp_recvmsg:2146
46 : [ 0.103712] READ finished. mseq: 14 mseq_ack:0 -
[7916{0} 5038:51962] skb: ffff8801231b3700 sdp_recvmsg:2146
47 : [ 0.103728] READ finished. mseq: 15 mseq_ack:0 -
[7916{0} 5038:51962] skb: ffff8801231b3540 sdp_recvmsg:2146
48 : [ 0.104631] RX SDP_MID_DATA +55 c:55->55 mseq:16 ack:0 -
[0{0} 5038:51962] skb: ffff8801231b3380 sdp_process_rx_skb:489
49 : [ 0.104632] Waking up sleepers -
[0{0} 5038:51962] skb: (null) sdp_bzcopy_write_space:602
50 : [ 0.104641] RX SDP_MID_DATA +55 c:55->55 mseq:17 ack:0 -
[7916{0} 5038:51962] skb: ffff8801231b31c0 sdp_process_rx_skb:489
51 : [ 0.104642] Waking up sleepers -
[7916{0} 5038:51962] skb: (null) sdp_bzcopy_write_space:602
52 : [ 0.104653] RX SDP_MID_DATA +55 c:55->55 mseq:18 ack:0 -
[7916{0} 5038:51962] skb: ffff88011e0b8cc0 sdp_process_rx_skb:489
53 : [ 0.104653] Waking up sleepers -
[7916{0} 5038:51962] skb: (null) sdp_bzcopy_write_space:602
54 : [ 0.104657] READ finished. mseq: 16 mseq_ack:0 -
[7916{0} 5038:51962] skb: ffff8801231b3380 sdp_recvmsg:2146
55 : [ 0.104661] RX SDP_MID_DATA +55 c:55->55 mseq:19 ack:0 -
[7916{0} 5038:51962] skb: ffff88011e0b8b00 sdp_process_rx_skb:489
56 : [ 0.104662] Waking up sleepers -
[7916{0} 5038:51962] skb: (null) sdp_bzcopy_write_space:602
57 : [ 0.104674] READ finished. mseq: 17 mseq_ack:0 -
[7916{0} 5038:51962] skb: ffff8801231b31c0 sdp_recvmsg:2146
58 : [ 0.104688] READ finished. mseq: 18 mseq_ack:0 -
[7916{0} 5038:51962] skb: ffff88011e0b8cc0 sdp_recvmsg:2146
59 : [ 0.104703] READ finished. mseq: 19 mseq_ack:0 -
[7916{0} 5038:51962] skb: ffff88011e0b8b00 sdp_recvmsg:2146
60 : [ 0.105630] RX SDP_MID_DATA +55 c:55->55 mseq:20 ack:0 -
[0{0} 5038:51962] skb: ffff88011e0b8940 sdp_process_rx_skb:489
61 : [ 0.105632] Waking up sleepers -
[0{0} 5038:51962] skb: (null) sdp_bzcopy_write_space:602
62 : [ 0.105640] RX SDP_MID_DATA +55 c:55->55 mseq:21 ack:0 -
[7916{0} 5038:51962] skb: ffff88011e0b8780 sdp_process_rx_skb:489
63 : [ 0.105641] Waking up sleepers -
[7916{0} 5038:51962] skb: (null) sdp_bzcopy_write_space:602
64 : [ 0.105652] RX SDP_MID_DATA +55 c:55->55 mseq:22 ack:0 -
[7916{0} 5038:51962] skb: ffff88011e0b85c0 sdp_process_rx_skb:489
65 : [ 0.105652] Waking up sleepers -
[7916{0} 5038:51962] skb: (null) sdp_bzcopy_write_space:602
66 : [ 0.105656] READ finished. mseq: 20 mseq_ack:0 -
[7916{0} 5038:51962] skb: ffff88011e0b8940 sdp_recvmsg:2146
67 : [ 0.105661] RX SDP_MID_DATA +55 c:55->55 mseq:23 ack:0 -
[7916{0} 5038:51962] skb: ffff88011e0b8400 sdp_process_rx_skb:489
68 : [ 0.105661] Waking up sleepers -
[7916{0} 5038:51962] skb: (null) sdp_bzcopy_write_space:602
69 : [ 0.105673] READ finished. mseq: 21 mseq_ack:0 -
[7916{0} 5038:51962] skb: ffff88011e0b8780 sdp_recvmsg:2146
70 : [ 0.105687] READ finished. mseq: 22 mseq_ack:0 -
[7916{0} 5038:51962] skb: ffff88011e0b85c0 sdp_recvmsg:2146
71 : [ 0.105702] READ finished. mseq: 23 mseq_ack:0 -
[7916{0} 5038:51962] skb: ffff88011e0b8400 sdp_recvmsg:2146
72 : [ 0.106630] RX SDP_MID_DATA +55 c:55->55 mseq:24 ack:0 -
[0{0} 5038:51962] skb: ffff88011e0b8240 sdp_process_rx_skb:489
73 : [ 0.106631] Waking up sleepers -
[0{0} 5038:51962] skb: (null) sdp_bzcopy_write_space:602
74 : [ 0.106640] RX SDP_MID_DATA +55 c:55->55 mseq:25 ack:0 -
[7916{0} 5038:51962] skb: ffff88011e0b8080 sdp_process_rx_skb:489
75 : [ 0.106640] Waking up sleepers -
[7916{0} 5038:51962] skb: (null) sdp_bzcopy_write_space:602
76 : [ 0.106652] RX SDP_MID_DATA +55 c:55->55 mseq:26 ack:0 -
[7916{0} 5038:51962] skb: ffff88011b5a1d00 sdp_process_rx_skb:489
77 : [ 0.106653] Waking up sleepers -
[7916{0} 5038:51962] skb: (null) sdp_bzcopy_write_space:602
78 : [ 0.106656] READ finished. mseq: 24 mseq_ack:0 -
[7916{0} 5038:51962] skb: ffff88011e0b8240 sdp_recvmsg:2146
79 : [ 0.106661] RX SDP_MID_DATA +55 c:55->55 mseq:27 ack:0 -
[7916{0} 5038:51962] skb: ffff88011b5a1b40 sdp_process_rx_skb:489
80 : [ 0.106661] Waking up sleepers -
[7916{0} 5038:51962] skb: (null) sdp_bzcopy_write_space:602
81 : [ 0.106673] READ finished. mseq: 25 mseq_ack:0 -
[7916{0} 5038:51962] skb: ffff88011e0b8080 sdp_recvmsg:2146
82 : [ 0.106686] READ finished. mseq: 26 mseq_ack:0 -
[7916{0} 5038:51962] skb: ffff88011b5a1d00 sdp_recvmsg:2146
83 : [ 0.106701] READ finished. mseq: 27 mseq_ack:0 -
[7916{0} 5038:51962] skb: ffff88011b5a1b40 sdp_recvmsg:2146
84 : [ 0.107630] RX SDP_MID_DATA +55 c:55->55 mseq:28 ack:0 -
[0{0} 5038:51962] skb: ffff88011b5a1980 sdp_process_rx_skb:489
85 : [ 0.107632] Waking up sleepers -
[0{0} 5038:51962] skb: (null) sdp_bzcopy_write_space:602
86 : [ 0.107640] RX SDP_MID_DATA +55 c:55->55 mseq:29 ack:0 -
[7916{0} 5038:51962] skb: ffff88011b5a17c0 sdp_process_rx_skb:489
87 : [ 0.107641] Waking up sleepers -
[7916{0} 5038:51962] skb: (null) sdp_bzcopy_write_space:602
88 : [ 0.107652] RX SDP_MID_DATA +55 c:55->55 mseq:30 ack:0 -
[7916{0} 5038:51962] skb: ffff88011b5a1600 sdp_process_rx_skb:489
89 : [ 0.107653] Waking up sleepers -
[7916{0} 5038:51962] skb: (null) sdp_bzcopy_write_space:602
90 : [ 0.107658] TX: SDP_MID_DATA bufs: 53 mseq:1 ack:30 -
[7916{0} 5038:51962] skb: ffff88011b67c180 sdp_post_send:99
91 : [ 0.107662] READ finished. mseq: 28 mseq_ack:0 -
[7916{0} 5038:51962] skb: ffff88011b5a1980 sdp_recvmsg:2146
92 : [ 0.107661] RX SDP_MID_DATA +55 c:54->54 mseq:31 ack:0 -
[7916{0} 5038:51962] skb: ffff88011b5a1440 sdp_process_rx_skb:489
93 : [ 0.107661] Waking up sleepers -
[7916{0} 5038:51962] skb: (null) sdp_bzcopy_write_space:602
94 : [ 0.107677] tx completion. mseq:1 -
[7916{0} 5038:51962] skb: ffff88011b67c180 sdp_handle_send_comp:220
95 : [ 0.107678] READ finished. mseq: 29 mseq_ack:0 -
[7916{0} 5038:51962] skb: ffff88011b5a17c0 sdp_recvmsg:2146
96 : [ 0.107691] READ finished. mseq: 30 mseq_ack:0 -
[7916{0} 5038:51962] skb: ffff88011b5a1600 sdp_recvmsg:2146
97 : [ 0.107706] READ finished. mseq: 31 mseq_ack:0 -
[7916{0} 5038:51962] skb: ffff88011b5a1440 sdp_recvmsg:2146
98 : [ 0.107742] RX SDP_MID_DATA +55 c:54->55 mseq:32 ack:1 -
[0{0} 5038:51962] skb: ffff88011b5a1280 sdp_process_rx_skb:489
99 : [ 0.107743] Waking up sleepers -
[0{0} 5038:51962] skb: (null) sdp_bzcopy_write_space:602
Hope this helps,
Maarten
-----Oorspronkelijk bericht-----
Van: Amir Vadai [mailto:amirv at mellanox.co.il]
Verzonden: dinsdag 14 juli 2009 16:37
Aan: Jason Gunthorpe
CC: general-list
Onderwerp: Re: [ofa-general] SDP and stock kernel gets BUG?
OFED-1.5 SDP has many changes in the data path.
Performance should be improved - although it is still work in progress.
You could use /proc/sdp/sdpstats and /proc/sdp/sdpprf facilities that
has been added in 1.5.
(enabled in sdp.h by defining macros SDPSTATS_ON and SDP_PROFILING).
What packet sizes do you use? what is the setup in general?
Let me know if you find something interesting.
- Amir
On 07/14/2009 05:31 PM, Jason Gunthorpe wrote:
> On Tue, Jul 14, 2009 at 04:14:59PM +0300, Amir Vadai wrote:
>
>> Hi,
>>
>> I will post a fix soon.
>>
> Thanks Amir!
>
> BTW - we are testing SDP here and trying to track down a performance
> regression - using the 2.6.30.1 combined with OFED-1.5 SDP performs
> poorly while 2.6.27.10 combined with OFED-1.4 SDP performs well. We
> have not yet narrowed down what is going on.. Don't suppose you have
> any insight?
>
> Regards,
> Jason
>
More information about the general
mailing list