[openib-general] Userspace testing results (many kernels, many svn trees)
Nishanth Aravamudan
nacc at us.ibm.com
Sun Jan 8 20:59:48 PST 2006
Hello all,
Here are more results, where each section's heading indicates
server-client size (e.g. 32-65 is a 32-bit server and a 64-bit client),
only related to userspace; that is both machines are running (identical)
64-bit kernels. Each row is headed by the particular kernel which was
booted on the two machines (If the kernel is suffixed with svn, then it
indicates that the OpenIB kernel components were used. If not, then it
is a mainline tree without modification). The userpace (and kernel,
where applicable) svn revision is listed following the kernel version.
This e-mail is ridiculously long, but there is *a lot* of data here...
Enjoy!
FYI, all of the errors in the footnotes are from the client-side. If you
would like to see what the server-side said, please just ask.
netpipe over IB
32-32
avg b/w (Mbps) peak b/w (Mbps)
rdma_write
2.6.15-rc7-git4-svn (4662) 1036.73 1839.97
2.6.15-rc7-git5-svn (4663) 1036.18 1839.98
2.6.15-rc7-git6-svn (4670) 1036.08 1839.99
2.6.15-rc7-git6-svn (4692) 1035.65 1839.97
2.6.15-svn (4714) 1036.25 1839.93
2.6.15-svn (4785) 1035.81 1840
2.6.15-svn (4789) 1036.48 1839.97
2.6.15 (4789) 1035.46 1839.98
2.6.15-svn (4803) 1035.57 1840.01
2.6.15 (4803) 1035.67 1839.97
rdma_write with immediate
Errors across the board [1]
send_recv
Errors across the board [2]
send_recv with immediate
Errors across the board [3]
32-64
avg b/w (Mbps) peak b/w (Mbps)
rdma_write
Errors across the board [4]
rdma_write with immediate
Errors across the board [1]
send_recv
Errors across the board [2]
send_recv with immediate
Errors across the board [3]
64-32
avg b/w (Mbps) peak b/w (Mbps)
rdma_write
Errors across the board [4]
rdma_write with immediate
Errors across the board [1]
send_recv
Errors across the board [2]
send_recv with immediate
Errors across the board [3]
64-64
avg b/w (Mbps) peak b/w (Mbps)
rdma_write
2.6.15-rc7-git4-svn (4662) 1037.22 1840.01
2.6.15-rc7-git5-svn (4663) Errors [5]
2.6.15-rc7-git6-svn (4670) 1037.58 1839.99
2.6.15-rc7-git6-svn (4692) 1037.28 1839.97
2.6.15-svn (4714) 1036.83 1839.98
2.6.15-svn (4785) 1037.32 1840.01
2.6.15-svn (4789) 1036.99 1840
2.6.15 (4789) 1036.74 1839.97
2.6.15-svn (4803) 1037.82 1839.95
2.6.15 (4803) 1038.05 1839.95
rdma_write with immediate
Errors across the board [1]
send_recv
Errors across the board [2]
send_recv with immediate
Errors across the board [3]
pingpong
32-32
b/w (Mbps)
rc
2.6.15-rc7-git4-svn (4662) 962.96
2.6.15-rc7-git5-svn (4663) 959.49
2.6.15-rc7-git6-svn (4670) 956.59
2.6.15-rc7-git6-svn (4692) 961.30
2.6.15-svn (4714) 957.58
2.6.15-svn (4785) 961.83
2.6.15-svn (4789) 962.79
2.6.15-svn (4803) 960.01
2.6.15 (4789) 883.95
2.6.15 (4803) 957.97
srq
2.6.15-rc7-git4-svn (4662) 3318.61
2.6.15-rc7-git5-svn (4663) 3247.41
2.6.15-rc7-git6-svn (4670) 3320.13
2.6.15-rc7-git6-svn (4692) 3294.26
2.6.15-svn (4714) 3267.16
2.6.15-svn (4785) 3293.27
2.6.15-svn (4789) 3347.09
2.6.15 (4789) 3277.46
2.6.15-svn (4803) 3386.00
2.6.15 (4803) 3144.87
uc
2.6.15-rc7-git4-svn (4662) 966.24
2.6.15-rc7-git5-svn (4663) 968.46
2.6.15-rc7-git6-svn (4670) 965.45
2.6.15-rc7-git6-svn (4692) 964.91
2.6.15-svn (4714) 970.53
2.6.15-svn (4785) 962.19
2.6.15-svn (4789) 967.02
2.6.15 (4789) 969.64
2.6.15-svn (4803) 964.62
2.6.15 (4803) 967.62
ud
2.6.15-rc7-git4-svn (4662) 466.30
2.6.15-rc7-git5-svn (4663) 465.32
2.6.15-rc7-git6-svn (4670) 465.64
2.6.15-rc7-git6-svn (4692) 465.42
2.6.15-svn (4714) 463.22
2.6.15-svn (4785) 464.58
2.6.15-svn (4789) 465.56
2.6.15 (4789) 465.97
2.6.15-svn (4803) 462.77
2.6.15 (4803) 462.94
32-64
b/w (Mbps)
rc
2.6.15-rc7-git4-svn (4662) 975.59
2.6.15-rc7-git5-svn (4663) 971.97
2.6.15-rc7-git6-svn (4670) 972.89
2.6.15-rc7-git6-svn (4692) 974.41
2.6.15-svn (4714) 968.46
2.6.15-svn (4785) 972.20
2.6.15-svn (4789) 967.26
2.6.15 (4789) 973.15
2.6.15-svn (4803) 968.55
2.6.15 (4803) 968.26
srq
2.6.15-rc7-git4-svn (4662) 3347.26
2.6.15-rc7-git5-svn (4663) 3301.56
2.6.15-rc7-git6-svn (4670) Error [5]
2.6.15-rc7-git6-svn (4692) 3330.42
2.6.15-svn (4714) 3337.54
2.6.15-svn (4785) 334.15
2.6.15-svn (4789) 3378.32
2.6.15 (4789) 3397.06
2.6.15-svn (4803) 3427.79
2.6.15 (4803) 3429.95
uc
2.6.15-rc7-git4-svn (4662) 977.55
2.6.15-rc7-git5-svn (4663) 963.25
2.6.15-rc7-git6-svn (4670) 979.67
2.6.15-rc7-git6-svn (4692) 977.61
2.6.15-svn (4714) 976.04
2.6.15-svn (4785) 974.37
2.6.15-svn (4789) 974.40
2.6.15 (4789) 978.41
2.6.15-svn (4803) 975.75
2.6.15 (4803) 977.76
ud
2.6.15-rc7-git4-svn (4662) 469.33
2.6.15-rc7-git5-svn (4663) 468.64
2.6.15-rc7-git6-svn (4670) 468.90
2.6.15-rc7-git6-svn (4692) 468.07
2.6.15-svn (4714) 468.14
2.6.15-svn (4785) 467.97
2.6.15-svn (4789) 467.30
2.6.15 (4789) 468.58
2.6.15-svn (4803) 462.95
2.6.15 (4803) 466.87
64-32
b/w (Mbps)
rc
2.6.15-rc7-git4-svn (4662) 974.95
2.6.15-rc7-git5-svn (4663) 972.01
2.6.15-rc7-git6-svn (4670) 974.17
2.6.15-rc7-git6-svn (4692) 970.75
2.6.15-svn (4714) 970.04
2.6.15-svn (4785) 964.25
2.6.15-svn (4789) 973.72
2.6.15 (4789) 974.80
2.6.15-svn (4803) 969.02
2.6.15 (4803) 968.62
srq
2.6.15-rc7-git4-svn (4662) 3312.58
2.6.15-rc7-git5-svn (4663) 3354.46
2.6.15-rc7-git6-svn (4670) 3344.36
2.6.15-rc7-git6-svn (4692) 3300.56
2.6.15-svn (4714) 3337.71
2.6.15-svn (4785) 3364.79
2.6.15-svn (4789) Error [5]
2.6.15 (4789) 3307.39
2.6.15-svn (4803) 3430.13
2.6.15 (4803) 3415.47
uc
2.6.15-rc7-git4-svn (4662) 973.99
2.6.15-rc7-git5-svn (4663) 975.11
2.6.15-rc7-git6-svn (4670) 981.45
2.6.15-rc7-git6-svn (4692) 978.91
2.6.15-svn (4714) 977.61
2.6.15-svn (4785) 974.28
2.6.15-svn (4789) 976.14
2.6.15 (4789) 975.51
2.6.15-svn (4803) 973.96
2.6.15 (4803) 972.13
ud
2.6.15-rc7-git4-svn (4662) 469.70
2.6.15-rc7-git5-svn (4663) 467.81
2.6.15-rc7-git6-svn (4670) Error [5]
2.6.15-rc7-git6-svn (4692) 467.97
2.6.15-svn (4714) 469.01
2.6.15-svn (4785) 468.09
2.6.15-svn (4789) 468.41
2.6.15 (4789) 468.94
2.6.15-svn (4803) 467.44
2.6.15 (4803) 467.53
64-64
b/w (Mbps)
rc
2.6.15-rc7-git4-svn (4662) 980.93
2.6.15-rc7-git5-svn (4663) Error [5]
2.6.15-rc7-git6-svn (4670) 982.37
2.6.15-rc7-git6-svn (4692) 982.76
2.6.15-svn (4714) 968.68
2.6.15-svn (4785) 983.86
2.6.15-svn (4789) 982.61
2.6.15 (4789) 868.61
2.6.15-svn (4803) 982.68
2.6.15 (4803) 981.84
srq
2.6.15-rc7-git4-svn (4662) 3379.36
2.6.15-rc7-git5-svn (4663) Error [5]
2.6.15-rc7-git6-svn (4670) 3303.73
2.6.15-rc7-git6-svn (4692) 3354.80
2.6.15-svn (4714) 3345.55
2.6.15-svn (4785) 3376.58
2.6.15-svn (4789) 3408.89
2.6.15 (4789) 2546.47
2.6.15-svn (4803) 3461.84
2.6.15 (4803) 3450.72
uc
2.6.15-rc7-git4-svn (4662) 987.23
2.6.15-rc7-git5-svn (4663) Error [5]
2.6.15-rc7-git6-svn (4670) 989.21
2.6.15-rc7-git6-svn (4692) 989.24
2.6.15-svn (4714) 985.30
2.6.15-svn (4785) 984.91
2.6.15-svn (4789) 986.27
2.6.15 (4789) 983.80
2.6.15-svn (4803) 985.37
2.6.15 (4803) 990.13
ud
2.6.15-rc7-git4-svn (4662) 472.34
2.6.15-rc7-git5-svn (4663) Error [5]
2.6.15-rc7-git6-svn (4670) 471.73
2.6.15-rc7-git6-svn (4692) 471.38
2.6.15-svn (4714) 471.28
2.6.15-svn (4785) 471.79
2.6.15-svn (4789) 472.37
2.6.15 (4789) 471.68
2.6.15-svn (4803) 471.04
2.6.15 (4803) 470.53
perftest latency [6]
32-32
typical (us) min max
rdma_lat
2.6.15-rc7-git4-svn (4662) 3.26954e+09 3.21451e+09 9.55805e+10
2.6.15-rc7-git5-svn (4663) 3.26015e+09 3.20109e+09 4.08654e+10
2.6.15-rc7-git6-svn (4670) 3.26552e+09 3.21586e+09 4.27914e+10
2.6.15-rc7-git6-svn (4692) 3.2682e+09 3.2172e+09 4.30169e+10
2.6.15-svn (4714) 3.26686e+09 3.20512e+09 4.5814e+10
2.6.15 (4785) 3.27491e+09 3.21988e+09 4.00909e+10
2.6.15-svn (4789) 3.26283e+09 3.20378e+09 4.42879e+10
2.6.15 (4789) 3.26686e+09 3.21586e+09 4.35323e+10
2.6.15-svn (4803) 3.27491e+09 3.22659e+09 5.65581e+10
2.6.15 (4803) 3.27625e+09 3.2172e+09 3.89313e+10
read_lat
2.6.15-svn (4803) 7274600857.60 7041062010.88 21923123691.52
2.6.15 (4803) 7282653921.28 7065221201.92 20535312384.00
send_lat
2.6.15-svn (4803) 4954647429.12 4771440230.40 43512045240.21
2.6.15 (4803) 4969411379.20 4799625953.28 46604421693.32
write_lat
2.6.15-svn (4803) 3291018690.56 3235989422.08 44894487838.72
2.6.15 (4803) 3286992158.72 3234647244.80 42607417753.60
32-64
typical (us) min max
rdma_lat
2.6.15-rc7-git4-svn (4662) 0.750625 0.7375 10.2784
2.6.15-rc7-git5-svn (4663) 0.753437 0.73875 10.1547
2.6.15-rc7-git6-svn (4670) 0.754687 0.740938 10.4453
2.6.15-rc7-git6-svn (4692) 0.753437 0.74 9.07281
2.6.15-svn (4714) 0.75625 0.744375 9.80438
2.6.15-svn (4785) 0.742812 0.74125 9.77406
2.6.15-svn (4789) 0.756875 0.744687 9.83906
2.6.15 (4789) 0.753125 0.74 9.15125
2.6.15-svn (4803) 0.753437 0.741875 12.2256
2.6.15 (4803) 0.75875 0.745 11.3766
read_lat
2.6.15-svn (4803) 1.63 1.58 8.59
2.6.15 (4803) 1.63 1.59 8.98
send_lat
2.6.15-svn (4803) 1.12 1.09 10.51
2.6.15 (4803) 1.13 1.09 9.44
write_lat
2.6.15-svn (4803) 0.75 0.74 10.24
2.6.15 (4803) 0.76 0.74 11.84
64-32
typical (us) min max
rdma_lat
2.6.15-rc7-git4-svn (4662) 3.22659e+09 3.17693e+09 4.30155e+10
2.6.15-rc7-git5-svn (4663) 3.22257e+09 3.16485e+09 4.63898e+10
2.6.15-rc7-git6-svn (4670) 3.21586e+09 3.1662e+09 4.29565e+10
2.6.15-rc7-git6-svn (4692) 3.23733e+09 3.17022e+09 4.34558e+10
2.6.15-svn (4714) 3.22794e+09 3.17425e+09 1.09938e+11
2.6.15-svn (4785) 3.23499e+09 3.17828e+09 4.08023e+10
2.6.15-svn (4789) 3.22928e+09 3.17425e+09 5.84103e+10
2.6.15 (4789) 3.2172e+09 3.17425e+09 5.45797e+10
2.6.15-svn (4803) 3.22391e+09 3.17291e+09 4.00372e+10
2.6.15 (4803) 3.23331e+09 3.17559e+09 1.30541e+12
read_lat
2.6.15-svn (4803) 7296075694.08 7041062010.88 71653476270.08
2.6.15 (4803) 7282653921.28 7046430720.00 20787641712.64
send_lat
2.6.15-svn (4803) 4818416435.20 4674803466.24 46003126271.89
2.6.15 (4803) 4821100789.76 4689567416.32 42077257727.89
write_lat
2.6.15-svn (4803) 3245384663.04 3201092812.80 44909251788.80
2.6.15 (4803) 3244042485.76 3193039749.12 59760443392.00
64-64
typical (us) min max
rdma_lat
2.6.15-rc7-git4-svn (4662) Errors [5]
2.6.15-rc7-git5-svn (4663) Errors [5]
2.6.15-rc7-git6-svn (4670) 0.745625 0.731563 10.5003
2.6.15-rc7-git6-svn (4692) 0.745313 0.73375 10.3
2.6.15-svn (4714) 0.747812 0.736875 24.7016
2.6.15-svn (4785) 0.745 0.735313 10.5237
2.6.15-svn (4789) 0.7425 0.73125 9.94781
2.6.15 (4789) 0.745938 0.736563 11.0213
2.6.15-svn (4803) 0.742188 0.730938 9.91563
2.6.15 (4803) 0.7425 0.730625 15.0122
read_lat
2.6.15-svn (4803) 1.63 1.58 8.61
2.6.15 (4803) 1.63 1.58 44.82
send_lat
2.6.15-svn (4803) 1.10 1.07 11.21
2.6.15 (4803) 1.10 1.07 10.89
write_lat
2.6.15-svn (4803) 0.74 0.73 11.00
2.6.15 (4803) 0.75 0.74 10.53
perftest bandwidth
32-32
peak (MBps) avg
rdma_bw
2.6.15-rc7-git4-svn (4662) 4.34461e-07 4.34461e-07
2.6.15-rc7-git5-svn (4663) 4.34457e-07 4.34457e-07
2.6.15-rc7-git6-svn (4670) 4.34455e-07 4.34454e-07
2.6.15-rc7-git6-svn (4692) 4.3451e-07 4.3451e-07
2.6.15-svn (4714) 4.34508e-07 4.34508e-07
2.6.15 (4785) 4.34518e-07 4.34518e-07
2.6.15-svn (4789) 4.34452e-07 4.34452e-07
2.6.15 (4789) 4.34453e-07 4.34453e-07
2.6.15-svn (4803) 4.34503e-07 4.34502e-07
2.6.15 (4803) 4.34446e-07 4.34446e-07
read_bw
2.6.15-svn (4803) 0.00 0.00
2.6.15 (4803) 0.00 0.00
send_bw
2.6.15-svn (4803) 0.00 0.00
2.6.15 (4803) 0.00 0.00
write_bw
2.6.15-svn (4803) 0.00 0.00
2.6.15 (4803) 0.00 0.00
32-64
peak (MBps) avg
rdma_bw
2.6.15-rc7-git4-svn (4662) 1866.19 1866.17
2.6.15-rc7-git5-svn (4663) 1866.16 1866.16
2.6.15-rc7-git6-svn (4670) 1865.88 1865.87
2.6.15-rc7-git6-svn (4692) 1865.98 1865.97
2.6.15-svn (4714) 1865.88 1865.86
2.6.15-svn (4785) 1866.16 1866.14
2.6.15-svn (4789) 1865.95 1865.95
2.6.15 (4789) 1865.95 1865.92
2.6.15-svn (4803) 1865.92 1865.89
2.6.15 (4803) 1865.92 1865.9
read_bw
2.6.15-svn (4803) 1840.20 1840.19
2.6.15 (4803) 1840.74 1840.71
send_bw
2.6.15-svn (4803) 1840.16 1840.16
2.6.15 (4803) 1840.37 1840.36
write_bw
2.6.15-svn (4803) 1841.55 1841.52
2.6.15 (4803) 1841.32 1841.31
64-32
peak (MBps) avg
rdma_bw
2.6.15-rc7-git4-svn (4662) 4.3449e-07 4.34489e-07
2.6.15-rc7-git5-svn (4663) 4.34456e-07 4.34456e-07
2.6.15-rc7-git6-svn (4670) 4.34458e-07 4.34458e-07
2.6.15-rc7-git6-svn (4692) 4.34507e-07 4.34507e-07
2.6.15-svn (4714) 4.34463e-07 4.34463e-07
2.6.15-svn (4785) 4.34454e-07 4.34454e-07
2.6.15-svn (4789) 4.34455e-07 4.34455e-07
2.6.15 (4789) 4.34443e-07 4.34443e-07
2.6.15-svn (4803) 4.34458e-07 4.34458e-07
2.6.15 (4803) 4.34501e-07 4.34501e-07
read_bw
2.6.15-svn (4803) 0.00 0.00
2.6.15 (4803) 0.00 0.00
send_bw
2.6.15-svn (4803) 0.00 0.00
2.6.15 (4803) 0.00 0.00
write_bw
2.6.15-svn (4803) 0.00 0.00
2.6.15 (4803) 0.00 0.00
64-64
peak (MBps) avg
rdma_bw
2.6.15-rc7-git4-svn (4662) Errors [5]
2.6.15-rc7-git5-svn (4663) Errors [5]
2.6.15-rc7-git6-svn (4670) 1865.95 1865.93
2.6.15-rc7-git6-svn (4692) 1865.95 1865.94
2.6.15-svn (4714) 1865.92 1865.91
2.6.15-svn (4785) 1865.92 1865.9
2.6.15-svn (4789) 1865.92 1865.89
2.6.15 (4789) 1866.16 1866.15
2.6.15-svn (4803) 1866.19 1866.16
2.6.15 (4803) 1865.95 1865.93
read_bw
2.6.15-svn (4803) 1840.50 1840.48
2.6.15 (4803) 1840.47 1840.46
send_bw
2.6.15-svn (4803) 1840.40 1840.38
2.6.15 (4803) 1840.20 1840.20
write_bw
2.6.15-svn (4803) 1841.32 1841.31
2.6.15 (4803) 1841.35 1841.32
Thanks,
Nish
[1]
Preposting asynchronous receives (required for Infiniband)
Error, local polling may only be used with RDMA Write.
Try using vapi polling or event completion
Using RDMA Write communications with immediate data
[2]
Preposting asynchronous receives (required for Infiniband)
Error, local polling may only be used with RDMA Write.
Try using vapi polling or event completion
Using Send/Receive communications
[3]
Preposting asynchronous receives (required for Infiniband)
Error, local polling may only be used with RDMA Write.
Try using vapi polling or event completion
Using Send/Receive communications with immediate data
[4]
Preposting asynchronous receives (required for Infiniband)
NetPIPE: error writing or reading synchronization string: Connection reset by peer
Using RDMA Write communications
This seems to be an actual error, not necessarily a programmatic one
like [1]-[3]. It seems netPIPE over IB verbs is unhappy with mixing the
size of the server client (does ok when they are the same, though).
[5]
These errors are due to our test grid, and thus are not indicative of
errors in the code.
[6]
Just like with the results I posted earlier, all the perftest results
are seriously wrong for 32-bit clients (with both 32-bit and 64-bit
servers). I am not sure who else to notify beyond the general list (is
there a corresponding MAINTAINERS files like in the kernel proper for
the OpenIB code?)
More information about the general
mailing list