[ofa-general] Expected RDMA performance

[Chuck Hartley]

We have recently started using IB and are wondering if we are getting the
expected level of performance out of it.  Searching the Wiki and various
websites didn't reveal any tables showing what "typical" performance is.
Our hardware is several SuperMicro X7DBT-INF with onboard DDR HCA's, and a
24 port MT47396 switch.  We are running Fedora Core 6 (kernel
2.6.20-1.2948.fc6), 16GB memory, dual quad core 2.33GHz Xeons and the latest
BIOS/firmware versions for the components. Here is the output we get for
RDMA write BW (read test is similar):

                    RDMA_Write BW Test
Inline data is used up to 400 bytes message
Number of qp's running 1
Connection type : RC
Each Qp will post up to 100 messages each time
  local address:  LID 0x02, QPN 0x2d0409, PSN 0x7721a RKey 0x4a043100 VAddr
0x002aaaabafd000
  remote address: LID 0x04, QPN 0x50405, PSN 0xd8c024, RKey 0xe002600 VAddr
0x002aaaabaff000
Mtu : 2048
------------------------------------------------------------------
 #bytes #iterations    BW peak[MB/sec]    BW average[MB/sec]
      2        5000               2.67                  2.67
      4        5000               5.36                  5.35
      8        5000              10.75                 10.74
     16        5000              20.82                 20.78
     32        5000              41.60                 41.57
     64        5000              82.57                 82.45
    128        5000             156.77                156.73
    256        5000             230.09                229.86
    512        5000             677.01                676.31
   1024        5000            1101.94               1099.43
   2048        5000            1238.14               1237.90
   4096        5000            1288.37               1288.03
   8192        5000            1320.57               1320.40
  16384        5000            1330.15               1330.11
  32768        5000            1343.19               1343.19
  65536        5000            1347.03               1347.02
 131072        5000            1348.83               1348.82
 262144        5000            1341.16               1341.16
 524288        5000            1340.69               1340.69
1048576        5000            1341.46               1340.97
2097152        5000            1342.01               1342.01
4194304        5000            1342.10               1342.09
8388608        5000            1342.12               1342.12
------------------------------------------------------------------

Is this typical RDMA performance?  What is the maximum theoretical BW for
DDR IB - 1525MB/sec?  Or am I doing some math incorrectly?  No kernel
parameters have been modified, though I did not think much could be done to
affect RDMA performance.  We are having some issues with an SSD unit that I
want to understand, but need to make sure the basic IB installation is
working correctly first.  If the numbers above are good, then I'll move on
to the SSD question.  If the numbers are low, then I need some pointers to
what I should look at / change.

Thanks,
Chuck
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.openfabrics.org/pipermail/general/attachments/20071018/6a1b9183/attachment.html>