[ofa-general] low bi bw with qperf for hca loopback test

[Or Gerlitz]

Hi Johann,

Using qperf-0.4.0-1.ofed1.3.1 on a system with DDR HCA (connectx) and PCIe gen2,
I see a low BW result for bidirectional BW when doing HCA loopback test.

Do you have any insight what is the reason for that (and how it can be fixed...) ?

When I use a different benchmark tool, I get a result of 4.9 GB/s which is close to the 5.3 GB/s
limit of my DRAM (see http://www.crucial.com/support/memory_speeds.aspx second table, third row)

Or.

# qperf -m 65500 -t 5 172.25.5.77 rc_bw
rc_bw:
    bw  =  1.96 GB/sec <--- limit is DDR BW, OK

# qperf -m 65500 -t 5 172.25.5.77 rc_bi_bw
rc_bi_bw:
    bw  =  3.89 GB/sec <--- DDR BI BW, OK

# qperf -m 65500 -t 5 127.0.0.1 rc_bw
rc_bw:
    bw  =  2.52 GB/sec <--- not perfect but nice

# qperf -m 65500 -t 5 127.0.0.1 rc_bi_bw
rc_bi_bw:
    bw  =  2.52 GB/sec <--- why ???


# ib_rdma_bw -b 172.25.5.77
28911: | port=18515 | ib_port=1 | size=65536 | tx_depth=100 | iters=1000 | duplex=1 | cma=0 |
28911: Bandwidth peak (#0 to #961): 3737.36 MB/sec
28911: Bandwidth average: 3737.35 MB/sec


# ib_rdma_bw -b 127.0.0.1
28913: | port=18515 | ib_port=1 | size=65536 | tx_depth=100 | iters=1000 | duplex=1 | cma=0 |

28913: Bandwidth peak (#0 to #999): 4910.35 MB/sec
28913: Bandwidth average: 4910.32 MB/sec <--- OK, close to the DRAM limit
28913: Service Demand peak (#0 to #999): 530 cycles/KB