[ofa-general] Re: low bi bw with qperf for hca loopback test

[Johann George]

Hello Or.

Interesting results.  I assume in the loopback case, you are
running both instances of qperf on the same machine?  I'm
wondering if you are somehow CPU limited?  If you give qperf
the -v option, it will print out percentage CPU utilization.
Note that in its printout, a cpu is a core so 150%
utilization indicates 1.5 cores.

I'm also wondering (although most unlikely) if somehow both
instances of qperf are somehow stuck on the same CPU.  You
can use the -la and -ra options to set the processor
affinities of the client and server.

Finally, rc_bw and rc_bi_bw determine bandwidth using
Send/Receives.  I believe that the ib_rdma_bw utility uses
RDMA Writes.  For the equivalent test, use the qperf
rc_rdma_write_bw test.  you can get a list of all tests by
typing:

    qperf -h tests

Let me know what you find.

Thanks.

Johann

On Mon, Aug 11, 2008 at 03:46:37PM +0300, Or Gerlitz wrote:
> 
> Hi Johann,
> 
> Using qperf-0.4.0-1.ofed1.3.1 on a system with DDR HCA (connectx) and PCIe gen2,
> I see a low BW result for bidirectional BW when doing HCA loopback test.
> 
> Do you have any insight what is the reason for that (and how it can be fixed...) ?
> 
> When I use a different benchmark tool, I get a result of 4.9 GB/s which is close to the 5.3 GB/s
> limit of my DRAM (see http://www.crucial.com/support/memory_speeds.aspx second table, third row)
> 
> Or.
> 
> # qperf -m 65500 -t 5 172.25.5.77 rc_bw
> rc_bw:
>     bw  =  1.96 GB/sec <--- limit is DDR BW, OK
> 
> # qperf -m 65500 -t 5 172.25.5.77 rc_bi_bw
> rc_bi_bw:
>     bw  =  3.89 GB/sec <--- DDR BI BW, OK
> 
> # qperf -m 65500 -t 5 127.0.0.1 rc_bw
> rc_bw:
>     bw  =  2.52 GB/sec <--- not perfect but nice
> 
> # qperf -m 65500 -t 5 127.0.0.1 rc_bi_bw
> rc_bi_bw:
>     bw  =  2.52 GB/sec <--- why ???
> 
> 
> # ib_rdma_bw -b 172.25.5.77
> 28911: | port=18515 | ib_port=1 | size=65536 | tx_depth=100 | iters=1000 | duplex=1 | cma=0 |
> 28911: Bandwidth peak (#0 to #961): 3737.36 MB/sec
> 28911: Bandwidth average: 3737.35 MB/sec
> 
> 
> # ib_rdma_bw -b 127.0.0.1
> 28913: | port=18515 | ib_port=1 | size=65536 | tx_depth=100 | iters=1000 | duplex=1 | cma=0 |
> 
> 28913: Bandwidth peak (#0 to #999): 4910.35 MB/sec
> 28913: Bandwidth average: 4910.32 MB/sec <--- OK, close to the DRAM limit
> 28913: Service Demand peak (#0 to #999): 530 cycles/KB