[ewg] Tools to diagnose rdma performance issue?

[Chen, Mei-Jen]

Hi,

I am using OFED-3.5 , pretest-2.0, to check  rdma performance between two 4x QDR HCAs which connects with  a switch.
As the tested performance results are much slower than expected (please see attached result below, I expected more than 3000 MB/s).  There is probably something very wrong and I am struggling with find proper tools to diagnose/narrow down problems.  Any suggestion is appreciated.

The selected packages from OFED-3.5 were recompiled and installed on top of Linux 3.4 (with PREEMPT RT patch). The HCA driver comes from of kernel.org. 
So far I have tried perftest, qperf, ibv_rc_pingpong. They all have similar results.
Iblinkinfo has shown all links are up with correct speed (4X 10.0 Gbps). 

Thanks.


# ib_read_bw 192.168.200.2                         
---------------------------------------------------------------------------------------
Device not recognized to implement inline feature. Disabling it
---------------------------------------------------------------------------------------
                    RDMA_Read BW Test
 Dual-port       : OFF		Device : qib0
 Number of qps   : 1
 Connection type : RC
 TX depth        : 128
 CQ Moderation   : 100
 Mtu             : 2048B
 Link type       : IB
 Outstand reads  : 16
 rdma_cm QPs	 : OFF
 Data ex. method : Ethernet
---------------------------------------------------------------------------------------
 local address: LID 0x02 QPN 0x00a1 PSN 0x72c87b OUT 0x10 RKey 0x4f5000 VAddr 0x007f3190af2000
 remote address: LID 0x01 QPN 0x0095 PSN 0xca533a OUT 0x10 RKey 0x494a00 VAddr 0x007fe0f1f45000
---------------------------------------------------------------------------------------
 #bytes     #iterations    BW peak[MB/sec]    BW average[MB/sec]   MsgRate[Mpps]
 65536      1000           1709.24            1709.23		   0.027348
---------------------------------------------------------------------------------------

# ibstat
CA 'qib0'
	CA type: InfiniPath_QLE7340
	Number of ports: 1
	Firmware version: 
	Hardware version: 2
	Node GUID: 0x001175000070aafe
	System image GUID: 0x001175000070aafe
	Port 1:
		State: Active
		Physical state: LinkUp
		Rate: 40
		Base lid: 1
		LMC: 0
		SM lid: 1
		Capability mask: 0x0761086a
		Port GUID: 0x001175000070aafe
		Link layer: InfiniBand

______________________________________________________________________
This email has been scanned by the Symantec Email Security.cloud service.
For more information please visit http://www.symanteccloud.com
______________________________________________________________________