[ewg] Tools to diagnose rdma performance issue?

[Ido Shamai]

On 8/14/2013 11:20 PM, Chen, Mei-Jen wrote:

Hi Chen,

You definitely have a configuration error, as you should get around 3200 
MB/s, easily.
 From what I see here, the issue is probably using x4 PCIx slot, and 
PCIe gen2 (and not gen3) is configured.
In this case max throughout = PCI_LANES(4)*PCI_SPEED_PER_LANE(5 for 
gen2)*PCI_GEN2_ENCODING(0.8)*PCI_HEADERS_ADD(256/280)*PCI_FLOW_CTRL(0.95) = 
13.4 Gbit/a ~= 1700 MB/s.

So, you probably want to switch to x8 slot and change in BIOS to use 
PCIe generation 3.

Regards,
Ido

> Hi,
>
> I am using OFED-3.5 , pretest-2.0, to check  rdma performance between two 4x QDR HCAs which connects with  a switch.
> As the tested performance results are much slower than expected (please see attached result below, I expected more than 3000 MB/s).  There is probably something very wrong and I am struggling with find proper tools to diagnose/narrow down problems.  Any suggestion is appreciated.
>
> The selected packages from OFED-3.5 were recompiled and installed on top of Linux 3.4 (with PREEMPT RT patch). The HCA driver comes from of kernel.org.
> So far I have tried perftest, qperf, ibv_rc_pingpong. They all have similar results.
> Iblinkinfo has shown all links are up with correct speed (4X 10.0 Gbps).
>
> Thanks.
>
>
> # ib_read_bw 192.168.200.2
> ---------------------------------------------------------------------------------------
> Device not recognized to implement inline feature. Disabling it
> ---------------------------------------------------------------------------------------
>                      RDMA_Read BW Test
>   Dual-port       : OFF		Device : qib0
>   Number of qps   : 1
>   Connection type : RC
>   TX depth        : 128
>   CQ Moderation   : 100
>   Mtu             : 2048B
>   Link type       : IB
>   Outstand reads  : 16
>   rdma_cm QPs	 : OFF
>   Data ex. method : Ethernet
> ---------------------------------------------------------------------------------------
>   local address: LID 0x02 QPN 0x00a1 PSN 0x72c87b OUT 0x10 RKey 0x4f5000 VAddr 0x007f3190af2000
>   remote address: LID 0x01 QPN 0x0095 PSN 0xca533a OUT 0x10 RKey 0x494a00 VAddr 0x007fe0f1f45000
> ---------------------------------------------------------------------------------------
>   #bytes     #iterations    BW peak[MB/sec]    BW average[MB/sec]   MsgRate[Mpps]
>   65536      1000           1709.24            1709.23		   0.027348
> ---------------------------------------------------------------------------------------
>
> # ibstat
> CA 'qib0'
> 	CA type: InfiniPath_QLE7340
> 	Number of ports: 1
> 	Firmware version:
> 	Hardware version: 2
> 	Node GUID: 0x001175000070aafe
> 	System image GUID: 0x001175000070aafe
> 	Port 1:
> 		State: Active
> 		Physical state: LinkUp
> 		Rate: 40
> 		Base lid: 1
> 		LMC: 0
> 		SM lid: 1
> 		Capability mask: 0x0761086a
> 		Port GUID: 0x001175000070aafe
> 		Link layer: InfiniBand
>
> ______________________________________________________________________
> This email has been scanned by the Symantec Email Security.cloud service.
> For more information please visit http://www.symanteccloud.com
> ______________________________________________________________________
> _______________________________________________
> ewg mailing list
> ewg at lists.openfabrics.org
> http://lists.openfabrics.org/cgi-bin/mailman/listinfo/ewg
>