[ewg] 200m cable results in slower rdma read performance? [ CC Anti-Virus checked ]

[Rupert Dance]

Hi,

 

Have you run ibdiagnet to verify that your link width and speed is what you
expect on all links?

 

Thanks

 

Rupert Dance

 

Software Forge

 

From: ewg-bounces at lists.openfabrics.org
[mailto:ewg-bounces at lists.openfabrics.org] On Behalf Of
koen_segers at computacenter.com
Sent: Monday, October 17, 2011 3:22 AM
To: ewg at lists.openfabrics.org
Subject: [ewg] 200m cable results in slower rdma read performance? [ CC
Anti-Virus checked ]

 

Hi, 

 

In my test setup I have 3 servers of which 2 are residing in Datacenter1 and
the other in Datacenter2.

If I do a rdma test between datacenters, I get a much lower performance than
if I would do the test between servers residing in the same datacenter.

 

 

DC1: gpfsprod1n1, gpfsprod1n3

DC2: gpfsprod1n2

 

08:54:48|root at gpfsprod1n1:~ 0 # qperf -t 5  cic-gpfsprod1n2 rc_rdma_write_bw

rc_rdma_write_bw:

    bw  =  1.9 GB/sec

08:54:59|root at gpfsprod1n1:~ 0 # qperf -t 5  cic-gpfsprod1n3 rc_rdma_write_bw

rc_rdma_write_bw:

    bw  =  3.39 GB/sec

 

 

The setup contains two pairs of edge switches (on each datacenter one pair)
and two spine switches (each datacenter one), configured as a non blocking
fat tree.

So:

The servers are connected to the edge switches.

The spine switches are connected to all edge switches.

 

These are the cables we are using:

.         Length 5m Vendor Name: WLGORE Code: QSFP+ Vendor PN: 498385-B24
Vendor Rev: D Vendor SN xxxx

.         Length 200m Vendor Name: MOLEX Code: QSFP+ Vendor PN: 106410-1200
Vendor Rev: A Vendor SN xxxx

 

 

Can someone tell me why this is so? And maybe how I can solve this?

 

Best regards,

 

 

Koen Segers

Enterprise Consultant

 

Computacenter

Services & Solutions

 

Ikaroslaan 31

B-1930 Zaventem

Belgium

 

Tel: +32 2 704 94 67

Fax: +32 2 704 95 95

Mob: +32 497 909353

 <mailto:koen_segers at computacenter.com> koen_segers at computacenter.com

 <http://www.computacenter.com/benelux> www.computacenter.com/benelux

 




Visit us at http://www.computacenter.com/
Computacenter: Transforming IT service delivery.

========================== Disclaimer ==================================
The information in this email is confidential, and is intended solely for
the addressee(s). If you are not the intended recipient of this email please
let us know by reply and then delete it from your system; you should not
copy this message or disclose its contents to anyone. Due to the integrity
risk of sending emails over the Internet, Computacenter will accept no
liability for any comments and / or attachments contained within this email.
========================== Disclaimer ==================================

-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.openfabrics.org/pipermail/ewg/attachments/20111017/2a42bb61/attachment.html>