[ofa-general] DDR vs SDR performance
Gilad Shainer
Shainer at Mellanox.com
Wed Nov 28 07:17:45 PST 2007
Here are some notes. You can contact me directly for more info.
1. You do not compare the same HW. The single port IB HCAs provides
difference performance than the dual port devices. If you want to see
the difference between SDR and DDR, you need to use the same IB
configuration as well.
2. Saying that, with the single port DDR you should get around 1400MB/s
with the RDMA tests but:
- The benchmark you are using is not supported any more (well, for long
time now). You should use the IB send, IB write etc tests
- On Opteron, the HTxx00 chipset configuration is very important (not
just for IB performance)
- There is a difference of performance depends on the location of the
memory. If you will run the tests you will see numbers in the high 1300
and low 1100 (with your current chipset config)
Gilad.
-----Original Message-----
From: general-bounces at lists.openfabrics.org
[mailto:general-bounces at lists.openfabrics.org] On Behalf Of Stijn De
Smet
Sent: Wednesday, November 28, 2007 7:02 AM
To: Gilad Shainer
Cc: general at lists.openfabrics.org
Subject: Re: [ofa-general] DDR vs SDR performance
One ServerWorks HT2100 A PCI Express Bridge, one HT2100 B PCI Express
Bridge, and one ServerWorks HT1000 South Bridge
Regards,
Stijn
Gilad Shainer wrote:
> Is the chipset in your servers HT2000?
>
> Gilad.
>
> -----Original Message-----
> From: general-bounces at lists.openfabrics.org
> [mailto:general-bounces at lists.openfabrics.org] On Behalf Of Stijn De
> Smet
> Sent: Wednesday, November 28, 2007 6:43 AM
> To: general at lists.openfabrics.org
> Subject: [ofa-general] DDR vs SDR performance
>
> Hello,
>
> I have a problem with the DDR performance:
>
> Configuration:
> 2 servers (IBM x3755, equiped with 4 dualcore opteron and 16GB RAM)
> 3 HCA's installed (2 Cisco DDR(Cheetah) and 1 Cisco dual
> SDR(LionMini), all PCI-e x8), all DDR HCA's at newest Cisco Firmware
> v1.2.917 build 3.2.0.149, with label 'HCA.Cheetah-DDR.20'
>
> The DDR's are connected with a cable, and s3n1 is running a SM. The
> SDR boards are connected over a Cisco SFS-7000D, but the DDR
> performance is
> +- the same over this SFS-7000D
>
> Both servers are running SLES10-SP1 with Ofed 1.2.5.
>
>
> s3n1:~ # ibstatus
> Infiniband device 'mthca0' port 1 status: < -- DDR board #1, not
> connected
> default gid: fe80:0000:0000:0000:0005:ad00:000b:cb39
> base lid: 0x0
> sm lid: 0x0
> state: 1: DOWN
> phys state: 2: Polling
> rate: 10 Gb/sec (4X)
>
> Infiniband device 'mthca1' port 1 status: <--- DDR board #2,
> connected with cable
> default gid: fe80:0000:0000:0000:0005:ad00:000b:cb31
> base lid: 0x16
> sm lid: 0x16
> state: 4: ACTIVE
> phys state: 5: LinkUp
> rate: 20 Gb/sec (4X DDR)
>
> Infiniband device 'mthca2' port 1 status: <--- SDR board, only port 1
> connected to the SFS-7000D
> default gid: fe80:0000:0000:0000:0005:ad00:0008:a8d9
> base lid: 0x3
> sm lid: 0x2
> state: 4: ACTIVE
> phys state: 5: LinkUp
> rate: 10 Gb/sec (4X)
>
> Infiniband device 'mthca2' port 2 status:
> default gid: fe80:0000:0000:0000:0005:ad00:0008:a8da
> base lid: 0x0
> sm lid: 0x0
> state: 1: DOWN
> phys state: 2: Polling
> rate: 10 Gb/sec (4X)
>
>
> RDMA test of :
> -- SDR:
> s3n2:~ # ib_rdma_bw -d mthca2 gpfs3n1
> 7190: | port=18515 | ib_port=1 | size=65536 | tx_depth=100 |
> iters=1000
> | duplex=0 | cma=0 |
> 7190: Local address: LID 0x05, QPN 0x0408, PSN 0xf10f03 RKey 0x003b00
> VAddr 0x002ba7b9943000
> 7190: Remote address: LID 0x03, QPN 0x040a, PSN 0xa9cf5c, RKey
> 0x003e00 VAddr 0x002adb2f3bb000
>
>
> 7190: Bandwidth peak (#0 to #989): 937.129 MB/sec
> 7190: Bandwidth average: 937.095 MB/sec
> 7190: Service Demand peak (#0 to #989): 2709 cycles/KB
> 7190: Service Demand Avg : 2709 cycles/KB
>
> -- DDR
> s3n2:~ # ib_rdma_bw -d mthca1 gpfs3n1
> 7191: | port=18515 | ib_port=1 | size=65536 | tx_depth=100 |
> iters=1000
> | duplex=0 | cma=0 |
> 7191: Local address: LID 0x10, QPN 0x0405, PSN 0x5e19e RKey 0x002600
> VAddr 0x002b76eab20000
> 7191: Remote address: LID 0x16, QPN 0x0405, PSN 0xdd976e, RKey
> 0x80002900 VAddr 0x002ba8ed10e000
>
>
> 7191: Bandwidth peak (#0 to #990): 1139.32 MB/sec
> 7191: Bandwidth average: 1139.31 MB/sec
> 7191: Service Demand peak (#0 to #990): 2228 cycles/KB
> 7191: Service Demand Avg : 2228 cycles/KB
>
> So only 200MB/s increase between SDR and DDR With comparable
> hardware(x3655, dual dualcore opteron, 8GB RAM), I get a little bit
> better RDMA performance(1395MB/s so close to the PCI-e x8 limit), but
> even worse IPoIB and SDP performance with kernels 2.6.22 and
> 2.6.23.9 and Ofed 1.3b
>
>
>
> IPoIB test(iperf), IPoIB in connected mode, MTU 65520:
> #ib2 is SDR, ib1 is DDR
> #SDR:
> s3n2:~ # iperf -c cic-s3n1
> ------------------------------------------------------------
> Client connecting to cic-s3n1, TCP port 5001 TCP window size: 1.00
> MByte
> (default)
> ------------------------------------------------------------
> [ 3] local 192.168.1.2 port 50598 connected with 192.168.1.1 port
> 5001 [ 3] 0.0-10.0 sec 6.28 GBytes 5.40 Gbits/sec
>
> #DDR:
> s3n2:~ # iperf -c cic-s3n1
> ------------------------------------------------------------
> Client connecting to cic-s3n1, TCP port 5001 TCP window size: 1.00
> MByte
> (default)
> ------------------------------------------------------------
> [ 3] local 192.168.1.2 port 32935 connected with 192.168.1.1 port
> 5001 [ 3] 0.0-10.0 sec 6.91 GBytes 5.93 Gbits/sec
>
>
> Now the increase is only 0.5Gbit
>
> And finally a test with SDP:
>
> DDR:
> s3n2:~ # LD_PRELOAD=libsdp.so SIMPLE_LIBSDP="ok" iperf -c cic-s3n1
> ------------------------------------------------------------
> Client connecting to cic-s3n1, TCP port 5001 TCP window size: 3.91
> MByte
> (default)
> ------------------------------------------------------------
> [ 4] local 192.168.1.2 port 58186 connected with 192.168.1.1 port
> 5001 [ 4] 0.0-10.0 sec 7.72 GBytes 6.63 Gbits/sec
>
> #SDR:
> s3n2:~ # LD_PRELOAD=libsdp.so SIMPLE_LIBSDP="ok" iperf -c cic-s3n1
> ------------------------------------------------------------
> Client connecting to cic-s3n1, TCP port 5001 TCP window size: 3.91
> MByte
> (default)
> ------------------------------------------------------------
> [ 4] local 192.168.1.2 port 58187 connected with 192.168.1.1 port
> 5001 [ 4] 0.0-10.0 sec 7.70 GBytes 6.61 Gbits/sec
>
> With SDP there is even no difference anymore between the 2 boards.
>
>
> Even when using multiple connections(using 3 servers(s3s2,s3s3,s3s4),
> x3655, 2.6.22, connecting all to one(s3s1) over DDR):
> s3s2:~ # iperf -c cic-s3s1 -p 5001 -t 30
> ------------------------------------------------------------
> Client connecting to cic-s3s1, TCP port 5001 TCP window size: 1.00
> MByte
> (default)
> ------------------------------------------------------------
> [ 3] local 192.168.1.15 port 33576 connected with 192.168.1.14 port
> 5001 [ 3] 0.0-30.0 sec 5.94 GBytes 1.70 Gbits/sec s3s3:~ # iperf
> -c
> cic-s3s1 -p 5002 -t 30
> ------------------------------------------------------------
> Client connecting to cic-s3s1, TCP port 5002 TCP window size: 1.00
> MByte
> (default)
> ------------------------------------------------------------
> [ 3] local 192.168.1.16 port 53558 connected with 192.168.1.14 port
> 5002 [ 3] 0.0-30.0 sec 5.74 GBytes 1.64 Gbits/sec s3s4:~ # iperf
> -c
> cic-s3s1 -p 5003 -t 30
> ------------------------------------------------------------
> Client connecting to cic-s3s1, TCP port 5003 TCP window size: 1.00
> MByte
> (default)
> ------------------------------------------------------------
> [ 3] local 192.168.1.17 port 37169 connected with 192.168.1.14 port
> 5003 [ 3] 0.0-30.0 sec 5.79 GBytes 1.66 Gbits/sec
>
>
> This gives a total of 1.7+1.64+1.66Gbits/sec=5Gbits/sec
>
> Is this normal behavior(SDP and IPoIB not benefiting from DDR)?
>
>
> Regards,
> Stijn
> _______________________________________________
> general mailing list
> general at lists.openfabrics.org
> http://lists.openfabrics.org/cgi-bin/mailman/listinfo/general
>
> To unsubscribe, please visit
> http://openib.org/mailman/listinfo/openib-general
>
_______________________________________________
general mailing list
general at lists.openfabrics.org
http://lists.openfabrics.org/cgi-bin/mailman/listinfo/general
To unsubscribe, please visit
http://openib.org/mailman/listinfo/openib-general
More information about the general
mailing list