<html>
<body>
<font size=3><br>
The best way to evaluate an interconnect and its applicability to a given
problem is simply to run the application and see where time is
spent. If the majority of the time is not spent in the
interconnect, then focus the performance analysis on where the majority
of time is spent and tune accordingly. Yes, there are many
inefficiencies in the various protocols used - IB, PCIe, coherency,
etc. - we comprehend these since protocol creation is often termed a
house-of-cards design where trade-offs must occur to meet the overall
objectives. When we designed the IB and PCIe protocols, we
were very conscious of these trade-offs and our analysis showed that
people can build solutions that deliver excellent performance even they
choose wisely in the transactions used. To date, the
bandwidth and latency achieved in various benchmarks over these
interconnects shows that people can get this right. Hence, I
recommend you focus your performance analysis on where time is spent in
the applications you care about and apply the 80/20 rule for subsequent
analysis and tuning. The PCIe and IB protocols are largely set and
at best will evolve over time based on demonstrable market needs but some
of the fundamentals will be unlikely to change, e.g. IB's 8b/10b encoding
which is used from 2.5-10GT/s signaling.<br><br>
Mike<br><br>
<br>
At 01:43 AM 10/2/2008, Ramiro Alba Queipo wrote:<br>
<blockquote type=cite class=cite cite="">On Wed, 2008-10-01 at 14:21
-0700, Michael Krause wrote:<br>
> At 09:08 AM 10/1/2008, Ramiro Alba Queipo wrote:<br>
> > Hi all,<br>
> > <br>
> > We have an infiniband cluster of 22 nodes witch 20 Gbps
Mellanox<br>
> > MHGS18-XTC cards and I tried to make performance net tests both
to<br>
> > check<br>
> > hardware as to clarify concepts.<br>
> > <br>
> > Starting from the theoretic pick according to the infiniband
card<br>
> > (in my<br>
> > case 4X DDR => 20 Gbits/s => 2.5 Gbytes/s) we have some
limits:<br>
> > <br>
> > 1) Bus type: PCIe 8x => 250 Mbytes/lane => 250 * 8 = 2
Gbytes/s<br>
> > <br>
> > 2) According to a thread an users openmpi mail-list (???):<br>
> > <br>
> > The 16 Gbit/s number is the theoretical peak, IB is
coded 8/10 so<br>
> > out of the 20 Gbit/s 16 is what you get. On SDR
this number is <br>
> > (of course) 8 Gbit/s achievable (which is ~1000
MB/s) and I've <br>
> > seen well above 900 on MPI (this on 8x PCIe, 2x
margin) <br>
> > <br>
> > Is this true?<br>
> <br>
> IB uses 8b/10 encoding. This results in a 20% overhead on
every<br>
> frame. Further, IB protocol - header, CRC, flow control
credits, etc.<br>
> will consume additional bandwidth - the amount will vary with
workload<br>
> and traffic patters. Also, any fabric can experience
congestion which<br>
> may reduce throughput for any given data flow. <br>
> <br>
> PCIe uses 8b/10b encoding for both 2.5GT/s and 5.0 GT/s signaling
(the<br>
> next generation signaling is scrambled based so provides 2x the
data<br>
> bandwidth with significantly less encoding overhead). It also
has<br>
> protocol overheads conceptually similar to IB which will
consume<br>
> additional bandwidth (keep in mind many volume chipsets only support
a<br>
> 256B transaction size so a single IB frame may require 8-16
PCIe<br>
> transactions to process. There will also be application
/ device<br>
> driver control messages between the host and the I/O device which
will<br>
> consume additional bandwidth. <br>
> <br>
> Also keep in mind that the actual application bandwidth may be
further<br>
> gated by the memory subsystem, the I/O-to-memory latency, etc.
so<br>
> while the theoretical bandwidths may be quite high, they will
be<br>
> constrained by the interactions and the limitations within the
overall<br>
> hardware and software stacks. <br>
> <br>
> <br>
> > 3) According to other comment in the same thread:<br>
> > <br>
> > The data throughput limit for 8x PCIe is ~12 Gb/s.
The theoretical<br>
> > limit is 16 Gb/s, but each PCIe packet has a
whopping 20 byte<br>
> > overhead. If the adapter uses 64 byte packets, then
you see 1/3 of<br>
> > the throughput go to overhead.<br>
> > <br>
> > Could someone explain me that?<br>
> <br>
> DMA Read completions are often returned one cache line at a time
while<br>
> DMA Writes are often transmitted at the Max_Payload_Size of 256B
(some<br>
> chipsets do coalesce completions allowing up to the
Max_Payload_Size<br>
> to be returned). Depending upon the mix of transactions
required to<br>
> move an IB frame, the overheads may seem excessive.<br>
> <br>
> PCIe overheads vary with the transaction type, the flow control
credit<br>
> exchanges, CRC, etc. It is important to keep these in
mind when<br>
> evaluating the solution. <br>
> <br>
> > Then I got another comment about the matter:<br>
> > <br>
> > The best uni-directional performance I have heard of for PCIe
8x IB<br>
> > DDR is ~1,400 MB/s (11.2 Gb/s) with Lustre, which is about 55%
of<br>
> > the<br>
> > theoretical 20 Gb/s advertised speed.<br>
> > <br>
> > <br>
> >
---------------------------------------------------------------------<br>
> > <br>
> > <br>
> > Now, I did some tests (mpi used is OpenMPI) with the
following<br>
> > results:<br>
> > <br>
> > a) Using "Performance tests" from OFED 1.31<br>
> > <br>
> > ib_write_bw -a server -> 1347
MB/s<br>
> > <br>
> > b) Using hpcc (2 cores at diferent nodes) -> 1157 MB/s
(--mca<br>
> > mpi_leave_pinned 1)<br>
> > <br>
> > c) Using "OSU Micro-Benchmarks" in
"MPItests" from OFED 1.3.1<br>
> > <br>
> > 1) 2 cores from different nodes<br>
> > <br>
> > - mpirun -np 2 --hostfile pool osu_bibw
-> 2001.29 MB/s<br>
> > (bidirectional)<br>
> > - mpirun -np 2 --hostfile pool osu_bw
-> 1311.31 MB/s<br>
> > <br>
> > 2) 2 cores from the same node<br>
> > <br>
> > - mpirun -np 2 osu_bibw ->
2232 MB/s (bidirectional)<br>
> > - mpirun -np 2 osu_bw -> 2058
MB/s<br>
> > <br>
> > The questions are:<br>
> > <br>
> > - Are those results coherent with what it should be?<br>
> > - Why tests with the two core in the same node are better?<br>
> > - Should not the bidirectional test be a bit higher?<br>
> > - Why hpcc is so low? <br>
> <br>
> You would need to provide more information about the system
hardware,<br>
> the fabrics, etc. to make any rational response. There are
many<br><br>
Whe have DELL PowerEdge SC1435 nodes with two AMD 2350 processors
(2.0<br>
GHz processor frequency and 1.8 GHz of Integrated Memory Controller<br>
Speed). The fabrics is built from 20 Gbps Mellanox MHGS18-XTC cards
and<br>
a Flextrics 24 port 4X DDR switch, with 3 meter cables from Mellanox<br>
(MCC4L30-003 4X microGiGaCN latch, 30 AWG).<br><br>
> variables here and as I noted above, one cannot just derate
the<br>
> hardware by a fixed percentage and conclude there is a real problem
in<br>
> the solution stack. He is more complex. The
question you should<br>
> ask is whether the micro-benchmarks you are executing are a
realistic<br>
> reflection of the real workload. If not, then do any of these
numbers<br><br>
No I don't think they are. My main intention is to understand what I<br>
really have and why, and to check for link degradations. Keep in
mind<br>
that this is my first contact with infiniband problematics and
before<br>
the end of this year we will have 76 nodes (608 cores) with an<br>
infiniband net that will be used both for calculations and data,
using<br>
NFS-RDMA.<br><br>
Apart from our own test, what tests you would use to check for a
ready<br>
cluster?<br><br>
> matter at the end of the day especially if the total time
spent<br>
> within the interconnect stacks are relatively small or bursty.<br>
> <br>
> Mike <br><br>
<br><br>
-- <br>
Aquest missatge ha estat analitzat per MailScanner<br>
a la cerca de virus i d'altres continguts perillosos,<br>
i es considera que està net.<br>
For all your IT requirements visit:
<a href="http://www.transtec.co.uk/" eudora="autourl">
http://www.transtec.co.uk</a> </font></blockquote></body>
</html>