[ofa-general] Expected RDMA performance

[Michael Krause]

At 02:33 PM 10/20/2007, Peter Kjellstrom wrote:
>On Friday 19 October 2007, Michael Krause wrote:
> > At 08:20 AM 10/19/2007, Peter Kjellstrom wrote:
> > >On Thursday 18 October 2007, Chuck Hartley wrote:
>...
> > > > What is the maximum theoretical BW for
> > > > DDR IB - 1525MB/sec?
> > >
> > >No, it's 20 Gbps on the wire and 8/10 encoded so 16 Gbps effective which
> > > is 2000 MB/s (10-base) and 1907 MiB/s (2-base).
> >
> > There is also IB protocol overhead combined with driver / device control
> > traffic overhead (consumes device as well as PCI resources / bandwidth),
> > end-to-end control traffic  which is also a function of how the application
> > is constructed.   In general, hitting about 80-85% of the theoretical
> > maximum is possible.
>
>IB can do much better than that. On an SDR system I typically get 950 MB/s
>(10-base), 95%. This on 8x pci-express so the limitations of pci-e above does
>not bite. If IB DDR could strech it's legs (if we had faster pci-e, say
>pci-e-2.0...) then maybe we would see 95% there too :-).

While there are certainly marketing workloads that can hit such high 
efficiencies, the number of real world workloads is rather small.   There 
was one interconnect provider a few years back who used to demonstrate 95+% 
efficiency post 8b/10b encoding overhead by sending 1MB messages so the 
host interaction was to pull a single work request and then just issue DMA 
Read Requests.  Just like that they were at link rate.  However, most 
workloads are not single streams but a mix of streams with varying work 
request rates, sizes, etc.  I don't doubt that one can hit higher rates 
than 80-85% but expect most workloads to rarely exceed this value.  A 
couple years ago a reporter asked me about why some interconnects are at 
50-60% efficiency when measured in real environments.  We walked through 
the host / device as well as driver interactions, the ability of the 
platform to actually generate useful I/O work (some are processor / memory 
limited so improvements in the I/O subsystems has no real ROI), the 
protocol overheads, etc.  He was trying ascertain whether there was a story 
here about vendors basically hyping their technology using the various 
marketing numbers when in reality they could not actually deliver the 
performance under more than a contrived or limited set of workloads.   I 
convinced him there was no story here but it did illustrate my earlier 
industry talks about marketing hype vs. reality and how marketing does a 
great deal of harm due to lost credibility when it comes to running real 
world applications.

Mike 
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.openfabrics.org/pipermail/general/attachments/20071022/edfe8ddf/attachment.html>