[ofa-general] Expected RDMA performance

[Michael Krause]

At 08:20 AM 10/19/2007, Peter Kjellstrom wrote:
>On Thursday 18 October 2007, Chuck Hartley wrote:
>...
> > 8388608        5000            1342.12               1342.12
> > ------------------------------------------------------------------
> >
> > Is this typical RDMA performance?
>
>It's close to what I've seen on similar hw. ~1400 is what you can push 
>through
>the 8x pci-e of the intel 5000 chipset (confirmed by trying 4x pci-e which
>has shown ~700).
>
> > What is the maximum theoretical BW for
> > DDR IB - 1525MB/sec?
>
>No, it's 20 Gbps on the wire and 8/10 encoded so 16 Gbps effective which is
>2000 MB/s (10-base) and 1907 MiB/s (2-base).

There is also IB protocol overhead combined with driver / device control 
traffic overhead (consumes device as well as PCI resources / bandwidth), 
end-to-end control traffic  which is also a function of how the application 
is constructed.   In general, hitting about 80-85% of the theoretical 
maximum is possible.

>On our system (with a different HCA) we see quite a difference with
>snoop-filter off (bios option). With snoop off (our) application performance
>goes up (not very suprising) but IB performance goes down (latency 0.4us
>worse and bw ~1400->1200).

Mike 
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.openfabrics.org/pipermail/general/attachments/20071019/a28ee691/attachment.html>