<html>
<body>
<font size=3>At 02:06 AM 10/22/2007, Koen Segers wrote:<br>
<blockquote type=cite class=cite cite="">On Fri, 2007-10-19 at 09:09
-0700, Michael Krause wrote:<br>
> At 08:20 AM 10/19/2007, Peter Kjellstrom wrote:<br>
> > On Thursday 18 October 2007, Chuck Hartley wrote:<br>
> > ...<br>
> > > 8388608
5000
1342.12
1342.12<br>
> > >
------------------------------------------------------------------<br>
> > ><br>
> > > Is this typical RDMA performance?<br>
> > <br>
> > It's close to what I've seen on similar hw. ~1400 is what you
can<br>
> > push through <br>
> > the 8x pci-e of the intel 5000 chipset (confirmed by trying 4x
pci-e<br>
> > which <br>
> > has shown ~700).<br>
> > <br>
> > > What is the maximum theoretical BW for <br>
> > > DDR IB - 1525MB/sec?<br>
> > <br>
> > No, it's 20 Gbps on the wire and 8/10 encoded so 16 Gbps
effective<br>
> > which is <br>
> > 2000 MB/s (10-base) and 1907 MiB/s (2-base).<br>
> <br>
> There is also IB protocol overhead combined with driver /
device<br>
> control traffic overhead (consumes device as well as PCI resources
/<br>
> bandwidth), end-to-end control traffic which is also a
function of<br>
> how the application is constructed. In general, hitting
about 80-85%<br>
> of the theoretical maximum is possible.<br><br>
<br>
I'm very interested in this result. Can you elaborate this a bit
more?</font></blockquote><br>
In what regard?<br><br>
<blockquote type=cite class=cite cite=""><font size=3>Has anyone
documented the ib traffic control mechanism?</font></blockquote><br>
Driver-to-device interactions consume resources and contend for local I/O
bandwidth / local device processing<br><br>
Application-to-device interactions have similar impacts<br><br>
ULP exchanges such as SEND operations to communicate protection keys,
addresses, etc. <br><br>
Host OS / application execution to generate work as well as schedule /
process work and deal with any interrupts / polling mechanisms.
This can lead from zero to significant delays resulting in burst style
traffic patterns. Also many workloads may be small
transaction dominate so their efficiency will be significantly lower than
one that is large transaction dominant.<br><br>
And so forth.<br><br>
There are many variables and mileage will vary as a result. Some
will do quite well while others will not. This is why a good
range of benchmarks is required to evaluate whether a given solution is
reasonable for the targeted workloads or problem space.
Anyone can contrive something to do outstanding at one thing while
completely biting it in other areas. <br><br>
Mike<br><br>
<br><br>
<br>
<blockquote type=cite class=cite cite=""><font size=3>Regards,<br><br>
Koen Segers<br>
> <br>
> > On our system (with a different HCA) we see quite a difference
with <br>
> > snoop-filter off (bios option). With snoop off (our)
application<br>
> > performance <br>
> > goes up (not very suprising) but IB performance goes down
(latency<br>
> > 0.4us <br>
> > worse and bw ~1400->1200).<br>
> <br>
> Mike<br>
> _______________________________________________<br>
> general mailing list<br>
> general@lists.openfabrics.org<br>
>
<a href="http://lists.openfabrics.org/cgi-bin/mailman/listinfo/general" eudora="autourl">
http://lists.openfabrics.org/cgi-bin/mailman/listinfo/general</a><br>
> <br>
> To unsubscribe, please visit
<a href="http://openib.org/mailman/listinfo/openib-general" eudora="autourl">
http://openib.org/mailman/listinfo/openib-general</a><br>
*** Disclaimer ***<br><br>
Vlaamse Radio- en Televisieomroep<br>
Auguste Reyerslaan 52, 1043 Brussel<br><br>
nv van publiek recht<br>
BTW BE 0244.142.664<br>
RPR Brussel<br>
<a href="http://www.vrt.be/disclaimer" eudora="autourl">
http://www.vrt.be/disclaimer</a><br>
</font></blockquote></body>
</html>