[Users] Performance issue with FDR & Ubuntu

[German Anders]

Hi Susan/Rupert, first I want to thank you both for the responses.
   Regarding the tests, the thing about ib_write_bw or ib_read_bw is that
it used underneath RDMA, so the adapter maps a connection directly to a
memory location in each host. After the IP session gets established, the
RDMA API bypasses the IP stack in the host and copies the data directly
from, let's say host1 memory to host2 memory. This bypasses the IP stack,
kernel and other networking layers.
   So the RDMA connection is made at the HW level and indicates what the HW
is capable of, in these kind of test I'm getting almost 50Gb/s, so it
rocks! :). However, the iperf tests indicates what the OS and IP layers are
capable of, so that's why iperf will lead to different results. Also,
besides the PCIe gen2 or 3 is used the IP stack took some performance from
that and that's why is not getting the "best" values compared to RDMA.
Notice that I'm also using PCIe v3 so with iperf I can't get more than
24Gb/s really don't know why.

Best,


*German*

2016-01-11 20:16 GMT-03:00 Rupert Dance <rsdance at soft-forge.com>:

> We get 40+ when using FDR cards and PCIe Gen3 and ib_write_bw
>
>
>
> Rupert
>
>
>
> *From:* users-bounces at lists.openfabrics.org [mailto:
> users-bounces at lists.openfabrics.org] *On Behalf Of *Coulter, Susan K
> *Sent:* Monday, January 11, 2016 5:11 PM
> *To:* German Anders
> *Cc:* users at lists.openfabrics.org
> *Subject:* Re: [Users] Performance issue with FDR & Ubuntu
>
>
>
>
>
> On Dec 22, 2015, at 11:55 AM, German Anders <ganders at despegar.com> wrote:
>
>
>
> Is there anyone around that is using IB FDR and running an iperf between
> two hosts on the same net, are getting more than 24Gbits/sec?
>
>
>
> I just ran some quick iperf tests between 2 FDR HCAs - I get the same
> ~24Gbs.
>
>
>
> The constraint is often the PCI bus, not IPoIB itself.
>
>
>
>
>
> ==================================================
>
> Susan Coulter
>
> HPC Network Technical Lead
>
> (505) 667-8425
>
> “Once in a while you get shown the light
>
>     In the strangest of places if you look at it right”  *Robert Hunter*
>
> ==================================================
>
>
>
>
>
>
>
>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.openfabrics.org/pipermail/users/attachments/20160112/19271346/attachment.html>