[Users] Performance issue with FDR & Ubuntu

[Peter Kjellström]

On Tue, 12 Jan 2016 09:10:05 -0300
German Anders <ganders at despegar.com> wrote:

> Hi Susan/Rupert, first I want to thank you both for the responses.
>    Regarding the tests, the thing about ib_write_bw 
...
> in these kind of test I'm getting almost
> 50Gb/s, so it rocks! :). However, the iperf tests indicates what the
> OS and IP layers are capable of, so that's why iperf will lead to
> different results. Also, besides the PCIe gen2 or 3 is used the IP
> stack took some performance from that and that's why is not getting
> the "best" values compared to RDMA. Notice that I'm also using PCIe
> v3 so with iperf I can't get more than 24Gb/s really don't know why.

Most customers running Infiniband use some form of native protocol. It
is not expected that IPoIB will perform at wirespeed. A socket app can
use rsocket, MPI uses verbs, Lustre uses LNET which uses verbs, ...

In short. IPoIB isn't a high performance way to use Infiniband (and I
think your 24 Gpbs iperf single stream sounds quite impressive).

/Peter K