[libfabric-users] Optimisation Tips for verbs provider

[Ilango, Arun]

Hi Valentino,

Libfabric has a set of tests available at https://github.com/ofiwg/fabtests. Can you run the fi_msg_bw test with the same size and iterations on your setup and check if you notice any variance? Also what version/commit number of libfabric are you using?

Thanks,
Arun.

From: Libfabric-users [mailto:libfabric-users-bounces at lists.openfabrics.org] On Behalf Of Valentino Picotti
Sent: Wednesday, September 07, 2016 7:48 AM
To: libfabric-users at lists.openfabrics.org
Subject: [libfabric-users] Optimisation Tips for verbs provider

Hi all,

I apologies in advance for the long email.

In the past month I've integrated libfabric in a project based on infiniband verbs with the aim to be provider independent. This project has a transport layer that makes the application independent from the transport implementation (that is chosen at compile time).
I worked only on the libfabric implementation of the transport layer and this was my first experience with RDMA APIs and hardware. What I did was to map the various ibv_* and rdma_* calls to fi_* calls and I got a working layer quite easily (after studying the libfabric terminology).
Now I'm trying to achieve the same performance of raw verbs.
I'm testing the transport layer with a one sided communication where a client sends the data to a server with the message API(fi_send/fi_recv). The client and the server run on two different nodes connected with one IB EDR link: i don't set processor affinity nor change power management policy. The depth of completion queues and the size of sent buffers are the same across the tests.
Running on the verbs transport layer I get a stable bandwidth of 22 Gbps, instead with libfabric over verbs I get a very floating bandwidth: from 0.4 Gbps to 19 Gbps in the same test[1]. The bandwidth is calculated as the number of buffers sent every 5 seconds.

This is how i setup the verbs provider:

  m_hints->caps = FI_MSG;
  m_hints->mode = FI_LOCAL_MR;
  m_hints->ep_attr->type = FI_EP_MSG;
  m_hints->domain_attr->threading = FI_THREAD_COMPLETION;
  m_hints->domain_attr->data_progress = FI_PROGRESS_MANUAL;
  m_hints->domain_attr->resource_mgmt = FI_RM_DISABLED;
  m_hints->fabric_attr->prov_name = strdup("verbs");

Furthermore I bind two completion queues to the endpoints: one with FI_SEND flag and the other with FI_RECV.

I can't figure out why I'm getting that high variance with libfabric.
Do you have any idea? I'm missing same optimisations tips for the verbs provider?

Thanks in advance,

Valentino


[1] Test run with depth queue of 1 and buffer size of 512KB

Example of a test output with libfarbic:

2016-09-07 - 15:10:56 t_server: INFO: Accepted connection

2016-09-07 - 15:10:56 t_server: INFO: Start receiving...

2016-09-07 - 15:11:01 t_server: INFO: Bandwith: 8.3324 Gb/s

2016-09-07 - 15:11:06 t_server: INFO: Bandwith: 15.831 Gb/s

2016-09-07 - 15:11:11 t_server: INFO: Bandwith: 19.1713 Gb/s

2016-09-07 - 15:11:16 t_server: INFO: Bandwith: 10.8825 Gb/s

2016-09-07 - 15:11:21 t_server: INFO: Bandwith: 8.07991 Gb/s

2016-09-07 - 15:11:26 t_server: INFO: Bandwith: 15.4015 Gb/s

2016-09-07 - 15:11:31 t_server: INFO: Bandwith: 20.4263 Gb/s

2016-09-07 - 15:11:36 t_server: INFO: Bandwith: 19.7023 Gb/s

2016-09-07 - 15:11:41 t_server: INFO: Bandwith: 10.474 Gb/s

2016-09-07 - 15:11:46 t_server: INFO: Bandwith: 17.4072 Gb/s

2016-09-07 - 15:11:51 t_server: INFO: Bandwith: 0.440402 Gb/s

2016-09-07 - 15:11:56 t_server: INFO: Bandwith: 2.73217 Gb/s

2016-09-07 - 15:12:01 t_server: INFO: Bandwith: 0.984822 Gb/s

2016-09-07 - 15:12:06 t_server: INFO: Bandwith: 2.93013 Gb/s

2016-09-07 - 15:12:11 t_server: INFO: Bandwith: 0.847248 Gb/s

2016-09-07 - 15:12:16 t_server: INFO: Bandwith: 7.72255 Gb/s

2016-09-07 - 15:12:21 t_server: INFO: Bandwith: 14.7849 Gb/s

2016-09-07 - 15:12:26 t_server: INFO: Bandwith: 12.9243 Gb/s

2016-09-07 - 15:12:31 t_server: INFO: Bandwith: 0.687027 Gb/s

2016-09-07 - 15:12:36 t_server: INFO: Bandwith: 1.44787 Gb/s

2016-09-07 - 15:12:41 t_server: INFO: Bandwith: 2.681 Gb/s

Example of a test output with raw verbs:

2016-09-07 - 16:36:00 t_server: INFO: Accepted connection

2016-09-07 - 16:36:00 t_server: INFO: Start receiving...

2016-09-07 - 16:36:05 t_server: INFO: Bandwith: 17.9491 Gb/s

2016-09-07 - 16:36:10 t_server: INFO: Bandwith: 23.4671 Gb/s

2016-09-07 - 16:36:15 t_server: INFO: Bandwith: 23.0368 Gb/s

2016-09-07 - 16:36:20 t_server: INFO: Bandwith: 22.9638 Gb/s

2016-09-07 - 16:36:25 t_server: INFO: Bandwith: 22.8203 Gb/s

2016-09-07 - 16:36:30 t_server: INFO: Bandwith: 20.058 Gb/s

2016-09-07 - 16:36:35 t_server: INFO: Bandwith: 22.5033 Gb/s

2016-09-07 - 16:36:40 t_server: INFO: Bandwith: 20.1754 Gb/s

2016-09-07 - 16:36:45 t_server: INFO: Bandwith: 22.5578 Gb/s

2016-09-07 - 16:36:50 t_server: INFO: Bandwith: 20.0588 Gb/s

2016-09-07 - 16:36:55 t_server: INFO: Bandwith: 22.2718 Gb/s

2016-09-07 - 16:37:00 t_server: INFO: Bandwith: 22.494 Gb/s

2016-09-07 - 16:37:05 t_server: INFO: Bandwith: 23.1836 Gb/s

2016-09-07 - 16:37:10 t_server: INFO: Bandwith: 23.0972 Gb/s

2016-09-07 - 16:37:15 t_server: INFO: Bandwith: 21.5033 Gb/s

2016-09-07 - 16:37:20 t_server: INFO: Bandwith: 18.5506 Gb/s

2016-09-07 - 16:37:25 t_server: INFO: Bandwith: 20.3709 Gb/s

2016-09-07 - 16:37:30 t_server: INFO: Bandwith: 21.3457 Gb/s

2016-09-07 - 16:37:35 t_server: INFO: Bandwith: 20.5059 Gb/s

2016-09-07 - 16:37:40 t_server: INFO: Bandwith: 22.4899 Gb/s

2016-09-07 - 16:37:45 t_server: INFO: Bandwith: 22.1266 Gb/s

2016-09-07 - 16:37:50 t_server: INFO: Bandwith: 22.4504 Gb/s


-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.openfabrics.org/pipermail/libfabric-users/attachments/20160907/7efa37ee/attachment.html>