<div dir="ltr">Hi all,<div><br></div><div>I apologies in advance for the long email.</div><div><br></div><div>In the past month I've integrated libfabric in a project based on infiniband verbs with the aim to be provider independent. This project has a transport layer that makes the application independent from the transport implementation (that is chosen at compile time).</div><div>I worked only on the libfabric implementation of the transport layer and this was my first experience with RDMA APIs and hardware. What I did was to map the various ibv_* and rdma_* calls to fi_* calls and I got a working layer quite easily (after studying the libfabric terminology).</div><div>Now I'm trying to achieve the same performance of raw verbs.</div><div>I'm testing the transport layer with a one sided communication where a client sends the data to a server with the message API(fi_send/fi_recv). The client and the server run on two different nodes connected with one IB EDR link: i don't set processor affinity nor change power management policy. The depth of completion queues and the size of sent buffers are the same across the tests.</div><div>Running on the verbs transport layer I get a stable bandwidth of 22 Gbps, instead with libfabric over verbs I get a very floating bandwidth: from 0.4 Gbps to 19 Gbps in the same test[1]. The bandwidth is calculated as the number of buffers sent every 5 seconds.</div><div><br></div><div>This is how i setup the verbs provider:</div><br>  m_hints->caps = FI_MSG;<br>  m_hints->mode = FI_LOCAL_MR;<br>  m_hints->ep_attr->type = FI_EP_MSG;<br>  m_hints->domain_attr->threading = FI_THREAD_COMPLETION;<br>  m_hints->domain_attr->data_progress = FI_PROGRESS_MANUAL;<br>  m_hints->domain_attr->resource_mgmt = FI_RM_DISABLED;<br>  m_hints->fabric_attr->prov_name = strdup("verbs");<div><br></div><div>Furthermore I bind two completion queues to the endpoints: one with FI_SEND flag and the other with FI_RECV.</div><div><br></div><div>I can't figure out why I'm getting that high variance with libfabric.</div><div>Do you have any idea? I'm missing same optimisations tips for the verbs provider?</div><div><br></div><div>Thanks in advance,</div><div><br></div><div>Valentino</div><div><br></div><div><br></div><div>[1] Test run with depth queue of 1 and buffer size of 512KB </div><div><br></div><div>Example of a test output with libfarbic:</div><div><p style="margin:0px;font-size:12px;line-height:normal;font-family:'andale mono';color:rgb(41,249,20);background-color:rgb(0,0,0)"><span style="font-variant-ligatures:no-common-ligatures">2016-09-07 - 15:10:56 t_server: INFO: Accepted connection</span></p>
<p style="margin:0px;font-size:12px;line-height:normal;font-family:'andale mono';color:rgb(41,249,20);background-color:rgb(0,0,0)"><span style="font-variant-ligatures:no-common-ligatures">2016-09-07 - 15:10:56 t_server: INFO: Start receiving...</span></p>
<p style="margin:0px;font-size:12px;line-height:normal;font-family:'andale mono';color:rgb(41,249,20);background-color:rgb(0,0,0)"><span style="font-variant-ligatures:no-common-ligatures">2016-09-07 - 15:11:01 t_server: INFO: Bandwith: 8.3324 Gb/s</span></p>
<p style="margin:0px;font-size:12px;line-height:normal;font-family:'andale mono';color:rgb(41,249,20);background-color:rgb(0,0,0)"><span style="font-variant-ligatures:no-common-ligatures">2016-09-07 - 15:11:06 t_server: INFO: Bandwith: 15.831 Gb/s</span></p>
<p style="margin:0px;font-size:12px;line-height:normal;font-family:'andale mono';color:rgb(41,249,20);background-color:rgb(0,0,0)"><span style="font-variant-ligatures:no-common-ligatures">2016-09-07 - 15:11:11 t_server: INFO: Bandwith: 19.1713 Gb/s</span></p>
<p style="margin:0px;font-size:12px;line-height:normal;font-family:'andale mono';color:rgb(41,249,20);background-color:rgb(0,0,0)"><span style="font-variant-ligatures:no-common-ligatures">2016-09-07 - 15:11:16 t_server: INFO: Bandwith: 10.8825 Gb/s</span></p>
<p style="margin:0px;font-size:12px;line-height:normal;font-family:'andale mono';color:rgb(41,249,20);background-color:rgb(0,0,0)"><span style="font-variant-ligatures:no-common-ligatures">2016-09-07 - 15:11:21 t_server: INFO: Bandwith: 8.07991 Gb/s</span></p>
<p style="margin:0px;font-size:12px;line-height:normal;font-family:'andale mono';color:rgb(41,249,20);background-color:rgb(0,0,0)"><span style="font-variant-ligatures:no-common-ligatures">2016-09-07 - 15:11:26 t_server: INFO: Bandwith: 15.4015 Gb/s</span></p>
<p style="margin:0px;font-size:12px;line-height:normal;font-family:'andale mono';color:rgb(41,249,20);background-color:rgb(0,0,0)"><span style="font-variant-ligatures:no-common-ligatures">2016-09-07 - 15:11:31 t_server: INFO: Bandwith: 20.4263 Gb/s</span></p>
<p style="margin:0px;font-size:12px;line-height:normal;font-family:'andale mono';color:rgb(41,249,20);background-color:rgb(0,0,0)"><span style="font-variant-ligatures:no-common-ligatures">2016-09-07 - 15:11:36 t_server: INFO: Bandwith: 19.7023 Gb/s</span></p>
<p style="margin:0px;font-size:12px;line-height:normal;font-family:'andale mono';color:rgb(41,249,20);background-color:rgb(0,0,0)"><span style="font-variant-ligatures:no-common-ligatures">2016-09-07 - 15:11:41 t_server: INFO: Bandwith: 10.474 Gb/s</span></p>
<p style="margin:0px;font-size:12px;line-height:normal;font-family:'andale mono';color:rgb(41,249,20);background-color:rgb(0,0,0)"><span style="font-variant-ligatures:no-common-ligatures">2016-09-07 - 15:11:46 t_server: INFO: Bandwith: 17.4072 Gb/s</span></p>
<p style="margin:0px;font-size:12px;line-height:normal;font-family:'andale mono';color:rgb(41,249,20);background-color:rgb(0,0,0)"><span style="font-variant-ligatures:no-common-ligatures">2016-09-07 - 15:11:51 t_server: INFO: Bandwith: 0.440402 Gb/s</span></p>
<p style="margin:0px;font-size:12px;line-height:normal;font-family:'andale mono';color:rgb(41,249,20);background-color:rgb(0,0,0)"><span style="font-variant-ligatures:no-common-ligatures">2016-09-07 - 15:11:56 t_server: INFO: Bandwith: 2.73217 Gb/s</span></p>
<p style="margin:0px;font-size:12px;line-height:normal;font-family:'andale mono';color:rgb(41,249,20);background-color:rgb(0,0,0)"><span style="font-variant-ligatures:no-common-ligatures">2016-09-07 - 15:12:01 t_server: INFO: Bandwith: 0.984822 Gb/s</span></p>
<p style="margin:0px;font-size:12px;line-height:normal;font-family:'andale mono';color:rgb(41,249,20);background-color:rgb(0,0,0)"><span style="font-variant-ligatures:no-common-ligatures">2016-09-07 - 15:12:06 t_server: INFO: Bandwith: 2.93013 Gb/s</span></p>
<p style="margin:0px;font-size:12px;line-height:normal;font-family:'andale mono';color:rgb(41,249,20);background-color:rgb(0,0,0)"><span style="font-variant-ligatures:no-common-ligatures">2016-09-07 - 15:12:11 t_server: INFO: Bandwith: 0.847248 Gb/s</span></p>
<p style="margin:0px;font-size:12px;line-height:normal;font-family:'andale mono';color:rgb(41,249,20);background-color:rgb(0,0,0)"><span style="font-variant-ligatures:no-common-ligatures">2016-09-07 - 15:12:16 t_server: INFO: Bandwith: 7.72255 Gb/s</span></p>
<p style="margin:0px;font-size:12px;line-height:normal;font-family:'andale mono';color:rgb(41,249,20);background-color:rgb(0,0,0)"><span style="font-variant-ligatures:no-common-ligatures">2016-09-07 - 15:12:21 t_server: INFO: Bandwith: 14.7849 Gb/s</span></p>
<p style="margin:0px;font-size:12px;line-height:normal;font-family:'andale mono';color:rgb(41,249,20);background-color:rgb(0,0,0)"><span style="font-variant-ligatures:no-common-ligatures">2016-09-07 - 15:12:26 t_server: INFO: Bandwith: 12.9243 Gb/s</span></p>
<p style="margin:0px;font-size:12px;line-height:normal;font-family:'andale mono';color:rgb(41,249,20);background-color:rgb(0,0,0)"><span style="font-variant-ligatures:no-common-ligatures">2016-09-07 - 15:12:31 t_server: INFO: Bandwith: 0.687027 Gb/s</span></p>
<p style="margin:0px;font-size:12px;line-height:normal;font-family:'andale mono';color:rgb(41,249,20);background-color:rgb(0,0,0)"><span style="font-variant-ligatures:no-common-ligatures">2016-09-07 - 15:12:36 t_server: INFO: Bandwith: 1.44787 Gb/s</span></p>
<p style="margin:0px;font-size:12px;line-height:normal;font-family:'andale mono';color:rgb(41,249,20);background-color:rgb(0,0,0)"><span style="font-variant-ligatures:no-common-ligatures">2016-09-07 - 15:12:41 t_server: INFO: Bandwith: 2.681 Gb/s</span></p></div><div><br></div><div>Example of a test output with raw verbs:</div><div><p style="margin:0px;font-size:12px;line-height:normal;font-family:'andale mono';color:rgb(41,249,20);background-color:rgb(0,0,0)"><span style="font-variant-ligatures:no-common-ligatures">2016-09-07 - 16:36:00 t_server: INFO: Accepted connection</span></p>
<p style="margin:0px;font-size:12px;line-height:normal;font-family:'andale mono';color:rgb(41,249,20);background-color:rgb(0,0,0)"><span style="font-variant-ligatures:no-common-ligatures">2016-09-07 - 16:36:00 t_server: INFO: Start receiving...</span></p>
<p style="margin:0px;font-size:12px;line-height:normal;font-family:'andale mono';color:rgb(41,249,20);background-color:rgb(0,0,0)"><span style="font-variant-ligatures:no-common-ligatures">2016-09-07 - 16:36:05 t_server: INFO: Bandwith: 17.9491 Gb/s</span></p>
<p style="margin:0px;font-size:12px;line-height:normal;font-family:'andale mono';color:rgb(41,249,20);background-color:rgb(0,0,0)"><span style="font-variant-ligatures:no-common-ligatures">2016-09-07 - 16:36:10 t_server: INFO: Bandwith: 23.4671 Gb/s</span></p>
<p style="margin:0px;font-size:12px;line-height:normal;font-family:'andale mono';color:rgb(41,249,20);background-color:rgb(0,0,0)"><span style="font-variant-ligatures:no-common-ligatures">2016-09-07 - 16:36:15 t_server: INFO: Bandwith: 23.0368 Gb/s</span></p>
<p style="margin:0px;font-size:12px;line-height:normal;font-family:'andale mono';color:rgb(41,249,20);background-color:rgb(0,0,0)"><span style="font-variant-ligatures:no-common-ligatures">2016-09-07 - 16:36:20 t_server: INFO: Bandwith: 22.9638 Gb/s</span></p>
<p style="margin:0px;font-size:12px;line-height:normal;font-family:'andale mono';color:rgb(41,249,20);background-color:rgb(0,0,0)"><span style="font-variant-ligatures:no-common-ligatures">2016-09-07 - 16:36:25 t_server: INFO: Bandwith: 22.8203 Gb/s</span></p>
<p style="margin:0px;font-size:12px;line-height:normal;font-family:'andale mono';color:rgb(41,249,20);background-color:rgb(0,0,0)"><span style="font-variant-ligatures:no-common-ligatures">2016-09-07 - 16:36:30 t_server: INFO: Bandwith: 20.058 Gb/s</span></p>
<p style="margin:0px;font-size:12px;line-height:normal;font-family:'andale mono';color:rgb(41,249,20);background-color:rgb(0,0,0)"><span style="font-variant-ligatures:no-common-ligatures">2016-09-07 - 16:36:35 t_server: INFO: Bandwith: 22.5033 Gb/s</span></p>
<p style="margin:0px;font-size:12px;line-height:normal;font-family:'andale mono';color:rgb(41,249,20);background-color:rgb(0,0,0)"><span style="font-variant-ligatures:no-common-ligatures">2016-09-07 - 16:36:40 t_server: INFO: Bandwith: 20.1754 Gb/s</span></p>
<p style="margin:0px;font-size:12px;line-height:normal;font-family:'andale mono';color:rgb(41,249,20);background-color:rgb(0,0,0)"><span style="font-variant-ligatures:no-common-ligatures">2016-09-07 - 16:36:45 t_server: INFO: Bandwith: 22.5578 Gb/s</span></p>
<p style="margin:0px;font-size:12px;line-height:normal;font-family:'andale mono';color:rgb(41,249,20);background-color:rgb(0,0,0)"><span style="font-variant-ligatures:no-common-ligatures">2016-09-07 - 16:36:50 t_server: INFO: Bandwith: 20.0588 Gb/s</span></p>
<p style="margin:0px;font-size:12px;line-height:normal;font-family:'andale mono';color:rgb(41,249,20);background-color:rgb(0,0,0)"><span style="font-variant-ligatures:no-common-ligatures">2016-09-07 - 16:36:55 t_server: INFO: Bandwith: 22.2718 Gb/s</span></p>
<p style="margin:0px;font-size:12px;line-height:normal;font-family:'andale mono';color:rgb(41,249,20);background-color:rgb(0,0,0)"><span style="font-variant-ligatures:no-common-ligatures">2016-09-07 - 16:37:00 t_server: INFO: Bandwith: 22.494 Gb/s</span></p>
<p style="margin:0px;font-size:12px;line-height:normal;font-family:'andale mono';color:rgb(41,249,20);background-color:rgb(0,0,0)"><span style="font-variant-ligatures:no-common-ligatures">2016-09-07 - 16:37:05 t_server: INFO: Bandwith: 23.1836 Gb/s</span></p>
<p style="margin:0px;font-size:12px;line-height:normal;font-family:'andale mono';color:rgb(41,249,20);background-color:rgb(0,0,0)"><span style="font-variant-ligatures:no-common-ligatures">2016-09-07 - 16:37:10 t_server: INFO: Bandwith: 23.0972 Gb/s</span></p>
<p style="margin:0px;font-size:12px;line-height:normal;font-family:'andale mono';color:rgb(41,249,20);background-color:rgb(0,0,0)"><span style="font-variant-ligatures:no-common-ligatures">2016-09-07 - 16:37:15 t_server: INFO: Bandwith: 21.5033 Gb/s</span></p>
<p style="margin:0px;font-size:12px;line-height:normal;font-family:'andale mono';color:rgb(41,249,20);background-color:rgb(0,0,0)"><span style="font-variant-ligatures:no-common-ligatures">2016-09-07 - 16:37:20 t_server: INFO: Bandwith: 18.5506 Gb/s</span></p>
<p style="margin:0px;font-size:12px;line-height:normal;font-family:'andale mono';color:rgb(41,249,20);background-color:rgb(0,0,0)"><span style="font-variant-ligatures:no-common-ligatures">2016-09-07 - 16:37:25 t_server: INFO: Bandwith: 20.3709 Gb/s</span></p>
<p style="margin:0px;font-size:12px;line-height:normal;font-family:'andale mono';color:rgb(41,249,20);background-color:rgb(0,0,0)"><span style="font-variant-ligatures:no-common-ligatures">2016-09-07 - 16:37:30 t_server: INFO: Bandwith: 21.3457 Gb/s</span></p>
<p style="margin:0px;font-size:12px;line-height:normal;font-family:'andale mono';color:rgb(41,249,20);background-color:rgb(0,0,0)"><span style="font-variant-ligatures:no-common-ligatures">2016-09-07 - 16:37:35 t_server: INFO: Bandwith: 20.5059 Gb/s</span></p>
<p style="margin:0px;font-size:12px;line-height:normal;font-family:'andale mono';color:rgb(41,249,20);background-color:rgb(0,0,0)"><span style="font-variant-ligatures:no-common-ligatures">2016-09-07 - 16:37:40 t_server: INFO: Bandwith: 22.4899 Gb/s</span></p>
<p style="margin:0px;font-size:12px;line-height:normal;font-family:'andale mono';color:rgb(41,249,20);background-color:rgb(0,0,0)"><span style="font-variant-ligatures:no-common-ligatures">2016-09-07 - 16:37:45 t_server: INFO: Bandwith: 22.1266 Gb/s</span></p>
<p style="margin:0px;font-size:12px;line-height:normal;font-family:'andale mono';color:rgb(41,249,20);background-color:rgb(0,0,0)"><span style="font-variant-ligatures:no-common-ligatures">2016-09-07 - 16:37:50 t_server: INFO: Bandwith: 22.4504 Gb/s</span></p></div><div><br></div><div><br></div></div>