<html xmlns:v="urn:schemas-microsoft-com:vml" xmlns:o="urn:schemas-microsoft-com:office:office" xmlns:w="urn:schemas-microsoft-com:office:word" xmlns:m="http://schemas.microsoft.com/office/2004/12/omml" xmlns="http://www.w3.org/TR/REC-html40">
<head>
<meta http-equiv="Content-Type" content="text/html; charset=utf-8">
<meta name="Generator" content="Microsoft Word 15 (filtered medium)">
<style><!--
/* Font Definitions */
@font-face
{font-family:"Cambria Math";
panose-1:2 4 5 3 5 4 6 3 2 4;}
@font-face
{font-family:Calibri;
panose-1:2 15 5 2 2 2 4 3 2 4;}
@font-face
{font-family:"andale mono";
panose-1:0 0 0 0 0 0 0 0 0 0;}
/* Style Definitions */
p.MsoNormal, li.MsoNormal, div.MsoNormal
{margin:0in;
margin-bottom:.0001pt;
font-size:12.0pt;
font-family:"Times New Roman",serif;}
a:link, span.MsoHyperlink
{mso-style-priority:99;
color:#0563C1;
text-decoration:underline;}
a:visited, span.MsoHyperlinkFollowed
{mso-style-priority:99;
color:#954F72;
text-decoration:underline;}
p
{mso-style-priority:99;
mso-margin-top-alt:auto;
margin-right:0in;
mso-margin-bottom-alt:auto;
margin-left:0in;
font-size:12.0pt;
font-family:"Times New Roman",serif;}
span.EmailStyle18
{mso-style-type:personal-reply;
font-family:"Calibri",sans-serif;
color:#1F497D;}
.MsoChpDefault
{mso-style-type:export-only;
font-family:"Calibri",sans-serif;}
@page WordSection1
{size:8.5in 11.0in;
margin:1.0in 1.0in 1.0in 1.0in;}
div.WordSection1
{page:WordSection1;}
--></style><!--[if gte mso 9]><xml>
<o:shapedefaults v:ext="edit" spidmax="1026" />
</xml><![endif]--><!--[if gte mso 9]><xml>
<o:shapelayout v:ext="edit">
<o:idmap v:ext="edit" data="1" />
</o:shapelayout></xml><![endif]-->
</head>
<body lang="EN-US" link="#0563C1" vlink="#954F72">
<div class="WordSection1">
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:"Calibri",sans-serif;color:#1F497D">Hi Valentino,<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:"Calibri",sans-serif;color:#1F497D"><o:p> </o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:"Calibri",sans-serif;color:#1F497D">Libfabric has a set of tests available at https://github.com/ofiwg/fabtests. Can you run the fi_msg_bw test with the same size and iterations on your setup and
check if you notice any variance? Also what version/commit number of libfabric are you using?<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:"Calibri",sans-serif;color:#1F497D"><o:p> </o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:"Calibri",sans-serif;color:#1F497D">Thanks,<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:"Calibri",sans-serif;color:#1F497D">Arun.<o:p></o:p></span></p>
<p class="MsoNormal"><a name="_MailEndCompose"><span style="font-size:11.0pt;font-family:"Calibri",sans-serif;color:#1F497D"><o:p> </o:p></span></a></p>
<p class="MsoNormal"><a name="_____replyseparator"></a><b><span style="font-size:11.0pt;font-family:"Calibri",sans-serif">From:</span></b><span style="font-size:11.0pt;font-family:"Calibri",sans-serif"> Libfabric-users [mailto:libfabric-users-bounces@lists.openfabrics.org]
<b>On Behalf Of </b>Valentino Picotti<br>
<b>Sent:</b> Wednesday, September 07, 2016 7:48 AM<br>
<b>To:</b> libfabric-users@lists.openfabrics.org<br>
<b>Subject:</b> [libfabric-users] Optimisation Tips for verbs provider<o:p></o:p></span></p>
<p class="MsoNormal"><o:p> </o:p></p>
<div>
<p class="MsoNormal">Hi all,<o:p></o:p></p>
<div>
<p class="MsoNormal"><o:p> </o:p></p>
</div>
<div>
<p class="MsoNormal">I apologies in advance for the long email.<o:p></o:p></p>
</div>
<div>
<p class="MsoNormal"><o:p> </o:p></p>
</div>
<div>
<p class="MsoNormal">In the past month I've integrated libfabric in a project based on infiniband verbs with the aim to be provider independent. This project has a transport layer that makes the application independent from the transport implementation (that
is chosen at compile time).<o:p></o:p></p>
</div>
<div>
<p class="MsoNormal">I worked only on the libfabric implementation of the transport layer and this was my first experience with RDMA APIs and hardware. What I did was to map the various ibv_* and rdma_* calls to fi_* calls and I got a working layer quite easily
(after studying the libfabric terminology).<o:p></o:p></p>
</div>
<div>
<p class="MsoNormal">Now I'm trying to achieve the same performance of raw verbs.<o:p></o:p></p>
</div>
<div>
<p class="MsoNormal">I'm testing the transport layer with a one sided communication where a client sends the data to a server with the message API(fi_send/fi_recv). The client and the server run on two different nodes connected with one IB EDR link: i don't
set processor affinity nor change power management policy. The depth of completion queues and the size of sent buffers are the same across the tests.<o:p></o:p></p>
</div>
<div>
<p class="MsoNormal">Running on the verbs transport layer I get a stable bandwidth of 22 Gbps, instead with libfabric over verbs I get a very floating bandwidth: from 0.4 Gbps to 19 Gbps in the same test[1]. The bandwidth is calculated as the number of buffers
sent every 5 seconds.<o:p></o:p></p>
</div>
<div>
<p class="MsoNormal"><o:p> </o:p></p>
</div>
<div>
<p class="MsoNormal">This is how i setup the verbs provider:<o:p></o:p></p>
</div>
<p class="MsoNormal"><br>
m_hints->caps = FI_MSG;<br>
m_hints->mode = FI_LOCAL_MR;<br>
m_hints->ep_attr->type = FI_EP_MSG;<br>
m_hints->domain_attr->threading = FI_THREAD_COMPLETION;<br>
m_hints->domain_attr->data_progress = FI_PROGRESS_MANUAL;<br>
m_hints->domain_attr->resource_mgmt = FI_RM_DISABLED;<br>
m_hints->fabric_attr->prov_name = strdup("verbs");<o:p></o:p></p>
<div>
<p class="MsoNormal"><o:p> </o:p></p>
</div>
<div>
<p class="MsoNormal">Furthermore I bind two completion queues to the endpoints: one with FI_SEND flag and the other with FI_RECV.<o:p></o:p></p>
</div>
<div>
<p class="MsoNormal"><o:p> </o:p></p>
</div>
<div>
<p class="MsoNormal">I can't figure out why I'm getting that high variance with libfabric.<o:p></o:p></p>
</div>
<div>
<p class="MsoNormal">Do you have any idea? I'm missing same optimisations tips for the verbs provider?<o:p></o:p></p>
</div>
<div>
<p class="MsoNormal"><o:p> </o:p></p>
</div>
<div>
<p class="MsoNormal">Thanks in advance,<o:p></o:p></p>
</div>
<div>
<p class="MsoNormal"><o:p> </o:p></p>
</div>
<div>
<p class="MsoNormal">Valentino<o:p></o:p></p>
</div>
<div>
<p class="MsoNormal"><o:p> </o:p></p>
</div>
<div>
<p class="MsoNormal"><o:p> </o:p></p>
</div>
<div>
<p class="MsoNormal">[1] Test run with depth queue of 1 and buffer size of 512KB <o:p></o:p></p>
</div>
<div>
<p class="MsoNormal"><o:p> </o:p></p>
</div>
<div>
<p class="MsoNormal">Example of a test output with libfarbic:<o:p></o:p></p>
</div>
<div>
<p style="margin:0in;margin-bottom:.0001pt;background:black"><span style="font-size:9.0pt;font-family:"andale mono",serif;color:#29F914">2016-09-07 - 15:10:56 t_server: INFO: Accepted connection<o:p></o:p></span></p>
<p style="margin:0in;margin-bottom:.0001pt;background:black"><span style="font-size:9.0pt;font-family:"andale mono",serif;color:#29F914">2016-09-07 - 15:10:56 t_server: INFO: Start receiving...<o:p></o:p></span></p>
<p style="margin:0in;margin-bottom:.0001pt;background:black"><span style="font-size:9.0pt;font-family:"andale mono",serif;color:#29F914">2016-09-07 - 15:11:01 t_server: INFO: Bandwith: 8.3324 Gb/s<o:p></o:p></span></p>
<p style="margin:0in;margin-bottom:.0001pt;background:black"><span style="font-size:9.0pt;font-family:"andale mono",serif;color:#29F914">2016-09-07 - 15:11:06 t_server: INFO: Bandwith: 15.831 Gb/s<o:p></o:p></span></p>
<p style="margin:0in;margin-bottom:.0001pt;background:black"><span style="font-size:9.0pt;font-family:"andale mono",serif;color:#29F914">2016-09-07 - 15:11:11 t_server: INFO: Bandwith: 19.1713 Gb/s<o:p></o:p></span></p>
<p style="margin:0in;margin-bottom:.0001pt;background:black"><span style="font-size:9.0pt;font-family:"andale mono",serif;color:#29F914">2016-09-07 - 15:11:16 t_server: INFO: Bandwith: 10.8825 Gb/s<o:p></o:p></span></p>
<p style="margin:0in;margin-bottom:.0001pt;background:black"><span style="font-size:9.0pt;font-family:"andale mono",serif;color:#29F914">2016-09-07 - 15:11:21 t_server: INFO: Bandwith: 8.07991 Gb/s<o:p></o:p></span></p>
<p style="margin:0in;margin-bottom:.0001pt;background:black"><span style="font-size:9.0pt;font-family:"andale mono",serif;color:#29F914">2016-09-07 - 15:11:26 t_server: INFO: Bandwith: 15.4015 Gb/s<o:p></o:p></span></p>
<p style="margin:0in;margin-bottom:.0001pt;background:black"><span style="font-size:9.0pt;font-family:"andale mono",serif;color:#29F914">2016-09-07 - 15:11:31 t_server: INFO: Bandwith: 20.4263 Gb/s<o:p></o:p></span></p>
<p style="margin:0in;margin-bottom:.0001pt;background:black"><span style="font-size:9.0pt;font-family:"andale mono",serif;color:#29F914">2016-09-07 - 15:11:36 t_server: INFO: Bandwith: 19.7023 Gb/s<o:p></o:p></span></p>
<p style="margin:0in;margin-bottom:.0001pt;background:black"><span style="font-size:9.0pt;font-family:"andale mono",serif;color:#29F914">2016-09-07 - 15:11:41 t_server: INFO: Bandwith: 10.474 Gb/s<o:p></o:p></span></p>
<p style="margin:0in;margin-bottom:.0001pt;background:black"><span style="font-size:9.0pt;font-family:"andale mono",serif;color:#29F914">2016-09-07 - 15:11:46 t_server: INFO: Bandwith: 17.4072 Gb/s<o:p></o:p></span></p>
<p style="margin:0in;margin-bottom:.0001pt;background:black"><span style="font-size:9.0pt;font-family:"andale mono",serif;color:#29F914">2016-09-07 - 15:11:51 t_server: INFO: Bandwith: 0.440402 Gb/s<o:p></o:p></span></p>
<p style="margin:0in;margin-bottom:.0001pt;background:black"><span style="font-size:9.0pt;font-family:"andale mono",serif;color:#29F914">2016-09-07 - 15:11:56 t_server: INFO: Bandwith: 2.73217 Gb/s<o:p></o:p></span></p>
<p style="margin:0in;margin-bottom:.0001pt;background:black"><span style="font-size:9.0pt;font-family:"andale mono",serif;color:#29F914">2016-09-07 - 15:12:01 t_server: INFO: Bandwith: 0.984822 Gb/s<o:p></o:p></span></p>
<p style="margin:0in;margin-bottom:.0001pt;background:black"><span style="font-size:9.0pt;font-family:"andale mono",serif;color:#29F914">2016-09-07 - 15:12:06 t_server: INFO: Bandwith: 2.93013 Gb/s<o:p></o:p></span></p>
<p style="margin:0in;margin-bottom:.0001pt;background:black"><span style="font-size:9.0pt;font-family:"andale mono",serif;color:#29F914">2016-09-07 - 15:12:11 t_server: INFO: Bandwith: 0.847248 Gb/s<o:p></o:p></span></p>
<p style="margin:0in;margin-bottom:.0001pt;background:black"><span style="font-size:9.0pt;font-family:"andale mono",serif;color:#29F914">2016-09-07 - 15:12:16 t_server: INFO: Bandwith: 7.72255 Gb/s<o:p></o:p></span></p>
<p style="margin:0in;margin-bottom:.0001pt;background:black"><span style="font-size:9.0pt;font-family:"andale mono",serif;color:#29F914">2016-09-07 - 15:12:21 t_server: INFO: Bandwith: 14.7849 Gb/s<o:p></o:p></span></p>
<p style="margin:0in;margin-bottom:.0001pt;background:black"><span style="font-size:9.0pt;font-family:"andale mono",serif;color:#29F914">2016-09-07 - 15:12:26 t_server: INFO: Bandwith: 12.9243 Gb/s<o:p></o:p></span></p>
<p style="margin:0in;margin-bottom:.0001pt;background:black"><span style="font-size:9.0pt;font-family:"andale mono",serif;color:#29F914">2016-09-07 - 15:12:31 t_server: INFO: Bandwith: 0.687027 Gb/s<o:p></o:p></span></p>
<p style="margin:0in;margin-bottom:.0001pt;background:black"><span style="font-size:9.0pt;font-family:"andale mono",serif;color:#29F914">2016-09-07 - 15:12:36 t_server: INFO: Bandwith: 1.44787 Gb/s<o:p></o:p></span></p>
<p style="margin:0in;margin-bottom:.0001pt;background:black"><span style="font-size:9.0pt;font-family:"andale mono",serif;color:#29F914">2016-09-07 - 15:12:41 t_server: INFO: Bandwith: 2.681 Gb/s<o:p></o:p></span></p>
</div>
<div>
<p class="MsoNormal"><o:p> </o:p></p>
</div>
<div>
<p class="MsoNormal">Example of a test output with raw verbs:<o:p></o:p></p>
</div>
<div>
<p style="margin:0in;margin-bottom:.0001pt;background:black"><span style="font-size:9.0pt;font-family:"andale mono",serif;color:#29F914">2016-09-07 - 16:36:00 t_server: INFO: Accepted connection<o:p></o:p></span></p>
<p style="margin:0in;margin-bottom:.0001pt;background:black"><span style="font-size:9.0pt;font-family:"andale mono",serif;color:#29F914">2016-09-07 - 16:36:00 t_server: INFO: Start receiving...<o:p></o:p></span></p>
<p style="margin:0in;margin-bottom:.0001pt;background:black"><span style="font-size:9.0pt;font-family:"andale mono",serif;color:#29F914">2016-09-07 - 16:36:05 t_server: INFO: Bandwith: 17.9491 Gb/s<o:p></o:p></span></p>
<p style="margin:0in;margin-bottom:.0001pt;background:black"><span style="font-size:9.0pt;font-family:"andale mono",serif;color:#29F914">2016-09-07 - 16:36:10 t_server: INFO: Bandwith: 23.4671 Gb/s<o:p></o:p></span></p>
<p style="margin:0in;margin-bottom:.0001pt;background:black"><span style="font-size:9.0pt;font-family:"andale mono",serif;color:#29F914">2016-09-07 - 16:36:15 t_server: INFO: Bandwith: 23.0368 Gb/s<o:p></o:p></span></p>
<p style="margin:0in;margin-bottom:.0001pt;background:black"><span style="font-size:9.0pt;font-family:"andale mono",serif;color:#29F914">2016-09-07 - 16:36:20 t_server: INFO: Bandwith: 22.9638 Gb/s<o:p></o:p></span></p>
<p style="margin:0in;margin-bottom:.0001pt;background:black"><span style="font-size:9.0pt;font-family:"andale mono",serif;color:#29F914">2016-09-07 - 16:36:25 t_server: INFO: Bandwith: 22.8203 Gb/s<o:p></o:p></span></p>
<p style="margin:0in;margin-bottom:.0001pt;background:black"><span style="font-size:9.0pt;font-family:"andale mono",serif;color:#29F914">2016-09-07 - 16:36:30 t_server: INFO: Bandwith: 20.058 Gb/s<o:p></o:p></span></p>
<p style="margin:0in;margin-bottom:.0001pt;background:black"><span style="font-size:9.0pt;font-family:"andale mono",serif;color:#29F914">2016-09-07 - 16:36:35 t_server: INFO: Bandwith: 22.5033 Gb/s<o:p></o:p></span></p>
<p style="margin:0in;margin-bottom:.0001pt;background:black"><span style="font-size:9.0pt;font-family:"andale mono",serif;color:#29F914">2016-09-07 - 16:36:40 t_server: INFO: Bandwith: 20.1754 Gb/s<o:p></o:p></span></p>
<p style="margin:0in;margin-bottom:.0001pt;background:black"><span style="font-size:9.0pt;font-family:"andale mono",serif;color:#29F914">2016-09-07 - 16:36:45 t_server: INFO: Bandwith: 22.5578 Gb/s<o:p></o:p></span></p>
<p style="margin:0in;margin-bottom:.0001pt;background:black"><span style="font-size:9.0pt;font-family:"andale mono",serif;color:#29F914">2016-09-07 - 16:36:50 t_server: INFO: Bandwith: 20.0588 Gb/s<o:p></o:p></span></p>
<p style="margin:0in;margin-bottom:.0001pt;background:black"><span style="font-size:9.0pt;font-family:"andale mono",serif;color:#29F914">2016-09-07 - 16:36:55 t_server: INFO: Bandwith: 22.2718 Gb/s<o:p></o:p></span></p>
<p style="margin:0in;margin-bottom:.0001pt;background:black"><span style="font-size:9.0pt;font-family:"andale mono",serif;color:#29F914">2016-09-07 - 16:37:00 t_server: INFO: Bandwith: 22.494 Gb/s<o:p></o:p></span></p>
<p style="margin:0in;margin-bottom:.0001pt;background:black"><span style="font-size:9.0pt;font-family:"andale mono",serif;color:#29F914">2016-09-07 - 16:37:05 t_server: INFO: Bandwith: 23.1836 Gb/s<o:p></o:p></span></p>
<p style="margin:0in;margin-bottom:.0001pt;background:black"><span style="font-size:9.0pt;font-family:"andale mono",serif;color:#29F914">2016-09-07 - 16:37:10 t_server: INFO: Bandwith: 23.0972 Gb/s<o:p></o:p></span></p>
<p style="margin:0in;margin-bottom:.0001pt;background:black"><span style="font-size:9.0pt;font-family:"andale mono",serif;color:#29F914">2016-09-07 - 16:37:15 t_server: INFO: Bandwith: 21.5033 Gb/s<o:p></o:p></span></p>
<p style="margin:0in;margin-bottom:.0001pt;background:black"><span style="font-size:9.0pt;font-family:"andale mono",serif;color:#29F914">2016-09-07 - 16:37:20 t_server: INFO: Bandwith: 18.5506 Gb/s<o:p></o:p></span></p>
<p style="margin:0in;margin-bottom:.0001pt;background:black"><span style="font-size:9.0pt;font-family:"andale mono",serif;color:#29F914">2016-09-07 - 16:37:25 t_server: INFO: Bandwith: 20.3709 Gb/s<o:p></o:p></span></p>
<p style="margin:0in;margin-bottom:.0001pt;background:black"><span style="font-size:9.0pt;font-family:"andale mono",serif;color:#29F914">2016-09-07 - 16:37:30 t_server: INFO: Bandwith: 21.3457 Gb/s<o:p></o:p></span></p>
<p style="margin:0in;margin-bottom:.0001pt;background:black"><span style="font-size:9.0pt;font-family:"andale mono",serif;color:#29F914">2016-09-07 - 16:37:35 t_server: INFO: Bandwith: 20.5059 Gb/s<o:p></o:p></span></p>
<p style="margin:0in;margin-bottom:.0001pt;background:black"><span style="font-size:9.0pt;font-family:"andale mono",serif;color:#29F914">2016-09-07 - 16:37:40 t_server: INFO: Bandwith: 22.4899 Gb/s<o:p></o:p></span></p>
<p style="margin:0in;margin-bottom:.0001pt;background:black"><span style="font-size:9.0pt;font-family:"andale mono",serif;color:#29F914">2016-09-07 - 16:37:45 t_server: INFO: Bandwith: 22.1266 Gb/s<o:p></o:p></span></p>
<p style="margin:0in;margin-bottom:.0001pt;background:black"><span style="font-size:9.0pt;font-family:"andale mono",serif;color:#29F914">2016-09-07 - 16:37:50 t_server: INFO: Bandwith: 22.4504 Gb/s<o:p></o:p></span></p>
</div>
<div>
<p class="MsoNormal"><o:p> </o:p></p>
</div>
<div>
<p class="MsoNormal"><o:p> </o:p></p>
</div>
</div>
</div>
</body>
</html>