<div dir="ltr">Thanks for the reply Arun,<div><br></div><div><div class="gmail_extra"><div class="gmail_quote"><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left-width:1px;border-left-color:rgb(204,204,204);border-left-style:solid;padding-left:1ex"><div lang="EN-US"><div>
<p class="MsoNormal"><span style="font-size:11pt;font-family:calibri,sans-serif;color:rgb(31,73,125)">You can run the fi_msg_bw test with other completion-waiting options like sread and fd too. I don’t get any variance when using them as well. </span></p></div></div></blockquote><div>There is no difference between cq_sread and spin on cq_read since the verbs cq_sread implementation spins on cq_read.</div><div><br></div><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left-width:1px;border-left-color:rgb(204,204,204);border-left-style:solid;padding-left:1ex"><div lang="EN-US"><div><p class="MsoNormal"><span style="font-size:11pt;font-family:calibri,sans-serif;color:rgb(31,73,125)">The test calls fi_cq_read
after posting every window size number of sends/recvs. The window size is an adjustable parameter. You can view all available options by calling fi_msg_bw -h.</span></p></div></div></blockquote><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left-width:1px;border-left-color:rgb(204,204,204);border-left-style:solid;padding-left:1ex"><div lang="EN-US"><div>
<p class="MsoNormal"><span style="font-size:11pt;font-family:calibri,sans-serif;color:rgb(31,73,125)">May be you could try posting a bunch of sends/recvs and then collect the completions for the bunch. Is there a need to post the messages one by one? If that’s
the case please try using spin wait for getting the completion. But even that might not guarantee consistent numbers. </span></p></div></div></blockquote><div><br></div><div>I run the test and configured it in a such way that is similar to our client/server test. So first of all I made sure to never call <a href="https://github.com/ofiwg/fabtests/blob/master/benchmarks/benchmark_shared.c#L188" target="_blank">bw_tx/rx_comp()</a>, setting the window size equal to the number of iterations. Then applied the following modifications to the shared.c file.</div><div><br></div><div>Here is my diff of shared.c:<br></div><div><br></div><div><div>diff --git a/common/shared.c b/common/shared.c</div><div>index 1709443..f93a209 100644</div><div>--- a/common/shared.c</div><div>+++ b/common/shared.c</div><div>@@ -344,7 +344,7 @@ int ft_alloc_ep_res(struct fi_info *fi)<br></div><div> </div><div> if (opts.options & FT_OPT_TX_CQ) {</div><div> ft_cq_set_wait_attr();</div><div>- cq_attr.size = fi->tx_attr->size;</div><div>+ cq_attr.size = 1;//fi->tx_attr->size;</div><div> ret = fi_cq_open(domain, &cq_attr, &txcq, &txcq);</div><div> if (ret) {</div><div> FT_PRINTERR("fi_cq_open", ret);</div><div><br></div><div>@@ -363,7 +363,7 @@ int ft_alloc_ep_res(struct fi_info *fi)</div><div> </div><div> if (opts.options & FT_OPT_RX_CQ) {</div><div> ft_cq_set_wait_attr();</div><div>- cq_attr.size = fi->rx_attr->size;</div><div>+ cq_attr.size = 1;//fi->rx_attr->size;</div><div> ret = fi_cq_open(domain, &cq_attr, &rxcq, &rxcq);</div><div> if (ret) {</div><div> FT_PRINTERR("fi_cq_open", ret);</div><div><br></div><div>@@ -1224,9 +1224,12 @@ static int ft_spin_for_comp(struct fid_cq *cq, uint64_t *cur,<br></div><div> } else if (timeout >= 0) {</div><div> clock_gettime(CLOCK_MONOTONIC, &b);</div><div> if ((b.tv_sec - a.tv_sec) > timeout) {</div><div>- fprintf(stderr, "%ds timeout expired\n", timeout);</div><div>- return -FI_ENODATA;</div><div>+ //fprintf(stderr, "Total: %d %ds timeout expired\n", total, timeout);</div><div>+ return 0;//-FI_ENODATA;</div><div> }</div><div>+ } else if (ret == -FI_EAGAIN && timeout == -1) {</div><div>+ //fprintf(stdout, "Iter: %d\n", total);</div><div>+ return 0;</div><div> }</div><div> }</div></div><div><br></div><div>The first two edits ensure to initialise the cqs with a depth of 1, the same of our client/server test. The third edit involves <a href="https://github.com/ofiwg/fabtests/blob/master/common/shared.c#L1205" target="_blank">this function</a>.</div><div>The first else-if statement displayed in the third diff is the path of the server process in the case a fi_cq_read, called after a failed fi_recv (-FI_EAGAIN), returns -FI_EAGAIN:</div><div>Returning zero form this function implies an immediate retry of a post receive operation, in other words we don't exit the while loop of the FT_POST macro.</div><div>The second else-if statement, whose condition of activation is the same of an else, is the path chosen when:</div><div>- ft_spin_for_comp is called with timeout equals to -1, i.e. in the client process</div><div>- fi_cq_read, called after a failed fi_send (-FI_EAGAIN), returns -FI_EAGAIN.</div><div><br></div><div>With this modifications, the msg_bw test stays in the FT_POST macro until a post operation is successful.</div><div>The send/receive loop of the test is reduced to the following:</div><div>- post a work request</div><div>- make a non blocking read on queue</div><div><br></div><div>This is what I get:</div><div><p style="margin:0px;font-size:12px;line-height:normal;font-family:'andale mono';color:rgb(41,249,20);background-color:rgb(0,0,0)"><span>./picotti/fabtests/bin/fi_msg_<wbr>bw -t queue -c spin -f verbs -S $((1024*512)) -I 10000 -W 10000 -w 0 -s 10.23.4.166</span></p>
<p style="margin:0px;font-size:12px;line-height:normal;font-family:'andale mono';color:rgb(41,249,20);background-color:rgb(0,0,0)">bytes iters total time MB/sec usec/xfer Mxfers/sec<br></p>
<p style="margin:0px;font-size:12px;line-height:normal;font-family:'andale mono';color:rgb(41,249,20);background-color:rgb(0,0,0)"><span>512k 10k 4.8g 16.58s 316.16 1658.28 0.00</span></p></div><div><p style="margin:0px;font-size:12px;line-height:normal;font-family:'andale mono';color:rgb(41,249,20);background-color:rgb(0,0,0)"><span>bytes iters total time MB/sec usec/xfer Mxfers/sec</span></p>
<p style="margin:0px;font-size:12px;line-height:normal;font-family:'andale mono';color:rgb(41,249,20);background-color:rgb(0,0,0)"><span>512k 10k 4.8g 6.43s 815.20 643.14 0.00</span></p></div><div><span><p style="margin:0px;font-size:12px;line-height:normal;font-family:'andale mono';color:rgb(41,249,20);background-color:rgb(0,0,0)"><span>bytes iters total time MB/sec usec/xfer Mxfers/sec</span></p>
<p style="margin:0px;font-size:12px;line-height:normal;font-family:'andale mono';color:rgb(41,249,20);background-color:rgb(0,0,0)"><span>512k 10k 4.8g 16.86s 311.04 1685.61 0.00</span></p></span></div><div><p style="margin:0px;font-size:12px;line-height:normal;font-family:'andale mono';color:rgb(41,249,20);background-color:rgb(0,0,0)"><span>bytes iters total time MB/sec usec/xfer Mxfers/sec</span></p>
<p style="margin:0px;font-size:12px;line-height:normal;font-family:'andale mono';color:rgb(41,249,20);background-color:rgb(0,0,0)"><span>512k 10k 4.8g 13.15s 398.61 1315.30 0.00</span></p></div><div><br></div><div>This results are aligned with the bandwidth of our client/server test.</div><div><br></div><div><div> </div><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left-width:1px;border-left-color:rgb(204,204,204);border-left-style:solid;padding-left:1ex"><div lang="EN-US"><p class="MsoNormal"><span style="font-size:11pt;font-family:calibri,sans-serif;color:rgb(31,73,125)">Currently, there is an issue of unpredictable transfer times if the sender overruns the receiver.</span></p></div></blockquote></div><div><br></div><div>So I have reproduced this issue with the msg_bw test? Is a libfarbic issue?</div><div>I don't understand how the bw_tx/rx_comp avoids this problem.</div><div><br></div><div><br></div><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left-width:1px;border-left-color:rgb(204,204,204);border-left-style:solid;padding-left:1ex"><div lang="EN-US"><div><p class="MsoNormal"></p>
<p class="MsoNormal"><span style="font-size:11pt;font-family:calibri,sans-serif;color:rgb(31,73,125)">-Arun.</span></p></div></div></blockquote><div><br></div><div>Thanks Arun,</div><div><br></div><div>Valentino.<br></div><div><br></div><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left-width:1px;border-left-color:rgb(204,204,204);border-left-style:solid;padding-left:1ex"><div lang="EN-US"><div><p class="MsoNormal"><span style="font-size:11pt;font-family:calibri,sans-serif;color:rgb(31,73,125)"><u></u><u></u></span></p>
<p class="MsoNormal"><a name="m_2151765066196944266_m_-2371084464376655905_m_5694920665038086619_m_-5999422344560847129__MailEndCompose"><span style="font-size:11pt;font-family:calibri,sans-serif;color:rgb(31,73,125)"><u></u> <u></u></span></a></p>
<p class="MsoNormal"><a name="m_2151765066196944266_m_-2371084464376655905_m_5694920665038086619_m_-5999422344560847129______replyseparator"></a><b><span style="font-size:11pt;font-family:calibri,sans-serif">From:</span></b><span style="font-size:11pt;font-family:calibri,sans-serif"> Libfabric-users [mailto:<a href="mailto:libfabric-users-bounces@lists.openfabrics.org" target="_blank">libfabric-users-bounce<wbr>s@lists.openfabrics.org</a>]
<b>On Behalf Of </b>Valentino Picotti<br>
<b>Sent:</b> Thursday, September 08, 2016 5:12 AM<br>
<b>To:</b> <a href="mailto:libfabric-users@lists.openfabrics.org" target="_blank">libfabric-users@lists.openfabr<wbr>ics.org</a><br>
<b>Subject:</b> [libfabric-users] Fwd: Optimisation Tips for verbs provider<u></u><u></u></span></p><div><div>
<p class="MsoNormal"><u></u> <u></u></p>
<div>
<p class="MsoNormal">I forgot to CC the list, here is my reply to Arun:<u></u><u></u></p>
<div>
<p class="MsoNormal"><u></u> <u></u></p>
<div>
<p class="MsoNormal" style="margin-bottom:12pt">---------- Forwarded message ----------<br>
From: <b>Valentino Picotti</b> <<a href="mailto:valentino.picotti@gmail.com" target="_blank">valentino.picotti@gmail.com</a>><br>
Date: 8 September 2016 at 14:00<br>
Subject: Re: [libfabric-users] Optimisation Tips for verbs provider<br>
To: "Ilango, Arun" <<a href="mailto:arun.ilango@intel.com" target="_blank">arun.ilango@intel.com</a>><br>
<br>
<u></u><u></u></p>
<div>
<p class="MsoNormal">Thanks for the reply,<u></u><u></u></p>
<div>
<p class="MsoNormal"><u></u> <u></u></p>
</div>
<div>
<p class="MsoNormal">I run the fi_msg_bw with CQ size and window size of 1 and I got the following result:<u></u><u></u></p>
</div>
<div>
<p style="margin:0in 0in 0.0001pt;background-color:black"><span style="font-size:9pt;font-family:'andale mono',serif;color:rgb(41,249,20)">bytes iters total time MB/sec usec/xfer Mxfers/sec<u></u><u></u></span></p>
<p style="margin:0in 0in 0.0001pt;background-color:black"><span style="font-size:9pt;font-family:'andale mono',serif;color:rgb(41,249,20)">512k 1m 488g 189.50s 2766.76 189.50 0.01<u></u><u></u></span></p>
</div>
<div>
<p class="MsoNormal">21,61 Gbps is an excellent result. I'm using libfabric 1.3.0 from the latest tarball. <u></u><u></u></p>
</div>
<div>
<p class="MsoNormal"><u></u> <u></u></p>
</div>
<div>
<p class="MsoNormal">So the problem is in my transport layer.<u></u><u></u></p>
</div>
<div>
<p class="MsoNormal">My fabric initialisation doesn't differs too much from the fi_msg_bw one, so the problem might be in the main loop.<u></u><u></u></p>
</div>
<div>
<p class="MsoNormal">At a first glance, It seems that i call fi_cq_read less often than the bw test.<u></u><u></u></p>
</div>
<div>
<p class="MsoNormal">In the test the sequence is:<u></u><u></u></p>
</div>
<div>
<p class="MsoNormal">- post work request ft_post_tx/rx<u></u><u></u></p>
</div>
<div>
<p class="MsoNormal">- spin on a completion bw_(tx/rx)_comp<u></u><u></u></p>
</div>
<div>
<p class="MsoNormal"><u></u> <u></u></p>
</div>
<div>
<p class="MsoNormal">In my client/server main loop:<u></u><u></u></p>
</div>
<div>
<p class="MsoNormal">- call fi_cq_read<u></u><u></u></p>
</div>
<div>
<p class="MsoNormal">- post work request<u></u><u></u></p>
</div>
<div>
<p class="MsoNormal"><u></u> <u></u></p>
</div>
<div>
<p class="MsoNormal">I don't spin waiting for completions, could this be the reason?<u></u><u></u></p>
</div>
<div>
<p class="MsoNormal"><u></u> <u></u></p>
</div>
<div>
<p class="MsoNormal">Thanks,<u></u><u></u></p>
</div>
<div>
<p class="MsoNormal">Valentino<u></u><u></u></p>
</div>
<div>
<p class="MsoNormal"><u></u> <u></u></p>
</div>
</div>
<div>
<div>
<div>
<p class="MsoNormal"><u></u> <u></u></p>
<div>
<p class="MsoNormal">On 7 September 2016 at 19:17, Ilango, Arun <<a href="mailto:arun.ilango@intel.com" target="_blank">arun.ilango@intel.com</a>> wrote:<u></u><u></u></p>
<blockquote style="border-style:none none none solid;border-left-color:rgb(204,204,204);border-left-width:1pt;padding:0in 0in 0in 6pt;margin-left:4.8pt;margin-right:0in">
<div>
<div>
<p class="MsoNormal"><span style="font-size:11pt;font-family:calibri,sans-serif;color:rgb(31,73,125)">Hi Valentino,</span><u></u><u></u></p>
<p class="MsoNormal"><span style="font-size:11pt;font-family:calibri,sans-serif;color:rgb(31,73,125)"> </span><u></u><u></u></p>
<p class="MsoNormal"><span style="font-size:11pt;font-family:calibri,sans-serif;color:rgb(31,73,125)">Libfabric has a set of tests available at
<a href="https://github.com/ofiwg/fabtests" target="_blank">https://github.com/ofiwg/fabte<wbr>sts</a>. Can you run the fi_msg_bw test with the same size and iterations on your setup and check if you notice any variance? Also what version/commit number of libfabric
are you using?</span><u></u><u></u></p>
<p class="MsoNormal"><span style="font-size:11pt;font-family:calibri,sans-serif;color:rgb(31,73,125)"> </span><u></u><u></u></p>
<p class="MsoNormal"><span style="font-size:11pt;font-family:calibri,sans-serif;color:rgb(31,73,125)">Thanks,</span><u></u><u></u></p>
<p class="MsoNormal"><span style="font-size:11pt;font-family:calibri,sans-serif;color:rgb(31,73,125)">Arun.</span><u></u><u></u></p>
<p class="MsoNormal"><a name="m_2151765066196944266_m_-2371084464376655905_m_5694920665038086619_m_-5999422344560847129_m_6693636125052867833_m_6391039724988073"><span style="font-size:11pt;font-family:calibri,sans-serif;color:rgb(31,73,125)"> </span><u></u><u></u></a></p>
<p class="MsoNormal"><b><span style="font-size:11pt;font-family:calibri,sans-serif">From:</span></b><span style="font-size:11pt;font-family:calibri,sans-serif"> Libfabric-users [mailto:<a href="mailto:libfabric-users-bounces@lists.openfabrics.org" target="_blank">libfabric-users-bounce<wbr>s@lists.openfabrics.org</a>]
<b>On Behalf Of </b>Valentino Picotti<br>
<b>Sent:</b> Wednesday, September 07, 2016 7:48 AM<br>
<b>To:</b> <a href="mailto:libfabric-users@lists.openfabrics.org" target="_blank">
libfabric-users@lists.openfabr<wbr>ics.org</a><br>
<b>Subject:</b> [libfabric-users] Optimisation Tips for verbs provider</span><u></u><u></u></p>
<div>
<div>
<p class="MsoNormal"> <u></u><u></u></p>
<div>
<p class="MsoNormal">Hi all,<u></u><u></u></p>
<div>
<p class="MsoNormal"> <u></u><u></u></p>
</div>
<div>
<p class="MsoNormal">I apologies in advance for the long email.<u></u><u></u></p>
</div>
<div>
<p class="MsoNormal"> <u></u><u></u></p>
</div>
<div>
<p class="MsoNormal">In the past month I've integrated libfabric in a project based on infiniband verbs with the aim to be provider independent. This project has a transport layer that makes the application
independent from the transport implementation (that is chosen at compile time).<u></u><u></u></p>
</div>
<div>
<p class="MsoNormal">I worked only on the libfabric implementation of the transport layer and this was my first experience with RDMA APIs and hardware. What I did was to map the various ibv_* and rdma_*
calls to fi_* calls and I got a working layer quite easily (after studying the libfabric terminology).<u></u><u></u></p>
</div>
<div>
<p class="MsoNormal">Now I'm trying to achieve the same performance of raw verbs.<u></u><u></u></p>
</div>
<div>
<p class="MsoNormal">I'm testing the transport layer with a one sided communication where a client sends the data to a server with the message API(fi_send/fi_recv). The client and the server run on
two different nodes connected with one IB EDR link: i don't set processor affinity nor change power management policy. The depth of completion queues and the size of sent buffers are the same across the tests.<u></u><u></u></p>
</div>
<div>
<p class="MsoNormal">Running on the verbs transport layer I get a stable bandwidth of 22 Gbps, instead with libfabric over verbs I get a very floating bandwidth: from 0.4 Gbps to 19 Gbps in the same
test[1]. The bandwidth is calculated as the number of buffers sent every 5 seconds.<u></u><u></u></p>
</div>
<div>
<p class="MsoNormal"> <u></u><u></u></p>
</div>
<div>
<p class="MsoNormal">This is how i setup the verbs provider:<u></u><u></u></p>
</div>
<p class="MsoNormal"><br>
m_hints->caps = FI_MSG;<br>
m_hints->mode = FI_LOCAL_MR;<br>
m_hints->ep_attr->type = FI_EP_MSG;<br>
m_hints->domain_attr->threadin<wbr>g = FI_THREAD_COMPLETION;<br>
m_hints->domain_attr->data_pro<wbr>gress = FI_PROGRESS_MANUAL;<br>
m_hints->domain_attr->resource<wbr>_mgmt = FI_RM_DISABLED;<br>
m_hints->fabric_attr->prov_nam<wbr>e = strdup("verbs");<u></u><u></u></p>
<div>
<p class="MsoNormal"> <u></u><u></u></p>
</div>
<div>
<p class="MsoNormal">Furthermore I bind two completion queues to the endpoints: one with FI_SEND flag and the other with FI_RECV.<u></u><u></u></p>
</div>
<div>
<p class="MsoNormal"> <u></u><u></u></p>
</div>
<div>
<p class="MsoNormal">I can't figure out why I'm getting that high variance with libfabric.<u></u><u></u></p>
</div>
<div>
<p class="MsoNormal">Do you have any idea? I'm missing same optimisations tips for the verbs provider?<u></u><u></u></p>
</div>
<div>
<p class="MsoNormal"> <u></u><u></u></p>
</div>
<div>
<p class="MsoNormal">Thanks in advance,<u></u><u></u></p>
</div>
<div>
<p class="MsoNormal"> <u></u><u></u></p>
</div>
<div>
<p class="MsoNormal">Valentino<u></u><u></u></p>
</div>
<div>
<p class="MsoNormal"> <u></u><u></u></p>
</div>
<div>
<p class="MsoNormal"> <u></u><u></u></p>
</div>
<div>
<p class="MsoNormal">[1] Test run with depth queue of 1 and buffer size of 512KB <u></u><u></u></p>
</div>
<div>
<p class="MsoNormal"> <u></u><u></u></p>
</div>
<div>
<p class="MsoNormal">Example of a test output with libfarbic:<u></u><u></u></p>
</div>
<div>
<p style="margin:0in 0in 0.0001pt;background-color:black"><span style="font-size:9pt;font-family:'andale mono',serif;color:rgb(41,249,20)">2016-09-07 - 15:10:56 t_server: INFO: Accepted connection</span><u></u><u></u></p>
<p style="margin:0in 0in 0.0001pt;background-color:black"><span style="font-size:9pt;font-family:'andale mono',serif;color:rgb(41,249,20)">2016-09-07 - 15:10:56 t_server: INFO: Start receiving...</span><u></u><u></u></p>
<p style="margin:0in 0in 0.0001pt;background-color:black"><span style="font-size:9pt;font-family:'andale mono',serif;color:rgb(41,249,20)">2016-09-07 - 15:11:01 t_server: INFO: Bandwith: 8.3324 Gb/s</span><u></u><u></u></p>
<p style="margin:0in 0in 0.0001pt;background-color:black"><span style="font-size:9pt;font-family:'andale mono',serif;color:rgb(41,249,20)">2016-09-07 - 15:11:06 t_server: INFO: Bandwith: 15.831 Gb/s</span><u></u><u></u></p>
<p style="margin:0in 0in 0.0001pt;background-color:black"><span style="font-size:9pt;font-family:'andale mono',serif;color:rgb(41,249,20)">2016-09-07 - 15:11:11 t_server: INFO: Bandwith: 19.1713 Gb/s</span><u></u><u></u></p>
<p style="margin:0in 0in 0.0001pt;background-color:black"><span style="font-size:9pt;font-family:'andale mono',serif;color:rgb(41,249,20)">2016-09-07 - 15:11:16 t_server: INFO: Bandwith: 10.8825 Gb/s</span><u></u><u></u></p>
<p style="margin:0in 0in 0.0001pt;background-color:black"><span style="font-size:9pt;font-family:'andale mono',serif;color:rgb(41,249,20)">2016-09-07 - 15:11:21 t_server: INFO: Bandwith: 8.07991 Gb/s</span><u></u><u></u></p>
<p style="margin:0in 0in 0.0001pt;background-color:black"><span style="font-size:9pt;font-family:'andale mono',serif;color:rgb(41,249,20)">2016-09-07 - 15:11:26 t_server: INFO: Bandwith: 15.4015 Gb/s</span><u></u><u></u></p>
<p style="margin:0in 0in 0.0001pt;background-color:black"><span style="font-size:9pt;font-family:'andale mono',serif;color:rgb(41,249,20)">2016-09-07 - 15:11:31 t_server: INFO: Bandwith: 20.4263 Gb/s</span><u></u><u></u></p>
<p style="margin:0in 0in 0.0001pt;background-color:black"><span style="font-size:9pt;font-family:'andale mono',serif;color:rgb(41,249,20)">2016-09-07 - 15:11:36 t_server: INFO: Bandwith: 19.7023 Gb/s</span><u></u><u></u></p>
<p style="margin:0in 0in 0.0001pt;background-color:black"><span style="font-size:9pt;font-family:'andale mono',serif;color:rgb(41,249,20)">2016-09-07 - 15:11:41 t_server: INFO: Bandwith: 10.474 Gb/s</span><u></u><u></u></p>
<p style="margin:0in 0in 0.0001pt;background-color:black"><span style="font-size:9pt;font-family:'andale mono',serif;color:rgb(41,249,20)">2016-09-07 - 15:11:46 t_server: INFO: Bandwith: 17.4072 Gb/s</span><u></u><u></u></p>
<p style="margin:0in 0in 0.0001pt;background-color:black"><span style="font-size:9pt;font-family:'andale mono',serif;color:rgb(41,249,20)">2016-09-07 - 15:11:51 t_server: INFO: Bandwith: 0.440402 Gb/s</span><u></u><u></u></p>
<p style="margin:0in 0in 0.0001pt;background-color:black"><span style="font-size:9pt;font-family:'andale mono',serif;color:rgb(41,249,20)">2016-09-07 - 15:11:56 t_server: INFO: Bandwith: 2.73217 Gb/s</span><u></u><u></u></p>
<p style="margin:0in 0in 0.0001pt;background-color:black"><span style="font-size:9pt;font-family:'andale mono',serif;color:rgb(41,249,20)">2016-09-07 - 15:12:01 t_server: INFO: Bandwith: 0.984822 Gb/s</span><u></u><u></u></p>
<p style="margin:0in 0in 0.0001pt;background-color:black"><span style="font-size:9pt;font-family:'andale mono',serif;color:rgb(41,249,20)">2016-09-07 - 15:12:06 t_server: INFO: Bandwith: 2.93013 Gb/s</span><u></u><u></u></p>
<p style="margin:0in 0in 0.0001pt;background-color:black"><span style="font-size:9pt;font-family:'andale mono',serif;color:rgb(41,249,20)">2016-09-07 - 15:12:11 t_server: INFO: Bandwith: 0.847248 Gb/s</span><u></u><u></u></p>
<p style="margin:0in 0in 0.0001pt;background-color:black"><span style="font-size:9pt;font-family:'andale mono',serif;color:rgb(41,249,20)">2016-09-07 - 15:12:16 t_server: INFO: Bandwith: 7.72255 Gb/s</span><u></u><u></u></p>
<p style="margin:0in 0in 0.0001pt;background-color:black"><span style="font-size:9pt;font-family:'andale mono',serif;color:rgb(41,249,20)">2016-09-07 - 15:12:21 t_server: INFO: Bandwith: 14.7849 Gb/s</span><u></u><u></u></p>
<p style="margin:0in 0in 0.0001pt;background-color:black"><span style="font-size:9pt;font-family:'andale mono',serif;color:rgb(41,249,20)">2016-09-07 - 15:12:26 t_server: INFO: Bandwith: 12.9243 Gb/s</span><u></u><u></u></p>
<p style="margin:0in 0in 0.0001pt;background-color:black"><span style="font-size:9pt;font-family:'andale mono',serif;color:rgb(41,249,20)">2016-09-07 - 15:12:31 t_server: INFO: Bandwith: 0.687027 Gb/s</span><u></u><u></u></p>
<p style="margin:0in 0in 0.0001pt;background-color:black"><span style="font-size:9pt;font-family:'andale mono',serif;color:rgb(41,249,20)">2016-09-07 - 15:12:36 t_server: INFO: Bandwith: 1.44787 Gb/s</span><u></u><u></u></p>
<p style="margin:0in 0in 0.0001pt;background-color:black"><span style="font-size:9pt;font-family:'andale mono',serif;color:rgb(41,249,20)">2016-09-07 - 15:12:41 t_server: INFO: Bandwith: 2.681 Gb/s</span><u></u><u></u></p>
</div>
<div>
<p class="MsoNormal"> <u></u><u></u></p>
</div>
<div>
<p class="MsoNormal">Example of a test output with raw verbs:<u></u><u></u></p>
</div>
<div>
<p style="margin:0in 0in 0.0001pt;background-color:black"><span style="font-size:9pt;font-family:'andale mono',serif;color:rgb(41,249,20)">2016-09-07 - 16:36:00 t_server: INFO: Accepted connection</span><u></u><u></u></p>
<p style="margin:0in 0in 0.0001pt;background-color:black"><span style="font-size:9pt;font-family:'andale mono',serif;color:rgb(41,249,20)">2016-09-07 - 16:36:00 t_server: INFO: Start receiving...</span><u></u><u></u></p>
<p style="margin:0in 0in 0.0001pt;background-color:black"><span style="font-size:9pt;font-family:'andale mono',serif;color:rgb(41,249,20)">2016-09-07 - 16:36:05 t_server: INFO: Bandwith: 17.9491 Gb/s</span><u></u><u></u></p>
<p style="margin:0in 0in 0.0001pt;background-color:black"><span style="font-size:9pt;font-family:'andale mono',serif;color:rgb(41,249,20)">2016-09-07 - 16:36:10 t_server: INFO: Bandwith: 23.4671 Gb/s</span><u></u><u></u></p>
<p style="margin:0in 0in 0.0001pt;background-color:black"><span style="font-size:9pt;font-family:'andale mono',serif;color:rgb(41,249,20)">2016-09-07 - 16:36:15 t_server: INFO: Bandwith: 23.0368 Gb/s</span><u></u><u></u></p>
<p style="margin:0in 0in 0.0001pt;background-color:black"><span style="font-size:9pt;font-family:'andale mono',serif;color:rgb(41,249,20)">2016-09-07 - 16:36:20 t_server: INFO: Bandwith: 22.9638 Gb/s</span><u></u><u></u></p>
<p style="margin:0in 0in 0.0001pt;background-color:black"><span style="font-size:9pt;font-family:'andale mono',serif;color:rgb(41,249,20)">2016-09-07 - 16:36:25 t_server: INFO: Bandwith: 22.8203 Gb/s</span><u></u><u></u></p>
<p style="margin:0in 0in 0.0001pt;background-color:black"><span style="font-size:9pt;font-family:'andale mono',serif;color:rgb(41,249,20)">2016-09-07 - 16:36:30 t_server: INFO: Bandwith: 20.058 Gb/s</span><u></u><u></u></p>
<p style="margin:0in 0in 0.0001pt;background-color:black"><span style="font-size:9pt;font-family:'andale mono',serif;color:rgb(41,249,20)">2016-09-07 - 16:36:35 t_server: INFO: Bandwith: 22.5033 Gb/s</span><u></u><u></u></p>
<p style="margin:0in 0in 0.0001pt;background-color:black"><span style="font-size:9pt;font-family:'andale mono',serif;color:rgb(41,249,20)">2016-09-07 - 16:36:40 t_server: INFO: Bandwith: 20.1754 Gb/s</span><u></u><u></u></p>
<p style="margin:0in 0in 0.0001pt;background-color:black"><span style="font-size:9pt;font-family:'andale mono',serif;color:rgb(41,249,20)">2016-09-07 - 16:36:45 t_server: INFO: Bandwith: 22.5578 Gb/s</span><u></u><u></u></p>
<p style="margin:0in 0in 0.0001pt;background-color:black"><span style="font-size:9pt;font-family:'andale mono',serif;color:rgb(41,249,20)">2016-09-07 - 16:36:50 t_server: INFO: Bandwith: 20.0588 Gb/s</span><u></u><u></u></p>
<p style="margin:0in 0in 0.0001pt;background-color:black"><span style="font-size:9pt;font-family:'andale mono',serif;color:rgb(41,249,20)">2016-09-07 - 16:36:55 t_server: INFO: Bandwith: 22.2718 Gb/s</span><u></u><u></u></p>
<p style="margin:0in 0in 0.0001pt;background-color:black"><span style="font-size:9pt;font-family:'andale mono',serif;color:rgb(41,249,20)">2016-09-07 - 16:37:00 t_server: INFO: Bandwith: 22.494 Gb/s</span><u></u><u></u></p>
<p style="margin:0in 0in 0.0001pt;background-color:black"><span style="font-size:9pt;font-family:'andale mono',serif;color:rgb(41,249,20)">2016-09-07 - 16:37:05 t_server: INFO: Bandwith: 23.1836 Gb/s</span><u></u><u></u></p>
<p style="margin:0in 0in 0.0001pt;background-color:black"><span style="font-size:9pt;font-family:'andale mono',serif;color:rgb(41,249,20)">2016-09-07 - 16:37:10 t_server: INFO: Bandwith: 23.0972 Gb/s</span><u></u><u></u></p>
<p style="margin:0in 0in 0.0001pt;background-color:black"><span style="font-size:9pt;font-family:'andale mono',serif;color:rgb(41,249,20)">2016-09-07 - 16:37:15 t_server: INFO: Bandwith: 21.5033 Gb/s</span><u></u><u></u></p>
<p style="margin:0in 0in 0.0001pt;background-color:black"><span style="font-size:9pt;font-family:'andale mono',serif;color:rgb(41,249,20)">2016-09-07 - 16:37:20 t_server: INFO: Bandwith: 18.5506 Gb/s</span><u></u><u></u></p>
<p style="margin:0in 0in 0.0001pt;background-color:black"><span style="font-size:9pt;font-family:'andale mono',serif;color:rgb(41,249,20)">2016-09-07 - 16:37:25 t_server: INFO: Bandwith: 20.3709 Gb/s</span><u></u><u></u></p>
<p style="margin:0in 0in 0.0001pt;background-color:black"><span style="font-size:9pt;font-family:'andale mono',serif;color:rgb(41,249,20)">2016-09-07 - 16:37:30 t_server: INFO: Bandwith: 21.3457 Gb/s</span><u></u><u></u></p>
<p style="margin:0in 0in 0.0001pt;background-color:black"><span style="font-size:9pt;font-family:'andale mono',serif;color:rgb(41,249,20)">2016-09-07 - 16:37:35 t_server: INFO: Bandwith: 20.5059 Gb/s</span><u></u><u></u></p>
<p style="margin:0in 0in 0.0001pt;background-color:black"><span style="font-size:9pt;font-family:'andale mono',serif;color:rgb(41,249,20)">2016-09-07 - 16:37:40 t_server: INFO: Bandwith: 22.4899 Gb/s</span><u></u><u></u></p>
<p style="margin:0in 0in 0.0001pt;background-color:black"><span style="font-size:9pt;font-family:'andale mono',serif;color:rgb(41,249,20)">2016-09-07 - 16:37:45 t_server: INFO: Bandwith: 22.1266 Gb/s</span><u></u><u></u></p>
<p style="margin:0in 0in 0.0001pt;background-color:black"><span style="font-size:9pt;font-family:'andale mono',serif;color:rgb(41,249,20)">2016-09-07 - 16:37:50 t_server: INFO: Bandwith: 22.4504 Gb/s</span><u></u><u></u></p>
</div>
<div>
<p class="MsoNormal"> <u></u><u></u></p>
</div>
<div>
<p class="MsoNormal"> <u></u><u></u></p>
</div>
</div>
</div>
</div>
</div>
</div>
</blockquote>
</div>
<p class="MsoNormal"><u></u> <u></u></p>
</div>
</div>
</div>
</div>
<p class="MsoNormal"><u></u> <u></u></p>
</div>
</div>
</div></div></div>
</div>
</blockquote></div><br></div></div></div>