<html xmlns:v="urn:schemas-microsoft-com:vml" xmlns:o="urn:schemas-microsoft-com:office:office" xmlns:w="urn:schemas-microsoft-com:office:word" xmlns:m="http://schemas.microsoft.com/office/2004/12/omml" xmlns="http://www.w3.org/TR/REC-html40">
<head>
<meta http-equiv="Content-Type" content="text/html; charset=utf-8">
<meta name="Generator" content="Microsoft Word 15 (filtered medium)">
<style><!--
/* Font Definitions */
@font-face
{font-family:"Cambria Math";
panose-1:2 4 5 3 5 4 6 3 2 4;}
@font-face
{font-family:Calibri;
panose-1:2 15 5 2 2 2 4 3 2 4;}
@font-face
{font-family:"andale mono";
panose-1:0 0 0 0 0 0 0 0 0 0;}
/* Style Definitions */
p.MsoNormal, li.MsoNormal, div.MsoNormal
{margin:0in;
margin-bottom:.0001pt;
font-size:12.0pt;
font-family:"Times New Roman",serif;}
a:link, span.MsoHyperlink
{mso-style-priority:99;
color:blue;
text-decoration:underline;}
a:visited, span.MsoHyperlinkFollowed
{mso-style-priority:99;
color:purple;
text-decoration:underline;}
p
{mso-style-priority:99;
mso-margin-top-alt:auto;
margin-right:0in;
mso-margin-bottom-alt:auto;
margin-left:0in;
font-size:12.0pt;
font-family:"Times New Roman",serif;}
span.EmailStyle18
{mso-style-type:personal-reply;
font-family:"Calibri",sans-serif;
color:#1F497D;}
.MsoChpDefault
{mso-style-type:export-only;
font-family:"Calibri",sans-serif;}
@page WordSection1
{size:8.5in 11.0in;
margin:1.0in 1.0in 1.0in 1.0in;}
div.WordSection1
{page:WordSection1;}
--></style><!--[if gte mso 9]><xml>
<o:shapedefaults v:ext="edit" spidmax="1026" />
</xml><![endif]--><!--[if gte mso 9]><xml>
<o:shapelayout v:ext="edit">
<o:idmap v:ext="edit" data="1" />
</o:shapelayout></xml><![endif]-->
</head>
<body lang="EN-US" link="blue" vlink="purple">
<div class="WordSection1">
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:"Calibri",sans-serif;color:#1F497D">Valentino,<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:"Calibri",sans-serif;color:#1F497D"><o:p> </o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:"Calibri",sans-serif;color:#1F497D">You can run the fi_msg_bw test with other completion-waiting options like sread and fd too. I don’t get any variance when using them as well. The test calls fi_cq_read
after posting every window size number of sends/recvs. The window size is an adjustable parameter. You can view all available options by calling fi_msg_bw -h.<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:"Calibri",sans-serif;color:#1F497D"><o:p> </o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:"Calibri",sans-serif;color:#1F497D">May be you could try posting a bunch of sends/recvs and then collect the completions for the bunch. Is there a need to post the messages one by one? If that’s
the case please try using spin wait for getting the completion. But even that might not guarantee consistent numbers. Currently, there is an issue of unpredictable transfer times if the sender overruns the receiver. Posting recvs in advance can avoid this
issue.<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:"Calibri",sans-serif;color:#1F497D"><o:p> </o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:"Calibri",sans-serif;color:#1F497D">-Arun.<o:p></o:p></span></p>
<p class="MsoNormal"><a name="_MailEndCompose"><span style="font-size:11.0pt;font-family:"Calibri",sans-serif;color:#1F497D"><o:p> </o:p></span></a></p>
<p class="MsoNormal"><a name="_____replyseparator"></a><b><span style="font-size:11.0pt;font-family:"Calibri",sans-serif">From:</span></b><span style="font-size:11.0pt;font-family:"Calibri",sans-serif"> Libfabric-users [mailto:libfabric-users-bounces@lists.openfabrics.org]
<b>On Behalf Of </b>Valentino Picotti<br>
<b>Sent:</b> Thursday, September 08, 2016 5:12 AM<br>
<b>To:</b> libfabric-users@lists.openfabrics.org<br>
<b>Subject:</b> [libfabric-users] Fwd: Optimisation Tips for verbs provider<o:p></o:p></span></p>
<p class="MsoNormal"><o:p> </o:p></p>
<div>
<p class="MsoNormal">I forgot to CC the list, here is my reply to Arun:<o:p></o:p></p>
<div>
<p class="MsoNormal"><o:p> </o:p></p>
<div>
<p class="MsoNormal" style="margin-bottom:12.0pt">---------- Forwarded message ----------<br>
From: <b>Valentino Picotti</b> <<a href="mailto:valentino.picotti@gmail.com">valentino.picotti@gmail.com</a>><br>
Date: 8 September 2016 at 14:00<br>
Subject: Re: [libfabric-users] Optimisation Tips for verbs provider<br>
To: "Ilango, Arun" <<a href="mailto:arun.ilango@intel.com">arun.ilango@intel.com</a>><br>
<br>
<o:p></o:p></p>
<div>
<p class="MsoNormal">Thanks for the reply,<o:p></o:p></p>
<div>
<p class="MsoNormal"><o:p> </o:p></p>
</div>
<div>
<p class="MsoNormal">I run the fi_msg_bw with CQ size and window size of 1 and I got the following result:<o:p></o:p></p>
</div>
<div>
<p style="margin:0in;margin-bottom:.0001pt;background:black"><span style="font-size:9.0pt;font-family:"andale mono",serif;color:#29F914">bytes iters total time MB/sec usec/xfer Mxfers/sec<o:p></o:p></span></p>
<p style="margin:0in;margin-bottom:.0001pt;background:black"><span style="font-size:9.0pt;font-family:"andale mono",serif;color:#29F914">512k 1m 488g 189.50s 2766.76 189.50 0.01<o:p></o:p></span></p>
</div>
<div>
<p class="MsoNormal">21,61 Gbps is an excellent result. I'm using libfabric 1.3.0 from the latest tarball. <o:p></o:p></p>
</div>
<div>
<p class="MsoNormal"><o:p> </o:p></p>
</div>
<div>
<p class="MsoNormal">So the problem is in my transport layer.<o:p></o:p></p>
</div>
<div>
<p class="MsoNormal">My fabric initialisation doesn't differs too much from the fi_msg_bw one, so the problem might be in the main loop.<o:p></o:p></p>
</div>
<div>
<p class="MsoNormal">At a first glance, It seems that i call fi_cq_read less often than the bw test.<o:p></o:p></p>
</div>
<div>
<p class="MsoNormal">In the test the sequence is:<o:p></o:p></p>
</div>
<div>
<p class="MsoNormal">- post work request ft_post_tx/rx<o:p></o:p></p>
</div>
<div>
<p class="MsoNormal">- spin on a completion bw_(tx/rx)_comp<o:p></o:p></p>
</div>
<div>
<p class="MsoNormal"><o:p> </o:p></p>
</div>
<div>
<p class="MsoNormal">In my client/server main loop:<o:p></o:p></p>
</div>
<div>
<p class="MsoNormal">- call fi_cq_read<o:p></o:p></p>
</div>
<div>
<p class="MsoNormal">- post work request<o:p></o:p></p>
</div>
<div>
<p class="MsoNormal"><o:p> </o:p></p>
</div>
<div>
<p class="MsoNormal">I don't spin waiting for completions, could this be the reason?<o:p></o:p></p>
</div>
<div>
<p class="MsoNormal"><o:p> </o:p></p>
</div>
<div>
<p class="MsoNormal">Thanks,<o:p></o:p></p>
</div>
<div>
<p class="MsoNormal">Valentino<o:p></o:p></p>
</div>
<div>
<p class="MsoNormal"><o:p> </o:p></p>
</div>
</div>
<div>
<div>
<div>
<p class="MsoNormal"><o:p> </o:p></p>
<div>
<p class="MsoNormal">On 7 September 2016 at 19:17, Ilango, Arun <<a href="mailto:arun.ilango@intel.com" target="_blank">arun.ilango@intel.com</a>> wrote:<o:p></o:p></p>
<blockquote style="border:none;border-left:solid #CCCCCC 1.0pt;padding:0in 0in 0in 6.0pt;margin-left:4.8pt;margin-right:0in">
<div>
<div>
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto"><span style="font-size:11.0pt;font-family:"Calibri",sans-serif;color:#1F497D">Hi Valentino,</span><o:p></o:p></p>
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto"><span style="font-size:11.0pt;font-family:"Calibri",sans-serif;color:#1F497D"> </span><o:p></o:p></p>
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto"><span style="font-size:11.0pt;font-family:"Calibri",sans-serif;color:#1F497D">Libfabric has a set of tests available at
<a href="https://github.com/ofiwg/fabtests" target="_blank">https://github.com/ofiwg/fabtests</a>. Can you run the fi_msg_bw test with the same size and iterations on your setup and check if you notice any variance? Also what version/commit number of libfabric
are you using?</span><o:p></o:p></p>
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto"><span style="font-size:11.0pt;font-family:"Calibri",sans-serif;color:#1F497D"> </span><o:p></o:p></p>
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto"><span style="font-size:11.0pt;font-family:"Calibri",sans-serif;color:#1F497D">Thanks,</span><o:p></o:p></p>
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto"><span style="font-size:11.0pt;font-family:"Calibri",sans-serif;color:#1F497D">Arun.</span><o:p></o:p></p>
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto"><a name="m_6693636125052867833_m_6391039724988073"><span style="font-size:11.0pt;font-family:"Calibri",sans-serif;color:#1F497D"> </span><o:p></o:p></a></p>
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto"><b><span style="font-size:11.0pt;font-family:"Calibri",sans-serif">From:</span></b><span style="font-size:11.0pt;font-family:"Calibri",sans-serif"> Libfabric-users [mailto:<a href="mailto:libfabric-users-bounces@lists.openfabrics.org" target="_blank">libfabric-users-bounces@lists.openfabrics.org</a>]
<b>On Behalf Of </b>Valentino Picotti<br>
<b>Sent:</b> Wednesday, September 07, 2016 7:48 AM<br>
<b>To:</b> <a href="mailto:libfabric-users@lists.openfabrics.org" target="_blank">
libfabric-users@lists.openfabrics.org</a><br>
<b>Subject:</b> [libfabric-users] Optimisation Tips for verbs provider</span><o:p></o:p></p>
<div>
<div>
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto"> <o:p></o:p></p>
<div>
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto">Hi all,<o:p></o:p></p>
<div>
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto"> <o:p></o:p></p>
</div>
<div>
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto">I apologies in advance for the long email.<o:p></o:p></p>
</div>
<div>
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto"> <o:p></o:p></p>
</div>
<div>
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto">In the past month I've integrated libfabric in a project based on infiniband verbs with the aim to be provider independent. This project has a transport layer that makes the application
independent from the transport implementation (that is chosen at compile time).<o:p></o:p></p>
</div>
<div>
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto">I worked only on the libfabric implementation of the transport layer and this was my first experience with RDMA APIs and hardware. What I did was to map the various ibv_* and rdma_*
calls to fi_* calls and I got a working layer quite easily (after studying the libfabric terminology).<o:p></o:p></p>
</div>
<div>
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto">Now I'm trying to achieve the same performance of raw verbs.<o:p></o:p></p>
</div>
<div>
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto">I'm testing the transport layer with a one sided communication where a client sends the data to a server with the message API(fi_send/fi_recv). The client and the server run on
two different nodes connected with one IB EDR link: i don't set processor affinity nor change power management policy. The depth of completion queues and the size of sent buffers are the same across the tests.<o:p></o:p></p>
</div>
<div>
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto">Running on the verbs transport layer I get a stable bandwidth of 22 Gbps, instead with libfabric over verbs I get a very floating bandwidth: from 0.4 Gbps to 19 Gbps in the same
test[1]. The bandwidth is calculated as the number of buffers sent every 5 seconds.<o:p></o:p></p>
</div>
<div>
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto"> <o:p></o:p></p>
</div>
<div>
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto">This is how i setup the verbs provider:<o:p></o:p></p>
</div>
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto"><br>
m_hints->caps = FI_MSG;<br>
m_hints->mode = FI_LOCAL_MR;<br>
m_hints->ep_attr->type = FI_EP_MSG;<br>
m_hints->domain_attr->threading = FI_THREAD_COMPLETION;<br>
m_hints->domain_attr->data_progress = FI_PROGRESS_MANUAL;<br>
m_hints->domain_attr->resource_mgmt = FI_RM_DISABLED;<br>
m_hints->fabric_attr->prov_name = strdup("verbs");<o:p></o:p></p>
<div>
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto"> <o:p></o:p></p>
</div>
<div>
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto">Furthermore I bind two completion queues to the endpoints: one with FI_SEND flag and the other with FI_RECV.<o:p></o:p></p>
</div>
<div>
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto"> <o:p></o:p></p>
</div>
<div>
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto">I can't figure out why I'm getting that high variance with libfabric.<o:p></o:p></p>
</div>
<div>
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto">Do you have any idea? I'm missing same optimisations tips for the verbs provider?<o:p></o:p></p>
</div>
<div>
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto"> <o:p></o:p></p>
</div>
<div>
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto">Thanks in advance,<o:p></o:p></p>
</div>
<div>
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto"> <o:p></o:p></p>
</div>
<div>
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto">Valentino<o:p></o:p></p>
</div>
<div>
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto"> <o:p></o:p></p>
</div>
<div>
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto"> <o:p></o:p></p>
</div>
<div>
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto">[1] Test run with depth queue of 1 and buffer size of 512KB <o:p></o:p></p>
</div>
<div>
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto"> <o:p></o:p></p>
</div>
<div>
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto">Example of a test output with libfarbic:<o:p></o:p></p>
</div>
<div>
<p style="margin:0in;margin-bottom:.0001pt;background:black"><span style="font-size:9.0pt;font-family:"andale mono",serif;color:#29F914">2016-09-07 - 15:10:56 t_server: INFO: Accepted connection</span><o:p></o:p></p>
<p style="margin:0in;margin-bottom:.0001pt;background:black"><span style="font-size:9.0pt;font-family:"andale mono",serif;color:#29F914">2016-09-07 - 15:10:56 t_server: INFO: Start receiving...</span><o:p></o:p></p>
<p style="margin:0in;margin-bottom:.0001pt;background:black"><span style="font-size:9.0pt;font-family:"andale mono",serif;color:#29F914">2016-09-07 - 15:11:01 t_server: INFO: Bandwith: 8.3324 Gb/s</span><o:p></o:p></p>
<p style="margin:0in;margin-bottom:.0001pt;background:black"><span style="font-size:9.0pt;font-family:"andale mono",serif;color:#29F914">2016-09-07 - 15:11:06 t_server: INFO: Bandwith: 15.831 Gb/s</span><o:p></o:p></p>
<p style="margin:0in;margin-bottom:.0001pt;background:black"><span style="font-size:9.0pt;font-family:"andale mono",serif;color:#29F914">2016-09-07 - 15:11:11 t_server: INFO: Bandwith: 19.1713 Gb/s</span><o:p></o:p></p>
<p style="margin:0in;margin-bottom:.0001pt;background:black"><span style="font-size:9.0pt;font-family:"andale mono",serif;color:#29F914">2016-09-07 - 15:11:16 t_server: INFO: Bandwith: 10.8825 Gb/s</span><o:p></o:p></p>
<p style="margin:0in;margin-bottom:.0001pt;background:black"><span style="font-size:9.0pt;font-family:"andale mono",serif;color:#29F914">2016-09-07 - 15:11:21 t_server: INFO: Bandwith: 8.07991 Gb/s</span><o:p></o:p></p>
<p style="margin:0in;margin-bottom:.0001pt;background:black"><span style="font-size:9.0pt;font-family:"andale mono",serif;color:#29F914">2016-09-07 - 15:11:26 t_server: INFO: Bandwith: 15.4015 Gb/s</span><o:p></o:p></p>
<p style="margin:0in;margin-bottom:.0001pt;background:black"><span style="font-size:9.0pt;font-family:"andale mono",serif;color:#29F914">2016-09-07 - 15:11:31 t_server: INFO: Bandwith: 20.4263 Gb/s</span><o:p></o:p></p>
<p style="margin:0in;margin-bottom:.0001pt;background:black"><span style="font-size:9.0pt;font-family:"andale mono",serif;color:#29F914">2016-09-07 - 15:11:36 t_server: INFO: Bandwith: 19.7023 Gb/s</span><o:p></o:p></p>
<p style="margin:0in;margin-bottom:.0001pt;background:black"><span style="font-size:9.0pt;font-family:"andale mono",serif;color:#29F914">2016-09-07 - 15:11:41 t_server: INFO: Bandwith: 10.474 Gb/s</span><o:p></o:p></p>
<p style="margin:0in;margin-bottom:.0001pt;background:black"><span style="font-size:9.0pt;font-family:"andale mono",serif;color:#29F914">2016-09-07 - 15:11:46 t_server: INFO: Bandwith: 17.4072 Gb/s</span><o:p></o:p></p>
<p style="margin:0in;margin-bottom:.0001pt;background:black"><span style="font-size:9.0pt;font-family:"andale mono",serif;color:#29F914">2016-09-07 - 15:11:51 t_server: INFO: Bandwith: 0.440402 Gb/s</span><o:p></o:p></p>
<p style="margin:0in;margin-bottom:.0001pt;background:black"><span style="font-size:9.0pt;font-family:"andale mono",serif;color:#29F914">2016-09-07 - 15:11:56 t_server: INFO: Bandwith: 2.73217 Gb/s</span><o:p></o:p></p>
<p style="margin:0in;margin-bottom:.0001pt;background:black"><span style="font-size:9.0pt;font-family:"andale mono",serif;color:#29F914">2016-09-07 - 15:12:01 t_server: INFO: Bandwith: 0.984822 Gb/s</span><o:p></o:p></p>
<p style="margin:0in;margin-bottom:.0001pt;background:black"><span style="font-size:9.0pt;font-family:"andale mono",serif;color:#29F914">2016-09-07 - 15:12:06 t_server: INFO: Bandwith: 2.93013 Gb/s</span><o:p></o:p></p>
<p style="margin:0in;margin-bottom:.0001pt;background:black"><span style="font-size:9.0pt;font-family:"andale mono",serif;color:#29F914">2016-09-07 - 15:12:11 t_server: INFO: Bandwith: 0.847248 Gb/s</span><o:p></o:p></p>
<p style="margin:0in;margin-bottom:.0001pt;background:black"><span style="font-size:9.0pt;font-family:"andale mono",serif;color:#29F914">2016-09-07 - 15:12:16 t_server: INFO: Bandwith: 7.72255 Gb/s</span><o:p></o:p></p>
<p style="margin:0in;margin-bottom:.0001pt;background:black"><span style="font-size:9.0pt;font-family:"andale mono",serif;color:#29F914">2016-09-07 - 15:12:21 t_server: INFO: Bandwith: 14.7849 Gb/s</span><o:p></o:p></p>
<p style="margin:0in;margin-bottom:.0001pt;background:black"><span style="font-size:9.0pt;font-family:"andale mono",serif;color:#29F914">2016-09-07 - 15:12:26 t_server: INFO: Bandwith: 12.9243 Gb/s</span><o:p></o:p></p>
<p style="margin:0in;margin-bottom:.0001pt;background:black"><span style="font-size:9.0pt;font-family:"andale mono",serif;color:#29F914">2016-09-07 - 15:12:31 t_server: INFO: Bandwith: 0.687027 Gb/s</span><o:p></o:p></p>
<p style="margin:0in;margin-bottom:.0001pt;background:black"><span style="font-size:9.0pt;font-family:"andale mono",serif;color:#29F914">2016-09-07 - 15:12:36 t_server: INFO: Bandwith: 1.44787 Gb/s</span><o:p></o:p></p>
<p style="margin:0in;margin-bottom:.0001pt;background:black"><span style="font-size:9.0pt;font-family:"andale mono",serif;color:#29F914">2016-09-07 - 15:12:41 t_server: INFO: Bandwith: 2.681 Gb/s</span><o:p></o:p></p>
</div>
<div>
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto"> <o:p></o:p></p>
</div>
<div>
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto">Example of a test output with raw verbs:<o:p></o:p></p>
</div>
<div>
<p style="margin:0in;margin-bottom:.0001pt;background:black"><span style="font-size:9.0pt;font-family:"andale mono",serif;color:#29F914">2016-09-07 - 16:36:00 t_server: INFO: Accepted connection</span><o:p></o:p></p>
<p style="margin:0in;margin-bottom:.0001pt;background:black"><span style="font-size:9.0pt;font-family:"andale mono",serif;color:#29F914">2016-09-07 - 16:36:00 t_server: INFO: Start receiving...</span><o:p></o:p></p>
<p style="margin:0in;margin-bottom:.0001pt;background:black"><span style="font-size:9.0pt;font-family:"andale mono",serif;color:#29F914">2016-09-07 - 16:36:05 t_server: INFO: Bandwith: 17.9491 Gb/s</span><o:p></o:p></p>
<p style="margin:0in;margin-bottom:.0001pt;background:black"><span style="font-size:9.0pt;font-family:"andale mono",serif;color:#29F914">2016-09-07 - 16:36:10 t_server: INFO: Bandwith: 23.4671 Gb/s</span><o:p></o:p></p>
<p style="margin:0in;margin-bottom:.0001pt;background:black"><span style="font-size:9.0pt;font-family:"andale mono",serif;color:#29F914">2016-09-07 - 16:36:15 t_server: INFO: Bandwith: 23.0368 Gb/s</span><o:p></o:p></p>
<p style="margin:0in;margin-bottom:.0001pt;background:black"><span style="font-size:9.0pt;font-family:"andale mono",serif;color:#29F914">2016-09-07 - 16:36:20 t_server: INFO: Bandwith: 22.9638 Gb/s</span><o:p></o:p></p>
<p style="margin:0in;margin-bottom:.0001pt;background:black"><span style="font-size:9.0pt;font-family:"andale mono",serif;color:#29F914">2016-09-07 - 16:36:25 t_server: INFO: Bandwith: 22.8203 Gb/s</span><o:p></o:p></p>
<p style="margin:0in;margin-bottom:.0001pt;background:black"><span style="font-size:9.0pt;font-family:"andale mono",serif;color:#29F914">2016-09-07 - 16:36:30 t_server: INFO: Bandwith: 20.058 Gb/s</span><o:p></o:p></p>
<p style="margin:0in;margin-bottom:.0001pt;background:black"><span style="font-size:9.0pt;font-family:"andale mono",serif;color:#29F914">2016-09-07 - 16:36:35 t_server: INFO: Bandwith: 22.5033 Gb/s</span><o:p></o:p></p>
<p style="margin:0in;margin-bottom:.0001pt;background:black"><span style="font-size:9.0pt;font-family:"andale mono",serif;color:#29F914">2016-09-07 - 16:36:40 t_server: INFO: Bandwith: 20.1754 Gb/s</span><o:p></o:p></p>
<p style="margin:0in;margin-bottom:.0001pt;background:black"><span style="font-size:9.0pt;font-family:"andale mono",serif;color:#29F914">2016-09-07 - 16:36:45 t_server: INFO: Bandwith: 22.5578 Gb/s</span><o:p></o:p></p>
<p style="margin:0in;margin-bottom:.0001pt;background:black"><span style="font-size:9.0pt;font-family:"andale mono",serif;color:#29F914">2016-09-07 - 16:36:50 t_server: INFO: Bandwith: 20.0588 Gb/s</span><o:p></o:p></p>
<p style="margin:0in;margin-bottom:.0001pt;background:black"><span style="font-size:9.0pt;font-family:"andale mono",serif;color:#29F914">2016-09-07 - 16:36:55 t_server: INFO: Bandwith: 22.2718 Gb/s</span><o:p></o:p></p>
<p style="margin:0in;margin-bottom:.0001pt;background:black"><span style="font-size:9.0pt;font-family:"andale mono",serif;color:#29F914">2016-09-07 - 16:37:00 t_server: INFO: Bandwith: 22.494 Gb/s</span><o:p></o:p></p>
<p style="margin:0in;margin-bottom:.0001pt;background:black"><span style="font-size:9.0pt;font-family:"andale mono",serif;color:#29F914">2016-09-07 - 16:37:05 t_server: INFO: Bandwith: 23.1836 Gb/s</span><o:p></o:p></p>
<p style="margin:0in;margin-bottom:.0001pt;background:black"><span style="font-size:9.0pt;font-family:"andale mono",serif;color:#29F914">2016-09-07 - 16:37:10 t_server: INFO: Bandwith: 23.0972 Gb/s</span><o:p></o:p></p>
<p style="margin:0in;margin-bottom:.0001pt;background:black"><span style="font-size:9.0pt;font-family:"andale mono",serif;color:#29F914">2016-09-07 - 16:37:15 t_server: INFO: Bandwith: 21.5033 Gb/s</span><o:p></o:p></p>
<p style="margin:0in;margin-bottom:.0001pt;background:black"><span style="font-size:9.0pt;font-family:"andale mono",serif;color:#29F914">2016-09-07 - 16:37:20 t_server: INFO: Bandwith: 18.5506 Gb/s</span><o:p></o:p></p>
<p style="margin:0in;margin-bottom:.0001pt;background:black"><span style="font-size:9.0pt;font-family:"andale mono",serif;color:#29F914">2016-09-07 - 16:37:25 t_server: INFO: Bandwith: 20.3709 Gb/s</span><o:p></o:p></p>
<p style="margin:0in;margin-bottom:.0001pt;background:black"><span style="font-size:9.0pt;font-family:"andale mono",serif;color:#29F914">2016-09-07 - 16:37:30 t_server: INFO: Bandwith: 21.3457 Gb/s</span><o:p></o:p></p>
<p style="margin:0in;margin-bottom:.0001pt;background:black"><span style="font-size:9.0pt;font-family:"andale mono",serif;color:#29F914">2016-09-07 - 16:37:35 t_server: INFO: Bandwith: 20.5059 Gb/s</span><o:p></o:p></p>
<p style="margin:0in;margin-bottom:.0001pt;background:black"><span style="font-size:9.0pt;font-family:"andale mono",serif;color:#29F914">2016-09-07 - 16:37:40 t_server: INFO: Bandwith: 22.4899 Gb/s</span><o:p></o:p></p>
<p style="margin:0in;margin-bottom:.0001pt;background:black"><span style="font-size:9.0pt;font-family:"andale mono",serif;color:#29F914">2016-09-07 - 16:37:45 t_server: INFO: Bandwith: 22.1266 Gb/s</span><o:p></o:p></p>
<p style="margin:0in;margin-bottom:.0001pt;background:black"><span style="font-size:9.0pt;font-family:"andale mono",serif;color:#29F914">2016-09-07 - 16:37:50 t_server: INFO: Bandwith: 22.4504 Gb/s</span><o:p></o:p></p>
</div>
<div>
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto"> <o:p></o:p></p>
</div>
<div>
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto"> <o:p></o:p></p>
</div>
</div>
</div>
</div>
</div>
</div>
</blockquote>
</div>
<p class="MsoNormal"><o:p> </o:p></p>
</div>
</div>
</div>
</div>
<p class="MsoNormal"><o:p> </o:p></p>
</div>
</div>
</div>
</body>
</html>