<div dir="ltr">I use a separate CQ for each endpoint.<div><br></div><div>Thanks,</div><div>Supun..</div></div><div class="gmail_extra"><br><div class="gmail_quote">On Tue, Mar 21, 2017 at 5:03 PM, Ilango, Arun <span dir="ltr"><<a href="mailto:arun.ilango@intel.com" target="_blank">arun.ilango@intel.com</a>></span> wrote:<br><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex">
<div lang="EN-US" link="blue" vlink="purple">
<div class="m_8747884564277293854WordSection1">
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:"Calibri",sans-serif;color:#1f497d">I see. Do you use a common CQ or a CQ per endpoint? We do have a lock in the CQ to serialize access to it.<u></u><u></u></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:"Calibri",sans-serif;color:#1f497d"><u></u> <u></u></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:"Calibri",sans-serif;color:#1f497d">-Arun.<u></u><u></u></span></p>
<p class="MsoNormal"><a name="m_8747884564277293854__MailEndCompose"><span style="font-size:11.0pt;font-family:"Calibri",sans-serif;color:#1f497d"><u></u> <u></u></span></a></p>
<p class="MsoNormal"><a name="m_8747884564277293854______replyseparator"></a><b><span style="font-size:11.0pt;font-family:"Calibri",sans-serif">From:</span></b><span style="font-size:11.0pt;font-family:"Calibri",sans-serif"> Supun Kamburugamuve [mailto:<a href="mailto:skamburugamuve@gmail.com" target="_blank">skamburugamuve@gmail.<wbr>com</a>]
<br>
<b>Sent:</b> Tuesday, March 21, 2017 1:07 PM<br>
<b>To:</b> Ilango, Arun <<a href="mailto:arun.ilango@intel.com" target="_blank">arun.ilango@intel.com</a>><br>
<b>Cc:</b> Hefty, Sean <<a href="mailto:sean.hefty@intel.com" target="_blank">sean.hefty@intel.com</a>>; <a href="mailto:libfabric-users@lists.openfabrics.org" target="_blank">libfabric-users@lists.<wbr>openfabrics.org</a></span></p><div><div class="h5"><br>
<b>Subject:</b> Re: [libfabric-users] Verbs send message slow down<u></u><u></u></div></div><p></p><div><div class="h5">
<p class="MsoNormal"><u></u> <u></u></p>
<div>
<p class="MsoNormal">Thanks, Arun, will try this. This happens with multiple connections, though. The program works when there is a small number of nodes sending data. For example, for 4 nodes it works. For 8 Nodes one or two QP's become slow.<u></u><u></u></p>
<div>
<p class="MsoNormal"><u></u> <u></u></p>
</div>
<div>
<p class="MsoNormal">Supun..<u></u><u></u></p>
</div>
</div>
<div>
<p class="MsoNormal"><u></u> <u></u></p>
<div>
<p class="MsoNormal">On Tue, Mar 21, 2017 at 3:26 PM, Ilango, Arun <<a href="mailto:arun.ilango@intel.com" target="_blank">arun.ilango@intel.com</a>> wrote:<u></u><u></u></p>
<blockquote style="border:none;border-left:solid #cccccc 1.0pt;padding:0in 0in 0in 6.0pt;margin-left:4.8pt;margin-top:5.0pt;margin-right:0in;margin-bottom:5.0pt">
<p class="MsoNormal">Hi Supun,<br>
<br>
I tried to reproduce the issue but I'm not hitting it. I have modified fabtests bandwidth test to read multiple completions (=window_size) at a time on the receive side. I have the patch on the following branch:<br>
<a href="https://github.com/a-ilango/fabtests/tree/comp_debug" target="_blank">https://github.com/a-ilango/<wbr>fabtests/tree/comp_debug</a><br>
<br>
I'm using fi_cq_read. Please try out fi_msg_bw and see if you get the issue.<br>
<br>
Thanks,<br>
Arun.<u></u><u></u></p>
<div>
<div>
<p class="MsoNormal"><br>
-----Original Message-----<br>
From: Libfabric-users [mailto:<a href="mailto:libfabric-users-bounces@lists.openfabrics.org" target="_blank">libfabric-users-<wbr>bounces@lists.openfabrics.org</a>] On Behalf Of Ilango, Arun<br>
Sent: Tuesday, March 21, 2017 11:07 AM<br>
To: Hefty, Sean <<a href="mailto:sean.hefty@intel.com" target="_blank">sean.hefty@intel.com</a>>; Supun Kamburugamuve <<a href="mailto:skamburugamuve@gmail.com" target="_blank">skamburugamuve@gmail.com</a>><br>
Cc: <a href="mailto:libfabric-users@lists.openfabrics.org" target="_blank">libfabric-users@lists.<wbr>openfabrics.org</a><br>
Subject: Re: [libfabric-users] Verbs send message slow down<br>
<br>
This might need some debugging. In the verbs provider, we call ibv_poll_cq with num_entries set to 1 and repeatedly call it when the app has requested multiple completions. I'm not sure if this is the issue but let me check.<br>
<br>
-Arun.<br>
<br>
-----Original Message-----<br>
From: Libfabric-users [mailto:<a href="mailto:libfabric-users-bounces@lists.openfabrics.org" target="_blank">libfabric-users-<wbr>bounces@lists.openfabrics.org</a>] On Behalf Of Hefty, Sean<br>
Sent: Tuesday, March 21, 2017 10:53 AM<br>
To: Supun Kamburugamuve <<a href="mailto:skamburugamuve@gmail.com" target="_blank">skamburugamuve@gmail.com</a>><br>
Cc: <a href="mailto:libfabric-users@lists.openfabrics.org" target="_blank">libfabric-users@lists.<wbr>openfabrics.org</a><br>
Subject: Re: [libfabric-users] Verbs send message slow down<br>
<br>
> I found a workaround to the problem. Previously I was doing cq_read<br>
> for multiple completions. If I do cq_read for a single completion the<br>
> problem goes away. Is this the expected behavior or something wrong<br>
> with my program or a bug in libfabric?<br>
<br>
This is not expected behavior. Something else is going goofy here.<br>
______________________________<wbr>_________________<br>
Libfabric-users mailing list<br>
<a href="mailto:Libfabric-users@lists.openfabrics.org" target="_blank">Libfabric-users@lists.<wbr>openfabrics.org</a><br>
<a href="http://lists.openfabrics.org/mailman/listinfo/libfabric-users" target="_blank">http://lists.openfabrics.org/<wbr>mailman/listinfo/libfabric-<wbr>users</a><br>
______________________________<wbr>_________________<br>
Libfabric-users mailing list<br>
<a href="mailto:Libfabric-users@lists.openfabrics.org" target="_blank">Libfabric-users@lists.<wbr>openfabrics.org</a><br>
<a href="http://lists.openfabrics.org/mailman/listinfo/libfabric-users" target="_blank">http://lists.openfabrics.org/<wbr>mailman/listinfo/libfabric-<wbr>users</a><u></u><u></u></p>
</div>
</div>
</blockquote>
</div>
<p class="MsoNormal"><u></u> <u></u></p>
</div>
</div></div></div>
</div>
</blockquote></div><br></div>