[libfabric-users] Verbs send message slow down

[Supun Kamburugamuve]

I use a separate CQ for each endpoint.

Thanks,
Supun..

On Tue, Mar 21, 2017 at 5:03 PM, Ilango, Arun <arun.ilango at intel.com> wrote:

> I see. Do you use a common CQ or a CQ per endpoint? We do have a lock in
> the CQ to serialize access to it.
>
>
>
> -Arun.
>
>
>
> *From:* Supun Kamburugamuve [mailto:skamburugamuve at gmail.com]
> *Sent:* Tuesday, March 21, 2017 1:07 PM
> *To:* Ilango, Arun <arun.ilango at intel.com>
> *Cc:* Hefty, Sean <sean.hefty at intel.com>; libfabric-users at lists.
> openfabrics.org
>
> *Subject:* Re: [libfabric-users] Verbs send message slow down
>
>
>
> Thanks, Arun, will try this. This happens with multiple connections,
> though. The program works when there is a small number of nodes sending
> data. For example, for 4 nodes it works. For 8 Nodes one or two QP's become
> slow.
>
>
>
> Supun..
>
>
>
> On Tue, Mar 21, 2017 at 3:26 PM, Ilango, Arun <arun.ilango at intel.com>
> wrote:
>
> Hi Supun,
>
> I tried to reproduce the issue but I'm not hitting it. I have modified
> fabtests bandwidth test to read multiple completions (=window_size) at a
> time on the receive side. I have the patch on the following branch:
> https://github.com/a-ilango/fabtests/tree/comp_debug
>
> I'm using fi_cq_read. Please try out fi_msg_bw and see if you get the
> issue.
>
> Thanks,
> Arun.
>
>
> -----Original Message-----
> From: Libfabric-users [mailto:libfabric-users-
> bounces at lists.openfabrics.org] On Behalf Of Ilango, Arun
> Sent: Tuesday, March 21, 2017 11:07 AM
> To: Hefty, Sean <sean.hefty at intel.com>; Supun Kamburugamuve <
> skamburugamuve at gmail.com>
> Cc: libfabric-users at lists.openfabrics.org
> Subject: Re: [libfabric-users] Verbs send message slow down
>
> This might need some debugging. In the verbs provider, we call ibv_poll_cq
> with num_entries set to 1 and repeatedly call it when the app has requested
> multiple completions. I'm not sure if this is the issue but let me check.
>
> -Arun.
>
> -----Original Message-----
> From: Libfabric-users [mailto:libfabric-users-
> bounces at lists.openfabrics.org] On Behalf Of Hefty, Sean
> Sent: Tuesday, March 21, 2017 10:53 AM
> To: Supun Kamburugamuve <skamburugamuve at gmail.com>
> Cc: libfabric-users at lists.openfabrics.org
> Subject: Re: [libfabric-users] Verbs send message slow down
>
> > I found a workaround to the problem. Previously I was doing cq_read
> > for multiple completions. If I do cq_read for a single completion the
> > problem goes away. Is this the expected behavior or something wrong
> > with my program or a bug in libfabric?
>
> This is not expected behavior.  Something else is going goofy here.
> _______________________________________________
> Libfabric-users mailing list
> Libfabric-users at lists.openfabrics.org
> http://lists.openfabrics.org/mailman/listinfo/libfabric-users
> _______________________________________________
> Libfabric-users mailing list
> Libfabric-users at lists.openfabrics.org
> http://lists.openfabrics.org/mailman/listinfo/libfabric-users
>
>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.openfabrics.org/pipermail/libfabric-users/attachments/20170321/0eb40dd8/attachment.html>