[libfabric-users] Libfabric : Facing issues with FI_VERSION(1, 5) in fi_getinfo()

Fri Feb 23 11:25:41 PST 2018

Hi Arun,

I have tried using the steps mentioned by you but it is still not working. I also tried to run the fabtest but it was also not returning verbs provider. I have installed Libfabric v1.5.1 using the following steps-

  1.  ./configure --prefix=$HOME/libfabric151 --enable-verbs=yes --enable-rxm=yes --disable-mlx
  2.  make
  3.  make install

Please find the attached logs for more details.

Following is my .bashrc configuration for libfabric-

[cid:image006.png at 01D3ACB2.2CF1DA00]

I am using the following command for compiling my program -
cc -o libfabric_test_version libfabric_test_version.c -I $LIBFABRIC_HOME/include -L $LIBFABRIC_HOME/lib -lfabric

Sample program is also attached with this mail.

Can you please let us know if we are doing anything wrong. It will be really great if you please let us know how to resolve this issue.

Thanks,
Subhadeep

From: Ilango, Arun [mailto:arun.ilango at intel.com]
Sent: Wednesday, February 21, 2018 1:30 PM
To: Subhadeep Bhattacharya <sb17v at my.fsu.edu<mailto:sb17v at my.fsu.edu>>; Hefty, Sean <sean.hefty at intel.com<mailto:sean.hefty at intel.com>>
Cc: Gladkov, Dmitry <dmitry.gladkov at intel.com<mailto:dmitry.gladkov at intel.com>>; libfabric-users at lists.openfabrics.org<mailto:libfabric-users at lists.openfabrics.org>
Subject: RE: Libfabric : Facing issues with FI_VERSION(1, 5) in fi_getinfo()

Subhadeep,

There are two ways to use RDM endpoint in verbs provider: one is the verbs/RDM path that's internal to the provider and the other is through the ofi_rxm utility provider which layers on top of verbs provider. The former would be deprecated from libfabric 1.7 onwards and it's advisable to use ofi_rxm.

To use the internal verbs/RDM path, FI_CONTEXT mode bit needs to be set in hints. To use ofi_rxm FI_MR_LOCAL domain attribute mr_mode bit needs to be set.

Can you try with the following added to your code:

hints->domain_attr->mr_mode = FI_MR_LOCAL;
hints->mode = FI_CONTEXT;

Note: Passing FI_LOCAL_MR in hints->mode as you have in the code should have worked for ofi_rxm but there is a bug which currently prevents that. It should be fixed in the next release.

Thanks,
Arun.

From: Subhadeep Bhattacharya [mailto:sb17v at my.fsu.edu]
Sent: Tuesday, February 20, 2018 6:10 PM
To: Ilango, Arun <arun.ilango at intel.com<mailto:arun.ilango at intel.com>>; Hefty, Sean <sean.hefty at intel.com<mailto:sean.hefty at intel.com>>
Cc: Gladkov, Dmitry <dmitry.gladkov at intel.com<mailto:dmitry.gladkov at intel.com>>; libfabric-users at lists.openfabrics.org<mailto:libfabric-users at lists.openfabrics.org>
Subject: RE: Libfabric : Facing issues with FI_VERSION(1, 5) in fi_getinfo()

Hi,

The IP address I am using (10.10.16.2 and 10.10.16.3) are seems to be fine and they are associated with IPoIB devices. I am getting the verbs as a provider while using FI_VERSION(1,4). But for FI_VERSION(1,5) I am not getting the verbs provider.

PFA the logs -

logs_for_v15 - For FI_VERSION(1,5)
logs_for_v14 - For FI_VERSION(1,4)

Thanks,
Subhadeep

From: Ilango, Arun [mailto:arun.ilango at intel.com]
Sent: Tuesday, February 20, 2018 2:14 PM
To: Hefty, Sean <sean.hefty at intel.com<mailto:sean.hefty at intel.com>>; Subhadeep Bhattacharya <sb17v at my.fsu.edu<mailto:sb17v at my.fsu.edu>>
Cc: Gladkov, Dmitry <dmitry.gladkov at intel.com<mailto:dmitry.gladkov at intel.com>>; libfabric-users at lists.openfabrics.org<mailto:libfabric-users at lists.openfabrics.org>
Subject: RE: Libfabric : Facing issues with FI_VERSION(1, 5) in fi_getinfo()

Subhadeep,

Can you run your program with FI_LOG_LEVEL=info and share the logs?

Thanks,
Arun.

From: Hefty, Sean
Sent: Tuesday, February 20, 2018 9:05 AM
To: Subhadeep Bhattacharya <sb17v at my.fsu.edu<mailto:sb17v at my.fsu.edu>>; Ilango, Arun <arun.ilango at intel.com<mailto:arun.ilango at intel.com>>
Cc: Gladkov, Dmitry <dmitry.gladkov at intel.com<mailto:dmitry.gladkov at intel.com>>; libfabric-users at lists.openfabrics.org<mailto:libfabric-users at lists.openfabrics.org>
Subject: RE: Libfabric : Facing issues with FI_VERSION(1, 5) in fi_getinfo()

Copying the verbs maintainers and libfabric-users list.  I think the most common problem is selecting IP addresses that are associated with IPoIB devices.

Btw, you may want to look at the fi_tostr() function for printing libfabric structures, bitmasks, etc.

- Sean

From: Subhadeep Bhattacharya [mailto:sb17v at my.fsu.edu]
Sent: Monday, February 19, 2018 9:01 AM
To: Hefty, Sean <sean.hefty at intel.com<mailto:sean.hefty at intel.com>>
Subject: Libfabric : Facing issues with FI_VERSION(1, 5) in fi_getinfo()

Hi Sean,

I am trying to work with Libfabric in our cluster but currently facing some issues. While checking the 'fi_info' with the provider as 'verbs', I am getting verbs provider for the current node.

[cid:image001.png at 01D3AA8E.A09391E0]

I have installed 'v1.5.3' in our cluster and trying to write some code using Libfabric. But I am getting only 'sockets' as provider when giving FI_VERSION(1, 5) in fi_getinfo() function. I want to use verbs provider but the fi_getinfo is not returning it. Please find the images below for more details-

[cid:image002.png at 01D3AA8E.A09391E0]

[cid:image003.png at 01D3AA8E.A09391E0]

If I give FI_VERSION(1, 4), I am getting verbs also as provider.

[cid:image004.png at 01D3AA8E.A09391E0]

[cid:image005.png at 01D3AA8E.A09391E0]

Can you please clarify if I am doing anything wrong. It would be really great if you please mention how to resolve this issue.

Thanks and Regards,
Subhadeep

Graduate Research Assistant
Computer Architecture and SysTems Research Lab (CASTL)
Florida State University

-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.openfabrics.org/pipermail/libfabric-users/attachments/20180223/06d5343c/attachment.html>
-------------- next part --------------
A non-text attachment was scrubbed...
Name: image006.png
Type: image/png
Size: 13453 bytes
Desc: image006.png
URL: <http://lists.openfabrics.org/pipermail/libfabric-users/attachments/20180223/06d5343c/attachment.png>
-------------- next part --------------
A non-text attachment was scrubbed...
Name: image007.png
Type: image/png
Size: 17399 bytes
Desc: image007.png
URL: <http://lists.openfabrics.org/pipermail/libfabric-users/attachments/20180223/06d5343c/attachment-0001.png>
-------------- next part --------------
A non-text attachment was scrubbed...
Name: image008.png
Type: image/png
Size: 31290 bytes
Desc: image008.png
URL: <http://lists.openfabrics.org/pipermail/libfabric-users/attachments/20180223/06d5343c/attachment-0002.png>
-------------- next part --------------
A non-text attachment was scrubbed...
Name: image009.png
Type: image/png
Size: 8384 bytes
Desc: image009.png
URL: <http://lists.openfabrics.org/pipermail/libfabric-users/attachments/20180223/06d5343c/attachment-0003.png>
-------------- next part --------------
A non-text attachment was scrubbed...
Name: image010.png
Type: image/png
Size: 32234 bytes
Desc: image010.png
URL: <http://lists.openfabrics.org/pipermail/libfabric-users/attachments/20180223/06d5343c/attachment-0004.png>
-------------- next part --------------
A non-text attachment was scrubbed...
Name: image011.png
Type: image/png
Size: 11919 bytes
Desc: image011.png
URL: <http://lists.openfabrics.org/pipermail/libfabric-users/attachments/20180223/06d5343c/attachment-0005.png>
-------------- next part --------------
An embedded and charset-unspecified text was scrubbed...
Name: fabtest_log.txt
URL: <http://lists.openfabrics.org/pipermail/libfabric-users/attachments/20180223/06d5343c/attachment.txt>
-------------- next part --------------
An embedded and charset-unspecified text was scrubbed...
Name: fi_info_details.txt
URL: <http://lists.openfabrics.org/pipermail/libfabric-users/attachments/20180223/06d5343c/attachment-0001.txt>
-------------- next part --------------
An embedded and charset-unspecified text was scrubbed...
Name: libfabric_test_version.c
URL: <http://lists.openfabrics.org/pipermail/libfabric-users/attachments/20180223/06d5343c/attachment.c>
-------------- next part --------------
An embedded and charset-unspecified text was scrubbed...
Name: log.txt
URL: <http://lists.openfabrics.org/pipermail/libfabric-users/attachments/20180223/06d5343c/attachment-0002.txt>
-------------- next part --------------
An embedded and charset-unspecified text was scrubbed...
Name: makeinstalllog.txt
URL: <http://lists.openfabrics.org/pipermail/libfabric-users/attachments/20180223/06d5343c/attachment-0003.txt>
-------------- next part --------------
An embedded and charset-unspecified text was scrubbed...
Name: makelog.txt
URL: <http://lists.openfabrics.org/pipermail/libfabric-users/attachments/20180223/06d5343c/attachment-0004.txt>
-------------- next part --------------
An embedded and charset-unspecified text was scrubbed...
Name: configurelog.txt
URL: <http://lists.openfabrics.org/pipermail/libfabric-users/attachments/20180223/06d5343c/attachment-0005.txt>