<html xmlns:v="urn:schemas-microsoft-com:vml" xmlns:o="urn:schemas-microsoft-com:office:office" xmlns:w="urn:schemas-microsoft-com:office:word" xmlns:m="http://schemas.microsoft.com/office/2004/12/omml" xmlns="http://www.w3.org/TR/REC-html40">
<head>
<meta http-equiv="Content-Type" content="text/html; charset=us-ascii">
<meta name="Generator" content="Microsoft Word 15 (filtered medium)">
<style><!--
/* Font Definitions */
@font-face
{font-family:"Cambria Math";
panose-1:2 4 5 3 5 4 6 3 2 4;}
@font-face
{font-family:Calibri;
panose-1:2 15 5 2 2 2 4 3 2 4;}
/* Style Definitions */
p.MsoNormal, li.MsoNormal, div.MsoNormal
{margin:0in;
margin-bottom:.0001pt;
font-size:11.0pt;
font-family:"Calibri",sans-serif;}
a:link, span.MsoHyperlink
{mso-style-priority:99;
color:#0563C1;
text-decoration:underline;}
a:visited, span.MsoHyperlinkFollowed
{mso-style-priority:99;
color:#954F72;
text-decoration:underline;}
p.MsoListParagraph, li.MsoListParagraph, div.MsoListParagraph
{mso-style-priority:34;
margin-top:0in;
margin-right:0in;
margin-bottom:0in;
margin-left:.5in;
margin-bottom:.0001pt;
font-size:11.0pt;
font-family:"Calibri",sans-serif;}
p.msonormal0, li.msonormal0, div.msonormal0
{mso-style-name:msonormal;
mso-margin-top-alt:auto;
margin-right:0in;
mso-margin-bottom-alt:auto;
margin-left:0in;
font-size:11.0pt;
font-family:"Calibri",sans-serif;}
span.EmailStyle19
{mso-style-type:personal;
font-family:"Calibri",sans-serif;
color:windowtext;}
span.EmailStyle20
{mso-style-type:personal;
font-family:"Calibri",sans-serif;
color:#1F497D;}
span.EmailStyle21
{mso-style-type:personal;
font-family:"Calibri",sans-serif;
color:windowtext;}
span.EmailStyle22
{mso-style-type:personal;
font-family:"Calibri",sans-serif;
color:#1F497D;}
span.EmailStyle23
{mso-style-type:personal;
font-family:"Calibri",sans-serif;
color:windowtext;}
span.EmailStyle24
{mso-style-type:personal-reply;
font-family:"Calibri",sans-serif;
color:#1F497D;}
.MsoChpDefault
{mso-style-type:export-only;
font-size:10.0pt;}
@page WordSection1
{size:8.5in 11.0in;
margin:1.0in 1.0in 1.0in 1.0in;}
div.WordSection1
{page:WordSection1;}
/* List Definitions */
@list l0
{mso-list-id:414715555;
mso-list-template-ids:-1976275326;}
@list l1
{mso-list-id:948244985;
mso-list-type:hybrid;
mso-list-template-ids:-1073949826 67698703 67698713 67698715 67698703 67698713 67698715 67698703 67698713 67698715;}
@list l1:level1
{mso-level-tab-stop:none;
mso-level-number-position:left;
text-indent:-.25in;}
@list l1:level2
{mso-level-number-format:alpha-lower;
mso-level-tab-stop:none;
mso-level-number-position:left;
text-indent:-.25in;}
@list l1:level3
{mso-level-number-format:roman-lower;
mso-level-tab-stop:none;
mso-level-number-position:right;
text-indent:-9.0pt;}
@list l1:level4
{mso-level-tab-stop:none;
mso-level-number-position:left;
text-indent:-.25in;}
@list l1:level5
{mso-level-number-format:alpha-lower;
mso-level-tab-stop:none;
mso-level-number-position:left;
text-indent:-.25in;}
@list l1:level6
{mso-level-number-format:roman-lower;
mso-level-tab-stop:none;
mso-level-number-position:right;
text-indent:-9.0pt;}
@list l1:level7
{mso-level-tab-stop:none;
mso-level-number-position:left;
text-indent:-.25in;}
@list l1:level8
{mso-level-number-format:alpha-lower;
mso-level-tab-stop:none;
mso-level-number-position:left;
text-indent:-.25in;}
@list l1:level9
{mso-level-number-format:roman-lower;
mso-level-tab-stop:none;
mso-level-number-position:right;
text-indent:-9.0pt;}
@list l2
{mso-list-id:1469398549;
mso-list-template-ids:-1017058674;}
@list l2:level1
{mso-level-start-at:2;
mso-level-tab-stop:.5in;
mso-level-number-position:left;
text-indent:-.25in;}
@list l3
{mso-list-id:1506896240;
mso-list-type:hybrid;
mso-list-template-ids:704445374 67698705 67698713 67698715 67698703 67698713 67698715 67698703 67698713 67698715;}
@list l3:level1
{mso-level-text:"%1\)";
mso-level-tab-stop:none;
mso-level-number-position:left;
text-indent:-.25in;}
@list l3:level2
{mso-level-number-format:alpha-lower;
mso-level-tab-stop:none;
mso-level-number-position:left;
text-indent:-.25in;}
@list l3:level3
{mso-level-number-format:roman-lower;
mso-level-tab-stop:none;
mso-level-number-position:right;
text-indent:-9.0pt;}
@list l3:level4
{mso-level-tab-stop:none;
mso-level-number-position:left;
text-indent:-.25in;}
@list l3:level5
{mso-level-number-format:alpha-lower;
mso-level-tab-stop:none;
mso-level-number-position:left;
text-indent:-.25in;}
@list l3:level6
{mso-level-number-format:roman-lower;
mso-level-tab-stop:none;
mso-level-number-position:right;
text-indent:-9.0pt;}
@list l3:level7
{mso-level-tab-stop:none;
mso-level-number-position:left;
text-indent:-.25in;}
@list l3:level8
{mso-level-number-format:alpha-lower;
mso-level-tab-stop:none;
mso-level-number-position:left;
text-indent:-.25in;}
@list l3:level9
{mso-level-number-format:roman-lower;
mso-level-tab-stop:none;
mso-level-number-position:right;
text-indent:-9.0pt;}
ol
{margin-bottom:0in;}
ul
{margin-bottom:0in;}
--></style><!--[if gte mso 9]><xml>
<o:shapedefaults v:ext="edit" spidmax="1026" />
</xml><![endif]--><!--[if gte mso 9]><xml>
<o:shapelayout v:ext="edit">
<o:idmap v:ext="edit" data="1" />
</o:shapelayout></xml><![endif]-->
</head>
<body lang="EN-US" link="#0563C1" vlink="#954F72">
<div class="WordSection1">
<p class="MsoNormal"><span style="color:#1F497D">Subhadeep,<o:p></o:p></span></p>
<p class="MsoNormal"><span style="color:#1F497D"><o:p> </o:p></span></p>
<p class="MsoNormal"><span style="color:#1F497D">I don’t see anything unusual in the debug logs for fi_info output. It The other logs didn’t contain any debug output but that shouldn’t matter.<o:p></o:p></span></p>
<p class="MsoNormal"><span style="color:#1F497D"><o:p> </o:p></span></p>
<p class="MsoNormal"><span style="color:#1F497D">Btw, are you using libfabric v1.5.0? You might want to get an updated version: v1.6.1 (<a href="https://github.com/ofiwg/libfabric/releases/tag/v1.6.1">https://github.com/ofiwg/libfabric/releases/tag/v1.6.1</a>)<o:p></o:p></span></p>
<p class="MsoNormal"><span style="color:#1F497D"><o:p> </o:p></span></p>
<p class="MsoNormal"><span style="color:#1F497D">I believe v1.5.0 was probably not handling a NULL value for node and service correctly.<o:p></o:p></span></p>
<p class="MsoNormal"><span style="color:#1F497D"><o:p> </o:p></span></p>
<p class="MsoNormal"><span style="color:#1F497D">Thanks,<o:p></o:p></span></p>
<p class="MsoNormal"><span style="color:#1F497D">Arun.<o:p></o:p></span></p>
<p class="MsoNormal"><a name="_MailEndCompose"><span style="color:#1F497D"><o:p> </o:p></span></a></p>
<div>
<div style="border:none;border-top:solid #E1E1E1 1.0pt;padding:3.0pt 0in 0in 0in">
<p class="MsoNormal"><a name="_____replyseparator"></a><b>From:</b> Subhadeep Bhattacharya [mailto:sb17v@my.fsu.edu]
<br>
<b>Sent:</b> Monday, May 14, 2018 1:23 PM<br>
<b>To:</b> Ilango, Arun <arun.ilango@intel.com><br>
<b>Cc:</b> libfabric-users@lists.openfabrics.org<br>
<b>Subject:</b> RE: [libfabric-users] Libfabric : Facing issues with FI_VERSION(1, 5) in fi_getinfo()<o:p></o:p></p>
</div>
</div>
<p class="MsoNormal"><o:p> </o:p></p>
<p class="MsoNormal">Hi Arun,<o:p></o:p></p>
<p class="MsoNormal"><o:p> </o:p></p>
<p class="MsoNormal">I have built Libfabric with –enable-debug configuration option and set FI_LOG_LEVEL env variable as debug. Please find the attached log and output from fi_info -v output.<o:p></o:p></p>
<p class="MsoNormal"><o:p> </o:p></p>
<p class="MsoNormal">Thanks,<o:p></o:p></p>
<p class="MsoNormal">Subhadeep<o:p></o:p></p>
<p class="MsoNormal"><o:p> </o:p></p>
<div>
<div style="border:none;border-top:solid #E1E1E1 1.0pt;padding:3.0pt 0in 0in 0in">
<p class="MsoNormal"><b>From:</b> Ilango, Arun [<a href="mailto:arun.ilango@intel.com">mailto:arun.ilango@intel.com</a>]
<br>
<b>Sent:</b> Monday, May 14, 2018 1:35 PM<br>
<b>To:</b> Subhadeep Bhattacharya <<a href="mailto:sb17v@my.fsu.edu">sb17v@my.fsu.edu</a>>;
<a href="mailto:libfabric-users@lists.openfabrics.org">libfabric-users@lists.openfabrics.org</a><br>
<b>Subject:</b> RE: [libfabric-users] Libfabric : Facing issues with FI_VERSION(1, 5) in fi_getinfo()<o:p></o:p></p>
</div>
</div>
<p class="MsoNormal"><o:p> </o:p></p>
<p class="MsoNormal"><span style="color:#1F497D">Subhadeep,<o:p></o:p></span></p>
<p class="MsoNormal"><span style="color:#1F497D"><o:p> </o:p></span></p>
<p class="MsoNormal"><span style="color:#1F497D">I don’t see any debug level logs. Did you set the correct level in FI_LOG_LEVEL env variable? Also you would have to build libfabric with --enable-debug configure option.<o:p></o:p></span></p>
<p class="MsoNormal"><span style="color:#1F497D"><o:p> </o:p></span></p>
<p class="MsoNormal"><span style="color:#1F497D">Can you also try running fi_info -v to see if you get all the interfaces?<o:p></o:p></span></p>
<p class="MsoNormal"><span style="color:#1F497D"><o:p> </o:p></span></p>
<p class="MsoNormal"><span style="color:#1F497D">Thanks,<o:p></o:p></span></p>
<p class="MsoNormal"><span style="color:#1F497D">Arun.<o:p></o:p></span></p>
<p class="MsoNormal"><span style="color:#1F497D"><o:p> </o:p></span></p>
<div>
<div style="border:none;border-top:solid #E1E1E1 1.0pt;padding:3.0pt 0in 0in 0in">
<p class="MsoNormal"><b>From:</b> Subhadeep Bhattacharya [<a href="mailto:sb17v@my.fsu.edu">mailto:sb17v@my.fsu.edu</a>]
<br>
<b>Sent:</b> Monday, May 14, 2018 9:42 AM<br>
<b>To:</b> Ilango, Arun <<a href="mailto:arun.ilango@intel.com">arun.ilango@intel.com</a>>;
<a href="mailto:libfabric-users@lists.openfabrics.org">libfabric-users@lists.openfabrics.org</a><br>
<b>Subject:</b> RE: [libfabric-users] Libfabric : Facing issues with FI_VERSION(1, 5) in fi_getinfo()<o:p></o:p></p>
</div>
</div>
<p class="MsoNormal"><o:p> </o:p></p>
<p class="MsoNormal">Hi Arun,<o:p></o:p></p>
<p class="MsoNormal"><o:p> </o:p></p>
<p class="MsoNormal">Please find the attached debug level logs.<o:p></o:p></p>
<p class="MsoNormal"><o:p> </o:p></p>
<p class="MsoNormal"><o:p> </o:p></p>
<p class="MsoNormal">Thanks,<o:p></o:p></p>
<p class="MsoNormal">Subhadeep<o:p></o:p></p>
<p class="MsoNormal"><o:p> </o:p></p>
<div>
<div style="border:none;border-top:solid #E1E1E1 1.0pt;padding:3.0pt 0in 0in 0in">
<p class="MsoNormal"><b>From:</b> Ilango, Arun [<a href="mailto:arun.ilango@intel.com">mailto:arun.ilango@intel.com</a>]
<br>
<b>Sent:</b> Monday, May 14, 2018 12:29 PM<br>
<b>To:</b> Subhadeep Bhattacharya <<a href="mailto:sb17v@my.fsu.edu">sb17v@my.fsu.edu</a>>;
<a href="mailto:libfabric-users@lists.openfabrics.org">libfabric-users@lists.openfabrics.org</a><br>
<b>Subject:</b> RE: [libfabric-users] Libfabric : Facing issues with FI_VERSION(1, 5) in fi_getinfo()<o:p></o:p></p>
</div>
</div>
<p class="MsoNormal"><o:p> </o:p></p>
<p class="MsoNormal"><span style="color:#1F497D">Subhadeep,<o:p></o:p></span></p>
<p class="MsoNormal"><span style="color:#1F497D"><o:p> </o:p></span></p>
<p class="MsoNormal"><span style="color:#1F497D">For scenario in 1, verbs;ofi_rxm should return fi_info for all available interfaces. Can you check if the provider is returning just the fi_info for localhost or is it returning other fi_info in the list as well?
If you can share the debug level logs, I can take a look.<o:p></o:p></span></p>
<p class="MsoNormal"><span style="color:#1F497D"><o:p> </o:p></span></p>
<p class="MsoNormal"><span style="color:#1F497D">2) ofi_rxm doesn’t have atomics support yet but it has read and write functionality. Can you try just setting FI_WRITE and FI_READ in hints->caps?<o:p></o:p></span></p>
<p class="MsoNormal"><span style="color:#1F497D"><o:p> </o:p></span></p>
<p class="MsoNormal"><span style="color:#1F497D">Thanks,<o:p></o:p></span></p>
<p class="MsoNormal"><span style="color:#1F497D">Arun.<o:p></o:p></span></p>
<p class="MsoNormal"><span style="color:#1F497D"><o:p> </o:p></span></p>
<div>
<div style="border:none;border-top:solid #E1E1E1 1.0pt;padding:3.0pt 0in 0in 0in">
<p class="MsoNormal"><b>From:</b> Libfabric-users [<a href="mailto:libfabric-users-bounces@lists.openfabrics.org">mailto:libfabric-users-bounces@lists.openfabrics.org</a>]
<b>On Behalf Of </b>Subhadeep Bhattacharya<br>
<b>Sent:</b> Monday, May 14, 2018 8:23 AM<br>
<b>To:</b> <a href="mailto:libfabric-users@lists.openfabrics.org">libfabric-users@lists.openfabrics.org</a><br>
<b>Subject:</b> [libfabric-users] Libfabric : Facing issues with FI_VERSION(1, 5) in fi_getinfo()<o:p></o:p></p>
</div>
</div>
<p class="MsoNormal"><o:p> </o:p></p>
<p class="MsoNormal">Hi,<o:p></o:p></p>
<p class="MsoNormal"><o:p> </o:p></p>
<p class="MsoNormal">I have few questions regarding the usage of verbs;ofi_rxm provider.
<o:p></o:p></p>
<p class="MsoNormal" style="margin-left:.5in;text-indent:-.25in;mso-list:l3 level1 lfo3">
<![if !supportLists]><span style="mso-list:Ignore">1)<span style="font:7.0pt "Times New Roman"">
</span></span><![endif]>We have latest OFED driver installed in our cluster and it have mlx5_0. The inet address for infiniband is set to 10.10.16.2 for node 2. However, if I put node and service as NULL and flag as 0 while querying the fi_getinfo, it is fetching
the localhost address(127.0.0.1) instead of ib0(10.10.16.2). Can you please let us know how to solve this issue.<o:p></o:p></p>
<p class="MsoListParagraph"><o:p> </o:p></p>
<p class="MsoNormal" style="margin-left:.5in;text-indent:-.25in;mso-list:l3 level1 lfo3">
<![if !supportLists]><span style="mso-list:Ignore">2)<span style="font:7.0pt "Times New Roman"">
</span></span><![endif]>I am trying to get read, write and atomics functionality. I can see in the manual that verbs support FI_ATOMIC but ofi_rxm does not support it. Do I need to use both FI_EP_RDM and FI_EP_MSG as endpoint types with verbs;ofi_rxm to enable
atomics along with read and write.<o:p></o:p></p>
<p class="MsoListParagraph"><o:p> </o:p></p>
<p class="MsoNormal">We are trying to get read, write and atomics functionality using verbs;ofi_rxm provider of Libfabric. Can you please let us know, which options are necessary to set in hints before passing it to fi_getinfo.<o:p></o:p></p>
<p class="MsoNormal"><o:p> </o:p></p>
<p class="MsoNormal"><o:p> </o:p></p>
<p class="MsoNormal">Thanks and Regards,<o:p></o:p></p>
<p class="MsoNormal">Subhadeep<o:p></o:p></p>
<p class="MsoNormal"><o:p> </o:p></p>
</div>
</body>
</html>