<html>
<head>
<meta http-equiv="Content-Type" content="text/html; charset=Windows-1252">
<style type="text/css" style="display:none;"> P {margin-top:0;margin-bottom:0;} </style>
</head>
<body dir="ltr">
<div style="font-family: Calibri, Arial, Helvetica, sans-serif; font-size: 12pt; color: rgb(0, 0, 0);">
This sort of situation is where I miss something like the (now deprecated) 'sockets' provider. As documented at least, it supported the entire interface. Of course that would be work to maintain (and I'm not volunteering, heh), but a thing doesn't have to perform
well to be beneficial.</div>
<div style="font-family: Calibri, Arial, Helvetica, sans-serif; font-size: 12pt; color: rgb(0, 0, 0);">
<br>
</div>
<div style="font-family: Calibri, Arial, Helvetica, sans-serif; font-size: 12pt; color: rgb(0, 0, 0);">
greg</div>
<div style="font-family: Calibri, Arial, Helvetica, sans-serif; font-size: 12pt; color: rgb(0, 0, 0);">
<br>
</div>
<div id="appendonsend"></div>
<hr style="display:inline-block;width:98%" tabindex="-1">
<div id="divRplyFwdMsg" dir="ltr"><font face="Calibri, sans-serif" style="font-size:11pt" color="#000000"><b>From:</b> Libfabric-users <libfabric-users-bounces@lists.openfabrics.org> on behalf of Hefty, Sean <sean.hefty@intel.com><br>
<b>Sent:</b> Tuesday, March 23, 2021 4:52 PM<br>
<b>To:</b> Biddiscombe, John A. <john.biddiscombe@cscs.ch>; libfabric-users@lists.openfabrics.org <libfabric-users@lists.openfabrics.org><br>
<b>Subject:</b> Re: [libfabric-users] Queue size question</font>
<div> </div>
</div>
<style>
<!--
@font-face
{font-family:"Cambria Math"}
@font-face
{font-family:Calibri}
@font-face
{font-family:Consolas}
p.x_MsoNormal, li.x_MsoNormal, div.x_MsoNormal
{margin:0in;
font-size:11.0pt;
font-family:"Calibri",sans-serif}
a:link, span.x_MsoHyperlink
{color:#0563C1;
text-decoration:underline}
a:visited, span.x_MsoHyperlinkFollowed
{color:#954F72;
text-decoration:underline}
pre
{margin:0in;
margin-bottom:.0001pt;
font-size:10.0pt;
font-family:"Courier New"}
p.x_msonormal0, li.x_msonormal0, div.x_msonormal0
{margin:0in;
font-size:11.0pt;
font-family:"Calibri",sans-serif}
p.x_emailquote, li.x_emailquote, div.x_emailquote
{margin-right:0in;
margin-left:1.0pt;
border:none;
padding:0in;
font-size:11.0pt;
font-family:"Calibri",sans-serif}
span.x_HTMLPreformattedChar
{font-family:"Consolas",serif}
span.x_EmailStyle23
{font-family:"Calibri",sans-serif;
color:windowtext}
span.x_SpellE
{}
.x_MsoChpDefault
{font-size:10.0pt}
@page WordSection1
{margin:1.0in 1.0in 1.0in 1.0in}
div.x_WordSection1
{}
-->
</style>
<div lang="EN-US" link="#0563C1" vlink="#954F72" style="word-wrap:break-word">
<div class="x_WordSection1">
<p class="x_MsoNormal"><font size="2" face="Calibri"><span style="font-size:11.0pt">That feature is not supported.<span style="">
</span>It would need to be emulated in SW, which wouldn’t provide any real benefit.</span></font></p>
<p class="x_MsoNormal"><font size="2" face="Calibri"><span style="font-size:11.0pt"> </span></font></p>
<div style="border:none; border-left:solid blue 1.5pt; padding:0in 0in 0in 4.0pt">
<div>
<div style="border:none; border-top:solid #E1E1E1 1.0pt; padding:3.0pt 0in 0in 0in">
<p class="x_MsoNormal"><b><font size="2" face="Calibri"><span style="font-size:11.0pt; font-weight:bold">From:</span></font></b><span style=""> Biddiscombe, John A. <john.biddiscombe@cscs.ch>
<br>
<b><span style="font-weight:bold">Sent:</span></b> Tuesday, March 23, 2021 2:02 PM<br>
<b><span style="font-weight:bold">To:</span></b> Hefty, Sean <sean.hefty@intel.com>; libfabric-users@lists.openfabrics.org<br>
<b><span style="font-weight:bold">Subject:</span></b> Re: Queue size question</span></p>
</div>
</div>
<p class="x_MsoNormal"><font size="2" face="Calibri"><span style="font-size:11.0pt"> </span></font></p>
<div>
<div id="x_x_divtagdefaultwrapper">
<p><font size="3" color="black" face="Calibri"><span style="font-size:12.0pt; color:black">I thought I'd experiment with scalable endpoints as an alternative to the thread local endpoints, but I'm getting ENOSYS from the
<span class="x_SpellE">tcp;rxm_ofi</span> setup.</span></font></p>
<p><font size="3" color="black" face="Calibri"><span style="font-size:12.0pt; color:black"> </span></font></p>
<p><font size="3" color="black" face="Calibri"><span style="font-size:12.0pt; color:black">is that something I can work around with different flags, or is that something that
<span class="x_SpellE">jjust</span> isn't supported (feature matrix doesn't mention it)</span></font></p>
<p><font size="3" color="black" face="Calibri"><span style="font-size:12.0pt; color:black"> </span></font></p>
<p><font size="3" color="black" face="Calibri"><span style="font-size:12.0pt; color:black">Thanks</span></font></p>
<p><font size="3" color="black" face="Calibri"><span style="font-size:12.0pt; color:black">JB</span></font></p>
<p><font size="3" color="black" face="Calibri"><span style="font-size:12.0pt; color:black"> </span></font></p>
<pre><font size="2" color="olive" face="Courier New"><span style="font-size:10.0pt; color:olive">int</span></font><font color="gray"><span style="color:gray"> </span></font><span class="x_SpellE"><b><font color="#e5e4e1"><span style="color:#E5E4E1; font-weight:bold">fi_no_scalable_ep</span></font></b></span><font color="black"><span style="color:black">(</span></font><font color="#87cde6"><span style="color:#87CDE6">struct</span></font><font color="gray"><span style="color:gray"> </span></font><span class="x_SpellE"><font color="#87cde6"><span style="color:#87CDE6">fid_domain</span></font></span><font color="gray"><span style="color:gray"> </span></font><font color="black"><span style="color:black">*</span></font><font color="#a0dc2c"><span style="color:#A0DC2C">domain</span></font><font color="black"><span style="color:black">,</span></font><font color="gray"><span style="color:gray"> </span></font><font color="#87cde6"><span style="color:#87CDE6">struct</span></font><font color="gray"><span style="color:gray"> </span></font><span class="x_SpellE"><font color="#87cde6"><span style="color:#87CDE6">fi_info</span></font></span><font color="gray"><span style="color:gray"> </span></font><font color="black"><span style="color:black">*</span></font><font color="#a0dc2c"><span style="color:#A0DC2C">info</span></font><font color="black"><span style="color:black">,</span></font></pre>
<pre><font size="2" color="gray" face="Courier New"><span style="font-size:10.0pt; color:gray"><span style=""> </span></span></font><font color="#87cde6"><span style="color:#87CDE6">struct</span></font><font color="gray"><span style="color:gray"> </span></font><span class="x_SpellE"><font color="#87cde6"><span style="color:#87CDE6">fid_ep</span></font></span><font color="gray"><span style="color:gray"> </span></font><font color="black"><span style="color:black">**</span></font><span class="x_SpellE"><font color="#a0dc2c"><span style="color:#A0DC2C">sep</span></font></span><font color="black"><span style="color:black">,</span></font><font color="gray"><span style="color:gray"> </span></font><font color="olive"><span style="color:olive">void</span></font><font color="gray"><span style="color:gray"> </span></font><font color="black"><span style="color:black">*</span></font><font color="#a0dc2c"><span style="color:#A0DC2C">context</span></font><font color="black"><span style="color:black">)</span></font></pre>
<pre><font size="2" color="black" face="Courier New"><span style="font-size:10.0pt; color:black">{</span></font></pre>
<pre><font size="2" color="gray" face="Courier New"><span style="font-size:10.0pt; color:gray"><span style=""> </span></span></font><font color="#87cde6"><span style="color:#87CDE6">return</span></font><font color="gray"><span style="color:gray"> </span></font><font color="#e6e5e2"><span style="color:#E6E5E2">-</span></font><b><font color="#cc8c5b"><span style="color:#CC8C5B; font-weight:bold">FI_ENOSYS</span></font></b><font color="black"><span style="color:black">;</span></font></pre>
<pre><font size="2" color="black" face="Courier New"><span style="font-size:10.0pt; color:black">}</span></font></pre>
<pre style="margin-bottom:12.0pt"><font size="2" color="black" face="Courier New"><span style="font-size:10.0pt; color:black"> </span></font></pre>
<p class="x_MsoNormal"><font size="3" color="black" face="Calibri"><span style="font-size:12.0pt; color:black"> </span></font></p>
</div>
<div class="x_MsoNormal" align="center" style="text-align:center"><font size="2" face="Calibri"><span style="font-size:11.0pt">
<hr size="2" width="98%" align="center">
</span></font></div>
<div id="x_x_divRplyFwdMsg">
<p class="x_MsoNormal" style=""><b><font size="2" color="black" face="Calibri"><span style="font-size:11.0pt; color:black; font-weight:bold">From:</span></font></b><font color="black"><span style="color:black"> Hefty, Sean <<a href="mailto:sean.hefty@intel.com">sean.hefty@intel.com</a>><br>
<b><span style="font-weight:bold">Sent:</span></b> 23 March 2021 00:08:38<br>
<b><span style="font-weight:bold">To:</span></b> Biddiscombe, John A.; <a href="mailto:libfabric-users@lists.openfabrics.org">
libfabric-users@lists.openfabrics.org</a><br>
<b><span style="font-weight:bold">Subject:</span></b> RE: Queue size question</span></font><span style="">
</span></p>
<div>
<p class="x_MsoNormal"><font size="2" face="Calibri"><span style="font-size:11.0pt"> </span></font></p>
</div>
</div>
</div>
<div>
<p class="x_MsoNormal"><font size="2" face="Calibri"><span style="font-size:10.0pt">> I have a test that seems to run fine on
<span class="x_SpellE">tcp;ofi_rxm</span> - though this test is two ranks on<br>
> the same laptop, so it isn't really a very good test - however, I can throw anything at<br>
> it and it seems to reliably complete.<br>
> <br>
> On GNI, I get lockups and after much head scratching, I am wondering what the<br>
> significance of the <span class="x_SpellE">tx</span>/<span class="x_SpellE">rx</span> attribute size may be.<br>
> <br>
> On <span class="x_SpellE">tcp</span>/<span class="x_SpellE">ofi_rxm</span> the size reports as "size: 65536" and I can have 16 threads each sending<br>
> up to 128 messages in flight on one thread per endpoint, and a single receive endpoint<br>
> handling all receives - possibly 16*128 messages with posted receives = 2048.<br>
> <br>
> When I run on GNI, using two nodes, each reports <span class="x_SpellE">tx</span>/<span class="x_SpellE">rx</span>
<span class="x_SpellE">attr</span> "size: 500" - and I find<br>
> that when many messages are in flight, things can lock up because some posted sends are<br>
> never received. This seems to happen even when I drop down to 16 threads with 8 in<br>
> flight messages which ought to be 128 at a time - and I would have suspected that a<br>
> size of 500 (<span class="x_SpellE">cq</span> size limitation?) would handle this.<br>
> <br>
> Question 1 - what is the <span class="x_SpellE">tx</span>/<span class="x_SpellE">rx</span>
<span class="x_SpellE">attr</span> size really telling me?<br>
<br>
Unfortunately, this is provider dependent, and there's very little that can be done to define it crisper without forcing an implementation. In some cases it's related to a HW queue size. I suspect that may be the case with
<span class="x_SpellE">gni</span>. However, the HW queue size doesn't necessarily mean that the number is equal to the number of operations that can be queue. For example, it's possible for a send that requires 2
<span class="x_SpellE">iovecs</span> to consume 2 entries in the queue. But each operation consuming 1 entry is usually a safe assumption.
<br>
<br>
Someone familiar with <span class="x_SpellE">gni</span> will need to chime in on how it maps to their HW.<br>
<br>
> Question 2 - if I post more than the allowed receives or sends, should I not receive<br>
> some kind of error? (I have enabled resource management, so I might expect a retry code<br>
> when I attempt the send/<span class="x_SpellE">recv</span>)<br>
<br>
Yes, you should see -FI_EAGAIN when trying to post more operations that the queues support. There are checks like this in some providers -- I think
<span class="x_SpellE">rxm</span>, verbs, and <span class="x_SpellE">tcp</span> all do, and
<span class="x_SpellE">rxm</span> is actually forgiving about it by allowing queues to overflow. (Because it's easy to swamp a receiver, even with a reasonably well-written app.)<br>
<br>
> Ideally, I'd like to throttle the number of messages in flight according to what the<br>
> hardware reports its capabilities - which vars should I use from the <span class="x_SpellE">
fi_info</span> to do<br>
> this?<br>
<br>
Resource management is the correct setting. Manually limited your application to the
<span class="x_SpellE">tx</span>/<span class="x_SpellE">rx</span> sizes, and sizing the CQ appropriately should have done the trick.<br>
<br>
It sounds like this is a problem likely restricted to <span class="x_SpellE">gni</span>.<br>
<br>
- Sean</span></font></p>
</div>
</div>
</div>
</div>
</body>
</html>