<html>
<head>
<meta http-equiv="Content-Type" content="text/html; charset=Windows-1252">
<meta name="Generator" content="Microsoft Exchange Server">
<!-- converted from text --><style><!-- .EmailQuote { margin-left: 1pt; padding-left: 4pt; border-left: #800000 2px solid; } --></style>
</head>
<body>
<style type="text/css" style="display:none;"><!-- P {margin-top:0;margin-bottom:0;} --></style>
<div id="divtagdefaultwrapper" style="font-size:12pt;color:#000000;font-family:Calibri,Helvetica,sans-serif;" dir="ltr">
<p>FYI : If I use FI_RECV | FI_TRANSMIT then the endpoint enables (I still don't receive all messages - but that seems to be a different problem)<br>
</p>
</div>
<hr style="display:inline-block;width:98%" tabindex="-1">
<div id="divRplyFwdMsg" dir="ltr"><font face="Calibri, sans-serif" style="font-size:11pt" color="#000000"><b>From:</b> Libfabric-users <libfabric-users-bounces@lists.openfabrics.org> on behalf of Biddiscombe, John A. <john.biddiscombe@cscs.ch><br>
<b>Sent:</b> 12 February 2021 13:05:51<br>
<b>To:</b> Hefty, Sean; libfabric-users@lists.openfabrics.org<br>
<b>Subject:</b> Re: [libfabric-users] Not receiving messages from other ranks</font>
<div> </div>
</div>
<div>
<meta content="text/html; charset=UTF-8">
<style type="text/css" style="">
<!--
p
{margin-top:0;
margin-bottom:0}
-->
</style>
<div dir="ltr">
<div id="x_divtagdefaultwrapper" dir="ltr" style="font-size:12pt; color:#000000; font-family:Calibri,Helvetica,sans-serif">
<p>After more debugging (and switching to tcp;ofi_rxm since sockets doesn't seem to work), I am left with the following</p>
<p><br>
</p>
<p></p>
<div> // create a completion queue for tx <br>
fabric_info_->tx_attr->op_flags |= FI_COMPLETION;<br>
txcq_ = create_completion_queue(fabric_domain_, fabric_info_->tx_attr->size);<br>
#ifdef SEPARATE_RX_TX_ENDPOINTS<br>
// setup an endpoint for sending messages<br>
ep_tx_ = new_endpoint_active(fabric_domain_, fabric_info_, nullptr);<br>
bind_queue_to_endpoint(ep_tx_, txcq_, FI_TRANSMIT);<br>
bind_address_vector_to_endpoint(ep_tx_, av_);<br>
enable_endpoint(ep_tx_);<br>
#else<br>
bind_queue_to_endpoint(ep_rx_, txcq_, FI_TRANSMIT);<br>
#endif<br>
<br>
When the ifdef is not defined, I bind the txcq_ to the rx endpoint and ask for TRANSMIT completions, everything works fine.</div>
<div><br>
</div>
<div>When the ifdef is defined, then I fi_enable gives</div>
<div><br>
</div>
<div><span> ERROR fi_enable : Missing or unavailable completion queue</span><br>
</div>
<div><br>
</div>
<div>and I just don't know what's wrong. I'm creating a cq, binding it along with the AV, but it gives me the error.</div>
<div><br>
</div>
<div>Any ideas why this might happen? Hopefully by the time the USA wakes up, I'll have found the problem... (fingers crossed)</div>
<div><br>
</div>
<div>Thanks for any suggestions<br>
</div>
<div><br>
</div>
<div>JB<br>
</div>
<p></p>
</div>
<hr tabindex="-1" style="display:inline-block; width:98%">
<div id="x_divRplyFwdMsg" dir="ltr"><font face="Calibri, sans-serif" color="#000000" style="font-size:11pt"><b>From:</b> Hefty, Sean <sean.hefty@intel.com><br>
<b>Sent:</b> 10 February 2021 21:48:44<br>
<b>To:</b> Biddiscombe, John A.; libfabric-users@lists.openfabrics.org<br>
<b>Subject:</b> RE: Not receiving messages from other ranks</font>
<div> </div>
</div>
</div>
<font size="2"><span style="font-size:10pt;">
<div class="PlainText">> Which provider are you using? You may need to call cq read, even for the send side, to<br>
> ensure progress is being driven. If you're using rxm, I believe there's an environment<br>
> variable you can set to force auto-progress. ("fi_info -g rxm" might help discover the<br>
> name.).<br>
> <<br>
> I poll on the send endpoint tx CQ and I poll on the recv endpoint rx CQ - do you mean<br>
> that I should also create a dummy rx CQ on the send endpoint and poll that too be sure?<br>
<br>
I mean read the tx and rx CQs, which it sounds like you are doing.<br>
<br>
- Sean<br>
<br>
</div>
</span></font></div>
</body>
</html>