<html>
<head>
<meta http-equiv="Content-Type" content="text/html; charset=us-ascii">
</head>
<body style="word-wrap: break-word; -webkit-nbsp-mode: space; line-break: after-white-space;" class="">
Dear Libfabric Users & Developers,
<div class=""><br class="">
</div>
<div class="">using the verbs provider I use the FI_WAIT_FD wait object for the completion queue (CQ).</div>
<div class="">The resulting file descriptor (FD), associated to the <font face="Menlo" class="">
libibverbs</font> completion channel, is added to an <font face="Menlo" class="">
epoll</font> on waiting for EPOLLIN<font face="Menlo" class="">.</font></div>
<div class="">The FD signal triggers the callback where <font face="Menlo" class="">fi_cq_read</font> reads the completions in a non-blocking way.</div>
<div class=""><br class="">
</div>
<div class="">The problem is that, as soon as a connection is established, the FD keeps signalling as fast as the CPU allows for, even if there is no data transfer.</div>
<div class="">As a matter of fact, after the first call, <font face="Menlo" class="">fi_cq_read
</font>keeps returning EAGAIN, sign there there is no CQ entry to read.</div>
<div class=""><br class="">
</div>
<div class=""><u class="">I would expect the fd not to signal if there's nothing to read in the CQ. Shouldn't be this the case?</u></div>
<div class=""> </div>
<div class="">I read some <font face="Menlo" class="">libibverbs</font> documentation and this point still remains unclear to me. </div>
<div class=""><br class="">
</div>
<div class="">Below I post the CQ attributes and some function calls.</div>
<div class=""><br class="">
</div>
<div class="">Thank you very much for your help,</div>
<div class="">Carlo</div>
<div class=""><br class="">
</div>
<div class=""><br class="">
</div>
<div class="">
<div style="word-wrap: break-word; -webkit-nbsp-mode: space; line-break: after-white-space;" class="">
<span style="font-family: Times;" class="">_____________________</span></div>
<div style="word-wrap: break-word; -webkit-nbsp-mode: space; line-break: after-white-space;" class="">
<span style="font-family: Times;" class="">Carlo A. Gottardo</span></div>
<div style="word-wrap: break-word; -webkit-nbsp-mode: space; line-break: after-white-space;" class="">
<span style="font-family: Times;" class="">Postdoc at Nikhef</span></div>
<div style="word-wrap: break-word; -webkit-nbsp-mode: space; line-break: after-white-space;" class="">
<span style="font-family: Times;" class="">Skype: carlogottardo</span></div>
</div>
<div class=""><br class="">
</div>
<div class=""><br class="">
</div>
<div class=""><br class="">
</div>
<div class=""><i class="">System: Libfabric 1.12.1 / Centos7 / Mellanox Connect-X5</i></div>
<div class=""><br class="">
</div>
<div class="">CQ attributes</div>
<div class=""><br class="">
</div>
<div class=""><font face="Menlo" class=""><span class="Apple-tab-span" style="white-space:pre"></span>struct fi_cq_attr cq_attr;<br class="">
<span class="Apple-tab-span" style="white-space:pre"></span>cq_attr.size = MAX_CQ_ENTRIES;<br class="">
<span class="Apple-tab-span" style="white-space:pre"></span>cq_attr.flags = 0;<br class="">
<span class="Apple-tab-span" style="white-space:pre"></span>cq_attr.format = FI_CQ_FORMAT_DATA;<br class="">
<span class="Apple-tab-span" style="white-space:pre"></span>cq_attr.wait_obj= FI_WAIT_FD; <br class="">
<span class="Apple-tab-span" style="white-space:pre"></span>cq_attr.signaling_vector = 0;<br class="">
<span class="Apple-tab-span" style="white-space:pre"></span>cq_attr.wait_cond = FI_CQ_COND_NONE;<br class="">
<span class="Apple-tab-span" style="white-space:pre"></span>cq_attr.wait_set = NULL;</font></div>
<div class=""><br class="">
</div>
<div class="">the queue is open and bonded with </div>
<div class=""><br class="">
</div>
<div class=""><code class=""><span class="n"><span class="Apple-tab-span" style="white-space:pre"></span>fi_cq_open</span><span class="p">(</span><span class="n">rsocket</span><span class="o">-></span><span class="n">domain</span><span class="p">,</span>
<span class="o">&</span><span class="n">cq_attr</span><span class="p">,</span> <span class="o">
&</span><span class="n">rsocket</span><span class="o">-></span><span class="n">cq</span><span class="p">,</span>
<span class="nb">NULL</span><span class="p">)))</span></code></div>
<div class=""><code class=""><span class="p"><code class=""><span class="n"><span class="Apple-tab-span" style="white-space:pre"></span>fi_ep_bind</span><span class="p">((</span><span class="n">rsocket</span><span class="o">-></span><span class="n">ep</span><span class="p">),</span>
<span class="o">&</span><span class="n">rsocket</span><span class="o">-></span><span class="n">cq</span><span class="o">-></span><span class="n">fid</span><span class="p">,</span>
<span class="n">FI_TRANSMIT</span><span class="o">|</span><span class="n">FI_RECV</span><span class="p">)))</span></code></span></code></div>
<div class=""><code class=""><span class="p"><code class=""><span class="p"><br class="">
</span></code></span></code></div>
<div class="">when the connection is established the wait object is retrieved with</div>
<div class=""><font face="monospace" class=""><br class="">
</font></div>
<div class=""><code class=""><span class="n"><span class="Apple-tab-span" style="white-space:pre"></span>fi_control</span><span class="p">(</span><span class="o">&</span><span class="n">socket</span><span class="o">-></span><span class="n">cq</span><span class="o">-></span><span class="n">fid</span><span class="p">,</span>
<span class="n">FI_GETWAIT</span><span class="p">,</span> <span class="o">&</span><span class="n">socket</span><span class="o">-></span><span class="n">cqfd</span><span class="p">)</span></code></div>
<div class=""><code class=""><span class="p"><br class="">
</span></code></div>
<div class="">the file descriptor is assigned a callback </div>
<div class="">
<pre class="highlight code" lang="c"><span id="LC421" class="line" lang="c"> <span class="n">socket</span><span class="o">-></span><span class="n">cq_ev_ctx</span><span class="p">.</span><span class="n">fd</span> <span class="o">=</span> <span class="n">socket</span><span class="o">-></span><span class="n">cqfd</span><span class="p">;</span></span>
<span id="LC422" class="line" lang="c"> <span class="n">socket</span><span class="o">-></span><span class="n">cq_ev_ctx</span><span class="p">.</span><span class="n">data</span> <span class="o">=</span> <span class="n">socket</span><span class="p">;</span></span>
<span id="LC423" class="line" lang="c"> <span class="n">socket</span><span class="o">-></span><span class="n">cq_ev_ctx</span><span class="p">.</span><span class="n">cb</span> <span class="o">=</span> <span class="n">on_recv_socket_cq_event</span><span class="p">;</span></span>
<span id="LC424" class="line" lang="c"></span><br class=""></pre>
<pre class="highlight code" lang="c"><font face="Helvetica" class="">and finally the file descriptor is added to the the main and only epoll of the application, which waits for EPOLLIN.</font></pre>
<pre class="highlight code" lang="c"><font face="Helvetica" class="">In the </font>on_recv_socket_cq_event <font face="Helvetica" class="">callback</font>:</pre>
<pre class="highlight code" lang="c"><font face="Helvetica" class=""><span class="Apple-tab-span" style="white-space:pre"> </span></font><span class="n">fi_cq_read</span><span class="p">(</span><span class="n">socket</span><span class="o">-></span><span class="n">cq</span><span class="p">,</span> <span class="o">&</span><span class="n">completion_entries</span><span class="p">,</span> N<span class="p">);</span></pre>
<pre class="highlight code" lang="c"><br class=""></pre>
</div>
<div class="">
<div class="">
<div dir="auto" style="text-align: start; text-indent: 0px; word-wrap: break-word; -webkit-nbsp-mode: space; line-break: after-white-space;" class="">
<br class="Apple-interchange-newline">
</div>
<br class="Apple-interchange-newline">
<br class="Apple-interchange-newline">
</div>
<br class="">
</div>
</body>
</html>