<html>
<head>
<meta http-equiv="Content-Type" content="text/html; charset=us-ascii">
</head>
<body style="word-wrap: break-word; -webkit-nbsp-mode: space; line-break: after-white-space;" class="">
Dear Sean,
<div class=""><br class="">
</div>
<div class="">thank you for your previous message. <br class="">
<div class=""><br class="">
</div>
<div class="">I followed your advice and I created a structure that keeps track of all the completion queues and their fid.</div>
<div class="">Then I wrapped <font face="Menlo" class="">epoll_wait</font> in <font face="Menlo" class="">
fi_trywait</font></div>
<div class=""><br class="">
</div>
<div class=""><font face="Menlo" class=""><span class="Apple-tab-span" style="white-space:pre"></span>while(1){</font></div>
<div class=""><font face="Menlo" class=""><span class="Apple-tab-span" style="white-space:pre"></span>if( fi_trywait ( evloop->pfids.fabric, evloop->pfids.fid_set, evloop->pfids.count ) == FI_SUCCESS){</font></div>
<div class=""><font face="Menlo" class=""><br class="">
<span class="Apple-tab-span" style="white-space:pre"></span>nevents = epoll_wait(evloop->epollfd, evloop->events, MAX_EPOLL_EVENTS, EPOLL_TIMEOUT);<br class="">
<span class="Apple-tab-span" style="white-space:pre"></span>}</font></div>
<div class=""><font face="Menlo" class=""><span class="Apple-tab-span" style="white-space:pre"></span>...</font></div>
<div class=""><font face="Menlo" class=""><span class="Apple-tab-span" style="white-space:pre"></span>for(int i=0; i<nevents; i++){ process_event(...</font></div>
<div class=""><font face="Menlo" class=""><span class="Apple-tab-span" style="white-space:pre"></span>}</font></div>
<div class=""><font face="Menlo" class=""><br class="">
</font>
<div class="">however <font face="Menlo" class="">fi_trywait</font> calls, via <font face="Menlo" class="">
vrb_cq_trywait</font>, <font face="Menlo" class="">ibv_get_cq_event</font> which seems to be blocking (i.e. killing the program with GDB I see that the program is stuck there).</div>
<div class="">Following [1], I tried to make non-blocking by setting the O_NONBLOCK flag to the FD prior to adding it to
<font face="Menlo" class="">epoll </font>but this had no effect.</div>
<div class=""><br class="">
</div>
<div class="">My application is based on a single event loop (the epoll) running on a single thread and executing non-blocking functions so I wouldn't expect polling to happen elsewhere.</div>
<div class=""><br class="">
</div>
<div class="">Is there something wrong in what I am doing?</div>
<div class=""><br class="">
</div>
<div class="">Thank you very much for your help,</div>
<div class="">Carlo</div>
<div class=""> </div>
<div class=""><br class="">
</div>
<div class="">[1] <a href="https://www.rdmamojo.com/2013/03/09/ibv_get_cq_event/" class="">https://www.rdmamojo.com/2013/03/09/ibv_get_cq_event/</a></div>
<div class=""><br class="">
</div>
<div class="">
<div dir="auto" style="caret-color: rgb(0, 0, 0); color: rgb(0, 0, 0); letter-spacing: normal; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; word-spacing: 0px; -webkit-text-stroke-width: 0px; text-decoration: none; word-wrap: break-word; -webkit-nbsp-mode: space; line-break: after-white-space;" class="">
<div dir="auto" style="caret-color: rgb(0, 0, 0); color: rgb(0, 0, 0); letter-spacing: normal; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; word-spacing: 0px; -webkit-text-stroke-width: 0px; text-decoration: none; word-wrap: break-word; -webkit-nbsp-mode: space; line-break: after-white-space;" class="">
<div dir="auto" style="caret-color: rgb(0, 0, 0); color: rgb(0, 0, 0); letter-spacing: normal; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; word-spacing: 0px; -webkit-text-stroke-width: 0px; text-decoration: none; word-wrap: break-word; -webkit-nbsp-mode: space; line-break: after-white-space;" class="">
<div dir="auto" style="caret-color: rgb(0, 0, 0); color: rgb(0, 0, 0); letter-spacing: normal; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; word-spacing: 0px; -webkit-text-stroke-width: 0px; text-decoration: none; word-wrap: break-word; -webkit-nbsp-mode: space; line-break: after-white-space;" class="">
<div dir="auto" style="caret-color: rgb(0, 0, 0); color: rgb(0, 0, 0); letter-spacing: normal; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; word-spacing: 0px; -webkit-text-stroke-width: 0px; text-decoration: none; word-wrap: break-word; -webkit-nbsp-mode: space; line-break: after-white-space;" class="">
<div dir="auto" style="caret-color: rgb(0, 0, 0); color: rgb(0, 0, 0); letter-spacing: normal; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; word-spacing: 0px; -webkit-text-stroke-width: 0px; text-decoration: none; word-wrap: break-word; -webkit-nbsp-mode: space; line-break: after-white-space;" class="">
<div style="color: rgb(0, 0, 0); font-family: Helvetica; font-size: 12px; font-style: normal; font-variant-caps: normal; font-weight: normal; letter-spacing: normal; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; word-spacing: 0px; -webkit-text-stroke-width: 0px; word-wrap: break-word; -webkit-nbsp-mode: space; line-break: after-white-space;" class="">
<span style="font-family: Times;" class=""><br class="Apple-interchange-newline">
</span><span style="font-family: Times;" class="">_____________________</span></div>
<div style="color: rgb(0, 0, 0); font-family: Helvetica; font-size: 12px; font-style: normal; font-variant-caps: normal; font-weight: normal; letter-spacing: normal; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; word-spacing: 0px; -webkit-text-stroke-width: 0px; word-wrap: break-word; -webkit-nbsp-mode: space; line-break: after-white-space;" class="">
<span style="font-family: Times;" class="">Carlo A. Gottardo</span></div>
<div style="color: rgb(0, 0, 0); font-family: Helvetica; font-size: 12px; font-style: normal; font-variant-caps: normal; font-weight: normal; letter-spacing: normal; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; word-spacing: 0px; -webkit-text-stroke-width: 0px; word-wrap: break-word; -webkit-nbsp-mode: space; line-break: after-white-space;" class="">
<span style="font-family: Times;" class="">Postdoc at Nikhef</span></div>
</div>
</div>
</div>
</div>
</div>
<br class="Apple-interchange-newline">
</div>
<br class="Apple-interchange-newline">
<br class="Apple-interchange-newline">
</div>
<div><br class="">
<blockquote type="cite" class="">
<div class="">On 29 Jun 2021, at 18:53, Hefty, Sean <<a href="mailto:sean.hefty@intel.com" class="">sean.hefty@intel.com</a>> wrote:</div>
<br class="Apple-interchange-newline">
<div class="">
<div class="">Background:<br class="">
<br class="">
With verbs devices there are 2 queues in play here. The first is associated with the fd, where low-level events are in the kernel. This event is generated in response to an interrupt from the device. In order to limit how many interrupts the device generates,
the device must be manually reset before it will generate another event. The event indicates which CQ had entries added to it.<br class="">
<br class="">
The second queue contains the completion entries itself. That is what fi_cq_read is accessing.<br class="">
<br class="">
<br class="">
The reason the fd remains signaled is that the kernel event is never being read.<br class="">
<br class="">
If an application wants to wait directly on a wait object (fd) using OS specific calls (select/poll), it needs to call fi_trywait() prior to blocking.<br class="">
<br class="">
<a href="https://ofiwg.github.io/libfabric/v1.12.1/man/fi_poll.3.html" class="">https://ofiwg.github.io/libfabric/v1.12.1/man/fi_poll.3.html</a><br class="">
<br class="">
- Sean<br class="">
<br class="">
<br class="">
<blockquote type="cite" class="">using the verbs provider I use the FI_WAIT_FD wait object for the completion queue<br class="">
(CQ).<br class="">
The resulting file descriptor (FD), associated to the libibverbs completion channel, is<br class="">
added to an epoll on waiting for EPOLLIN.<br class="">
The FD signal triggers the callback where fi_cq_read reads the completions in a non-<br class="">
blocking way.<br class="">
<br class="">
The problem is that, as soon as a connection is established, the FD keeps signalling as<br class="">
fast as the CPU allows for, even if there is no data transfer.<br class="">
As a matter of fact, after the first call, fi_cq_read keeps returning EAGAIN, sign<br class="">
there there is no CQ entry to read.<br class="">
<br class="">
I would expect the fd not to signal if there's nothing to read in the CQ. Shouldn't be<br class="">
this the case?<br class="">
<br class="">
I read some libibverbs documentation and this point still remains unclear to me.<br class="">
<br class="">
Below I post the CQ attributes and some function calls.<br class="">
<br class="">
Thank you very much for your help,<br class="">
Carlo<br class="">
<br class="">
<br class="">
_____________________<br class="">
Carlo A. Gottardo<br class="">
Postdoc at Nikhef<br class="">
Skype: carlogottardo<br class="">
<br class="">
<br class="">
<br class="">
System: Libfabric 1.12.1 / Centos7 / Mellanox Connect-X5<br class="">
<br class="">
CQ attributes<br class="">
<br class="">
struct fi_cq_attr cq_attr;<br class="">
cq_attr.size = MAX_CQ_ENTRIES;<br class="">
cq_attr.flags = 0;<br class="">
cq_attr.format = FI_CQ_FORMAT_DATA;<br class="">
cq_attr.wait_obj= FI_WAIT_FD;<br class="">
cq_attr.signaling_vector = 0;<br class="">
cq_attr.wait_cond = FI_CQ_COND_NONE;<br class="">
cq_attr.wait_set = NULL;<br class="">
<br class="">
the queue is open and bonded with<br class="">
<br class="">
fi_cq_open(rsocket->domain, &cq_attr, &rsocket->cq, NULL)))<br class="">
fi_ep_bind((rsocket->ep), &rsocket->cq->fid, FI_TRANSMIT|FI_RECV)))<br class="">
<br class="">
<br class="">
when the connection is established the wait object is retrieved with<br class="">
<br class="">
<br class="">
fi_control(&socket->cq->fid, FI_GETWAIT, &socket->cqfd)<br class="">
<br class="">
<br class="">
the file descriptor is assigned a callback<br class="">
socket->cq_ev_ctx.fd = socket->cqfd;<br class="">
socket->cq_ev_ctx.data = socket;<br class="">
socket->cq_ev_ctx.cb = on_recv_socket_cq_event;<br class="">
<br class="">
and finally the file descriptor is added to the the main and only epoll of the<br class="">
application, which waits for EPOLLIN.<br class="">
In the on_recv_socket_cq_event callback:<br class="">
<span class="Apple-tab-span" style="white-space:pre"></span>fi_cq_read(socket->cq, &completion_entries, N);<br class="">
<br class="">
<br class="">
<br class="">
<br class="">
</blockquote>
<br class="">
</div>
</div>
</blockquote>
</div>
<br class="">
</div>
</div>
</body>
</html>