<html>
<head>
<meta http-equiv="Content-Type" content="text/html; charset=utf-8">
</head>
<body style="word-wrap: break-word; -webkit-nbsp-mode: space; line-break: after-white-space;" class="">
<div class="">Hi All,</div>
<div class=""><br class="">
</div>
I have some test code that depends on FI_REMOTE_CQ_DATA which I’ve debugged using the UDP;ofi_rxd provider.
<div class=""><br class="">
</div>
<div class="">I am trying to run that code using the gni provider on an XC40 but I’m not ever seeing the remote CQ events there. Is there something special I need to set up to get remote CQ events with gni?
<div class=""><br class="">
</div>
<div class="">I request:</div>
<div class=""><br class="">
</div>
<div class="">
<blockquote type="cite" class="">fi_info *hints = fi_allocinfo();<br class="">
hints->caps = FI_RMA | FI_REMOTE_WRITE | FI_RMA_EVENT;<br class="">
hints->mode = FI_CONTEXT | FI_CONTEXT2;<br class="">
hints->domain_attr->mr_mode = FI_MR_BASIC;<br class="">
hints->ep_attr->type = FI_EP_RDM;<br class="">
hints->tx_attr->msg_order = FI_ORDER_WAW | FI_ORDER_RMA_WAW;<br class="">
hints->rx_attr->msg_order = FI_ORDER_WAW | FI_ORDER_RMA_WAW;<br class="">
hints->rx_attr->caps = FI_RMA | FI_REMOTE_WRITE | FI_RMA_EVENT;</blockquote>
<br class="">
</div>
<div class="">And I successfully receive:</div>
<div class=""><br class="">
</div>
<div class=""></div>
<blockquote type="cite" class="">
<div class="">0: # Provider Fabric Domain Version EP_TYPE Protocol<br class="">
0: # gni gni /sys/class/gni/kgni0 1.1 FI_EP_RDM FI_EP_RDM<br class="">
0: # gni;ofi_rxd gni /sys/class/gni/kgni0 111.0 FI_EP_RDM FI_EP_RDM</div>
</blockquote>
<blockquote type="cite" class="">...<br class="">
</blockquote>
<blockquote type="cite" class="">
<div class="">1: # Provider Fabric Domain Version EP_TYPE Protocol<br class="">
1: # gni gni /sys/class/gni/kgni0 1.1 FI_EP_RDM FI_EP_RDM<br class="">
1: # gni;ofi_rxd gni /sys/class/gni/kgni0 111.0 FI_EP_RDM FI_EP_RDM</div>
</blockquote>
<br class="">
<div class="">I move through sequence of initialization calls that seem to be standard from what I can tell, resulting in an endpoint that is enabled successfully.</div>
<div class=""><br class="">
</div>
<div class="">
<blockquote type="cite" class="">static fi_context ep_ctx[2];<br class="">
check(fi_endpoint, domain, info, &ep, ep_ctx);<br class="">
check(fi_ep_bind, ep, &tx->fid, FI_TRANSMIT | FI_SELECTIVE_COMPLETION);<br class="">
check(fi_ep_bind, ep, &rx->fid, FI_RECV);<br class="">
check(fi_ep_bind, ep, &av->fid, 0);<br class="">
check(fi_enable, ep);</blockquote>
<br class="">
</div>
<div class="">Messages are sent with fi_writemsg and FI_REMOTE_CQ_DATA, and neither fail nor signal <span style="caret-color: rgb(0, 0, 0); color: rgb(0, 0, 0);" class="">FI_EAGAIN (this is a little alarming as I have tx/rx size of 500 and I send more than
that through the endpoint, I guess they just vanish into the ether).</span></div>
<div class=""><br class="">
<blockquote type="cite" class="">int e = fi_writemsg(ep, &msg, FI_REMOTE_CQ_DATA);<br class="">
<br class="">
if (likely(!e)) {<br class="">
return true;<br class="">
}<br class="">
<br class="">
if (likely(e == -FI_EAGAIN)) {<br class="">
return false;<br class="">
}<br class="">
<br class="">
fmt::print(stderr, "[{}] has unhandled tx error {}: {}\n", mpi::rank(), -e, fi_strerror(-e));<br class="">
</blockquote>
</div>
<div class=""><br class="">
</div>
<div class="">Unfortunately I never see any completions on the target rank (unlike <span style="caret-color: rgb(0, 0, 0); color: rgb(0, 0, 0);" class="">UDP;ofi_rxd where things are fine).</span></div>
<div class=""><span style="caret-color: rgb(0, 0, 0); color: rgb(0, 0, 0);" class=""><br class="">
</span></div>
<div class=""><font color="#000000" class="">Is there some magic that I need with gni to make</font> <span style="caret-color: rgb(0, 0, 0); color: rgb(0, 0, 0);" class="">FI_REMOTE_CQ_DATA</span><span style="color: rgb(0, 0, 0);" class=""> work?</span></div>
<div class=""><font color="#000000" class=""><br class="">
</font></div>
<div class=""><font color="#000000" class=""><span style="caret-color: rgb(0, 0, 0);" class="">Thanks,</span></font></div>
<div class=""><font color="#000000" class=""><span style="caret-color: rgb(0, 0, 0);" class="">Luke</span></font></div>
</div>
</body>
</html>