<html>
<head>
<meta http-equiv="Content-Type" content="text/html; charset=us-ascii">
<style type="text/css" style="display:none;"> P {margin-top:0;margin-bottom:0;} </style>
</head>
<body dir="ltr">
<div style="font-family: Calibri, Arial, Helvetica, sans-serif; font-size: 12pt; color: rgb(0, 0, 0);">
Following up my own question, I just realized that the fi_atomic(3) man page is explicit about this: fetching atomic ops always use FI_DELIVER_COMPLETE for their results, no matter what. So my question really only applies to RMA reads.</div>
<div style="font-family: Calibri, Arial, Helvetica, sans-serif; font-size: 12pt; color: rgb(0, 0, 0);">
<br>
</div>
<div style="font-family: Calibri, Arial, Helvetica, sans-serif; font-size: 12pt; color: rgb(0, 0, 0);">
greg</div>
<div style="font-family: Calibri, Arial, Helvetica, sans-serif; font-size: 12pt; color: rgb(0, 0, 0);">
<br>
</div>
<div id="appendonsend"></div>
<hr style="display:inline-block;width:98%" tabindex="-1">
<div id="divRplyFwdMsg" dir="ltr"><font face="Calibri, sans-serif" style="font-size:11pt" color="#000000"><b>From:</b> Libfabric-users <libfabric-users-bounces@lists.openfabrics.org> on behalf of Titus, Greg <gregory.titus@hpe.com><br>
<b>Sent:</b> Monday, February 10, 2020 1:08 PM<br>
<b>To:</b> libfabric-users@lists.openfabrics.org <libfabric-users@lists.openfabrics.org><br>
<b>Subject:</b> [libfabric-users] question about "default" FI_DELIVERY_COMPLETE semantics for RMA reads and fetching atomics</font>
<div> </div>
</div>
<style type="text/css" style="display:none">
<!--
p
{margin-top:0;
margin-bottom:0}
-->
</style>
<div dir="ltr">
<div style="font-family:Calibri,Arial,Helvetica,sans-serif; font-size:12pt; color:rgb(0,0,0)">
Greetings --</div>
<div style="font-family:Calibri,Arial,Helvetica,sans-serif; font-size:12pt; color:rgb(0,0,0)">
<br>
</div>
<div style="font-family:Calibri,Arial,Helvetica,sans-serif; font-size:12pt; color:rgb(0,0,0)">
The fi_cq(3) man page says that</div>
<div style="">
<blockquote style="color:rgb(0,0,0); font-family:Calibri,Arial,Helvetica,sans-serif; font-size:12pt; margin:0px 0px 0px 40px; border:none; padding:0px">
<div style="font-family:Calibri,Arial,Helvetica,sans-serif; font-size:12pt; color:rgb(0,0,0)">
<span>[FI_DELIVERY_COMPLETE] applies only to reliable endpoints. For opera</span>tions that return data to the initiator, such as RMA read or atomic-fetch, the source endpoint is also considered a destination endpoint. This is the default completion mode for
such operations.</div>
</blockquote>
<div style="font-family:Calibri,Arial,Helvetica,sans-serif; font-size:12pt; color:rgb(0,0,0)">
<br>
</div>
<div style="font-family:Calibri,Arial,Helvetica,sans-serif; font-size:12pt; color:rgb(0,0,0)">
Does "default completion mode" in that last sentence really mean "default", in that it's the completion mode that will be used if and only if no FI_{INJECT,TRANSMIT,DELIVERY,...}_COMPLETE flag is present in the hints to fi_getinfo() and/or the flags to fi_readmsg()
etc.? Or does it really mean that delivery_complete is guaranteed for transactions that produce local results, such that even if for example FI_TRANSMIT_COMPLETE is given in the hints, RMA reads and fetching atomics will nevertheless effectively use delivery_complete?</div>
<div style="font-family:Calibri,Arial,Helvetica,sans-serif; font-size:12pt; color:rgb(0,0,0)">
<br>
</div>
<div style="font-family:Calibri,Arial,Helvetica,sans-serif; font-size:12pt; color:rgb(0,0,0)">
(I'm pretty sure it's the former, but thought I should check.)</div>
<div style="font-family:Calibri,Arial,Helvetica,sans-serif; font-size:12pt; color:rgb(0,0,0)">
<br>
</div>
<div style="font-family:Calibri,Arial,Helvetica,sans-serif; font-size:12pt; color:rgb(0,0,0)">
If it is indeed the former, when an <span style="font-family:Calibri,Arial,Helvetica,sans-serif; background-color:rgb(255,255,255); display:inline!important">RMA read or fetching atomic is done and </span>FI_TRANSMIT_COMPLETE is in force, is there any way to
tell when the data has arrived?</div>
<div style="font-family:Calibri,Arial,Helvetica,sans-serif; font-size:12pt; color:rgb(0,0,0)">
<br>
</div>
<div style="font-family:Calibri,Arial,Helvetica,sans-serif; font-size:12pt; color:rgb(0,0,0)">
thanks,</div>
<div style="font-family:Calibri,Arial,Helvetica,sans-serif; font-size:12pt; color:rgb(0,0,0)">
greg</div>
</div>
</div>
</body>
</html>